Method and device for caching massive data

Abstract

The present invention is applicable to the technical field of data processing, and provides a method and device for caching massive data. The method includes: converting the original data obtained in the database into data in a standardized matrix format, and converting the converted data into mat File format cache; when receiving the user's data request information, judge whether there is all the data corresponding to the data request information in the cached mat file; if not, use the statistical model algorithm to obtain the missing data from the database data; the missing data is converted into data in a standardized matrix format and filled into the mat file, and the data corresponding to the data request information in the filled mat file is fed back to the User; convert the missing data into data in a standardized matrix format and cache it in a mat file format. Through the invention, data storage redundancy can be greatly reduced, and the efficiency of data storage and reading can be improved.

Description

technical field

[0001] The invention belongs to the technical field of data processing, and in particular relates to a method and device for caching massive data. Background technique

[0002] With the advent of the big data era, the demand for processing and storage of massive data is increasing. At present, relational databases are mainly used to store and process massive data. However, since the relational database stores record-type data, it is easy to have a large amount of data redundancy, and it is very inconvenient to use data conversion in statistical analysis. Moreover, relational databases request different data and need to be associated with different tables, and the query interface is used separately, and the efficiency of data acquisition is low. In addition, although the relational database provides a memory-level cache function, the data will be cached in the server's memory when the data is requested, and the next time the same request will be much faster....

Examples

Embodiment 1

[0024] figure 1 The implementation flow of the massive data caching method provided by the first embodiment of the present invention is shown, and the process is described in detail as follows:

[0025] In step S101, the original data acquired in the database is converted into data in a standardized matrix format, and the converted data is cached in a mat file format.

[0026] In this embodiment, the data in the standardized matrix format is matlab standard matrix data. The mat file is a standard format for matlab data storage. mat files are standard binary files that can be saved and loaded in ASCII code.

[0027] Among them, converting the raw data obtained in the database into a standardized matrix format specifically includes:

[0028] Obtain data in the cell format from the database; specifically, obtain the original data from the database through multiple data adapters, the original data is obtained from the database to the matlab platform in a cell matrix format, eac...

Embodiment 2

[0055] figure 2 An example of the architecture of the massive data cache device provided by the second embodiment of the present invention is shown, as shown in figure 2 As shown, the architecture of the massive data cache device includes a database layer, an adapter layer, a data interface routing layer, a data extraction and cache scheduling layer, and a graphic user interaction layer from bottom to top. The specific applications of each layer are as follows:

[0056] 1) Database layer

[0057] The database layer mainly stores original data through the database, and extracts the original data through a series of stored procedures.

[0058] 2) Adapter layer

[0059] The adapter layer mainly extracts original data from the database, and converts the extracted original data into matlab standard matrix data. The adapter layer includes a plurality of data adapters, each of which handles a type of data.

[0060] The specific application of the adapter layer is as follows: fi...

Embodiment 3

[0073] image 3 The composition structure of the mass data cache device provided by the third embodiment of the present invention is shown, and for the convenience of description, only the parts related to the embodiment of the present invention are shown.

[0074] The mass data cache device can be a software unit, a hardware unit or a combination of software and hardware running in the terminal device, or it can be integrated into the terminal device as an independent pendant or run in the application system of the terminal device middle.

[0075] The massive data cache device includes a first cache unit 31 , a judgment unit 32 , a data acquisition unit 33 and a data filling unit 34 . Among them, the specific functions of each unit are as follows:

[0076] The first cache unit 31 is used to convert the raw data obtained in the database into data in a standardized matrix format, and cache the converted data in a mat file format;

[0077] A judging unit 32, configured to jud...

technical field

[0001] The invention belongs to the technical field of data processing, and in particular relates to a method and device for caching massive data. Background technique

[0002] With the advent of the big data era, the demand for processing and storage of massive data is increasing. At present, relational databases are mainly used to store and process massive data. However, since the relational database stores record-type data, it is easy to have a large amount of data redundancy, and it is very inconvenient to use data conversion in statistical analysis. Moreover, relational databases request different data and need to be associated with different tables, and the query interface is used separately, and the efficiency of data acquisition is low. In addition, although the relational database provides a memory-level cache function, the data will be cached in the server's memory when the data is requested, and the next time the same request will be much faster....

Examples

Embodiment 1

[0024] figure 1 The implementation flow of the massive data caching method provided by the first embodiment of the present invention is shown, and the process is described in detail as follows:

[0025] In step S101, the original data acquired in the database is converted into data in a standardized matrix format, and the converted data is cached in a mat file format.

[0026] In this embodiment, the data in the standardized matrix format is matlab standard matrix data. The mat file is a standard format for matlab data storage. mat files are standard binary files that can be saved and loaded in ASCII code.

[0027] Among them, converting the raw data obtained in the database into a standardized matrix format specifically includes:

[0028] Obtain data in the cell format from the database; specifically, obtain the original data from the database through multiple data adapters, the original data is obtained from the database to the matlab platform in a cell matrix format, eac...

Embodiment 2

[0055] figure 2 An example of the architecture of the massive data cache device provided by the second embodiment of the present invention is shown, as shown in figure 2 As shown, the architecture of the massive data cache device includes a database layer, an adapter layer, a data interface routing layer, a data extraction and cache scheduling layer, and a graphic user interaction layer from bottom to top. The specific applications of each layer are as follows:

[0056] 1) Database layer

[0057] The database layer mainly stores original data through the database, and extracts the original data through a series of stored procedures.

[0058] 2) Adapter layer

[0059] The adapter layer mainly extracts original data from the database, and converts the extracted original data into matlab standard matrix data. The adapter layer includes a plurality of data adapters, each of which handles a type of data.

[0060] The specific application of the adapter layer is as follows: fi...

Embodiment 3

[0073] image 3 The composition structure of the mass data cache device provided by the third embodiment of the present invention is shown, and for the convenience of description, only the parts related to the embodiment of the present invention are shown.

[0074] The mass data cache device can be a software unit, a hardware unit or a combination of software and hardware running in the terminal device, or it can be integrated into the terminal device as an independent pendant or run in the application system of the terminal device middle.

[0075] The massive data cache device includes a first cache unit 31 , a judgment unit 32 , a data acquisition unit 33 and a data filling unit 34 . Among them, the specific functions of each unit are as follows:

[0076] The first cache unit 31 is used to convert the raw data obtained in the database into data in a standardized matrix format, and cache the converted data in a mat file format;

[0077] A judging unit 32, configured to jud...

Images