Cloud storage service platform using data analysis

Abstract

The invention provides a cloud storage service platform utilizing data analysis, and relates to the technical field of cloud servers, a system structure in the service platform is composed of a data layer, a business layer and a presentation layer, MongoDB-Hadoop is adopted as an adapter in the system, a MongoDB database is adopted as a database of basic storage, and the data layer is connected with the presentation layer. The MongoDB-Hadoop adapter interacts with an external computer framework, storage is performed by registering a driver, acquiring a data name, acquiring data information and accessing data elements in the system, and different access interfaces are used for different users, so that the system is more flexible, a storage task can be completed more efficiently, the problem of data cloud storage can be solved, and the storage efficiency is improved. The MongoDB-based data-intensive cloud storage system has the advantages that the MongoDB-based data-intensive cloud storage system can realize data processing, so that the use requirements of users on intensive data and network service requirements are met, and compared with the existing cloud storage system, the MongoDB-based data-intensive cloud storage system has the advantage that the storage performance and the query performance are greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cloud servers, and in particular to a cloud storage service platform utilizing data analysis. Background Art

[0002] Data storage is one of the hot topics of current research. The more commonly used storage methods are unstructured storage mode and semi-structured storage mode. Currently used sensors, location services and social networks will apply a lot of semi-structured spatial data, as well as unstructured spatial data. There are two main implementation methods used by traditional data storage systems: spatially extending relational databases and adding spatial engines to the upper layer of the database. These two ideas can greatly improve the consistency and integrity issues of database storage.

[0003] The existing technology has the problem of massive data being difficult to store, access, and maintain in a complete manner using relational databases, and it is even more difficult to expand the database. Summary of the invention

[0004] Technical issues solved

[0005] In view of the deficiencies in the prior art, the present invention provides a cloud storage service platform using data analysis, which solves the problem that relational databases are difficult to fully implement storage, access, and maintenance, and even more difficult to implement database expansion.

[0006] Technical Solution

[0007] To achieve the above objectives, the present invention is implemented through the following technical solutions: a cloud storage service platform using data analysis, wherein the system structure inside the service platform consists of a data layer, a business layer, and a presentation layer, wherein the system uses MongoDB-Hadoop as an adapter, uses MongoDB database as a database for basic storage, and interacts with an external computer framework through the MongoDB-Hadoop adapter, wherein the system stores data by registering a driver, obtaining a data name, obtaining data information, and accessing data elements, wherein:

[0008] The data layer uniformly stores the user uploaded data in the MongoDB database, and uses the HDFS distributed file system to calculate the stored data. The data layer divides the stored data into two categories: shared data and private data;

[0009] The business layer is located on top of the data layer, data is accessed through the business layer, and the MongoDB database inside the business layer needs to be encapsulated;

[0010] The presentation layer is responsible for the display of the system, selects the B / S architecture as the overall architecture of the presentation layer, and stores, updates, and uploads various data in the browser.

[0011] Preferably, the data sets within the data layer are divided into two categories, the first category is responsible for storing spatial data, and the second category is responsible for storing cloud data.

[0012] Preferably, after the shared data in the data layer is stored in the unified database, a filter is started to maintain data security, and the private data in the data layer is stored in a separate database for the user, and permission control is started to maintain data security.

[0013] Preferably, all data modules in the system structure use MongoDB's Java program to complete data interaction and continuously mine and analyze data in their own algorithm library.

[0014] Preferably, the system uses an access interface to connect the data layer and the presentation layer together. The access interface can provide the necessary interface functions to the client. After obtaining the interface function command, all service programs complete deletion, update, modification, search and other tasks, and the results obtained must be fed back to the client in a timely manner.

[0015] Preferably, the business layer includes an input module, a download module, a search module, an update module, a processing module and a deletion module.

[0016] Preferably, the data layer records the spatial reference system, data information and user description through a metadata set to determine the relationship between the file and the database set.

[0017] Beneficial Effects

[0018] The present invention provides a cloud storage service platform using data analysis. It has the following beneficial effects:

[0019] The present invention adopts the system structure of the service platform composed of the data layer, the business layer and the presentation layer, and selects MongoDB-Hadoop as the adapter. The storage process is realized by registering the driver, obtaining the data name, obtaining the data information, and accessing the data elements. Different access interfaces are used for different users. The system is more flexible and can complete the storage task more efficiently. It can not only solve the data cloud storage problem, but also realize data processing, thereby meeting the user's use requirements and network service requirements for intensive data. Compared with the existing cloud storage system, the data-intensive cloud storage system based on MongoDB has greatly improved storage performance and query performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a system architecture diagram of the present invention;

[0021] Figure 2 It is a data layer storage structure diagram of the present invention;

[0022] Figure 3 The model diagram for importing and exporting disk information of the presentation layer of the present invention;

[0023] Figure 4 A data-intensive cloud storage system storage process of the present invention;

[0024] Figure 5 The batch computing workflow of the present invention;

[0025] Figure 6 This is a graph showing the comparative experimental results of the data batch storage performance of the present invention;

[0026] Figure 7 This is a comparison chart of the specific data query performance experiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Specific embodiment one:

[0029] like Figure 1-5 As shown, a cloud storage service platform using data analysis, the system structure inside the service platform consists of a data layer, a business layer and a presentation layer. The system uses MongoDB-Hadoop as an adapter, uses MongoDB database as the basic storage database, and interacts with the external computer framework through the MongoDB-Hadoop adapter. The system stores data by registering a driver, obtaining a data name, obtaining data information, and accessing data elements.

[0030] The data layer stores the user-uploaded data in a unified manner in the MongoDB database, and uses the HDFS distributed file system to calculate the stored data. The data layer divides the stored data into two categories: shared data and private data.

[0031] The business layer is located on top of the data layer. Data is accessed through the business layer, and the MongoDB database inside the business layer needs to be encapsulated.

[0032] The presentation layer is responsible for the display of the system. It selects the B / S architecture as the overall architecture of the presentation layer and stores, updates, and uploads various data in the browser.

[0033] The data collection within the data layer is divided into two categories. The first category is responsible for storing spatial data, and the second category is responsible for storing cloud data. Since the data layer will provide storage services to different users, the stored data must be classified into shared data and private data. After the shared data within the data layer is stored in a unified database, the filter is enabled to maintain data security. The private data within the data layer is stored in the user's separate database, and permission control is enabled to maintain data security.

[0034] Various different data are stored through spatial data sets. When storing data, users can either select an existing data set within the storage structure or create a new data set. To achieve better storage results, each spatial object will correspond to a json object. At the same time, when storing, "file" and "layer" attributes must be added to facilitate users to better determine the source file and the corresponding different layers to which the spatial object belongs when querying. The system data layer records the spatial reference system, data information and user description through metadata sets to determine the relationship between files and database sets. When a data set requires a new field, the user can directly add it to the uploaded document without the need for re-design. This not only reduces the number of times the data structure is used, but also prevents large data sets, avoids storage redundancy, and uses a sharded replica architecture to implement cluster architecture design.

[0035] The system uses an access interface to connect the data layer and the presentation layer together. The access interface can provide the necessary interface functions to the client. After obtaining the interface function command, all service programs complete deletion, update, modification, search and other tasks, and the results obtained must be fed back to the client in a timely manner.

[0036] The business layer encapsulates the MongoDB database to ensure the smooth operation of functions required by users in daily life. At the same time, in order to optimize the work efficiency of the system, the complex logical operations within the data storage layer are hidden, and the reading and writing functions and management processes of user data are simplified. The business layer includes input module, download module, search module, update module, processing module and deletion module. All data modules in the system structure use MongoDB's Java program to complete data interaction and continuously mine and analyze data in their own algorithm library. Specific embodiment 2:

[0038] like Figure 1-5 As shown, according to the system hardware design software, different data use different conversion interfaces and storage interfaces. The storage process is realized by registering the driver, obtaining the data name, obtaining the data information, and accessing the data elements. The system software storage process is as follows Figure 4 As shown,

[0039] The data group information is obtained through cloud computing. The formula is as follows:

[0040]

[0041] In the above formula, x represents the acquired data group information. represents the storage layer information, and ε represents the data layer information. Specific embodiment three:

[0043] like Figure 1-5 As shown in the figure, after the storage system obtains the relevant parameters in the MongoDB-Hadoop adapter, it will process the data in slices, transfer the obtained data to the mapper, process the data in MongoDB, define the obtained programming language by reading, continuously index and filter new data, and realize batch computing through cloud computing. The calculation process is shown below

[0044] 1) Use the MongoDB-Hadoop adapter to detect the obtained MongoDB data set and split all the obtained data;

[0045] 2) Split the obtained data and distribute it to different computing nodes;

[0046] 3) Split the data in the computing nodes in Hadoop, analyze the situation after segmentation, obtain the data in MongoDB, and implement local calculation in Mapper;

[0047] 4) Combine the calculations in Recuder and output the calculation results to MongoDB data, thus achieving batch calculations. Specific embodiment four:

[0049] like Figure 1-5 As shown in the figure, an operating system is constructed in a cluster environment composed of three physical machines. The system performance is determined by analyzing the difference between this system, that is, the cloud storage system based on MongoDB and the cloud storage system based on Sqlite. The application parameters of the experimental process are shown in Table 1.

[0050]

[0051] Table 1 Experimental parameters

[0052] The experimental results of data batch storage performance comparison are as follows: Figure 6 As shown, according to Figure 6It can be seen that the overall cluster efficiency of the data in this system is better than that of the traditional cloud storage system based on the relational database Sqlite. After multiple statistics, the storage performance of the system in this article is 2.8 times that of the traditional system. The cluster efficiency will not fluctuate significantly with the increase of experimental samples. It can be seen that this system has obvious advantages over the cloud storage system based on the relational database Sqlite. Even if the amount of data increases, the insertion efficiency of the system can be effectively guaranteed.

[0053] Analyze 1,000 data with specific performance, display the results based on data level information and row and column number information, and compare the query performance of different systems, such as Figure 7 As shown in the figure, when the amount of data is small, there is little difference between the traditional cloud storage system and the cloud storage system studied in this paper. However, as the amount of data increases, the query time spent by the cloud storage system based on the relational database Sqlite becomes longer and longer, and the upward slope is very obvious. In comparison, the upward slope of this system is relatively gentle. It can be seen that the query performance of this system is better.

[0054] Further analysis Figure 7 The experimental results show that as the amount of data increases, the performance of this system in data storage and retrieval is significantly better than that of the Sqlite-based cloud storage system, and it can adapt well to the cloud platform system. MongoDB has the ability to reduce the coupling between the client, storage service and each site. This system no longer stores data in an integrated structure, but logically separates shared data and private data, so that the storage service and client are no longer completely dependent on the site. Even if the site cannot work properly, the storage service and client can still perform development and storage functions normally. This system reduces the pressure on the service site, allowing the service site to process a large number of orders at the same time. Different storage points work at the same time, making the system's storage space richer and the system more flexible. When the database is working, the system can appropriately adjust the MongoDB cluster according to its own characteristics. In addition, MongoDB also has the advantage of good fault tolerance, which enhances the system performance as a whole.

[0055] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprising a reference structure" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0056] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A cloud storage service platform using data analysis, characterized in that: The system structure inside the service platform consists of a data layer, a business layer, and a presentation layer. The system uses MongoDB-Hadoop as an adapter, uses MongoDB database as the basic storage database, and interacts with the external computer framework through the MongoDB-Hadoop adapter. The system stores data by registering a driver, obtaining a data name, obtaining data information, and accessing data elements. The data layer uniformly stores the user uploaded data in the MongoDB database, and uses the HDFS distributed file system to calculate the stored data. The data layer divides the stored data into two categories: shared data and private data; The business layer is located on top of the data layer, data is accessed through the business layer, and the MongoDB database inside the business layer needs to be encapsulated; The presentation layer is responsible for the display of the system, selects the B / S architecture as the overall architecture of the presentation layer, and stores, updates, and uploads various data in the browser.

2. The cloud storage service platform using data analysis according to claim 1, characterized in that: The data collection within the data layer is divided into two categories, the first category is responsible for storing spatial data, and the second category is responsible for storing cloud data.

3. The cloud storage service platform using data analysis according to claim 1, characterized in that: After the shared data in the data layer is stored in the unified database, the filter is started to maintain data security, and the private data in the data layer is stored in the user's separate database, and the authority control is started to maintain data security.

4. The cloud storage service platform using data analysis according to claim 1, characterized in that: All data modules in the system structure use MongoDB's Java program to complete data interaction and continuously mine and analyze data in their own algorithm library.

5. The cloud storage service platform using data analysis according to claim 1, characterized in that: The system uses an access interface to connect the data layer and the presentation layer together. The access interface can provide the client with necessary interface functions. After obtaining the interface function command, all service programs complete deletion, update, modification, search and other tasks, and the results obtained must be fed back to the client in a timely manner.

6. The cloud storage service platform using data analysis according to claim 1, characterized in that: The business layer includes an input module, a download module, a search module, an update module, a processing module and a deletion module.

7. The cloud storage service platform using data analysis according to claim 1, characterized in that: The data layer records the spatial reference system, data information and user description through a metadata set, and determines the relationship between files and database sets.

Images