Secure storage system for internet data

Abstract

The invention discloses a secure storage system for internet data. The system comprises a data acquisition module, a data layering module, an encryption and decryption module and a database, wherein the data acquisition module is used for acquiring internet data generated in each network platform, and classifying the internet data according to data types; the data layering module is used for distributing storage spaces for different types of Internet data and grouping and merging the same type of Internet data; the encryption and decryption module is used for distributing corresponding secret keys and encryption algorithms for different data types of internet data; and the database is used for receiving and storing the encrypted internet data. Through the scheme, invalid data can be released, and the security of the data information is improved while the data information is accessed and deleted.

Description

Technical Field

[0001] The present invention belongs to the technical field of information security, and in particular relates to a secure storage system for Internet data. Background Art

[0002] With the rapid development of Internet technology, data information security issues are becoming increasingly serious. Although network platforms all have information encryption functions, some sensitive information is still easily leaked during storage, especially in the financial field. Any information leakage will affect the vital interests of users, affecting their property and information security. The data may include: name, ID card, email address, mobile phone number, bank card, or other sensitive information in other business areas. In addition, since a large amount of Internet data is stored in the server, including a large amount of invalid data, it takes up a lot of storage space.

[0003] Currently, some companies desensitize information on the upper system platform to address the issue of sensitive information security, which can protect user information security to a certain extent. In addition, sensitive information will be encrypting at the bottom data layer, but each external business system needs to perform encryption and decryption processing separately. Summary of the invention

[0004] The embodiment of the present invention provides a secure storage system for Internet data, which groups and merges data information and performs centralized encryption and decryption processing, thereby facilitating access to and deletion of data information and improving the security of data information.

[0005] The embodiment of the present invention provides a secure storage system for Internet data. The system is applied to the server side and includes: a data acquisition module, a data layering module, an encryption and decryption module, and a database; wherein:

[0006] The data acquisition module is used to obtain Internet data generated in various Internet platforms and classify the Internet data according to data types;

[0007] The data layering module is used to allocate storage space to different types of Internet data, and to group and merge Internet data of the same type;

[0008] The encryption and decryption module is used to allocate corresponding keys and encryption algorithms to Internet data in different data formats;

[0009] The database is used to receive and store encrypted Internet data.

[0010] Furthermore, the data types include: time series data and non-time series data; the data acquisition module organizes the time series data into a plurality of time series files, and organizes the non-time series data into a plurality of non-time series files;

[0011] After classifying the Internet data according to the data type, the data acquisition module records the access times N and the time interval T of the user accessing each time-series file and / or non-time-series file in real time, and sets a time interval threshold TS and an access times threshold NS; when T > TS, compare N with NS, if N < NS, mark the corresponding time-series file and / or non-time-series file as invalid data, and transfer it from the original storage space to the cloud storage.

[0012] Further, the data layering module is specifically used to store the time-series files in the first storage space and the non-time-series files in the second storage space.

[0013] Further, the data layering module is also specifically used to layer the first storage space, including a data layer and a buffer; divide the data layer into L1, L2, L3…LN levels according to the time range of the time-series files, and store the time-series files falling within the corresponding time range in the corresponding levels.

[0014] Further, it is characterized in that the ratio of the time range of the lower level to the time range of the upper level is set as a fixed value.

[0015] Further, the data layering module is also specifically used to layer the second storage space, including a data layer and a buffer; divide the data layer into M1, M2, M3…MN levels according to the size range of the non-time-series files, and store the non-time-series files falling within the corresponding size range in the corresponding levels.

[0016] Further, the encryption and decryption module is specifically used to set one or more different encryption and decryption algorithms for the Internet data of different data formats according to the data format of the Internet data, and provide the corresponding keys; the data formats include: structured data, unstructured data, and semi-structured data.

[0017] Further, a communication channel is allocated to the user terminal and the database in advance, a communication connection between the user terminal and the database is established, and the user terminal queries the database based on the established communication connection.

[0018] Further, the database has multiple storage sectors and is also used to store the encryption and decryption algorithms and the corresponding keys.

[0019] Compared with the prior art, the beneficial effects of the present invention are:

[0020] The embodiments of the present invention organize and manage time series data and non-time series data by grouping and merging them. While ensuring efficient data writing, it can perform efficient retrieval and analysis based on the time range of time series data or the size of non-time series data. At the same time, invalid data is marked according to the number of accesses and time intervals, and transferred to the cloud to alleviate the storage pressure of the server. Finally, different encryption and decryption algorithms are set for data in different formats, and information is centrally encrypted and decrypted for storage to improve information security.

[0021] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 It is a structural diagram of a secure storage system for Internet data provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0024] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0025] Embodiment 1

[0026] The present invention proposes a secure storage system for Internet data, aiming to facilitate the management of Internet data while improving the security of Internet data.

[0027] Figure 1 This is a structural diagram of a secure storage system for Internet data provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where a user-side server stores Internet data from different sources, and the secure storage system for Internet data is applied on the server side.

[0028] like Figure 1 As shown, the secure storage system for Internet data provided by the embodiment of the present invention includes: a data acquisition module 10, a data layering module 20, an encryption and decryption module 30, and a database 40.

[0029] Specifically, the data acquisition module 10 is used to obtain the Internet data generated in each Internet platform and classify the Internet data according to the data type; the data stratification module 20 is used to allocate storage space to different types of Internet data and group and merge the Internet data of the same type; the encryption and decryption module 30 is used to allocate corresponding keys and encryption algorithms to Internet data of different data formats; the database 40 is used to receive and store the encrypted Internet data.

[0030] It should be noted that the Internet data in this embodiment is generated by users connecting to the Internet platform through a client server. The client server can be a terminal device with networking function such as a computer, a mobile phone, a tablet, etc. The Internet platform can be a service device with resource sharing such as software, a web page, a system, etc. For example: chat software, hownet platform, etc. When users access the Internet platform, they often need to enter various information, such as: account password, mobile phone number, email address, bank card and other sensitive information. The data acquisition module 10 captures the Internet data generated during the access process.

[0031] In special scenarios, the server can also be an enterprise-level server, and the Internet platform can be multiple internal business systems. The business system generates a large amount of data when processing business and uploads it to the server for storage. This data may involve the core secrets of the enterprise and needs to be stored securely.

[0032] In this embodiment, the Internet data is classified by the data acquisition module 10, grouped in the data layering module 20, encrypted and decrypted in the encryption and decryption module 30, and finally stored in the database 40. When a third-party networked device needs to access the Internet data in the database 40, it needs to decrypt the encrypted ciphertext into plaintext according to the corresponding key.

[0033] Furthermore, the data types include: time series data and non-time series data; the data acquisition module organizes the time series data into a plurality of time series files, and organizes the non-time series data into a plurality of non-time series files.

[0034] Among them, time series data is widely available, including operation and maintenance data generated by IT equipment, monitoring data generated by sensors, etc. Time series data has a strong timeliness and contains rich data value. With the continuous development of mobile Internet, time series data is growing in speed and scale. Therefore, it is necessary to organize and manage time series data to ensure efficient data writing while supporting efficient retrieval and analysis of data within a specific time range.

[0035] Exemplarily, with the relaxation of the requirements for data consistency and the improvement of access performance in Internet applications, the key-value model is increasingly widely used to manage non-relational data. The key-value storage system has been widely applied as a novel storage solution. The key-value storage manages a set of key-value pairs, where a key-value pair refers to the mapping relationship from a key to a value.

[0036] Furthermore, the time-series data and non-time-series data are respectively written into key-value pairs, and the LSM-tree is used as a storage engine to manage the key-value pairs. That is, the randomly written data is converted into sequential writing. The data records are sorted by the primary key in the memory first. When the amount of data cached in the memory reaches the set threshold, these data are sequentially written to the disk and saved as independent files. These independent files need to be merged to generate one or more new files to reduce the number of data files and eliminate data redundancy in the files, thereby optimizing the disk space utilization rate and improving the query performance of the system.

[0037] Furthermore, after classifying the Internet data according to the data type, the data acquisition module 10 records the access times N and the time interval T of the user accessing each time-series file and / or non-time-series file in real time, and sets the time interval threshold TS and the access times threshold NS. When T > TS, N is compared with NS. If N < NS, the corresponding time-series file and / or non-time-series file are marked as invalid data and transferred from the original storage space to the cloud storage. To release the memory pressure, it is necessary to automatically clean up the time-series files and / or non-time-series files that have not been used for a long time stored in the memory and transfer them from the original storage location to the cloud.

[0038] Specifically, after the time-series file and / or non-time-series file are stored in the memory, the data acquisition module 10 starts to count the access times N and the time interval T. When no one accesses the data after the TS time period from the last access time, the relationship between the access times N and the access times threshold NS is judged. If the access times N also do not reach the access times threshold NS, the data is judged as infrequently used and marked as invalid data. Among them, the access times threshold NS is not a fixed value, which is obtained by the ratio of the time period from the data entry time T0 into the memory to the current time TN to the preset fixed time period TS, and is dynamically adjusted. That is, the ratio of (TN - T0) / TS is rounded in real time as the access times threshold NS.

[0039] Furthermore, the data stratification module 20 is specifically used to store the time-series files in the first storage space and the non-time-series files in the second storage space. Preferably, the memory is divided according to the performance data of the server, and the performance data of the server includes: hard disk performance index, memory performance index and processor performance index, etc. The data stratification module 20 queries and determines the storage sector with sufficient space from the pre-set storage table according to the Internet data acquired by the data acquisition module 10 and the size of the data, and divides the address of the storage sector.

[0040] As mentioned above, independent files need to be merged to generate one or more new files to reduce the number of data files. Specifically, this embodiment classifies the first storage space data layer by file time range, stores the time series files that fall within the corresponding time range in the corresponding layer for merging; and classifies the second storage space data layer by file size range, stores the non-time series files that fall within the corresponding size range in the corresponding layer.

[0041] The data stratification module 20 is also specifically used to stratify the first storage space, including a data layer and a cache area; divide the data layer into L1, L2, L3...LN layers according to the time range of the time series file, and store the time series files falling into the corresponding time range in the corresponding layer.

[0042] The ratio of the time range of the next level to the time range of the previous level is set as a constant.

[0043] The first storage space data layer is graded by the file time range, including: the file first enters the cache area, and the time range of the L1 layer is pre-set, and the time range here can be adjusted. From the L1 layer to the LN layer, the time range of each layer increases proportionally, and the capacity of each layer also increases proportionally. For example: the time range of the L1 layer is 1 hour, and the magnification is 10, then the time range of the L2 layer is 10 hours, and the time range of the L3 layer is 100 hours. Its capacity also increases accordingly in proportion, and the time range of the time window is connected in sequence from beginning to end. The time series files that fall within the corresponding time range are stored in the corresponding level. If the number of files in a certain level exceeds the capacity of that level, it will be merged with the next level.

[0044] By merging time series files according to the file time range, the initial time window and merge multiple can be set according to the usage during the merge process to adjust the merge effect. The target level of the time series file can be quickly determined, which greatly reduces the number of merges during the merge process and reduces the system overhead during the merge. Each layer corresponds to a time range, and the number of layers to which the time series file belongs can be used to manage the data lifecycle.

[0045] The data stratification module 20 is also specifically used to stratify the second storage space, including a data layer and a cache area; according to the size range of the non-sequential files, the data layer is divided into M1, M2, M3...MN layers, and the non-sequential files falling into the corresponding size range are stored in the corresponding layer.

[0046] The second storage space data layer is graded by file size range, including: merging several files of similar size into one file. This merging strategy can effectively reduce the number of non-sequential files. During the merging process, it is necessary to first traverse all non-sequential files, assign non-sequential files of similar size to the same level, and calculate the average size of non-sequential files in the level. Then, multiply the average size by 0.5 and 1.5, which are used as the lower and upper bounds of the level respectively. For unallocated non-sequential files in the cache area, compare them with all existing levels. If the size of the non-sequential file is between the upper and lower bounds of a certain level, put the non-sequential file into the level, and update the upper and lower bound values ​​of the level. If there is no suitable level, create a new level containing the file.

[0047] The encryption and decryption module 30 is specifically used to set one or more different encryption and decryption algorithms for Internet data in different data formats according to the data format of the Internet data, and provide corresponding keys; the data format includes: structured data, unstructured data and semi-structured data.

[0048] For different Internet platforms, the data types collected by the data collection module 10 are rich and varied, including table data, HTML web page files, XML files, RDF (Resource Description Framework) data, text data, graph (social network) data, multimedia data (audio / video / image), etc. These data can be divided into different data formats such as structured data, unstructured data and semi-structured data.

[0049] The first type of data format: structured data, including numbers, symbols, etc.

[0050] Structured data is highly organized and neatly formatted data. Structured data is a type of data that can be easily put into tables and spreadsheets. It is typically represented and stored using a relational database, and is presented as two-dimensional data. The general characteristics are: data is in rows, a row of data represents the information of an entity, and the attributes of each row of data are the same.

[0051] The storage and arrangement of structured data are very regular, also known as quantitative data, which is information that can be represented by data or a unified structure. Typical structured data include: credit card number, date, financial amount, phone number, address, product name, etc. For example, enterprise ERP, financial system; medical HIS database; education card; government administrative approval; other core databases, etc.

[0052] The second type of data format: semi-structured data, including log files, XML documents, JSON documents, Email, etc.

[0053] Semi-structured data is a form of structured data that does not conform to the data model structure associated with a relational database or other data tables, but contains relevant tags to separate semantic elements and hierarchize records and fields. It is also called a self-describing structure.

[0054] With semi-structured data, entities belonging to the same category can have different attributes, and even if they are grouped together, the order of these attributes is not important. Semi-structured data is very scalable.

[0055] The third type of data format: unstructured data, including video, audio, pictures, images, documents, text, etc.

[0056] Unstructured data is irregular or incomplete data with no predefined data model. Its fields are of variable length and each field record may consist of repeatable or non-repeatable subfields. It may be textual or non-textual, and may be human or machine generated. Unstructured data constitutes the vast majority of available data on the web, and it is growing every year. As more information becomes available on the web, much of it is unstructured.

[0057] Human-generated unstructured data includes: text files (word processing, spreadsheets, presentations, emails, logs, etc.), emails, social media data, website data, mobile data (text messages, locations, etc.), communication data (chat, instant messaging, phone recordings, collaboration software, etc.), media data (MP3, digital photos, audio files, video files, etc.), business application data.

[0058] Machine-generated unstructured data includes: satellite imagery (weather data, topography, military activities, etc.), scientific data (oil and gas exploration, space exploration, seismic imagery, atmospheric data, etc.), digital surveillance (surveillance photos and videos, etc.), sensor data (traffic, weather, ocean sensors, etc.).

[0059] Furthermore, the encryption and decryption unit 30 pre-stores a variety of encryption and decryption algorithms, such as: Data Encryption Standard Algorithm (DES), Triple Data Encryption Standard Algorithm (3DES), Advanced Encryption Standard Algorithm (AES), RSA public key algorithm, Digital Signature Algorithm (DSA), Elliptic Curve Cryptography Algorithm (ECC), Message Digest Algorithm (MD5), Secure Hash Algorithm (SHA), etc.

[0060] Symmetric keys use the same key in the encryption and decryption process. Common symmetric encryption algorithms include DES, 3DES, AES, RC5, RC6, etc. However, if the same key is used on the client, the client can use the key to crack everyone's ciphertext. If each client and server use different keys, the server needs to manage all the different keys.

[0061] Asymmetric keys use different keys in the encryption and decryption processes. Common asymmetric encryption includes RSA, etc. The server generates a pair of public and private keys. The private key is stored on the server and sent to the client. The client's plain text is encrypted with the public key and the ciphertext needs to be decrypted with the private key. Compared with symmetric key encryption, asymmetric encryption does not require sharing keys between the client and the server. As long as the private key is not sent to any user, even if the public key is intercepted on the Internet, it cannot be decrypted. Only the stolen public key is useless.

[0062] In addition, a load balancing unit can be set in the encryption and decryption module 30 to dynamically distribute a large number of encryption or decryption tasks to different cache units to avoid system crashes caused by too many encryption or decryption tasks. At the same time, when some encryption tasks fail, all encryption tasks can be prevented from being completely paralyzed.

[0063] Furthermore, a communication channel is allocated in advance for the user terminal and the database, and a communication connection is established between the user terminal and the database, so that the user terminal queries the database based on the established communication connection.

[0064] The third-party device can establish network communication with the database by the user inputting personal account, password or verification code and other information. The third-party device can be a computer, mobile phone and other devices. The connection pool pre-allocates communication channels for the user terminal and the database. The connection pool is a buffer pool technology for creating and managing connections, which stores the connection configuration of each time series file and / or non-time series data. The target time series file and / or non-time series data is stored in the database. The user selects the target time series file and / or non-time series data from the database. The server obtains the user's selection instruction, selects the communication channel corresponding to each data file from the allocated communication channel in the connection pool, connects to the third-party device, and sends the corresponding key and encryption algorithm to the third-party device to realize the operation of the target time series file and / or non-time series data. By allocating communication channels through the connection pool, the target time series file and / or non-time series data can be quickly selected from the database, thereby improving the efficiency of data query.

[0065] The database has multiple storage sectors, which are also used to store encryption and decryption algorithms and corresponding keys. Access rights are set in the database. Only verified legitimate third-party devices can access database data. Third-party devices need to send authentication information to the database and can access the database after passing the authentication. The database stores the ciphertext of the keys of multiple algorithms. The ciphertext of the keys of multiple algorithms in the database can be stored in the form of algorithm code + ciphertext of the key. The third-party device reads the ciphertext of the keys of these encryption algorithms one by one.

[0066] A secure storage system for Internet data provided by an embodiment of the present invention first organizes and manages time series data and non-time series data by grouping and merging them, while ensuring efficient data writing, and can perform efficient retrieval and analysis based on the time range of the time series data or the size of the non-time series data. Secondly, invalid data is marked according to the number of accesses and time intervals, and transferred to the cloud to alleviate the storage pressure of the server. Then, different encryption and decryption algorithms are set for data in different formats, and information is centrally encrypted and decrypted for storage to improve the security of the information, and a load balancing unit is set to avoid system crashes during the encryption process. Finally, communication channels are allocated in advance for third-party user terminals and databases to improve the efficiency of data queries.

[0067] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0068] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A secure storage system for Internet data, characterized in that: The system applied to the server side includes: a data acquisition module, a data layering module, an encryption and decryption module, and a database; wherein, The data acquisition module is used to obtain Internet data generated in each Internet platform and classify the Internet data according to the data type; The data layering module is used to allocate storage space for different types of Internet data and group and merge the Internet data of the same type; The encryption and decryption module is used to allocate corresponding keys and encryption algorithms for Internet data of different data formats; The database is used to receive and store the encrypted Internet data.

2. The method according to claim 1, characterized in that The data type includes: time series data, non-time series data; the data acquisition module organizes the time series data into several time series files and the non-time series data into several non-time series files; After classifying the Internet data according to the data type, the data acquisition module records in real time the access times N and time interval T of the user accessing each time series file and / or non-time series file, and sets a time interval threshold TS and an access times threshold NS; when T>TS, compare N with NS, if N<NS, mark the corresponding time series file and / or non-time series file as invalid data and transfer it from the original storage space to cloud storage.

3. The method according to claim 2, characterized in that The data layering module is specifically used to store the time series files in the first storage space and the non-time series files in the second storage space.

4. The method according to claim 3, characterized in that The data layering module is also specifically used to layer the first storage space, including a data layer and a buffer; divide the data layer into L1, L2, L3…LN levels according to the time range of the time series files, and store the time series files falling within the corresponding time range in the corresponding levels.

5. The method according to claim 4, characterized in that The ratio of the time range of the next level to the time range of the previous level is set as a fixed value.

6. The method according to claim 4, characterized in that The data layering module is also specifically used to layer the second storage space, including a data layer and a buffer; divide the data layer into M1, M2, M3…MN levels according to the size range of the non-time series files, and store the non-time series files falling within the corresponding size range in the corresponding levels.

7. The method according to claim 1, characterized in that The encryption and decryption module is specifically used to set one or more different encryption and decryption algorithms for Internet data of different data formats according to the data format of the Internet data and provide the corresponding keys; The data format includes: structured data, unstructured data, and semi-structured data.

8. The method according to claim 7, characterized in that A communication channel is allocated for the user side and the database in advance, a communication connection between the user side and the database is established, and the user side queries the database based on the established communication connection.

9. The method according to claim 8, characterized in that The database has multiple storage sectors and is also used to store encryption and decryption algorithms and the corresponding keys.

Images