Secure cloud storage monitoring system and method based on virtual machine

Abstract

The present application discloses a secure cloud storage monitoring system and method based on a virtual machine, including: a virtual machine obtains data to be stored; establishes a hash-based memory data structure to optimize the data to be stored; uses a first static hash function to encode the data to be stored, and stores the encoded data to be stored. The present application provides a secure cloud storage monitoring system and method based on a virtual machine. As an efficient and scalable cloud storage monitoring system, the system is based on a virtual machine and combines a special hash memory data structure and encoding technology to improve the efficiency and immediacy of data processing while ensuring the privacy and security of the data.

Description

Technical Field

[0001] The present application relates to the field of information security technology, and in particular to a virtual machine-based secure cloud storage monitoring system and method. Background Art

[0002] With the rapid growth of enterprise data, data management and archive management are facing increasing challenges. Enterprises need to efficiently and securely store and analyze large amounts of semi-structured or unstructured data, such as contract documents, customer information, financial statements, etc. Traditional relational databases and file systems can no longer meet the needs of enterprises for data storage and analysis.

[0003] Cloud storage provides enterprises with a flexible and scalable data management solution. However, as the scale of cloud storage systems continues to expand, their complexity is also increasing. In response to this challenge, the monitoring and security protection of cloud storage systems are becoming increasingly important. It is necessary to collect and analyze system operation logs in real time to detect potential security threats and performance issues, while also ensuring the privacy and compliance of stored data.

[0004] Traditional relational database management systems (RDBMS), such as PostgreSQL or Oracle databases, were developed primarily to store relational data with well-defined schemas and support transactional reads and updates. However, the data to be stored, such as logs or metrics, is usually not updated, and its attributes are dynamic and multidimensional. NoSQL document stores, such as MongoDB or CouchDB, achieve better horizontal scalability and relax the schema requirements of data when ingesting data. However, the log-structured merge tree used by many NoSQL databases mainly supports efficient access to data through primary keys. Due to the high overhead introduced by the creation and maintenance of secondary indexes, NoSQL databases often require users to pre-define index data attributes, and full-text searches incur a lot of storage overhead.

[0005] Therefore, cloud storage monitoring systems that use traditional storage technology generally have problems such as low processing efficiency and insufficient security guarantees. Summary of the invention

[0006] In view of this, the present application provides a secure cloud storage monitoring system and method based on virtual machines. As an efficient and scalable cloud storage monitoring system, the system is based on virtual machines and combines special hash memory data structures and encoding technologies to improve the efficiency and real-time performance of enterprise unstructured data processing while ensuring data privacy and security.

[0007] The present application provides a virtual machine-based secure cloud storage monitoring method, comprising:

[0008] The virtual machine obtains the data to be stored;

[0009] Establishing a hash-based memory data structure to optimize the data to be stored;

[0010] The data to be stored is encoded using a first static hash function, and the encoded data to be stored is stored.

[0011] Optionally, establishing a hash-based memory data structure, and optimizing the data to be stored based on the memory data structure, includes:

[0012] Establishing a hash-based inverted index structure to perform inverted indexing on the data to be stored;

[0013] A monitoring point is set up to continuously track the inflow rate and data pattern of the data to be stored, and redundant publishing lists are merged online, where the publishing lists are used to describe the position and frequency information of any word in the inverted index structure.

[0014] Optionally, establishing a hash-based inverted index structure to perform inverted indexing on the data to be stored includes:

[0015] Decomposing the data to be stored into a plurality of word units;

[0016] Generate a hash fingerprint of each word using a hash algorithm, wherein the hash algorithm is MurmurHash or CityHash;

[0017] Add the hash fingerprint and corresponding ID of each word to the inverted list.

[0018] Optionally, merge redundant release lists online, including:

[0019] Identify duplicate data entries in the release list based on the hash fingerprint of each token;

[0020] Duplicate data entries are merged so that only the latest version data is retained for each data entry.

[0021] Optionally, the method further comprises:

[0022] Dynamically adjust the size of the publishing list according to data load conditions;

[0023] A dynamic merging strategy is formulated, and based on the dynamic merging strategy, a merging operation of the publishing list is executed in batches within a preset time based on the hash fingerprint of each word.

[0024] Optionally, encoding the data to be stored by using a first static hash function includes:

[0025] Judging the word units and identifying sensitive word units;

[0026] Mapping the sensitive word to a reference to a non-personal identifier;

[0027] The reference is encoded using a first static hash function.

[0028] Optionally, encoding the reference using a first static hash function comprises:

[0029] Constructing a first static hash function, wherein the first static hash function is SHA-256 or MD5;

[0030] Using the first static hash function, converting the sensitive word into a hash value;

[0031] The hash value is used to construct a publishing list corresponding to the sensitive word.

[0032] Optionally, the method further comprises:

[0033] evaluating the validity period of the first static hash function;

[0034] Before the expiration of the validity period, the first static Hash function is replaced by a second static Hash function.

[0035] Optionally, before storing the encoded data to be stored, the method further includes:

[0036] The encoded data to be stored is compressed.

[0037] The embodiment of the present application also provides a virtual machine-based secure cloud storage monitoring system, including:

[0038] An acquisition module, used to acquire data to be stored;

[0039] An optimization module, used to establish a hash-based memory data structure to optimize the data to be stored;

[0040] The encoding module is used to encode the data to be stored by using a first static hash function, and store the encoded data to be stored.

[0041] The present application provides a secure cloud storage monitoring system and method based on a virtual machine, and introduces a hash-based inverted index structure. This special hash memory data structure can greatly improve the storage and retrieval efficiency of an enterprise's unstructured data, and is very suitable for the scenario of massive unstructured data management in an enterprise. In addition, in the embodiment of the present application, a static hash function is used to encode the stored data, providing an irreversible and non-directly associated reference (identification) for privacy-sensitive words, greatly improving the security and privacy protection capabilities of the stored data, and meeting the compliance requirements of enterprise data management. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the disclosed embodiments in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A flowchart of a secure cloud storage monitoring method based on a virtual machine provided in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of the structure of a virtual machine-based secure cloud storage monitoring system provided in one embodiment. DETAILED DESCRIPTION

[0045] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

[0046] It should be clear that the following embodiments of the present disclosure are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present disclosure from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. The present disclosure can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present disclosure. It should be noted that the following embodiments and features in the embodiments can be combined with each other in the absence of conflict. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present disclosure.

[0047] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0048] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present disclosure. The drawings only show components related to the present disclosure rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0049] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0050] Embodiment 1

[0051] In modern computing environments, especially in the field of cloud computing, the security of data storage and processing is a key issue in enterprise data management. With the increasing popularity of virtualization technology, a secure cloud storage monitoring system based on virtual machines has become a key solution to ensure data storage security in cloud environments. This technical solution proposes an efficient and scalable cloud storage monitoring system, which is based on virtual machines and combines special hash memory data structures and other optimization technologies to improve the efficiency and immediacy of data processing while ensuring data privacy and security.

[0052] In the embodiments of the present invention, a virtual machine refers to a computer system that is simulated on a physical machine through virtualization technology and is identical to the hardware. It can independently run operating systems and applications, and use a virtualization management program (hypervisor) to manage interactions with physical hardware. In a cloud storage system, virtual machines can dynamically allocate resources and achieve elastic scaling to optimize computing performance and storage efficiency.

[0053] The embodiment of the present invention adopts a cloud storage system architecture based on virtual machines: the cloud storage system is a highly scalable storage solution that separates data storage from computing resources through virtual machine technology and provides services through the network. The basic architecture usually includes the following key components:

[0054] Storage Nodes: A cluster of physical or virtual servers that carry data and are responsible for the actual storage and backup of data.

[0055] Management Nodes: Control storage nodes, manage data distribution, replication, and cluster health.

[0056] Front-end Load Balancers: manage data requests, implement load balancing and failover, and ensure high availability of user access.

[0057] API Gateways: Provides a series of interfaces that allow clients to interact with storage systems through RESTful APIs.

[0058] The storage system has the following functions:

[0059] Dynamic resource management of virtual machines: Combined with the monitoring system, virtual machine resource allocation is automatically adjusted. For example, CPU and memory resources are increased when data access demand peaks to ensure efficient data processing speed.

[0060] Data redundancy and backup strategy: Take advantage of virtualization technology to achieve multiple copies of data, real-time backup and rapid recovery to cope with possible hardware failures and data loss.

[0061] Data distribution and sharding: To optimize storage and retrieval efficiency, data is physically distributed and sharded. The inverted index connects these distributed data into a large index table, allowing data to be quickly indexed and retrieved.

[0062] Encryption and security: Data is often stored encrypted in storage nodes. In addition, data transmitted over the network also uses encryption protocols (such as TLS / SSL) to ensure data privacy and security.

[0063] In order to meet the needs of different applications and ensure system availability, the architecture design of cloud storage systems needs to be highly reliable and scalable. The elastic design allows the system to increase storage space and computing power on demand without interrupting services. By using virtualization technology and dynamic resource management, cloud storage systems can not only effectively process and store large amounts of data, but also adjust resources at any time according to system load.

[0064] The following is an explanation of the technical terms of the embodiments of the present invention:

[0065] Hash algorithms, such as MurmurHash or CityHash, are used to convert input (such as word units) into fixed-length outputs (hash values) that are random and avoid conflicts as much as possible.

[0066] Static hash functions: such as SHA-256 or MD5, used to encode sensitive information into irreversible and anonymous references to enhance data privacy and security.

[0067] MurmurHash: MurmurHash is a non-cryptographic hash function designed to efficiently hash data. It provides good dispersion and low collision probability when computing hash values. Due to its excellent performance, MurmurHash is often used to generate hash values ​​in software development, especially when processing large amounts of data.

[0068] CityHash: CityHash is similar to MurmurHash and is also a non-cryptographic hash function. Google developed CityHash to quickly process large amounts of data and generate unique hash values. It is designed to provide high-quality hashing functions for long strings and effectively reduce hash collisions when processing similar strings.

[0069] SHA-256: SHA-256 (Secure Hash Algorithm 256-bit), a member of the SHA-2 (Secure Hash Algorithm family), is a cryptographic hash function. SHA-256 can compress messages of any length into a fixed-length (256-bit) summary. Because the hash value it outputs is highly unique and irreversible, it is widely used in data integrity verification and information security.

[0070] MD5: MD5 (Message Digest Algorithm Version 5) is also a widely used cryptographic hash function that can generate a 128-bit (16-byte) hash value. Although MD5 has been partially eliminated due to its security (more vulnerable to collision attacks), it is still used for consistency checks in some non-security-critical applications.

[0071] Posting list (inverted index): The posting list is a component of the inverted index that stores a list of specific terms sorted by document ID. In a search engine, when an embodiment of the present invention indexes a set of documents and searches based on a keyword, the search engine will use the posting list in the inverted index to quickly find all documents containing the keyword.

[0072] Figure 1 A flowchart of a virtual machine-based secure cloud storage monitoring method provided in an embodiment of the present disclosure is provided. The virtual machine-based secure cloud storage monitoring method includes steps S1 to S3.

[0073] S1. The virtual machine obtains the data to be stored;

[0074] In the embodiment of the present invention, the virtual machine is responsible for receiving and parsing the user's input data and converting it into a format that can be further processed by the storage system. Specifically:

[0075] Receive data: The virtual machine receives data sent from the user or other systems through the network or other communication methods. For example, communication protocols such as HTTP(S), FTP, WebSocket or RESTful API are used to upload and download data. In addition, the monitoring service running in the virtual machine will monitor the preset network port in real time to process any incoming data requests in a timely manner.

[0076] Data parsing and verification: Format parsing of received data. For example, data in JSON, XML, or other formats may need to be converted into data structures for internal use. At the same time, the integrity and security of the data also need to be verified to ensure that the data has not been tampered with. In addition, a cache mechanism is established in the virtual machine memory to temporarily store a large amount of incoming data streams to reduce the impact of instantaneous data peaks on the storage system.

[0077] Preprocessing: Perform necessary data cleaning and deduplication processing before storage to optimize storage efficiency and data quality.

[0078] Specifically, the virtual machine configures a network monitoring service to listen to a specific port to receive data; the virtual machine verifies and decodes the received data packets to ensure that the data is complete and in the correct format; the data caching mechanism is implemented to temporarily store the data and perform pre-processing on demand to reduce redundancy.

[0079] S2. Establishing a hash-based memory data structure, and optimizing the data to be stored based on the memory data structure;

[0080] In step S2, the establishment of the data structure aims to efficiently manage the data to be stored, achieve fast retrieval and data security. Data optimization further improves storage and query efficiency and reduces the impact of redundant data on system performance.

[0081] S2 specifically includes:

[0082] S21. Establishing a hash-based inverted index structure to perform inverted indexing on the data to be stored;

[0083] The hash-based inverted index structure is a data structure that is particularly suitable for text retrieval and data mining, especially when dealing with large-scale data sets. In this structure, a hash table is used to store indexes to achieve fast search and insertion operations. The key to this structure is that it can quickly locate all documents containing a specific word.

[0084] The inverted index consists of two parts:

[0085] Vocabulary: Stores unique tokens (such as words, phrases, or vocabulary tokens) that appear in all documents.

[0086] Inverted Lists: For each token in the vocabulary, an inverted list is maintained which contains references (such as document IDs) to all documents containing that token.

[0087] Hash-based implementation:

[0088] In a hash-based inverted index:

[0089] Token hash table: A hash table is used to replace the traditional vocabulary, where the key is the hash value of the token and the value is a pointer to the inverted list.

[0090] Inverted List: Similar to the inverted list in the base structure, but usually with some optimizations, such as merging duplicate document entries.

[0091] For example, there are three documents:

[0092] Document 1 Content: "Secure Storage System"

[0093] Document 2 Content: "Storage System Efficiency"

[0094] Document 3 Content: "Security Monitoring"

[0095] The steps to build an inverted index are as follows:

[0096] Tokenization: Break the content of each document into tokens.

[0097] Hash calculation: Calculate a hash value for each word.

[0098] Create an index: Add the hash value of each word and the corresponding document ID to the inverted list.

[0099] The result of building an inverted index may be as follows, where H(word) represents the hash value of the word:

[0100] Hash table:

[0101] H ("safe")-> points to the inverted list [document 1, document 3]

[0102] H ("Storage")->points to the inverted list [Document 1, Document 2]

[0103] H("system")->points to the inverted list [document 1, document 2]

[0104] H ("efficiency")->points to the inverted list [Document 2]

[0105] H ("monitor")->points to the inverted list [document 3]

[0106] When searching for documents containing "security" and "storage", the hash function can be used to quickly locate the inverted list and find the document set containing these two terms. Since the inverted lists of the two terms "security" and "storage" both contain "document 1", it can be quickly determined that document 1 contains all the terms to be retrieved.

[0107] The hash-based inverted index significantly improves the efficiency of retrieval, especially when facing huge data sets, by reducing the number of string comparisons for terms and effectively utilizing the properties of hashes to achieve fast searches.

[0108] Specifically, in S21, a hash-based inverted index structure is established to perform inverted indexing on the data to be stored, including the following steps S211-S213:

[0109] S211. Decomposing the data to be stored into a plurality of word units;

[0110] In S211, tokenization technology is used to decompose text strings into meaningful units, called tokens, according to grammar and vocabulary rules of different languages ​​such as Chinese and English.

[0111] S212. Generate a hash fingerprint of each word using a hash algorithm, wherein the hash algorithm is MurmurHash or CityHash;

[0112] A hash fingerprint refers to a unique hash value (fingerprint) of each word obtained by a hash algorithm. In the embodiment of the present application, it is necessary to select a hash algorithm that can balance the probability of collision and computational efficiency. For a monitoring system, an ideal hash algorithm should have high randomness and low collision rate to ensure that the fingerprint of each word is as unique as possible. In addition, the embodiment of the present application can map a word to its hash fingerprint while ensuring that the process is fast and stable. Specifically, a mapping mechanism can be established to maintain a mapping table in the system to record the relationship between a word and its corresponding hash fingerprint. In this way, the constructed hash fingerprint mapping table can be used to achieve fast word search and comparison in the monitoring system. When a new word request is received, the system first calculates its hash fingerprint, and then quickly searches in the mapping table to verify the existence of the word.

[0113] In step S212, the MurmurHash and CityHash hash algorithms are used to generate the hash fingerprint of each word. The following are the detailed steps of the specific implementation:

[0114] 1. Generate hash fingerprint using MurmurHash algorithm

[0115] Initialize the hash function: Load the MurmurHash algorithm library and initialize a hash function instance for each word.

[0116] Calculate the hash value: Call the MurmurHash function for each token after segmentation and output a hash value of 128 bits or other appropriate length.

[0117] For example, for the word "key", the process is as follows:

[0118] Word: "key"

[0119] Hash fingerprint (MurmurHash):,"11f50ed2c5a3a3b5d260b9df465811b4"

[0120] 2. Generate hash fingerprint using CityHash algorithm

[0121] Initialize the hash function: Introduce the CityHash algorithm and set the hash function for each word.

[0122] Perform hash operation: Perform CityHash calculation on each word to obtain a hash fingerprint of fixed length.

[0123] For example, for the word "data", the process is as follows:

[0124] Word: "data"

[0125] Hash fingerprint (CityHash): "9ae0ea9e3c9c8e988b6a1850e17ea195"

[0126] In the actual inverted index creation process, the hash fingerprint is generated as follows:

[0127] Apply MurmurHash and CityHash algorithms to token "management":

[0128] Word: "management"

[0129] MurmurHash fingerprint: eg,"2a94f8fa2ccb1e29ed2a1bcfe3c208f3"

[0130] CityHash fingerprint: eg,"3bcdb75cb8f63f6c29be1a369d9e041a"

[0131] Apply MurmurHash and CityHash algorithms to the word "technique":

[0132] Word: "technology"

[0133] MurmurHash fingerprint: eg,"d3bfbc2dd0f42d1a0446c8e0f1b79ae8"

[0134] CityHash fingerprint: eg,"64c2f54b5e4dc9cc9a5b76465726443c"

[0135] After each word is processed by the hash function, it will get a unique hash fingerprint.

[0136] S213. Add the hash fingerprint and corresponding ID of each word to the inverted list.

[0137] In S213, a hash table is developed as the basic data structure of the inverted index, wherein the key is the hash fingerprint of the word unit and the value is the document ID list. For each hash fingerprint item of the word unit in the hash table, the document ID in which the corresponding word unit appears is added.

[0138] For example, if there is a piece of data to be stored: "Key management and data encryption technology", the processing flow is as follows:

[0139] Preprocessing: Remove meaningless characters and get the keyword: "key management data encryption technology".

[0140] Tokenization: Segment the keywords to obtain token sequences: "key", "management", "data", "encryption", "technology".

[0141] Hash calculation: Apply the MurmurHash algorithm to each word to obtain a set of hash fingerprints.

[0142] Index addition: Associate the hash fingerprint of each term with the document ID and add it to the inverted index list.

[0143] The results are as follows:

[0144] Hash table:

[0145] H("Key")->Link to [Document ID1]

[0146] H("Management")->Link to [Document ID1]

[0147] H("data")->Link to [Document ID1]

[0148] H("encrypted")->link to [document ID1]

[0149] H("Technology")->Link to [Document ID1]

[0150] In another example, consider a blogging platform with the following simplified article content:

[0151] Article 1: "Cloud storage provides efficient data backup"

[0152] Article 2: "Data security is crucial in cloud computing"

[0153] Article 3: "Improving the Reliability of Cloud Services"

[0154] When establishing an inverted index for this blog platform, the embodiment of the present invention performs the following steps to implement the hash fingerprint technology and optimize the word unit matching process:

[0155] Tokenization:

[0156] Each article is broken down into a series of tokens (e.g., “cloud storage,” “efficiency,” “data,” “backup,” “security,” “cloud computing,” “critical,” “improvement,” “cloud service,” “reliability”).

[0157] Calculate the hash fingerprint (Hashing):

[0158] A hash function is applied to each word element to generate a unique hash value (fingerprint). For example, an embodiment of the present invention uses a hash function H so that the word element is converted into the following:

[0159] H("Cloud Storage")->2a6d

[0160] H("data")->8ef4 ...

[0161] Create index:

[0162] Generate an inverted index based on the hash fingerprint of each word. For each new word's fingerprint, if it does not exist in the index, create a new entry in the index and associate it with the ID of the current article; if it exists, add the ID of the current article to the inverted list of the corresponding word fingerprint.

[0163] The result is:

[0164] Hash table (inverted index):

[0165] 2a6d->[Article 1, Article 3]

[0166] 8ef4->[Article 1, Article 2] ...

[0167] Index storage example:

[0168] 2a6d:{Article 1, Article 3}

[0169] 8ef4:{Article 1, Article 2} ...

[0170] This process simplifies the indexing and retrieval process of the article content, so that the embodiment of the present invention can very quickly find all articles containing specific words, while also greatly saving storage space.

[0171] It can be seen from this that the embodiment of the present invention needs to select a hash function that can generate a unique identifier without causing a large number of conflicts, so as to speed up the retrieval speed and improve the storage efficiency.

[0172] The embodiment of the present invention uses a high-performance memory data structure, such as Bloom Filter, to quickly determine whether an article ID already exists in the inverted list of a certain word, thereby avoiding inserting duplicate article IDs into the list.

[0173] S22. Establish monitoring points, continuously track the inflow rate and data pattern of the data to be stored, and merge redundant publishing lists online, where the publishing lists are used to describe the position and frequency information of any word in the inverted index structure.

[0174] Posting List is a term used in the inverted index structure to describe a list that stores information about the location of a particular term (usually a keyword or phrase) in which documents it appears. Each term has a corresponding posting list entry in the vocabulary. Posting lists are a key component in building text search engines and some database indexing systems.

[0175] Each entry (release) in the release list consists of:

[0176] Document ID: represents the document number or unique identifier that contains this term. In a distributed system, this can also be the path to the file or other unique resource locator.

[0177] Position information: The position of a word in a document, which can be a single position or a series of positions.

[0178] Frequency information: how often a word appears in a document.

[0179] Other possible information: such as the context of the word in the document, etc.

[0180] Its significance lies in:

[0181] Indexing and retrieval: Posting lists allow search engines or databases to quickly index and retrieve information. For example, if you want to find all documents containing the term "cloud storage", the search engine will look up the posting list corresponding to "cloud storage" in the vocabulary, then extract the document ID from the list, and thus find all relevant documents.

[0182] Data storage efficiency: With the structure of the release list, the system can effectively store and manage the occurrence of keywords without having to store repeated word strings. This not only saves storage space, but also improves retrieval efficiency.

[0183] Retrieval optimization: The published list can be further optimized as required. For example, the retrieval performance of the list can be optimized through jump tables, hash tables or other efficient data structures.

[0184] In S22, setting up a monitoring point generally includes configuring monitoring software to monitor the data flow in the virtual machine. The monitoring point will record the data inflow rate and data pattern, such as packet size, request frequency, etc., in real time.

[0185] When designing data processing strategies for a secure cloud storage monitoring system based on virtual machines, improving storage efficiency and reducing unnecessary data duplication are the key points of optimization. Using online list merging technology, storage utilization can be improved and redundant information can be reduced while monitoring data streams in real time.

[0186] Specifically including steps S221 and S222:

[0187] S221. Based on the hash fingerprint of each word, identify duplicate data entries in the release list;

[0188] In S221, it is necessary to identify duplicate data entries in the existing publication list. For example, if a word appears in multiple documents, and these documents have a same document tag (such as the same version number or document type), these entries will be considered duplicates.

[0189] Specifically, S221 constructs a hash mapping table with the hash fingerprint of each data entry as the key and the location and frequency information as the value. The mapping table is updated in real time to reflect the fingerprint of the newly added data entry and its related information. The mapping table is traversed and the hash value is used to quickly locate potential duplicate entries. For each new entry added to the mapping table, a matching entry is checked by comparing the hash fingerprint. For each matching hash fingerprint, all duplicate publication list entries are marked in preparation for merging.

[0190] S222. Merge duplicate data entries so that each data entry only retains the latest version data.

[0191] Set the merge strategy, for example, determine the trigger conditions for the merge operation, priority settings, etc. Select the version rules for retaining data entries, for example, you can retain the latest version of the data.

[0192] Perform a merge operation based on the tags to merge the information of the duplicate entries (such as document location and frequency) into the retained entries. Remove duplicate and older versions of data entries from the release list, which is a deduplication operation.

[0193] After the merge operation is complete, the publication list is updated to ensure that it reflects the latest, non-duplicate information. In addition, if necessary, relevant caches and indexes are updated to keep the data consistent and up-to-date across the system.

[0194] It should be noted that in S222, the actual merging operation is performed, and the merging can be completed in various ways, such as integrating duplicate document references into one, or reorganizing the data in the list according to certain logical rules to reduce duplication and redundancy, that is, deduplication operation.

[0195] In addition, when the release list expands, timely and accurate deduplication is essential to maintain system efficiency and storage optimization. Therefore, for different data load levels, an algorithm needs to be implemented to automatically manage the expanded release list and deduplication operations to ensure that the data is always up-to-date and non-duplicate. The following are the specific steps A1-A2:

[0196] A1. Dynamically adjust the size of the release list according to the data load;

[0197] In A1, the data load of the system needs to be constantly monitored, which involves multiple aspects such as the data inflow rate, data volume, and query frequency. According to the monitoring results, the size of the release list is adjusted in real time. This includes expanding memory allocation, enabling new data structures to reduce search time, etc. In addition, it is necessary to manually determine when the release list size needs to be adjusted, such as when the system reaches a certain CPU or memory usage threshold.

[0198] A2. Formulate a dynamic merging strategy, and based on the dynamic merging strategy, execute the merging operation of the publishing list in batches within a preset time based on the hash fingerprint of each word.

[0199] Dynamically formulate merging strategies based on application scenarios and system performance requirements, and select appropriate merging strategies according to different scenarios and requirements. Specifically, the merging strategy can be real-time or periodic. For real-time systems, small batches may be more likely to be merged quickly, while batch processing systems may choose to merge large batches during non-peak hours.

[0200] S3. Encode the data to be stored using a first static hash function, and store the encoded data to be stored.

[0201] The "static hash function" in S3 refers to a deterministic hash function that does not change during system operation. This function is typically used to map input (such as a word or other type of identifier) ​​to a fixed-size, seemingly random string of data (hash value). This mapping process is an encoding operation used to convert data into another form. In this case, a "reference" refers to an identifier that points to a specific data or object, such as a pointer or index to a data item stored in a database or file system.

[0202] The purpose of using the first static hash function to encode the list and the references between the tokens is to maintain the anonymity and privacy of the data. This is achieved by replacing the data items and their identifiers (such as document IDs or tokens) with unique codes generated by the hash function, which improves the non-identification of the data.

[0203] It should be noted that the storage data in S2 is optimized to improve the efficiency of data storage and retrieval, while S3 is more targeted at sensitive data. Sensitive data needs to be irreversibly encoded in order to hide the plaintext information of the sensitive data, with the aim of improving storage security.

[0204] Specifically, in S3, the data to be stored is encoded using a first static hash function, including the following steps S31-S33:

[0205] S31. Judging the word element and identifying the sensitive word element;

[0206] For example, judgment can be made through preset sensitive word libraries, natural language processing (NLP) tools, etc.

[0207] S32. Mapping the sensitive word to a reference to a non-personal identifier;

[0208] For each sensitive word, create a reference replacement of a non-personal identifier. These reference replacements can be randomly generated identifiers or an anonymized code generated by a specific algorithm. In the data record, all sensitive words are replaced with the corresponding non-personal identifier reference to desensitize the data.

[0209] S33. Encode the reference using a first static hash function.

[0210] In S33, an appropriate static hash function is selected, such as SHA-256 or MD5, to convert sensitive words into hash values ​​to increase data security. A hash function is applied to each non-personal identifier reference to generate a unique hash value for each sensitive word. A new release list is constructed using the hash value spectrum, which contains the encoded word hash values ​​and their corresponding location information in the storage system.

[0211] The step of encoding the reference using the first static hash function includes the following steps S331-S333:

[0212] S331. Construct a first static hash function, wherein the first static hash function is SHA-256 or MD5;

[0213] S332. Using the first static hash function, convert the sensitive word into a hash value;

[0214] Bulk hash non-personal identifier references using a static hash function of your choice, and ensure that the same hash function version or configuration is used for each reference to maintain encoding consistency.

[0215] S333. Use the hash value to construct a publishing list corresponding to the sensitive word.

[0216] The hash value of each reference encoding is filled into the publication list, and each hash value corresponds to one or more locations of the original data in the storage system.

[0217] An example is used to illustrate the implementation of the above steps:

[0218] Assume that the non-personal identifier is referenced as "REF001" and represents a piece of sensitive data that needs to be protected.

[0219] Choose a hash function:

[0220] It was determined that SHA-256 is a suitable static hash function for this use case.

[0221] Perform the SHA-256 hash function on "REF001" to obtain a unique hash fingerprint.

[0222] Construct the encoded release list, and now the release list content is updated to the hash fingerprint, and has no obvious direct connection with the storage location of the actual data, thus maintaining the privacy of the data.

[0223] Example of an encoded release list:

[0224] Hash value: "5e88489..."->[document ID1 location, document ID2 location]

[0225] The above steps ensure that the original sensitive data is well protected through the reference of non-personal identifiers and subsequent hashing, while maintaining the functionality of the retrieval system.

[0226] In addition, using static hash functions is very effective in enhancing data security because it provides a non-reversible and non-direct reference to the stored word. However, in order to combat security attacks that may occur over time, such as rainbow table attacks, these hash functions need to be updated regularly. Here is how to do it:

[0227] B1. Evaluate the validity period of the first static hash function;

[0228] Based on the external security environment, evaluate the validity period of the static hash function and estimate the length of time the function can be used safely without leaking critical information.

[0229] B2. Before the expiration of the validity period, use a second static hash function to replace the first static hash function.

[0230] Make a plan to replace the old hash function before the end of the validity period determined by the evaluation. Among them, the second static hash function, such as SHA-3 encoding, can adjust its hash mechanism as needed during system operation, such as further adding salt or changing hash parameters to adapt to changing data protection needs.

[0231] In addition, before storing the encoded data to be stored, the encoded data to be stored may be compressed, and conventional compression techniques may be used, which will not be described here.

[0232] For example, in the field of cloud computing, real-time monitoring of logs and analysis of user behavior are key tasks to ensure service quality and security. For this application scenario, the embodiments of the present invention provide the following technical advantages:

[0233] Real-time monitoring of logs: The embodiments of the present invention can monitor and analyze log data generated from the cloud infrastructure in real time. Using the hash inverted index structure, specific events or patterns can be quickly retrieved, greatly improving the speed of detecting and responding to problems.

[0234] User behavior analysis: Analyzing user activities on the cloud platform is crucial to improving user experience and security management. By building an inverted index and combining it with a hash algorithm, the system can instantly track user activities, such as file access, service requests, etc., and analyze these behavior patterns to quickly discover abnormal behaviors and prevent potential security threats.

[0235] For example, considering the continuous flow of logs and the continuous generation of user behaviors in cloud services, the monitoring system needs to efficiently process, classify and store a large number of log and behavior data records. The system implements monitoring through the following steps:

[0236] After collecting real-time log data and user behavior records, they are divided into separately indexable terms.

[0237] A special hash algorithm (such as MurmurHash) is used to generate a hash fingerprint for each word and stored in an inverted index structure.

[0238] Update and optimize indexes in real time, such as dynamically adjusting the size of release lists and performing batch merge operations, to maintain the timeliness and accuracy of data.

[0239] For sensitive information, such as user identity, static hash functions are applied for encoding to enhance data security and privacy protection.

[0240] Through this approach, cloud service providers can provide their users with more reliable service guarantees while effectively maintaining the privacy of user data and the overall security of the system.

[0241] In addition, for the actual scenario of storing user log files in a cloud service environment: In order to ensure the security of stored data and efficiently process logs, the specific implementation steps include:

[0242] Step 1: Data collection and preprocessing

[0243] When users interact with services, the cloud service provider's virtual machines collect users' log data in real time through monitoring software, such as access records, error reports, and system calls.

[0244] Data preprocessing includes organizing log formats, filtering out useless information, and extracting key data points such as timestamps, event types, and resource identifiers.

[0245] Step 2: Create word segmentation and inverted index

[0246] Use text analysis technology to decompose log content into a series of tokens to facilitate faster data search and analysis in subsequent steps.

[0247] Create an inverted index, apply the MurmurHash algorithm to each word to generate a unique hash fingerprint, and store the result in the index structure. The inverted index is crucial for efficient log data retrieval.

[0248] Step 3: Monitoring point setting and data flow tracking

[0249] Set up monitoring points to track the inflow rate and pattern of log data in real time, as well as any potential abnormal behavior.

[0250] Through continuous analysis, the monitoring system can detect and defend against potential security threats, such as frequent login attempts, unauthorized data access, or abnormal system operation behavior.

[0251] Step 4: Online deduplication and efficient storage

[0252] By merging redundant publication lists online, unnecessary storage space usage is reduced while maintaining the latest data version.

[0253] The publishing list can be dynamically resized based on the data load, optimizing memory usage and query response time.

[0254] Step 5: Sensitive information handling and secure coding

[0255] A mapping operation is performed on the identified sensitive tokens, such as those containing user personal information, to convert them into references to non-personal identifiers.

[0256] These references are encoded using static hash functions such as SHA-256 or MD5 to generate irreversible unique codes, strengthening data security and privacy protection.

[0257] Through the above steps, the secure cloud storage monitoring system efficiently completes the processing and storage of log data while ensuring the privacy of user data, enabling operators to easily conduct post-analysis of past events and improving the overall security of data storage in the cloud environment.

[0258] The present application provides a secure cloud storage monitoring system and method based on a virtual machine, and introduces a hash-based inverted index structure. This special hash memory data structure can greatly improve the storage and retrieval efficiency of an enterprise's unstructured data, and is very suitable for the scenario of massive unstructured data management in an enterprise. In addition, in the embodiment of the present application, a static hash function is used to encode the stored data, providing an irreversible and non-directly associated reference (identification) for privacy-sensitive words, greatly improving the security and privacy protection capabilities of the stored data, and meeting the compliance requirements of enterprise data management.

[0259] Embodiment 2

[0260] Figure 2 A schematic diagram of a secure cloud storage monitoring system based on a virtual machine provided in an embodiment of the present disclosure is provided. The secure cloud storage monitoring system based on a virtual machine 200 includes:

[0261] An acquisition module 201 is used to acquire data to be stored;

[0262] The optimization module 202 is used to establish a hash-based memory data structure to optimize the data to be stored;

[0263] The encoding module 203 is used to encode the data to be stored by using a first static hash function, and store the encoded data to be stored.

[0264] Optionally, the optimization module 202 is used to establish a hash-based memory data structure, and optimize the data to be stored based on the memory data structure, including:

[0265] Establishing a hash-based inverted index structure to perform inverted indexing on the data to be stored;

[0266] A monitoring point is set up to continuously track the inflow rate and data pattern of the data to be stored, and redundant publishing lists are merged online, where the publishing lists are used to describe the position and frequency information of any word in the inverted index structure.

[0267] Optionally, establishing a hash-based inverted index structure to perform inverted indexing on the data to be stored includes:

[0268] Decomposing the data to be stored into a plurality of word units;

[0269] Generate a hash fingerprint of each word using a hash algorithm, wherein the hash algorithm is MurmurHash or CityHash;

[0270] Add the hash fingerprint and corresponding ID of each word to the inverted list.

[0271] Optionally, merge redundant release lists online, including:

[0272] Identify duplicate data entries in the release list based on the hash fingerprint of each token;

[0273] Duplicate data entries are merged so that only the latest version data is retained for each data entry.

[0274] Optionally, the system further comprises:

[0275] An adjustment module, used for dynamically adjusting the size of the publishing list according to data load conditions;

[0276] The merging module is used to formulate a dynamic merging strategy, and based on the dynamic merging strategy, batch execute the merging operation of the publishing list based on the hash fingerprint of each word element within a preset time.

[0277] Optionally, encoding the data to be stored by using a first static hash function includes:

[0278] Judging the word units and identifying sensitive word units;

[0279] Mapping the sensitive word to a reference to a non-personal identifier;

[0280] The reference is encoded using a first static hash function.

[0281] Optionally, encoding the reference using a first static hash function comprises:

[0282] Constructing a first static hash function, wherein the first static hash function is SHA-256 or MD5;

[0283] Using the first static hash function, converting the sensitive word into a hash value;

[0284] The hash value is used to construct a publishing list corresponding to the sensitive word.

[0285] Optionally, the system further comprises:

[0286] An evaluation module, used for evaluating the validity period of the first static hash function;

[0287] A replacement module is used to replace the first static hash function with a second static hash function before the end of the validity period.

[0288] Optionally, the system further comprises:

[0289] A compression module is used to compress the encoded data to be stored.

[0290] The present application provides a secure cloud storage monitoring system based on a virtual machine, and introduces a hash-based inverted index structure. This special hash memory data structure can greatly improve the storage and retrieval efficiency of an enterprise's unstructured data, and is very suitable for the scenario of massive unstructured data management in an enterprise. In addition, in the embodiment of the present application, a static hash function is used to encode the stored data, providing an irreversible and non-directly associated reference (identification) for privacy-sensitive words, greatly improving the security and privacy protection capabilities of the stored data, and meeting the compliance requirements of enterprise data management.

[0291] The system of the embodiments of the present disclosure can execute the method provided by the embodiments of the present disclosure, and the implementation principles are similar. The actions performed by each module in the system of each embodiment of the present disclosure correspond to the steps in the method of each embodiment of the present disclosure. For the detailed functional description of each module of the system, please refer to the description in the corresponding method shown in the previous text, which will not be repeated here.

[0292] The above is only an optional implementation method for some implementation scenarios of the present disclosure. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the scheme of the present disclosure, other similar implementation methods based on the technical ideas of the present disclosure are also within the protection scope of the embodiments of the present disclosure.

Claims

1. A secure cloud storage monitoring method based on a virtual machine, characterized in that: The method comprises: The virtual machine obtains the data to be stored; Establishing a hash-based memory data structure, and optimizing the data to be stored based on the memory data structure; Encoding the data to be stored by using a first static hash function, and storing the encoded data to be stored; The establishing of a hash-based memory data structure and optimizing the data to be stored based on the memory data structure include: Establishing a hash-based inverted index structure to perform inverted indexing on the data to be stored; Setting up monitoring points to continuously track the inflow rate and data pattern of the data to be stored, and online merging redundant publishing lists, where the publishing lists are used to describe the position and frequency information of any word in the inverted index structure; The step of establishing a hash-based inverted index structure and performing inverted indexing on the data to be stored includes: Decomposing the data to be stored into a plurality of word units; Use hash algorithm to generate hash fingerprint of each word; Add the hash fingerprint and corresponding ID of each word to the inverted list; The encoding of the data to be stored by using a first static hash function includes: Judging the word units and identifying sensitive word units; Mapping the sensitive word to a reference to a non-personal identifier; The reference is encoded using a first static hash function.

2. The method according to claim 1, characterized in that The hash algorithm is MurmurHash or CityHash.

3. The method according to claim 1, characterized in that Online merging of redundant release lists, including: Identify duplicate data entries in the release list based on the hash fingerprint of each token; Duplicate data entries are merged so that only the latest version data is retained for each data entry.

4. The method according to claim 3, characterized in that The method further comprises: Dynamically adjust the size of the publishing list according to data load conditions; A dynamic merging strategy is formulated, and based on the dynamic merging strategy, a merging operation of the publishing list is executed in batches within a preset time based on the hash fingerprint of each word.

5. The method according to claim 1, characterized in that Encoding the reference using a first static hash function includes: Constructing a first static hash function, wherein the first static hash function is SHA-256 or MD5; Using the first static hash function, converting the sensitive word into a hash value; The hash value is used to construct a publishing list corresponding to the sensitive word.

6. The method according to claim 5, characterized in that The method further comprises: evaluating the validity period of the first static hash function; Before the expiration of the validity period, the first static Hash function is replaced by a second static Hash function.

7. The method according to claim 1, characterized in that Before storing the encoded data to be stored, the method further includes: The encoded data to be stored is compressed.

8. A secure cloud storage monitoring system based on a virtual machine, characterized in that: Executing the method according to any one of claims 1 to 7, comprising: An acquisition module, used to acquire data to be stored; An optimization module, used to establish a hash-based memory data structure to optimize the data to be stored; The encoding module is used to encode the data to be stored by using a first static hash function, and store the encoded data to be stored.

Images