Internet big data information collection and storage method and system based on machine learning

Abstract

The invention discloses an internet big data information collection and storage method and system based on machine learning, and relates to the technical field of data security. Collecting the original data stream and inputting the original data stream into a pre-trained lightweight real-time intrusion detection model to obtain an effective data stream; performing semantic association partitioning processing on the effective data stream through a preset algorithm to obtain a plurality of data word blocks; for each data word block, based on the encryption key, performing encryption processing on the data word block through a preset encryption algorithm to obtain a ciphertext data block, and synchronously generating a searchable encryption index unit of the ciphertext data block; constructing a Merkel tree based on the hash values of all the ciphertext data blocks; and storing the Merkel tree and all the ciphertext data blocks in a storage block. By screening out unsafe data and matching with a design scheme of semantic association partitioning, data block level encryption and searchable encryption indexes, the dual problems of insufficient security and low efficiency existing in Internet data storage are thoroughly solved.

Description

Technical Field

[0001] This invention belongs to the field of data security technology, specifically relating to a method and system for collecting and storing Internet big data information based on machine learning. Background Technology

[0002] With the rapid development of internet technology, big data is experiencing explosive growth. Internet data sources encompass multiple scenarios, including social media platforms, news media, industry websites, and IoT terminals. The resulting data streams are characterized by massive volume, heterogeneity, real-time nature, and complexity, posing significant challenges to the efficient collection and secure storage of big data. Current methods for collecting and storing internet big data still suffer from numerous technical shortcomings in practical applications, making it difficult to meet the demands for massive, real-time, and secure storage.

[0003] First, in the big data collection stage, the raw data stream of the Internet contains a large amount of malicious intrusion data, invalid and redundant data, and abnormal interference data. Traditional data collection methods often use simple rule-based filtering to filter the raw data stream, lacking efficient real-time detection capabilities. They cannot accurately identify intrusion data and abnormal data hidden in the massive data stream. This not only leads to a large amount of redundancy in the subsequently stored data and occupies too much storage resources, but may also cause security risks such as data leakage and storage system paralysis due to the mixing of malicious intrusion data. At the same time, it reduces the accuracy and effectiveness of data collection, making it difficult to support subsequent data processing and applications.

[0004] Secondly, in the data segmentation process, existing technologies mostly use an equal division method based on data size to segment data without considering the semantic relationships between data. This results in the data stream after segmentation lacking logic and coherence. This defect not only requires the processing of a large number of irrelevant data blocks during subsequent data encryption, retrieval, and retrieval processes, significantly reducing data processing efficiency and increasing system computing load, but also directly affects the encryption effect and security of subsequent data security storage, ultimately leading to low data storage security and efficiency. Summary of the Invention

[0005] The purpose of this invention is to solve the problems of low security and efficiency in data storage when dealing with Internet data storage, and to propose a method and system for collecting and storing Internet big data information based on machine learning.

[0006] In a first aspect of this invention, a method for collecting and storing big data information on the Internet based on machine learning is first proposed, the method comprising: Raw data streams are collected from internet data sources and then input into a pre-trained lightweight real-time intrusion detection model to obtain effective data streams. The effective data stream is semantically associated and segmented using a preset algorithm to obtain multiple data blocks. For each data block, based on the encryption key, the data block is encrypted using a preset encryption algorithm to obtain a ciphertext data block, and a searchable encrypted index unit for the ciphertext data block is generated simultaneously; the encryption key is generated based on preset rules; the index unit contains encrypted tags based on keywords; Construct a Merkle tree based on the hash values ​​of all ciphertext data blocks; Save the Merkle tree and all ciphertext data blocks to the storage block.

[0007] This solution filters insecure data at the source through lightweight real-time intrusion detection. Combined with semantic association block partitioning, data block-level encryption, and searchable encrypted indexes, it achieves full-process security for Internet data storage through multi-dimensional encryption protection, Merkle tree integrity verification, and block storage. It also reduces processing resource consumption through lightweight detection, optimizes data organization logic through semantic partitioning, and solves the problem of encrypted data retrieval through searchable indexes. Meanwhile, block processing supports parallel storage and maintenance, effectively balancing the security of Internet data storage with processing and storage efficiency. It solves the dual problems of insufficient security and low efficiency in Internet data storage across the entire chain from data filtering, processing, encryption to storage.

[0008] Optionally, inputting the raw data stream into a pre-trained lightweight real-time intrusion detection model to obtain an effective data stream includes: The original data stream is sorted in chronological order to obtain a time-series sequence, and the time-series sequence is segmented according to a preset time period to obtain multiple sub-time-series sequences; Substituting the target sub-time series sequence into a bidirectional GRU layer yields the fused hidden features of each time sequence in the target sub-time series sequence; the target sub-time series sequence is any one of multiple sub-time series sequences; The weights corresponding to each fused hidden feature are obtained by linear mapping of the fused hidden features across all time series and normalization by Softmax. The global key features are obtained by weighted fusion of the fused hidden features according to the weights corresponding to each fused hidden feature. The global key features are substituted into the residual network to obtain residual features, and the residual features are substituted into the Softmax classifier to obtain the data anomaly probability. If the probability of data anomaly is greater than a preset anomaly threshold, the target sub-time series is removed, all removed sub-time series are obtained, and the effective data stream is obtained by splicing them together according to time.

[0009] This solution removes only abnormal sub-sequences exceeding the threshold and concatenates them with compliant sequences to obtain the valid data stream. It leverages a lightweight bidirectional GRU and residual network architecture to fully exploit the temporal correlation and global key features of internet data streams while ensuring real-time performance. This improves the accuracy of intrusion and abnormal data detection, avoids the accidental deletion of valid data, and adapts to the real-time detection requirements of streaming data through time-series segmentation processing. The lightweight model design reduces resource consumption in data processing, balancing accuracy, real-time performance, and efficiency in intrusion detection. It accurately filters insecure data at the data source, building a solid first line of defense for the security of subsequent data storage, while ensuring the integrity and processing efficiency of the valid data stream and preventing invalid data from consuming subsequent storage and processing resources.

[0010] Optionally, the effective data stream can be semantically correlated and segmented using a preset algorithm to obtain multiple data blocks, including: Semantic relevance features are extracted from each data item in the effective data stream to obtain a structured data matrix; Through formula Calculate the fuzzy data likelihood of each data item in the effective data stream; in, For fuzzy data likelihood, it represents the core feature vector in a given semantic block. and fuzzy semantic relationships At that time, structured data matrix The probability of conforming to this cluster structure, where h is the fuzzification parameter. This is a block index, with values ​​ranging from 1 to G. For the first Data points in semantic blocks, where E is a simplified representation of the clustering prototype and The meaning is the same. For the first The overall membership degree of data within a class block, where λ is the data index within the block, with a value ranging from 1 to H. For the first The individual membership degree of the λth data point in the class block, γ=v is the mean constraint condition, γ is the mean factor of the data in the block, v is the preset mean threshold, and d is the correlation strength between the data point and the block center. By maximizing the likelihood of fuzzy data Update clustering prototype membership degree The process continues until multiple data blocks are obtained through convergence.

[0011] This solution first extracts semantically relevant features from effective data streams to construct a structured data matrix. Then, it quantifies the semantic correlation of data using a fuzzy data likelihood formula that integrates multiple dimensions. With the goal of maximizing this likelihood, it iteratively optimizes clustering until convergence is achieved, thus completing the block division. This approach not only breaks through the limitations of traditional block division and accurately achieves semantic aggregation of data, ensuring high semantic correlation within the same block and clear boundaries between blocks, but also adapts to the batch processing requirements of streaming data through algorithmic standardization. Furthermore, it provides a reasonable granularity foundation for subsequent block-level encryption and searchable index construction, improving the targeting and efficiency of subsequent storage operations and reducing resource waste.

[0012] Optionally, based on the encryption key, the data block is encrypted using a preset encryption algorithm to obtain a ciphertext data block, and a searchable encrypted index unit for the ciphertext data block is generated simultaneously, including: Based on the identifier ID and encryption key of the data block, an algorithm is used. Obtain the data block subkey; Where SK is the subkey of the data block, HMAC is the hash-based message authentication code, SHA3 is the Secure Hash Algorithm 3, MK is the encryption key, BlockID is the identifier ID of the data block, and TS is the timestamp. The target key is obtained by encrypting the data block subkey twice using secret elliptic curve cryptography. Based on the target key, the data block is encrypted using a preset encryption algorithm to obtain a ciphertext data block; Keywords are extracted from the data block, the extracted keywords are encrypted with the encryption key, and then an encrypted tag is generated by SHA3-256 hashing to obtain a searchable encrypted index unit.

[0013] This solution employs a two-layer encryption mechanism—master key-derived subkeys and quadratic elliptic curve encryption—combined with timestamps to achieve block-level personalized and highly secure encryption. This significantly enhances the anti-cracking and anti-tampering capabilities of encrypted data blocks. Simultaneously generated searchable encrypted index units enable accurate and rapid retrieval of encrypted data while ensuring that keyword encryption does not reveal data semantics. This addresses the pain point of difficult encrypted data retrieval. Furthermore, the design of simultaneous encryption and index generation adapts to the processing rhythm of semantic blocks, improving the overall data storage encryption efficiency and subsequent access efficiency, thus balancing data storage security and retrieval practicality.

[0014] Optionally, constructing a Merkle tree based on the hash values ​​of all ciphertext data blocks includes: For each ciphertext data block, a unique hash value is calculated using the SHA-3-512 hash algorithm; Based on the IPFS content addressing mechanism, a unique identifier for each ciphertext data block is generated by mapping a unique hash value. Obtain the hash values ​​of all ciphertext blocks, sort them in order, and group them into multiple groups of hash values ​​by a preset number; Calculate the hash value of the intermediate node in each group as the next level node of the tree, and repeat this process until all nodes are aggregated into a root node hash value to obtain the Merkle tree.

[0015] This solution generates unique hash values ​​and maps them to unique identifiers using the IPFS content addressing mechanism. It then iteratively calculates intermediate nodes by grouping the hash values ​​until a root node is generated, constructing a Merkle tree. This achieves both unique identification and precise location of encrypted data blocks. Furthermore, the hierarchical hash structure of the Merkle tree allows for rapid verification of the integrity and authenticity of single or multiple encrypted data blocks. Any data tampering can be detected promptly through node hash value comparison. The tree structure is also adapted to the batch storage characteristics of encrypted data blocks, connecting with the previously mentioned block-level encryption and semantic segmentation. This strengthens data storage security against tampering and loss while improving the efficiency of data integrity verification and addressing access, providing a secure verification foundation for subsequent block storage.

[0016] In a second aspect of this invention, a machine learning-based internet big data information collection and storage system is proposed, comprising: The data filtering module is used to collect raw data streams from Internet data sources and input the raw data streams into a pre-trained lightweight real-time intrusion detection model to obtain effective data streams. The data segmentation module is used to perform semantic association segmentation processing on the effective data stream using a preset algorithm to obtain multiple data blocks. The data encryption module is used to encrypt each data block using a preset encryption algorithm based on an encryption key to obtain a ciphertext data block, and simultaneously generate a searchable encrypted index unit for the ciphertext data block; the encryption key is generated based on preset rules; the index unit contains encrypted tags based on keywords; The data query tree building module is used to construct a Merkle tree based on the hash values ​​of all encrypted data blocks; The data storage module is used to save the Merkle tree and all encrypted data blocks into the storage area.

[0017] Optionally, the data filtering module: The data sorting module is used to sort the original data stream in chronological order to obtain a time-series sequence, and to segment the time-series sequence into multiple sub-time-series sequences according to a preset time period. The fusion hidden feature extraction module is used to substitute the target sub-time series sequence into a bidirectional GRU layer to obtain the fusion hidden feature of each time sequence in the target sub-time series sequence; the target sub-time series sequence is any one of multiple sub-time series sequences; The feature weight determination module is used to perform linear mapping on the fused hidden features of all time series and normalize them using Softmax to obtain the weight corresponding to each fused hidden feature. The global key feature extraction module is used to perform weighted fusion of the fused hidden features according to the weights corresponding to each fused hidden feature to obtain global key features. The data anomaly probability determination module is used to substitute the global key features into the residual network to obtain residual features, and substitute the residual features into the Softmax classifier to obtain the data anomaly probability. The effective data stream generation module is used to remove the target sub-time series if the probability of data anomaly is greater than a preset anomaly threshold, obtain all the removed sub-time series, and concatenate them according to time to obtain the effective data stream.

[0018] Optionally, the data segmentation module includes: The structured data matrix extraction module is used to extract semantically relevant features from each piece of data in the effective data stream to obtain a structured data matrix; The fuzzy data likelihood calculation module is used to calculate the likelihood of data using a formula. Calculate the fuzzy data likelihood of each data item in the effective data stream; in, For fuzzy data likelihood, it represents the core feature vector in a given semantic block. and fuzzy semantic relationships At that time, structured data matrix The probability of conforming to this cluster structure, where h is the fuzzification parameter. This is a block index, with values ​​ranging from 1 to G. For the first Data points in semantic blocks, where E is a simplified representation of the clustering prototype and The meaning is the same. For the first The overall membership degree of data within a class block, where λ is the data index within the block, with a value ranging from 1 to H. For the first The individual membership degree of the λth data point in the class block, γ=v is the mean constraint condition, γ is the mean factor of the data in the block, v is the preset mean threshold, and d is the correlation strength between the data point and the block center. The fuzzy data likelihood update module is used to update the fuzzy data likelihood by maximizing the fuzzy data likelihood. Update clustering prototype membership degree The process continues until multiple data blocks are obtained through convergence.

[0019] Optionally, the data encryption module includes: The data block subkey determination module is used to determine the data block's identifier ID and encryption key through an algorithm. Obtain the data block subkey; Where SK is the subkey of the data block, HMAC is the hash-based message authentication code, SHA3 is the Secure Hash Algorithm 3, MK is the encryption key, BlockID is the identifier ID of the data block, and TS is the timestamp. The secondary encryption module is used to perform secondary encryption on the data block subkey using secret elliptic curve cryptography to obtain the target key; The ciphertext data block generation module is used to encrypt the data block using a preset encryption algorithm based on the target key to obtain a ciphertext data block; A searchable encrypted index unit generation module is used to extract keywords from the data block, encrypt the extracted keywords with the encryption key, and then generate an encrypted tag through SHA3-256 hashing to obtain a searchable encrypted index unit.

[0020] Optionally, the data query tree construction module includes: The unique hash value calculation module is used to calculate a unique hash value for each ciphertext data block using the SHA-3-512 hash algorithm; The unique identifier generation module is used for the IPFS-based content addressing mechanism to generate a unique identifier for each ciphertext data block through unique hash value mapping. The ciphertext block grouping module is used to obtain the hash values ​​of all ciphertext blocks, sort them in order, and group them into multiple groups of hash values ​​by a preset number; The Merkle tree building module is used to calculate the hash value of the intermediate node in each group as the next level node of the tree. This process is repeated until all nodes are aggregated into a root node hash value to obtain the Merkle tree.

[0021] The beneficial effects of this invention are: This invention proposes a machine learning-based method for collecting and storing big data information on the internet. It filters out insecure data at the source through lightweight real-time intrusion detection, combined with a design scheme of semantic association-based segmentation, data block-level encryption, and searchable encrypted indexes. On the one hand, it relies on multi-dimensional encryption protection, Merkle tree integrity verification, and block storage to achieve full-process security protection for internet data storage. On the other hand, it reduces resource consumption in data processing through lightweight detection, optimizes data organization logic through semantic segmentation, and overcomes the pain point of encrypted data retrieval with searchable indexes. Simultaneously, the segmentation processing mode supports parallel storage and maintenance of data, effectively balancing the security of internet data storage with processing and storage efficiency. From data filtering, processing, encryption to storage, this method thoroughly solves the dual problems of insufficient security and low efficiency in internet data storage. Attached Figure Description

[0022] The invention will now be further described with reference to the accompanying drawings.

[0023] Figure 1 A flowchart illustrating a machine learning-based method for collecting and storing big data information on the Internet, provided as an embodiment of the present invention; Figure 2 A flowchart of a data stream segmentation method provided in an embodiment of the present invention; Figure 3 This is a framework diagram of an Internet big data information collection and storage system based on machine learning, provided for an embodiment of the present invention. Detailed Implementation

[0024] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0025] Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] This invention provides a method for collecting and storing big data information from the internet based on machine learning. See also... Figure 1 , Figure 1 A flowchart illustrating a machine learning-based method for collecting and storing big data information on the internet, provided as an embodiment of the present invention. The method includes the following steps: S101 collects raw data streams from Internet data sources and inputs the raw data streams into a pre-trained lightweight real-time intrusion detection model to obtain effective data streams; S102, using a preset algorithm to perform semantic association and block segmentation on the effective data stream to obtain multiple data blocks; S103, for each data block, based on the encryption key, the data block is encrypted using a preset encryption algorithm to obtain a ciphertext data block, and a searchable encrypted index unit for the ciphertext data block is generated simultaneously. S104, Construct a Merkle tree based on the hash values ​​of all ciphertext data blocks; S105, save the Merkle tree and all ciphertext data blocks to the storage block; The encryption key is generated based on preset rules; the index unit contains encryption tags based on keywords.

[0027] In one implementation, the encryption key is generated based on the AES-256 encryption algorithm, and the preset rule is the rule of the AES-256 encryption algorithm; the Merkle tree stores the storage location of the corresponding ciphertext data block.

[0028] In one embodiment, see Figure 2 , Figure 2 A flowchart of a data stream segmentation method is provided, which obtains an effective data stream by inputting the raw data stream into a pre-trained lightweight real-time intrusion detection model, including: S1021, Sort the original data stream in chronological order to obtain a time series sequence, and divide the time series sequence into segments according to a preset time period to obtain multiple sub-time series sequences; S1022, Substitute the target sub-time series into the bidirectional GRU layer to obtain the fused hidden features of each time series in the target sub-time series; S1023, perform linear mapping on the fused hidden features of all time series and normalize them using Softmax to obtain the weights corresponding to each fused hidden feature; S1024, The fusion hidden features are weighted and fused according to the weights corresponding to each fusion hidden feature to obtain the global key features; S1025, substitute the global key features into the residual network to obtain residual features, and substitute the residual features into the Softmax classifier to obtain the data anomaly probability; S1026, If the probability of data anomaly is greater than the preset anomaly threshold, the target sub-time series is removed, all removed sub-time series are obtained, and the effective data stream is obtained by splicing them together according to time. The target sub-time series sequence is any one of multiple sub-time series sequences.

[0029] In one implementation, both the preset time period and the preset anomaly threshold are determined by technical personnel; if the probability of data anomaly is less than or equal to the preset anomaly threshold, the data is retained.

[0030] In one implementation, the bidirectional GRU layer includes a forward GRU layer and a backward GRU layer. The temporal sequence is substituted into the forward GRU layer, and the sequence is processed sequentially to obtain the forward hidden features corresponding to each time step in the temporal sequence. The temporal sequence is then substituted into the backward GRU layer, and the sequence is processed in reverse order to obtain the backward hidden features corresponding to each time step in the temporal sequence. The forward and backward hidden features corresponding to each time step are concatenated to obtain the fused hidden features. The hidden layer dimension of the bidirectional GRU layer is 128, the GRU gate activation function is Sigmoi, and the hidden state is Tanh.

[0031] In one implementation, existing models often use bidirectional GRU layers for temporal data feature extraction, and attention mechanisms are mostly used to strengthen the weights of key features (such as in machine translation and image segmentation). ResNet primarily addresses the gradient decay problem in deep networks. However, connecting these three elements into a closed loop of temporal feature extraction, key attack feature focusing, and deep feature classification is specifically tailored for security scenarios with strong temporal characteristics and complex attack patterns, such as network traffic.

[0032] In one embodiment, obtaining multiple data blocks by semantic association and block segmentation of a valid data stream using a preset algorithm includes: Semantic relevance features are extracted from each data point in the valid data stream to obtain a structured data matrix; Through formula Calculate the fuzzy data likelihood of each data point in the valid data stream; in, For fuzzy data likelihood, it represents the core feature vector in a given semantic block. and fuzzy semantic relationships At that time, structured data matrix The probability of conforming to this cluster structure, where h is the fuzzification parameter. This is a block index, with values ​​ranging from 1 to G. For the first Data points in semantic blocks, where E is a simplified representation of the clustering prototype and The meaning is the same. For the first The overall membership degree of data within a class block, where λ is the data index within the block, with a value ranging from 1 to H. For the first The individual membership degree of the λth data point in the class block, γ=v is the mean constraint condition, γ is the mean factor of the data in the block, v is the preset mean threshold, and d is the correlation strength between the data point and the block center. By maximizing the likelihood of fuzzy data Update clustering prototype membership degree The process continues until multiple data blocks are obtained through convergence.

[0033] In one implementation, semantically relevant features are extracted from each piece of data in the effective data stream to obtain a structured data matrix. Each piece of data is then substituted into a semantic extraction model to obtain the structured data matrix. The semantic extraction model is an existing model, such as Sentence-BERT, SimCSE, ContrastiveBERT, BERT-Of-Theseus, etc.

[0034] In one implementation, the given semantic blocks are pre-defined by technicians, who also initialize the core feature vectors of the blocks. The core feature vector To randomly select a preset number (determined by technical personnel) of data points from the structured data matrix as the initial core feature vector, the core feature vector of the block is updated in weight according to the membership degree of each data point to the current block, that is, the fuzzy semantic relationship. Based on the updated core feature vector, the semantic similarity of each data point to each block is recalculated. The similarity is calculated by Matusita distance and then transformed into a new membership degree, so that data with better semantic matching are more concentrated in the same block.

[0035] In one implementation, if the change in the membership degree of the data to the block is less than 0.01 during the iteration process, then the fuzzy data likelihood is... When the maximum value is reached, the block at this point is called a data block.

[0036] In one embodiment, based on an encryption key, a ciphertext data block is obtained by encrypting a data block using a preset encryption algorithm, and a searchable encrypted index unit for the ciphertext data block is generated simultaneously, comprising: Based on the identifier ID and encryption key of the data block, an algorithm is used. Obtain the data block subkey; Where SK is the subkey of the data block, HMAC is the hash-based message authentication code, SHA3 is the Secure Hash Algorithm 3, MK is the encryption key, BlockID is the identifier ID of the data block, and TS is the timestamp. The target key is obtained by encrypting the data block subkey twice using secret elliptic curve cryptography. Based on the target key, the data blocks are encrypted using a preset encryption algorithm to obtain ciphertext data blocks; Keywords are extracted from data blocks, encrypted with an encryption key, and then encrypted tags are generated using SHA3-256 hashing to obtain searchable encrypted index units.

[0037] In one implementation, the encryption key is generated based on the AES-256 encryption algorithm; the preset encryption algorithm is the AES-256-GCM algorithm.

[0038] In one implementation, keywords are extracted from data blocks by word frequency inverse document frequency.

[0039] In one embodiment, constructing a Merkle tree based on the hash values ​​of all ciphertext data blocks includes: For each ciphertext data block, a unique hash value is calculated using the SHA-3-512 hash algorithm; Based on the IPFS content addressing mechanism, a unique identifier for each ciphertext data block is generated by mapping a unique hash value. Obtain the hash values ​​of all ciphertext blocks, sort them in order, and group them into multiple groups of hash values ​​by a preset number; Calculate the hash value of the intermediate node in each group as the next level node of the tree, and repeat this process until all nodes are aggregated into a root node hash value to obtain the Merkle tree.

[0040] In one implementation, the pairwise hash aggregation logic of the traditional Merkle tree is optimized by adopting dynamic grouping and batch hashing mechanism. When the number of ciphertext blocks is odd, the last hash value is not repeated. Instead, zero-padding is used to complete the grouping to avoid tree structure imbalance.

[0041] In one implementation, the preset number of groupings is determined by technical personnel. For example, if four leaf nodes are aggregated into one intermediate node, then the hash value of the intermediate node is... Ha, Hb, Hc, and Hd are the four leaf nodes. It involves byte concatenation; during the construction process, the parent node pointer and hierarchy information of each node are recorded, generating tree structure metadata including tree height, total number of nodes, and leaf node mapping relationship, which are used for subsequent integrity verification.

[0042] In one implementation, once the Merkle tree is constructed, the searchable cryptographic index unit is mapped to the corresponding position in the Merkle tree.

[0043] Based on the same inventive concept, this invention also provides an internet big data information collection and storage system based on machine learning. See also Figure 2 , Figure 2 A framework diagram of an Internet big data information collection and storage system based on machine learning, provided for embodiments of the present invention, includes: The data filtering module is used to collect raw data streams from Internet data sources and input the raw data streams into a pre-trained lightweight real-time intrusion detection model to obtain effective data streams. The data segmentation module is used to perform semantic association segmentation on the effective data stream using a preset algorithm to obtain multiple data blocks. The data encryption module is used to encrypt each data block using a preset encryption algorithm based on an encryption key to obtain a ciphertext data block, and simultaneously generate a searchable encrypted index unit for the ciphertext data block; the encryption key is generated based on preset rules; the index unit contains encrypted tags based on keywords; The data query tree building module is used to construct a Merkle tree based on the hash values ​​of all encrypted data blocks; The data storage module is used to save the Merkle tree and all encrypted data blocks into the storage area.

[0044] This invention provides a machine learning-based internet big data information collection and storage system. It filters insecure data at the source through lightweight real-time intrusion detection, and combines semantic association-based segmentation, data block-level encryption, and searchable encrypted indexes. This achieves full-process security for internet data storage through multi-dimensional encryption protection, Merkle tree integrity verification, and block storage. Furthermore, it reduces processing resource consumption through lightweight detection, optimizes data organization logic through semantic segmentation, and solves the problem of encrypted data retrieval through searchable indexes. Simultaneously, segmented processing supports parallel storage and maintenance, effectively balancing the security of internet data storage with processing and storage efficiency. From data filtering, processing, encryption to storage, it solves the dual problems of insufficient security and low efficiency in internet data storage.

[0045] The foregoing has described one embodiment of the present invention in detail, but this content is merely a preferred embodiment and should not be considered as limiting the scope of the present invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the scope of the claims of this invention.

Claims

1. A method for collecting and storing big data information on the Internet based on machine learning, characterized in that: The method includes: Raw data streams are collected from internet data sources and then input into a pre-trained lightweight real-time intrusion detection model to obtain effective data streams. The effective data stream is semantically associated and segmented using a preset algorithm to obtain multiple data blocks. For each data block, based on the encryption key, the data block is encrypted using a preset encryption algorithm to obtain a ciphertext data block, and a searchable encrypted index unit for the ciphertext data block is generated simultaneously; the encryption key is generated based on preset rules; the index unit contains encrypted tags based on keywords; Construct a Merkle tree based on the hash values ​​of all ciphertext data blocks; Save the Merkle tree and all ciphertext data blocks to the storage block.

2. The method for collecting and storing Internet big data information based on machine learning according to claim 1, characterized in that, The raw data stream is input into a pre-trained lightweight real-time intrusion detection model to obtain an effective data stream, including: The original data stream is sorted in chronological order to obtain a time-series sequence, and the time-series sequence is segmented according to a preset time period to obtain multiple sub-time-series sequences; Substituting the target sub-time series sequence into a bidirectional GRU layer yields the fused hidden features of each time sequence in the target sub-time series sequence; the target sub-time series sequence is any one of multiple sub-time series sequences; The weights corresponding to each fused hidden feature are obtained by linear mapping of the fused hidden features across all time series and normalization by Softmax. The global key features are obtained by weighted fusion of the fused hidden features according to the weights corresponding to each fused hidden feature. The global key features are substituted into the residual network to obtain residual features, and the residual features are substituted into the Softmax classifier to obtain the data anomaly probability. If the probability of data anomaly is greater than a preset anomaly threshold, the target sub-time series is removed, all removed sub-time series are obtained, and the effective data stream is obtained by splicing them together according to time.

3. The method for collecting and storing Internet big data information based on machine learning according to claim 1, characterized in that, Multiple data blocks are obtained by semantic association and block segmentation of the effective data stream using a preset algorithm, including: Semantic relevance features are extracted from each data item in the effective data stream to obtain a structured data matrix; Through formula Calculate the fuzzy data likelihood of each data item in the effective data stream; in, For fuzzy data likelihood, it represents the core feature vector in a given semantic block. and fuzzy semantic relationships At that time, structured data matrix The probability of conforming to this cluster structure, where h is the fuzzification parameter. This is a block index, with values ​​ranging from 1 to G. For the first Data points in semantic blocks, where E is a simplified representation of the clustering prototype and The meaning is the same. For the first The overall membership degree of data within a class block, where λ is the data index within the block, with a value ranging from 1 to H. For the first The individual membership degree of the λth data point in the class block, γ=v is the mean constraint condition, γ is the mean factor of the data in the block, v is the preset mean threshold, and d is the correlation strength between the data point and the block center. By maximizing the likelihood of fuzzy data Update clustering prototype membership degree The process continues until multiple data blocks are obtained through convergence.

4. The method for collecting and storing Internet big data information based on machine learning according to claim 1, characterized in that, Based on the encryption key, a ciphertext data block is obtained by encrypting data blocks using a preset encryption algorithm, and a searchable encrypted index unit for the ciphertext data block is generated simultaneously, including: Based on the identifier ID and encryption key of the data block, an algorithm is used. Obtain the data block subkey; Where SK is the subkey of the data block, HMAC is the hash-based message authentication code, SHA3 is the Secure Hash Algorithm 3, MK is the encryption key, BlockID is the identifier ID of the data block, and TS is the timestamp. The target key is obtained by encrypting the data block subkey twice using secret elliptic curve cryptography. Based on the target key, the data block is encrypted using a preset encryption algorithm to obtain a ciphertext data block; Keywords are extracted from the data block, the extracted keywords are encrypted with the encryption key, and then an encrypted tag is generated by SHA3-256 hashing to obtain a searchable encrypted index unit.

5. The method for collecting and storing Internet big data information based on machine learning according to claim 1, characterized in that, Constructing a Merkle tree based on the hash values ​​of all ciphertext data blocks includes: For each ciphertext data block, a unique hash value is calculated using the SHA-3-512 hash algorithm; Based on the IPFS content addressing mechanism, a unique identifier for each ciphertext data block is generated by mapping a unique hash value. Obtain the hash values ​​of all ciphertext blocks, sort them in order, and group them into multiple groups of hash values ​​by a preset number; Calculate the hash value of the intermediate node in each group as the next level node of the tree, and repeat this process until all nodes are aggregated into a root node hash value to obtain the Merkle tree.

6. A machine learning-based internet big data information collection and storage system, characterized in that, The system includes: The data filtering module is used to collect raw data streams from Internet data sources and input the raw data streams into a pre-trained lightweight real-time intrusion detection model to obtain effective data streams. The data segmentation module is used to perform semantic association segmentation processing on the effective data stream using a preset algorithm to obtain multiple data blocks. The data encryption module is used to encrypt each data block using a preset encryption algorithm based on an encryption key to obtain a ciphertext data block, and simultaneously generate a searchable encrypted index unit for the ciphertext data block; the encryption key is generated based on preset rules; the index unit contains encrypted tags based on keywords; The data query tree building module is used to construct a Merkle tree based on the hash values ​​of all encrypted data blocks; The data storage module is used to save the Merkle tree and all encrypted data blocks into the storage area.

7. The Internet big data information collection and storage system based on machine learning according to claim 6, characterized in that, The data filtering module: The data sorting module is used to sort the original data stream in chronological order to obtain a time-series sequence, and to segment the time-series sequence into multiple sub-time-series sequences according to a preset time period. The fusion hidden feature extraction module is used to substitute the target sub-time series sequence into the bidirectional GRU layer to obtain the fusion hidden features of each time series in the target sub-time series sequence; The target sub-time series sequence is any one of a plurality of sub-time series sequences; The feature weight determination module is used to perform linear mapping on the fused hidden features of all time series and normalize them using Softmax to obtain the weight corresponding to each fused hidden feature. The global key feature extraction module is used to perform weighted fusion of the fused hidden features according to the weights corresponding to each fused hidden feature to obtain global key features. The data anomaly probability determination module is used to substitute the global key features into the residual network to obtain residual features, and substitute the residual features into the Softmax classifier to obtain the data anomaly probability. The effective data stream generation module is used to remove the target sub-time series if the probability of data anomaly is greater than a preset anomaly threshold, obtain all the removed sub-time series, and concatenate them according to time to obtain the effective data stream.

8. The Internet big data information collection and storage system based on machine learning according to claim 6, characterized in that, The data segmentation module includes: The structured data matrix extraction module is used to extract semantically relevant features from each piece of data in the effective data stream to obtain a structured data matrix; The fuzzy data likelihood calculation module is used to calculate the likelihood of fuzzy data using a formula. Calculate the fuzzy data likelihood of each data item in the effective data stream; in, For fuzzy data likelihood, it represents the core feature vector in a given semantic block. and fuzzy semantic relationships At that time, structured data matrix The probability of conforming to this cluster structure, where h is the fuzzification parameter. This is a block index, with values ​​ranging from 1 to G. For the first Data points in semantic blocks, where E is a simplified representation of the clustering prototype and The meaning is the same. For the first The overall membership degree of data within a class block, where λ is the data index within the block, with a value ranging from 1 to H. For the first The individual membership degree of the λth data point in the class block, γ=v is the mean constraint condition, γ is the mean factor of the data in the block, v is the preset mean threshold, and d is the correlation strength between the data point and the block center. The fuzzy data likelihood update module is used to update the fuzzy data likelihood by maximizing the fuzzy data likelihood. Update clustering prototype membership degree The process continues until multiple data blocks are obtained through convergence.

9. The Internet big data information collection and storage system based on machine learning according to claim 6, characterized in that, The data encryption module includes: The data block subkey determination module is used to determine the data block's identifier ID and encryption key through an algorithm. Obtain the data block subkey; Where SK is the subkey of the data block, HMAC is the hash-based message authentication code, SHA3 is the Secure Hash Algorithm 3, MK is the encryption key, BlockID is the identifier ID of the data block, and TS is the timestamp. The secondary encryption module is used to perform secondary encryption on the data block subkey using secret elliptic curve cryptography to obtain the target key; The ciphertext data block generation module is used to encrypt the data block using a preset encryption algorithm based on the target key to obtain a ciphertext data block; A searchable encrypted index unit generation module is used to extract keywords from the data block, encrypt the extracted keywords with the encryption key, and then generate an encrypted tag through SHA3-256 hashing to obtain a searchable encrypted index unit.

10. The Internet big data information collection and storage system based on machine learning according to claim 6, characterized in that, The data query tree construction module includes: The unique hash value calculation module is used to calculate a unique hash value for each ciphertext data block using the SHA-3-512 hash algorithm; The unique identifier generation module is used for the IPFS-based content addressing mechanism to generate a unique identifier for each ciphertext data block through unique hash value mapping. The ciphertext block grouping module is used to obtain the hash values ​​of all ciphertext blocks, sort them in order, and group them into multiple groups of hash values ​​by a preset number; The Merkle tree building module is used to calculate the hash value of the intermediate node in each group as the next level node of the tree. This process is repeated until all nodes are aggregated into a root node hash value to obtain the Merkle tree.

Images