A big data security transmission method combining blockchain and machine learning

Abstract

The application provides a big data security transmission method combining blockchain and machine learning in the technical field of information security and distributed computing, and comprises the following steps: S1, a transmission end performs data slicing and preprocessing on big data to be transmitted to obtain data blocks; S2, real-time security monitoring data is input into a data transmission risk identification model to obtain a transmission risk identification result, an encryption rule is matched from an encryption rule set based on the transmission risk identification result, data blocks are encrypted to obtain data encryption blocks, and the data encryption blocks are compressed to obtain data compression blocks; S3, data fingerprints of the data compression blocks are calculated and uploaded to a blockchain, and the data compression blocks are sent to a receiving end; S4, the receiving end performs security protection through an abnormal transmission behavior identification model during the receiving process; and S5, integrity check is performed through the data fingerprints, decompression, decryption and splicing are performed, and big data is obtained. The application has the advantages that the security and efficiency of big data transmission are greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information security and distributed computing, and particularly relates to a big data security transmission method combining blockchain and machine learning. BACKGROUND

[0002] Big data is generally defined as a data set with three characteristics of huge volume, variety of types and fast processing speed. Such a data set not only has a large scale, but also has high complexity and real-time characteristics, which poses a serious challenge to traditional data processing technology.

[0003] With the increasing application scenarios, the scale, complexity and transmission requirements of big data have triggered prominent security and efficiency problems, which have become a key bottleneck restricting its further development. Specifically, the existing big data transmission technology faces the following main challenges:

[0004] 1. Inherent limitations of static encryption algorithm:

[0005] Although static encryption algorithms (such as AES, RSA) are widely used for data protection, they have inherent defects in dynamic and large-scale data transmission environments. The core problem is that they rely on predefined fixed keys and encryption modes, lack the ability to adapt to real-time threats (such as increasingly intelligent variants of man-in-the-middle attacks) in the transmission process, and are difficult to effectively detect and respond to data theft or tampering behavior during transmission.

[0006] 2. Vulnerability of centralized verification mechanism:

[0007] The current mechanism for ensuring data integrity relies heavily on the audit of trusted third-party institutions (such as certificate authorities CA). This centralized architecture has a single point of failure risk: once the central server is attacked by a distributed denial of service (DDoS) or fails, the entire verification system will be paralyzed, causing data loss or service interruption.

[0008] 3. Bottleneck of direct application of blockchain technology:

[0009] Blockchain is explored as a replacement due to its decentralization and tamper-proof characteristics, but it has significant bottlenecks when applied to large-scale data transmission: a, storage and efficiency bottleneck: the mechanism of full node storing complete blocks will introduce serious processing delay when dealing with GB-level or even TB-level large data (such as medical images, satellite remote sensing data), and the consensus mechanism (such as Proof of Work PoW) may reduce the transmission efficiency to less than one tenth of the traditional method; b, lack of dynamic defense capability: blockchain mainly focuses on transaction verification and recording, and essentially lacks active awareness of the risk of transmission channels (such as the network layer), and cannot analyze the transmission traffic pattern in real time to intelligently identify abnormal behavior (such as DDoS attacks, data leakage signs), and is limited to recording transactions after the fact, making it difficult to provide active real-time defense responses during transmission.

[0010] In summary, the existing technical system is difficult to effectively balance the security and efficiency of the two core needs when dealing with the challenge of large data transmission: traditional static encryption and centralized verification architecture has serious security risks and low reliability; although blockchain technology improves security, its high storage overhead and inherent delay problem cannot meet the efficiency requirements of high-throughput data transmission, and lacks dynamic intelligent protection capabilities for transmission channels. Therefore, how to provide a large data security transmission method combining blockchain and machine learning to improve the security and efficiency of large data transmission has become a technical problem to be solved. SUMMARY

[0011] The technical problem to be solved by the present application is to provide a large data security transmission method combining blockchain and machine learning to improve the security and efficiency of large data transmission.

[0012] The present application is implemented as follows: a large data security transmission method combining blockchain and machine learning, comprising the following steps:

[0013] Step S1, creating a data transmission risk identification model, training the data transmission risk identification model and deploying it to the transmission end; creating an abnormal transmission behavior identification model, training the abnormal transmission behavior identification model and deploying it to the receiving end;

[0014] Step S2, setting an encryption rule set including standard encryption rules, intermediate encryption rules and high-level encryption rules, and preloading the encryption rule set into the transmission end and the receiving end;

[0015] Step S3, the transmission end obtains large data whose data type is structured data, semi-structured data or unstructured data, and performs data slicing and preprocessing on the large data to obtain a plurality of data blocks;

[0016] Step S4, the transmission end acquires real-time safety monitoring data, inputs the real-time safety monitoring data into a deployed data transmission risk identification model to obtain a transmission risk identification result, matches a corresponding encryption rule from a pre-set encryption rule set based on the transmission risk identification result, encrypts each data block based on the matched encryption rule to obtain a data encryption block, and compresses each data encryption block to obtain a data compression block; the encryption rule is a standard encryption rule, an intermediate encryption rule, or a high-level encryption rule;

[0017] Step S5, the transmission end calculates data fingerprints of each data compression block, constructs a Merkle tree based on each data fingerprint, uploads a root hash value of the Merkle tree to a blockchain, acquires blockchain transaction information, and sequentially sends the blockchain transaction information and each data compression block to a receiving end;

[0018] Step S6, the receiving end receives the transmitted blockchain transaction information and each data compression block in real time, and performs security defense through the deployed abnormal transmission behavior identification model during the receiving process;

[0019] Step S7, after the receiving end performs integrity verification on each data compression block through the blockchain transaction information, the receiving end decompresses each data compression block to obtain a data encryption block, decrypts the data encryption block through the pre-set encryption rule set to obtain a data block, splices each data block to obtain big data, and completes transmission of the big data;

[0020] Step S8, the receiving end records a big data receiving log in real time, encrypts the big data receiving log into an encrypted log through the encryption rule set and uploads the encrypted log to a blockchain, and iteratively optimizes the deployed abnormal transmission behavior identification model through the big data receiving log.

[0021] Further, in the step S1, after the data transmission risk identification model is created and trained, the data transmission risk identification model is deployed to the transmission end, and specifically:

[0022] A data transmission risk identification model is created based on a data input preprocessing layer, a risk feature extraction layer, a risk feature fusion layer, and a risk prediction output layer;

[0023] The data input preprocessing layer is constructed based on a network traffic preprocessing module, a system log preprocessing module, a network device state preprocessing module, a user behavior preprocessing module, a vulnerability scanning preprocessing module, and a threat intelligence preprocessing module; the network traffic preprocessing module, the system log preprocessing module, the network device state preprocessing module, the user behavior preprocessing module, the vulnerability scanning preprocessing module, and the threat intelligence preprocessing module are all constructed based on a standardization unit and an embedding unit;

[0024] The network traffic preprocessing module is configured to normalize and preliminarily embed network traffic data to obtain a network traffic embedding vector; the system log preprocessing module is configured to normalize and preliminarily embed system log data to obtain a system log embedding vector; the network device state preprocessing module is configured to normalize and preliminarily embed network device state data to obtain a network device state embedding vector; the user behavior preprocessing module is configured to normalize and preliminarily embed user behavior data to obtain a user behavior embedding vector; the vulnerability scanning preprocessing module is configured to normalize and preliminarily embed vulnerability scanning data to obtain a vulnerability scanning embedding vector; and the threat intelligence preprocessing module is configured to normalize and preliminarily embed threat intelligence data to obtain a threat intelligence embedding vector.

[0025] The risk feature extraction layer is constructed based on a network traffic feature extraction module, a system log feature extraction module, a network device state feature extraction module, a user behavior feature extraction module, a vulnerability scanning feature extraction module, and a threat intelligence feature extraction module; the network traffic feature extraction module, the system log feature extraction module, the network device state feature extraction module, the user behavior feature extraction module, the vulnerability scanning feature extraction module, and the threat intelligence feature extraction module are each constructed based on a convolutional neural network unit, a gated recurrent unit, and a concatenation unit.

[0026] The network traffic feature extraction module is configured to extract network traffic risk features from the network traffic embedding vector; the system log feature extraction module is configured to extract system log risk features from the system log embedding vector; the network device state feature extraction module is configured to extract network device state risk features from the network device state embedding vector; the user behavior feature extraction module is configured to extract user behavior risk features from the user behavior embedding vector; the vulnerability scanning feature extraction module is configured to extract vulnerability scanning risk features from the vulnerability scanning embedding vector; and the threat intelligence feature extraction module is configured to extract threat intelligence risk features from the threat intelligence embedding vector.

[0027] The risk feature fusion layer is configured to fuse the network traffic risk features, the system log risk features, the network device state risk features, the user behavior risk features, the vulnerability scanning risk features, and the threat intelligence risk features through a multi-head self-attention mechanism unit to obtain risk fusion features.

[0028] The risk prediction output layer is constructed based on a risk item identification module, a risk level classification module, and a result output module.

[0029] The risk item identification module is configured to infer the risk fusion feature to obtain a risk item probability distribution; the risk level classification module is configured to infer the risk fusion feature to obtain a risk level probability distribution; and the result output module is configured to output a transmission risk identification result carrying a risk item and a risk level according to the risk item probability distribution and the risk level probability distribution; and the risk level is low risk, medium risk, or high risk.

[0030] The risk optimization function of the data transmission risk identification model is set to use an Adam optimizer.

[0031] The risk loss function of the data transmission risk identification model is set to be:

[0032] wherein, represents a loss value of the risk loss function; represents a risk item cross-entropy loss, and a classification cross-entropy loss is used; represents a risk level cross-entropy loss, and a classification cross-entropy loss is used; both represent weight coefficients.

[0033] A large amount of historical security monitoring data including at least network flow data, system log data, network device state data, user behavior data, vulnerability scanning data, and threat intelligence data is acquired.

[0034] ​The network traffic data is preprocessed by at least data cleaning including removing noise data and removing duplicate data, data standardization including uniform time format and standardized numerical field, and feature extraction including session features and traffic features; the system log data is preprocessed by at least data cleaning including formatting log and removing irrelevant log, data standardization including uniform log level and uniform time format, and feature extraction including event features and text features; the network device state data is preprocessed by at least data cleaning including removing abnormal state data and removing duplicate data, data standardization including uniform device identification and standardized numerical field, and feature extraction including performance features and state features; the user behavior data is preprocessed by at least data cleaning including removing invalid behavior data and removing duplicate data, data standardization including uniform user identification and uniform time format, and feature extraction including behavior features and time features; the vulnerability scanning data is preprocessed by at least data cleaning including removing invalid vulnerability data and removing duplicate data, data standardization including uniform vulnerability level and uniform time format, and feature extraction including vulnerability features and risk features; the threat intelligence data is preprocessed by at least data cleaning including removing invalid intelligence data and removing duplicate data, data standardization including uniform threat type and uniform time format, and feature extraction including threat features and credibility features, so as to complete the preprocessing of the historical security monitoring data;

[0035] The preprocessed historical security monitoring data is labeled to construct a risk data set including at least risk items and risk levels; the risk data set is divided into a first training set, a first validation set and a first test set by a stratified sampling method in a ratio of 8:1:1;

[0036] The data transmission risk identification model is trained by the first training set, a risk optimization function and a risk loss function until a preset first early stopping condition is met, and the trained data transmission risk identification model is verified and tested by the first validation set and the first test set respectively, and the data transmission risk identification model that passes the test is deployed to the transmission end.

[0037] Further, in the step S1, the creating an abnormal transmission behavior identification model, the abnormal transmission behavior identification model is trained and deployed to the receiving end, specifically:

[0038] An abnormal transmission behavior identification model is created based on an abnormal feature extraction layer, an abnormal feature fusion layer and an abnormal prediction output layer;

[0039] The abnormal feature extraction layer is constructed based on a multi-scale time sequence channel, a spatial dependence channel, a behavior pattern channel, and a feature aggregation module; the multi-scale time sequence channel is used to extract multi-scale time sequence features from the input network transmission data stream through a bidirectional gate recurrent unit and a cavity convolution layer; the spatial dependence channel is used to extract spatial dependence features from the input network transmission data stream through a graph convolution network unit; the behavior pattern channel is used to extract behavior pattern features from the input network transmission data stream through a self-attention mechanism and a statistical feature extractor; and the feature aggregation module is used to aggregate the multi-scale time sequence features, the spatial dependence features, and the behavior pattern features through a gated feature fusion unit to obtain context features.

[0040] The abnormal feature fusion layer is used to perform cross-modal fusion on the context features through a hierarchical attention mechanism and a space-time compression module to obtain comprehensive behavior representation.

[0041] The abnormal prediction output layer is used to map the comprehensive behavior representation to an abnormal behavior probability distribution and classify and output a transmission behavior recognition result.

[0042] The behavior optimization function of the abnormal transmission behavior recognition model is set to use a Lookahead optimizer and a RAdam optimizer.

[0043] The behavior loss function of the abnormal transmission behavior recognition model is set to use a classification cross-entropy loss.

[0044] A large amount of historical network transmission data streams including at least source IP addresses, destination IP addresses, ports, communication protocols, packet sizes, transmission intervals, and identification bit distributions are obtained.

[0045] Each of the historical network transmission data streams is preprocessed including at least data cleaning, feature extraction and conversion, data dimension reduction, data bucketing, and data formatting.

[0046] Each of the preprocessed historical network transmission data streams is labeled to construct a behavior dataset including at least abnormal behaviors; and the behavior dataset is divided into a second training set, a second validation set, and a second test set in a ratio of 8:1:1 through a hierarchical sampling method.

[0047] The abnormal transmission behavior recognition model is trained through the second training set, the behavior optimization function, and the behavior loss function until a preset second early stopping condition is met, and the trained abnormal transmission behavior recognition model is verified and tested through the second validation set and the second test set, respectively, and the abnormal transmission behavior recognition model that passes the test is deployed to a receiving end.

[0048] Further, in the step S2, the standard encryption rule is specifically:

[0049] obtaining a current date string, performing MD5 hash calculation on the date string to obtain a hash value H1, extracting the first 8 bytes from the hash value H1 as a dynamic key K1; converting the large data to be encrypted into a UTF-8 encoded byte array B1, generating an 8-byte dynamic salt value Y1, appending the dynamic salt value Y1 to the front of the byte array B1 to obtain augmented data S1, encrypting the augmented data S1 by calling the dynamic key K1 through the DES algorithm to obtain ciphertext data C1, and performing Base64 encoding on the ciphertext data C1 to obtain a data encryption block;

[0050] The intermediate encryption rule is specifically:

[0051] obtaining a current date string, performing SHA-256 hash calculation on the date string to obtain a 32-byte hash value H2, taking the first 16 bytes of the hash value H2 as a dynamic key K2 and the last 16 bytes as a dynamic key K3; converting the large data to be encrypted into a UTF-8 encoded byte array B2, dividing the byte array B2 into a subarray B21 and a subarray B22, encrypting the subarray B21 by calling the dynamic key K2 through the AES-128 algorithm to obtain an encryption block EA1, encrypting the subarray B22 by calling the dynamic key K3 through the Blowfish algorithm to obtain an encryption block EB1, splicing the encryption block EA1 and the encryption block EB1 to obtain spliced data P1, and performing a 5-bit cyclic shift on the spliced data P1 in byte units to obtain transformed data T1; performing a position exchange operation on every two consecutive bytes in the transformed data T1 to obtain ciphertext data C2, and performing Base32 encoding on the ciphertext data C2 to obtain a data encryption block;

[0052] The high-level encryption rule is specifically:

[0053] obtaining a current date string, performing SHA-512 hash calculation on the date string to obtain a 64-byte hash value H3, taking the first 32 bytes of the hash value H3 as a dynamic key K4, the middle 16 bytes as a dynamic key K5, and the last 16 bytes as a dynamic salt value Y2; converting the large data to be encrypted into a UTF-8 encoded byte array B3, dividing the byte array B3 into a subarray B31, a subarray B32, and a subarray B33, encrypting the subarray B31 by calling the dynamic key K4 through the AES-256 algorithm to obtain an encrypted block EA2, encrypting the subarray B32 by calling the dynamic key K5 through the 3DES algorithm to obtain an encrypted block EB2, performing a byte-by-byte XOR operation on the subarray B32 and the dynamic salt value Y2 to obtain an encrypted block EC2, splicing the encrypted block EA2, the encrypted block EB2, and the encrypted block EC2 to obtain spliced data P2, exchanging the high 4 bits and the low 4 bits in each byte of the spliced data P2 to obtain transformed data Q, dividing the transformed data Q into 16-byte blocks, reordering each 16-byte block in a preset reverse order to obtain ciphertext data C3, and performing Base16 encoding on the ciphertext data C3 to obtain a data encryption block.

[0054] Further, the step S3 is specifically:

[0055] The transmission end obtains large data to be transmitted, and the data type of the large data is structured data, semi-structured data, or unstructured data; and performs data slicing on the large data based on a preset file size to obtain a plurality of data blocks, and performs preprocessing including at least data cleaning and data standardization on each data block.

[0056] Further, the step S4 is specifically:

[0057] The transmission end obtains real-time security monitoring data including at least network traffic data, system log data, network device state data, user behavior data, vulnerability scanning data, and threat intelligence data, performs preprocessing including at least data cleaning and data standardization on each real-time security monitoring data through a streaming computing engine, inputs each preprocessed real-time security monitoring data into a deployed data transmission risk identification model, and the data transmission risk identification model performs parallel inference through multiple GPUs to output a transmission risk identification result carrying a risk item and a risk level;

[0058] Analyzing the transmission risk identification result, when the risk level is low risk, matching a standard encryption rule from the encryption rule set; when the risk level is medium risk, matching a medium-level encryption rule from the encryption rule set; and when the risk level is high risk, matching a high-level encryption rule from the encryption rule set;

[0059] The data blocks are encrypted in parallel based on the matched encryption rules to obtain corresponding data encrypted blocks, the data encrypted blocks are named based on the current date string and the data block number, and the data encrypted blocks are compressed by a DEFLATE algorithm to obtain data compressed blocks; the encryption rules are standard encryption rules, intermediate encryption rules or high-level encryption rules.

[0060] Further, the step S5 is specifically:

[0061] The transmission end calculates data fingerprints of the data compressed blocks by an SM3 algorithm, constructs a Merkle tree based on the data fingerprints of the data compressed blocks, uploads a root hash value of the Merkle tree to a blockchain, waits for the root hash value to be passed by a blockchain consensus, acquires blockchain transaction information including at least a transaction hash, a block height and a block hash fed back by the blockchain, and sequentially sends the blockchain transaction information and the data compressed blocks to the receiving end by a secure communication protocol.

[0062] Further, the step S6 is specifically:

[0063] The receiving end receives the data compressed blocks in real time, collects real-time network transmission data streams including at least a source IP address, a destination IP address, a port, a communication protocol, a packet size, a transmission interval and an identification bit distribution during the receiving process, pre-processes the real-time network transmission data streams by a stream computing engine including at least data cleaning and data standardization, inputs the pre-processed real-time network transmission data streams into a deployed abnormal transmission behavior recognition model, and the abnormal transmission behavior recognition model outputs a transmission behavior recognition result by parallel inference of multiple GPUs.

[0064] The receiving end analyzes the transmission behavior recognition result, stops receiving the data compressed blocks when the transmission behavior recognition result carries an abnormal transmission behavior, and pushes a warning notification to a pre-associated management terminal to perform security defense.

[0065] Further, the step S7 is specifically:

[0066] The receiving end acquires the root hash value from the blockchain based on the blockchain transaction information, checks the integrity of the data compressed blocks based on the root hash value, decompresses the data compressed blocks by a DEFLATE algorithm to obtain data encrypted blocks, matches an encryption rule from a pre-set encryption rule set based on the encoding format of the data encrypted blocks, decrypts and checks the data encrypted blocks based on the matched encryption rule and a date string carried by the file name of the data encrypted blocks to obtain data blocks, splices the data blocks based on the data block number carried by the file name of the data encrypted blocks to obtain big data, and completes transmission of the big data.

[0067] Further, the step S8 is specifically:

[0068] The receiving end records at least the real-time security monitoring data, the receiving time, and the big data receiving log of the blockchain transaction information, encrypts the big data receiving log into an encrypted log through the high-level encryption rule in the encryption rule set, and uploads the blockchain; builds an incremental data set through the big data receiving log, labels the abnormal behavior of the incremental data set, and iteratively optimizes the deployed abnormal transmission behavior identification model.

[0069] The advantages of the present application are:

[0070] 1. After creating and training the data transmission risk identification model and deploying it to the transmission end, create and train the abnormal transmission behavior identification model and deploy it to the receiving end; set the encryption rule set containing the standard encryption rule, the intermediate encryption rule and the high-level encryption rule, and preset it into the transmission end and the receiving end; the transmission end obtains the big data whose data type to be transmitted is structured data, semi-structured data or unstructured data, and preprocesses the big data to obtain a plurality of data blocks; the transmission end obtains real-time security monitoring data, inputs the real-time security monitoring data into the data transmission risk identification model to obtain a transmission risk identification result, matches the corresponding encryption rule from the encryption rule set based on the transmission risk identification result, encrypts each data block based on the matched encryption rule to obtain a data encryption block, compresses each data encryption block to obtain a data compression block, and calculates the data fingerprint of each data compression block, constructs a Merkle tree based on each data fingerprint, uploads the root hash value of the Merkle tree to the blockchain, obtains the blockchain transaction information, and sends the blockchain transaction information and each data compression block to the receiving end in turn; the receiving end performs security defense through the abnormal transmission behavior identification model during the receiving process, performs integrity verification on each data compression block through the blockchain transaction information, decompresses each data compression block to obtain a data encryption block, decrypts the data encryption block through the encryption rule set to obtain a data block, splices each data block to obtain big data to complete the transmission of the big data, and records the big data receiving log in real time, encrypts the big data receiving log into an encrypted log through the encryption rule set and uploads the blockchain, and iteratively optimizes the deployed abnormal transmission behavior identification model through the big data receiving log; that is, through the machine learning model (data transmission risk identification model, abnormal transmission behavior identification model), dynamically perceive the transmission risk (transmission end) and real-time identify the abnormal behavior (receiving end), realize adaptive encryption strength adjustment and active defense; at the same time, the blockchain only stores the root hash value of the Merkle tree constructed by the data fingerprint, avoids the efficiency bottleneck of directly uploading the big data on the chain under the premise of guaranteeing the data integrity and the source credibility, thereby greatly improving the security and efficiency of the big data transmission under the double protection of dynamic threat protection and lightweight credible verification.

[0071] 2、By combining machine learning risk assessment models (data transmission risk identification model and abnormal transmission behavior identification model) with blockchain, a dynamic security architecture is constructed, which provides multi-layer defense: based on real-time security monitoring data, dynamically match encryption rules (such as standard, intermediate or high level), adapt to different risk levels; at the same time, the receiving end prevents attacks in real time through the deployed abnormal behavior model, this integration significantly improves security and adaptability, reduces the risk of data leakage or tampering, especially suitable for handling high-sensitive big data.

[0072] 3、By calculating data fingerprints, constructing Merkle trees, and uploading root hash values to the blockchain, the integrity of big data during transmission is ensured; the distributed ledger characteristics of the blockchain ensure that the root hash value is tamper-proof, while the integrity check (using blockchain transaction information) provides end-to-end verifiability, which makes any data tampering behavior can be quickly detected, preventing man-in-the-middle attacks or data corruption, greatly improving the reliability of big data transmission.

[0073] 4、Through data sharding, preprocessing and data compression, the processing efficiency of big data is optimized, reducing bandwidth demand and transmission delay: data sharding allows parallel processing, compression reduces data volume, combined with risk-based encryption rule selection, unnecessary high-strength encryption is avoided (such as using high-level encryption only when risk is high), effectively improving transmission speed and system throughput, especially suitable for real-time transmission scenarios of massive big data.

[0074] 5、By recording big data receiving logs, encrypting and uploading to the blockchain, and optimizing the abnormal transmission behavior identification model, a closed-loop feedback mechanism is introduced; machine learning models can be iteratively optimized based on actual log data, improving abnormal behavior detection accuracy and defense capabilities; at the same time, big data receiving logs are encrypted and uploaded to the blockchain to ensure the reliability and auditability of big data receiving logs, enabling the system to adapt to new threats and long-term improve security performance.

[0075] 6、By supporting multiple data types (structured, semi-structured and unstructured data), and providing a general security framework through pre-set encryption rule sets; that is, from data acquisition to decryption splicing, the entire process is applicable to various big data scenarios (such as enterprise data transmission, Internet of Things device communication), with high universality.

[0076] 7. By deeply integrating blockchain and machine learning technologies, a smart and dynamic big data security transmission system has been constructed. Its core advantages lie in using real-time risk assessment to drive adaptive encryption strength adjustment (such as dynamically matching multi-level encryption rules according to transmission risks), using blockchain to ensure the immutability and traceability of data throughout the entire chain (through Merkle root hashing on-chain and log encryption for evidence storage), and continuously improving defense accuracy through the closed-loop optimization mechanism of machine learning models (based on the iterative abnormal behavior identification model based on receiver logs). This not only significantly enhances the transmission security and integrity of mixed structured / semi-structured / unstructured data, but also greatly optimizes transmission efficiency (data fragmentation compression and lightweight verification) and system resource utilization (layered encryption to reduce computing power consumption). Ultimately, while effectively resisting complex network attacks, it provides a scalable, low-cost, and highly compliant integrated solution for big data applications in multiple fields.

[0077] 8. By integrating six key security data sources—network traffic data, system log data, network device status data, user behavior data, vulnerability scanning data, and threat intelligence data—it covers multiple dimensions such as network transmission activities, system operating status, device health, user operations, system vulnerabilities, and external threat intelligence. This greatly expands the scope of risk identification, significantly improves the model's ability to capture complex, covert, or cross-data source collaborative attacks, and avoids risk assessment blind spots caused by data silos.

[0078] 9. By designing independent preprocessing modules (including standardization and embedding units) and feature extraction modules (based on CNN+GRU+concatenation units) for each type of data source, the most suitable preprocessing (such as specific data cleaning and standardization methods) and feature extraction (such as CNN capturing spatial patterns and GRU capturing temporal dependencies) can be performed for the characteristics of different data types (such as time series, log text, and status indicators). This avoids information loss or noise introduction caused by mixed processing of different types of data, and can generate high-quality, risk-identification-oriented embedding vectors and risk features. This significantly improves the data quality and feature effectiveness of the input model, laying a solid foundation for subsequent fusion and prediction, and is a key prerequisite for the model's high accuracy.

[0079] 10、Risk feature fusion layer uses multi-head self-attention mechanism, which can dynamically learn the importance weight and correlation between different data source risk features, is particularly good at capturing long-distance dependence and nonlinear interaction, and the "multi-head" design allows the model to focus on the associated information from different representation subspaces at the same time, which enables the data transmission risk identification model to intelligently fuse risk features from six different sources, understand the complex mutual influence between them (for example, the suspicious behavior of a user combined with the known vulnerabilities of a certain device and the related attack patterns in external threat intelligence), and generate more comprehensive and accurate risk fusion features, significantly improving the ability to identify complex threats.

[0080] 11、Risk prediction output layer simultaneously performs risk item identification (what risk) and risk level classification (low, medium, high risk) two tasks, shares the underlying risk fusion features, utilizes the correlation between tasks, shares feature representation, is more efficient than training two independent models, and can promote each other; the output result (risk item + level) is more operationally instructive than a single risk item or a single level, and the risk loss function allows the importance of the two tasks to be flexibly adjusted through weight coefficients a and b; that is, the data transmission risk identification model can provide more comprehensive risk information (not only know what risk, but also know its severity) once reasoning, which is convenient for subsequent differential security measures (such as strengthening encryption level, alarm, blocking, audit enhancement), and multi-task learning helps to improve the generalization ability of the model.

[0081] 12、By specifying the specific steps of data cleaning, standardization and feature extraction (such as removing noise / repeated data, unifying format / identification, extracting session features / event features, etc.) for each type of historical security monitoring data, the data quality and consistency are ensured; stratified sampling is performed on the labeled data (8:1:1 division of training / validation / test sets), which helps to maintain the balance of data distribution and makes the transmission risk identification result more reliable; Adam optimizer (adaptive learning rate, suitable for complex non-convex optimization) and early stopping strategy (based on validation set performance to prevent overfitting) are used; after training, the model is optimized and evaluated on the test set to ensure that the model performance meets the standard before deployment, and the whole process design is scientific and standardized, which maximizes the effectiveness of the training data, the stability of the model training, and the generalization performance and reliability of the finally deployed model.

[0082] 13. By constructing a modular deep learning model (data transmission risk identification model) based on multi-source heterogeneous data (network traffic, system logs, device status, user behavior, vulnerability scanning, and threat intelligence), using specialized data preprocessing (standardization and embedding) and feature extraction (combination of CNN and GRU), combining multi-head self-attention mechanism to realize risk feature fusion, and using multi-task learning framework to output risk items and levels (low / medium / high risk) simultaneously, the model is finally deployed on the transmission end, significantly improving the comprehensiveness, accuracy, real-time performance, and operability of data transmission risk identification, and providing an efficient and reliable intelligent solution for active defense.

[0083] 14. The abnormal feature extraction layer adopts a multi-channel design (multi-scale time sequence channel, spatial dependence channel, and behavior pattern channel), and combines a feature aggregation module for context feature integration. The multi-scale time sequence channel uses bidirectional gated recurrent units (BiGRU) and cavity convolution to capture transmission behavior features of different time granularities. The spatial dependence channel processes network topology relationships (such as dependencies between IP addresses) through a graph convolution network (GCN). The behavior pattern channel uses a self-attention mechanism and a statistical feature extractor to identify abnormal patterns in behavior patterns. The abnormal feature fusion layer further realizes cross-modal fusion through a hierarchical attention mechanism and a space-time compression module. This multi-dimensional feature extraction and fusion mechanism overcomes the limitations of traditional single-feature models (such as using only time sequence models), can comprehensively cover the time sequence, spatial, and behavior pattern features of network transmission data, and reduces feature redundancy through the feature aggregation module, enhancing the robustness and generalization ability of the model.

[0084] 15. The behavior optimization function uses a combination of Lookahead optimizer and RAdam optimizer, and the behavior loss function uses classification cross-entropy loss. The Lookahead optimizer stabilizes the training process through a forward-looking mechanism, and the RAdam optimizer (Rectified Adam) adaptively adjusts the learning rate to avoid gradient vanishing / explosion. The loss function optimizes the model output for classification tasks, and introduces an early stopping condition (second early stopping condition) during training, and divides the data set using a hierarchical sampling method (8:1:1 ratio). This optimization strategy significantly improves training efficiency - the combination of Lookahead and RAdam reduces the number of training iterations, while the early stopping mechanism prevents overfitting and saves computing resources. Hierarchical sampling ensures balanced data distribution, reduces sample bias (such as insufficient abnormal behavior samples), and improves the performance of the abnormal transmission behavior identification model on the validation and test sets.

[0085] 16、Historical network transmission data stream is preprocessed through multiple steps, including data cleaning, feature extraction and conversion, data dimension reduction, data bucketing, and data formatting, and combined with abnormal behavior labeling to build a behavior dataset, i.e., optimizing for network transmission characteristics (such as source IP address, destination IP address, packet size, etc.), ensuring high quality and consistency of input data; for example, data dimension reduction (such as PCA or t-SNE) reduces noise and dimension disaster, data bucketing (such as grouping based on transmission interval) enhances feature interpretability, and data formatting adapts to model input requirements, which directly improves the training effect of the abnormal transmission behavior recognition model and reduces false positive rate caused by data; at the same time, stratified sampling method (8:1:1) ensures the representativeness of data set division, making the abnormal transmission behavior recognition model more reliable after deployment.

[0086] 17、By building a multi-channel collaborative abnormal transmission behavior recognition model (fusing multi-scale time sequence channel, spatial dependence channel and behavior pattern channel), combined with feature aggregation and cross-modal fusion mechanism, the comprehensiveness and accuracy of abnormal transmission behavior detection are significantly improved; using Lookahead and RAdam dual optimizers and classification cross-entropy loss function, combined with stratified sampling data division and early stopping mechanism, the training efficiency and model generalization ability are greatly optimized; at the same time, the systematic data preprocessing process (including cleaning, dimension reduction, bucketing, etc.) ensures the input quality, and the model is directly deployed to the receiving end after strict verification, realizing low-latency real-time anomaly recognition, and providing efficient, reliable and scalable intelligent protection capability for secure transmission in blockchain environment.

[0087] 18、By setting standard encryption rules to dynamically generate MD5 hash keys (K1) and random salt values (Y1) based on the current date, both the timeliness and anti-cracking ability of the keys are ensured, and the predictability of encrypted data is eliminated, effectively defending against rainbow table attacks and pattern analysis; at the same time, the DES algorithm combined with the dynamic salt value addition mechanism ensures the efficient processing capability of large data encryption, and the final output of Base64 encoding provides cross-platform compatibility, making the scheme balance security strength, running efficiency and system universality.

[0088] 19、By setting intermediate encryption rules to dynamically generate double keys (K2 and K3) based on the current date, combining AES-128 and Blowfish encryption algorithms to parallel encrypt the segmented data, and implementing multi-layer confusion through cyclic bit shift and byte exchange, the system security is significantly improved - dynamic keys can prevent long-term key leakage risk, mixed algorithm design reduces the probability of single-point attack, and post-processing operation enhances the anti-analysis ability of ciphertext; at the same time, data block strategy and Base32 encoding optimize processing efficiency and compatibility, ensuring encryption strength while providing efficient and reliable end-to-end protection for big data applications.

[0089] 20. By setting advanced encryption rules, using date-driven dynamic key generation (SHA-512 hash-derived multi-level keys and salts) and hybrid encryption mechanisms (AES-256, 3DES, and XOR salt operations for parallel data block processing), combined with multiple data transformations (byte bit swapping, block reversal and rearrangement) and Base16 standardized output, we achieve efficient and crack-resistant big data encryption, significantly improving security (dynamic keys reduce the risk of long-term exposure, hybrid algorithms enhance anti-attack redundancy, and bit transformations disrupt data patterns). At the same time, we take practicality into account (parallel block processing optimizes computational efficiency, and Base16 encoding ensures cross-platform compatibility), forming an encryption system that is dynamic, obfuscated, and robust, suitable for big data scenarios with high security requirements.

[0090] 21. By pre-segmenting any type of large data to be transmitted (including structured, semi-structured, or unstructured data) into fragments of a preset file size and performing preprocessing operations such as data cleaning and standardization, the system achieves synergistic optimization of transmission efficiency, data processing quality, and system robustness. Integrating the fragmentation mechanism and preprocessing into the transmission stage effectively reduces network bandwidth pressure and transmission latency. At the same time, by eliminating data errors and format differences in advance through cleaning and standardization, the system not only significantly improves the accuracy of subsequent analysis results but also greatly reduces the resource overhead at the receiving end. The independence of fragmentation processing further ensures that a single point of failure does not affect the overall task, enabling the solution to flexibly adapt to diverse application scenarios while ensuring high reliability. Ultimately, this provides efficient, reliable, and scalable technical support for large data transmission and applications.

[0091] 22. By integrating multi-source security monitoring data in real time through a streaming computing engine, and combining it with a risk identification model based on multi-GPU parallel inference, the transmission risk level is accurately assessed. Based on the dynamic risk level (low / medium / high), differentiated encryption rules are adaptively matched to achieve intelligent resource scheduling. At the same time, parallel encryption, structured naming, and DEFLATE compression technologies are adopted to significantly improve data processing efficiency while ensuring high security (such as enabling strong encryption in high-risk scenarios). This forms an innovative secure transmission system that combines real-time response capabilities, optimized computing performance, elastic resource allocation, and full-link closed-loop protection.

[0092] 23. By integrating data compression, the SM3 national cryptographic algorithm, Merkle trees, and blockchain technology, a highly efficient, secure, and reliable data transmission and verification mechanism is constructed: First, the SM3 algorithm is used to generate high-strength tamper-proof fingerprints for compressed data blocks, and the Merkle tree structure improves the verification efficiency of massive data to logarithmic level; then, the root hash value is anchored to the blockchain, and decentralized notarization and timestamp authentication are achieved through the consensus mechanism to ensure the credibility of the data source and the traceability of its history; finally, the compressed blocks are incrementally transmitted in conjunction with secure communication protocols (such as HTTPS / TLS), and blockchain notarization certificates (transaction hash, block height, etc.) are transmitted synchronously, which significantly optimizes bandwidth utilization (reduces transmission costs), strengthens end-to-end data integrity and non-repudiation, and provides the receiving end with lightweight verification capabilities without third-party intervention, which is especially suitable for cross-domain trusted exchange scenarios of highly sensitive, large-volume data.

[0093] 24. The system performs millisecond-level preprocessing of multi-dimensional network transmission data streams (including source / destination IP, protocol characteristics, etc.) through a streaming computing engine, and accurately detects threats by combining an anomaly detection model with multi-GPU parallel inference. Once abnormal transmission behavior is detected, data reception is immediately interrupted and an early warning is automatically pushed, which significantly improves the system's response speed and proactive defense. While reducing the false alarm and false negative rates, it effectively blocks potential attack chains and has the comprehensive advantages of high-precision detection, optimized resource utilization, and automated security protection.

[0094] 25. By comprehensively utilizing the trusted root hash value of the blockchain to ensure the integrity of the data source, combining DEFLATE for efficient compression to save transmission bandwidth, and implementing parallel verification, decompression, and decryption through block processing; at the same time, using the date string carried in the file name to dynamically generate the decryption key to improve security, and achieving accurate splicing based on the data block number, and finally through multiple verification mechanisms (root hash verification, post-decryption verification) and flexible encryption rule matching, the reliability, efficiency, and scalability of big data transmission are significantly improved while ensuring tamper-proof and strong encryption throughout the data transmission process.

[0095] 26. By constructing a highly reliable closed-loop security system, and by recording and encrypting the big data receiving logs containing security monitoring data and blockchain transaction information in real time, an immutable data evidence is formed. At the same time, based on incremental datasets, abnormal behaviors are dynamically labeled and the abnormal transmission behavior identification model is optimized, which significantly enhances data tamper resistance, real-time response capability, intelligent analysis accuracy (adaptive incremental learning model) and resource efficiency. It can prevent man-in-the-middle attacks and data leakage risks, and continuously improve the accuracy of security protection. Attached Figure Description

[0096] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0097] Figure 1 is a flowchart of a big data secure transmission method combining blockchain and machine learning. DETAILED DESCRIPTION

[0098] The technical solution in the embodiments of the present application has the following general idea: through machine learning model dynamic perception of transmission risk and real-time identification of abnormal behavior, adaptive encryption strength adjustment and active defense are realized; at the same time, under the premise of guaranteeing data integrity and source credibility, the efficiency bottleneck of directly chaining big data is avoided by using the root hash value of the Merkle tree constructed by storing only data fingerprints in the blockchain, so that the security and efficiency of big data transmission are improved under the double protection of dynamic threat protection and lightweight credible verification.

[0099] Please refer to Figure 1 The preferred embodiment of the big data secure transmission method combining blockchain and machine learning of the present application comprises the following steps:

[0100] Step S1, a data transmission risk identification model is created, the data transmission risk identification model is trained and deployed to the transmission end; an abnormal transmission behavior identification model is created, the abnormal transmission behavior identification model is trained and deployed to the receiving end;

[0101] Step S2, an encryption rule set containing standard encryption rules, intermediate encryption rules and high-level encryption rules is set, and the encryption rule set is preloaded into the transmission end and the receiving end;

[0102] Step S3, the transmission end acquires big data whose data type to be transmitted is structured data, semi-structured data or unstructured data, performs data slicing and preprocessing on the big data to obtain a plurality of data blocks;

[0103] Step S4, the transmission end acquires real-time security monitoring data, inputs the real-time security monitoring data into the deployed data transmission risk identification model to obtain a transmission risk identification result, matches a corresponding encryption rule from the preloaded encryption rule set based on the transmission risk identification result, encrypts each data block based on the matched encryption rule to obtain data encryption blocks, and compresses each data encryption block to obtain data compression blocks; the encryption rule is a standard encryption rule, an intermediate encryption rule or a high-level encryption rule;

[0104] Step S5, the transmission end calculates the data fingerprints of each data compression block, constructs a Merkle tree based on each data fingerprint, uploads the root hash value of the Merkle tree to the blockchain, acquires blockchain transaction information, and sequentially sends the blockchain transaction information and each data compression block to the receiving end;

[0105] Step S6, the receiving end receives the transmitted blockchain transaction information and each data compression block in real time, and performs security defense through the deployed abnormal transmission behavior identification model during the receiving process;

[0106] Step S7, after the receiving end performs integrity check on each data compression block through the blockchain transaction information, the receiving end decompresses each data compression block to obtain a data encryption block, decrypts the data encryption block through the preset encryption rule set to obtain a data block, splices each data block to obtain big data, and completes transmission of the big data;

[0107] Step S8, the receiving end records a big data receiving log in real time, encrypts the big data receiving log into an encrypted log through the encryption rule set and uploads the encrypted log to a blockchain, and iteratively optimizes the deployed abnormal transmission behavior identification model through the big data receiving log.

[0108] In the step S1, the creating a data transmission risk identification model, after training the data transmission risk identification model, the data transmission risk identification model is deployed to the transmission end, and specifically comprises:

[0109] A data transmission risk identification model is created based on a data input preprocessing layer, a risk feature extraction layer, a risk feature fusion layer and a risk prediction output layer;

[0110] The data input preprocessing layer is constructed based on a network traffic preprocessing module, a system log preprocessing module, a network device state preprocessing module, a user behavior preprocessing module, a vulnerability scanning preprocessing module and a threat intelligence preprocessing module; the network traffic preprocessing module, the system log preprocessing module, the network device state preprocessing module, the user behavior preprocessing module, the vulnerability scanning preprocessing module and the threat intelligence preprocessing module are all constructed based on a standardization unit and an embedding unit; the standardization unit uses Z-score standardization to process numerical data; the embedding unit uses word embedding or positional embedding to process sequence or classification data, and the specific structure is a linear fully connected layer (dimension 128) plus a ReLU activation function, which outputs a unified embedding vector;

[0111] The network traffic preprocessing module is used for standardizing and preliminarily embedding network traffic data to obtain a network traffic embedding vector; the system log preprocessing module is used for standardizing and preliminarily embedding system log data to obtain a system log embedding vector; the network device state preprocessing module is used for standardizing and preliminarily embedding network device state data to obtain a network device state embedding vector; the user behavior preprocessing module is used for standardizing and preliminarily embedding user behavior data to obtain a user behavior embedding vector; the vulnerability scanning preprocessing module is used for standardizing and preliminarily embedding vulnerability scanning data to obtain a vulnerability scanning embedding vector; and the threat intelligence preprocessing module is used for standardizing and preliminarily embedding threat intelligence data to obtain a threat intelligence embedding vector.

[0112] The risk feature extraction layer is constructed based on a network traffic feature extraction module, a system log feature extraction module, a network device state feature extraction module, a user behavior feature extraction module, a vulnerability scanning feature extraction module, and a threat intelligence feature extraction module; each of the network traffic feature extraction module, the system log feature extraction module, the network device state feature extraction module, the user behavior feature extraction module, the vulnerability scanning feature extraction module, and the threat intelligence feature extraction module is constructed based on a convolutional neural network unit, a gated recurrent unit, and a concatenation unit; that is, each module is composed of a double-channel feature extraction path of a convolutional neural network unit (CNN) and a gated recurrent unit (GRU); the convolutional neural network unit is used for extracting local spatial features (e.g., 2 CNN layers, a convolution kernel size of 3x3, and ReLU activation); and the gated recurrent unit is used for extracting time series features (e.g., 1 layer of GRU and 128 hidden units), and the outputs of the two channels are concatenated into a comprehensive feature vector (dimension 256) through a concatenation unit, that is, the corresponding risk features are concatenated.

[0113] The network traffic feature extraction module is used for extracting network traffic risk features from the network traffic embedding vector; the system log feature extraction module is used for extracting system log risk features from the system log embedding vector; the network device state feature extraction module is used for extracting network device state risk features from the network device state embedding vector; the user behavior feature extraction module is used for extracting user behavior risk features from the user behavior embedding vector; the vulnerability scanning feature extraction module is used for extracting vulnerability scanning risk features from the vulnerability scanning embedding vector; and the threat intelligence feature extraction module is used for extracting threat intelligence risk features from the threat intelligence embedding vector.

[0114] The risk feature fusion layer is configured to fuse the network traffic risk features, the system log risk features, the network device state risk features, the user behavior risk features, the vulnerability scanning risk features, and the threat intelligence risk features through a multi-head self-attention mechanism unit to obtain risk fusion features; the multi-head self-attention mechanism unit includes multi-head attention calculation (8 heads, key-query-value mechanism) and residual connection, and outputs a fusion feature vector (dimension 256), i.e., the risk fusion features, for capturing global dependency between outputs of the risk feature extraction layer modules (for example, vulnerability scanning data may strengthen the risk signal of threat intelligence data), reducing data redundancy and noise, and improving the discriminability of the risk features; the residual connection ensures gradient stability.

[0115] The risk prediction output layer is constructed based on a risk item identification module, a risk level classification module, and a result output module.

[0116] The risk item identification module is configured to infer the risk fusion features to obtain a risk item probability distribution; the risk level classification module is configured to infer the risk fusion features to obtain a risk level probability distribution; and the result output module is configured to output a transmission risk identification result carrying a risk item and a risk level according to the risk item probability distribution and the risk level probability distribution; the risk level is low risk, medium risk, or high risk; the risk item identification module outputs the risk item probability distribution through a fully connected layer (dimension 128) connected to an output layer (fully connected layer dimension 10, corresponding to the number of risk item categories) with a dropout (dropout rate 0.3) and a softmax activation function; and the risk level classification module outputs the risk level probability distribution through a fully connected layer (dimension 64) connected to an output layer (fully connected layer dimension 3, corresponding to the number of risk level categories) with a softmax activation function.

[0117] The risk optimization function of the data transmission risk identification model is set to use an Adam optimizer (learning rate 0.001, beta1=0.9, beta2=0.999), which is suitable for multi-task and non-convex optimization problems and accelerates training convergence;

[0118] The risk loss function of the data transmission risk identification model is set as follows: ;

[0119] wherein, represents a loss value of the risk loss function; represents a risk item cross-entropy loss, which uses a categorical cross-entropy loss (Categorical Cross-Entropy Loss); represents a risk level cross-entropy loss, which uses a categorical cross-entropy loss; both represent weight coefficients;

[0120] ; ;

[0121] wherein, denotes the true risk item label; denotes the risk item prediction probability; denotes the true risk level label; denotes the risk level prediction probability;

[0122] Obtain a large amount of historical security monitoring data including at least network traffic data, system log data, network device status data, user behavior data, vulnerability scanning data, and threat intelligence data;

[0123] Network traffic data includes: a, traffic features: including the size of the data packet, the rate of traffic, the type of transmission protocol (such as TCP, UDP, HTTP, HTTPS, etc.); For example, the HTTP traffic of normal web browsing is usually small data packets and high-frequency interaction, while file downloading shows large data packets and long duration traffic; b, record the source IP address and destination IP address of the data packet, as well as the port number; By analyzing the flow direction, abnormal communication paths can be found; For example, if it is found that an internal server frequently interacts with a certain unknown IP address outside, it may imply a risk of data leakage; c, session information: the duration of each network session, connection state (such as establishment, continuation, interruption, etc.); Abnormal sessions that are not disconnected for a long time may indicate potential malicious connections.

[0124] System log data includes: a, host log: operating system log of servers, terminal devices, recording system startup, shutdown, user login / logout, permission change, software installation / uninstallation, etc. events; For example, frequent failed login attempts may indicate a brute force attack; b, application log: logs generated by various network applications (such as databases, web servers, mail servers, etc.); For example, web server logs record user access pages, request methods (GET, POST, etc.), return status codes (200 indicates normal, 404 indicates page not found, 500 indicates server internal error); If 500 errors occur frequently, it may indicate that the application has vulnerabilities; c, security device log: log records of security devices such as firewalls, intrusion detection systems (IDS), intrusion prevention systems (IPS); Firewall logs will record allowed or denied traffic information, IDS / IPS logs will record detected attack behavior, attack type (such as SQL injection, cross-site scripting, etc.).

[0125] Network device status data includes: a, device performance indicators: CPU usage, memory usage, interface bandwidth utilization of network devices (such as routers, switches), etc.; if the CPU or memory usage of the device is high for a long time, it may affect network performance, and even cause device failure; b, device configuration information: device network interface configuration (such as IP address, subnet mask, VLAN configuration, etc.), routing table information, etc.; by monitoring the changes of configuration information, unauthorized configuration tampering behavior can be found in time; c, device running status: online / offline status of the device, port status (such as whether the port is open, whether it has failed, etc.); for example, if a key network port is found to be suddenly closed, it may affect network connectivity.

[0126] User behavior data includes: a, user identity information: login account, user role, permission level, etc.; by analyzing the matching of user identity and behavior, abnormal behavior can be found; for example, a low-privilege user accessing high-privilege resources may indicate a risk of privilege abuse; b, user operation records: various operation behaviors of users in the network, such as file access, application usage, network resource access, etc.; for example, a user frequently accessing external suspicious websites or downloading a large number of files from unknown sources may indicate abnormal user behavior; c, user access pattern: user access time, access frequency, access path, etc.; if it is found that a user frequently accesses certain sensitive resources during non-working hours, further investigation may be needed.

[0127] Vulnerability scanning data includes: a, vulnerability detection results: by regularly scanning the network system for vulnerabilities, record the types of vulnerabilities found (such as operating system vulnerabilities, application vulnerabilities, configuration vulnerabilities, etc.), vulnerability levels (high, medium, low) and the devices or systems where the vulnerabilities are located; b, vulnerability repair situation: record the time of vulnerability repair, repair method and verification results after repair; by continuously monitoring the vulnerability repair situation, it can ensure that the network system eliminates security risks in time.

[0128] Threat intelligence data includes: a, external threat intelligence: external threat intelligence obtained from security vendors, industry alliances, etc., including malicious IP address list, malicious domain name list, malicious software feature library, etc.; by comparing these intelligence with internal network data, potential external threats can be found in time; b, internal threat intelligence: threat intelligence generated by enterprise internal security event analysis, audit, etc., such as internal user abnormal behavior pattern, internal network malicious software propagation path, etc.; these intelligence helps enterprises better cope with internal threats.

[0129] The network traffic data is preprocessed by at least data cleaning including removing noise data and removing duplicate data, data standardization including uniform time format and standardized numerical field, and feature extraction including session features and traffic features; the system log data is preprocessed by at least data cleaning including formatting log and removing irrelevant log, data standardization including uniform log level and uniform time format, and feature extraction including event features and text features; the network device state data is preprocessed by at least data cleaning including removing abnormal state data and removing duplicate data, data standardization including uniform device identification and standardized numerical field, and feature extraction including performance features and state features; the user behavior data is preprocessed by at least data cleaning including removing invalid behavior data and removing duplicate data, data standardization including uniform user identification and uniform time format, and feature extraction including behavior features and time features; the vulnerability scanning data is preprocessed by at least data cleaning including removing invalid vulnerability data and removing duplicate data, data standardization including uniform vulnerability level and uniform time format, and feature extraction including vulnerability features and risk features; the threat intelligence data is preprocessed by at least data cleaning including removing invalid intelligence data and removing duplicate data, data standardization including uniform threat type and uniform time format, and feature extraction including threat features and credibility features, so as to complete the preprocessing of the historical security monitoring data;

[0130] The historical security monitoring data after preprocessing is labeled to construct a risk data set including at least risk items and risk levels; the risk data set is divided into a first training set, a first validation set and a first test set by a stratified sampling method in a ratio of 8:1:1;

[0131] The data transmission risk identification model is trained by the first training set, a risk optimization function and a risk loss function until a preset first early stopping condition is met, and the trained data transmission risk identification model is verified and tested by the first validation set and the first test set respectively, and the data transmission risk identification model that passes the test is deployed to the transmission end.

[0132] In the step S1, the abnormal transmission behavior identification model is created, and after the abnormal transmission behavior identification model is trained, it is deployed to the receiving end, specifically:

[0133] An abnormal transmission behavior identification model is created based on an abnormal feature extraction layer, an abnormal feature fusion layer and an abnormal prediction output layer;

[0134] The abnormal feature extraction layer is constructed based on a multi-scale time sequence channel, a spatial dependence channel, a behavior pattern channel, and a feature aggregation module; the multi-scale time sequence channel is used for extracting multi-scale time sequence features from an input network transmission data stream through a bidirectional gated recurrent unit and a hollow convolution layer; the bidirectional gated recurrent unit (Bi-GRU) extracts bidirectional long short-term dependencies (such as burst traffic duration) in the network transmission data stream, and the hollow convolution layer (dilation = 1, 2, 4) captures periodic patterns (such as DDoS attack pulses) of different granularities from the network transmission data stream, and outputs multi-scale time sequence features including bidirectional long short-term dependencies and periodic patterns; the spatial dependence channel is used for extracting spatial dependence features from the input network transmission data stream through a graph convolution network unit; that is, an IP-port topology graph is constructed through the graph convolution network unit (GCN), the communication mode between nodes is learned through an adjacency matrix, and an irregular connection mode (such as abnormal port access in scanning behavior) is identified; the behavior pattern channel is used for extracting behavior pattern features from the input network transmission data stream through a self-attention mechanism and a statistical feature extractor; the self-attention mechanism is used for focusing on key behavior fragments (such as TCP flag abnormal sequences), and the statistical feature extractor is used for calculating non-stationary indicators such as entropy and variance in real time; the feature aggregation module is used for aggregating multi-scale time sequence features, spatial dependence features, and behavior pattern features through a gated feature fusion unit to obtain context features; that is, a learnable weight gating mechanism is adopted to dynamically weight and fuse three-channel output features;

[0135] The abnormal feature fusion layer is used for cross-modal fusion of the context features through a hierarchical attention mechanism and a space-time compression module to obtain comprehensive behavior representations; the hierarchical attention mechanism adopts a multi-level attention network, the first level attention allocates feature importance in the time dimension (such as weighting the period of sudden traffic increase), and the second level attention strengthens the association of abnormal nodes in the space dimension (such as C&C communication nodes); the space-time compression module adopts an adaptive pooling layer, which is used for compressing a variable-length sequence into a fixed-dimension behavior fingerprint vector and retaining the statistical distribution characteristics (such as skewness / kurtosis of traffic distribution) of time sequence features;

[0136] The abnormal prediction output layer is used for mapping the comprehensive behavior representations to an abnormal behavior probability distribution and classifying and outputting transmission behavior recognition results; the abnormal prediction output layer is constructed based on a classifier and a confidence calibration module; the classifier adopts a network structure of a full connection layer + GELU activation function, a main branch predicts normal / abnormal, and an auxiliary branch outputs abnormal types (such as DDoS / port scanning); the confidence calibration module adopts a temperature scaling layer, which is used for adjusting the output probability distribution through a learnable temperature parameter and solving the confidence offset problem in the class imbalance scenario;

[0137] The behavior optimization function of the abnormal transmission behavior identification model is set to use a Lookahead optimizer and a RAdam optimizer; the learning rate is cosine annealing scheduling (initial value 0.001); the gradient clipping threshold is 1.0;

[0138] The behavior loss function of the abnormal transmission behavior identification model is set to use a classification cross-entropy loss.

[0139] A large amount of historical network transmission data streams including at least source IP addresses, destination IP addresses, ports, communication protocols, packet sizes, transmission intervals, and identification bit distributions are obtained;

[0140] The historical network transmission data streams are preprocessed including at least data cleaning, feature extraction and conversion, data dimension reduction, data binning, and data formatting; the data cleaning includes at least removing duplicate data (checking whether there are duplicate records in the data set, if there are duplicate records, the duplicate items can be deleted and only one record is kept), handling missing values (checking whether there are missing values in the data, for missing values, the following methods can be used: a, deleting records containing missing values; b, filling missing values with mean, median or mode; c, for classification features (such as communication protocols), filling missing values with the most common category), and filtering invalid data (checking whether the data conforms to the logic, for example, whether the port number is within the range of 0-65535, whether the IP address conforms to the format, etc. If invalid data is found, it can be deleted or corrected); data binning, that is, for certain features (such as packet size and transmission interval), they can be binned to simplify the data, for example, packet size can be divided into "small", "medium", and "large" intervals, and transmission interval can be divided into "short", "medium", and "long" intervals;

[0141] The behavior data set is constructed after labeling the preprocessed historical network transmission data streams including at least abnormal behaviors; the behavior data set is divided into a second training set, a second validation set, and a second test set in a ratio of 8:1:1 by stratified sampling method;

[0142] The abnormal transmission behavior identification model is trained through the second training set, the behavior optimization function, and the behavior loss function until the preset second early stopping condition is met, the trained abnormal transmission behavior identification model is verified and tested through the second validation set and the second test set respectively, and the abnormal transmission behavior identification model that passes the test is deployed to the receiving end.

[0143] In the step S2, the standard encryption rule is specifically:

[0144] Get the current date string (format YYYYMMDD), MD5 hash the date string to get hash value H1, extract the first 8 bytes from the hash value H1 as dynamic key K1; Convert the data to be encrypted into a UTF-8 encoded byte array B1, generate an 8-byte dynamic salt value Y1, append the dynamic salt value Y1 to the front of the byte array B1 to get augmented data S1, encrypt the augmented data S1 using the dynamic key K1 through the DES algorithm to get ciphertext data C1, and encode the ciphertext data C1 through Base64 to get the data encryption block;

[0145] The DES algorithm is weak, but the dynamic key is based on the date and increases variability; the salt value prevents replay attacks; the step logic is rigorous: date hashing -> key generation -> data salt augmentation -> DES encryption -> Base64 encoding, suitable for low-risk large data transmission or low-sensitive data processing.

[0146] The intermediate encryption rule is specifically:

[0147] Get the current date string, SHA-256 hash the date string to get a 32-byte hash value H2, take the first 16 bytes of the hash value H2 as dynamic key K2, and the last 16 bytes as dynamic key K3; Convert the data to be encrypted into a UTF-8 encoded byte array B2, divide the byte array B2 into subarrays B21 and B22 (if the data size is odd, subarray B22 has one more byte), encrypt subarray B21 through AES-128 algorithm using dynamic key K2 to get encryption block EA1, encrypt subarray B22 through Blowfish algorithm using dynamic key K3 to get encryption block EB1, concatenate encryption block EA1 and encryption block EB1 to get concatenated data P1, and circularly shift 5 bits of the concatenated data P1 in byte units to get transformed data T1, perform a position exchange operation (i.e., byte[i] and byte[i+1] are exchanged, the index starts from 0) on every two consecutive bytes in the transformed data T1 to get ciphertext data C2, and encode the ciphertext data C2 through Base32 to get the data encryption block;

[0148] SHA-256 and AES enhance the strength of hashing and encryption; data segmentation and two encryption algorithms provide multi-layer protection; circular shift and byte exchange add confusion; step logic: date hashing -> key segmentation -> data segmentation -> parallel encryption -> concatenation -> shift transformation -> byte exchange; suitable for medium-risk large data transmission or medium-sensitive data such as configuration files.

[0149] The high-level encryption rule is specifically:

[0150] Obtaining a current date string, performing SHA-512 hash calculation on the date string to obtain a 64-byte hash value H3, taking the first 32 bytes of the hash value H3 as a dynamic key K4, the middle 16 bytes as a dynamic key K5, and the last 16 bytes as a dynamic salt value Y2; converting the large data to be encrypted into a UTF-8 encoded byte array B3, dividing the byte array B3 into subarrays B31, B32 and B33 (if the size is not divisible, the subarray B33 is slightly larger), encrypting the subarray B31 by calling the dynamic key K4 through the AES-256 algorithm to obtain an encrypted block EA2, encrypting the subarray B32 by calling the dynamic key K5 through the 3DES algorithm to obtain an encrypted block EB2, performing a byte-by-byte XOR operation on the subarray B32 and the dynamic salt value Y2 to obtain an encrypted block EC2, splicing the encrypted block EA2, the encrypted block EB2 and the encrypted block EC2 to obtain spliced data P2, exchanging the high 4 bits and the low 4 bits in each byte of the spliced data P2 (i.e., exchanging the [7-4] bits and the [3-0] bits), obtaining transformed data Q, dividing the transformed data Q into 16-byte blocks, reordering each 16-byte block in a preset reverse order (such as block 1→block n, and inverting to block n→block 1) to obtain ciphertext data C3, and performing Base16 encoding on the ciphertext data C3 to obtain a data encryption block.

[0151] SHA-512 and AES-256 provide high-strength hash and encryption; triple block processing (AES, 3DES, XOR salt) increases complexity; bit-level confusion and block permutation add deep confusion and diffusion; step logic: date hash→key salt division→data division→parallel encryption and XOR→splicing→bit confusion→block permutation; suitable for high-risk large data transmission or high-sensitive data such as passwords or identity information.

[0152] Converting the large data to be encrypted into a UTF-8 encoded byte array, i.e., converting structured data, semi-structured data or unstructured data into a byte array first to ensure universality.

[0153] The step S3 is specifically:

[0154] The transmission end obtains large data to be transmitted, and the data type of the large data is structured data, semi-structured data or unstructured large data; based on a preset file size, the large data is data fragmented to obtain a plurality of data blocks, and each data block is preprocessed including at least data cleaning and data standardization.

[0155] The step S4 is specifically:

[0156] The transmission end obtains real-time security monitoring data including at least network traffic data, system log data, network device state data, user behavior data, vulnerability scanning data, and threat intelligence data, pre-processes each of the real-time security monitoring data through a stream computing engine, including at least data cleaning and data standardization, inputs each of the pre-processed real-time security monitoring data into a deployed data transmission risk identification model, and the data transmission risk identification model outputs a transmission risk identification result carrying a risk item and a risk level through parallel inference of multiple GPUs;

[0157] The transmission risk identification result is analyzed, a standard encryption rule is matched from the encryption rule set when the risk level is low, a medium encryption rule is matched from the encryption rule set when the risk level is medium, and a high-level encryption rule is matched from the encryption rule set when the risk level is high.

[0158] Each of the data blocks is encrypted in parallel based on the matched encryption rule to obtain corresponding data encryption blocks, each of the data encryption blocks is named based on the current date string and the data block number, and each of the data encryption blocks is compressed to obtain a data compression block through a DEFLATE algorithm. The encryption rule is a standard encryption rule, a medium encryption rule, or a high-level encryption rule.

[0159] The DEFLATE algorithm is a lossless data compression algorithm that combines LZ77 algorithm and Huffman coding; LZ77 compression finds repeated strings in input data and uses pointers and lengths to represent repeated patterns; Huffman coding encodes the dictionary matching generated by LZ77 compression to further compress data.

[0160] The step S5 specifically includes:

[0161] The transmission end calculates the data fingerprints of each of the data compression blocks through an SM3 algorithm, constructs a Merkle tree based on the data fingerprints of each of the data compression blocks, uploads the root hash value of the Merkle tree to a blockchain, waits for the root hash value to be passed by blockchain consensus, obtains blockchain transaction information including at least transaction hash, block height, and block hash fed back by the blockchain, and sequentially sends the blockchain transaction information and each data compression block to the receiving end through a secure communication protocol.

[0162] The basic data for data fingerprint calculation includes the content of the data compression block and the file name.

[0163] Merkle Tree, also known as Hash Tree, is a tree-like data structure based on hash functions. It aggregates the hash values of a large amount of data layer by layer through hierarchical hashing, and finally generates a unique root hash value (Merkle Root), thereby efficiently verifying the integrity and consistency of the data.

[0164] The step S6 is specifically:

[0165] The receiving end receives each of the data compression blocks transmitted in real time, collects real-time network transmission data streams including at least source IP address, destination IP address, port, communication protocol, packet size, transmission interval and identification bit distribution during the receiving process, performs preprocessing including at least data cleaning and data standardization on each of the real-time network transmission data streams through a streaming computing engine, inputs each of the preprocessed real-time network transmission data streams into a deployed abnormal transmission behavior identification model, and the abnormal transmission behavior identification model outputs a transmission behavior identification result through parallel inference of multiple GPUs;

[0166] The receiving end analyzes the transmission behavior identification result, stops receiving the data compression blocks when the transmission behavior identification result carries abnormal transmission behavior, and pushes a warning notification to a pre-associated management terminal to perform security defense.

[0167] The step S7 is specifically:

[0168] The receiving end obtains a root hash value from the blockchain through the blockchain transaction information, performs integrity check on each data compression block based on the root hash value, decompresses each data compression block through a DEFLATE algorithm to obtain a data encryption block, matches an encryption rule from a pre-set encryption rule set based on the encoding format of the data encryption block, decrypts and checks each data encryption block based on the matched encryption rule and the date string carried by the file name of the data encryption block to obtain a data block, and splices each data block based on the data block number carried by the file name of the data encryption block to obtain big data, thereby completing transmission of the big data.

[0169] The step S8 is specifically:

[0170] The receiving end records a big data receiving log including at least real-time security monitoring data, receiving time and blockchain transaction information in real time, encrypts the big data receiving log into an encrypted log through a high-level encryption rule in the encryption rule set and uploads the encrypted log to the blockchain, constructs an incremental data set through the big data receiving log, labels the incremental data set for abnormal behavior, and iteratively optimizes the deployed abnormal transmission behavior identification model.

[0171] The big data receiving log further comprises data block feature information, transmission behavior details, system resource state, identity authentication information, and threat intelligence correlation data; the data block feature information comprises: a, data block check value: the hash value (such as SM3 value) of each received data block, used for comparison with the original fingerprint of the transmission end; b, compression / encryption metadata: compression rate, encryption rule type (standard / intermediate / advanced), and dynamic key generation parameter (such as date string); c, data block integrity state: check result (success / failure) and failure reason (such as hash mismatch); the transmission behavior details comprise: a, network layer indicators: transmission delay, packet loss rate, and bandwidth occupancy rate; b, abnormal behavior markers: original risk score output by an abnormal transmission behavior identification model and specific features (such as abnormal port access frequency) triggering an early warning; c, defense action records: operations (such as suspension of receiving and data block retransmission request) triggered by security defense; the system resource state comprises: a, receiving end load: CPU / memory occupancy rate and disk I / O speed; b, decryption / decompression performance: single data block processing time consumption and parallel task number; the identity authentication information comprises: a, digital certificate fingerprint of the transmission end / receiving end; b, TLS / SSL protocol version and key exchange algorithm of the communication session; the threat intelligence correlation data comprises: a, real-time threat intelligence matching result (such as whether the source IP is located in a known malicious IP library); b, CVE number related to the received data in the vulnerability scanning result.

[0172] Although the specific embodiments of the present application are described above, those skilled in the art should understand that the specific embodiments described are only illustrative, and are not intended to limit the scope of the present application, and equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present application should be covered within the scope of the claims of the present application.

Claims

1. A method for secure big data transmission combining blockchain and machine learning, characterized in that: The method comprises the following steps: Step S1, creating a data transmission risk identification model, training the data transmission risk identification model, and deploying the trained data transmission risk identification model to a transmission end; creating an abnormal transmission behavior identification model, training the abnormal transmission behavior identification model, and deploying the trained abnormal transmission behavior identification model to a receiving end; Step S2, setting an encryption rule set comprising a standard encryption rule, an intermediate encryption rule, and a high-level encryption rule, and preloading the encryption rule set into the transmission end and the receiving end; Step S3, the transmission end obtains big data to be transmitted, wherein the data type of the big data is structured data, semi-structured data, or unstructured data; based on a preset file size, the big data is data-fragmented to obtain a plurality of data blocks, and each data block is preprocessed, including data cleaning and data standardization; Step S4, the transmission end obtains real-time security monitoring data including network traffic data, system log data, network device state data, user behavior data, vulnerability scanning data, and threat intelligence data, and preprocesses each real-time security monitoring data through a streaming computing engine, including data cleaning and data standardization; the preprocessed real-time security monitoring data is input into the deployed data transmission risk identification model, and the data transmission risk identification model outputs a transmission risk identification result carrying a risk item and a risk level through multi-GPU parallel inference; analyzing the transmission risk identification result, when the risk level is low, matching a standard encryption rule from the encryption rule set; when the risk level is medium, matching an intermediate encryption rule from the encryption rule set; when the risk level is high, matching a high-level encryption rule from the encryption rule set; based on the matched encryption rule, each data block is encrypted in parallel to obtain a corresponding data encryption block, each data encryption block is named based on the current date string and the data block number, and each data encryption block is compressed to obtain a data compression block through the DEFLATE algorithm; the encryption rule is a standard encryption rule, an intermediate encryption rule, or a high-level encryption rule; Step S5, the transmission end calculates the data fingerprint of each data compression block through the SM3 algorithm, constructs a Merkle tree based on the data fingerprint of each data compression block, uploads the root hash value of the Merkle tree to the blockchain, waits for the root hash value to be passed by the blockchain consensus, obtains the blockchain transaction information including the transaction hash, the block height, and the block hash fed back by the blockchain, and sends the blockchain transaction information and each data compression block to the receiving end in sequence through a secure communication protocol; Step S6, the receiving end receives the transmitted blockchain transaction information and each data compression block in real time, collects at least real-time network transmission data streams including source IP address, destination IP address, port, communication protocol, packet size, transmission interval and identification bit distribution during the receiving process, pre-processes each real-time network transmission data stream at least including data cleaning and data standardization through a stream computing engine, inputs each pre-processed real-time network transmission data stream into a deployed abnormal transmission behavior identification model, and the abnormal transmission behavior identification model outputs a transmission behavior identification result through parallel inference of multiple GPUs; The receiving end analyzes the transmission behavior identification result, stops receiving the data compression block when the transmission behavior identification result carries abnormal transmission behavior, and pushes a warning notification to a pre-associated management terminal to perform security defense; Step S7, after the receiving end performs integrity check on each data compression block through the blockchain transaction information, the receiving end decompresses each data compression block to obtain a data encryption block, decrypts the data encryption block through a preset encryption rule set to obtain a data block, splices each data block to obtain big data, and completes transmission of the big data; Step S8, the receiving end records a big data receiving log in real time, encrypts the big data receiving log into an encrypted log through the encryption rule set and uploads the encrypted log to a blockchain, and iteratively optimizes the deployed abnormal transmission behavior identification model through the big data receiving log; In the step S1, the data transmission risk identification model is created, and after the data transmission risk identification model is trained, the data transmission risk identification model is deployed to the transmission end, specifically: A data transmission risk identification model is created based on a data input preprocessing layer, a risk feature extraction layer, a risk feature fusion layer and a risk prediction output layer; The data input preprocessing layer is constructed based on a network traffic preprocessing module, a system log preprocessing module, a network device state preprocessing module, a user behavior preprocessing module, a vulnerability scanning preprocessing module and a threat intelligence preprocessing module; the network traffic preprocessing module, the system log preprocessing module, the network device state preprocessing module, the user behavior preprocessing module, the vulnerability scanning preprocessing module and the threat intelligence preprocessing module are each constructed based on a standardization unit and an embedding unit; The network traffic preprocessing module is used for standardizing and initially embedding network traffic data to obtain a network traffic embedding vector; The system log preprocessing module is used for standardizing and initially embedding system log data to obtain a system log embedding vector; the network device state preprocessing module is used for standardizing and initially embedding network device state data to obtain a network device state embedding vector; the user behavior preprocessing module is used for standardizing and initially embedding user behavior data to obtain a user behavior embedding vector; the vulnerability scanning preprocessing module is used for standardizing and initially embedding vulnerability scanning data to obtain a vulnerability scanning embedding vector; and the threat intelligence preprocessing module is used for standardizing and initially embedding threat intelligence data to obtain a threat intelligence embedding vector; The risk feature extraction layer is constructed based on a network traffic feature extraction module, a system log feature extraction module, a network device state feature extraction module, a user behavior feature extraction module, a vulnerability scanning feature extraction module, and a threat intelligence feature extraction module; the network traffic feature extraction module, the system log feature extraction module, the network device state feature extraction module, the user behavior feature extraction module, the vulnerability scanning feature extraction module, and the threat intelligence feature extraction module are all constructed based on a convolutional neural network unit, a gated recurrent unit, and a concatenation unit; The network traffic feature extraction module is configured to extract network traffic risk features from a network traffic embedding vector; the system log feature extraction module is configured to extract system log risk features from a system log embedding vector; the network device state feature extraction module is configured to extract network device state risk features from a network device state embedding vector; the user behavior feature extraction module is configured to extract user behavior risk features from a user behavior embedding vector; the vulnerability scanning feature extraction module is configured to extract vulnerability scanning risk features from a vulnerability scanning embedding vector; and the threat intelligence feature extraction module is configured to extract threat intelligence risk features from a threat intelligence embedding vector; The risk feature fusion layer is configured to fuse the network traffic risk features, the system log risk features, the network device state risk features, the user behavior risk features, the vulnerability scanning risk features, and the threat intelligence risk features through a multi-head self-attention mechanism unit to obtain risk fusion features; The risk prediction output layer is constructed based on a risk item identification module, a risk level classification module, and a result output module; The risk item identification module is configured to infer the risk fusion features to obtain a risk item probability distribution; the risk level classification module is configured to infer the risk fusion features to obtain a risk level probability distribution; and the result output module is configured to output a transmission risk identification result carrying a risk item and a risk level according to the risk item probability distribution and the risk level probability distribution; the risk level is low risk, medium risk, or high risk; The risk optimization function of the data transmission risk identification model is set to use an Adam optimizer; The risk loss function of the data transmission risk identification model is set as: ; wherein, represents a loss value of a risk loss function; represents a risk item cross-entropy loss, using a classification cross-entropy loss; represents a risk level cross-entropy loss, using a classification cross-entropy loss; and both represent a weight coefficient; A large amount of historical security monitoring data including at least network traffic data, system log data, network device state data, user behavior data, vulnerability scanning data, and threat intelligence data is acquired; The network traffic data is preprocessed by at least data cleaning including removing noise data and removing duplicate data, data standardization including uniform time format and standardized numerical field, and feature extraction including session features and traffic features; the system log data is preprocessed by at least data cleaning including formatting log and removing irrelevant log, data standardization including uniform log level and uniform time format, and feature extraction including event features and text features; the network device state data is preprocessed by at least data cleaning including removing abnormal state data and removing duplicate data, data standardization including uniform device identification and standardized numerical field, and feature extraction including performance features and state features; the user behavior data is preprocessed by at least data cleaning including removing invalid behavior data and removing duplicate data, data standardization including uniform user identification and uniform time format, and feature extraction including behavior features and time features; the vulnerability scanning data is preprocessed by at least data cleaning including removing invalid vulnerability data and removing duplicate data, data standardization including uniform vulnerability level and uniform time format, and feature extraction including vulnerability features and risk features; and the threat intelligence data is preprocessed by at least data cleaning including removing invalid intelligence data and removing duplicate data, data standardization including uniform threat type and uniform time format, and feature extraction including threat features and credibility features, so as to complete the preprocessing of the historical security monitoring data; The preprocessed historical security monitoring data is labeled to construct a risk data set including risk items and risk levels; and the risk data set is divided into a first training set, a first validation set and a first test set by a stratified sampling method in a ratio of 8:1:1; The data transmission risk identification model is trained by the first training set, a risk optimization function and a risk loss function until a preset first early stopping condition is met, and the trained data transmission risk identification model is verified and tested by the first validation set and the first test set respectively, and the data transmission risk identification model that passes the test is deployed to a transmission end; In step S2, the standard encryption rule is specifically: A current date string is obtained, MD5 hash calculation is performed on the date string to obtain a hash value H1, and the first 8 bytes of the hash value H1 are extracted as a dynamic key K1; the big data to be encrypted is converted into a UTF-8 encoded byte array B1, an 8-byte dynamic salt value Y1 is generated, the dynamic salt value Y1 is appended to the front of the byte array B1 to obtain augmented data S1, the augmented data S1 is encrypted by calling the dynamic key K1 through a DES algorithm to obtain ciphertext data C1, and the ciphertext data C1 is Base64 encoded to obtain a data encryption block; The intermediate encryption rule is specifically: Obtaining a current date string, performing SHA-256 hash calculation on the date string to obtain a 32-byte hash value H2, taking the first 16 bytes of the hash value H2 as a dynamic key K2 and the last 16 bytes as a dynamic key K3; converting the large data to be encrypted into a UTF-8 encoded byte array B2, dividing the byte array B2 into a subarray B21 and a subarray B22, encrypting the subarray B21 by calling the dynamic key K2 through the AES-128 algorithm to obtain an encrypted block EA1, encrypting the subarray B22 by calling the dynamic key K3 through the Blowfish algorithm to obtain an encrypted block EB1, splicing the encrypted block EA1 and the encrypted block EB1 to obtain spliced data P1, and performing cyclic bit shift of 5 bits on the spliced data P1 to obtain transformed data T1; performing position exchange operation on every two consecutive bytes in the transformed data T1 to obtain ciphertext data C2, and performing Base32 encoding on the ciphertext data C2 to obtain a data encryption block; The advanced encryption rule is specifically: Obtaining a current date string, performing SHA-512 hash calculation on the date string to obtain a 64-byte hash value H3, taking the first 32 bytes of the hash value H3 as a dynamic key K4, the middle 16 bytes as a dynamic key K5, and the last 16 bytes as a dynamic salt value Y2; converting the large data to be encrypted into a UTF-8 encoded byte array B3, dividing the byte array B3 into a subarray B31, a subarray B32 and a subarray B33, encrypting the subarray B31 by calling the dynamic key K4 through the AES-256 algorithm to obtain an encrypted block EA2, encrypting the subarray B32 by calling the dynamic key K5 through the 3DES algorithm to obtain an encrypted block EB2, performing byte-by-byte XOR operation on the subarray B32 and the dynamic salt value Y2 to obtain an encrypted block EC2, splicing the encrypted block EA2, the encrypted block EB2 and the encrypted block EC2 to obtain spliced data P2, exchanging the high 4 bits and the low 4 bits in each byte of the spliced data P2 to obtain transformed data Q, dividing the transformed data Q into 16-byte blocks, reordering each 16-byte block in a preset reverse order to obtain ciphertext data C3, and performing Base16 encoding on the ciphertext data C3 to obtain a data encryption block.

2. The method for secure transmission of big data by combining blockchain and machine learning according to claim 1, characterized in that: In the step S1, the creating an abnormal transmission behavior identification model, and the abnormal transmission behavior identification model is deployed to a receiving end after being trained, specifically: Creating an abnormal transmission behavior identification model based on an abnormal feature extraction layer, an abnormal feature fusion layer and an abnormal prediction output layer; The abnormal feature extraction layer is constructed based on a multi-scale time sequence channel, a spatial dependence channel, a behavior pattern channel, and a feature aggregation module; the multi-scale time sequence channel is used to extract multi-scale time sequence features from an input network transmission data stream through a bidirectional gated recurrent unit and a cavity convolution layer; the spatial dependence channel is used to extract spatial dependence features from the input network transmission data stream through a graph convolution network unit; and the behavior pattern channel is used to extract behavior pattern features from the input network transmission data stream through a self-attention mechanism and a statistical feature extractor. The feature aggregation module is used to aggregate the multi-scale time sequence features, the spatial dependence features, and the behavior pattern features through a gated feature fusion unit to obtain context features. The abnormal feature fusion layer is used to perform cross-modal fusion on the context features through a hierarchical attention mechanism and a space-time compression module to obtain comprehensive behavior representations. The abnormal prediction output layer is used to map the comprehensive behavior representations to an abnormal behavior probability distribution and classify and output a transmission behavior recognition result. The behavior optimization function of the abnormal transmission behavior recognition model is set to use a Lookahead optimizer and a RAdam optimizer. The behavior loss function of the abnormal transmission behavior recognition model is set to use a classification cross-entropy loss. A large amount of historical network transmission data streams including at least source IP addresses, destination IP addresses, ports, communication protocols, packet sizes, transmission intervals, and identification bit distributions are obtained. The historical network transmission data streams are preprocessed through at least data cleaning, feature extraction and conversion, data dimension reduction, data bucketing, and data formatting. The behavior dataset is constructed after the historical network transmission data streams are preprocessed and labeled with at least abnormal behaviors. The behavior dataset is divided into a second training set, a second validation set, and a second test set in a ratio of 8:1:1 through a hierarchical sampling method. The abnormal transmission behavior recognition model is trained through the second training set, the behavior optimization function, and the behavior loss function until a preset second early stopping condition is met, and the trained abnormal transmission behavior recognition model is verified and tested through the second validation set and the second test set, respectively. The abnormal transmission behavior recognition model that passes the test is deployed to a receiving end.

3. The big data secure transmission method combining blockchain and machine learning as described in claim 1, characterized in that: The step S7 is specifically: The receiving end obtains a root hash value from the blockchain through the blockchain transaction information, performs integrity verification on each data compression block based on the root hash value, decompresses each data compression block to obtain a data encryption block through a DEFLATE algorithm, matches an encryption rule from a preset encryption rule set based on the encoding format of the data encryption block, decrypts and verifies each data encryption block based on the matched encryption rule and the date string carried by the file name of the data encryption block to obtain a data block, and splices each data block based on the data block number carried by the file name of the data encryption block to obtain big data, thereby completing the transmission of the big data.

4. The method for secure transmission of big data by combining blockchain and machine learning according to claim 1, wherein: The step S8 is specifically: The receiving end records a big data receiving log including at least real-time security monitoring data, receiving time and blockchain transaction information, encrypts the big data receiving log into an encrypted log through a high-level encryption rule in the encryption rule set and uploads the encrypted log to the blockchain; An incremental data set is constructed through the big data receiving log, the incremental data set is labeled for abnormal behavior, and the deployed abnormal transmission behavior recognition model is iteratively optimized.

Images