A cybersecurity big data security processing method based on artificial intelligence

Abstract

This invention discloses an artificial intelligence-based method for big data security processing in the field of network security technology. The method collects encrypted traffic and extracts its transport layer metadata without decrypting the network traffic; preprocesses the metadata and constructs a temporal feature sequence; analyzes the feature sequence using a hybrid deep learning model composed of a one-dimensional convolutional neural network and a long short-term memory network to identify malicious encrypted traffic; and executes a real-time security response based on the identification results. This invention achieves high-precision detection of encryption-masked attacks by deeply mining the behavioral features in the encrypted traffic metadata, while strictly protecting user communication privacy.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a network security big data security processing method based on artificial intelligence. Background Technology

[0002] With the widespread adoption of encryption protocols such as HTTPS and TLS, over 95% of websites worldwide use encrypted transmission. While this protects user privacy, it also renders traditional deep packet inspection (DPI) techniques ineffective because they cannot parse encrypted payloads. Some "decrypt-detect-encrypt" schemes not only violate privacy regulations due to the need to obtain keys but also incur high computational costs, making them unsuitable for real-time requirements. Alternative solutions, such as detection methods based on port, protocol characteristics, or manual statistical features, suffer from high false positive rates, poor generalization capabilities, and an inability to identify deep malicious patterns. More seriously, attackers exploit encrypted channels to carry out "encryption-masked attacks," such as transmitting malicious code or establishing C&C communication, making it difficult to balance privacy protection and security oversight.

[0003] While existing technologies offer various AI-based cybersecurity solutions—such as CN113098867A, which uses historical data to build threat cloud maps for interface protection; CN113839963A, which generates vulnerability detection samples through training; CN120017369A, which uses multi-class samples for threat detection; and CN120389880A, which integrates multimodal data with knowledge graphs for vulnerability detection—none of these solutions directly address the core challenge of deep feature mining of encrypted traffic metadata. Their data foundations, model architecture designs, and application scenarios differ fundamentally from the problem of "real-time malicious behavior identification without decrypting encrypted traffic" that this invention aims to solve.

[0004] Therefore, there is an urgent need for a highly efficient and accurate encrypted traffic detection technology that does not rely on decryption, to fill the gaps in the traditional security system. Summary of the Invention

[0005] To address the technical problems existing in the prior art, this invention provides a network security big data security processing method based on artificial intelligence, comprising the following steps: S101. Encrypted traffic metadata collection: Non-intrusive capture of raw encrypted data packets from network communication links and extraction of metadata features; the metadata features include basic transmission features, timing features, and distribution features, and the extraction process does not require decryption of the encrypted traffic payload; S102, Metadata preprocessing and sequence construction: Data cleaning and feature standardization are performed on the metadata features, and a fixed-length feature input sequence is constructed to provide input for the deep learning model; S103. Deep learning model construction and training: Construct a hybrid deep learning model and train the hybrid deep learning model using the preprocessed feature input sequence; the hybrid deep learning model adopts a hybrid architecture combining a one-dimensional convolutional neural network and a long short-term memory network. S104. Real-time detection and response execution: Deploy the trained hybrid deep learning model into the inference engine to perform threat detection on real-time traffic and execute response actions based on the detection results. S105. Online learning and model iteration: Collect low-confidence traffic samples, label them, and perform incremental training to achieve continuous optimization and adaptive evolution of the model.

[0006] As a preferred embodiment of this application, in step S101, the encrypted traffic metadata collection includes: deploying data collection probes at key network nodes to capture raw encrypted data packets in a bypass monitoring mode; parsing the header information of each raw encrypted data packet and extracting five-tuple information, wherein the five-tuple includes source address, destination address, source port, destination port, and protocol type; associating data packets as data streams based on the five-tuple and extracting the metadata features; the basic transmission features include the number of data packets in the uplink and downlink directions, total number of bytes, byte ratio, statistical values ​​of data packet length, session duration, statistics of transmission control protocol flags, and data retransmission rate; the timing features include statistical values ​​of data packet arrival time intervals and burstiness indicators of traffic rate; the distribution features include a histogram of data packet length distribution and information entropy.

[0007] As a preferred embodiment of this application, in step S102, the metadata preprocessing and sequence construction includes: cleaning the metadata features, including identifying and replacing outliers, and using forward padding to handle missing values; standardizing the cleaned metadata features to eliminate the influence of dimensions; constructing a fixed-length feature sequence, using zero-value vectors to pad sequences with insufficient length, and using a sliding window to truncate subsequences for sequences with excessive length.

[0008] As a preferred embodiment of this application, in step S103, the one-dimensional convolutional neural network is used to extract local correlation features in the feature input sequence, and the long short-term memory network is used to extract long-term dependencies in the feature input sequence; during training, the cross-entropy loss function and adaptive moment estimation algorithm are used, and an early stopping strategy is adopted to prevent overfitting.

[0009] As a preferred embodiment of this application, in step S104, the real-time detection and response execution includes: inputting the feature input sequence into the inference engine to obtain a malicious probability value; making a decision on the malicious probability value according to a preset threshold; when the malicious probability value exceeds a first threshold, executing a blocking action and generating an alarm; when the malicious probability value is in the middle range, generating an alarm and recording a log; and when the malicious probability value is lower than a second threshold, allowing traffic.

[0010] As a preferred embodiment of this application, in step S105, the online learning and model iteration includes: collecting the low-confidence traffic samples and labeling them using external threat intelligence sources or manual judgment; using an incremental learning strategy to train the model based on existing model parameters and the newly added samples; and deploying the new model to the production environment through model hot update technology.

[0011] This application also provides an artificial intelligence-based network security big data security processing system for implementing the methods described above. The system includes various modules that work collaboratively with data streams through clearly defined interfaces. The metadata acquisition engine module, deployed on the critical path of the network, is implemented based on a high-performance packet processing framework and is used to capture raw network traffic and extract metadata features. The streaming preprocessing and serialization module includes data cleaning logic, standardization transformation and sequence construction unit, used to clean, standardize and serialize the metadata features to generate feature input sequences; The AI ​​inference service module is deployed in the form of a microservice cluster, supports dynamic batch processing, and is used to load hybrid deep learning models and perform inference on the feature input sequence. The strategy decision and execution engine module integrates a configurable security policy rule base, which is used to execute corresponding security response operations based on the results of the inference. The online learning and model management module is used to manage the model lifecycle and perform model iterations.

[0012] As a preferred embodiment of this application, the online learning and model management platform module includes a sample data lake sub-component, an annotation workbench sub-component, an incremental training pipeline sub-component, and a model repository sub-component; the sample data lake sub-component is used to store collected sample data; the annotation workbench sub-component supports a combination of automated and manual sample annotation; the incremental training pipeline sub-component is used to trigger and execute incremental training tasks for the model; and the model repository sub-component is used for version management, storage, and verification of the model, and to update the verified new model to the AI ​​inference service module through continuous integration and continuous deployment pipelines.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: 1) Achieving a balance between privacy protection and security detection: This method does not require decryption of traffic content throughout the process, but only analyzes externally observable metadata, fundamentally avoiding the risk of leakage of user communication content, fully complying with data privacy protection regulations, and effectively resolving the contradiction between security supervision and privacy protection in encrypted environments.

[0014] 2) Significantly improved detection accuracy and robustness: The CNN-LSTM hybrid model designed for encrypted traffic characteristics can automatically learn complex local patterns and long-term dependencies in metadata. It has high accuracy and low false alarm rate in identifying encrypted malicious traffic such as C&C communication and data leakage, and is not affected by evasion methods such as port spoofing and protocol obfuscation.

[0015] 3) Possesses adaptive evolution capability: Through an online learning mechanism, the system can continuously collect suspected samples and incrementally update the model, enabling it to quickly adapt to new attack methods and constantly changing encrypted traffic characteristics, significantly improving the detection capability against unknown threats and variant attacks, and reducing long-term operation and maintenance costs.

[0016] 4) Meets high real-time requirements and is easy to implement: Low overhead in metadata processing and model inference, meeting millisecond-level real-time detection needs in high-concurrency network environments. The solution requires no changes to the existing network architecture and encryption processes, and can be easily integrated into existing security devices such as firewalls and IDS, offering flexible deployment and strong compatibility. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of the network security big data security processing method provided in the embodiments of the present invention.

[0020] Figure 2 This is a schematic diagram of encrypted traffic metadata collection provided in an embodiment of the present invention.

[0021] Figure 3 This is a schematic diagram of metadata preprocessing and sequence construction provided in an embodiment of the present invention.

[0022] Figure 4 This is a schematic diagram illustrating the construction and training of a deep learning model provided in an embodiment of the present invention.

[0023] Figure 5 This is a schematic diagram of real-time detection and response execution provided in an embodiment of the present invention.

[0024] Figure 6 This is a schematic diagram of online learning and model iteration provided in an embodiment of the present invention.

[0025] Figure 7 This is an architecture diagram of the network security big data security processing system provided in the embodiments of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0027] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.

[0028] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" and "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. When the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.

[0029] Example 1 This invention provides a method for secure big data processing in cybersecurity based on artificial intelligence. For example... Figure 1 As shown, the specific steps include: S101, Encrypted Traffic Metadata Collection This step aims to non-intrusively capture raw encrypted data packets from network communication links and extract a set of metadata features that fully characterize traffic behavior patterns without requiring decryption of the payload content, providing a data foundation for subsequent intelligent analysis. For example... Figure 2 As shown, the specific implementation is as follows: First, deploy data acquisition probes at core network hubs, such as internet border gateways, data center aggregation switches, or the front end of critical servers. These probes operate in bypass listening mode, acquiring copies of all original encrypted data packets flowing through network nodes via port mirroring or network splitting technology, thus ensuring zero interference with normal business traffic.

[0030] The data acquisition probe performs preliminary parsing on each captured raw encrypted data packet, extracting five key elements—source address, destination address, source port, destination port, and protocol type—based on its Internet Protocol header and transport layer header information; these are known as the "five-tuple." Using this five-tuple as an identifier, the system performs real-time association and aggregation of all raw encrypted data packets belonging to the same bidirectional communication session, forming a complete data flow context.

[0031] For each active data stream, the system initiates an independent metadata extraction routine. This routine runs continuously throughout the data stream's lifecycle, performing calculations solely based on the header, timing, and statistical information of the original encrypted data packets, resolutely avoiding any access to or decryption of the encrypted payload content of transport layer security protocols or application layer protocols. The extracted metadata primarily covers the following three dimensions: The first dimension is basic transmission characteristics, which includes the number of data packets and the total number of bytes in the upstream and downstream directions of the statistical data stream, and the calculation of the upstream / downstream byte ratio. This ratio is indicative of behavioral patterns such as data leakage (upstream significantly greater than downstream) or large file downloads (downstream significantly greater than upstream). Simultaneously, the average, standard deviation, and multiple quantiles of the data packet length in each direction are calculated to characterize the concentration and dispersion trends of data packet size. Furthermore, the total session duration is recorded, and for Transmission Control Protocol (TCP) traffic, the distribution of its flags (such as push, urgent, end, reset) is analyzed, and the data retransmission rate is estimated to reflect connection stability and potential anomalies.

[0032] The second dimension is temporal characteristics. This is achieved by calculating the sequence of arrival time intervals of consecutive data packets and extracting their mean, standard deviation, extreme values, and multiple quantiles to quantify the burstiness and rhythm of traffic. Furthermore, the system calculates the traffic rate within consecutive short time windows, identifies burst events by setting dynamic thresholds, and then statistically analyzes the burst frequency, average burst duration, and the proportion of bytes transmitted during a burst to the total number of bytes transmitted. This temporal pattern is crucial for detecting malicious activities such as heartbeat signals in command and control channels or intermittent data leakage.

[0033] The third dimension is the distribution characteristics. The system divides the length values ​​of all data packets into a series of predefined intervals, counts the number of data packets falling into each interval, and forms a histogram of data packet length distribution. This visually shows whether the traffic consists of a large number of small packets, a small number of large packets, or a mixture of sizes. At the same time, information entropy is calculated based on this length distribution. The level of entropy reflects the uniformity or concentration of the length distribution, which helps to distinguish between randomized encrypted traffic and protocol traffic with a fixed pattern.

[0034] The data acquisition probe maintains a dynamically updated metadata feature vector for each data stream. When the data stream ends naturally or reaches a preset fixed time analysis window, the data acquisition probe encapsulates the final feature vector within this time analysis window, along with the timestamp and the data stream 5-tuple identifier, into a complete metadata record and outputs it to the downstream processing pipeline.

[0035] S102, Metadata Preprocessing and Sequence Construction The goal of this step is to transform the raw metadata from the acquisition layer into clean, standardized, and uniformly formatted feature sequences, providing high-quality input for deep learning models. For example... Figure 3 As shown, the specific implementation is as follows: First, data cleaning is performed. After receiving the raw metadata records, the system verifies each feature value according to a predefined reasonable range. For extreme values ​​that exceed the physically reasonable range (such as abnormally large packet lengths or impossible session durations), the system identifies them as outliers and replaces them with the median of that feature in historical data, or directly marks them as missing. Subsequently, for any missing values, the system uses forward imputation, that is, it fills in the missing values ​​using the corresponding feature value from the previous time window in the same data stream to ensure data continuity.

[0036] After cleaning, the system proceeds to the feature standardization stage. Since different features have drastically different dimensions and numerical ranges—for example, byte counts can reach millions, while ratios range between zero and one—direct use would cause model training to be dominated by large numerical features. Therefore, the system employs the Z-score standardization method to uniformly scale each numerical feature. Specifically, during offline training, the system pre-calculates the arithmetic mean and standard deviation of each feature in the entire training dataset. During online processing, for any input feature value, its corresponding mean is subtracted and then divided by the standard deviation, thus converting it into a standardized value with a mean of zero and a standard deviation of one. This process eliminates the influence of dimensions and accelerates model convergence.

[0037] Finally, a fixed-length feature input sequence is constructed. The duration of network sessions varies, resulting in variable lengths for their corresponding metadata feature vector sequences. To accommodate deep learning models with fixed input size requirements, the system implements a sequence normalization strategy. The system sets a fixed time step (e.g., feature vectors from ten consecutive time windows are considered as an analysis unit). For a specific data stream, if the number of generated feature vectors exceeds this set length, a sliding window approach is used to sequentially extract multiple subsequences of the set length from beginning to end, each subsequence serving as an independent sample. This increases the data volume and captures behavioral changes at different stages of the session. If the number of feature vectors is insufficient, zero-value vectors are used to pad the end of the sequence until the set length is reached. Ultimately, each sample to be analyzed is represented as a two-dimensional feature sequence tensor, where the first dimension is the fixed time step, and the second dimension is the total number of dimensions of all metadata features.

[0038] S103, Deep Learning Model Construction and Training The core of this step is to build a hybrid deep learning model that can automatically learn from preprocessed temporal metadata sequences and accurately distinguish between malicious and normal behavior. For example... Figure 4 As shown, the specific implementation is as follows: The input to a deep learning model is a preprocessed two-dimensional tensor of feature sequences. The forward propagation of this feature sequence follows a carefully designed hierarchical structure. First, the feature sequence is fed into a one-dimensional convolutional neural network (CNN). The convolutional kernels of the CNN slide along the time axis, designed to efficiently extract correlation patterns and short-term fluctuations between multiple features within a local timeframe. For example, a convolutional kernel with a width of 3 can simultaneously observe the states of dozens of features at three adjacent time points, thus learning local combination patterns such as "a surge in uplink packet length accompanied by a sharp decrease in downlink packet length over three consecutive time points," which is highly effective in detecting the initial stages of data leakage. Deep learning models typically contain two to three consecutive layers of one-dimensional CNNs, each followed by a non-linear activation function and a max-pooling layer. The pooling layer is used to reduce dimensionality and enhance the spatial invariance of features. Subsequent CNN layers progressively abstract more discriminative high-level local feature sequences and compress the sequence length.

[0039] Subsequently, the high-level local feature sequences extracted by the one-dimensional convolutional neural network layers are fed into long short-term memory (LSM) network layers. LSM networks, through their ingenious internal input, forget, and output gate mechanisms, selectively remember or ignore information, thereby modeling long-term dependencies and dynamic evolution trends across the entire input sequence. This is crucial for identifying long-term patterns such as periodic behaviors (e.g., heartbeats in command and control channels), slowly increasing scanning activity, or gradual shifts in behavioral patterns within a session. Deep learning models typically employ stacked two-layer LSM networks to enhance their ability to model complex temporal dynamics.

[0040] To further improve the performance of deep learning models, an attention mechanism is introduced after the Long Short-Term Memory (LSTM) network layer. This attention mechanism allows the deep learning model to dynamically assign different importance weights to features at different time steps in the sequence, enabling it to adaptively focus on key time segments most relevant to malicious behavior. The deep learning model computes a weighted context vector that integrates information from all time steps, highlighting key components and thus enhancing the model's discriminative power and interpretability.

[0041] Finally, the context vector generated via the attention mechanism is fed into a fully connected neural network for final classification. The fully connected neural network consists of several fully connected layers, employing non-linear activation and random deactivation techniques to prevent overfitting. The final layer of the fully connected neural network is a two-dimensional Softmax classifier that outputs a two-dimensional probability vector, representing the confidence levels of whether the current input sequence belongs to "normal" traffic and "malicious" traffic, respectively, with the sum of the two confidence levels being one.

[0042] Training deep learning models relies on large-scale, high-quality labeled datasets. Normal traffic samples need to broadly cover various business types and encryption scenarios, while malicious traffic samples need to include diverse real and simulated attack traffic. The dataset is randomly shuffled and divided into training, validation, and test sets in approximately a 7:1:2 ratio. During training, the cross-entropy loss function is used to measure the difference between the model's predicted probability distribution and the true labels, and an adaptive moment estimation algorithm is employed for optimization. The performance of the deep learning model is monitored on independent validation sets during training, and an early stopping strategy is used to prevent overfitting. Ultimately, the model parameters that perform best on the validation set are retained.

[0043] S104, Real-time Detection and Response Execution This step deploys the fully trained deep learning model into a real-world production environment, enabling millisecond-level real-time threat detection and automated handling of encrypted traffic. For example... Figure 5 As shown, the specific implementation is as follows: The first step is model deployment. The trained deep learning model file is integrated into the inference engine deployed at critical network nodes (such as next-generation firewalls, unified threat management devices, or dedicated detection probes). This inference engine is highly optimized to leverage hardware acceleration for efficient forward propagation computation.

[0044] Next, a real-time inference pipeline is constructed. The streaming preprocessing and serialization module continuously operates in a streaming manner, generating standardized, fixed-length feature sequence 2D tensors in real time for each active data stream. These feature sequence 2D tensors are submitted to the inference engine immediately. Upon receiving the request, the inference engine quickly executes model calculations and outputs the malicious probability value of that traffic segment.

[0045] Based on the malicious probability value output by the deep learning model, the system executes decisions and responses. Administrators can pre-set one or more judgment thresholds. When the malicious probability exceeds the highest level threshold, the system determines it as high-confidence malicious traffic, and the policy execution engine will immediately take blocking actions, such as issuing instructions to the firewall through the application programming interface to drop all subsequent packets of the data stream and reset the connection. At the same time, it will generate the highest level security alarm event and report it to the security information and event management platform. When the malicious probability is in the intermediate range, the system may only generate an alarm and perform deep logging for further analysis. If the probability is below the security threshold, it is determined as normal traffic, and the system does not intervene in any way, ensuring seamless business operations.

[0046] S105, Online Learning and Model Iteration To address the continuous evolution of cyberattack methods, the system incorporates an online learning and model iteration mechanism, enabling its detection capabilities to adapt and evolve. For example... Figure 6 As shown, the specific implementation is as follows: The system continuously runs a closed-loop model optimization process. First, during production testing, the system selectively collects metadata and related contextual information from traffic samples where the deep learning model's decision confidence is relatively low (e.g., probability values ​​close to the decision threshold). These "fuzzy" samples represent cases at the model's current knowledge boundary and are most likely to contain novel attack patterns or rare variations of normal business operations.

[0047] The collected samples are fed into a semi-automated annotation pipeline. The system first uses externally updated threat intelligence sources (such as the VirusTotal API and commercial threat subscriptions) for automated initial screening and annotation. Samples that the automated system cannot identify are pushed to the security analyst console for manual review and final annotation by experts. All annotated samples are stored in a dedicated incremental training sample library.

[0048] Periodically, the system initiates offline model update tasks. The update process employs an incremental learning strategy, not retraining from scratch, but rather training on the existing production model parameters primarily using newly added labeled samples. To prevent the model from forgetting old knowledge while learning new knowledge, some historical core samples are used in conjunction with the training process. After training, the new model undergoes rigorous evaluation on independent test sets containing both new and old threats to ensure its overall performance meets release standards. Once the evaluation is passed, a secure hot-update technology seamlessly replaces the old model in the production environment with the new model, thus upgrading and iterating detection capabilities without service interruption and achieving system self-evolution.

[0049] Through the complete closed loop formed by the above five steps, this invention achieves continuous, high-precision, and adaptive security awareness and proactive protection of encrypted network traffic.

[0050] Example 2 This invention also provides an artificial intelligence-based network security big data security processing system. The system is a comprehensive hardware and software system integrating data acquisition, real-time analysis, intelligent decision-making, and continuous learning. Each module works collaboratively through clearly defined interfaces and data flows. For example... Figure 7 As shown, it specifically includes: Metadata Acquisition Engine Module The metadata acquisition engine module is the system's data awareness layer, responsible for capturing raw network traffic and extracting core metadata at high speed and without loss. Deployed on critical network paths, this module is typically implemented using a high-performance packet processing framework to ensure line-rate processing capabilities. Its workflow begins with real-time ingestion of mirrored or split data in the network link. For each arriving packet, the engine quickly parses its network and transport layer headers, merging them into the corresponding data flow session context based on the five-tuple information. The engine maintains a set of dynamic accumulators and state machines for each active flow, updating dozens of fundamental metrics in real time, including bidirectional packet count, byte count, packet length statistics, arrival time interval sequence, and the number of times transmission control protocol flags appear. These calculations are strictly limited to the unencrypted header and spatiotemporal information. Periodically or at the end of a flow, the engine encapsulates the aggregated and calculated structured metadata feature vector, along with flow identifiers and time window information, into a standard-format message and publishes it to the high-speed message middleware bus, achieving decoupling and buffering from subsequent processing modules.

[0051] Streaming preprocessing and serialization module The streaming preprocessing and serialization module is the data normalization layer, responsible for transforming the raw metadata stream into a normalized sequence that the model can consume. This module acts as a consumer of the message middleware, continuously subscribing to metadata messages. Internally, it maintains a finite-length FIFO cache for each active stream, storing processed feature vectors from the stream over several recent time windows. For a newly arrived metadata message, the module first performs data cleaning logic, filtering or correcting outliers by comparing them with a pre-loaded feature value domain table. Then, it applies the normalization parameters determined during training to normalize the vector, eliminating the influence of dimensions. Next, the module retrieves the historical feature vectors from the cache and concatenates them with the new vector in chronological order, padding with zero vectors if the length is insufficient, thus constructing a feature sequence of fixed time length. Finally, this sequence is assembled into a model inference request containing the necessary context and sent to the backend intelligent analysis service.

[0052] AI Inference Service Module The AI ​​inference service module is the intelligent core of the system, carrying out and executing the computation of the deep learning model. This module is typically deployed as a microservice cluster, possessing elastic scalability. It receives inference requests from the preprocessing module. To improve computational efficiency, the service employs dynamic batch processing technology, aggregating multiple independent requests into a batch tensor within an extremely short latency window. This batch tensor is fed into a hybrid model of a one-dimensional convolutional neural network and a long short-term memory network, already loaded into a high-speed computing device. The model performs forward propagation, outputting a probability value for each request's traffic as malicious. The service then maps the probability results to the original request identifiers and returns the inference results. This module's design ensures high throughput and low latency, meeting the requirements of real-time detection.

[0053] Strategy Decision and Execution Engine Module The strategy decision-making and execution engine module is the system's response and control layer, responsible for translating the conclusions of intelligent analysis into practical actions. This module receives result streams from the AI ​​inference service module. Internally, it maintains a configurable security policy rule base, with rules defining the corresponding actions and alert levels for different malicious probability ranges. The engine matches the probability value in each inference result with the policy rules to generate specific execution instructions. These instructions may include: immediate blocking of high-threat traffic by issuing blocking policies through the application programming interface of a next-generation firewall or software-defined network controller; generating structured security alert events of different levels and sending them to the security information and event management system or log management platform; or allowing normal traffic without any action. This module ensures a closed loop of detection and response, directly improving the initiative and effectiveness of security protection.

[0054] Online learning and model management module The online learning and model management module is the evolutionary hub of the system, responsible for managing the entire lifecycle of the model and driving its continuous optimization. This is a management-oriented system comprising multiple sub-components. The sample data lake sub-component continuously collects low-confidence judgment samples, confirmed threat samples, and external intelligence from the production environment, forming a raw data reserve. The annotation workbench sub-component provides a human-computer interaction interface and automated toolchain, supporting security experts in efficiently reviewing and accurately annotating suspicious samples. The incremental training pipeline sub-component, triggered periodically or quantitatively, extracts new annotated samples from the data lake, mixes them with some historical data, and initiates fine-tuning training tasks based on the current production model in an independent training cluster, producing a new model version with superior performance. The model repository sub-component manages, stores, and verifies model assets through versioning. Finally, through automated continuous integration and continuous deployment pipelines, verified new models are securely and smoothly rolled out to the production environment of the AI ​​inference service module, thereby completing iterative upgrades of detection capabilities without interrupting business operations. The entire platform provides comprehensive operational visibility through a monitoring dashboard.

[0055] The above five core modules are connected in a loosely coupled manner (mainly relying on message queues and API calls) to form a scalable, highly available, and self-evolving intelligent security detection and response system for encrypted traffic.

[0056] Example 3 The present invention also provides an electronic device, including: a processor, a transmitting device, an input device, an output device, and a memory. The processor may be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application. The memory may be implemented using a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), and is used to store computer program code. The computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes a method as described in any of the above possible implementation methods.

[0057] Example 4 The present invention also provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a processor of an electronic device, cause the processor to perform a method as described in any of the above possible implementations.

[0058] In the description of this specification, the references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0059] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for secure big data processing in cybersecurity based on artificial intelligence, characterized in that, Includes the following steps: S101. Encrypted traffic metadata collection: Non-intrusive capture of raw encrypted data packets from network communication links and extraction of metadata features; the metadata features include basic transmission features, timing features, and distribution features, and the extraction process does not require decryption of the encrypted traffic payload; S102, Metadata preprocessing and sequence construction: Data cleaning and feature standardization are performed on the metadata features, and a fixed-length feature input sequence is constructed to provide input for the deep learning model; S103. Deep learning model construction and training: Construct a hybrid deep learning model and train the hybrid deep learning model using the preprocessed feature input sequence; the hybrid deep learning model adopts a hybrid architecture combining a one-dimensional convolutional neural network and a long short-term memory network. S104. Real-time detection and response execution: Deploy the trained hybrid deep learning model into the inference engine to perform threat detection on real-time traffic and execute response actions based on the detection results. S105. Online learning and model iteration: collect low-confidence traffic samples, label and incrementally train them to achieve continuous optimization and adaptive evolution of the model. In step S101, the encrypted traffic metadata collection includes: deploying data collection probes at key network nodes to capture raw encrypted data packets in a bypass monitoring mode; parsing the header information of each raw encrypted data packet and extracting the five-tuple information, wherein the five-tuple includes source address, destination address, source port, destination port and protocol type; associating data packets as data streams based on the five-tuple and extracting the metadata features.

2. The method according to claim 1, characterized in that, In step S101, the basic transmission characteristics include the number of data packets in the uplink and downlink directions, the total number of bytes, the byte ratio, the statistical value of data packet length, the session duration, the statistics of transmission control protocol flag bits, and the data retransmission rate; the timing characteristics include the statistical value of the data packet arrival time interval and the burst index of the traffic rate; the distribution characteristics include the histogram of data packet length distribution and information entropy.

3. The method according to claim 1, characterized in that, In step S102, the metadata preprocessing and sequence construction include: cleaning the metadata features, including identifying and replacing outliers and using forward padding to handle missing values; standardizing the cleaned metadata features to eliminate the influence of dimensions; constructing a fixed-length feature sequence, using zero-value vectors to pad sequences with insufficient length, and using a sliding window to truncate subsequences for sequences with excessive length.

4. The method according to claim 1, characterized in that, In step S103, the one-dimensional convolutional neural network is used to extract local correlation features in the feature input sequence, and the long short-term memory network is used to extract long-term dependencies in the feature input sequence. During training, the cross-entropy loss function and adaptive moment estimation algorithm are used, and an early stopping strategy is adopted to prevent overfitting.

5. The method according to claim 1, characterized in that, In step S104, the real-time detection and response execution includes: inputting the feature input sequence into the inference engine to obtain a malicious probability value; making a decision on the malicious probability value according to a preset threshold; when the malicious probability value exceeds a first threshold, executing a blocking action and generating an alarm; when the malicious probability value is in the middle range, generating an alarm and recording a log; and when the malicious probability value is lower than a second threshold, allowing traffic.

6. The method according to claim 1, characterized in that, In step S105, the online learning and model iteration includes: collecting the low-confidence traffic samples and labeling them using external threat intelligence sources or manual judgment; using an incremental learning strategy to train the model based on existing model parameters and the newly added samples; and deploying the new model to the production environment using model hot update technology.

7. A network security big data security processing system based on artificial intelligence, wherein the system applies the method as described in any one of claims 1 to 6, characterized in that, The system comprises various modules that work together with data streams through clearly defined interfaces: The metadata acquisition engine module, deployed on the critical path of the network, is implemented based on a high-performance packet processing framework and is used to capture raw network traffic and extract metadata features. The streaming preprocessing and serialization module includes data cleaning logic, standardization transformation and sequence construction unit, used to clean, standardize and serialize the metadata features to generate feature input sequences; The AI ​​inference service module is deployed in the form of a microservice cluster, supports dynamic batch processing, and is used to load hybrid deep learning models and perform inference on the feature input sequence. The strategy decision and execution engine module integrates a configurable security policy rule base, which is used to execute corresponding security response operations based on the results of the inference. The online learning and model management platform module is used to manage the model lifecycle and perform model iterations.

8. The system according to claim 7, characterized in that, The online learning and model management platform module includes a sample data lake sub-component, an annotation workbench sub-component, an incremental training pipeline sub-component, and a model repository sub-component. The sample data lake sub-component is used to store collected sample data. The annotation workbench sub-component supports a combination of automated and manual sample annotation. The incremental training pipeline sub-component is used to trigger and execute incremental training tasks for the model. The model repository sub-component is used for version management, storage, and verification of models, and updates the AI ​​inference service module with verified new models through continuous integration and continuous deployment pipelines.

9. An electronic device comprising a processor, a memory, and a computer program stored in the memory, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-6.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images