Cvoca feature extraction method and system based on multi-modal large model

Abstract

The application discloses a CVOCA feature extraction method and system based on a multi-modal large model, belongs to the technical field of multi-modal feature extraction, and comprises data input; data preprocessing, processing the input data to generate standardized input; feature uniform representation, mapping visual, text and time sequence data to a uniform feature space by using a cross-modal uniform encoder; feature optimization, dynamically adjusting the mode weight based on a dynamic feature optimizer to realize efficient feature fusion; context perception modeling, capturing long-range time sequence dependence; adversarial optimization, improving the model robustness through adversarial training; a feature storage and updating system, adopting an expert network and a gating mechanism to realize dynamic updating of a feature library; and the application solves the defects of the existing network behavior analysis technology in multi-modal data fusion, calculation efficiency, adversarial robustness and dynamic adaptability, and improves the technical performance and application value.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of multi-modal feature extraction, and particularly relates to a CVOCA feature extraction method and system based on a multi-modal large model. BACKGROUND

[0002] In recent years, multi-modal models based on contrast learning (such as CLIP) have become the core technology for cross-modal alignment, which realizes cross-modal retrieval, classification and other tasks by mapping images and texts to a unified semantic space. However, with the complication of large model application scenarios (such as network behavior analysis and industrial quality inspection), the existing technology faces the following challenges:

[0003] Inadequate long text processing capability: due to the absolute position encoding limitation, the text input of traditional CLIP is limited to 77 tokens, and the effective semantic length is less than 20 tokens, resulting in loss of fine-grained information.

[0004] Defects in dynamic feature fusion: existing methods mostly use static feature splicing (such as simply concatenating image and text features), lack of context-based adaptive weight distribution, and are difficult to capture the timing association and core attributes in multi-modal data.

[0005] Weakness in adversarial robustness: the feature extraction process is easily disturbed by adversarial samples, and existing defense means rely on data augmentation, lacking an active optimization mechanism in the feature space.

[0006] Computational efficiency bottleneck: when processing high-resolution network data (such as full-flow matrices), the complexity of the traditional Transformer architecture results in high memory usage, making it difficult to meet real-time requirements.

[0007] Secondly, in the existing technology, Long-CLIP is aligned and fine-tuned (PCA extracts key features) with the core attributes through position encoding hierarchical interpolation (the first 20 bits are reserved, and the last 57 bits are extended interpolation), to improve the long text processing capability of CLIP. However, there is no dynamic weight mechanism, and multi-modal feature fusion still relies on static splicing, making it difficult to adapt to timing changes in network behavior; without introducing feature space gradient constraints, the performance of adversarial samples decreases significantly, and the memory usage is still as high as 16GB when processing high-resolution data, making it impossible to adapt to edge device deployment requirements.

[0008] In the prior art, jina-clip-v1 is a joint optimization of text-image and text-text contrast loss, adopts a multi-stage training strategy (short text pre-training + long text fine-tuning + difficult negative sample optimization), and improves the performance of cross-modal and pure text tasks. However, the text-image alignment and text-text alignment are optimized independently, and the unified dynamic feature space is not realized, which limits the accuracy of multi-modal correlation detection; the three-stage training process is complex, and the inference delay reaches milliseconds, which cannot meet the real-time response requirements of network behavior analysis.

[0009] In the prior art, the lightweight image description generation based on CLIP combines the CLIP encoder and the multi-head sparse attention mechanism, reduces the computational complexity through Top-K sparsification, and realizes lightweight image description generation. However, the attention mechanism is only used for single-modal internal, and the dynamic interaction optimization of image-text features is not realized; no adversarial training module is designed, and the generated description is easily disturbed by input noise.

[0010] In the prior art, self-training Chinese CLIP multi-modal retrieval is to construct a Chinese CLIP model, realize cross-modal fast retrieval through approximate nearest neighbor algorithm (ANN), and support text-image bidirectional matching. However, there is no dynamic weight visualization tool provided, it is difficult to locate the key behavior features; the position encoding is not optimized, and the long text retrieval accuracy is reduced by 15% compared with the English model.

[0011] The existing network behavior analysis technology faces the following core bottlenecks, which restrict the safety protection efficiency in the era of large models:

[0012] Multi-modal data fragmentation problem: traditional methods (such as CNN processing traffic images, RNN analyzing log text) use independent models to process different modal data, which makes it difficult to realize cross-modal semantic alignment of network traffic, operation logs, system calls and other multi-source heterogeneous information, and cannot capture cross-modal correlation behavior features.

[0013] Feature extraction efficiency and resource consumption contradiction: when the existing large model directly processes high-resolution network data (such as full-flow packet capture data), the computational complexity increases in a quadratic manner (such as the attention mechanism of Transformer), which leads to high memory occupation and poor real-time performance.

[0014] Data evidence: tests show that the traditional ViT model processes a 4096x4096 traffic matrix, and the GPU memory occupation exceeds 24GB, which cannot be deployed on edge devices.

[0015] Feature vulnerability in adversarial environment: by injecting noise traffic (such as DNS tunnel obfuscation) and changing log formats, the robustness of traditional feature extraction models is significantly reduced.

[0016] Technical defects: existing methods rely on passive defense such as data augmentation, and lack active adversarial optimization mechanism in feature space.

[0017] Dynamic behavior pattern adaptation is insufficient: it presents long cycle, low frequency and multi-stage characteristics, and traditional feature extraction methods based on fixed time window are difficult to capture cross-session context association, with a false negative rate of more than 35%. SUMMARY

[0018] In view of the problems mentioned in the background art, the present application proposes a CVOCA feature extraction method and system based on a multi-modal large model, focusing on the problems of cross-modal fragmentation, low computational efficiency, vulnerability to attack, and insufficient dynamic adaptability of network behavior analysis in the era of large models. Through the CVOCA feature architecture, technical breakthroughs are achieved in multi-modal semantic alignment, dynamic resource optimization, active defense enhancement, and context-aware modeling, providing an efficient and reliable network behavior feature extraction scheme for the next generation of intelligent security systems.

[0019] Technical solution: in order to solve the above technical problems, the technical solution adopted by the present application is as follows:

[0020] A CVOCA feature extraction method based on a multi-modal large model, comprising the following steps:

[0021] S1: data input, including network traffic data, log text data, and system call sequence;

[0022] S2: data preprocessing, processing network traffic data, log text data, and system call sequence to generate standardized input;

[0023] S3: feature uniform representation, using a cross-modal uniform encoder to map visual, text, and time series data to a unified feature space;

[0024] S4: feature optimization, based on a dynamic feature optimizer, dynamically adjusting the modal weight and realizing efficient feature fusion;

[0025] S5: context-aware modeling, using causal Transformer combined with local attention window to capture long-range temporal dependence;

[0026] S6: adversarial optimization, improving model robustness through adversarial training;

[0027] S7: feature storage and updating system, using an expert network and a gating mechanism to realize dynamic updating of the feature library.

[0028] As a preferred, in S2, the specific preprocessing content includes:

[0029] S21: network traffic imaging: slice the original traffic data according to the time window and convert it into a gray spectrum image;

[0030] S22: Log text structuring: regular expression is used to extract key fields;

[0031] S23: Building protocol-behavior keyword dictionary: annotating protocol keywords for regular matching extracted fields;

[0032] S24: System call sequence coding: mapping system call ID to 32-dimensional word vector;

[0033] Generate time series feature blocks through sliding window.

[0034] As a preferred, the specific process of S3 is:

[0035] S31: Visual coding, using improved ViT-B / 32 model, adding local sensitive hash module to embedding layer for local feature enhancement of traffic spectrum graph;

[0036] S32: Text coding, based on improved BERT architecture, introducing dynamic position encoding;

[0037] S33: Contrastive learning alignment, setting double-flow contrastive loss function to align visual and text feature spaces;

[0038] S34: Time series coding, using Transformer encoder for processing, converting each system call word vector in the sequence into a context-aware time series feature vector.

[0039] As a preferred, the specific process of S4 is:

[0040] S41: Introducing differentiable gating mechanism, calculating modality importance weight based on MLP network;

[0041] S42: Dynamic feature fusion;

[0042] S43: Resource-aware scheduling: dynamically adjusting feature slice size according to real-time state of device video memory.

[0043] As a preferred, the specific process of S5 is:

[0044] S51: Causal Transformer architecture: setting multi-head attention with causal mask;

[0045] S52: Session-level state tracking: using LSTM to maintain session state vector.

[0046] As a preferred, the specific process of S6 is:

[0047] S61: Adversarial sample generation;

[0048] S62: Based on FGSM algorithm, calculating adversarial loss.

[0049] As preferred, the specific process of S7 is:

[0050] MoE incremental learning: set up an expert hybrid system to calculate each expert's focus on a specific disturbance type;

[0051] Feature version management: store feature update records based on blockchain to ensure traceability.

[0052] A CVOCA feature extraction system based on a multi-modal large model, which implements the CVOCA feature extraction method based on a multi-modal large model described in any of the above, comprises:

[0053] A multi-modal input preprocessing module for standardizing the input heterogeneous data;

[0054] A cross-modal unified encoder for mapping multi-source information to a shared semantic space and completing the initialization modeling of inter-modal semantic association;

[0055] A dynamic feature optimizer for adaptively adjusting the weights and suppressing redundancies of the preliminary features based on real-time feedback mechanisms;

[0056] A context-aware modeling module for incorporating timing association and scene dependency information to construct high-order feature representations with logical coherence;

[0057] An adversarial robustness enhancement module for improving feature anti-interference capability through adversarial training and noise injection strategies;

[0058] A feature storage and update system for realizing incremental accumulation and version management of feature knowledge, forming a continuously evolving multi-modal feature knowledge base.

[0059] Advantages: Compared with the prior art, the present application has the following advantages:

[0060] (1) The present application solves the core pain points of existing network behavior analysis technology in multi-modal data fusion, computational efficiency, adversarial robustness and dynamic adaptability through the innovative CVOCA multi-modal feature extraction architecture, significantly improving the technical performance and application value.

[0061] (2) The present application breaks through the fragmentation of multi-modal data and realizes efficient semantic alignment. Traditional methods use independent models to process visual, textual and timing multi-modal data, resulting in a high loss rate of cross-modal associated features of up to 35%, making it difficult to capture the deep correlation between multi-modal data. The present application maps multi-modal data to a unified feature space through a dual-flow contrastive learning framework, achieving cross-modal semantic alignment: In the CTU-13 dataset test, the cross-modal detection accuracy is improved to 98.7% (traditional method is 91.2%), with an improvement of 7.5%.

[0062] In existing technologies, the correlation pattern between abnormal software encryption behavior (text logs) and abnormal traffic surges (visual spectrograms) can be successfully identified, with a detection rate improvement of 27.4%.

[0063] (3) This invention significantly improves computational efficiency and is suitable for edge deployment. Existing large models consume excessive amounts of GPU memory and have long computational latency when processing high-resolution network data, which cannot meet the resource constraints and real-time requirements of edge devices. This invention introduces a hierarchical feature slice injection and dynamic resource scheduling mechanism, which greatly optimizes computational efficiency: when processing a 4K×4K traffic matrix, GPU memory usage is reduced to 9.2GB (compared to 24GB in the traditional method), a reduction of 62%. Inference latency is reduced from seconds to milliseconds (less than 15ms), meeting the requirements of real-time detection. On edge devices (ARM Cortex-A72), feature extraction speed reaches 153 frames / second (compared to 52 frames / second in the traditional method), an improvement of 194%.

[0064] (4) This invention enhances adversarial robustness. The detection accuracy of traditional feature extraction models drops sharply to 54.3%, failing to effectively cope with advanced techniques such as DNS tunneling obfuscation and HTTP requests. This invention significantly improves model robustness through feature space gradient inversion (GRL) and adversarial distillation training: in the adversarial test set with injected noisy traffic and change logs, the detection accuracy remains at 89.5% (54.3% for traditional methods), an improvement of 64.8%. The success rate of external defense is improved by 65.2%.

[0065] (5) This invention improves dynamic adaptability. It adopts MoE incremental learning and dynamic Prompt engineering to enhance the model's dynamic adaptability. After importing the new feature template, the model update time is shortened to 2 hours (72 hours for traditional methods), a speedup of 96%. The early recognition rate of APT is improved to 82.4% (less than 50% for traditional methods). Through the USB offline update mechanism, it supports edge devices to quickly load the latest feature library in environments without network access.

[0066] (6) This invention can optimize user experience and improve operation and maintenance efficiency; existing systems are complex to operate, security analysts need to manually analyze multi-source data, the false alarm rate is high, and the operation and maintenance burden is heavy; this invention optimizes user experience through a visual dashboard and one-click policy deployment; the average response time is shortened to less than 5 minutes (traditional methods > 30 minutes), a reduction of 83.3%. The false alarm rate is reduced to 1.2% (traditional methods 5.8%), significantly reducing the ineffective workload of operation and maintenance personnel.

[0067] (7) The present application aims at the core problems of multi-modal data fragmentation, low computational efficiency, vulnerability to confrontation, and insufficient dynamic adaptability in the prior art. Through cross-modal unified encoding, hierarchical feature optimization, adversarial training, and dynamic learning, the overall performance of network behavior analysis is improved. The above technical effects are verified in experimental data and actual scenarios, forming a complete closed loop from problem identification to technical breakthrough to effect landing, fully embodying the practical value and innovation of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 is a CVOCA multi-modal feature extraction system architecture schematic diagram of the present application;

[0069] Figure 2 is a schematic diagram of the front-end interaction logic of the present application;

[0070] Figure 3 is a schematic diagram of the background processing logic of the present application;

[0071] Figure 4 is a data processing flow schematic diagram of the present application;

[0072] Figure 5 is a CVOCA system network topology diagram of the present application;

[0073] Figure 6 is a CVOCA feature extraction timing diagram of the present application. DETAILED DESCRIPTION

[0074] The present application will be further illustrated below in conjunction with specific embodiments, which are implemented on the premise of the technical solutions of the present application, and it should be understood that these embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0075] Embodiment 1

[0076] The explanations of special abbreviations or self-defined terms used in this embodiment are as follows:

[0077] CVOCA: Cross-modal Volumetric Optimization Context-aware Adversarial Robustness, a multi-modal feature extraction architecture that integrates cross-modal unified encoding, dynamic feature optimization, context-aware modeling, and adversarial robustness enhancement technology, for improving the accuracy and robustness of network behavior analysis.

[0078] CLIP: Contrastive Language-Image Pre-training, a pre-training model for images and texts. It is a neural network model trained on large-scale image-text pair datasets through contrastive learning, which can learn the semantic correlation between images and texts.

[0079] ViT: Vision Transformer, an image encoder based on Transformer architecture for extracting image features. In this invention, ViT is used to process network traffic imaging data.

[0080] BERT: Bidirectional Encoder Representations from Transformers, a text encoder based on Transformer architecture for extracting text features. In this invention, BERT is used to process text log data in network behavior.

[0081] InfoNCE: Information Noise-Contrastive Estimation, a commonly used contrastive loss function for learning to distinguish positive and negative samples. In this invention, InfoNCE is used for contrastive learning in cross-modal unified encoding.

[0082] GPU: Graphics Processing Unit, a specialized electronic circuit for parallel computing, commonly used to accelerate deep learning tasks such as model training and inference in this invention.

[0083] Prompt Engineering: a technique for guiding pre-trained language models to perform specific tasks by setting specific text prompts (Prompts). In this invention, Prompt Engineering is used to dynamically generate protocol-related semantic features.

[0084] NLP: Natural Language Processing, a branch of computer science and artificial intelligence that studies how to make computers understand and process human language. In this invention, NLP technology is used to process text data in network behavior.

[0085] CNN: Convolutional Neural Network, a deep learning model commonly used in image processing and computer vision tasks. In this invention, CNN can be used as an alternative to ViT for image feature extraction.

[0086] Transformer: A neural network architecture based on self-attention mechanism, which has achieved great success in natural language processing and computer vision. In the invention, the Transformer architecture is used for text encoding and context-aware modeling.

[0087] CUDA: Compute Unified Device Architecture, a parallel computing platform and programming model developed by NVIDIA, used for general-purpose computing on GPUs, supporting the acceleration of deep learning tasks in the invention.

[0088] As shown in Figures 1-4 The multi-modal large model-based CVOCA feature extraction method provided by the embodiment includes the following steps:

[0089] S1: Data input;

[0090] Input network traffic data, log text data, and system call sequence; Network traffic: mirror to the network card through SPAN, and capture using libpcap. Operation log: syslog-ng collects to / var / log / . System call: strace tool captures process behavior.

[0091] libpcap: Packet Capture Library, an open-source, cross-platform network packet capture library, which is a basic component in the field of network analysis and security tools.

[0092] syslog-ng: System Logging - Next Generation, an open-source log management tool for Linux / Unix systems, designed to replace the traditional syslogd daemon, providing more powerful, flexible, and reliable log collection, transmission, processing, and storage functions.

[0093] S2: Data preprocessing;

[0094] Use the multi-modal input preprocessing module to process multi-source data such as network traffic, log text, and system calls, and generate standardized input.

[0095] Network traffic imaging: slice the original traffic data (such as PCAP) by time window (default 1 second) and convert it into a grayscale spectrum image, specifically:

[0096]

[0097] Where M i , jRepresents the computed value for the cell located at coordinates (i,j) in the two-dimensional matrix (grayscale spectrogram). This value typically corresponds to the intensity of a pixel in the image. i Represents the time slice, j represents the protocol type index, and the weight table is defined according to the RFC standard. Applying CLAHE enhances the visibility of low-frequency features. The count() function represents the calculation of the aggregate value defined by the expression inside the parentheses. In the context of network traffic analysis, this usually refers to counting or summing the attributes of network packets that meet certain conditions. size Represents the size of a single network packet, usually measured in bytes (Bytes). protocol weight [j] represents the weight value associated with the jth protocol type. According to the document description, this weight table is defined according to the RFC standard.

[0098] libpcap parses traffic-time window slices (1 second) - generates grayscale spectrogram.

[0099] Log text structuring: Extract key fields (such as IP addresses, HTTP methods, return codes) using regular expressions.

[0100] Build protocol-behavior keyword dictionary (contains 14 types of protocol key operation labels such as HTTP / HTTPS / DNS).

[0101] Regular matching to extract fields - label protocol keywords (such as HTTP GET).

[0102] System call sequence encoding: Map system call IDs to 32-dimensional word vectors (pre-trained on Linux kernel call datasets).

[0103] ID mapping to 32-dimensional word vectors - time series block (window size 256).

[0104] Generate time series feature blocks through sliding windows (window size 256, step size 64).

[0105] S3: Unified representation of features;

[0106] Use cross-modal unified encoder to map visual, textual, and time series data to a unified feature space.

[0107] Visual encoding: Use ViT-LSH model (pre-trained on ImageNet + internal traffic dataset).

[0108] ImageNet is a large visual database (dataset) designed for visual object recognition software research. It contains over 14 million manually annotated images covering more than 20,000 categories. In the field of deep learning, especially in computer vision, ImageNet is widely used as a dataset for model pre-training.

[0109] An improved ViT-B / 32 model is adopted, and a local sensitive hashing (LSH) module is added to the embedding layer to enhance the local features of the traffic spectrum graph:

[0110]

[0111] where, is the weight matrix; LSH(x) represents the result of the Locality-Sensitive Hashing (LSH) transformation on the input vector x. x represents the input vector, representing high-dimensional data points that need to be hashed, such as user behavior features, network traffic features, etc. sign represents the sign function, which maps the output of the hash function to binary hash codes {-1,1}. b hash represents the bias term, used to adjust the threshold of the hash function, so that the hash function can more flexibly adapt to different data distributions, and improve the discrimination of the hash code. W hash represents the weight matrix, used to map the input vector x to the hash space, defining the direction of the hash function, so that similar input vectors have similar hash codes in the hash space.

[0112] Text encoding: use BERT-DPE model (pre-trained on Wikipedia + security log corpus).

[0113] Based on the improvement of BERT architecture, dynamic position encoding (DPE) is introduced, specifically:

[0114]

[0115] where, (pos,i) is a learnable scaling factor that adapts to different protocol text lengths; PE (pos,2i) represents the position encoding value at position pos, with a dimension of 2i. pos is the position index, representing the position of an element in the sequence, for example, in natural language processing, it represents the position of a word in a sentence. i represents the dimension used to specify which dimension of the encoding is currently being calculated. 2i represents the even index of the dimension. In position encoding, sine and cosine functions are usually used to generate encoding of different dimensions, and 2i is used to select the dimension corresponding to the sine function. d modelThe dimensionality of the model, i.e., the length of the position encoding vector, is usually the same as the hidden layer dimension of the model in models such as the Transformer.

[0116] Contrastive learning alignment:

[0117] Set up a double-flow contrastive loss function L align , which forces the visual and text feature spaces to align, specifically:

[0118]

[0119] where S() is the cosine similarity, τ = 0.05, and N is the batch size; v i represents the visual feature vector of the i-th sample. In multi-modal learning, visual features usually come from visual data such as images, videos, etc. i represents the text feature vector of the i-th sample. In multi-modal learning, text features usually come from text data such as text descriptions, labels, etc. j represents the text feature vector of the j-th sample. When calculating the alignment loss, the similarity between the current visual feature and all other text features needs to be considered, so j is used to traverse all text features.

[0120] Temporal encoding: use Temporal Convolutional Network (TCN). Temporal Convolutional Network is a neural network architecture specially designed for processing sequential data such as time series, natural language, audio, video, etc.

[0121] Specific process of temporal encoding:

[0122] Input preparation: receive pre-processed system call sequence data, i.e., sequences that have been mapped to 32-dimensional word vectors.

[0123] These vector sequences are usually organized in time series feature blocks generated by a sliding window mentioned in the protocol-behavior keyword dictionary, with each block representing system call activity within a time slice.

[0124] Encoding architecture selection and core processing:

[0125] Use a Transformer-based architecture as the core encoder. This echoes the causal Transformer architecture used in context-aware modeling, indicating that temporal encoding is its basis or component.

[0126] Core operations:

[0127] Self-Attention Mechanism: Transformer encoder layers utilize a self-attention mechanism to compute relationship weights between each system call vector (word vector) in the sequence and all other call vectors. This allows the model to capture long-range dependencies and complex patterns between calls, regardless of their position in the sequence.

[0128] Positional Encoding: Since Transformers inherently do not contain sequence order information, positional encoding needs to be added. This can be standard sinusoidal / cosine encoding, or a variant of the dynamic positional encoding mentioned in the text encoding, to inject absolute or relative position information of each call occurrence in the sequence into the model.

[0129] Feed-Forward Neural Network: Each encoder layer also contains a feed-forward neural network (FFN) that performs a non-linear transformation on the output of self-attention, increasing the model's expressive power.

[0130] Residual Connection and Layer Normalization: After each sub-layer (self-attention and FFN), a residual connection and layer normalization are applied to stabilize the training process and facilitate learning in deep networks.

[0131] Output Representation: After processing by several layers of Transformer encoders, each system call word vector in the sequence is converted into a context-aware temporal feature vector.

[0132] These output vectors not only contain the original call semantic information (from word embeddings), but also incorporate temporal dependency information in the context of the entire sequence block.

[0133] The output of the entire sequence block can be represented as a series of context vectors, or a fixed-length temporal feature vector obtained by pooling (such as average pooling, max pooling, or taking the vector at the beginning / special token of the sequence), for subsequent multi-modal fusion.

[0134] Integration with Overall Scheme:

[0135] Unified Feature Space: The output of the temporal encoder (context vectors or pooled feature vectors) needs to be aligned in dimension and semantics with other modalities (such as visual features, text features) to enter the "unified feature space". This usually means that the dimension of the output vector needs to match the output dimension of other modality encoders (for example, both mapped to 256 dimensions or 512 dimensions).

[0136] Foundation for high-level modeling: The context-aware features obtained through temporal encoding are the direct input for "context-aware modeling" (causal Transformer combined with LSTM for session-level state tracking). The temporal encoder is responsible for extracting basic, fine-grained inter-call dependencies, while the context-aware modeling module may be responsible for integrating this information on a longer time scale or across sessions.

[0137] S4: Feature optimization;

[0138] Traditional static feature fusion cannot adapt to the dynamic changes of network behavior. The application is based on a dynamic feature optimizer, which dynamically adjusts the modal weight and realizes efficient feature fusion. The differentiable gating mechanism calculates the importance weight of the modal, specifically:

[0139]

[0140] Where W m represents the attention weight of the mth element (such as a word, image region, etc.). In the attention mechanism, the weight is used to represent the importance or relevance of the element. f m is the modal feature, and C is the global context vector. is the concatenation operation, and MLP represents a multi-layer perceptron used to map the feature vector and context vector to the space of attention weights. Softmax represents the softmax function used to convert the output of the multi-layer perceptron into a probability distribution, ensuring that the sum of the attention weights of all elements is 1.

[0141] Feature fusion, specifically:

[0142]

[0143] Where Proj m is the modal projection layer, f fusion represents the fused feature vector. In multi-modal fusion, it is often necessary to fuse the feature vectors of different modalities into a unified representation for subsequent task processing, f fusion represents this fused result. m represents the index or number of elements, used to traverse and identify the elements that need to be fused. W m represents the attention weight of the mth element. In the attention mechanism, the weight is used to represent the importance or relevance of the element, used for weighted fusion of the feature vector, so that important elements occupy a larger proportion in the fused feature vector. f m is the modal feature.

[0144] Resource-aware scheduling: dynamically adjust the feature slice size according to the real-time state of the device memory.

[0145] Resource-aware scheduling:

[0146] 1) Input parameters include matrix, mem_status; matrix represents the high-resolution network traffic spectrum graph to be processed, and mem_status represents the real-time status of the device video memory. The value is 'high': the video memory is sufficient (such as server side ≥ 16GB), and the value is other values (such as 'low'): the video memory is tight (such as edge device ≤ 8GB).

[0147] 2) The judgment logic is: max_patch = 128 if mem_status == 'high' else 64.

[0148] When the video memory is sufficient (mem_status == 'high'): set the slice size to 128x128, large slice reduces the number of calculations and improves processing efficiency; When the video memory is insufficient (other states): set the slice size to 64x64, small slice reduces the single memory occupation to avoid overflow.

[0149] 3) The actual slicing is performed by calling the split_matrix() function, which divides the original matrix into several sub-blocks according to the calculated size and outputs them.

[0150] S5: Context modeling;

[0151] Use the context-aware modeling module to capture cross-session long-time dependencies.

[0152] Input: fused feature sequence .

[0153] Causal Transformer architecture:

[0154] Build multi-head attention with causal mask:

[0155]

[0156] Where M causal is the causal mask (Causal mask) matrix, which ensures the time sequence dependency, Q represents the query (Query) matrix, which represents the features of the current element or position that needs to be focused on. K represents the key (Key) matrix, which represents the features of all elements or positions in the sequence, and generates attention weights by calculating the similarity with Q. V represents the value (Value) matrix, which contains the actual information of all elements or positions in the sequence, and generates the final output by weighted sum according to the attention weight. T represents the transpose operation; d k represents the dimension of the key (Key dimension), and the dot product result will be scaled to prevent the softmax function from saturating when calculating the attention weight. Mcausal Causal mask matrix, which ensures that only the current position and previous positions are considered when calculating attention, preventing the model from seeing future information. Element-wise multiplication, also known as Hadamard product, which multiplies the elements at corresponding positions in two matrices to obtain a new matrix.

[0157] Introducing a local attention window (window size 256), reducing the computational complexity from to .

[0158] Session-level state tracking: using LSTM to maintain the session state vector, the specific calculation formula is:

[0159]

[0160] where f t is the current time feature, h t is the hidden state; h t-1 represents the hidden state of the previous time. In the LSTM network, the hidden state is used to transmit long-term dependencies in the sequence, which contains all the information up to the previous time. LSTM network controls the flow and memory of information through mechanisms such as update gate, forget gate and output gate, so as to capture long-term dependencies in the sequence.

[0161] Output: context-enhanced features .

[0162] S6: adversarial optimization;

[0163] Based on the adversarial robustness enhancement module, the model robustness is improved through adversarial training.

[0164] Adversarial sample generation, specifically:

[0165]

[0166] where represents the input adversarial perturbation, which is added to the original input sample to generate an adversarial sample, thereby misleading the model to make incorrect predictions. represents the perturbation amplitude, which is a hyperparameter used to control the size of the perturbation. Smaller values generate adversarial samples that are more difficult for the model to detect, but may also have less impact on the model's predictions. sign represents the sign function, which is used to extract the sign information of the gradient , that is, only the direction of the gradient is retained, and its size is ignored. denotes the gradient operator acting on the loss function L, which represents the gradient of the input x. L represents the loss function, which is used to quantify the difference between the model's prediction and the true label. f(x) represents the model's prediction for the input x. y represents the true label or expected output for the input x.

[0167] Based on the FGSM algorithm, the adversarial loss is calculated , specifically:

[0168]

[0169] where, denotes the input adversarial perturbation. It is usually added to the input data to generate an adversarial sample, which misleads the model to make incorrect predictions. f(x) represents the output value after processing or mapping the input x. In adversarial training, this usually represents the model's prediction for the original input x. f(x+δ) represents the new output obtained after adding the perturbation δ to the input. In adversarial training, this represents the model's prediction for the input x+δ after adding the adversarial perturbation.

[0170] S7: Feature storage and update system;

[0171] Feature update and version management, supporting incremental update and version management of feature library.

[0172] Storage format: HDF5 (feature vector) + JSON (metadata).

[0173] Version control: Git-like branch management, supporting rollback.

[0174] MoE incremental learning: Set up 8 expert hybrid systems, each expert focuses on a specific perturbation type:

[0175]

[0176] where, g i is the gating network, E i is the expert network; f update represents the updated value. This value is usually calculated by some update rule or algorithm, which reflects the latest state of the system or model at a certain time.

[0177] As Figure 1As shown, the CVOCA feature extraction system based on the multi-modal large model provided by the embodiment constructs an end-to-end intelligent processing architecture, the core of which is driven by six modules in cooperation: first, the multi-modal input preprocessing module realizes the standardization alignment and noise filtering of heterogeneous data such as text, image and audio; then the cross-modal unified encoder maps the multi-source information to the shared semantic space, completes the initialization modeling of the semantic correlation between modes; on this basis, the dynamic feature optimizer adjusts the adaptive weight and suppresses the redundancy of the preliminary features based on the real-time feedback mechanism; then the context perception modeling module integrates the timing correlation and scene dependence information, and constructs a high-order feature representation with logical coherence; in order to ensure the safety of the system, the anti-robustness enhancement module enhances the anti-interference ability of the features through the strategies of adversarial training and noise injection; finally, the feature storage and updating system realizes the incremental accumulation and version management of feature knowledge, forming a continuously evolving multi-modal feature knowledge base. Through the closed loop setting of modal fusion, dynamic optimization and knowledge precipitation, the system significantly improves the feature expression quality and decision reliability of multi-modal data in complex scenarios.

[0178] Multi-modal input preprocessing module:

[0179] Technical problem: Network behavior data has the characteristics of high noise and multi-format (PCAP flow packet, JSON log, system call sequence, etc.), and needs to be standardized.

[0180] The specific implementation content includes:

[0181] Network traffic imaging: slice the original traffic data (such as PCAP) according to the time window (default 1 second) and convert it into a gray spectrum image.

[0182] Log text structuring: regular expressions are used to extract key fields.

[0183] Constructing protocol-behavior keyword dictionary (including key operation labels of 14 types of protocols such as HTTP / HTTPS / DNS).

[0184] System call sequence encoding: map system call ID to 32-dimensional word vector (pre-trained on Linux kernel call dataset).

[0185] Generate timing feature blocks through sliding window (window size 256, step 64).

[0186] Cross-modal unified encoder: solve the semantic gap problem of visual, text and timing modalities, and map visual, text and timing data to a unified feature space.

[0187] The specific implementation content includes: visual encoding branch, text encoding branch, contrastive learning alignment; adopt a dual-flow contrastive learning framework to map multi-modal data to a unified feature space.

[0188] Based on improved ViT (visual encoder) and BERT (text encoder), combined with local sensitive hashing (LSH) and dynamic position encoding (DPE), semantic alignment is realized.

[0189] Dynamic feature optimizer:

[0190] Technical problem: traditional static feature fusion cannot adapt to the dynamic changes of network behavior.

[0191] The specific implementation content includes: introducing differentiable gating mechanism, dynamically adjusting the fusion weight of each modal feature; based on MLP network, the importance weight of modal is calculated to realize adaptive feature fusion; resource perception scheduling: according to the real-time state of device memory, dynamically adjust the feature slice size.

[0192] Dynamically adjust the importance of modal to improve the efficiency of fusion.

[0193] Context-aware modeling module:

[0194] Technical problem: traditional methods are difficult to capture long-term dependencies across sessions.

[0195] Implementation scheme: adopt causal Transformer combined with local attention window to capture long-range time series dependence. Causal mask and session-level state tracking (LSTM) optimize the time series logic and computational efficiency.

[0196] Causal Transformer architecture:

[0197] Set multi-head attention with causal mask:

[0198]

[0199] where, is a lower triangular mask matrix, which ensures the time series dependence. Q represents the query (Query) matrix, which is used to represent the features of the current element or position that need to be focused on. K represents the key (Key) matrix, which is used to represent the features of all elements or positions in the sequence. By calculating the similarity with Q, the attention weight is generated. V represents the value (Value) matrix, which contains the actual information of all elements or positions in the sequence. According to the attention weight, the weighted sum is generated to generate the final output. T represents the transpose operation; d k represents the dimension of the key (Key dimension), and the dot product result will be scaled to prevent the softmax function from saturating when calculating the attention weight. M causal represents the causal mask (Causal mask) matrix, which ensures that only the current position and previous positions are considered when calculating attention, preventing the model from seeing future information. Element-wise multiplication, also known as Hadamard product, which multiplies the elements at corresponding positions in two matrices to get a new matrix.

[0200] Introducing a local attention window (window size 256) reduces the computational complexity from to .

[0201] Session-level state tracking: use LSTM to maintain the session state vector, the specific formula is:

[0202]

[0203] Where f t is the current time feature, h t is the hidden state; h t-1 represents the hidden state at the last time. In the LSTM network, the hidden state is used to transmit long-term dependencies in the sequence, which contains all the information up to the last time. LSTM network controls the flow and memory of information through mechanisms such as update gate, forget gate and output gate, so as to capture long-term dependencies in the sequence.

[0204] Adversarial robustness enhancement module: embed gradient reversal layer (GRL) and build adversarial distillation training framework.

[0205] The present application improves the defense ability of the model through the adversarial training target function.

[0206] Implementation scheme: define a neural network module named DynamicFusion, which inherits from nn.Module, aiming to realize the dynamic fusion of visual, text and sequence three modal features. The gradient reversal coefficient λ = 0.1. Specifically, it includes function input, return and specific judgment process:

[0207] 1. Class definition and initialization:

[0208] Use class DynamicFusion(nn.Module) to define a class named DynamicFusion, which inherits from PyTorch's nn.Module class, indicating that it is a neural network module.

[0209] Use def __init__(self, dim), the initialization function of the class, which receives an argument dim, representing the dimension of the input feature.

[0210] Use super().__init__(), call the initialization function of the parent class.

[0211] A multi-layer perceptron (MLP) is defined using self.mlp = nn.Sequential(...), which includes two fully connected layers and a ReLU activation function. The first fully connected layer maps the input dimension from dim to dim, and the second fully connected layer maps the dimension from dim to 3, outputting three weight values corresponding to the fusion weights of the visual, text, and sequence modalities.

[0212] 2. Forward propagation function:

[0213] The forward propagation function is defined using def forward(self, visual, text, seq), which receives three arguments: visual, text, and seq, representing the features of the visual, text, and sequence modalities, respectively.

[0214] The input features of the three modalities are concatenated along the last dimension using context = torch.cat([visual, text, seq], dim=-1) to form a joint feature representation context.

[0215] The concatenated feature context is input into the MLP using weights = F.softmax(self.mlp(context), dim=-1) to obtain the fusion weights of the three modalities, which are then normalized by the softmax function to ensure that the sum of the three weights is 1.

[0216] The fused feature is obtained by weighting and summing the features of the three modalities using the normalized weights: fused = weights[:,0].unsqueeze(1) visual + weights[:,1].unsqueeze(1) text + weights[:,2].unsqueeze(1) seq. Here, weights[:,0], weights[:,1], and weights[:,2] represent the fusion weights of the visual, text, and sequence modalities, respectively, and unsqueeze(1) is used to adjust the dimensions of the weights for broadcasting multiplication with the features.

[0217] Finally, the fused feature is returned.

[0218] 3. Specific judgment process:

[0219] During the forward propagation, the features of the three modalities are first concatenated to form a more comprehensive contextual representation.

[0220] Then, an MLP is learned to map this contextual representation to the fusion weights for the three modalities. The structure of the MLP is set so that it can capture the complex interaction between different modalities and output the fusion weights suitable for the current input.

[0221] Finally, the features of the three modalities are fused into a unified feature representation through weighted summation. This fusion method not only retains the unique information of each modality, but also realizes the interaction and complementarity of information between modalities through weight adjustment.

[0222] Implementation: Define a function named adversarial_loss, which aims to calculate the prediction difference between the model for the input data and its adversarial samples, i.e. adversarial loss. Specifically, it includes function input, return and specific judgment process:

[0223] 1. Function definition and input parameters:

[0224] The function is defined as def adversarial_loss(model, x, y, epsilon=0.01), which receives four parameters: model: the neural network model to be evaluated. x: input data, usually the original input of the model. y: true label, usually used to calculate standard loss (such as cross-entropy loss). epsilon: the amplitude of adversarial perturbation, controls the difference between the adversarial sample and the original input, default is 0.01.

[0225] 2. Generate adversarial samples (problem exists):

[0226] Through x_adv = x + epsilon torch.sign(x.grad), try to generate adversarial samples x_adv. However, there is a key problem here: x.grad is undefined before forward propagation and backpropagation. The correct adversarial sample generation process should include:

[0227] a. Forward propagation on input x to get model prediction.

[0228] b. Calculate the loss function based on the prediction and the true label (or target label).

[0229] c. Backpropagation on the loss function to calculate the gradient of input x.

[0230] d. Generate adversarial samples x_adv according to the gradient information.

[0231] 3. Model prediction (disable gradient calculation):

[0232] with torch.no_grad(), a context manager is used to disable gradient calculation, to reduce memory consumption and speed up computation, since only forward propagation is needed at this stage, without backward propagation.

[0233] f_x = model(x), forward propagation is performed on the original input x to obtain the prediction result f_x of the model.

[0234] f_adv = model(x_adv), forward propagation is performed on the generated adversarial sample x_adv to obtain the prediction result f_adv of the adversarial sample.

[0235] 4. Calculate adversarial loss:

[0236] loss = F.mse_loss(f_x, f_adv), the mean square error loss (MSE Loss) between the prediction result f_x of the original input and the prediction result f_adv of the adversarial sample is calculated. This loss function measures the difference between the model's predictions for the original input and the adversarial sample.

[0237] 5. Return the loss value:

[0238] The function returns the calculated adversarial loss value loss, which can be used to evaluate the model's robustness to adversarial perturbations, or as part of the training process to optimize the model to improve its adversarial robustness.

[0239] Feature space active defense, robustness improved to 89.5%.

[0240] The adversarial training objective function is:

[0241]

[0242] The first term constrains the feature smoothness, the second term enforces the perturbation consistency, and λ represents a hyperparameter that balances the weight of the gradient norm term and the KL divergence term in the loss function. By adjusting the value of λ, the application can control the trade-off between complexity and robustness of the model. λ=0.5; E(x,y) represents the expected value, which is the average of all possible input-label pairs (x,y). In adversarial training, the application usually considers all possible input and label combinations to evaluate the performance of the model on the entire data distribution. F represents the Frobenius norm. Represents the adversarial perturbation, the tiny input noise generated by FGSM, simulates DNS tunnel obfuscation / log changes, etc. f(x) represents the feature extraction function, the CVOCA core model, which maps multi-modal input (traffic / log / call sequence) into a unified feature vector. x represents the original input, the pre-processed normalized data (such as 4096x4096 traffic grayscale image, structured log text). K represents the feature dimension, the dimension of the feature vector f(x) (such as visual 1024 dimensions, text 768 dimensions), which determines the calculation space of KL divergence. L represents the adversarial loss, the objective function that quantifies the robustness of the model, the smaller the value, the stronger the adversarial ability. Represents the gradient operator, which calculates the partial derivative (Jacobian matrix) of the feature f(x) with respect to the input x, measuring the feature sensitivity.

[0243] Feature storage and update system: Hybrid Expert System (MoE) is used to support online learning of feature extraction module.

[0244] Technical key: Expert network and gating mechanism realize dynamic update of feature library.

[0245] Specific content includes: MoE incremental learning, feature version management.

[0246] Feature version management: Store feature update records based on blockchain to ensure traceability. By defining a class named FeatureVersionControl, the core idea is to store feature update records based on blockchain to ensure traceability. Including the specific function input, return and specific judgment process:

[0247] 1. Class definition and initialization:

[0248] Define a class named FeatureVersionControl, that is, class FeatureVersionControl, which encapsulates the feature version control function.

[0249] Define the initialization function (constructor) of the class, that is, def __init__(self, blockchain), which receives a parameter blockchain, indicating a pre-created blockchain object.

[0250] In the initialization process, the incoming blockchain object is assigned to the instance variable self.chain. This allows the instance of the FeatureVersionControl class to access and manipulate this blockchain for storing feature update records.

[0251] 2. Commit feature hash value to blockchain (commit function):

[0252] Use def commit(self, feature_hash), which defines a method named commit that receives a parameter feature_hash, which represents the hash value of the feature that needs to be recorded. Hash values are usually used to uniquely identify the content of the feature, so even if the feature content changes slightly, its hash value will be significantly different.

[0253] Use block = create_block(feature_hash, self.chain[-1].hash), which is the core operation of the commit method. It calls a hypothetical function create_block. The function's role is to create a new blockchain block. Use feature_hash as the data content of the new block, which records the current feature that needs version control. Use self.chain[-1].hash to represent the hash value of the last block (i.e., the latest block) in the blockchain. Including the hash value of the previous block in the new block is a key mechanism in blockchain technology that ensures data cannot be changed and the chain structure. If someone tries to change the historical data in the blockchain, the hash values of all subsequent blocks will change, making it easy to detect.

[0254] Use self.chain.append(block) to add the newly created block to the end of the self.chain blockchain. This step completes the process of permanently writing feature updates to the blockchain. Due to the immutable nature of the blockchain, once the feature hash is written to the blockchain, it is difficult to modify or delete, thereby ensuring the traceability and credibility of the feature version history.

[0255] Hierarchical Feature Injection and Resource Optimization: Set up hierarchical slice injection technology to dynamically allocate high-resolution data to different levels of the model.

[0256] Key technology: adaptive slicing and local block injection, optimize edge device deployment efficiency.

[0257] Hierarchical Feature Injection (Edge Device): Define a function named hierarchical_injection, which aims to implement hierarchical feature injection technology to optimize the deployment efficiency of the model on edge devices. Including function input, return, and specific judgment process:

[0258] 1. Function definition and input parameters:

[0259] The function is defined as def hierarchical_injection(x, model, mem_status), which receives three parameters: x: input data, usually an image or feature map, as the original data that needs to be processed. model: neural network model, the function will perform feature injection at different levels of this model. mem_status: memory status indication, used to determine the size of the extracted patches to adapt to different resource constraints.

[0260] Patch extraction: dynamically determine the patch size based on the memory status. If the memory status is 'low' (indicating that memory resources are tight), use a smaller patch size (32x32) to reduce memory occupancy; otherwise, use a larger patch size (64x64) to capture more rich feature information. Then extract patches of the specified size from the input data x. These patches will be injected as additional feature information into different levels of the model. The extract_patches function is responsible for implementing the specific logic of patch extraction.

[0261] Hierarchical feature injection: iterate through each layer of the model, preparing for feature injection; through if i % 2 == 0, implement the key logic of hierarchical injection, that is, inject features every other layer. This setting aims to balance the frequency of feature injection and the processing efficiency of the model, avoiding excessive feature injection leading to model performance degradation.

[0262] Use inject_patch = patches[i / / 2] to implement the selection of patches to be injected according to the current level. Since injection is performed every other layer, you can index the patch array patches by i / / 2 to ensure that each injection layer gets a unique patch.

[0263] Use layer_input = torch.cat([layer_input, inject_patch], dim=1) to inject the patch into the input of the current layer. Here it is assumed that layer_input has been defined outside the loop and is gradually built in each loop, representing the input data before the current layer is processed. By using the torch.cat function, the patch inject_patch is concatenated with the original input layer_input in the specified dimension (dim=1), realizing the fusion of features.

[0264] Use layer_output = layer(layer_input) to pass the injected input to the current layer for processing to obtain the output layer_output of the current layer.

[0265] Return output: The function returns the output of the last layer, layer_output, which is the final output of the model after the hierarchical feature injection process.

[0266] Through the above hardware and software configuration, scheme logic setting and data flow optimization, the present application is significantly superior to the prior art in terms of cross-modal feature extraction efficiency, adversarial robustness, edge computing adaptability, etc., providing efficient and reliable technical support for network behavior analysis. At the same time, the memory occupation is reduced by 62%, and the adaptive resource limited scene is adapted.

[0267] The system in this embodiment can be applied in the scenarios of financial transactions and industrial Internet of Things intrusion detection.

[0268] Financial transactions:

[0269] Input: visual modality: transaction interface screenshot; text modality: operation log (>1000 per second); time series modality: API call sequence (length >10,000).

[0270] Processing flow:

[0271] Divide the API sequence into windows (window size 256) and slice them into grayscale images.

[0272] Extract visual features through the ViT-LSH encoder.

[0273] Dynamically assign weights (visual 0.6, text 0.3, time series 0.1).

[0274] Detect the spatial-temporal offset between the interface screenshot and the log operation, and trigger an alarm.

[0275] Effect: False positive rate reduced to 1.2%, detection delay <50ms.

[0276] Industrial Internet of Things intrusion detection:

[0277] Layered slicing of Modbus TCP protocol traffic (64x64 per layer).

[0278] Dynamic Prompt generation: "Modbus [function code exception] request".

[0279] Efficiency: Feature extraction speed 153 frames / second (traditional method 52 frames / second).

[0280] Memory usage: 4.3GB (traditional solution 11.2GB).

[0281] The application solves the core problems of multi-modal fragmentation, low computational efficiency, long-range dependence loss and vulnerability to attacks in network behavior analysis through four technologies of cross-modal dynamic fusion architecture, hierarchical computation optimization, causal context modeling and active defense, and provides a reliable feature extraction basis for building the next generation of intelligent security systems.

[0282] The experimental index results of the traditional method and the scheme of the application are compared, as shown in Table 1.

[0283] Table 1 Comparison of results

[0284]

[0285] The technical scheme of the application can be integrated into three types of products: the next generation intelligent security gateway (NISG), the cloud security monitoring platform (CSMP) and the industrial edge security terminal (IEST). The following explains from three dimensions of application scenarios, functional characteristics and interactive design:

[0286] I. Application scenarios and product positioning

[0287] Table 2 Application scenarios and product positioning

[0288]

[0289] II. Functional characteristics and interactive design

[0290] 1. Intelligent security gateway (NISG)

[0291] The NISG-Dashboard visualization platform matched with the application builds a multi-modal network security decision center. Its interactive interface adopts a three-column layout. After user login, the user can perform the closed-loop path execution of monitoring network segment filtering, weight dynamic optimization, real-time graph analysis, report export and strategy deployment. The feature extraction rules and security policies are linked and configured to ensure the whole-process visualization control from multi-modal traffic profiling to abnormal event handling, significantly improving the traceability and response efficiency of complex network security events.

[0292] 2. Security monitoring platform (CSMP)

[0293] The CSMP-Console multi-tenant security management platform matched with the application builds an intelligent security operation center. Its interactive interface adopts a hierarchical layout and provides one-key adoption or manual correction dual mode. In typical operation and maintenance scenarios, the administrator can quickly locate the abnormal tenant through the score sudden drop warning (such as 45 threshold triggering), and after confirming the high-risk operation through call chain backtracking, the system automatically recommends a fine-grained control strategy containing time, user and action elements, realizes the whole-process closed-loop control from behavior profiling to strategy deployment, and significantly improves the response accuracy and handling efficiency in a multi-tenant environment.

[0294] 3. Industry Edge Terminal (IEST);

[0295] The IEST-Device industrial security terminal provided by the application is an intelligent protection device specially set for an industrial control system, the hardware interaction and function modules of which are deeply integrated with the requirements of an industrial scene. The operation process of the device is highly automated: after power-on, the latest feature library synchronized from the cloud is automatically loaded, the protocol deep detection is started when accessing the industrial network, when the abnormal operation triggers an alarm, the management personnel can quickly locate the specific details (such as illegal function code combination) through the touch screen, and use the USB interface to export the complete event report containing the timestamp and protocol field, forming a complete closed-loop handling capability from protocol analysis, abnormal early warning to evidence solidification, effectively improving the real-time defense level of the industrial control system against unknown disturbances.

[0296] III. Mapping of core technology on product side

[0297] Table 3. Mapping of core technology on product side

[0298]

[0299] The technical solution of the application realizes the explicit transformation of technical capability to user experience, converts the complex multi-modal feature extraction process into an operable and understandable interactive interface, and significantly reduces the technical threshold of safety operation personnel.

[0300] As shown in Table 4 and Table 5, the hardware and software environment of the application is:

[0301] Table 4. Hardware environment

[0302]

[0303] Table 5. Software environment

[0304]

[0305] The front-end interaction logic of the system of the application is as shown in Table 6. Figure 2 The front end is responsible for user interaction and visual display.

[0306] RESTful API (Representation State Transfer Application Programming Interface) call path: / api / v1 / config / feature_weight.

[0307] In the request body example, "visual_weight (visual weight)": 0.6, "text_weight (text weight)": 0.3, "sequence_weight (time sequence weight)": 0.1.

[0308] The system background processing logic: the microservice architecture adopts modular design, four core services realize high-performance RPC (Remote Procedure Call) communication through the gRPC framework, specifically:

[0309] 1) Feature Extractor is responsible for multi-modal data preprocessing, parallel processing of visual, text and time series data through thread pool technology, and completes data encoding and format unification.

[0310] 2) Inference Engine carries CVOCA core model, supports batch_size=32 batch inference ability, realizes end-to-end processing of dynamic optimization and context modeling.

[0311] 3) Adversarial Trainer generates adversarial samples based on FGSM algorithm every day, continuously enhances the robustness of the system through model fine-tuning mechanism.

[0312] 4) Feature Store builds a dual-mode knowledge base, maintains a large feature vector library, and deposits disturbance mode templates, integrates Faiss vector retrieval engine to realize millisecond-level nearest neighbor search. Each service is designed to be decoupled to ensure independent scalability, and the inference service supports horizontal expansion to handle sudden traffic, and the storage service uses Faiss quantization compression and index optimization technology to efficiently manage hundreds of millions of feature data, forming a closed-loop processing link of data preprocessing-model inference-security enhancement-knowledge sedimentation.

[0313] As shown in Figure 5 , the system adopts a three-layer distributed architecture to realize full-scene safety monitoring: in the data center layer, the server node realizes full-flow mirror collection by accessing the SPAN port of the core switch, and is deeply integrated with the SIEM security information management system to complete flow metadata extraction, abnormal alarm aggregation and compliance audit functions; The lightweight edge device is deployed in the industrial field layer, directly connected to PLC / DCS and other industrial controllers, analyzes industrial instructions through Modbus / OPC UA and other protocols, and uses 4G / 5G network to return the preprocessed behavior features to the cloud in real time, effectively solving the problem of wired deployment in industrial environments.

[0314] As shown in Figure 6As shown, after the user configures the feature weight through the front-end interface, the front-end sends the configuration information to the / api / config interface of the back-end through a POST request. After receiving the request, the back-end starts the Feature Extractor to collect data. After the Feature Extractor completes data collection, it requests batch inference from the Inference Engine. The Inference Engine then obtains the adversarial model from the Adversarial Trainer and stores the feature vectors obtained through inference into the Feature Store. The Feature Store pushes the real-time feature stream to the front-end, and the front-end updates the abnormal event graph and displays it to the user after receiving the feature stream.

[0315] The present application solves the core pain points of existing network behavior analysis technology in multi-modal data fusion, computational efficiency, adversarial robustness and dynamic adaptability through the innovative CVOCA multi-modal feature extraction architecture, significantly improving the technical performance and application value.

[0316] Embodiment 2

[0317] The core goal of the present application is to solve the problems of multi-modal data fusion, computational efficiency, adversarial robustness and dynamic adaptability in network behavior analysis through cross-modal unified encoding, dynamic feature optimization, context-aware modeling and adversarial robustness enhancement.

[0318] The difference between this embodiment and embodiment 1 is that this embodiment can replace the key modules in embodiment 1 with multiple implementation methods and scalable application scenarios. These replacement schemes can achieve the purpose of the present application, while providing differentiated choices in technical paths or application fields. Specifically:

[0319] I. Replacement implementation of cross-modal unified encoding

[0320] Main scheme: use ViT (visual encoder) combined with BERT (text encoder) to realize multi-modal data alignment through contrastive learning.

[0321] Replacement scheme:

[0322] 1. CLIP model variant;

[0323] Implementation: directly use the CLIP (Contrastive Language-Image Pretraining) model to process multi-modal data using the pre-trained image-text alignment capability.

[0324] Features: CLIP has been pre-trained on large-scale data, and transfer learning can quickly adapt to the network behavior analysis scenario, simplifying the training process.

[0325] 2. ALIGN architecture;

[0326] Implementation: Use ALIGN model, combined with EfficientNet (visual) and BERT (text) encoders, align modal features through large-scale noisy data.

[0327] Features: More robust to unstructured network logs and other noisy data, suitable for complex input scenarios.

[0328] 3. ResNet + RoBERTa combination;

[0329] Implementation: Use ResNet-50 to encode visual data and RoBERTa (Robustly Optimized BERT Pretraining Approach) to encode text data, and realize feature alignment through shared projection layers.

[0330] Features: ResNet has high computational efficiency, and RoBERTa is good at long text processing, suitable for resource-constrained environments.

[0331] 4. Multi-modal graph neural network (GNN);

[0332] Implementation: Model different modal data as graph nodes and inter-modal relationships as edges, and use GNN to extract cross-modal features.

[0333] Features: Explicitly model inter-modal relationships to enhance semantic correlation extraction capabilities.

[0334] II. Replacement implementation of dynamic feature optimization

[0335] Main scheme: Use differentiable gating mechanism to dynamically adjust multi-modal feature weights.

[0336] Alternative scheme:

[0337] 1. Attention mechanism fusion;

[0338] Implementation: Use multi-head attention mechanism to calculate the correlation between modal features and automatically generate fusion weights.

[0339] Features: Capture fine-grained interaction relationships to improve feature fusion accuracy.

[0340] 2. Adaptive weighting strategy;

[0341] Implementation: Calculate uncertainty based on modal data entropy or variance, and dynamically allocate weights (reduce the weight of modal with high uncertainty).

[0342] Features: No additional network layers, more efficient computation.

[0343] 3. Reinforcement Learning Scheduling;

[0344] Implementation: Train reinforcement learning agents to dynamically choose modalities based on device load or data distribution.

[0345] Feature: Adaptive optimization of long-term performance, suitable for dynamic environments.

[0346] 4. Bayesian Feature Fusion;

[0347] Implementation: Use Bayesian networks to model inter-modal conditional dependencies and perform probabilistic weighted fusion.

[0348] Feature: Provides uncertainty estimates, enhancing decision robustness.

[0349] Three, alternative implementation of context-aware modeling

[0350] Main scheme: Use Transformer architecture to capture long-range temporal dependencies.

[0351] Alternative:

[0352] 1. RNN / LSTM network;

[0353] Implementation: Use bidirectional LSTM to process temporal features and capture context dependencies.

[0354] Feature: High computational efficiency, low memory usage, suitable for short temporal data.

[0355] 2. Temporal Convolutional Network (TCN);

[0356] Implementation: Use dilated convolution to model long-term dependencies and reduce parameter quantity.

[0357] Feature: Fast inference speed, suitable for edge device deployment.

[0358] 3. Graph Convolutional Network (GCN);

[0359] Implementation: Model network sessions as graph structures, and GCN extracts temporal associations between nodes.

[0360] Feature: Explicitly model session jump relationships, improve path recognition ability.

[0361] 4. Causal Convolutional Network;

[0362] Implementation: Use causal convolution to ensure that feature extraction conforms to temporal logic.

[0363] Feature: Low latency, suitable for real-time detection scenarios.

[0364] Four, alternative implementation of adversarial robustness enhancement

[0365] Main solution: Use gradient reversal layer (GRL) and adversarial distillation training to enhance robustness.

[0366] Alternative solution:

[0367] 1. Virtual adversarial training (VAT);

[0368] Implementation: Generate virtual perturbations in feature space to optimize model smoothness.

[0369] Features: No label required, high training flexibility.

[0370] 2. Generative adversarial network (GAN);

[0371] Implementation: Train generator to produce adversarial samples, and discriminator to optimize feature robustness.

[0372] Features: Generate diverse adversarial samples to improve defense breadth.

[0373] 3. Feature space regularization;

[0374] Implementation: Introduce Lipschitz constraint to limit feature gradient norm.

[0375] Features: Theoretically guarantees feature smoothness, stable defense effect.

[0376] 4. Self-supervised adversarial learning;

[0377] Implementation: Enhance model robustness to local perturbations through mask reconstruction task.

[0378] Features: No additional adversarial samples required, low training cost.

[0379] Five, other application scenarios

[0380] Main scenario: Network behavior analysis (e.g. security gateway, cloud monitoring, industrial edge).

[0381] Extended scenario:

[0382] 1. Medical image analysis;

[0383] Application method: Fusion of CT images (vision), medical records (text), and physiological signals (time series).

[0384] Effect: Improve multi-modal diagnosis accuracy and reduce misdiagnosis.

[0385] 2. Financial risk control;

[0386] Application method: Fusion of transaction data (time series), user behavior logs (text), and market news (text).

[0387] Effect: Real-time identification of abnormal transactions, reduce financial risk.

[0388] 3. Autonomous driving;

[0389] Application mode: Fusion of vehicle-mounted camera (vision), laser radar (point cloud), navigation instruction (text), realize environment perception.

[0390] Effect: Improve multi-source data fusion accuracy, enhance driving safety.

[0391] 4. Intelligent customer service;

[0392] Application mode: Fusion of user voice (audio), dialogue text (text), emotional image (vision), realize intent recognition.

[0393] Effect: Improve response accuracy, optimize user experience.

[0394] The above is only the preferred embodiment of the present application, it should be pointed out that, for those skilled in the art, without departing from the principles of the present application, can make a number of improvements and refinements, these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A CVOCA feature extraction method based on a multi-modal large model, characterized in that: Comprising the following steps: S1: data input, including network traffic data, log text data, system call sequence; S2: data preprocessing, processing network traffic data, log text data, system call sequence, generating standardized input, converting original traffic data into grayscale spectrum graph; S3: unified representation of features, mapping visual, text, and time series data to a unified feature space using a cross-modal unified encoder; S3 includes the following sub-steps: S31: visual encoding, using an improved ViT-B / 32 model, adding a local sensitive hash module to the embedding layer for local feature enhancement of the grayscale spectrum graph; S32: text encoding, based on BERT architecture improvement, introducing dynamic position encoding; S33: contrastive learning alignment, setting a double-flow contrastive loss function to align the visual and text feature spaces; A dual-stream contrastive loss function L is set align , which forces the alignment of visual and textual feature spaces, specifically: where S() is the cosine similarity, τ = 0.05, and N is the batch size; v i represents the visual feature vector of the i-th sample; v j represents the visual feature vector of the j-th sample; t i represents the text feature vector of the i-th sample; t j represents the text feature vector of the j-th sample; S34: time series encoding, using a Transformer encoder to process each system call word vector in the sequence into a context-aware time series feature vector; S4: feature optimization, based on a dynamic feature optimizer, dynamically adjusting modal weights and achieving efficient feature fusion; S4 includes the following sub-steps: S41: introducing a differentiable gating mechanism, calculating modal importance weights based on an MLP network; Specifically: where W m represents the attention weight of the mth element; where element is a feature vector corresponding to a word or a feature vector corresponding to an image region or a feature vector corresponding to a sequence position; f m is the modal feature, and C is the global context vector; is a concatenation operation, and MLP represents a multi-layer perceptron; Softmax represents a softmax function, which is used to convert the output of the multi-layer perceptron into a probability distribution, ensuring that the sum of the attention weights of all elements is 1; S42: dynamic feature fusion; Specifically: Wherein, Proj m is a modal projection layer, f fusion represents the fused feature vector; m represents the index of the element; W m represents the attention weight of the mth element; f m is a modal feature; S43: resource-aware scheduling: dynamically adjusting feature slice size according to real-time device memory state; S5: context-aware modeling, using causal Transformer combined with local attention window to capture long-range temporal dependencies; S6: adversarial optimization, improving model robustness through adversarial training; S6 includes the following sub-steps: S61: generating adversarial samples; Specifically: wherein, δ represents an input adversarial perturbation amount; ∈ represents a perturbation amplitude; sign represents a sign function; denotes a gradient operator; L denotes a loss function; f(x) denotes a prediction result of a model on an input x; y denotes a true label or an expected output of the input x; S62: based on FGSM algorithm, calculating adversarial loss; Specifically: Where δ represents the input adversarial perturbation; f(x) represents the output value after processing or mapping the input x; f(x+δ) represents the new output obtained after adding perturbation δ to the input; S7: feature storage and update system, using an expert network and a gating mechanism to achieve dynamic updating of the feature library.

2. The multi-modal large model-based CVOCA feature extraction method according to claim 1, characterized in that: In S2, data preprocessing includes: S21: network traffic imaging: slice the original traffic data by time window and convert it into a grayscale spectrum graph; S22: log text structuring: using regular expressions to extract key fields; S23: building a protocol-behavior keyword dictionary: annotating protocol keywords for regular matching extracted fields; S24: system call sequence encoding: mapping system call IDs to 32-dimensional word vectors and generating time series feature blocks through sliding windows. 3.The multi-modal large model based CVOCA feature extraction method according to claim 1, characterized in that: The specific process of S5 is: S51: causal Transformer architecture: setting multi-head attention with causal mask; S52: session-level state tracking: using LSTM to maintain session state vectors.

4. The multi-modal large model-based CVOCA feature extraction method according to claim 1, characterized in that: The specific process of S7 is: MoE incremental learning: setting an expert mixing system to calculate each expert's focus on specific disturbance types; Feature version management: based on blockchain to store feature update records to ensure traceability.

5. A multi-modal large model based CVOCA feature extraction system, realizing the multi-modal large model based CVOCA feature extraction method according to any one of claims 1 to 4, characterized in that, It includes: Multimodal input preprocessing module: for standardizing the input heterogeneous data; Cross-modal unified encoder: used to map multi-source information to a shared semantic space, completing the initial modeling of semantic correlation between modalities; Dynamic feature optimizer: based on real-time feedback mechanism, it adjusts the weights of preliminary features and suppresses redundancy adaptively; Context-aware modeling module: used to integrate timing correlation and scene-dependent information, building high-order feature representation with logical coherence; Adversarial robustness enhancement module: through adversarial training and noise injection strategies, it improves the anti-interference ability of features; Feature storage and update system: used to realize incremental accumulation and version management of feature knowledge, forming a continuously evolving multi-modal feature knowledge base.

Images