Data security risk assessment method based on big data model

Abstract

The invention discloses a data security risk assessment method based on a big data model, and relates to the technical field of data security, and the method comprises the steps: collecting and preprocessing multi-source data, collecting security-related data from network equipment, a server and an application system, carrying out the preprocessing, carrying out the adaptive feature extraction, and carrying out the data security risk assessment. The feature importance is evaluated by calculating the mutual information amount of features and risk tags, a standardized feature vector set is constructed, multi-model collaborative analysis is performed, feature vectors are input into a cascade collaborative network composed of an anomaly detection model, a threat recognition model, a correlation analysis model and a prediction model, and a risk risk is obtained. Through cross-model feature transmission and a bidirectional information feedback mechanism, deep collaborative analysis and multi-model deep fusion decision making are carried out, a weight is calculated according to historical accuracy of each model, a comprehensive risk score is calculated by adopting dynamic gating deep fusion, and a dynamic threshold value is calculated based on a sliding time window. And the risk is divided into three levels of high risk, medium risk and low risk.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data security, in particular to a data security risk assessment method based on a big data model. BACKGROUND

[0002] Data security risk assessment is an important technical direction in the field of information security, and its core goal is to identify, analyze and quantify the security threats faced by data assets. Traditional risk assessment methods are usually based on the following technical routes: The first is a rule-based assessment method, which matches and detects data access behavior and system logs by predefining a set of security rules. This method relies on expert experience to establish a rule library, and triggers a risk alert when a data behavior matches a certain rule; The second is a statistical analysis method, which analyzes historical security event data using statistical methods to calculate the occurrence probability and impact of various risk factors, and establishes a risk assessment index system; The third is a machine learning method, which uses machine learning algorithms to train historical security data, establishes a classification model, and is used to identify abnormal behavior and assess risk levels; The closest prior art to the present application is a data security risk assessment system based on machine learning, which mainly includes the following technical processes: collecting security logs, access records and traffic data from network devices, servers and application systems; Extracting predefined security features from raw data, including access frequency, access time, data size, user behavior characteristics and other fixed-dimensional feature parameters to form a feature vector; Using a labeled historical data set to train a classification model, using a support vector machine or random forest machine learning algorithm to establish a risk identification model; Input the feature vector of the data to be evaluated into the trained model, and the model outputs the risk classification result and gives a risk score; Generate a risk report according to the evaluation results, and trigger an alarm mechanism when a high-risk event is detected; The above prior art has the following technical defects: Defect one: fixed feature extraction leads to insufficient adaptability, the existing technology uses a predefined fixed feature set for extraction, which has obvious limitations, because data security threats continue to evolve, and new attack methods emerge in an endless stream. The fixed feature set cannot capture new threat features, and when attackers use new attack patterns, the system will not be able to identify the threat because the features of the pattern are not included in the predefined feature set. This feature fixation results in poor generalization of the model to new threats; Defect two: model independent operation leads to information island, the existing technology even adopts multiple models, each model is independently operated, lacks information interaction and cooperation mechanism, cannot realize knowledge sharing and complementary enhancement among models, leads to that the overall system performance cannot surpass single optimal model, this mode is difficult to deal with complex and changeable security scene, because different types of security threats have different characteristic modes, single model is difficult to simultaneously optimize the identification effect of multiple threats; Defect three: lack of multi-dimensional correlation analysis capability, the existing technology mainly analyzes single data source independently, lacks depth correlation analysis of multi-source data, in actual scene, data security risk often shows as combination of multiple abnormal events, single point data analysis is difficult to find such associated threat; Defect four: static evaluation mechanism lacks dynamic prediction capability, the existing technology mainly carries out risk assessment of current state, lacks prediction of risk evolution trend, because the evaluation model can only judge the risk state at current time, cannot predict the risk situation that may appear in the future, this leads to that the system can only carry out post-detection but not pre-warning; Defect five: model updating mechanism is imperfect, the model training of the existing technology usually adopts offline batch mode, model parameters are fixed for a long time, because of lacking online learning and incremental updating mechanism, the model cannot absorb new threat intelligence and security event data in time, leading to that the model performance gradually declines; Defect six: multi-model fusion method is simple and rough, the existing multi-model scheme usually adopts simple weighted average or voting mechanism to carry out result fusion, this fusion mode ignores the internal correlation between model outputs and the particularity of current sample, cannot dynamically adjust the contribution degree of each model according to specific scene, and also cannot handle the output conflict among models. SUMMARY

[0003] The purpose of the present application is to provide a data security risk assessment method based on big data model.

[0004] To achieve the above purpose, the present application provides the following technical scheme: a data security risk assessment method based on big data model, comprising the following steps: S1, multi-source data acquisition and preprocessing: collecting security related data from network equipment, server, application system, carrying out integrity check, format standardization, abnormal value detection and data deduplication processing on the collected original data, forming a preprocessed data set; S2, adaptive feature extraction: extracting statistical features, behavior features and content features from the preprocessed data set, evaluating feature importance by calculating mutual information between features and risk labels, screening features with importance score higher than a preset threshold, and normalizing the screened features to construct a standardized feature vector set; S3, multi-model cascade collaborative analysis: input the feature vector set into the cascade collaborative network composed of anomaly detection model, threat identification model, correlation analysis model and prediction model, share intermediate representation between models through cross-model feature transmission layer, and realize real-time information interaction through bidirectional information feedback channel; The anomaly detection model adopts a density-based anomaly detection algorithm to identify abnormal data deviating from normal behavior patterns, outputs an anomaly score and an anomaly feature embedding vector, and the anomaly feature embedding vector is transmitted to the threat identification model and the correlation analysis model as auxiliary input; The threat identification model receives the original feature vector and the anomaly feature embedding vector, identifies known threat types through feature matching method, outputs threat score and threat type coding vector, and the threat type coding vector is fed back to the anomaly detection model for adjusting the anomaly judgment boundary; The correlation analysis model fuses the anomaly feature embedding vector to construct an enhanced security event graph, uses graph neural network technology to discover deep correlation between multiple security events, and outputs correlation score and event correlation topological feature, which is transmitted to the prediction model to enhance the time series modeling capability; The prediction model comprehensively receives original time series features, event correlation topological features and intermediate state vectors of each model, uses an enhanced long short-term memory network to analyze risk evolution trend, outputs prediction score and trend gradient vector, and the trend gradient vector is back propagated to other three models to dynamically adjust the sensitivity parameters of each model; S4, multi-model deep fusion decision: joint coding of the output results of the four models and their intermediate representations, using a dynamic gated fusion network based on multi-head attention mechanism to calculate the adaptive weight of each model on the current sample, evaluating the confidence of each model output through uncertainty quantification, introducing conflict detection and coordination mechanism to handle the scoring differences between models, and finally generating comprehensive risk score and decision confidence, wherein the weight of the anomaly detection model is , the weight of the threat identification model is , the weight of the correlation analysis model is , and the weight of the prediction model is , each weight satisfies + + + =1; S5, dynamic threshold comparison and risk level determination: based on the historical risk score in the sliding time window, calculate the dynamic threshold, compare the comprehensive risk score with the dynamic threshold, and divide the risk into three levels of high risk, medium risk and low risk according to the comparison result; S6, response decision and model optimization: generate corresponding alarm information and response suggestions according to the risk level, collect feedback data of risk assessment and response treatment, and update model parameters by using incremental learning method when the accumulated feedback data reaches the preset threshold.

[0005] As a further scheme of the present application: the adaptive feature extraction in step S2 specifically comprises: S2.1, constructing an initial feature pool: extracting mean, variance, maximum value of data field, minimum value of data field, median feature for numerical data field, extracting value distribution and entropy value feature for categorical data field, and extracting length and element frequency feature for sequence data; S2.2, calculating feature importance score: calculating mutual information between each feature and risk label by using mutual information method, evaluating the contribution of feature to sample classification purity by using Gini coefficient method, and calculating the correlation between feature and risk label by using correlation coefficient method, and giving comprehensive importance score to each feature by integrating the scores of the three methods; S2.3, feature screening: deleting features with comprehensive importance score lower than the preset threshold, and retaining the feature with higher importance score for two features with correlation coefficient greater than 0.9; S2.4, feature normalization: mapping feature values to the interval of 0 to 1 by using minimum-maximum normalization method for numerical features, and converting category features into numerical representation by using one-hot encoding method.

[0006] As a further scheme of the present application: the analysis process of the anomaly detection model in step S3 specifically comprises: S3.1.1, establishing a normal behavior baseline model: defining the normal behavior range by learning the distribution characteristics of historical normal data; S3.1.2, calculating local density: for each data point, searching its neighbor data points, calculating the average distance to neighbor data points, wherein the value range of is 5 to 20, and the reciprocal of the average distance is taken as the local density value of the data point; S3.1.3, anomaly determination: when the local density value of a data point is lower than the average density value of data points in its neighborhood by a preset proportion, the data point is determined to be abnormal, and the value range of the preset proportion is 0.5 to 0.8; S3.1.4, using isolation forest algorithm as auxiliary detection, isolating data points by randomly building decision trees, recording the number of partitions required to isolate each data point, and determining the data point to be abnormal when the number of partitions is less than the preset number; S3.1.5, outputting anomaly score by integrating the results of the two algorithms.

[0007] As a further scheme of the present application: the synergistic interaction mechanism between the models in step S3 specifically includes: S3.2.1, constructing a shared feature space: projecting the intermediate layer feature representations of the four models into a unified shared feature space, The dimension of the shared feature space is the larger value among the intermediate feature dimensions of the models, and the projection process adopts a learnable linear transformation matrix; S3.2.2, cross-attention calculation: for any two models And model , calculate the attention weight of model to model , The attention weight represents the influence degree of the features of model on the decision of model , and the attention calculation adopts a scaled dot-product attention mechanism; S3.2.3, feature enhancement transmission: each model weights and aggregates the shared features of other models according to the attention weight, The aggregated cross-model features are spliced with their own features to form enhanced feature representations for subsequent calculations; S3.2.4, gradient synergistic optimization: in the model training stage, the loss functions of the four models are associated through the shared feature space, The parameter update of a single model is constrained by both its own loss and the cross-model consistency loss, The cross-model consistency loss is defined as the weighted sum of the risk judgment differences of each model for the same input sample; S3.2.5, dynamic gating mechanism: dynamically calculate the gating coefficients of information transmission between models according to the feature distribution of the current input data, When it is detected that the output confidence of a certain model is lower than a preset threshold, increase the gating coefficient of the model receiving information from other models, When it is detected that the difference between the output of a certain model and other models exceeds a preset divergence threshold, start the divergence mediation mechanism for joint reasoning.

[0008] As a further scheme of the present application: the analysis process of the correlation analysis model in step S3 specifically includes: S3.3.1, constructing a security event graph: abstracting various security events as nodes in the graph, and representing the association relationship between events as edges, the association relationship includes temporal association, causal association, subject association and object association; S3.3.2, attack chain path search: search suspicious association patterns through graph traversal algorithm to identify attack chain paths; S3.3.3, sequence pattern mining: using sequence pattern mining algorithm to find the frequently occurring event sequence patterns; S3.3.4, calculating the correlation strength score: comprehensively considering the time interval between events, correlation tightness and the rarity of the pattern, calculating the correlation strength score and outputting.

[0009] As a further scheme of the application: the multi-model deep fusion decision in step S4 specifically includes: S4.1, joint representation coding: splicing the output scores of the four models with their corresponding intermediate representation vectors, and encoding into a unified dimension fusion input vector through a multi-layer perception machine; S4.2, dynamic gating weight calculation: inputting the fusion input vector into the gating fusion network, which includes the query layer, the key layer and the value layer, calculating the dynamic weight of each model under the current sample condition through the multi-head attention mechanism, the dynamic weight is self-adaptively adjusted according to the input sample features, calculating the exponential value of the accuracy of each model, and dividing the exponential value of each model by the sum of the exponential values of all models to obtain the weight of the model; S4.3, uncertainty perception fusion: calculating the uncertainty estimation value of each model output, using the Monte Carlo Dropout method to perform multiple forward propagations to obtain the prediction distribution, quantifying the uncertainty according to the variance of the prediction distribution, and reducing the contribution of the high uncertainty model in the fusion calculation; S4.4, conflict detection and coordination: constructing a model output consistency matrix, calculating the Jensen-Shannon divergence between the outputs of any two models, triggering the coordination mechanism when the divergence exceeds the conflict threshold, the coordination mechanism comprehensively determines which model's output to adopt according to the historical reliability, current confidence and sample similarity of each model, and generates a conflict explanation report; S4.5, synergistic consistency enhancement: when the output trends of three or more models are consistent, starting the synergistic enhancement mechanism, multiplying the final score by the synergistic enhancement coefficient based on the fusion score, and the synergistic enhancement coefficient is positively correlated with the number of consistent models; when only a single model gives an extreme score, starting the independent verification mechanism, calling the explanatory module of the model to generate judgment basis, and combining the feedback of other models to decide whether to adopt the extreme score.

[0010] As a further scheme of the application: the calculation process of the dynamic threshold in step S5 specifically includes: S5.1, setting a sliding time window, collecting all historical risk score data within the sliding time window; S5.2, calculating the mean of the historical risk score data And the standard deviation ; S5.3, setting the high-risk threshold value as , set the high-risk threshold as ; S5.4, set threshold boundary: set the upper limit of the high-risk threshold as the absolute high-risk threshold, and set the lower limit of the medium-risk threshold as the minimum alarm threshold, and when the calculated dynamic threshold exceeds the boundary range, limit it to the boundary value; S5.5, threshold smoothing processing: adopt an exponential weighted moving average method, and the new threshold value is equal to the current calculated threshold multiplied by a smoothing coefficient plus the historical threshold multiplied by (1-smoothing coefficient) , wherein the value of the smoothing coefficient is 0.2.

[0011] As a further scheme of the application: the risk level determination in step S5 specifically includes: when the comprehensive risk score exceeds the high-risk threshold, it is determined as high risk; when the comprehensive risk score is between the medium-risk threshold and the high-risk threshold, it is determined as medium risk; when the comprehensive risk score is lower than the medium-risk threshold and higher than the normal baseline, it is determined as low risk.

[0012] As a further scheme of the application: the incremental learning mode to update the model parameters in step S6 specifically includes: S6.1, label and preprocess the collected feedback data to form a training sample set; S6.2, input the training sample into the existing model to calculate the loss function value of the model on the training sample; S6.3, update the model parameters through a gradient descent algorithm, and the update amplitude is controlled by a learning rate, and the value of the learning rate is 0.001; S6.4, verify the performance of the updated model, and the verification indexes include accuracy, recall rate, precision and score; S6.5, when the accuracy and score of the updated model improve more than 5% compared with before the update, and any index does not decrease more than 2%, deploy the updated model to the production environment; when the performance of the updated model does not meet the above requirements, give up this update, and continue to use the original model.

[0013] As a further scheme of the application: the multi-model deep fusion decision in step S4 further includes context-aware fusion, specifically including: S4.6, cooperative scene awareness: comprehensive analysis of the output scores of the four models, intermediate representation similarity and cross-model attention weight distribution, identify the scene type to which the current risk belongs through a scene classification network, the input of the scene classification network is the joint encoding vector of the four model states, when the anomaly detection model score is higher than 0.7 and the threat identification model score is lower than 0.3, identify as an unknown threat scene; When the correlation analysis model score is higher than 0.7 and the standard deviation of the scores of the other three models is greater than 0.2, identify as an advanced persistent threat scene; When the prediction model trend value increases by more than the preset increase threshold and the scores of the other three models are in the interval of 0.3 to 0.6, identify as a risk warning scene; S4.7, scene-specific fusion strategy: for unknown threat scenes, the anomaly detection model weight is set to 0.6, the threat identification model weight is set to 0.15, the correlation analysis model weight is set to 0.15, and the prediction model weight is set to 0.1; S4.8, cooperative optimization target: define a global cooperative loss function that includes the independent prediction loss of the four models, the fusion prediction loss, and the inter-model consistency regularization term, optimize the overall system performance through end-to-end training, so that each model maintains professional ability while achieving cooperative complementation; For advanced persistent threat scenes, the anomaly detection model weight is set to 0.15, the threat identification model weight is set to 0.2, the correlation analysis model weight is set to 0.45, and the prediction model weight is set to 0.2; For risk warning scenes, the anomaly detection model weight is set to 0.2, the threat identification model weight is set to 0.2, the correlation analysis model weight is set to 0.1, and the prediction model weight is set to 0.5.

[0014] As a further scheme of the present application: the prediction model in step S3 uses a long short-term memory network for risk trend prediction, specifically including: S3.4.1, construct historical risk data into a time series, the input of the time series is a risk feature sequence of N historical time windows; S3.4.2, the long short-term memory network captures the long-term dependence relationship in the risk data through the cooperative action of the forgetting gate, the input gate and the output gate; S3.4.3, the network output is a risk prediction value for the future time window, wherein the value range of is 1 to 10; S3.4.4, analyze the change trend of the current risk factors, identify risk growth factors, and calculate the contribution of each risk growth factor to the future risk; S3.4.5, output the future risk level prediction and risk change trend.

[0015] Compared with the prior art, the present application has the beneficial effects that: Firstly, the present application can automatically discover and extract new threat features through the adaptive feature extraction mechanism and multi-source data fusion technology, and no longer relies on the pre-defined fixed feature set, thereby improving the identification accuracy of new threats.

[0016] Secondly, the present application deeply couples four special models of anomaly detection, threat identification, correlation analysis and trend prediction through the construction of a cascaded collaborative network, each model shares intermediate representations through a cross-model feature transmission layer, and realizes real-time interaction through a bidirectional information feedback channel, forming an inseparable organic whole, which can simultaneously optimize the identification effect of multiple threat types, and significantly improves the detection success rate for complex attack chains and combined threats.

[0017] Thirdly, the present application can comprehensively analyze cross-data source, cross-time and cross-user correlation anomalies through the construction of a multi-dimensional correlation analysis framework, effectively identify hidden complex threats, and improve the detection ability of advanced persistent threats.

[0018] Fourthly, the present application can predict potential risk situations in advance through the introduction of a time series prediction model and a risk evolution analysis mechanism, realize active defense, and gain more disposal time for security response.

[0019] Fifthly, the present application can absorb new threat intelligence in real time through an online learning and incremental updating mechanism, continuously optimize model parameters, and significantly improve the long-term running stability of the system.

[0020] Sixthly, the present application realizes adaptive fusion weight calculation at the sample level through a dynamic gated fusion network based on an attention mechanism and an uncertainty quantification technology, cooperates with a conflict detection and coordination mechanism, effectively solves the problem of output divergence of multiple models, and significantly enhances the explainability and reliability of fusion decision. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a system overall architecture schematic diagram of the present application; Figure 2 is a complete technical flowchart of the present application; Figure 3 is a system interaction flowchart of the present application; Figure 4 is an adaptive feature extraction flowchart of the present application; Figure 5 is a multi-model deep fusion decision flowchart of the present application; Figure 6 is a dynamic threshold calculation flowchart of the present application. DETAILED DESCRIPTION

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Figure 1 As shown, this invention provides a data security risk assessment method based on a big data model. The overall technical architecture includes five core layers: a data acquisition layer, a feature processing layer, a model analysis layer, a risk assessment layer, and a response decision layer. The data acquisition layer is responsible for collecting raw security data from multiple data sources in real time, including network traffic data, system log data, user behavior data, and external threat intelligence data. This layer connects to various data sources through standardized interfaces to ensure the integrity and real-time nature of data collection. After preliminary formatting, the collected data is uniformly transmitted to the feature processing layer. The feature processing layer performs in-depth processing on the collected raw data, including four sub-processes: data cleaning, feature extraction, feature selection, and feature enhancement. The data cleaning module is responsible for removing noisy data and processing missing and outlier values. The feature extraction module extracts multi-dimensional security features from the raw data; The feature selection module uses algorithms to filter out a subset of features that contribute significantly to risk assessment. The feature enhancement module generates more expressive high-order features through feature combination and transformation. The processed feature data forms a standardized feature vector, which is input into the model analysis layer. The model analysis layer is the core layer of the system and integrates four dedicated analysis models. The anomaly detection model is responsible for identifying abnormal behaviors that deviate from the normal pattern. The threat identification model is responsible for identifying known threat types and new attack patterns; The correlation analysis model is responsible for discovering the correlations between multi-dimensional data and identifying complex attack chains; The predictive model is responsible for analyzing the evolution trend of risks and predicting potential future security threats. The models are tightly coupled through a cascaded collaborative network. The models share intermediate representation vectors through cross-model feature transfer layers and transmit adjustment signals through bidirectional information feedback channels. The analysis processes of each model influence each other and evolve collaboratively. The final output includes not only independent scores but also collaborative state vectors between models. This information is jointly transmitted to the risk assessment layer. The risk assessment layer integrates the output results of the model analysis layer to perform risk quantification calculation, level classification, and trend analysis. The risk quantification calculation module calculates the comprehensive risk score based on the output results of multiple models and collaborative state vectors. The risk level classification module divides risks into different levels based on risk scores and dynamic thresholds; The risk trend analysis module combines historical data and prediction results to analyze the evolution trend of the risk, and the evaluation result is transmitted to a response decision layer, the response decision layer generates alarm information and response suggestions according to the risk evaluation result, and collects evaluation feedback for model optimization, and the alarm generation module generates a hierarchical alarm according to the risk level and the emergency degree; The response suggestion module gives disposal suggestions according to the threat type and the risk level; The model feedback module collects actual disposal results and new threat samples, and feeds back to the model analysis layer through a model update optimization mechanism, so that the model continuously learns and the performance is improved, for example Figure 2 As shown in the figure, the complete technical process of the application is as follows: Step S1: Multi-source data acquisition, the system acquires security-related data from multiple data sources in real time through a pre-configured data acquisition interface, the data types acquired include network layer data (IP address, port number, protocol type, data packet size, traffic rate), application layer data (HTTP request, database query, file access record), system layer data (login log, system call, process information, resource usage) and external intelligence data (known malicious IP list, vulnerability information, attack feature library), the acquisition process adopts a non-blocking asynchronous mode, so that the normal operation of the business system is not affected, the raw data acquired is labeled according to a unified timestamp and data source identifier, and stored in a distributed data buffer; Step S2: Data preprocessing and cleaning, the raw data acquired is preprocessed to improve data quality, first, data integrity check is performed, missing fields are identified and processed, data records with missing key fields are discarded, and non-key field missing is filled by using a statistical method, then data format standardization is performed, different formats of timestamps are uniformly converted to a standard format, different encodings of text are uniformly converted to UTF-8 encoding, then abnormal value detection is performed, obviously unreasonable data is identified and corrected, and finally data deduplication is performed, repeated data records in the same time window are merged and processed; Step S3: Adaptive feature extraction, as shown in the figure, Figure 4 Multi-dimensional security features are extracted from the preprocessed data, basic features include statistical features (access frequency, data volume statistics, time distribution), behavior features (operation sequence pattern, access path feature) and content features (sensitive keyword frequency, data type distribution), for time series data, time series features include trend features, periodic features and mutation features, for network traffic data, flow features include connection duration, packet interval and traffic burst features, the calculation of feature importance score adopts a comprehensive evaluation method, for feature And risk label The mutual information amount is calculated according to the following formula: ; wherein, denotes a feature taking value and the risk label taking value , denotes a feature taking value , denotes a risk label taking value , Step S4: constructing a feature vector set, organizing the extracted features of each type into a structured feature vector, each to-be-evaluated object corresponding to a feature vector, each dimension of the vector representing a feature value, for numerical features, using the min-max normalization method: ; wherein, denotes an original feature value, denotes the historical minimum value of the feature, denotes the historical maximum value of the feature, denotes the normalized feature value; Step S5: feature dimension reduction and selection, since the original feature dimension is high, there is feature redundancy and noise interference, which needs to be reduced and selected, the feature dimension reduction uses the principal component analysis method, which maps the high-dimensional feature space to the low-dimensional space while retaining the main feature information, calculates the covariance matrix of the feature vector set, performs eigenvalue decomposition on the covariance matrix, selects a number of principal components with high ranking of eigenvalues, and projects the original feature vector into the subspace composed of these principal components; Step S6: multi-model cascading collaborative analysis, inputting the processed feature vector into the cascading collaborative network, which tightly connects the four special analysis models through the cross-model feature transmission layer and the bidirectional information feedback channel, the specific information flow process is as follows: the anomaly detection model uses a density-based anomaly detection algorithm, for each data point , calculates the average distance of its nearest neighbor data points, and the local density is defined as: ; wherein, denotes the local density of data point , denotes the number of neighbors, denotes the distance from data point to its th nearest neighbor, when When the average density of other data points in its neighborhood is significantly lower, it is determined to be abnormal, and the threat identification model uses feature matching method to calculate the similarity between the current feature vector and the known threat features in the threat feature library, and uses cosine similarity measurement method: ; wherein, represents the current feature vector, represents the known threat feature vector in the threat feature library, represents the inner product of two vectors, and respectively represent the modulus of two vectors, the correlation analysis model uses graph analysis technology to abstract each type of security event as a node in the graph, and the correlation between events is represented as an edge, the correlation includes time sequence correlation (events occurring before and after), causal correlation (one event leading to another event), subject correlation (events involving the same subject) and object correlation (events targeting the same target), suspicious correlation patterns are searched through graph traversal algorithm, attack chain paths are identified, and the prediction model uses long short-term memory network model for time series prediction, the model input is the risk feature sequence of the last N time windows, and the output is the risk prediction value of the next M time windows, wherein the value range of is 1 to 10, and the long short-term memory network captures long-term dependencies through the synergistic effect of the forget gate , the input gate and the output gate ; ; ; wherein, represents the sigmoid activation function, , , respectively represent the weight matrix of the forget gate, the input gate and the output gate, , , respectively represent the corresponding bias vector, represents the hidden state at the last time, represents the input at the current time; First, the original feature vector enters the input layer of the four models at the same time; In the first round of forward propagation, each model independently calculates the initial intermediate representation; Then, the intermediate representation of each model is projected to the shared feature space through the cross-model feature transmission layer, and each model obtains relevant information from the other three models through the attention mechanism and integrates it into its own representation; ​Next, a second round of forward propagation is performed, and each model calculates its output based on the enhanced representation; Finally, the trend gradient vector of the prediction model is backpropagated to other models, dynamically adjusting the sensitivity parameters of each model to complete a full round of collaborative analysis.

[0023] Step S7: Fuse results from multiple models, such as Figure 5 As shown, the analysis results of the four specialized models and their collaborative state vectors are deeply fused. The fusion process employs a dynamic gating fusion network based on a multi-head attention mechanism. ; in, Indicates the first The accuracy of each model on the validation set, and the calculation process of the dynamic gated fusion network, are as follows: First, the output scores of the four models are concatenated with the intermediate representations to form a fused input matrix. ; Then, calculate the query matrix. Key matrix Value matrix ; Next, calculate multi-head attention: ; The dynamic weights of each model are calculated using gating units: ,in For the first The hidden states of a model For global context vectors; Final fusion score: The Coherence function measures the degree of consistency between the outputs of the four models. The synergistic enhancement coefficient; ; in, , , , These represent the normalized output scores of the anomaly detection model, threat identification model, correlation analysis model, and prediction model, respectively. , , , These represent the corresponding weights and consistency adjustment factors, respectively. Based on the standard deviation of the output scores of each model calculate: when When <0.1, =1.2; When 0.1≤ When ≤0.3, =1.0; when When >0.3, =0.8; Step S8: Calculate the overall risk score. The calculation comprehensively considers the degree of anomaly, the severity of the threat, the complexity of the correlation, and future trends: ; in, , , , These represent the weighting coefficients for anomaly degree, threat severity, correlation complexity, and future trend, respectively. =0.25, =0.30, =0.25, = 0.20, the final comprehensive risk score is a value between 0 and 100, the higher the value, the higher the risk; Step S9: Dynamic threshold comparison, such as Figure 6 As shown, the calculated comprehensive risk score is compared with the dynamic threshold. The dynamic threshold is calculated based on the historical risk score distribution within the sliding time window. All risk scores within the sliding time window are collected, and their mean is calculated. and standard deviation : ; ; in, This indicates the number of risk scores within the sliding time window. Indicates the first Individual risk scores, high-risk threshold Set as: ; Medium risk threshold Set as: ; Threshold smoothing uses an exponentially weighted moving average method: ; in, This indicates the updated threshold. This indicates the threshold value currently being calculated. Indicates historical threshold, =0.2 is the smoothing coefficient. Step S10: Risk level determination. Based on the dynamic threshold comparison results, the risk is divided into different levels. When the comprehensive risk score is... Exceeding the high-risk threshold At that time, it was determined to be high-risk; when Between the medium risk threshold and high risk threshold When the risk level is between 0 and 1, it is classified as medium risk. when Below the medium risk threshold However, if the risk level is higher than the normal baseline, it is considered low risk. Step S11: Generate alarms and response suggestions. For high-risk situations, the system immediately generates high-priority alarms. Alarm information includes the time of risk occurrence, data assets involved, risk type, risk score, confidence level, and preliminary risk description. The alarms are sent to security management personnel through multiple channels, including real-time pop-ups on the system interface, email notifications, and SMS notifications. At the same time, the system generates corresponding response suggestions based on the identified threat type and risk characteristics. Step S12: Collect feedback data. The system continuously collects feedback data on risk assessment and response handling, including the verification results of alarm authenticity, the impact of actual risks, and the effectiveness of response measures. The feedback data is entered into the system by the safety management personnel after the handling is completed. Step S13: Incrementally update model parameters. When a model update is needed, incremental learning is used to update the model parameters. Newly collected feedback data is labeled and preprocessed to form a training sample set. The training samples are input into the existing model, the loss function of the model on the new samples is calculated, and the model parameters are updated using the gradient descent algorithm. ; in, This represents the updated model parameters. This represents the model parameters before the update. =0.001 is the learning rate. This represents the gradient of the loss function with respect to the model parameters. Step S14: Verify model performance. Perform performance verification on the updated model. Verification metrics include accuracy. Recall rate Accuracy and Fraction: ; ; ; ; in, This indicates the number of true positives. Indicates the number of true negatives. Indicates the number of false positives. represents the number of false negatives, when the accuracy of the updated model and The updated model is deployed to the production environment when the accuracy of the updated model is improved by more than 5% compared with that before the update, and no index decreases by more than 2%; Otherwise, the update is abandoned, and the original model is continued to be used, as shown in the following formula: Figure 3 As shown in the figure, the system interaction process of the application shows the cooperation relationship and data flow direction among the components. In the data acquisition stage, the data source continuously pushes real-time data to the data acquisition module, and the acquisition module processes the received data in a format. In the feature processing stage, the feature processing module receives the original data, performs data cleaning and feature extraction, and simultaneously analyzes the processed feature vectors in the parallel analysis stage. In the result fusion stage, the four models respectively complete the calculation and return the respective analysis results to the fusion engine for dynamic gate depth fusion. In the risk assessment stage, the risk assessment module receives the fusion result and calculates the comprehensive risk score. In the alarm response stage, different response strategies are taken according to the risk level. In the feedback collection stage, the user feeds back the disposal result to the update management module after disposing the alarm. In the model optimization stage, when the accumulated feedback data reaches a certain amount, the model update process is triggered. The application also provides a context-aware fusion mechanism, which identifies the scene type to which the current risk belongs according to the output mode of the four models: Unknown threat scenario identification: when the abnormal detection model score is higher than 0.7 and the threat identification model score is lower than 0.3, it is identified as an unknown threat scenario, for this scenario, the fusion strategy of abnormal detection model weight 0.6, threat identification model weight 0.15, correlation analysis model weight 0.15 and prediction model weight 0.1 is adopted, advanced persistent threat scenario identification: when the correlation analysis model score is higher than 0.7 and the standard deviation of the scores of the other three models is greater than 0.2, it is identified as an advanced persistent threat scenario, for this scenario, the fusion strategy of abnormal detection model weight 0.15, threat identification model weight 0.2, correlation analysis model weight 0.45 and prediction model weight 0.2 is adopted, risk warning scenario identification: when the prediction model trend value increases by more than the preset increase threshold and the scores of the other three models are in the interval of 0.3 to 0.6, it is identified as a risk warning scenario, for this scenario, the fusion strategy of abnormal detection model weight 0.2, threat identification model weight 0.2, correlation analysis model weight 0.1 and prediction model weight 0.5 is adopted, the threat intelligence library construction and maintenance process of the present application includes: threat intelligence collection, collecting threat intelligence information from multiple channels, including public threat intelligence platforms, vulnerability announcements released by security manufacturers, attack reports of security communities, and new threats discovered by the system itself, intelligence standardization processing, converting the collected threat intelligence of different formats and different sources into a unified standardized format, intelligence quality evaluation, filtering low-quality information, intelligence storage, storing the processed threat intelligence into the intelligence library, supporting quick retrieval in multiple ways such as threat type, feature keyword, IP address and domain name, intelligence dynamic updating, periodically pulling information from intelligence sources and adding it to the intelligence library.

[0024] Uncertainty quantification process: enable Dropout for each model and perform 20 times forward propagation , obtain prediction values , calculate the prediction mean and variance , the uncertainty estimate of model is = , when fusing, the model weight is inversely proportional to its uncertainty: ; Conflict detection and coordination mechanism: calculate the Jensen-Shannon divergence between the outputs of any two models: ; where, , denotes the KL divergence; The collaborative training method of the present application is as follows: Define a global collaborative loss function wherein is the independent prediction loss of the th model, is the fusion prediction loss, is the inter-model consistency regularization term; Consistency regularization term is calculated as follows: for sample pairs that should give similar judgments, the similarity of the intermediate representations of the four models is constrained to be higher than a threshold; For sample pairs that should give different judgments, the similarity of the intermediate representations is constrained to be lower than a threshold; The training process adopts an end-to-end manner, and the gradient is back-propagated through the cross-model feature transmission layer to realize joint optimization of the parameters of the four models; This training method enables each model to focus on its own task while producing intermediate representations that are beneficial to other models, forming a synergistic complementary relationship, and the overall system performance significantly exceeds that of simple combination of independently trained models.

[0025] The above only describes the preferred embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes, and any modifications, equivalent replacements, improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A data security risk assessment method based on a big data model, characterized in that: Includes the following steps: S1. Multi-source data acquisition and preprocessing: Collect security-related data from network devices, servers, and application systems, and perform integrity checks, format standardization, outlier detection, and deduplication on the collected raw data to form a preprocessed dataset; S2. Adaptive Feature Extraction: Extract statistical features, behavioral features, and content features from the preprocessed dataset. Evaluate feature importance by calculating the mutual information between features and risk labels. Filter features with importance scores higher than a preset threshold and normalize the filtered features to construct a standardized feature vector set. S3, Multi-model Cascaded Collaborative Analysis: The feature vector set is input into a cascaded collaborative network consisting of an anomaly detection model, a threat identification model, a correlation analysis model, and a prediction model. The models share intermediate representations through a cross-model feature transfer layer and achieve real-time information interaction through a two-way information feedback channel. The anomaly detection model uses a density-based anomaly detection algorithm to identify anomalous data that deviates from normal behavior patterns, and outputs anomaly scores and anomaly feature embedding vectors. The anomaly feature embedding vectors are passed to the threat identification model and the correlation analysis model as auxiliary inputs. The threat identification model receives the original feature vector and the abnormal feature embedding vector, identifies known threat types through feature matching methods, and outputs a threat score and a threat type encoding vector. The threat type encoding vector is fed back to the anomaly detection model to adjust the anomaly judgment boundary. The correlation analysis model integrates anomaly feature embedding vectors to construct an enhanced security event graph, uses graph neural network technology to discover deep correlations between multiple security events, and outputs correlation scores and event correlation topology features. The event correlation topology features are passed to the prediction model to enhance the time series modeling capability. The prediction model comprehensively receives the original time-series features, event-related topological features, and intermediate state vectors of each model. It uses an enhanced long short-term memory network to analyze the risk evolution trend and outputs a prediction score and a trend gradient vector. The trend gradient vector is backpropagated to the other three models to dynamically adjust the sensitivity parameters of each model. S4. Multi-model Deep Fusion Decision Making: The outputs and intermediate representations of the four models are jointly encoded. A dynamic gated fusion network based on a multi-head attention mechanism is used to calculate the adaptive weights of each model on the current sample. The confidence of each model's output is evaluated through uncertainty quantification. A conflict detection and coordination mechanism is introduced to handle scoring discrepancies between models. Finally, a comprehensive risk score and decision confidence are generated, where the weight of the anomaly detection model is... The weights of the threat identification model are The weights of the association analysis model are The weights of the prediction model are Each weight satisfies + + + =1; S5. Dynamic Threshold Comparison and Risk Level Determination: Calculate the dynamic threshold based on the historical risk score within the sliding time window, compare the comprehensive risk score with the dynamic threshold, and classify the risk into three levels: high risk, medium risk, and low risk based on the comparison results. S6. Response Decision and Model Optimization: Generate corresponding alarm information and response suggestions based on the risk level, collect feedback data on risk assessment and response handling, and update model parameters using incremental learning when the accumulated amount of feedback data reaches a preset threshold.

2. The data security risk assessment method based on a big data model according to claim 1, characterized in that: The adaptive feature extraction in step S2 specifically includes: S2.1 Constructing the initial feature pool: Extract the mean, variance, maximum value, minimum value, and median features from numerical data fields; extract the value distribution and entropy features from categorical data fields; and extract the length and element frequency features from sequence data. S2.2 Calculate the feature importance score: The mutual information method is used to calculate the mutual information between each feature and the risk label, the Gini coefficient method is used to evaluate the contribution of the feature to the classification purity of the sample, and the correlation coefficient method is used to calculate the correlation between the feature and the risk label. The scores of the three methods are combined to assign a comprehensive importance score to each feature. S2.3 Feature Filtering: Delete features with a comprehensive importance score below a preset threshold. For two features with a correlation coefficient greater than 0.9, retain the feature with the higher importance score. S2.4 Feature Normalization: For numerical features, the min-max normalization method is used to map the feature values ​​to the interval between 0 and 1. For categorical features, the one-hot encoding method is used to convert them into numerical representations.

3. The data security risk assessment method based on a big data model according to claim 1, characterized in that: The analysis process of the anomaly detection model in step S3 specifically includes: S3.1.1 Establish a baseline model for normal behavior: Define the range of normal behavior by learning the distribution characteristics of historical normal data; S3.1.2 Calculate local density: For each data point, search its... Nearest neighbor data points, calculated to The average distance of the nearest neighbors, of which The value ranges from 5 to 20, and the reciprocal of this average distance is used as the local density value of the data point. S3.1.3 Anomaly Detection: When the local density value of a data point is lower than a preset proportion of the average density value of data points in its neighborhood, the data point is determined to be an anomaly. The preset proportion ranges from 0.5 to 0.

8. S3.1.

4. The Isolation Forest algorithm is used as an auxiliary detection method. Data points are isolated by randomly constructing decision trees. The number of splits required to isolate each data point is recorded. Data points with fewer than the preset number of splits are judged as abnormal. S3.1.

5. Output an anomaly score by combining the results of the two algorithms.

4. The data security risk assessment method based on a big data model according to claim 1, characterized in that: The collaborative interaction mechanism between the models in step S3 specifically includes: S3.2.1 Constructing a shared feature space: Projecting the intermediate layer feature representations of the four models into a unified shared feature space. The dimension of the shared feature space is the larger value among the intermediate feature dimensions of each model, and the projection process uses a learnable linear transformation matrix; S3.2.2 Cross-attention calculation: For any two models and model Computational model For the model Attention weights Attention weight representation model Features of the model The degree of influence of the decision is calculated using a scaled dot product attention mechanism; S3.2.3 Feature Enhancement Transmission: Each model performs weighted aggregation of shared features from other models based on attention weights. The aggregated cross-model features are concatenated with the features of the model itself to form an enhanced feature representation for subsequent calculations; S3.2.4 Gradient Co-optimization: During the model training phase, the loss functions of the four models establish gradient correlations by sharing a feature space. The parameter updates of a single model are constrained by both its own loss and cross-model consistency loss. Cross-model consistency loss is defined as the weighted sum of the differences in risk assessments of different models for the same input sample; S3.2.5 Dynamic Gating Mechanism: Based on the feature distribution of the current input data, dynamically calculate the gating coefficients for information transmission between models. When the confidence level of a model's output is detected to be lower than a preset threshold, the gating coefficient for that model to receive information from other models is increased. When the difference between the output of a certain model and other models exceeds the preset divergence threshold, the divergence mediation mechanism is activated to perform joint inference.

5. The data security risk assessment method based on a big data model according to claim 1, characterized in that: The analysis process of the correlation analysis model in step S3 specifically includes: S3.3.1 Constructing a security event graph: Abstracting various security events into nodes in the graph, and representing the relationships between events as edges, including temporal relationships, causal relationships, subject relationships, and object relationships; S3.3.2 Attack Chain Path Search: Search for suspicious association patterns using a graph traversal algorithm to identify attack chain paths; S3.3.3 Sequence Pattern Mining: Sequence pattern mining algorithms are used to discover frequently occurring event sequence patterns; S3.3.4 Calculate the association strength score: Taking into account the time interval between events, the tightness of association, and the rarity of the pattern, calculate the association strength score and output it.

6. The data security risk assessment method based on a big data model according to claim 1, characterized in that: The multi-model deep fusion decision in step S4 specifically includes: S4.1 Joint Representation Encoding: The output scores of the four models are concatenated with their corresponding intermediate representation vectors and encoded into a unified dimension fusion input vector through a multilayer perceptron. S4.2 Dynamic Gated Weight Calculation: The fused input vector is input into the gated fusion network, which includes a query layer, a key layer, and a value layer. The dynamic weight of each model under the current sample conditions is calculated through a multi-head attention mechanism. The dynamic weight is adaptively adjusted according to the features of the input sample. The exponential value of the accuracy of each model is calculated. The exponential value of each model is divided by the sum of the exponential values ​​of all models to obtain the weight of that model. S4.3 Uncertainty-aware fusion: Calculate the uncertainty estimate of each model output, use the Monte Carlo Dropout method to perform multiple forward propagations to obtain the prediction distribution, quantify the uncertainty based on the variance of the prediction distribution, and reduce the contribution of high uncertainty models in the fusion calculation; S4.4 Conflict Detection and Coordination: Construct a model output consistency matrix, calculate the Jensen-Shannon divergence between any two model outputs, and trigger a coordination mechanism when the divergence exceeds the conflict threshold. The coordination mechanism comprehensively determines which model's output to adopt based on the historical reliability, current confidence level, and sample similarity of each model, and generates a conflict explanation report. S4.5, Cooperative Consistency Enhancement: When the output trends of three or more models are consistent, the cooperative enhancement mechanism is activated. The final score is multiplied by the cooperative enhancement coefficient based on the fusion score. The cooperative enhancement coefficient is positively correlated with the number of consistent models. When only a single model gives an extreme score, the independent verification mechanism is activated. The interpretive module of the model is called to generate the judgment criteria. The extreme score is then combined with the feedback from other models to decide whether to accept it.

7. The data security risk assessment method based on a big data model according to claim 1, characterized in that: The calculation process of the dynamic threshold in step S5 specifically includes: S5.1 Set a sliding time window and collect all historical risk score data within the sliding time window; S5.2 Calculate the mean of historical risk score data and standard deviation ; S5.3, Set a high-risk threshold as follows The medium-risk threshold is set as follows: ; S5.4 Setting threshold boundaries: Set the upper limit of the high-risk threshold as the absolute high-risk threshold, and set the lower limit of the medium-risk threshold as the minimum alarm threshold. When the calculated dynamic threshold exceeds the boundary range, limit it to the boundary value. S5.5 Threshold Smoothing: An exponentially weighted moving average method is used, and the new threshold is equal to the current calculated threshold multiplied by the smoothing coefficient. Add the historical threshold multiplied by ( ), where the smoothing coefficient The value is 0.2; The risk level determination in step S5 specifically includes: When the comprehensive risk score exceeds the high-risk threshold, it is judged as high-risk; When the comprehensive risk score falls between the medium risk threshold and the high risk threshold, it is classified as medium risk; When the overall risk score is below the medium risk threshold but above the normal baseline, it is considered low risk.

8. The data security risk assessment method based on a big data model according to claim 1, characterized in that: The incremental learning method for updating model parameters in step S6 specifically includes: S6.

1. Label and preprocess the collected feedback data to form a training sample set; S6.2 Input the training samples into the existing model and calculate the loss function value of the model on the training samples; S6.3 Update the model parameters using the gradient descent algorithm. The update magnitude is controlled by the learning rate, which is 0.

001. S6.

4. Validate the performance of the updated model, including accuracy, recall, precision, and... Fraction; S6.5, When the accuracy of the updated model and When the score improves by more than 5% compared to before the update, and no metric decreases by more than 2%, the updated model will be deployed to the production environment. If the performance of the updated model does not meet the above requirements, abandon the update and continue using the original model.

9. The data security risk assessment method based on a big data model according to claim 1, characterized in that: The multi-model deep fusion decision in step S4 also includes context-aware fusion, specifically including: S4.6 Collaborative Scene Awareness: By comprehensively analyzing the output scores, intermediate representation similarity, and cross-model attention weight distribution of the four models, the scene classification network identifies the scene type to which the current risk belongs. The input of the scene classification network is the joint encoding vector of the four model states. When the anomaly detection model score is higher than 0.7 and the threat recognition model score is lower than 0.3, it is identified as an unknown threat scene. When the correlation analysis model score is higher than 0.7 and the standard deviation of the scores of the other three models is greater than 0.2, it is identified as an advanced persistent threat scenario. When the increase in the trend value of the prediction model exceeds the preset increase threshold and the scores of the other three models are in the range of 0.3 to 0.6, it is identified as a risk warning scenario; S4.7 Scenario-Specific Fusion Strategy: For scenarios with unknown threats, the weight of the anomaly detection model is set to 0.6, the weight of the threat identification model is set to 0.15, the weight of the correlation analysis model is set to 0.15, and the weight of the prediction model is set to 0.

1. S4.8 Collaborative Optimization Objective: Define a global collaborative loss function, which includes the independent prediction loss, fusion prediction loss, and inter-model consistency regularization term of the four models. Optimize the overall system performance through end-to-end training, so that each model can achieve synergistic complementarity while maintaining its specialized capabilities. For advanced persistent threat scenarios, the weight of the anomaly detection model is set to 0.15, the weight of the threat identification model is set to 0.2, the weight of the correlation analysis model is set to 0.45, and the weight of the prediction model is set to 0.

2. For risk warning scenarios, the weight of the anomaly detection model is set to 0.2, the weight of the threat identification model is set to 0.2, the weight of the correlation analysis model is set to 0.1, and the weight of the prediction model is set to 0.

5.

10. The data security risk assessment method based on a big data model according to claim 1, characterized in that: The prediction model in step S3 uses a long short-term memory network for risk trend prediction, specifically including: S3.4.1 Construct historical risk data into a time series, with the input of the time series being a sequence of risk characteristics for N historical time windows; S3.4.2 Long Short-Term Memory Networks capture long-term dependencies in risky data through the synergistic effect of forget gates, input gates, and output gates; S3.4.3, Network Output for the Future Risk prediction values ​​for each time window, among which The value range is from 1 to 10; S3.4.4 Analyze the changing trends of current risk factors, identify risk growth factors, and calculate the contribution of each risk growth factor to future risks; S3.4.5 Output future risk level predictions and risk change trends.

Images