Network data maintenance method and system based on big data and artificial intelligence

Abstract

The invention discloses a network data maintenance method and system based on big data and artificial intelligence. The method comprises the following steps: constructing a network data multi-dimensional feature space, extracting time sequence features by using an optimized long and short-term memory network, dividing feature subsets according to data traffic fluctuation, and integrating key features by improving an Adaboost algorithm; abnormal data are recognized through anomaly detection analysis, repairing or reconstruction operation is executed according to priorities, and finally the data are stored and an associated index is established. The system comprises seven units including a network data multi-dimensional feature space construction unit and a long-short-term memory network feature extraction unit, and all the units are sequentially connected to cooperatively work. According to the scheme, efficient analysis, precise maintenance and ordered management of the network data are realized based on the multi-dimensional parameters of the network data by fusing an innovative algorithm, and the integrity and availability of the network data are effectively guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of network data maintenance, and in particular to a network data maintenance method and system based on big data and artificial intelligence. Background Art

[0002] With the rapid development of big data and artificial intelligence technologies, the scale of network data is exploding, with increasingly diverse data types and more frequent transmission and interaction. As a core asset in the digital age, the integrity, accuracy, and availability of network data directly impact the stable operation of network services, enterprise decision-making, and the protection of user privacy. However, the complex and ever-changing network environment presents numerous risks during data transmission, storage, and processing, such as data anomalies, loss, and corruption. This makes network data maintenance a critical component in ensuring network security and efficient operation.

[0003] Existing network data maintenance technologies have significant limitations. First, they lack the ability to handle the dynamic characteristics and time series relationships of network data. Traditional methods often view network data in isolation and lack effective modeling of both long-term and short-term dependencies across time. In real-world network environments, network data characteristics such as traffic flow and transmission delays continuously evolve over time, exhibiting significant temporal and dynamic characteristics. However, traditional technologies struggle to capture these time series characteristics, making it impossible to accurately predict and respond to abnormal data changes. This significantly reduces the timeliness and effectiveness of data maintenance.

[0004] Second, traditional data maintenance technologies have limited capabilities for feature extraction and integration of complex network data. Network data is multidimensional and heterogeneous, encompassing numerous attributes such as data type, transmission protocol, and traffic volume, with complex relationships between these attributes. The single feature extraction and processing methods employed by traditional technologies fail to fully tap into the potential connections between data and make it difficult to comprehensively analyze all data parameters. Faced with massive amounts of complex network data, traditional methods struggle to accurately identify key features, making it prone to misjudgments or omissions during data anomaly detection and repair. These methods fail to meet the high-precision requirements for network data maintenance in today's complex network environments. Summary of the Invention

[0005] In order to overcome the shortcomings and deficiencies of the existing technology, the present invention provides a network data maintenance method and system based on big data and artificial intelligence.

[0006] The technical solution adopted by the present invention is a network data maintenance method based on big data and artificial intelligence, comprising the following steps:

[0007] Step S1: Constructing a multidimensional feature space of network data based on the network operating environment, classifying and identifying network data according to data type, transmission protocol, and flow size attributes, establishing corresponding data nodes in the multidimensional feature space, and constructing connection edges between nodes based on data interaction relationships and time series relationships to form an initial network data feature network structure;

[0008] Step S2: Using an optimized long short-term memory network to extract time series features from the initial network data feature network structure, by designing a multi-layer memory unit structure and combining a gating mechanism to perform deep learning on the long-term and short-term dependencies of the network data in the time dimension, a preliminary feature set including time series feature vectors is obtained;

[0009] Step S3: Divide the preliminary feature set into intervals according to the data flow fluctuation range to form multiple data feature subsets, each data feature subset corresponding to a different data flow fluctuation interval;

[0010] Step S4: Using an improved Adaboost algorithm to process the data feature subset, by redesigning the weight update strategy of the weak classifier and combining the transmission delay, packet loss rate, and error rate parameters of the network data, the key features in the data feature subset are weighted and integrated to obtain an enhanced feature vector set;

[0011] Step S5: performing anomaly detection analysis on the enhanced feature vector set, and identifying abnormal feature vectors that deviate from the normal distribution based on the historical normal feature distribution properties of the network data;

[0012] Step S6: performing data repair or data reconstruction operations on the network data corresponding to the abnormal feature vector according to the priority and importance parameters of the network data;

[0013] Step S7: storing the maintained network data in a preset network data repository, and establishing an association index between the maintained network data, the original data identifier, and the maintenance operation record.

[0014] Furthermore, the optimized long short-term memory network is designed to update the memory unit state model formula as follows:

[0015]

[0016] Among them, C t Represents the state of the memory unit at time t; C t-1 represents the state of the memory unit at time t-1; f t is the output value of the forget gate at time t, which is determined by the timestamp interval of the network data and the data change frequency parameter, and is used to control the degree of retention of the memory unit state at the previous moment; it is the output value of the input gate at the t th moment, which is calculated according to the real-time traffic of network data and data type change parameters, and is used to control the inflow of input information at the current moment; is the candidate memory cell state calculated at the current moment; and represents multiplication of corresponding elements of matrices.

[0017] Further, the improved Adaboost algorithm is designed, and a weak classifier weight updating model formula is:

[0018]

[0019] wherein, α m is the weight of the m th weak classifier; e m is the classification error of the m th weak classifier; β m is a weighted correction coefficient, which is dynamically adjusted by a transmission bandwidth utilization rate and a data retransmission number parameter of network data, and is used to correct the weight of the weak classifier, so that the algorithm is more suitable for network data characteristics.

[0020] Further, in the step S3, the division of the data traffic fluctuation interval is dynamically adjusted according to a historical traffic peak value, a historical traffic valley value parameter and a standard deviation parameter of traffic change, and an adjustment formula is:

[0021] I n =[μ-k×σ,μ+k×σ]

[0022] wherein, I n is the n th data traffic fluctuation interval; μ is the average value of historical network data traffic; σ is the standard deviation of historical network data traffic; and k is an adjustment coefficient, which is set according to the stability parameter of network data traffic change.

[0023] Further, in the step S4, when the improved Adaboost algorithm is used to process the data feature subset, the credibility score of each feature vector in the strengthened feature vector set is calculated in combination with a source IP address credibility and a destination IP address security parameter of network data, and a scoring formula is:

[0024] S v =w1×R s +w2×S d

[0025] wherein, S v is the credibility score of the feature vector; R s is the source IP address credibility parameter; S d is the destination IP address security parameter; and w1 and w2 are weight coefficients set according to network security policies.

[0026] Furthermore, in step S5, the anomaly detection analysis adopts a density anomaly detection method based on normal traffic pattern parameters and data transmission pattern parameters of network data. During the detection process, the anomaly degree formula is defined as:

[0027]

[0028] Among them, D x is the abnormality of the eigenvector x; d xy is the distance between feature vector x and feature vector y, calculated by the Manhattan distance of network data feature vectors; ∈ is the preset distance threshold; χ is the indicator function, when d xy When <∈, the value of χ is 1, otherwise it is 0.

[0029] Furthermore, in step S6, when performing data repair or data reconstruction operations, the integrity check value and timeliness parameters of the network data are introduced to construct an operation priority model formula:

[0030]

[0031] Among them, P o is the operation priority; C i is the integrity check value of the network data; T e is the timeliness parameter of network data; w3 and w4 are weight coefficients set according to the network data maintenance requirements.

[0032] Furthermore, in step S2, when using the optimized long short-term memory network to extract time series features, differentiated gating mechanism parameters are designed for different types of network data. The parameters are dynamically updated according to the real-time traffic trend and data transmission direction parameters of the data. The update formula is:

[0033]

[0034] Among them, G p is the gating mechanism parameter after the pth update; G p-1 is the gating mechanism parameter of the last time; η is the learning rate; F t is the real-time traffic of current data; F max is the preset maximum flow rate; D d D is the current data transmission direction parameter; max is the preset maximum transmission direction parameter value; ΔG is the parameter update step size.

[0035] Furthermore, in step S4, when the improved Adaboost algorithm is used to process the data feature subset, the classification boundary of the weak classifier is dynamically adjusted in combination with the protocol type diversity and port usage frequency parameters of the network data. The adjustment formula is:

[0036]

[0037] Among them, B m is the classification boundary of the mth weak classifier; B m-1 is the last classification boundary; γ is the adjustment coefficient; N p The number of protocol types for the current data; N pmax The maximum number of preset protocol types; F u The frequency used by the current port; F umax is the preset maximum port usage frequency; ΔB is the classification boundary adjustment step.

[0038] A network data maintenance system based on big data and artificial intelligence, which includes:

[0039] A network data multi-dimensional feature space construction unit, used to construct a network data multi-dimensional feature space based on a network operating environment;

[0040] A long short-term memory network feature extraction unit is connected to the network data multidimensional feature space construction unit and is used to use the optimized long short-term memory network to extract time series features from the initial network data feature network structure;

[0041] a data feature interval division unit, connected to the long short-term memory network feature extraction unit, for dividing the preliminary feature set into intervals according to the data flow fluctuation amplitude;

[0042] an improved Adaboost algorithm processing unit, connected to the data feature interval division unit, for processing the data feature subset using the improved Adaboost algorithm;

[0043] an anomaly detection and analysis unit, connected to the improved Adaboost algorithm processing unit, for performing anomaly detection and analysis on the enhanced feature vector set;

[0044] a data maintenance operation execution unit, connected to the anomaly detection and analysis unit, configured to perform a data repair or data reconstruction operation on the network data corresponding to the anomaly feature vector;

[0045] The data storage index building unit is connected to the data maintenance operation execution unit and is used to store the maintained network data in a preset network data repository and to establish an associated index between the maintained network data and the original data identifier and the maintenance operation record.

[0046] Beneficial effects: The present invention proposes a network data maintenance method and system based on big data and artificial intelligence. The present invention transforms multidimensional attributes such as data type and transmission protocol into structured nodes and connection edges by constructing a multidimensional feature space of network data, laying the foundation for in-depth analysis; the optimized long-short-term memory network uses multi-layer memory units and gating mechanisms to dynamically update the memory unit status according to parameters such as timestamp interval and data change frequency, accurately extract the long-term and short-term dependencies of network data in the time dimension, and effectively solve the problem of insufficient processing of time series features by traditional technologies. The Adaboost algorithm is improved to redesign the weight update strategy of the weak classifier, and the weighted correction coefficient is adjusted in combination with parameters such as transmission delay and packet loss rate. At the same time, the classification boundary is dynamically optimized according to the diversity of protocol types and the frequency of port use, so as to achieve efficient integration of complex network data features and enhancement of key features. In the anomaly detection link, the interval is divided based on the fluctuation of data traffic, and the density anomaly detection method and multi-dimensional parameters are used to construct an anomaly evaluation model to accurately identify abnormal data; for abnormal data, the repair or reconstruction priority is determined according to parameters such as integrity check value and timeliness. The various units of the system work closely together to form a complete link from feature construction to storage index, and ultimately establish an associated index of data and operation records, significantly improving the accuracy, timeliness and intelligence of network data maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of the method steps of the present invention;

[0048] Figure 2 It is a diagram of the system unit composition of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that, unless there is a conflict, the embodiments and features in the embodiments of this application can be combined with each other. The present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] like Figure 1 As shown, the network data maintenance method based on big data and artificial intelligence includes the following steps:

[0051] Step S1: Constructing a multidimensional feature space of network data based on the network operating environment, classifying and identifying network data according to data type, transmission protocol, and flow size attributes, establishing corresponding data nodes in the multidimensional feature space, and constructing connection edges between nodes based on data interaction relationships and time series relationships to form an initial network data feature network structure;

[0052] Specifically, the core task of step S1 is to construct a multi-dimensional feature space for network data based on the network operating environment. During the construction process, network data must be strictly classified and identified according to attributes such as data type, transmission protocol, and traffic size. Data types cover various forms such as text, images, and videos; transmission protocols include TCP, UDP, HTTP, etc.; and traffic size is divided according to the number of bytes of data transmitted. After establishing the corresponding data nodes in the multi-dimensional feature space, the connection edges between the nodes are constructed based on the data interaction relationship and the time series relationship. The data interaction relationship reflects the association between transmission, calling, etc. between different data; the time series relationship reflects the order and continuity of the data in the time dimension. In this way, the initial network data feature network structure is formed, which provides a structured data foundation for subsequent feature extraction and analysis.

[0053] Transforming complex network data into a calculable and analyzable characteristic network structure allows for a clear presentation of the relationships between previously scattered and isolated data. In terms of implementation, the network operating environment must first be comprehensively scanned to obtain the various attribute information of the network data. Using professional data acquisition tools, data types can be automatically identified, transmission protocols can be parsed, and traffic volume statistics can be calculated. Data interaction relationships and time series relationships are determined by analyzing information such as the source address, destination address, and timestamp of network data packets. During the construction process, the accuracy and completeness of the information must be ensured, and the attributes of nodes and edges must be labeled in detail so that subsequent steps can be used for in-depth analysis.

[0054] Step S2: Using an optimized long short-term memory network to extract time series features from the initial network data feature network structure, by designing a multi-layer memory unit structure and combining a gating mechanism to perform deep learning on the long-term and short-term dependencies of the network data in the time dimension, a preliminary feature set including time series feature vectors is obtained;

[0055] Specifically, step S2 uses an optimized long-short-term memory network to extract time series features from the initial network data feature network structure. The optimized long-short-term memory network is designed with a multi-layer memory unit structure that can store and process information in the time dimension of network data. Through a gating mechanism, the network can precisely control the inflow, outflow, and retention of information. The operation of the gating mechanism is based on parameters such as the timestamp interval of the network data and the frequency of data changes. These parameters determine the degree of attention and processing method of the memory unit for information at different time points. Through this design, the network can deeply learn the long-term and short-term dependencies of network data in the time dimension, thereby obtaining a preliminary feature set containing time series feature vectors.

[0056] From a time series perspective, this approach mines the inherent characteristics and changing patterns of network data, providing more valuable information for subsequent anomaly detection and maintenance operations. During implementation, the number of layers in the long-short-term memory network and the number of memory units in each layer must be determined. This needs to be appropriately set based on the complexity and scale of the network data. During training, historical network data is used as input, and network parameters are adjusted to enable the network to accurately capture the time series characteristics of the data. The parameters of the gating mechanism are dynamically adjusted based on real-time data such as timestamp intervals and data change frequency to ensure that the network can effectively process different types of time series data. After training is complete, the initial network data feature network structure is input into the optimized long-short-term memory network to extract a preliminary feature set containing time series feature vectors.

[0057] Step S3: Divide the preliminary feature set into intervals according to the data flow fluctuation range to form multiple data feature subsets, each data feature subset corresponding to a different data flow fluctuation interval;

[0058] Specifically, step S3 divides the preliminary feature set into intervals according to the data traffic fluctuation amplitude to form multiple data feature subsets. The data traffic fluctuation amplitude reflects the degree of change in network data traffic over different time periods. By analyzing it, the data can be divided into different fluctuation intervals. Each data feature subset corresponds to a different data traffic fluctuation interval. This division method facilitates the subsequent use of different processing strategies for data with different fluctuation characteristics. During the division process, it is necessary to consider the historical traffic peak and valley parameters of the network data, as well as the standard deviation parameters of the traffic changes. These parameters can help determine reasonable interval boundaries, so that the division results are more consistent with the actual data fluctuations.

[0059] Gathering data with similar traffic fluctuation characteristics together facilitates subsequent targeted analysis and processing, improving the efficiency and accuracy of data maintenance. In terms of implementation, we must first conduct a statistical analysis of the data traffic fluctuation amplitude in the preliminary feature set, and calculate parameters such as the peak value, valley value, and standard deviation of the historical traffic. Then, based on these parameters, determine a reasonable interval division method, which can be equal-width division or adaptive division based on data distribution. During the division process, it is necessary to ensure that the data in each interval has similar traffic fluctuation characteristics, while considering the continuity and integrity between intervals. After the division is completed, the data in the preliminary feature set is allocated to the corresponding data feature subset according to the traffic fluctuation interval to which it belongs.

[0060] Step S4: Using an improved Adaboost algorithm to process the data feature subset, by redesigning the weight update strategy of the weak classifier and combining the transmission delay, packet loss rate, and error rate parameters of the network data, the key features in the data feature subset are weighted and integrated to obtain an enhanced feature vector set;

[0061] Specifically, step S4 uses the improved Adaboost algorithm to process the data feature subset. The improved Adaboost algorithm redesigns the weight update strategy of the weak classifier, combines the parameters such as the transmission delay, packet loss rate, and error rate of the network data, and performs weighted integration on the key features in the data feature subset. The transmission delay reflects the transmission time of the data in the network, the packet loss rate represents the proportion of data lost during the transmission process, and the error rate reflects the accuracy of data transmission. These parameters can reflect the quality and stability of the network data. By integrating them into the weight update process of the weak classifier, the algorithm can pay more attention to the features that have an important impact on the quality of the network data. Through this weighted integration, a strengthened feature vector set is obtained, in which the key features are highlighted and strengthened.

[0062] This step improves the ability to extract and integrate network data features and enhances the ability to identify data anomalies and quality issues. During implementation, the first thing to do is to select an appropriate type of weak classifier, such as a decision tree stump. Then, based on parameters such as the transmission delay, packet loss rate, and error rate of the network data, the weights of the weak classifiers are initialized. During the training process, the weights of the weak classifiers are continuously adjusted to make them pay more attention to samples that are difficult to classify and features that have a significant impact on the quality of network data. Through multiple iterations, multiple weak classifiers are combined into a strong classifier, and the data feature subset is processed to obtain a strengthened feature vector set. During the processing process, the various parameters of the network data are monitored in real time, and the weights of the weak classifiers are dynamically adjusted according to the changes in the parameters to ensure the adaptability and accuracy of the algorithm.

[0063] Step S5: performing anomaly detection analysis on the enhanced feature vector set, and identifying abnormal feature vectors that deviate from the normal distribution based on the historical normal feature distribution properties of the network data;

[0064] Specifically, step S5 performs anomaly detection analysis on the enhanced feature vector set. Based on the historical normal feature distribution properties of the network data, abnormal feature vectors that deviate from the normal distribution are identified. Under normal circumstances, network data typically has a certain distribution pattern. By analyzing historical data, a distribution model of normal features can be established. When the enhanced feature vector deviates from this distribution model, it is considered an abnormal feature vector. During the detection process, the normal traffic pattern parameters and data transmission pattern parameters of the network data are taken into account. These parameters can help more accurately determine whether the feature vector is abnormal.

[0065] This step promptly detects anomalies in network data, providing a basis for subsequent data repair and maintenance. In terms of implementation, a large amount of historical network data must first be collected, analyzed, and processed to establish a distribution model of normal features. This model can be established using a variety of methods, including statistical methods and machine learning. The enhanced feature vector set is then compared with the normal feature distribution model, and the degree of deviation of each feature vector from the normal distribution is calculated. When the degree of deviation exceeds a preset threshold, the feature vector is marked as an abnormal feature vector. During the detection process, the normal feature distribution model is continuously updated and optimized to adapt to the dynamic changes in network data and improve the accuracy of anomaly detection.

[0066] Step S6: performing data repair or data reconstruction operations on the network data corresponding to the abnormal feature vector according to the priority and importance parameters of the network data;

[0067] Specifically, step S6 performs data repair or reconstruction on the network data corresponding to the abnormal feature vector based on the network data's priority and importance parameters. The network data's priority reflects its importance and urgency within the business process, while the importance parameter reflects the data's impact on system operations and decision-making. By comprehensively considering these two parameters, the appropriate maintenance actions for the abnormal data and the order in which they should be performed are determined. High-priority and highly important data is prioritized for repair or reconstruction to ensure its availability and accuracy.

[0068] This step rationally allocates resources, efficiently handles abnormal data, and minimizes its impact on network systems and services. During implementation, a system for assessing the priority and importance of network data must be established, with defined criteria and methods. Next, the network data corresponding to the abnormal feature vectors must be evaluated to determine its priority and importance parameters. Based on these parameters, a corresponding data repair or reconstruction plan can be developed. Data repair operations can include error correction and data supplementation; data reconstruction operations can reorganize data structures and optimize data storage. During the execution of these operations, strict adherence to the plan must be ensured to ensure operational effectiveness and reliability.

[0069] Step S7: storing the maintained network data in a preset network data repository, and establishing an association index between the maintained network data, the original data identifier, and the maintenance operation record.

[0070] Specifically, step S7 stores the maintained network data in a pre-set network data repository and establishes an index linking the maintained network data with the original data identifier and maintenance operation records. The maintained network data has been repaired or reconstructed, improving its quality and requiring proper storage for subsequent use. The original data identifier uniquely identifies each piece of network data. By establishing an association with the maintained network data, the data change history can be traced. The maintenance operation record details the maintenance operations performed on the data, including information such as the operation type, operation time, and operator, providing a basis for data management and auditing.

[0071] This step establishes a comprehensive network data storage and management system to ensure data traceability and maintainability. In terms of implementation, the first step is to select an appropriate network data repository based on factors such as the data's size, type, and access requirements. The maintained network data is then stored in the repository according to a specific format and structure. During the storage process, each piece of maintained network data is assigned a unique identifier and associated with the original data identifier. At the same time, maintenance operations performed on the data are recorded, and an association is established between the maintenance operation records and the maintained network data. Finally, an efficient association index is established to facilitate rapid query and retrieval of data and related information, improving the efficiency and convenience of data management.

[0072] Preferably, in the optimized long short-term memory network, the memory unit state update model formula is designed as follows:

[0073]

[0074] Among them, C t Represents the state of the memory unit at time t; C t-1 represents the state of the memory unit at time t-1; f t is the output value of the forget gate at time t, which is determined by the timestamp interval of the network data and the data change frequency parameter, and is used to control the degree of retention of the memory unit state at the previous moment; i t is the output value of the input gate at time t, which is calculated based on the real-time traffic of network data and the data type change parameters, and is used to control the inflow of input information at the current moment; is the candidate memory cell state calculated at the current moment; ⊙ represents the multiplication of corresponding matrix elements.

[0075] Specifically, during the optimized LSTM network processing, a specific memory cell state update mechanism was designed to accurately update the memory cell state. This mechanism comprehensively considers the current input information and the memory cell state at the previous moment. A forget gate controls the degree of retention of the previous state. Its value is determined by parameters such as the timestamp interval and data change frequency of the network data. The shorter the timestamp interval and the higher the data change frequency, the smaller the forget gate output value, and the less the previous state is retained. The input gate controls the inflow of current input information. It is calculated based on parameters such as the real-time traffic volume and data type changes of the network data. The greater the real-time traffic volume and the more frequent the data type changes, the larger the input gate output value, indicating more current information inflow. This dynamic adjustment enables the network to better capture the temporal characteristics of network data and improves its ability to learn long-term and short-term data dependencies. During implementation, relevant parameter data must be collected first, and the forget gate and input gate output values ​​are calculated before performing the memory cell state update. This ensures that the model can effectively process network data with diverse characteristics.

[0076] Preferably, in the improved Adaboost algorithm, the weak classifier weight update model formula is designed as follows:

[0077]

[0078] Among them, α m is the weight of the mth weak classifier; e m is the classification error of the mth weak classifier; β m It is a weighted correction coefficient, whose value is dynamically adjusted by the transmission bandwidth utilization rate of network data and the number of data retransmission parameters. It is used to correct the weight of the weak classifier to make the algorithm more adaptable to the characteristics of network data.

[0079] Specifically, the redesigned attention calculation method in the Adaboost algorithm is improved by introducing a weighted coefficient matrix to strengthen focus on key threat features. The weighted coefficient matrix is ​​generated based on parameters such as the urgency of the cybersecurity threat, the credibility of the attack source, and the difficulty of defense. The higher the urgency, the lower the credibility of the attack source, and the greater the difficulty of defense, the greater the weight of the corresponding feature. This design enables the algorithm to dynamically adjust attention allocation based on the actual threat situation when processing feature sequences, focusing more resources on high-risk features and improving its ability to identify complex threats. During implementation, threat parameters are collected in real time, and a weighted coefficient matrix is ​​generated through specific calculations. This matrix is ​​then integrated into the attention calculation process of the Transformer algorithm. By optimizing model parameters using a large amount of training data, the algorithm can accurately assess the impact of each parameter on the threat, achieve precise weighted aggregation of threat features, and enhance the system's perception and response capabilities to cybersecurity threats.

[0080] Preferably, in the step S3, the division of the data flow fluctuation interval is dynamically adjusted according to the historical flow peak value, valley value parameter and standard deviation parameter of network data, and the adjustment formula is:

[0081] I n =[μ-k×σ,μ+k×σ]

[0082] Wherein, I n is the nth data flow fluctuation interval; μ is the average value of the historical flow of network data; σ is the standard deviation of the historical flow of network data; k is the adjustment coefficient, which is set according to the stability parameter of network data flow change.

[0083] Specifically, the time window length adaptive adjustment mechanism in step S3 dynamically optimizes the time window according to the historical flow characteristics of network data. This mechanism considers the threat historical frequency and network flow fluctuation. When the historical frequency is high and the flow fluctuation is large, the time window is shortened to capture the rapidly changing threat characteristics; otherwise, the time window is increased to ensure that enough threat information is covered. Through this dynamic adjustment, the time window is matched with the threat change rhythm, and the rationality of feature sequence division is improved. In implementation, the historical flow peak value, valley value and change standard deviation are calculated by continuously monitoring network data. The appropriate time window length is calculated by using these parameters. With the change of network environment, the calculation result is updated in real time to ensure that the time window can adapt to the change of data characteristics in time and provide accurate data units for subsequent analysis.

[0084] Preferably, in the step S4, when the improved Adaboost algorithm is used to process the data feature subset, the source IP address reputation and destination IP address security parameters of network data are combined to score the credibility of each feature vector in the strengthened feature vector set, and the scoring formula is:

[0085] S v =w1×R s +w2×S d

[0086] Wherein, S v is the credibility score of the feature vector; R s is the source IP address reputation parameter; S d is the destination IP address security parameter; w1 and w2 are weight coefficients set according to network security policy.

[0087] Specifically, in the process of improving the Transformer algorithm to deal with feature sequences, a feature vector importance scoring mechanism based on network asset value and attack path depth is introduced. Network asset value reflects the importance of assets affected by threats, and attack path depth reflects the complexity of threat propagation, both of which determine the importance of feature vectors. By linearly combining these two parameters, the feature vectors are scored, and the higher the score, the more attention the feature vector will receive in subsequent processing. In implementation, first determine the asset value evaluation standard and attack path depth calculation method, collect relevant parameters and calculate the importance score for each threat scenario corresponding to the feature vector. According to the scoring results, the feature vectors are screened and weighted, highlighting key features and suppressing secondary information, so that the algorithm can focus more on features that have a significant impact on network security, improving the accuracy and efficiency of threat analysis.

[0088] Preferably, in step S5, the anomaly detection analysis adopts a density anomaly detection method based on normal traffic pattern parameters and data transmission pattern parameters of network data. During the detection process, the anomaly degree formula is defined as:

[0089]

[0090] where D x is the anomaly degree of feature vector x; d xy is the distance between feature vector x and feature vector y, calculated by the Manhattan distance of network data feature vectors; ∈ is a pre-set distance threshold; χ is an indicator function, which is 1 when d xy <∈, otherwise 0.

[0091] Specifically, the clustering analysis of step S5 adopts a density-based algorithm, combined with the propagation speed and impact range parameters of network security threats, to identify threat information clusters. Threats with fast propagation speed and wide impact range will form areas with higher density, and the density is determined by calculating the number of other vectors within the neighborhood of the feature vector that satisfy the distance threshold. When the density reaches the threshold, a cluster is formed. This method can effectively discover threat groups with different propagation characteristics and facilitate classification processing. In implementation, a suitable distance threshold is set, the propagation speed and impact range data of threats are collected, and the feature vector density is calculated. According to the density distribution, clusters are divided and numbered. By adjusting the threshold and parameter weights, the clustering results are optimized to ensure that similar threats are accurately classified into one category, providing clear classification basis for subsequent threat analysis and response.

[0092] Preferably, in step S6, when performing data repair or data reconstruction operations, the integrity check value and timeliness parameters of network data are introduced to construct an operation priority model formula:

[0093]

[0094] Among them, P o is the operation priority; C i is the integrity check value of the network data; T e is the timeliness parameter of network data; w3 and w4 are weight coefficients set according to the network data maintenance requirements.

[0095] Specifically, when establishing the associated index in step S6, the index association strength model introduced optimizes the index structure through the parameters of tracing difficulty and repair time. Threat information with high tracing difficulty and long repair time has a stronger association strength with network nodes and timestamps, and occupies a more important position in the index. This design enables the system to quickly locate and handle complex threats and improve response efficiency. During implementation, first formulate the evaluation criteria for tracing difficulty and repair time, and for each threat information cluster, collect relevant parameters and calculate the association strength. Build an index based on the strength results, and closely associate the threat information with the corresponding network nodes and timestamps. Maintain the validity of the index by regularly updating parameters and recalculating the strength, ensuring that key information can be quickly retrieved when a network security incident occurs, supporting rapid decision-making and response.

[0096] Preferably, in step S2, when using the optimized long short-term memory network to extract time series features, differentiated gating mechanism parameters are designed for different types of network data. The parameters are dynamically updated according to the real-time traffic trend and data transmission direction parameters of the data. The update formula is:

[0097]

[0098] Among them, G p is the gating mechanism parameter after the pth update; G p-1 is the gating mechanism parameter of the last time; η is the learning rate; F t is the real-time traffic of current data; F max is the preset maximum flow rate; D d D is the current data transmission direction parameter; max is the preset maximum transmission direction parameter value; ΔG is the parameter update step size.

[0099] Specifically, in step S2, differentiated feature extraction for different types of network data is achieved by dynamically updating the weight matrix. The weight matrix is ​​adjusted according to the threat level and attack frequency parameters of the node. Nodes with high threat levels and high attack frequencies have greater feature extraction weights. This mechanism enables the model to focus on high-risk nodes in a targeted manner based on the real-time threat situation, thereby improving the accuracy of feature extraction. During implementation, the node status is monitored in real time to obtain threat level and attack frequency data. The current parameters are compared with the preset maximum values, and the weight matrix is ​​iteratively updated in combination with the learning rate and update step size. During the operation of the graph neural network, the updated weight matrix is ​​used to extract features from different types of nodes to ensure that the model can adapt to threat changes in a timely manner, highlight key node features, and provide more valuable information for subsequent analysis.

[0100] Preferably, in step S4, when the improved Adaboost algorithm is used to process the data feature subset, the classification boundary of the weak classifier is dynamically adjusted in combination with the protocol type diversity and port usage frequency parameters of the network data, and the adjustment formula is:

[0101]

[0102] Among them, B m is the classification boundary of the mth weak classifier; B m-1 is the last classification boundary; γ is the adjustment coefficient; N p The number of protocol types for the current data; N pmax The maximum number of preset protocol types; F u The frequency used by the current port; F umax is the preset maximum port usage frequency; ΔB is the classification boundary adjustment step.

[0103] Specifically, in step S4, the Transformer algorithm's multi-head attention mechanism is improved to manage groups, allocating attention heads to different groups based on the threat's lateral spread and vertical penetration parameters. The lateral spread parameter reflects the threat's propagation within the same network layer, while the vertical penetration parameter reflects the threat's ability to cross layers. Dynamic grouping is achieved by calculating the sum of the elements of the parameter vector and taking the modulus of the number of groups. Different groups optimize processing strategies for threats with different propagation patterns, improving the algorithm's adaptability to complex threats. During implementation, data on the threat's lateral spread and vertical penetration is collected in real time, and the parameter vector is calculated and grouped. During algorithm execution, each group independently processes the corresponding type of threat characteristics. By integrating the results of each group, the multi-dimensional characteristics of the threat are fully captured. The grouping strategy is regularly adjusted based on the network security situation to ensure that the algorithm can effectively respond to changing threat scenarios and enhance the system's overall protection capabilities.

[0104] like Figure 2As shown, a network data maintenance system based on big data and artificial intelligence includes:

[0105] A network data multi-dimensional feature space construction unit, used to construct a network data multi-dimensional feature space based on a network operating environment;

[0106] A long short-term memory network feature extraction unit is connected to the network data multidimensional feature space construction unit and is used to use the optimized long short-term memory network to extract time series features from the initial network data feature network structure;

[0107] a data feature interval division unit, connected to the long short-term memory network feature extraction unit, for dividing the preliminary feature set into intervals according to the data flow fluctuation amplitude;

[0108] an improved Adaboost algorithm processing unit, connected to the data feature interval division unit, for processing the data feature subset using the improved Adaboost algorithm;

[0109] an anomaly detection and analysis unit, connected to the improved Adaboost algorithm processing unit, for performing anomaly detection and analysis on the enhanced feature vector set;

[0110] a data maintenance operation execution unit, connected to the anomaly detection and analysis unit, configured to perform a data repair or data reconstruction operation on the network data corresponding to the anomaly feature vector;

[0111] The data storage index building unit is connected to the data maintenance operation execution unit and is used to store the maintained network data in a preset network data repository and to establish an associated index between the maintained network data and the original data identifier and the maintenance operation record.

[0112] This invention addresses the problem of traditional technologies' weak ability to process time series features of network data. By constructing a multidimensional feature space for network data, it converts multidimensional attributes such as data types and transmission protocols into structured nodes and connection edges, laying the foundation for in-depth analysis. On top of this, the optimized long-short-term memory network plays a core role. Its multi-layer memory unit structure and gating mechanism can dynamically update the memory unit state based on parameters such as the timestamp interval of network data and the frequency of data changes, thereby deeply mining the long-term and short-term dependencies of data in the time dimension. This mechanism enables the system to accurately capture the temporal evolution of network data traffic changes, transmission delays and other features. Compared with traditional methods, it greatly improves the perception and prediction capabilities of dynamic changes in data, and effectively solves the problem of processing data time series features.

[0113] Improving the Adaboost algorithm is key to addressing the shortcomings of traditional technologies in extracting and integrating complex network data features. This algorithm redesigns the weak classifier weight update strategy, combining core network data parameters such as transmission delay, packet loss rate, and error rate. It dynamically adjusts the weak classifier weights through weighted correction coefficients and optimizes classification boundaries based on parameters such as protocol type diversity and port usage frequency. At the same time, the system comprehensively considers multi-dimensional parameters such as source IP address reputation and integrity check values ​​in the data processing process, utilizing a series of steps such as data traffic fluctuation interval division, credibility scoring, anomaly detection, and operation priority determination. This enables the system to fully explore the potential connections between network data and accurately identify key features, significantly improving the accuracy of anomaly detection and the effectiveness of data repair, and achieving efficient integration and in-depth maintenance of complex network data.

[0114] Furthermore, the seven units of this invention work closely together, forming a complete and smooth data processing chain, from constructing the multidimensional feature space of network data to storing and indexing the maintained data. Efficient data interaction and collaborative work between these units ensure the coherence and systematic nature of the entire maintenance process, further enhancing the system's ability to manage network data throughout its lifecycle and providing a solid foundation for its stable and reliable operation.

[0115] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "disposed," "installed," "connected," "connected," and "fixed" should be understood in a broad sense. For example, they may refer to fixed connections, detachable connections, or integral connections; they may refer to mechanical connections or electrical connections; they may refer to direct connections or indirect connections through an intermediate medium; and they may refer to internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.

[0116] Although embodiments of the present invention have been shown and described, it will be understood by those skilled in the art that various equivalent changes, modifications, substitutions and variations may be made to these embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A network data maintenance method based on big data and artificial intelligence, characterized in that: The following steps are involved: Step S1: Constructing a multidimensional feature space of network data based on the network operating environment, classifying and identifying network data according to data type, transmission protocol, and flow size attributes, establishing corresponding data nodes in the multidimensional feature space, and constructing connection edges between nodes based on data interaction relationships and time series relationships to form an initial network data feature network structure; Step S2: Using an optimized long short-term memory network to extract time series features from the initial network data feature network structure, by designing a multi-layer memory unit structure and combining a gating mechanism to perform deep learning on the long-term and short-term dependencies of the network data in the time dimension, a preliminary feature set including time series feature vectors is obtained; Step S3: Divide the preliminary feature set into intervals according to the data flow fluctuation range to form multiple data feature subsets, each data feature subset corresponding to a different data flow fluctuation interval; Step S4: Using an improved Adaboost algorithm to process the data feature subset, by redesigning the weight update strategy of the weak classifier and combining the transmission delay, packet loss rate, and error rate parameters of the network data, the key features in the data feature subset are weighted and integrated to obtain an enhanced feature vector set; Step S5: performing anomaly detection analysis on the enhanced feature vector set, and identifying abnormal feature vectors that deviate from the normal distribution based on the historical normal feature distribution properties of the network data; Step S6: performing data repair or data reconstruction operations on the network data corresponding to the abnormal feature vector according to the priority and importance parameters of the network data; Step S7: storing the maintained network data in a preset network data repository, and establishing an association index between the maintained network data, the original data identifier, and the maintenance operation record.

2. The network data maintenance method based on big data and artificial intelligence according to claim 1 is characterized in that: The optimized long short-term memory network is designed to update the memory unit state model formula as follows: Among them, C t Represents the state of the memory unit at time t; C t-1 represents the state of the memory unit at time t-1; f t is the output value of the forget gate at time t, which is determined by the timestamp interval of the network data and the data change frequency parameter, and is used to control the degree of retention of the memory unit state at the previous moment; i t is the output value of the input gate at time t, which is calculated based on the real-time traffic of network data and the data type change parameters, and is used to control the inflow of input information at the current moment; is the candidate memory cell state calculated at the current moment; ⊙ represents the multiplication of corresponding matrix elements.

3. The network data maintenance method based on big data and artificial intelligence according to claim 1 is characterized in that: The improved Adaboost algorithm designs a weak classifier weight update model formula as follows: Among them, α m is the weight of the mth weak classifier; e m is the classification error of the mth weak classifier; β m It is a weighted correction coefficient, whose value is dynamically adjusted by the transmission bandwidth utilization rate of network data and the number of data retransmission parameters. It is used to correct the weight of the weak classifier to make the algorithm more adaptable to the characteristics of network data.

4. The network data maintenance method based on big data and artificial intelligence according to claim 1 is characterized in that: In step S3, the division of the data flow fluctuation interval is dynamically adjusted based on the historical flow peak and valley parameters of the network data and the standard deviation parameter of the flow change. The adjustment formula is: I n =[μ-k×σ,μ+k×σ] Among them, I n is the nth data flow fluctuation interval; μ is the average value of the historical network data flow; σ is the standard deviation of the historical network data flow; k is the adjustment coefficient, which is set according to the stability parameter of the network data flow change.

5. The network data maintenance method based on big data and artificial intelligence according to claim 1 is characterized in that: In step S4, when the improved Adaboost algorithm is used to process the data feature subset, the credibility of each feature vector in the enhanced feature vector set is scored in combination with the source IP address credibility and the destination IP address security parameter of the network data. The scoring formula is: S v =w1×R s +w2×S d Among them, S v R is the credibility score of the feature vector; s is the source IP address reputation parameter; S d is the destination IP address security parameter; w1 and w2 are weight coefficients set according to the network security policy.

6. The network data maintenance method based on big data and artificial intelligence according to claim 1 is characterized in that: In step S5, the anomaly detection analysis adopts a density anomaly detection method based on the normal traffic pattern parameters and data transmission pattern parameters of the network data. During the detection process, the anomaly degree formula is defined as: Among them, D x is the abnormality of the eigenvector x; d xy is the distance between feature vector x and feature vector y, calculated by the Manhattan distance of network data feature vectors; ∈ is the preset distance threshold; χ is the indicator function, when d xy When <∈, the value of χ is 1, otherwise it is 0.

7. The network data maintenance method based on big data and artificial intelligence according to claim 1 is characterized in that: In step S6, when performing data repair or data reconstruction operations, the integrity check value and timeliness parameters of the network data are introduced to construct an operation priority model formula: Among them, P o is the operation priority; C i is the integrity check value of the network data; T e is the timeliness parameter of network data; w3 and w4 are weight coefficients set according to the network data maintenance requirements.

8. The network data maintenance method based on big data and artificial intelligence according to claim 1 is characterized in that: In step S2, when using the optimized long short-term memory network to extract time series features, differentiated gating mechanism parameters are designed for different types of network data. The parameters are dynamically updated according to the real-time traffic trend and data transmission direction parameters of the data. The update formula is: Among them, G p is the gating mechanism parameter after the pth update; G p-1 is the gating mechanism parameter of the last time; η is the learning rate; F t is the real-time traffic of current data; F max is the preset maximum flow rate; D d D is the current data transmission direction parameter; max is the preset maximum transmission direction parameter value; ΔG is the parameter update step size.

9. The network data maintenance method based on big data and artificial intelligence according to claim 1 is characterized in that: In step S4, when the improved Adaboost algorithm is used to process the data feature subset, the classification boundary of the weak classifier is dynamically adjusted in combination with the protocol type diversity and port usage frequency parameters of the network data. The adjustment formula is: Among them, B m is the classification boundary of the mth weak classifier; B m-1 is the last classification boundary; γ is the adjustment coefficient; N p The number of protocol types for the current data; N pmax The maximum number of preset protocol types; F u The frequency used by the current port; F umax is the preset maximum port usage frequency; ΔB is the classification boundary adjustment step.

10. The network data maintenance system based on big data and artificial intelligence is characterized by: The system includes: A network data multi-dimensional feature space construction unit, used to construct a network data multi-dimensional feature space based on a network operating environment; A long short-term memory network feature extraction unit is connected to the network data multidimensional feature space construction unit and is used to use the optimized long short-term memory network to extract time series features from the initial network data feature network structure; a data feature interval division unit, connected to the long short-term memory network feature extraction unit, for dividing the preliminary feature set into intervals according to the data flow fluctuation amplitude; an improved Adaboost algorithm processing unit, connected to the data feature interval division unit, for processing the data feature subset using the improved Adaboost algorithm; an anomaly detection and analysis unit, connected to the improved Adaboost algorithm processing unit, for performing anomaly detection and analysis on the enhanced feature vector set; a data maintenance operation execution unit, connected to the anomaly detection and analysis unit, configured to perform a data repair or data reconstruction operation on the network data corresponding to the anomaly feature vector; The data storage index building unit is connected to the data maintenance operation execution unit and is used to store the maintained network data in a preset network data repository and to establish an associated index between the maintained network data and the original data identifier and the maintenance operation record.

Images