Abnormal data analysis method and system based on optical fiber communication

Abstract

The application relates to an abnormal data analysis method and system based on optical fiber communication, which comprises the following steps: collecting physical layer parameters, digital features and link state data in real time through a receiver to obtain a complete data set; calculating a reference digital spectrum under a normal state based on the complete data set, generating a digital residual spectrum through the difference between a real-time digital spectrum and the reference digital spectrum, and performing dimension reduction fusion on the digital residual spectrum and the physical layer parameters to obtain a low-dimensional feature vector; when a reconstruction error exceeds a dynamic threshold value, determining that the data is abnormal, and then classifying the abnormal data through a support vector machine; based on the classification of the abnormal data, calculating the intensity variance of the low-dimensional feature vector through a time window, recursively dividing the link data into a normal subset or an abnormal subset through a decision tree algorithm, and locating an abnormal node position; based on the abnormal node position, if there is abnormal data classification with a proportion below a preset threshold value, generating synthetic data of the digital residual spectrum and optical power through a synthetic minority over-sampling technique.

Description

Technical Field

[0001] This application relates to the field of optical fiber communication technology, and in particular to an anomaly data analysis method and system based on optical fiber communication. Background Technology

[0002] Optical fiber communication refers to a communication method that uses optical fibers as the transmission medium to transmit information through optical signals. It focuses on the signal transmission technology of the physical layer, such as optical modulation, optical fiber transmission characteristics, and optical amplification. It also involves the performance optimization of optical transmitting and receiving devices. It has significant advantages such as large transmission bandwidth, low loss, resistance to electromagnetic interference, and strong confidentiality. It is the core pillar technology of modern communication networks and is widely used in backbone networks, metropolitan area networks, data center interconnection, and mobile communication backhaul scenarios.

[0003] As the scale of fiber optic communication networks expands, communication failures and network attacks lead to an exponential increase in the amount of abnormal data. In fiber optic communication systems, abnormal data, such as phase jitter, power loss, and data loss, can degrade signal transmission quality and even cause link failures.

[0004] In real-world scenarios, abnormal data is often scarce, while traditional supervised learning methods rely on a large number of labeled abnormal samples to achieve effective training. As a result, the model struggles to fully learn abnormal features, limiting the accuracy of anomaly analysis.

[0005] Based on this, this application provides an anomaly data analysis method and system based on optical fiber communication. Summary of the Invention

[0006] To address the problem that outlier data is often scarce, and that traditional supervised learning methods rely on a large number of labeled outlier samples for effective training, making it difficult for models to fully learn outlier features and thus limiting the accuracy of outlier analysis, this application provides an outlier data analysis method and system based on optical fiber communication.

[0007] Firstly, this application provides an anomaly data analysis method based on optical fiber communication, employing the following technical solution: including: The receiver collects physical layer parameters, digital features, and link status data in real time. The collected data is then subjected to noise suppression and missing value completion to obtain a complete dataset. The baseline digital spectrum under normal conditions is calculated based on the complete dataset. A digital residual spectrum is generated by the difference between the real-time digital spectrum and the baseline digital spectrum. The digital residual spectrum and physical layer parameters are then dimensionality-reduced and fused to obtain a low-dimensional feature vector. An autoencoder is used to perform unsupervised anomaly detection on the low-dimensional feature vector. When the reconstruction error exceeds a dynamic threshold, it is determined to be abnormal data. Then, a support vector machine is used to classify the abnormal data. Based on abnormal data classification, the intensity variance of the low-dimensional feature vector is calculated through a preset time window, the intensity variance change curve is plotted along the transmission link, and the link data is recursively divided into normal subsets or abnormal subsets using a decision tree algorithm to locate the abnormal node positions. Based on the location of the abnormal node, if the proportion of the abnormal data classification is lower than a preset threshold, synthetic minority class oversampling technology is used to generate the synthetic data of the digital residual spectrum and optical power.

[0008] Preferably, the step of acquiring physical layer parameters, digital features, and link status data in real time via a receiver, and performing noise suppression and missing value completion on the acquired data to obtain a complete dataset includes: The receiver's digital signal processing module synchronously acquires the physical layer parameters, digital features, and link status data. The physical layer parameters include optical power, center wavelength, polarization state, and phase noise. The digital features include bit error rate, constellation diagram distortion, signal amplitude variance, and forward error correction overhead. The link status data includes node connection status, transmission rate, and optical switch switching records. The acquired raw data is decomposed into high-frequency noise components and low-frequency signal components by wavelet transform, and the high-frequency components are processed by a soft threshold function. The denoised signal is dynamically corrected by Kalman filtering to obtain denoised data; If the data missing rate is less than the threshold in a single time period, then time series linear interpolation is used to fill the missing data. If the data missing rate is not less than the threshold or the continuous missing duration exceeds the preset time within a single time period, the LSTM prediction model trained with historical data of the same type of link is called, and the physical layer parameters of the current link are combined to generate the complete data. The completed data is aligned by timestamp, normalized, and integrated into a structured data matrix to form the complete dataset. The row dimension of the structured data matrix is ​​the time series sampling points, and the column dimension is the feature combination of the physical layer parameters, the digital features, and the link status data.

[0009] Preferably, the step of calculating the baseline digital spectrum under normal conditions based on the complete dataset, generating a digital residual spectrum by the difference between the real-time digital spectrum and the baseline digital spectrum, and performing dimensionality reduction and fusion of the digital residual spectrum and physical layer parameters to obtain a low-dimensional feature vector includes: Extract normal state data segments without anomaly labels from the complete dataset, perform power spectrum estimation using the Welch periodogram method, and calculate the digital spectrum mean of the normal state data segments as the baseline digital spectrum. Synchronous power spectrum estimation is performed on the real-time acquired physical layer parameters and digital features to obtain the real-time digital spectrum; The amplitude difference between the real-time digital spectrum and the reference digital spectrum is calculated to form a digital residual spectrum, and the frequency axis of the digital residual spectrum is aligned with the reference digital spectrum. The amplitude value sequence of the digital residual spectrum and the physical layer parameters are concatenated by timestamp to form a high-dimensional feature vector; The high-dimensional feature vector is reduced in dimensionality using a distributed neighborhood embedding algorithm to obtain the low-dimensional feature vector.

[0010] Preferably, the step of using an autoencoder to perform unsupervised anomaly detection on the low-dimensional feature vector, determining the data as anomalous when the reconstruction error exceeds a dynamic threshold, and then classifying the anomalous data using a support vector machine includes: A stacked autoencoder model containing an input layer, a hidden layer, and an output layer is constructed and trained using the low-dimensional feature vectors of the normal state in the complete dataset. The mean squared error is used as the loss function, and the model is iteratively optimized until the loss converges to obtain the autoencoder. The dynamic threshold is obtained by fitting the error distribution curve based on the kernel density and dynamically adjusting the threshold according to the preset initial threshold and the time decay factor. The low-dimensional feature vector is input into the autoencoder to calculate the reconstruction error. When the reconstruction error is greater than the dynamic threshold, the current data is determined to be abnormal data. A support vector machine model is constructed using radial basis function kernel function. The abnormal data and the corresponding physical layer parameters are input, and the abnormal data classification is output. The abnormal data classification includes non-lockout anomaly, lockout-relocking anomaly, or device anomaly.

[0011] Preferably, the step of classifying abnormal data, calculating the intensity variance of the low-dimensional feature vector through a preset time window, plotting the intensity variance change curve along the transmission link, recursively dividing the link data into normal subsets or abnormal subsets using a decision tree algorithm, and locating the abnormal node positions, includes: For the low-dimensional feature vector corresponding to the abnormal data classification, a sliding time window is set, and the strength variance of each dimension of the low-dimensional feature vector within the time window is calculated. Obtain the physical topology information of the transmission link, map the strength variance to the link transmission path according to the timestamp, and plot the strength variance as a function of node position along the link from the sending end to the receiving end. The horizontal axis of the strength curve is the node number, and the vertical axis is the strength variance. The intensity curve is input into a preset decision tree model. If the output is an abnormal subset, the position of the first node in the intensity curve that satisfies the condition that "the intensity variance exceeds the threshold and the parameter fluctuation matches the abnormal data classification" is extracted. Combined with the node's physical address, the abnormal node position is located.

[0012] Preferably, based on the location of the abnormal node, if the proportion of abnormal data classification is lower than a preset threshold, the synthetic minority class oversampling technique is used to generate the synthetic data of the digital residual spectrum and optical power, including: For the historical data within a preset period corresponding to the abnormal node location, the number of samples for each type of abnormal data category is counted, and the percentage of each type of abnormal sample in the total number of abnormal samples is calculated to obtain the category ratio. If the proportion of the classification is lower than the preset threshold, the corresponding anomaly is determined to be a minority anomaly, and the synthetic minority oversampling technique is used to generate the synthetic data of the digital residual spectrum and optical power.

[0013] Preferably, the step of generating the synthesized digital residual spectrum and optical power data using synthetic minority class oversampling technology includes: Extract the original data corresponding to the minority anomalies, as well as the digital residual spectrum and optical power time series data of the anomaly node locations; The synthetic minority class oversampling technique is used, with the feature vector of the digital residual spectrum and the temporal characteristics of the optical power as inputs, and the synthetic samples are generated by random interpolation in the feature space of the sample data corresponding to the minority class anomaly in the original data and the feature space of the nearest neighbor samples according to the preset number of nearest neighbor samples and the sampling ratio. Calculate the feature similarity between the synthetic sample and the sample data corresponding to minority class anomalies in the original data, and define the synthetic sample with the feature similarity exceeding a preset value as the synthetic data.

[0014] Secondly, this application discloses an anomaly data analysis device based on optical fiber communication, which adopts the following technical solution, including: The multi-source acquisition module is used to acquire physical layer parameters, digital features and link status data in real time through the receiver, and to perform noise suppression and missing value completion on the acquired data to obtain a complete dataset. The feature vector module is used to calculate the baseline digital spectrum under normal conditions based on the complete dataset, generate a digital residual spectrum by the difference between the real-time digital spectrum and the baseline digital spectrum, and perform dimensionality reduction and fusion on the digital residual spectrum and physical layer parameters to obtain a low-dimensional feature vector. The anomaly classification module is used to perform unsupervised anomaly detection on the low-dimensional feature vector using an autoencoder. When the reconstruction error exceeds a dynamic threshold, it is determined to be abnormal data, and then the abnormal data is classified by a support vector machine. The node localization module is used to classify abnormal data, calculate the intensity variance of the low-dimensional feature vector through a preset time window, plot the intensity variance change curve along the transmission link, and recursively divide the link data into normal subsets or abnormal subsets using a decision tree algorithm to locate the abnormal node positions. The sample synthesis module is used to generate the synthesized data of the digital residual spectrum and optical power based on the location of the abnormal node and if the proportion of the abnormal data classification is lower than a preset threshold, by using a synthetic minority class oversampling technique.

[0015] Thirdly, this application also provides a control device, the device comprising: It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed, such as the above-described fiber optic communication-based anomaly data analysis method.

[0016] Fourthly, this application also provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed as described above regarding the abnormal data analysis method based on optical fiber communication.

[0017] In summary, this application addresses the issues of noise interference and missing data in fiber optic links through acquisition and dynamic repair technologies, providing a high-quality data foundation for subsequent analysis. Next, it introduces digital residual spectrum analysis to capture weak anomalous signals by comparing the baseline and real-time spectra, and combines this with dimensionality reduction techniques to effectively compress high-dimensional features. Then, it employs an anomaly recognition architecture that combines unsupervised and supervised learning, ensuring both the detection capability of unknown anomalies and accurate classification of anomaly types. Finally, it maps abstract data anomalies to physical node faults using intensity variance curves and decision tree topology localization algorithms. To address model bias caused by insufficient anomalous samples in communication systems, it uses feature space interpolation synthesis technology to dynamically optimize sample distribution, thus solving the problem of scarce anomalous samples in real-world scenarios. This facilitates improved training performance of the analysis model, thereby enhancing the accuracy of model analysis. Attached Figure Description

[0018] Figure 1 This is a flowchart illustrating an anomaly data analysis method based on optical fiber communication.

[0019] Figure 2 This is a structural block diagram of an anomaly data analysis device based on optical fiber communication. Detailed Implementation

[0020] The following combination Figure 1 - Figure 2 This application will be described in further detail.

[0021] Existing fiber optic link anomaly detection methods largely rely on fixed threshold alarms, which suffer from insufficient ability to detect weak anomalies, low fault location accuracy, and missed detections due to imbalanced anomaly sample categories. To address these shortcomings, this invention provides a comprehensive analysis solution. Through multi-dimensional data acquisition and dynamic repair, digital residual spectrum feature enhancement, deep learning anomaly detection and classification, decision tree topology localization, and minority class sample synthesis techniques, it achieves real-time and accurate identification of fiber optic link anomalies, fault node location, and sample distribution optimization, significantly improving the reliability and self-healing capabilities of communication networks.

[0022] Reference Figure 1 The embodiments of this application include at least steps S10 to S50.

[0023] S10 collects physical layer parameters, digital features, and link status data in real time through the receiver, performs noise suppression and missing value completion on the collected data, and obtains a complete dataset.

[0024] S20 calculates the baseline digital spectrum under normal conditions based on the complete dataset, generates a digital residual spectrum by the difference between the real-time digital spectrum and the baseline digital spectrum, and performs dimensionality reduction and fusion on the digital residual spectrum and physical layer parameters to obtain a low-dimensional feature vector.

[0025] S30 uses an autoencoder to perform unsupervised anomaly detection on low-dimensional feature vectors. When the reconstruction error exceeds a dynamic threshold, it is determined to be abnormal data, and then the abnormal data is classified by a support vector machine.

[0026] S40, based on abnormal data classification, calculates the intensity variance of low-dimensional feature vectors through a preset time window, plots the intensity variance change curve along the transmission link, and uses a decision tree algorithm to recursively divide the link data into normal subsets or abnormal subsets to locate the abnormal node positions.

[0027] S50: Based on the location of abnormal nodes, if the proportion of abnormal data classification is lower than a preset threshold, synthetic minority class oversampling technology is used to generate synthetic data of digital residual spectrum and optical power.

[0028] Specifically, through multi-dimensional data acquisition and preprocessing, noise is suppressed and missing values ​​are filled in, providing a high-quality data foundation for subsequent analysis; a baseline digital spectrum and residual spectrum are constructed, and low-dimensional features are extracted by combining dimensionality reduction and fusion to enhance the ability to represent abnormal signals; an autoencoder is used to achieve unsupervised anomaly detection, and SVM is used to accurately classify anomaly types, improving detection efficiency and accuracy; anomaly nodes are located by intensity variance curves and decision tree algorithms, and for minority anomaly samples, oversampling technology is used to generate synthetic data to optimize the distribution of training samples and reduce the occurrence of low-proportion anomaly misses, thereby solving the problem of scarce anomaly samples in real-world scenarios, which facilitates the training effect of the analysis model and improves the accuracy of model analysis.

[0029] In some embodiments, step S10 specifically includes the following steps: synchronously acquiring physical layer parameters, digital features, and link status data through the receiver's digital signal processing module. Physical layer parameters include optical power, center wavelength, polarization state, and phase noise; digital features include bit error rate, constellation diagram distortion, signal amplitude variance, and forward error correction overhead; and link status data includes node connection status, transmission rate, and optical switch switching records. The acquired raw data is decomposed into high-frequency noise components and low-frequency signal components using wavelet transform, and the high-frequency components are processed using a soft thresholding function. The denoised signal is then processed using Kalman filtering. Dynamic error correction is performed to obtain denoised data. If the data missing rate in a single time period is less than the threshold, time series linear interpolation is used to fill the missing data. If the data missing rate in a single time period is not less than the threshold or the continuous missing duration exceeds the preset time, the LSTM prediction model trained on historical data of the same link is called, and the physical layer parameters of the current link are combined to generate complete data. The complete data is aligned by timestamp, normalized and integrated into a structured data matrix to form a complete dataset. The row dimension of the structured data matrix is ​​the time series sampling points, and the column dimension is the feature combination of physical layer parameters, digital features and link status data.

[0030] Specifically, a holographic data base for the fiber optic communication system is constructed through multi-dimensional synchronous acquisition, encompassing physical layer parameters such as optical power and wavelength, digital features such as bit error rate and constellation diagram distortion, and link data such as node status, enabling comprehensive monitoring. Wavelet transform soft thresholding is used to separate noise and signal components, combined with Kalman filtering to dynamically correct errors, significantly improving the data signal-to-noise ratio. Interpolation or LSTM prediction completion strategies are intelligently selected for different missing scenarios to ensure data continuity and integrity. Finally, through time alignment, normalization, and structured integration, a time-series feature matrix is ​​formed, providing high-quality and standardized data input for subsequent residual spectrum analysis and anomaly detection, effectively solving the problems of noise interference, missing data, and format heterogeneity in fiber optic link data.

[0031] In some embodiments, step S20 specifically includes the following steps: extracting normal state data segments without anomaly markers from the complete dataset, performing power spectrum estimation using the Welch periodogram method, and calculating the digital spectrum mean of the normal state data segments as the reference digital spectrum; performing synchronous power spectrum estimation on the real-time acquired physical layer parameters and digital features to obtain the real-time digital spectrum; calculating the amplitude difference between the real-time digital spectrum and the reference digital spectrum to form a digital residual spectrum, with the frequency axis of the digital residual spectrum aligned with the reference digital spectrum; concatenating the amplitude value sequence of the digital residual spectrum with the physical layer parameters according to the timestamp to form a high-dimensional feature vector; and using a distributed neighborhood embedding algorithm to reduce the dimensionality of the high-dimensional feature vector to obtain a low-dimensional feature vector.

[0032] Specifically, by extracting normal state data segments and establishing a baseline digital spectrum using the Welch periodogram method, a reference benchmark is provided for anomaly detection. The difference between the real-time digital spectrum and the baseline spectrum is calculated to form a digital residual spectrum, effectively amplifying the features of weak anomalous signals. The residual spectrum amplitude sequence is concatenated with physical layer parameters to form a high-dimensional vector, fusing multimodal features. A distributed neighborhood embedding algorithm is used to reduce dimensionality, reducing data dimensionality while preserving the correlation of key features, thereby reducing subsequent computational complexity and improving model efficiency. This process realizes the transformation from raw data to highly recognizable low-dimensional features, laying the foundation for accurate anomaly detection and solving the problems of feature redundancy and difficulty in capturing weak anomalies in traditional methods.

[0033] In some embodiments, step S30 specifically includes the following steps: constructing a stacked autoencoder model containing an input layer, a hidden layer, and an output layer, and training it using low-dimensional feature vectors of normal states in the complete dataset, using mean squared error as the loss function, iteratively optimizing until the loss converges to obtain the autoencoder; fitting the error distribution curve based on kernel density, and dynamically adjusting the threshold according to a preset initial threshold and a time decay factor to obtain a dynamic threshold; inputting the low-dimensional feature vectors into the autoencoder, calculating the reconstruction error, and determining the current data as abnormal data when the reconstruction error is greater than the dynamic threshold; constructing a support vector machine model using radial basis function kernel functions, inputting abnormal data and corresponding physical layer parameters, and outputting an abnormal data classification, which includes non-lockout anomalies, lockout-relocking anomalies, or device anomalies.

[0034] Specifically, a stacked autoencoder is constructed and trained using low-dimensional feature vectors of normal states. The model is optimized using mean squared error as the loss function, enabling it to accurately reconstruct normal data patterns. Kernel density is used to fit the error distribution, and the threshold is dynamically adjusted by combining a time decay factor to improve the environmental adaptability of anomaly detection. Unsupervised anomaly detection is achieved by comparing the autoencoder reconstruction error with the dynamic threshold, overcoming the limitations of traditional fixed thresholds. An SVM model with radial basis function kernel function is used to classify abnormal data and physical layer parameters, outputting non-lockout anomalies, lockout-relocking anomalies, or device anomaly types. This achieves a leap from anomaly detection to accurate classification, solves the problem of ambiguous anomaly types in complex scenarios, and provides a classification basis for subsequent fault localization.

[0035] In some embodiments, step S40 specifically includes the following steps: for the low-dimensional feature vector corresponding to the abnormal data classification, set a sliding time window and calculate the intensity variance of each dimension of the low-dimensional feature vector within the time window; obtain the physical topology information of the transmission link, map the intensity variance to the link transmission path according to the timestamp, and draw an intensity curve of intensity variance changing with node position along the link from the sending end to the receiving end, where the horizontal axis of the intensity curve is the node number and the vertical axis is the intensity variance; input the intensity curve into a preset decision tree model, and if the output is an abnormal subset, extract the first node position in the intensity curve that satisfies "intensity variance exceeds the threshold and parameter fluctuation matches the abnormal data classification", and locate the abnormal node position by combining the node's physical address.

[0036] Specifically, a sliding time window is set for anomaly classification data to calculate the strength variance of low-dimensional feature vectors and capture abnormal fluctuation characteristics. The variance is mapped to the transmission path using the link's physical topology, and a node number-strength variance curve is plotted to visualize the anomaly. Data is recursively partitioned using a decision tree model, and the location of the first node whose variance exceeds a threshold and whose fluctuation matches the anomaly type is extracted. This location is then precisely determined by combining the node's physical address. This process transforms abstract feature variance into physical node coordinates, solving the problem of fuzzy traditional link fault location, enabling rapid source tracing of anomalies, providing precise targeting for subsequent maintenance, and improving the efficiency of fault diagnosis in fiber optic communication systems.

[0037] In some embodiments, step S50 specifically includes the following steps: for historical data within a preset period corresponding to the location of the abnormal node, count the number of samples for each type of abnormal data category, calculate the percentage of each type of abnormal sample in the total number of abnormal samples, and obtain the classification ratio; if there is a classification ratio lower than a preset threshold, the corresponding abnormality is determined to be a minority class abnormality, and synthetic minority class oversampling technology is used to generate synthetic data of digital residual spectrum and optical power.

[0038] Furthermore, the original data corresponding to minority class anomalies, as well as the digital residual spectrum and optical power time-series data of the anomaly node locations, are extracted. A synthetic minority class oversampling technique is adopted, which uses the feature vector of the digital residual spectrum and the temporal features of the optical power as inputs. Based on the preset number of nearest neighbor samples and the sampling ratio, synthetic samples are randomly interpolated in the feature space of the sample data corresponding to minority class anomalies in the original data and the sample data of nearest neighbor samples to generate synthetic samples. The feature similarity between the synthetic samples and the sample data corresponding to minority class anomalies in the original data is calculated, and the synthetic samples with feature similarity exceeding the preset value are defined as synthetic data.

[0039] Specifically, by statistically analyzing the classification proportions of various anomalies in historical data of abnormal nodes, minority anomalies with proportions below a threshold are identified, and supplementary data is generated using Synthetic Minority Oversampling Technique (SMOTE). Specifically, the original data, digital residual spectrum, and optical power temporal features of minority anomalies are extracted, and synthetic samples are generated by random interpolation in the feature space based on the number of nearest neighbor samples and the sampling fold. High-quality synthetic data is then selected based on feature similarity. This process solves the problem of anomaly sample class imbalance, enhances the feature expression of minority anomalies, improves the model's ability to identify rare anomaly types, avoids detection bias caused by sample scarcity, and provides balanced data support for model iteration and optimization.

[0040] In some embodiments, considering the optimization of anomaly classification performance by fusing extreme gradient boosting models, the corresponding processing steps are as follows: Construct an extreme gradient boosting classification model containing an input layer, a hidden layer, and an output layer; use low-dimensional feature vectors and corresponding anomaly data classification labels as training samples; optimize the model hyperparameters through grid search and five-fold cross-validation, including learning rate, tree depth, number of leaf nodes, and regularization coefficients λ and γ; use Cox proportional hazards regression with L1 penalty as the loss function; calculate the first and second derivatives of the loss function to optimize the decision tree structure; train using normal state data and supplementary samples generated by synthetic minority class oversampling technology; iteratively optimize until the loss converges; input the real-time low-dimensional feature vectors into the trained model; and output the classification results of non-lock-out anomalies, lock-out-relock anomalies, or device anomalies.

[0041] The implementation principle of an anomaly data analysis method based on optical fiber communication in this application embodiment is as follows: First, data acquisition and dynamic repair techniques are used to solve the problems of noise interference and missing data in optical fiber links, providing a high-quality data foundation for subsequent analysis. Then, a digital residual spectrum is introduced, and weak anomaly signals are captured through difference analysis between the baseline spectrum and the real-time spectrum. Dimensionality reduction techniques are combined to achieve effective compression of high-dimensional features. Next, an anomaly recognition architecture combining unsupervised and supervised learning is adopted, ensuring both the detection capability of unknown anomalies and accurate classification of anomaly types. Finally, through intensity variance curves and decision tree topology localization algorithms, abstract data anomalies are mapped to physical node faults. Addressing the model bias problem caused by insufficient anomaly samples in the communication system, feature space interpolation synthesis technology is used to dynamically optimize the sample distribution, thereby solving the problem of scarce anomaly samples in real-world scenarios. This facilitates improved training effects of the analysis model, thus enhancing the accuracy of model analysis.

[0042] Figure 1 This is a flowchart illustrating an anomaly data analysis method based on fiber optic communication in one embodiment. It should be understood that, although... Figure 1The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows; unless explicitly stated otherwise, there is no strict order requirement for the execution of these steps, and they can be executed in other orders; and Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0043] Based on the same technical concept, referring to Figure 2 This application also provides an anomaly data analysis device based on optical fiber communication, which adopts the following technical solution: The device includes: The multi-source acquisition module is used to acquire physical layer parameters, digital features and link status data in real time through the receiver, and to perform noise suppression and missing value completion on the acquired data to obtain a complete dataset. The feature vector module is used to calculate the baseline digital spectrum under normal conditions based on the complete dataset. It generates a digital residual spectrum by the difference between the real-time digital spectrum and the baseline digital spectrum, and performs dimensionality reduction and fusion on the digital residual spectrum and physical layer parameters to obtain a low-dimensional feature vector. The anomaly classification module is used to perform unsupervised anomaly detection on low-dimensional feature vectors using an autoencoder. When the reconstruction error exceeds a dynamic threshold, it is determined to be abnormal data, and then the abnormal data is classified by a support vector machine. The node localization module is used to classify abnormal data, calculate the strength variance of low-dimensional feature vectors through a preset time window, plot the strength variance change curve along the transmission link, and recursively divide the link data into normal subsets or abnormal subsets using a decision tree algorithm to locate the abnormal node positions. The sample synthesis module is used to generate synthesized digital residual spectrum and optical power data based on the location of abnormal nodes and when the proportion of abnormal data classification is lower than a preset threshold, by using synthetic minority class oversampling technology.

[0044] In some embodiments, the multi-source acquisition module is specifically used to synchronously acquire physical layer parameters, digital features and link status data through the digital signal processing module of the receiver. The physical layer parameters include optical power, center wavelength, polarization state and phase noise. The digital features include bit error rate, constellation diagram distortion, signal amplitude variance and forward error correction overhead. The link status data includes node connection status, transmission rate and optical switch switching records. The acquired raw data is decomposed into high-frequency noise components and low-frequency signal components by wavelet transform, and the high-frequency components are processed by a soft threshold function. The denoised signal is dynamically corrected by Kalman filtering to obtain denoised data; If the data missing rate is less than the threshold in a single time period, then time series linear interpolation is used to fill the missing data. If the data missing rate is not less than the threshold or the continuous missing duration exceeds the preset time within a single time period, the LSTM prediction model trained with historical data of the same link is called, and the physical layer parameters of the current link are combined to generate the complete data. The completed data is aligned by timestamp, normalized, and integrated into a structured data matrix to form a complete dataset. The row dimension of the structured data matrix consists of time series sampling points, and the column dimension consists of a combination of physical layer parameters, digital features, and link status data features.

[0045] In some embodiments, the feature vector module is specifically used to extract normal state data segments without abnormal labels from the complete dataset, use the Welch periodogram method to estimate the power spectrum, and calculate the digital spectrum mean of the normal state data segments as the reference digital spectrum. Synchronous power spectrum estimation is performed on the real-time acquired physical layer parameters and digital features to obtain the real-time digital spectrum; The amplitude difference between the real-time digital spectrum and the reference digital spectrum is calculated to form a digital residual spectrum, and the frequency axis of the digital residual spectrum is kept aligned with the reference digital spectrum. The amplitude value sequence of the digital residual spectrum and the physical layer parameters are concatenated by timestamp to form a high-dimensional feature vector; The distributed neighborhood embedding algorithm is used to reduce the dimensionality of high-dimensional feature vectors to obtain low-dimensional feature vectors.

[0046] In some embodiments, the anomaly classification module is specifically used to construct a stacked autoencoder model containing an input layer, a hidden layer and an output layer, and to train it using low-dimensional feature vectors of normal states in the complete dataset, using mean squared error as the loss function, and iteratively optimizing it until the loss converges to obtain the autoencoder. The dynamic threshold is obtained by fitting the error distribution curve based on the kernel density and dynamically adjusting the threshold according to the preset initial threshold and the time decay factor. The low-dimensional feature vector is input into the autoencoder to calculate the reconstruction error. When the reconstruction error is greater than the dynamic threshold, the current data is determined to be abnormal data. A support vector machine model is constructed using radial basis functions and kernel functions. The input is abnormal data and corresponding physical layer parameters, and the output is an abnormal data classification, which includes non-lockout anomalies, lockout-relocking anomalies, or equipment anomalies.

[0047] In some embodiments, the node localization module is specifically used to classify the low-dimensional feature vector corresponding to the abnormal data, set a sliding time window, and calculate the strength variance of each dimension of the low-dimensional feature vector within the time window. Obtain the physical topology information of the transmission link, map the strength variance to the link transmission path according to the timestamp, and plot the strength variance as a function of node position along the link from the sending end to the receiving end. The horizontal axis of the strength curve is the node number, and the vertical axis is the strength variance. Input the intensity curve into the preset decision tree model. If the output is an anomalous subset, extract the first node in the intensity curve that meets the condition that "the intensity variance exceeds the threshold and the parameter fluctuation matches the anomalous data classification". Combine the node's physical address to locate the anomalous node.

[0048] In some embodiments, the sample synthesis module is specifically used to count the number of samples for each type of abnormal data category for historical data within a preset period corresponding to the abnormal node location, calculate the percentage of each type of abnormal sample to the total number of abnormal samples, and obtain the classification ratio. If the proportion of a category is lower than a preset threshold, the corresponding anomaly is determined to be a minority anomaly, and synthetic minority oversampling technology is used to generate synthetic data of digital residual spectrum and optical power.

[0049] In some embodiments, the sample synthesis module is also used to extract the original data corresponding to minority anomalies and the digital residual spectrum and optical power time series data of the anomaly node locations; A synthetic minority oversampling technique is adopted. Based on the feature vector of the digital residual spectrum and the temporal characteristics of the optical power, the synthetic sample is generated by random interpolation in the feature space of the sample data corresponding to the minority anomaly in the original data and the feature space of the nearest sample according to the preset number of nearest neighbor samples and the sampling ratio. Calculate the feature similarity between the synthetic sample and the sample data corresponding to the minority class anomalies in the original data, and define the synthetic sample with a feature similarity exceeding a preset value as synthetic data.

[0050] This application also discloses a control device.

[0051] Specifically, the control device includes a memory and a processor, the memory storing a computer program that can be loaded and executed by the processor to perform the aforementioned abnormal data analysis method based on fiber optic communication.

[0052] This application also discloses a computer-readable storage medium.

[0053] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed, such as the above-described abnormal data analysis method based on fiber optic communication. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0054] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for analyzing abnormal data based on optical fiber communication, characterized in that, include: The receiver collects physical layer parameters, digital features, and link status data in real time. The collected data is then subjected to noise suppression and missing value completion to obtain a complete dataset. The baseline digital spectrum under normal conditions is calculated based on the complete dataset. A digital residual spectrum is generated by the difference between the real-time digital spectrum and the baseline digital spectrum. The digital residual spectrum and physical layer parameters are then dimensionality-reduced and fused to obtain a low-dimensional feature vector. An autoencoder is used to perform unsupervised anomaly detection on the low-dimensional feature vector. When the reconstruction error exceeds a dynamic threshold, it is determined to be abnormal data. Then, a support vector machine is used to classify the abnormal data. Based on abnormal data classification, the intensity variance of the low-dimensional feature vector is calculated through a preset time window, the intensity variance change curve is plotted along the transmission link, and the link data is recursively divided into normal subsets or abnormal subsets using a decision tree algorithm to locate the abnormal node positions. Based on the location of the abnormal nodes, if the proportion of abnormal data categories is lower than a preset threshold, synthetic minority class oversampling technology is used to generate the synthetic data of digital residual spectrum and optical power; The process involves real-time acquisition of physical layer parameters, digital features, and link status data via a receiver, followed by noise suppression and missing value completion of the acquired data to obtain a complete dataset, including: The receiver's digital signal processing module synchronously acquires the physical layer parameters, digital features, and link status data. The physical layer parameters include optical power, center wavelength, polarization state, and phase noise. The digital features include bit error rate, constellation diagram distortion, signal amplitude variance, and forward error correction overhead. The link status data includes node connection status, transmission rate, and optical switch switching records. The acquired raw data is decomposed into high-frequency noise components and low-frequency signal components by wavelet transform, and the high-frequency components are processed by a soft threshold function. The denoised signal is dynamically corrected by Kalman filtering to obtain denoised data; If the data missing rate is less than the threshold in a single time period, then time series linear interpolation is used to fill the missing data. If the data missing rate is not less than the threshold or the continuous missing duration exceeds the preset time within a single time period, the LSTM prediction model trained with historical data of the same type of link is called, and the physical layer parameters of the current link are combined to generate the complete data. The completed data is aligned by timestamp, normalized, and integrated into a structured data matrix to form the complete dataset. The row dimension of the structured data matrix is ​​the time series sampling points, and the column dimension is the feature combination of the physical layer parameters, the digital features, and the link status data.

2. The method for analyzing abnormal data based on optical fiber communication according to claim 1, characterized in that, The process involves calculating a baseline digital spectrum under normal conditions based on the complete dataset, generating a digital residual spectrum by the difference between the real-time digital spectrum and the baseline digital spectrum, and then performing dimensionality reduction and fusion on the digital residual spectrum and physical layer parameters to obtain a low-dimensional feature vector, including: Extract normal state data segments without anomaly labels from the complete dataset, perform power spectrum estimation using the Welch periodogram method, and calculate the digital spectrum mean of the normal state data segments as the baseline digital spectrum. Synchronous power spectrum estimation is performed on the real-time acquired physical layer parameters and digital features to obtain the real-time digital spectrum; The amplitude difference between the real-time digital spectrum and the reference digital spectrum is calculated to form a digital residual spectrum, and the frequency axis of the digital residual spectrum is aligned with the reference digital spectrum. The amplitude value sequence of the digital residual spectrum and the physical layer parameters are concatenated by timestamp to form a high-dimensional feature vector; The high-dimensional feature vector is reduced in dimensionality using a distributed neighborhood embedding algorithm to obtain the low-dimensional feature vector.

3. The method for analyzing abnormal data based on optical fiber communication according to claim 2, characterized in that, The process involves using an autoencoder to perform unsupervised anomaly detection on the low-dimensional feature vector. When the reconstruction error exceeds a dynamic threshold, the data is identified as anomalous. Then, a support vector machine is used to classify the anomalous data, including: A stacked autoencoder model containing an input layer, a hidden layer, and an output layer is constructed and trained using the low-dimensional feature vectors of the normal state in the complete dataset. The mean squared error is used as the loss function, and the model is iteratively optimized until the loss converges to obtain the autoencoder. The dynamic threshold is obtained by fitting the error distribution curve based on the kernel density and dynamically adjusting the threshold according to the preset initial threshold and the time decay factor. The low-dimensional feature vector is input into the autoencoder to calculate the reconstruction error. When the reconstruction error is greater than the dynamic threshold, the current data is determined to be abnormal data. A support vector machine model is constructed using radial basis function kernel function. The abnormal data and the corresponding physical layer parameters are input, and the abnormal data classification is output. The abnormal data classification includes non-lockout anomaly, lockout-relocking anomaly, or device anomaly.

4. The method for analyzing abnormal data based on optical fiber communication according to claim 3, characterized in that, The abnormal data classification method involves calculating the intensity variance of the low-dimensional feature vectors within a preset time window, plotting the intensity variance variation curve along the transmission link, recursively dividing the link data into normal or abnormal subsets using a decision tree algorithm, and locating the abnormal node positions, including: For the low-dimensional feature vector corresponding to the abnormal data classification, a sliding time window is set, and the strength variance of each dimension of the low-dimensional feature vector within the time window is calculated. Obtain the physical topology information of the transmission link, map the strength variance to the link transmission path according to the timestamp, and plot the strength variance as a function of node position along the link from the sending end to the receiving end. The horizontal axis of the strength curve is the node number, and the vertical axis is the strength variance. The intensity curve is input into a preset decision tree model. If the output is an abnormal subset, the position of the first node in the intensity curve that satisfies "the intensity variance exceeds the threshold and the parameter fluctuation matches the abnormal data classification" is extracted. Combined with the node's physical address, the abnormal node position is located.

5. The method for analyzing abnormal data based on optical fiber communication according to claim 3, characterized in that, Based on the location of the abnormal node, if the proportion of abnormal data classification is lower than a preset threshold, synthetic minority class oversampling technology is used to generate the synthetic data of digital residual spectrum and optical power, including: For the historical data within a preset period corresponding to the abnormal node location, the number of samples for each type of abnormal data category is counted, and the percentage of each type of abnormal sample in the total number of abnormal samples is calculated to obtain the category ratio. If the proportion of the classification is lower than the preset threshold, the corresponding anomaly is determined to be a minority anomaly, and the synthetic minority oversampling technique is used to generate the synthetic data of the digital residual spectrum and optical power.

6. The method for analyzing abnormal data based on optical fiber communication according to claim 5, characterized in that, The method of generating the synthesized digital residual spectrum and optical power data using synthetic minority class oversampling technology includes: Extract the original data corresponding to the minority anomalies, as well as the digital residual spectrum and optical power time series data of the anomaly node locations; The synthetic minority class oversampling technique is used, with the feature vector of the digital residual spectrum and the temporal characteristics of the optical power as inputs, and the synthetic samples are generated by random interpolation in the feature space of the sample data corresponding to the minority class anomaly in the original data and the feature space of the nearest neighbor samples according to the preset number of nearest neighbor samples and the sampling ratio. Calculate the feature similarity between the synthetic sample and the sample data corresponding to minority class anomalies in the original data, and define the synthetic sample whose feature similarity exceeds a preset value as the synthetic data.

7. The anomaly data analysis device according to any one of claims 1-6, based on the anomaly data analysis method of optical fiber communication, is characterized in that, The device includes: The multi-source acquisition module is used to acquire physical layer parameters, digital features and link status data in real time through the receiver, and to perform noise suppression and missing value completion on the acquired data to obtain a complete dataset. The feature vector module is used to calculate the baseline digital spectrum under normal conditions based on the complete dataset, generate a digital residual spectrum by the difference between the real-time digital spectrum and the baseline digital spectrum, and perform dimensionality reduction and fusion on the digital residual spectrum and physical layer parameters to obtain a low-dimensional feature vector. The anomaly classification module is used to perform unsupervised anomaly detection on the low-dimensional feature vector using an autoencoder. When the reconstruction error exceeds a dynamic threshold, it is determined to be abnormal data, and then the abnormal data is classified by a support vector machine. The node localization module is used to classify abnormal data, calculate the intensity variance of the low-dimensional feature vector through a preset time window, plot the intensity variance change curve along the transmission link, and recursively divide the link data into normal subsets or abnormal subsets using a decision tree algorithm to locate the abnormal node positions. The sample synthesis module is used to generate the synthesized data of the digital residual spectrum and optical power based on the location of the abnormal node and if the proportion of the abnormal data classification is lower than a preset threshold, by using a synthetic minority class oversampling technique.

8. A control device, characterized in that, The device includes: It includes a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and executed as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer program is stored that can be loaded by a processor and executed as described in any one of claims 1 to 6.

Images