Abnormal data analysis method and system based on optical fiber communication
By collecting and processing data in the optical fiber communication system, a reference digital spectrum and residual spectrum are constructed. Anomaly detection and classification are then performed by combining an autoencoder and support vector machine. This solves the problem of scarce abnormal data, enables accurate identification of abnormal data and location of fault nodes in the optical fiber communication system, and improves the accuracy of model analysis and network reliability.
Patent Information
- Application Number
- CN202511344857.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-19
- Publication Date
- 2025-12-16
AI Technical Summary
In fiber optic communication systems, abnormal data is scarce. Traditional supervised learning methods rely on a large number of labeled abnormal samples, which makes it difficult for the model to fully learn abnormal features, thus limiting the accuracy of anomaly analysis.
The receiver collects physical layer parameters, digital features, and link status data in real time, performs noise suppression and missing value completion, constructs a baseline digital spectrum and residual spectrum, uses an autoencoder for unsupervised anomaly detection, combines support vector machine classification, uses decision tree to locate anomaly nodes, and optimizes sample distribution through synthetic minority class oversampling technology.
It enables real-time and accurate identification of abnormal data and location of fault nodes in fiber optic communication systems, improves the accuracy of model analysis and the reliability of communication networks, and solves the problem of scarce abnormal samples.
Smart Images
Figure CN121150801A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of optical fiber communication, in particular to an abnormal data analysis method and system based on optical fiber communication. BACKGROUND
[0002] Optical fiber communication refers to a communication mode that transmits information through optical signals by using optical fibers as transmission media. The core focuses on signal transmission technology at the physical layer, such as optical modulation, optical fiber transmission characteristics, optical amplification, and the performance optimization of optical transmitters and receivers. Optical fiber communication has significant advantages such as large transmission bandwidth, low loss, resistance to electromagnetic interference, and strong security, and is a core support technology for modern communication networks, widely used in backbone networks, metropolitan area networks, data center interconnections, and mobile communication backhaul scenarios.
[0003] With the expansion of optical fiber communication network scale, the amount of abnormal data caused by communication faults and network attacks increases exponentially. In optical fiber communication systems, abnormal data such as phase jitter, power loss, and data loss can cause signal transmission quality to deteriorate, and even trigger link faults.
[0004] In actual scenarios, abnormal data is often scarce, and traditional supervised learning methods rely on a large number of labeled abnormal samples to achieve effective training. Therefore, the model is difficult to fully learn abnormal features, resulting in limited accuracy of abnormal analysis.
[0005] Based on this, the present application provides an abnormal data analysis method and system based on optical fiber communication. SUMMARY
[0006] In order to improve the problem that abnormal data is often scarce, and traditional supervised learning methods rely on a large number of labeled abnormal samples to achieve effective training, so the model is difficult to fully learn abnormal features, resulting in limited accuracy of abnormal analysis, the present application provides an abnormal data analysis method and system based on optical fiber communication.
[0007] In a first aspect, the present application provides an abnormal data analysis method based on optical fiber communication, which adopts the following technical solution: comprising: Real-time acquisition of physical layer parameters, digital features, and link state data by a receiver, noise suppression and missing value completion of the collected data to obtain a complete data set; Based on the complete data set, a baseline digital spectrum under normal state is calculated, a digital residual spectrum is generated by the difference between real-time digital spectrum and the baseline digital spectrum, and the digital residual spectrum and the physical layer parameters are dimensionally reduced and fused to obtain a low-dimensional feature vector; Using an autoencoder for unsupervised anomaly detection of the low-dimensional feature vector, when the reconstruction error exceeds a dynamic threshold, it is determined as abnormal data, and then a support vector machine is used to classify the abnormal data; Based on the abnormal data classification, the intensity variance of the low-dimensional feature vector is calculated through a preset time window, the intensity variance curve is drawn along the transmission link, the decision tree algorithm is used to recursively divide the link data into normal subsets or abnormal subsets, and the abnormal node position is located; Based on the abnormal node position, if the proportion of the abnormal data classification is lower than a preset threshold, a synthetic minority over-sampling technique is used to generate synthetic data of the digital residual spectrum and optical power.
[0008] Preferably, the physical layer parameters, digital features and link state data are collected in real time by the receiver, noise suppression and missing value completion are performed on the collected data to obtain a complete data set, including: The physical layer parameters, digital features and link state data are synchronously collected by the digital signal processing module of the receiver, the physical layer parameters include optical power, center wavelength, polarization state and phase noise, the digital features include bit error rate, constellation distortion, signal amplitude variance and forward error correction overhead, and the link state data includes node connection state, transmission rate and optical switch switching record; The collected original data is decomposed into high-frequency noise components and low-frequency signal components by wavelet transform, and the high-frequency components are processed by a soft threshold function; The denoised signal is dynamically error-corrected by Kalman filtering to obtain denoised data; If the data missing rate in a single period is less than a threshold, time series linear interpolation method is used to fill in; If the data missing rate in a single period is not less than a threshold or the continuous missing time length exceeds a preset time, an LSTM prediction model trained by historical data of similar links is called to generate complete data combined with the physical layer parameters of the current link; The complete data is aligned by timestamp, normalized and integrated into a structured data matrix to form the complete data set, and the row dimension of the structured data matrix is the time series sampling point, and the column dimension is the feature combination of the physical layer parameters, digital features and link state data.
[0009] Preferably, the reference digital spectrum under normal state is calculated based on the complete data set, the digital residual spectrum is generated by the difference between the real-time digital spectrum and the reference digital spectrum, and the digital residual spectrum and the physical layer parameters are reduced and fused to obtain a low-dimensional feature vector, including: The normal state data segment without abnormal label is extracted from the complete data set, the Welch periodogram method is used for power spectrum estimation, and the digital spectrum mean of the normal state data segment is calculated as the reference digital spectrum; The physical layer parameters and the digital features collected in real time are synchronously power spectrum estimated to obtain a real-time digital spectrum; calculating a difference between amplitudes of the real-time digital spectrum and the reference digital spectrum to form a digital residual spectrum, a frequency axis of the digital residual spectrum being kept aligned with the reference digital spectrum; splicing a sequence of amplitude values of the digital residual spectrum and the physical layer parameters into a high-dimensional feature vector according to time stamps; performing dimension reduction on the high-dimensional feature vector by using a distributed neighborhood embedding algorithm to obtain a low-dimensional feature vector.
[0010] Preferably, the low-dimensional feature vector is subjected to unsupervised anomaly detection by using an autoencoder, and when a reconstruction error exceeds a dynamic threshold, the current data is determined as abnormal data, and the abnormal data is classified by using a support vector machine, including: constructing a stacked autoencoder model including an input layer, a hidden layer and an output layer, and training the low-dimensional feature vector of the normal state in the complete data set to obtain the autoencoder by using a mean square error as a loss function and iteratively optimizing until the loss converges; fitting an error distribution curve based on a kernel density, and dynamically adjusting the threshold value according to a preset initial threshold value and a time decay factor to obtain the dynamic threshold value; inputting the low-dimensional feature vector into the autoencoder to calculate a reconstruction error, and determining the current data as the abnormal data when the reconstruction error is greater than the dynamic threshold value; constructing a support vector machine model by using a radial basis function kernel function, inputting the abnormal data and corresponding physical layer parameters, and outputting the abnormal data classification, the abnormal data classification including a non-lockout abnormality, a lockout relocking abnormality or a device abnormality.
[0011] Preferably, based on the abnormal data classification, the intensity variance of the low-dimensional feature vector is calculated through a preset time window, an intensity variance change curve is drawn along a transmission link, a decision tree algorithm is used to recursively divide the link data into a normal subset or an abnormal subset, and the abnormal node position is located, including: setting a sliding time window for the low-dimensional feature vector corresponding to the abnormal data classification, and calculating the intensity variance of each dimension of the low-dimensional feature vector within the time window; obtaining physical topology information of a transmission link, mapping the intensity variance to a link transmission path according to time stamps, and drawing an intensity curve of the intensity variance changing with node positions along the sending end to the receiving end of the link, the horizontal axis of the intensity curve being a node number and the vertical axis being the intensity variance; inputting the intensity curve into a preset decision tree model, and if the output is an abnormal subset, extracting the first node position in the intensity curve that satisfies “the intensity variance exceeds the threshold value and the parameter fluctuation matches the abnormal data classification”, and locating the abnormal node position in combination with the node physical address.
[0012] Preferably, based on the abnormal node position, if the abnormal data classification proportion is below a preset threshold, a synthetic minority over-sampling technique is used to generate synthetic data of the digital residual spectrum and optical power, comprising: For the historical data in the preset period corresponding to the abnormal node position, the sample number of each type of abnormal data classification is counted, the percentage of the number of each type of abnormal sample in the total number of abnormal samples is calculated, and the classification proportion is obtained. If the classification proportion is below the preset threshold, it is determined that the corresponding abnormality is a minority class abnormality, and a synthetic minority over-sampling technique is used to generate synthetic data of the digital residual spectrum and optical power.
[0013] Preferably, the synthetic minority over-sampling technique for generating synthetic data of the digital residual spectrum and optical power comprises: Extract the original data corresponding to the minority class abnormality and the digital residual spectrum and optical power time series data of the abnormal node position; Using the synthetic minority over-sampling technique, according to the feature vector of the digital residual spectrum and the time domain feature of the optical power as input, according to the preset number of nearest neighbor samples and the sampling rate, synthetic samples are generated by random interpolation in the feature space of the sample data corresponding to the minority class abnormality and the nearest neighbor samples in the original data. Calculate the feature similarity of the synthetic samples and the sample data corresponding to the minority class abnormality in the original data, and define the synthetic samples with feature similarity exceeding a preset value as the synthetic data.
[0014] In a second aspect, an abnormal data analysis device based on optical fiber communication is provided, which adopts the following technical scheme, comprising: A multi-source acquisition module is used to acquire physical layer parameters, digital features and link state data in real time through a receiver, to perform noise suppression and missing value completion on the acquired data, and to obtain a complete data set. A feature vector module is used to calculate a reference digital spectrum under normal state based on the complete data set, to generate a digital residual spectrum by the difference between a real-time digital spectrum and the reference digital spectrum, and to perform dimension reduction fusion on the digital residual spectrum and the physical layer parameters to obtain a low-dimensional feature vector. An abnormal classification module is used to perform unsupervised anomaly detection on the low-dimensional feature vector by using an automatic encoder, to determine as abnormal data when the reconstruction error exceeds a dynamic threshold, and to classify the abnormal data by using a support vector machine. A node positioning module is used to calculate the intensity variance of the low-dimensional feature vector based on abnormal data classification through a preset time window, to draw an intensity variance change curve along the transmission link, to recursively divide the link data into normal subsets or abnormal subsets by using a decision tree algorithm, and to locate the abnormal node position. a sample synthesis module configured to, based on the abnormal node position, if the proportion of the abnormal data classification is lower than a preset threshold, generate synthetic data of the digital residual spectrum and the optical power by using a synthetic minority over-sampling technique.
[0015] In a third aspect, the present application also provides a control device, which comprises: comprising a memory and a processor, the memory storing a computer program capable of being loaded and executed by the processor to implement the above abnormal data analysis method based on optical fiber communication.
[0016] In a fourth aspect, the present application also provides a computer readable storage medium storing a computer program capable of being loaded and executed by a processor to implement the above abnormal data analysis method based on optical fiber communication.
[0017] In summary, in the present application, the data noise interference and missing problems of the optical fiber link are solved by the collection and dynamic repair technology, and high-quality data base is provided for subsequent analysis; then the digital residual spectrum is introduced, the weak abnormal signal is captured by the difference analysis of the reference spectrum and the real-time spectrum, and the effective compression of high-dimensional features is realized by combining the dimension reduction technology; then the unsupervised and supervised learning collaborative abnormal recognition architecture is adopted, which not only guarantees the detection ability of unknown abnormalities, but also realizes the accurate classification of abnormal types; finally, the abstract data anomaly is mapped to the physical node fault by the intensity variance curve and the decision tree topology positioning algorithm, and the feature space interpolation synthesis technology is used to dynamically optimize the sample distribution to solve the problem of insufficient abnormal samples in the communication system, thereby solving the problem of insufficient number of abnormal samples in the actual scene, and further improving the training effect of the analysis model to improve the accuracy of the model analysis. BRIEF DESCRIPTION OF DRAWINGS
[0018] Figure 1 is a flowchart of an abnormal data analysis method based on optical fiber communication.
[0019] Figure 2 is a structural block diagram of an abnormal data analysis device based on optical fiber communication. DETAILED DESCRIPTION
[0020] The following will be described in detail in combination with Figure 1 - Figure 2 The present application will be further described in detail.
[0021] The existing optical fiber link anomaly detection mainly relies on fixed threshold alarm, and has problems of insufficient weak anomaly capture ability, low fault positioning accuracy, and missed detection caused by unbalanced anomaly sample categories. To solve the above defects, the present application provides a full-process analysis scheme, which realizes real-time accurate identification of optical fiber link anomaly, fault node positioning and sample distribution optimization through multi-dimensional data acquisition and dynamic repair, digital residual spectrum feature enhancement, deep learning anomaly detection and classification, decision tree topology positioning and minority class sample synthesis technology, and significantly improves the reliability and self-healing ability of the communication network.
[0022] Reference Figure 1 The embodiments of the present application at least include steps S10 to S50.
[0023] S10, the physical layer parameters, digital features and link state data are collected in real time by the receiver, noise suppression and missing value completion are performed on the collected data, and a complete data set is obtained.
[0024] S20, the reference digital spectrum under normal state is calculated based on the complete data set, the digital residual spectrum is generated by the difference between the real-time digital spectrum and the reference digital spectrum, and the digital residual spectrum and the physical layer parameters are reduced and fused to obtain a low-dimensional feature vector.
[0025] S30, the low-dimensional feature vector is detected for anomaly by using an automatic encoder, when the reconstruction error exceeds a dynamic threshold, it is determined as abnormal data, and the abnormal data is classified by using a support vector machine.
[0026] S40, based on the classification of abnormal data, the intensity variance of the low-dimensional feature vector is calculated through a preset time window, the intensity variance curve is drawn along the transmission link, the decision tree algorithm is used to recursively divide the link data into normal subsets or abnormal subsets, and the abnormal node position is located.
[0027] S50, based on the abnormal node position, if the proportion of abnormal data classification is lower than a preset threshold, the synthetic minority over-sampling technique is used to generate synthetic data of the digital residual spectrum and optical power.
[0028] Specifically, through multi-dimensional data acquisition and preprocessing, noise is suppressed and missing values are completed, providing a high-quality data basis for subsequent analysis; the reference digital spectrum and residual spectrum are constructed, the low-dimensional features are extracted by dimension reduction fusion, and the abnormal signal representation ability is enhanced; the automatic encoder is used to realize unsupervised anomaly detection, the SVM is combined to accurately classify the abnormal types, the detection efficiency and accuracy are improved; the abnormal node is located by the intensity variance curve and the decision tree algorithm, for the minority class abnormal sample, the over-sampling technique is used to generate synthetic data, the training sample distribution is optimized, and the low-proportion abnormal missed detection is reduced, thereby solving the problem of insufficient number of abnormal samples in actual scene, and further improving the training effect of the analysis model, so as to improve the accuracy of model analysis.
[0029] In some embodiments, step S10 specifically comprises the following steps: synchronously collecting physical layer parameters, digital features and link state data by a digital signal processing module of the receiver, the physical layer parameters including optical power, central wavelength, polarization state and phase noise, the digital features including bit error rate, constellation diagram distortion, signal amplitude variance and forward error correction overhead, and the link state data including node connection state, transmission rate and optical switch switching record; decomposing the collected original data into high-frequency noise components and low-frequency signal components by wavelet transform, and processing the high-frequency components by a soft threshold function; performing dynamic error correction on the denoised signals by Kalman filtering to obtain denoised data; if the data loss rate in a single time period is less than a threshold, filling by a time series linear interpolation method; if the data loss rate in a single time period is not less than the threshold or the continuous loss duration exceeds a preset time, calling an LSTM prediction model trained by historical data of similar links to generate completed data in combination with the physical layer parameters of the current link; aligning the completed data by time stamp, normalizing and integrating into a structured data matrix to form a complete data set, and the row dimension of the structured data matrix is the time series sampling point and the column dimension is the feature combination of the physical layer parameters, the digital features and the link state data.
[0030] Specifically, a holographic data base of the optical fiber communication system is constructed by multi-dimensional synchronous collection, covering physical layer parameters such as optical power and wavelength, digital features such as bit error rate and constellation diagram distortion, and link data such as node state, realizing full-element monitoring; the wavelet transform soft threshold denoising is used to separate noise and signal components, and the Kalman filtering is combined to dynamically correct errors, significantly improving the data signal-to-noise ratio; the interpolation or LSTM prediction completion strategy is intelligently selected according to different missing scenarios to ensure data continuity and integrity; finally, the time alignment, normalization and structured integration are performed to form a time series feature matrix, providing high-quality and standardized data input for subsequent residual spectrum analysis and anomaly detection, effectively solving the problems of noise interference, missing and format heterogeneity of optical fiber link data.
[0031] In some embodiments, step S20 specifically comprises the following steps: extracting a normal state data segment without an abnormal flag from the complete data set, performing power spectrum estimation by the Welch periodogram method, and calculating the digital spectrum mean of the normal state data segment as a reference digital spectrum; synchronously performing power spectrum estimation on the real-time collected physical layer parameters and digital features to obtain a real-time digital spectrum; calculating the amplitude difference between the real-time digital spectrum and the reference digital spectrum to form a digital residual spectrum, and the frequency axis of the digital residual spectrum is aligned with the reference digital spectrum; concatenating the amplitude value sequence of the digital residual spectrum and the physical layer parameters into a high-dimensional feature vector according to the time stamp; performing dimension reduction on the high-dimensional feature vector by the distributed neighborhood embedding algorithm to obtain a low-dimensional feature vector.
[0032] Specifically, by extracting the normal state data segment and using the Welch periodogram method to establish a reference digital spectrum, a reference benchmark is provided for anomaly detection; the difference between the real-time digital spectrum and the reference spectrum is calculated to form a digital residual spectrum, effectively amplifying the weak abnormal signal characteristics; the residual spectrum amplitude sequence is spliced with the physical layer parameters to form a high-dimensional vector, and multi-modal features are fused; the distributed neighborhood embedding algorithm is used for dimension reduction, which reduces the data dimension while retaining the correlation of key features, reduces the subsequent calculation complexity, and improves the model efficiency; this process realizes the transformation from raw data to high-recognizability low-dimensional features, lays the foundation for precise anomaly detection, and solves the problems of feature redundancy and difficulty in capturing weak anomalies in traditional methods.
[0033] In some embodiments, step S30 specifically comprises the following steps: a stack auto-encoder model comprising an input layer, a hidden layer and an output layer is constructed, and the low-dimensional feature vectors of the normal state in the complete data set are used for training, with mean square error as the loss function, iterative optimization to loss convergence, to obtain an auto-encoder; the error distribution curve is fitted based on kernel density, and the threshold value is dynamically adjusted according to the preset initial threshold value and time decay factor, to obtain a dynamic threshold value; the low-dimensional feature vectors are input into the auto-encoder, and the reconstruction error is calculated, and when the reconstruction error is greater than the dynamic threshold value, the current data is determined as abnormal data; a support vector machine model is constructed using a radial basis function kernel function, the abnormal data and corresponding physical layer parameters are input, and the abnormal data classification is output, including non-lockout abnormality, lockout relock abnormality or equipment abnormality.
[0034] Specifically, by constructing a stack auto-encoder and training it with low-dimensional feature vectors of the normal state, the model is optimized with mean square error as the loss function, so that it can accurately reconstruct the normal data pattern; the error distribution is fitted based on kernel density and combined with a time decay factor to dynamically adjust the threshold value, improving the environmental adaptability of anomaly judgment; unsupervised anomaly detection is achieved by comparing the auto-encoder reconstruction error with the dynamic threshold value, overcoming the limitations of traditional fixed threshold values; the SVM model using a radial basis function kernel function is used to classify abnormal data and physical layer parameters, outputting non-lockout abnormality, lockout relock abnormality or equipment abnormality type, achieving a leap from anomaly detection to precise classification, solving the problem of ambiguous abnormal types in complex scenarios, and providing classification basis for subsequent fault location.
[0035] In some embodiments, step S40 specifically comprises the following steps: classifying the low-dimensional feature vector corresponding to the abnormal data, setting a sliding time window, and calculating the intensity variance of each dimension of the low-dimensional feature vector within the time window; obtaining the physical topology information of the transmission link, mapping the intensity variance to the link transmission path according to the timestamp, and drawing an intensity curve of the intensity variance changing with the node position along the sending end to the receiving end of the link, wherein the horizontal axis of the intensity curve is the node number, and the vertical axis is the intensity variance; inputting the intensity curve into a preset decision tree model, and if the output is an abnormal subset, extracting the first node position in the intensity curve that satisfies the condition of "intensity variance exceeding threshold and parameter fluctuation matching abnormal data classification", and combining the node physical address to locate the abnormal node position.
[0036] Specifically, a sliding time window is set for the abnormal classification data to calculate the intensity variance of the low-dimensional feature vector, and the abnormal fluctuation characteristics are captured; the variance is mapped to the transmission path combined with the link physical topology, and the node number-intensity variance curve is drawn to realize abnormal visualization; the first node position with variance exceeding threshold and fluctuation matching abnormal type is extracted through recursive division of data by the decision tree model, and the physical address is combined to complete accurate positioning. This process converts the abstract feature variance into physical node coordinates, solves the problem of fuzzy traditional link fault positioning, realizes rapid tracing of abnormal points, provides accurate targeting for subsequent maintenance, and improves the fault troubleshooting efficiency of the optical fiber communication system.
[0037] In some embodiments, step S50 specifically comprises the following steps: for the historical data corresponding to the abnormal node position within a preset period, counting the number of samples of each type of abnormal data classification, calculating the percentage of the number of each type of abnormal sample in the total number of abnormal samples to obtain the classification proportion; if there is a classification proportion below a preset threshold, it is determined that the corresponding abnormality is a minority class abnormality, and then a synthetic minority over-sampling technique is used to generate synthetic data of the digital residual spectrum and optical power.
[0038] Further, the original data corresponding to the minority class abnormality and the digital residual spectrum and optical power time series data of the abnormal node position are extracted; a synthetic minority over-sampling technique is used, the feature vector of the digital residual spectrum and the time domain features of the optical power are taken as inputs, and according to a preset number of neighbor samples and a sampling rate, synthetic samples are randomly interpolated in the feature space of the sample data corresponding to the minority class abnormality in the original data and the neighbor samples; the feature similarity between the synthetic samples and the sample data corresponding to the minority class abnormality in the original data is calculated, and the synthetic samples with feature similarity exceeding a preset value are defined as synthetic data.
[0039] Specifically, by counting the classification proportion of each type of anomaly in the historical data of the abnormal node, minority class anomalies with a proportion below a threshold are identified, and synthetic minority over-sampling technique (SMOTE) is used to generate supplementary data; specifically, the original data, digital residual spectrum and optical power time sequence features of the minority class anomalies are extracted, and synthetic samples are generated by random interpolation in the feature space based on the number of neighboring samples and the sampling rate, and high-quality synthetic data is selected by feature similarity; this process solves the problem of class imbalance of abnormal samples, enhances the feature expression of minority class anomalies, improves the recognition ability of the model for rare abnormal types, avoids detection bias caused by sample scarcity, and provides balanced data support for model iteration and optimization.
[0040] In some embodiments, considering the fusion of the extreme gradient boosting model to optimize the anomaly classification performance, the corresponding processing steps are as follows: an extreme gradient boosting classification model containing an input layer, a hidden layer and an output layer is constructed, the low-dimensional feature vector and the corresponding anomaly data classification label are taken as training samples, the model hyperparameters including learning rate, tree depth, leaf node number and regularization coefficients λ and γ are optimized through grid search and five-fold cross-validation, Cox proportional hazards regression with L1 penalty term is used as the loss function, the first and second derivatives of the loss function are calculated to optimize the decision tree structure, the normal state data and the supplementary samples generated by the synthetic minority over-sampling technique are used for training, and the iteration is optimized until the loss converges; the real-time low-dimensional feature vector is input into the trained model, and the classification results of non-lockout anomaly, lockout relocking anomaly or device anomaly are output.
[0041] The implementation principle of an embodiment of the abnormal data analysis method based on optical fiber communication is as follows: noise interference and missing problems of optical fiber link data are solved through acquisition and dynamic repair technology, providing a high-quality data base for subsequent analysis; then the digital residual spectrum is introduced, and weak abnormal signals are captured through difference analysis of the reference spectrum and the real-time spectrum, and effective compression of high-dimensional features is realized in combination with dimension reduction technology; then an abnormal recognition architecture of unsupervised and supervised learning cooperation is adopted, which not only guarantees the detection ability of unknown anomalies, but also realizes accurate classification of abnormal types; finally, the abstract data anomaly is mapped to physical node failure through intensity variance curve and decision tree topology positioning algorithm, and for the model bias problem caused by insufficient abnormal samples in the communication system, feature space interpolation synthesis technology is used to dynamically optimize the sample distribution, thereby solving the problem of insufficient number of abnormal samples in the actual scene, and further improving the training effect of the analysis model, so as to improve the accuracy of model analysis.
[0042] Figure 1 A flowchart of an embodiment of the abnormal data analysis method based on optical fiber communication is shown. It should be understood that, although Figure 1The steps in the flowchart are displayed in sequence according to the indication of the arrows, but the steps are not necessarily executed in the order indicated by the arrows; unless otherwise specified herein, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other orders; and Figure 1 At least part of the steps in the flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the sub-steps or stages is not necessarily sequential, but can be alternately executed with other steps or at least part of the sub-steps or stages of other steps.
[0043] Based on the same technical concept, referring to Figure 2 The embodiment of the application also provides an abnormal data analysis device based on optical fiber communication, which adopts the following technical scheme, the device comprises: A multi-source acquisition module is configured to acquire physical layer parameters, digital features and link state data in real time through a receiver, perform noise suppression and missing value completion on the acquired data, and obtain a complete data set; A feature vector module is configured to calculate a reference digital spectrum in a normal state based on the complete data set, generate a digital residual spectrum through a difference between a real-time digital spectrum and the reference digital spectrum, and perform dimension reduction fusion on the digital residual spectrum and the physical layer parameters to obtain a low-dimensional feature vector; An abnormal classification module is configured to perform unsupervised anomaly detection on the low-dimensional feature vector by using an autoencoder, determine that the data is abnormal when a reconstruction error exceeds a dynamic threshold, and classify the abnormal data by using a support vector machine; A node positioning module is configured to calculate a strength variance of the low-dimensional feature vector through a preset time window based on the classification of the abnormal data, draw a strength variance change curve along a transmission link, recursively divide the link data into a normal subset or an abnormal subset by using a decision tree algorithm, and locate an abnormal node position; A sample synthesis module is configured to generate synthetic data of the digital residual spectrum and the optical power by using a synthetic minority over-sampling technique if the proportion of the abnormal data classification is lower than a preset threshold based on the abnormal node position.
[0044] In some embodiments, the multi-source acquisition module is specifically configured to synchronously acquire the physical layer parameters, the digital features and the link state data by a digital signal processing module of the receiver, the physical layer parameters include optical power, center wavelength, polarization state and phase noise, the digital features include bit error rate, constellation diagram distortion degree, signal amplitude variance and forward error correction overhead, and the link state data includes node connection state, transmission rate and optical switch switching record; The collected original data is decomposed into high-frequency noise components and low-frequency signal components by wavelet transform, and the high-frequency components are processed by using a soft threshold function; The denoised signal is dynamically error-corrected by Kalman filtering to obtain denoised data; If the data missing rate in a single period is less than a threshold, a time series linear interpolation method is used to fill in; If the data missing rate in a single period is not less than the threshold or the continuous missing duration exceeds a preset time, an LSTM prediction model trained by historical data of the same type of link is called to generate the completed data in combination with the physical layer parameters of the current link; The completed data is aligned by timestamp, normalized and integrated into a structured data matrix to form a complete data set, and the row dimension of the structured data matrix is the time series sampling point, and the column dimension is the feature combination of the physical layer parameters, digital features and link state data.
[0045] In some embodiments, the feature vector module is specifically configured to extract normal state data segments without abnormal labels from the complete data set, perform power spectrum estimation using the Welch periodogram method, and calculate the digital spectrum mean of the normal state data segments as the reference digital spectrum; Synchronous power spectrum estimation is performed on the real-time collected physical layer parameters and digital features to obtain real-time digital spectrum; The amplitude difference between the real-time digital spectrum and the reference digital spectrum is calculated to form a digital residual spectrum, and the frequency axis of the digital residual spectrum is aligned with the reference digital spectrum; The amplitude value sequence of the digital residual spectrum and the physical layer parameters are spliced into a high-dimensional feature vector according to the timestamp; The high-dimensional feature vector is reduced in dimension using the distributed neighborhood embedding algorithm to obtain a low-dimensional feature vector.
[0046] In some embodiments, the anomaly classification module is specifically configured to construct a stack autoencoder model including an input layer, a hidden layer and an output layer, and train the low-dimensional feature vectors of the normal state in the complete data set to obtain an autoencoder using the mean square error as the loss function and iteratively optimizing until the loss converges; The error distribution curve is fitted based on the kernel density, and the threshold is dynamically adjusted according to the preset initial threshold and time decay factor to obtain a dynamic threshold; The low-dimensional feature vector is input into the autoencoder, and the reconstruction error is calculated. When the reconstruction error is greater than the dynamic threshold, the current data is determined to be abnormal data; A support vector machine model is constructed using a radial basis function kernel function, the abnormal data and corresponding physical layer parameters are input, and the abnormal data classification is output, including non-lockout abnormality, lockout relocking abnormality or equipment abnormality.
[0047] In some embodiments, the node positioning module is specifically configured to set a sliding time window for the low-dimensional feature vector corresponding to the abnormal data classification, and calculate the intensity variance of each dimension of the low-dimensional feature vector in the time window. Obtaining physical topology information of a transmission link, mapping the intensity variance to the link transmission path according to the timestamp, drawing an intensity curve of the intensity variance changing with the node position along the transmission end to the receiving end of the link, the horizontal axis of the intensity curve being the node number and the vertical axis being the intensity variance; Inputting the intensity curve into a preset decision tree model, if the output is an abnormal subset, extracting the first node position in the intensity curve satisfying "the intensity variance exceeds the threshold value and the parameter fluctuation matches the abnormal data classification", and locating the abnormal node position in combination with the node physical address.
[0048] In some embodiments, the sample synthesis module is specifically configured to, for the historical data in a preset period corresponding to the abnormal node position, count the sample quantity of each type of abnormal data classification, calculate the percentage of the number of each type of abnormal sample in the total number of abnormal samples, and obtain the classification proportion; If there is a classification proportion lower than a preset threshold, it is determined that the corresponding abnormality is a minority class abnormality, and a minority class oversampling technology is used to generate synthetic data of the digital residual spectrum and the optical power.
[0049] In some embodiments, the sample synthesis module is further configured to extract the original data corresponding to the minority class abnormality and the digital residual spectrum and optical power time series data of the abnormal node position; Using the minority class oversampling technology, according to the feature vector of the digital residual spectrum and the time domain feature of the optical power as input, according to a preset number of neighbor samples and a sampling rate, synthetic samples are generated by random interpolation in the feature space of the sample data corresponding to the minority class abnormality in the original data and the neighbor samples; Calculating the feature similarity between the synthetic samples and the sample data corresponding to the minority class abnormality in the original data, and defining the synthetic samples with feature similarity exceeding a preset value as synthetic data.
[0050] Embodiments of the present application also disclose a control device.
[0051] Specifically, the control device includes a memory and a processor, and the memory stores a computer program capable of being loaded and executed by the processor to perform the above-mentioned abnormal data analysis method based on optical fiber communication.
[0052] Embodiments of the present application also disclose a computer readable storage medium.
[0053] Specifically, the computer readable storage medium stores a computer program capable of being loaded and executed by the processor to perform the above-mentioned abnormal data analysis method based on optical fiber communication, and the computer readable storage medium includes, for example, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media capable of storing program codes.
[0054] The above are all preferred embodiments of the present application, and do not limit the protection scope of the present application, so that: all equivalent changes made according to the structure, shape, principle of the present application should be covered within the protection scope of the present application.
Claims
1. An abnormal data analysis method based on optical fiber communication, characterized by, The application relates to a method for detecting abnormal data in a transmission link. The method comprises the following steps: Real-time acquisition of physical layer parameters, digital features and link state data by a receiver, noise suppression and missing value completion of the acquired data, and obtaining of a complete data set; Calculation of a reference digital spectrum under a normal state based on the complete data set, generation of a digital residual spectrum through a difference between a real-time digital spectrum and the reference digital spectrum, dimension reduction fusion of the digital residual spectrum and the physical layer parameters, and obtaining of a low-dimensional feature vector; Unsupervised anomaly detection of the low-dimensional feature vector by using an automatic encoder, determination of abnormal data when a reconstruction error exceeds a dynamic threshold, and classification of the abnormal data by using a support vector machine; Based on the classification of the abnormal data, calculation of intensity variance of the low-dimensional feature vector through a preset time window, drawing of an intensity variance change curve along the transmission link, recursive division of link data into a normal subset or an abnormal subset by using a decision tree algorithm, and positioning of an abnormal node position; 2. The abnormal data analysis method based on optical fiber communication according to claim 1, characterized in that, Based on the abnormal node position, if the proportion of the classification of the abnormal data is lower than a preset threshold, generating of synthetic data of the digital residual spectrum and optical power by using a synthetic minority over-sampling technique. The real-time acquisition of physical layer parameters, digital features and link state data by the receiver, the noise suppression and missing value completion of the acquired data, and the obtaining of the complete data set comprise the following steps: Synchronous acquisition of the physical layer parameters, the digital features and the link state data by a digital signal processing module of the receiver, wherein the physical layer parameters comprise optical power, central wavelength, polarization state and phase noise, the digital features comprise bit error rate, constellation diagram distortion degree, signal amplitude variance and forward error correction overhead, and the link state data comprises node connection state, transmission rate and optical switch switching record; Decomposition of the acquired original data into high-frequency noise components and low-frequency signal components by using a wavelet transform, and soft threshold function processing of the high-frequency components; Dynamic error correction of the denoised signal by using Kalman filtering, and obtaining of denoised data; If the data missing rate in a single period is less than a threshold, time series linear interpolation is used for filling; If the data missing rate in a single period is not less than the threshold or the continuous missing duration exceeds a preset time, an LSTM prediction model trained by using historical data of a similar link is called to generate the completed data in combination with the physical layer parameters of the current link; 3. The method of claim 2, wherein the method further comprises: The completed data is aligned according to timestamps, normalized and integrated into a structured data matrix to form the complete data set, and the row dimension of the structured data matrix is a time series sampling point, and the column dimension is a feature combination of the physical layer parameters, the digital features and the link state data. The calculation of the reference digital spectrum under the normal state based on the complete data set, the generation of the digital residual spectrum through the difference between the real-time digital spectrum and the reference digital spectrum, and the dimension reduction fusion of the digital residual spectrum and the physical layer parameters to obtain the low-dimensional feature vector comprise the following steps: Extraction of a normal state data segment without an abnormal label from the complete data set, power spectrum estimation by using a Welch periodogram method, calculation of a digital spectrum mean value of the normal state data segment as the reference digital spectrum, and calculation of a digital spectrum mean value of the normal state data segment as the reference digital spectrum. synchronously performing power spectrum estimation on the physical layer parameters and the digital features to obtain a real-time digital spectrum; calculating an amplitude difference between the real-time digital spectrum and the reference digital spectrum to form a digital residual spectrum, a frequency axis of the digital residual spectrum being aligned with the reference digital spectrum; concatenating a sequence of amplitude values of the digital residual spectrum and the physical layer parameters by timestamps to form a high-dimensional feature vector; performing dimension reduction on the high-dimensional feature vector by using a distributed neighborhood embedding algorithm to obtain a low-dimensional feature vector.
4. The method of claim 3, wherein the method further comprises: performing unsupervised anomaly detection on the low-dimensional feature vector by using an autoencoder, determining that the current data is abnormal data when a reconstruction error exceeds a dynamic threshold, and classifying the abnormal data by using a support vector machine, including: constructing a stacked autoencoder model including an input layer, a hidden layer and an output layer, and training the low-dimensional feature vector of the normal state in the complete data set to obtain the autoencoder by using a mean square error as a loss function and iteratively optimizing until the loss converges; fitting an error distribution curve based on a kernel density, and dynamically adjusting the threshold value according to a preset initial threshold value and a time decay factor to obtain the dynamic threshold value; inputting the low-dimensional feature vector into the autoencoder to calculate the reconstruction error, and determining that the current data is the abnormal data when the reconstruction error is greater than the dynamic threshold value; constructing a support vector machine model by using a radial basis function kernel function, inputting the abnormal data and the corresponding physical layer parameters, and outputting the abnormal data classification, the abnormal data classification including a non-lockout abnormality, a lockout relocking abnormality or a device abnormality.
5. The method of claim 4, wherein the method further comprises: based on the abnormal data classification, calculating the intensity variance of the low-dimensional feature vector by using a preset time window, drawing an intensity variance change curve along the transmission link, recursively dividing the link data into a normal subset or an abnormal subset by using a decision tree algorithm, and locating the abnormal node position, including: setting a sliding time window for the low-dimensional feature vector corresponding to the abnormal data classification, and calculating the intensity variance of each dimension of the low-dimensional feature vector within the time window; obtaining the physical topology information of the transmission link, mapping the intensity variance to the link transmission path according to the timestamps, and drawing an intensity curve of the intensity variance changing with the node position along the sending end to the receiving end of the link, the horizontal axis of the intensity curve being the node number and the vertical axis being the intensity variance; inputting the intensity curve into a preset decision tree model, if the output is an abnormal subset, extracting the first node position in the intensity curve that satisfies "the intensity variance exceeds the threshold value and the parameter fluctuation matches the abnormal data classification", and locating the abnormal node position combined with the node physical address.
6. The method of claim 4, wherein the method further comprises: based on the abnormal node position, if the proportion of the abnormal data classification is less than a preset threshold value, generating synthetic data of the digital residual spectrum and optical power by using a synthetic minority over-sampling technique, including: counting the number of samples of each type of abnormal data classification for historical data in a preset period corresponding to the abnormal node position, calculating the percentage of the number of each type of abnormal sample in the total number of abnormal samples to obtain the classification proportion. If the proportion of the classification is less than the preset threshold, it is determined that the corresponding anomaly is a minority class anomaly, and a synthetic minority over-sampling technique is used to generate the synthetic data of the digital residual spectrum and the optical power.
7. The method of claim 6, wherein the method further comprises: The synthetic data of the digital residual spectrum and the optical power generated by the synthetic minority over-sampling technique comprises: extracting the digital residual spectrum and the optical power time series data of the original data corresponding to the minority class anomaly and the abnormal node position; using the synthetic minority over-sampling technique, taking the feature vector of the digital residual spectrum and the time domain feature of the optical power as input, and according to the preset number of neighbor samples and the sampling rate, generating synthetic samples by random interpolation in the feature space of the sample data corresponding to the minority class anomaly in the original data and the neighbor samples; calculating the feature similarity of the synthetic samples and the sample data corresponding to the minority class anomaly in the original data, and defining the synthetic samples with feature similarity exceeding the preset value as the synthetic data.
8. An abnormal data analysis device based on optical fiber communication, characterized by, The device comprises: a multi-source acquisition module for acquiring physical layer parameters, digital features and link state data in real time through a receiver, performing noise suppression and missing value completion on the acquired data, and obtaining a complete data set; a feature vector module for calculating a reference digital spectrum under a normal state based on the complete data set, generating a digital residual spectrum by the difference between a real-time digital spectrum and the reference digital spectrum, and performing dimension reduction fusion on the digital residual spectrum and the physical layer parameters to obtain a low-dimensional feature vector; an anomaly classification module for performing unsupervised anomaly detection on the low-dimensional feature vector using an automatic encoder, determining abnormal data when the reconstruction error exceeds a dynamic threshold, and classifying the abnormal data using a support vector machine; a node positioning module for positioning the abnormal node position based on the classification of the abnormal data, calculating the intensity variance of the low-dimensional feature vector through a preset time window, drawing an intensity variance change curve along the transmission link, recursively dividing the link data into normal subsets or abnormal subsets using a decision tree algorithm, and positioning the abnormal node position; a sample synthesis module for generating synthetic data of the digital residual spectrum and the optical power using a synthetic minority over-sampling technique if the proportion of the abnormal data classification is less than a preset threshold based on the abnormal node position.
9. A control device, characterized by The device comprises: a memory and a processor, the memory storing a computer program capable of being loaded and executed by the processor to perform the method of any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, a computer program capable of being loaded and executed by the processor to perform the method of any one of claims 1 to 7.