Data link anomaly positioning method and device, electronic equipment and storage medium
By determining abnormal positioning indicators in the data link, extracting data characteristics and using autoencoder and machine learning models, the problem of abnormal positioning in complex data links is solved, monitoring efficiency and maintenance convenience are improved, cost is reduced, and the Operational performance of data links.
Patent Information
- Application Number
- CN202411056752.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2021-03-22
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art is difficult to efficiently locate abnormal links in complex large-scale data links, resulting in abnormal link operation, increasing maintenance costs, and affecting production efficiency.
By determining the transmission method of the data link, obtaining exception positioning indicators, extracting and processing data features, and using autoencoder and machine learning classification models to achieve accurate positioning of data link exceptions.
It improves the data link monitoring efficiency, reduces labor costs, facilitates data link maintenance, and improves the efficiency of data links.
Smart Images

Figure CN118941153A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of power systems, and in particular to a data link anomaly locating method, device, electronic equipment and storage medium. Background Art
[0002] With the advent of the big data era, more and more data is being generated, and data links play a very important role in the process of enterprise construction. Data runs through all aspects of business processing. With the continuous advancement of data platform construction, data resources are constantly enriched, and the scale of data link construction has gradually become large and complex. However, there are still problems such as difficulty in locating abnormal links in the data integration link. If the abnormal link cannot be located in a timely and effective manner, the link will not be able to operate normally, increase maintenance costs, and affect the production efficiency of the enterprise.
[0003] In the prior art, experts manually check the data link flow information, make judgments based on experience, and locate the abnormal links in the data link. However, due to the complexity of the data link scale, when faced with high-latitude, large-scale complex data, the prior art is difficult to achieve the requirements of abnormal location accuracy. Therefore, in order to ensure the high-quality and efficient operation of the data link, a method for locating data link anomalies is urgently needed to locate and diagnose the abnormal links that may exist in the data link, so as to improve the efficiency and effectiveness of the data integration link. Summary of the invention
[0004] The purpose of the present invention is to provide a data link anomaly locating method, device, electronic device and storage medium, which can improve the data link monitoring efficiency, reduce labor costs, facilitate data link maintenance, improve data link performance, and solve the above-mentioned problems existing in the prior art.
[0005] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0006] In a first aspect, the present invention provides a method for locating a data link anomaly, comprising the following steps:
[0007] Step 110: determine an abnormality location indicator according to the transmission mode of the data link, and obtain abnormality location data corresponding to each abnormality location indicator or a specific value corresponding to each abnormality location indicator; wherein the transmission mode of the data link is horizontal link transmission and / or vertical link transmission, and correspondingly, the abnormality location indicator is a horizontal link indicator and / or a vertical link indicator;
[0008] Step 120, processing the abnormal location data corresponding to each of the abnormal location indicators, extracting the data features of the abnormal location data, and determining the data features corresponding to each of the abnormal location indicators; Step 120 includes: Step 121, standardizing the abnormal location data corresponding to each of the abnormal location indicators to obtain standard abnormal location data; Step 122, pre-training the standard abnormal location data through an autoencoder to obtain data features that minimize reconstruction errors; wherein, extracting the data features of the abnormal location data is implemented by an autoencoder, and when the autoencoder is used for feature extraction, it is performed for different abnormal classifications, and the abnormal classifications include data timeliness alarms, data integrity alarms, and database connectivity alarms;
[0009] Step 130: Input the data features into a preset machine learning classification model to obtain an abnormal classification result.
[0010] Furthermore, the horizontal link indicators include at least one of the following: a horizontal link input data volume indicator, a horizontal link output data volume indicator, a horizontal link data transmission time consumption indicator, a horizontal link transmission task status indicator, a horizontal link time delay indicator, and a horizontal link checkpoint time delay indicator; the vertical link indicators include at least one of the following: a vertical link input data volume indicator, a vertical link output data volume indicator, a vertical link data transmission time consumption indicator, a vertical link local execution status indicator, a vertical link remote execution status indicator, and a vertical link callback status indicator.
[0011] Furthermore, the horizontal link input data volume indicator refers to the size of the input data in the horizontal link, the horizontal link output data volume indicator refers to the size of the output data in the horizontal link, the horizontal link data transmission time consumption indicator refers to the time consumed for each data transmission in the horizontal link, the horizontal link transmission task status indicator refers to the status of the transmission task in the horizontal link, the horizontal link time delay indicator refers to the difference between the operating system time of the last record processed by the replication process and the timestamp of this record in the trajectory file, and the horizontal link checkpoint time delay indicator refers to the time difference between the most recent checkpoint in the horizontal link and the current system time; the vertical link input data The data volume indicator refers to the size of the input data in the vertical link, the vertical link output data volume indicator refers to the size of the output data in the vertical link, the vertical link data transmission time consumption indicator refers to the time consumed for each data transmission in the vertical link, the vertical link local execution status indicator refers to the status of the local task in the case of cross-system remote task scheduling in the vertical link, the vertical link remote execution status indicator refers to the remote service called by the "DMS Scheduling" component through the local task configuration in the vertical link, wherein DMS is a scheduling management system, and the vertical link callback status indicator refers to the task triggered by a third-party system through an interface, and the asynchronous callback notifies the third-party system of the running result after the task is completed.
[0012] Furthermore, the abnormal classification results include data timeliness warnings, data integrity warnings, and database connectivity warnings.
[0013] Furthermore, before step 130, it also includes obtaining a training set corresponding to the anomaly location indicator, using a machine learning classification algorithm to perform model training, and obtaining a preset machine learning classification model; wherein the data in the training set includes anomaly location data corresponding to each of the anomaly location indicators, data features, and preset anomaly classification result labels.
[0014] Furthermore, the preset abnormal classification result label is obtained by encoding the preset abnormal classification result through one-hot encoding.
[0015] In a second aspect, the present invention further provides a data link anomaly locating device, the device comprising:
[0016] An abnormality location data acquisition module is used to determine an abnormality location index according to the transmission mode of the data link, and acquire abnormality location data corresponding to each of the abnormality location indexes;
[0017] A data feature determination module is used to process the abnormality location data corresponding to each of the abnormality location indicators to determine the data features corresponding to each of the abnormality location indicators; wherein the data feature determination module includes a standardization processing unit and a data feature determination unit, the standardization processing unit is used to perform standardization processing on the abnormality location data corresponding to each of the abnormality location indicators to obtain standard abnormality location data, and the data feature determination unit is used to pre-train the standard abnormality location data through an autoencoder to obtain data features that minimize reconstruction errors;
[0018] An abnormal classification result determination module is used to input the data features into a preset machine learning classification model to obtain an abnormal classification result;
[0019] The preset machine learning classification model determination module is used to obtain a training set corresponding to the anomaly location indicator before inputting the data features into the preset machine learning classification model to obtain the anomaly classification result, and use the machine learning classification algorithm to train the model to obtain the preset machine learning classification model; wherein the data in the training set includes anomaly location data corresponding to each anomaly location indicator, data features, and preset anomaly classification result labels.
[0020] In a third aspect, the present invention further provides an electronic device, the device comprising:
[0021] one or more processors;
[0022] a storage device for storing one or more programs,
[0023] When the one or more programs are executed by the one or more processors, the one or more processors implement the data link anomaly locating method.
[0024] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which implements the data link anomaly locating method when executed by a processor.
[0025] Compared with the prior art, the beneficial technical effects of the present invention are:
[0026] The data link anomaly locating method, device, electronic device and storage medium of the present invention solve the problem of data link anomaly locating, can effectively improve the data link monitoring efficiency, reduce labor costs, facilitate data link maintenance, and improve the efficiency of the data link. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0028] Figure 1 This is a flow chart of a data link anomaly locating method according to a first embodiment of the present invention;
[0029] Figure 2 A schematic diagram of an indicator system for locating data link anomalies in Embodiment 1 of the present invention;
[0030] Figure 3 This is a schematic diagram of the structure of a data link anomaly locating device according to Embodiment 2 of the present invention;
[0031] Figure 4 This is a schematic diagram of the structure of an electronic device according to Embodiment 3 of the present invention.
[0032] Explanation of the reference numerals: 210, abnormality location data acquisition module; 220, data feature determination module; 230, abnormality classification result determination module; 310, processor; 320, memory; 330, input device; 340, output device. DETAILED DESCRIPTION
[0033] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.
[0034] Embodiment 1
[0035] Figure 1 1 is a flow chart of a data link anomaly locating method according to a first embodiment of the present invention. This embodiment is applicable to the case of locating anomalies in large-scale data links. The method can be executed by a data link anomaly locating device. The device can be implemented by software and / or hardware. The device can be integrated in an electronic device such as a computer. Figure 1 As shown, the method specifically includes:
[0036] Step 110: determine anomaly location indicators according to the transmission mode of the data link, and obtain anomaly location data corresponding to each anomaly location indicator.
[0037] The transmission mode of the data link may be horizontal link transmission and / or vertical link transmission. For example, horizontal link transmission may be primary data transmission between national databases (headquarters), and vertical link transmission may be secondary data transmission between national databases and local databases (branches). By performing horizontal and vertical link monitoring on the data link, abnormal location under the two-level scheduling of the headquarters and branches can be achieved.
[0038] The abnormal location indicator may be an indicator corresponding to the transmission mode and related to the abnormal location. For example, the abnormal location indicator may be a horizontal link indicator and / or a vertical link indicator. Specifically, the abnormal location indicator may be determined based on the monitoring scheme, monitoring indicator list and data link transmission technology provided by the national database.
[0039] In an optional implementation of an embodiment of the present invention, an abnormality positioning indicator is determined according to a transmission mode of a data link, including: determining a lateral link indicator according to a transmission mode of a lateral link; and / or determining a longitudinal link indicator according to a transmission mode of a longitudinal link.
[0040] Figure 2 Schematic diagram of the indicator system for locating data link anomalies in the first embodiment of the present invention. Figure 2 As shown, in an optional implementation of an embodiment of the present invention, the lateral link indicator includes at least one of the following: a lateral link input data volume indicator, a lateral link output data volume indicator, a lateral link data transmission time consumption indicator, a lateral link transmission task status indicator, a lateral link time delay indicator, and a lateral link checkpoint time delay indicator.
[0041] Among them, the horizontal link input data volume indicator refers to the size of the input data in the horizontal link. The horizontal link output data volume indicator refers to the size of the output data in the horizontal link. The horizontal link data transmission time consumption indicator refers to the time consumed for each data transmission in the horizontal link. The horizontal link transmission task status indicator refers to the status of the transmission task in the horizontal link. The horizontal link time delay indicator (lagatchkpt) refers to the difference between the operating system time when the replication process processes the last record and the timestamp of this record in the trail file. The horizontal link checkpoint time delay indicator (timesincechkpt) refers to the time difference between the most recent checkpoint in the horizontal link and the current system time.
[0042] like Figure 2 As shown, in an optional implementation manner of an embodiment of the present invention, the longitudinal link indicator includes at least one of the following: a longitudinal link input data volume indicator, a longitudinal link output data volume indicator, a longitudinal link data transmission time consumption indicator, a longitudinal link local execution status indicator, a longitudinal link remote execution status indicator, and a longitudinal link callback status indicator.
[0043] Among them, the vertical link input data volume indicator refers to the size of the input data in the vertical link. The vertical link output data volume indicator refers to the size of the output data in the vertical link. The vertical link data transmission time consumption indicator refers to the time consumed for each data transmission in the vertical link. The vertical link local execution status indicator refers to the status of the local task in the case of cross-system remote task scheduling in the vertical link. The vertical link remote execution status indicator refers to the remote service called by the local task configuration in the vertical link through the "DMS Scheduling" component, where DMS is the scheduling management system. The vertical link callback status indicator refers to the third-party system triggering the task through the interface, and the asynchronous callback notifies the third-party system of the operation result after the task is completed.
[0044] The abnormality location data may be data corresponding to the abnormality location indicator. The abnormality location data may be a specific value corresponding to the indicator. The abnormality location data may be a value collected by monitoring the data link in real time, and may be a real value or a simulated value.
[0045] Step 120: Process the anomaly location data corresponding to each anomaly location indicator to determine the data features corresponding to each anomaly location indicator.
[0046] Among them, the processing of the abnormal location data can be to extract the data features of the abnormal location data as the data features of the corresponding abnormal location indicators. The abnormal location data corresponding to each abnormal location indicator can be large. The data link abnormal location method of the embodiment of the present invention can be applicable to large-scale data links. There can be multiple methods for extracting data features, for example, it can be implemented through a deep learning model; or, it can be implemented through an autoencoder, etc.
[0047] In an optional implementation of an embodiment of the present invention, the abnormality location data corresponding to each abnormality location indicator is processed to determine the data features corresponding to each abnormality location indicator, including: standardizing the abnormality location data corresponding to each abnormality location indicator to obtain standard abnormality location data; pre-training the standard abnormality location data through an autoencoder to obtain data features that minimize reconstruction errors.
[0048] Among them, the standardization processing can be dimensionless processing of the anomaly location data, so that the processed anomaly location data, that is, the standard anomaly location data, is located in the interval [0, 1]. The standardization processing can avoid the influence of the data dimension on the data characteristics and improve the accuracy of anomaly location.
[0049] For example, the anomaly location data corresponding to each anomaly location indicator can be expressed as: After standardization, dimensionless standard anomaly location data is obtained. iis the i-th abnormal location data corresponding to the current abnormal location indicator; max(x) is the maximum value of the abnormal location data corresponding to the current abnormal location indicator; min(x) is the minimum value of the abnormal location data corresponding to the current abnormal location indicator; y i is the standard anomaly location data obtained after the i-th anomaly location data corresponding to the current anomaly location indicator is standardized, y i It is dimensionless and lies in the interval [0, 1].
[0050] In an embodiment of the present invention, the data features of the abnormal location data can be extracted by using an autoencoder (AE). An autoencoder is an artificial neural network that can learn an efficient representation of input data through unsupervised learning. The pre-training of the autoencoder includes encoding, decoding and reconstruction errors.
[0051] Exemplarily, the pre-training using the autoencoder can be:
[0052] First, the standard anomaly location data y is encoded as the input of the hidden layer to obtain the encoded data h. Where h is the extracted feature. Encoding refers to the process of mapping y to the implicit representation h. Where y∈R, h∈R. The calculation formula of h can be h=σ h (W1y+b1). Among them, σ h is the activation function, W1 is the weight matrix, and b1 is the bias vector. W1 and b1 can be determined by random initialization and automatically updated in the subsequent model parameter adjustment. h It can be determined according to needs, for example, Sigmoid function, Tanh function or ReLU function can be selected.
[0053] Secondly, the encoded data h is decrypted to obtain the reconstructed data z. Decoding refers to the process of mapping the implicit representation h to the output layer z to reconstruct the input y. The calculation formula of z can be z = σ y (W2h+b2). Among them, σ y is the activation function, W2 is the weight matrix, and b2 is the bias vector. W2 and b2 can be determined by random initialization and automatically updated in the subsequent model parameter adjustment. y It can be determined according to needs, for example, Sigmoid function, Tanh function or ReLU function can be selected.
[0054] Finally, the loss function between y and z is calculated, and the parameters (such as the weight matrix) are adjusted to repeat the pre-training to minimize the reconstruction error until the loss function converges. The loss function can be determined by: Among them, the first term is the autoencoder loss function, z iis the output data, y i is the input data, the second term is the regularization term, λ is the regularization parameter, and w j is the weight matrix; λ can be a manually set value, w j It can be determined according to the current weight matrix.
[0055] In an embodiment of the present invention, the data features that minimize the reconstruction error can be obtained by retaining the first half of the autoencoder in the autoencoder model obtained above, that is, only using the autoencoder to extract data features. Using the autoencoder for feature extraction and combining it with a preset machine learning classification model can overcome the limitations of a single machine learning algorithm for anomaly location and improve the accuracy of anomaly location.
[0056] It should be noted that when the autoencoder is used for feature extraction, it can be performed for different abnormal classifications. In an optional implementation of the embodiment of the present invention, the abnormal classification includes: data timeliness alarm, data integrity alarm and database connectivity alarm.
[0057] Among them, data timeliness alarm refers to the abnormality of data transmission time on the data link, such as network interruption, transmission line abnormality, etc. Data integrity alarm refers to the abnormality of data volume transmitted on the data link, such as data loss, data redundancy, etc. Database connectivity alarm refers to the abnormality of data input source and output object, such as database connection interruption, database damage, etc.
[0058] When performing feature extraction, each abnormal classification can be encoded. For example, the abnormal classification can be one-hot encoded. Among them, one-hot encoding uses 0 and 1 to represent some parameters, uses an N-bit state register, and encodes N states.
[0059] For example, when the anomaly classification is data timeliness warning, data integrity warning, and database connectivity warning, the corresponding one-hot encodings may be: [100, 010, 001]. Processing the anomaly classification through one-hot encoding can facilitate model training.
[0060] Step 130: Input the data features into a preset machine learning classification model to obtain an abnormal classification result.
[0061] The data feature may be a feature h obtained by encoding through an autoencoder. In an embodiment of the present invention, the feature extracted by the autoencoder may be input into a machine learning classification model to train the machine learning model, which may overcome the limitation of anomaly location by a single machine learning algorithm and improve the accuracy of anomaly location.
[0062] The preset machine learning classification model may be a machine learning classification model obtained by pre-training according to the features. The preset machine learning classification model may be used to determine the abnormal classification result according to the features. There are many commonly used machine learning classification algorithms, such as support vector machine, logarithmic probability regression, random forest, and naive Bayes.
[0063] In an optional implementation of the embodiment of the present invention, the abnormal classification result includes: data timeliness alarm, data integrity alarm and database connectivity alarm.
[0064] In an optional implementation of an embodiment of the present invention, before inputting data features into a preset machine learning classification model to obtain anomaly classification results, it also includes: obtaining a training set corresponding to the anomaly location indicator, using a machine learning classification algorithm to perform model training, and obtaining a preset machine learning classification model; wherein the data in the training set includes anomaly location data corresponding to each anomaly location indicator, data features, and preset anomaly classification result labels.
[0065] The training set may be a sample set consisting of data features determined based on historical anomaly location data. For example, the training set may be features extracted by an autoencoder after being standardized based on historical anomaly location data. The training set may include a preset anomaly classification result label, which may be a one-hot encoding.
[0066] Exemplarily, the training of a preset machine learning classification model is described by taking a support vector machine as an example. In machine learning, a support vector machine (SVM) is a supervised learning model with an associated learning algorithm that analyzes data for classification and regression analysis. Given a set of training examples, each example is labeled as belonging to one or the other of two categories, and the SVM training algorithm builds a model that assigns new examples to one category or the other, making it a non-probabilistic binary linear classifier. The SVM model represents examples as points in space, mapping the examples of separate categories to be divided by as wide a clear gap as possible. New examples are then mapped to the same space and predicted to belong to which category based on which edge they fall. In addition to performing linear classification, SVM can also effectively perform nonlinear classification using kernel tricks, implicitly mapping its input to a high-dimensional feature space.
[0067] Since some classification algorithms (such as support vector machines) are binary classification machine learning algorithms, in order to apply these algorithms for multi-classification, commonly used methods include one-to-many and one-to-one.
[0068] Among them, the one-to-many method means that for a sample set containing N class labels, N binary classification models will be trained. In the training phase, samples of one class are taken as positive classes each time, and samples of all other classes are taken as negative classes to train N models. In the testing phase, if only one classifier predicts a sample as a positive class, the sample is classified into the class label corresponding to the positive class. If multiple classifiers predict the sample as a positive class, the sample is classified into the class corresponding to the result of the classifier with the highest classification confidence.
[0069] The one-to-one method means that in samples with N class labels, a classifier will be trained for every two classes of samples, so a total of In the prediction phase, new samples are submitted to all classifiers for classification and the The results are collected by voting, and the category with the most votes is taken as the final classification result of the sample.
[0070] In the embodiment of the present invention, the key to actually using the machine learning algorithm for training is the construction of a data set. For example, 1,000 pieces of abnormal location data of different abnormal types can be taken and randomly shuffled, and the data set can be divided into a training set, a validation set, and a test set at a ratio of 8:1:1. The data of the training set is used for pre-training of the autoencoder and training of the machine learning classifier. The model parameters are continuously optimized by comparing the accuracy of the validation set, and finally a higher classification accuracy is obtained on the test set. The obtained model is the preset machine learning classification model.
[0071] The preset machine learning classification model obtained in the embodiment of the present invention can determine the abnormal classification result, and then accurately locate the abnormal link according to the abnormal classification result. For example, if the abnormal classification result is a data timeliness alarm, the abnormality can be located as a network interruption or a transmission line abnormality. By determining the abnormal classification result by the preset machine learning classification model, it can be ensured that the abnormal link of the data link is checked and the problem is solved in time, the monitoring efficiency and maintenance efficiency of the data full-link monitoring tool are improved, and the normal and efficient operation of the data full-link system is guaranteed.
[0072] In an optional implementation of the embodiment of the present invention, the preset abnormal classification result label is obtained by encoding the preset abnormal classification result through one-hot encoding.
[0073] The technical solution of this embodiment determines the abnormal location indicator according to the transmission mode of the data link, and obtains the abnormal location data corresponding to each abnormal location indicator; processes the abnormal location data corresponding to each abnormal location indicator to determine the data features corresponding to each abnormal location indicator; inputs the data features into a preset machine learning classification model to obtain an abnormal classification result, thereby solving the problem of abnormal location monitoring of the data link, facilitating the determination of abnormal location, improving the efficiency of data link monitoring, facilitating the maintenance of the data link, and reducing labor costs.
[0074] Embodiment 2
[0075] Figure 3 Schematic diagram of the structure of the data link anomaly locating device according to the second embodiment of the present invention. Figure 3 The device includes: an abnormality positioning data acquisition module 210, a data feature determination module 220 and an abnormality classification result determination module 230.
[0076] The abnormality location data acquisition module 210 is used to determine abnormality location indicators according to the transmission mode of the data link, and to acquire abnormality location data corresponding to each abnormality location indicator.
[0077] The data feature determination module 220 is used to process the anomaly location data corresponding to each anomaly location indicator to determine the data feature corresponding to each anomaly location indicator.
[0078] The abnormal classification result determination module 230 is used to input data features into a preset machine learning classification model to obtain an abnormal classification result.
[0079] Optionally, the abnormality positioning data acquisition module 210 includes: an abnormality positioning index determination unit, used to determine the horizontal link index according to the transmission mode of the horizontal link; and / or, determine the vertical link index according to the transmission mode of the vertical link.
[0080] Optionally, the horizontal link indicators include at least one of the following: a horizontal link input data volume indicator, a horizontal link output data volume indicator, a horizontal link data transmission time consumption indicator, a horizontal link transmission task status indicator, a horizontal link time delay indicator, and a horizontal link checkpoint time delay indicator; the vertical link indicators include at least one of the following: a vertical link input data volume indicator, a vertical link output data volume indicator, a vertical link data transmission time consumption indicator, a vertical link local execution status indicator, a vertical link remote execution status indicator, and a vertical link callback status indicator.
[0081] Optionally, the data feature determination module 220 includes: a standardization processing unit, used to standardize the anomaly location data corresponding to each anomaly location indicator to obtain standard anomaly location data; a data feature determination unit, used to pre-train the standard anomaly location data through an autoencoder to obtain data features that minimize the reconstruction error.
[0082] Optionally, abnormal classification results include data timeliness alarms, data integrity alarms, and database connectivity alarms.
[0083] Optionally, the device also includes: a preset machine learning classification model determination module, which is used to obtain a training set corresponding to the anomaly location indicator before inputting the data features into the preset machine learning classification model to obtain the anomaly classification result, and use the machine learning classification algorithm to perform model training to obtain the preset machine learning classification model.
[0084] The data in the training set includes anomaly location data corresponding to each anomaly location indicator, data features, and preset anomaly classification result labels.
[0085] Optionally, the preset abnormal classification result label is obtained by encoding the preset abnormal classification result through one-hot encoding.
[0086] The data link anomaly locating device provided in the embodiment of the present invention can execute the data link anomaly locating method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.
[0087] Embodiment 3
[0088] Figure 4 is a schematic diagram of the structure of an electronic device according to a third embodiment of the present invention. Figure 4 As shown, the device includes:
[0089] one or more processors 310, Figure 4 A processor 310 is taken as an example;
[0090] Memory 320;
[0091] The device may further include: an input device 330 and an output device 340 .
[0092] The processor 310, memory 320, input device 330 and output device 340 in the device can be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.
[0093] The memory 320, as a non-transitory computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to a data link anomaly locating method in an embodiment of the present invention (for example, the attached Figure 4 The processor 310 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 320, that is, a data link anomaly locating method of the above method embodiment is implemented, namely:
[0094] Determine anomaly location indicators according to the transmission mode of the data link, and obtain anomaly location data corresponding to each of the anomaly location indicators;
[0095] Processing the abnormality location data corresponding to each of the abnormality location indicators to determine the data features corresponding to each of the abnormality location indicators;
[0096] The data features are input into a preset machine learning classification model to obtain an abnormal classification result.
[0097] The memory 320 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 320 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory 320 may optionally include a memory remotely arranged relative to the processor 310, and these remote memories may be connected to the terminal device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0098] The input device 330 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the computer device. The output device 340 may include a display device such as a display screen.
[0099] Embodiment 4
[0100] A computer-readable storage medium according to a fourth embodiment of the present invention stores a computer program, which, when executed by a processor, implements a data link anomaly locating method provided in an embodiment of the present invention:
[0101] Determine anomaly location indicators according to the transmission mode of the data link, and obtain anomaly location data corresponding to each of the anomaly location indicators;
[0102] Processing the abnormality location data corresponding to each of the abnormality location indicators to determine the data features corresponding to each of the abnormality location indicators;
[0103] The data features are input into a preset machine learning classification model to obtain an abnormal classification result.
[0104] Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: electrical connections with one or more wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this document, computer-readable storage media can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.
[0105] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0106] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.
[0107] Computer program code for performing the operations of the present invention may be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).
[0108] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.
Claims
1. A data link anomaly location method, characterized in that: The following steps are involved: Step 110: determine an abnormality location indicator according to the transmission mode of the data link, and obtain abnormality location data corresponding to each abnormality location indicator or a specific value corresponding to each abnormality location indicator; wherein the transmission mode of the data link is horizontal link transmission and / or vertical link transmission, and correspondingly, the abnormality location indicator is a horizontal link indicator and / or a vertical link indicator; Step 120, processing the abnormal location data corresponding to each of the abnormal location indicators, extracting the data features of the abnormal location data, and determining the data features corresponding to each of the abnormal location indicators; Step 120 includes: Step 121, standardizing the abnormal location data corresponding to each of the abnormal location indicators to obtain standard abnormal location data; Step 122, pre-training the standard abnormal location data through an autoencoder to obtain data features that minimize reconstruction errors; wherein, extracting the data features of the abnormal location data is implemented by an autoencoder, and when the autoencoder is used for feature extraction, it is performed for different abnormal classifications, and the abnormal classifications include data timeliness alarms, data integrity alarms, and database connectivity alarms; Step 130: Input the data features into a preset machine learning classification model to obtain an abnormal classification result.
2. The data link anomaly locating method according to claim 1, characterized in that: The lateral link indicator includes at least one of the following: a lateral link input data volume indicator, a lateral link output data volume indicator, a lateral link data transmission time consumption indicator, a lateral link transmission task status indicator, a lateral link time delay indicator, and a lateral link checkpoint time delay indicator; The longitudinal link indicator includes at least one of the following: a longitudinal link input data volume indicator, a longitudinal link output data volume indicator, a longitudinal link data transmission time consumption indicator, a longitudinal link local execution status indicator, a longitudinal link remote execution status indicator, and a longitudinal link callback status indicator.
3. The data link anomaly locating method according to claim 2, characterized in that: The lateral link input data volume indicator refers to the size of the input data in the lateral link, the lateral link output data volume indicator refers to the size of the output data in the lateral link, the lateral link data transmission time consumption indicator refers to the time consumed for each data transmission in the lateral link, the lateral link transmission task status indicator refers to the status of the transmission task in the lateral link, the lateral link time delay indicator refers to the difference between the operating system time when the replication process processes the last record and the timestamp of this record in the trajectory file, and the lateral link checkpoint time delay indicator refers to the time difference between the most recent checkpoint in the lateral link and the current system time; The longitudinal link input data volume indicator refers to the size of the input data in the longitudinal link, the longitudinal link output data volume indicator refers to the size of the output data in the longitudinal link, the longitudinal link data transmission time consumption indicator refers to the time consumed for each data transmission in the longitudinal link, the longitudinal link local execution status indicator refers to the status of the local task in the case of cross-system remote task scheduling in the longitudinal link, the longitudinal link remote execution status indicator refers to the remote service called by the "DMS Scheduling" component through the local task configuration in the longitudinal link, wherein DMS is a scheduling management system, and the longitudinal link callback status indicator refers to the task triggered by a third-party system through an interface, and the asynchronous callback notifies the third-party system of the running result after the task is completed.
4. The data link anomaly locating method according to claim 1, characterized in that: The abnormal classification results include data timeliness warnings, data integrity warnings, and database connectivity warnings.
5. The data link anomaly locating method according to any one of claims 1 to 4, characterized in that: Before step 130, it also includes obtaining a training set corresponding to the anomaly location indicator, using a machine learning classification algorithm to perform model training, and obtaining a preset machine learning classification model; wherein the data in the training set includes anomaly location data corresponding to each of the anomaly location indicators, data features, and preset anomaly classification result labels.
6. The data link anomaly locating method according to claim 5, characterized in that: The preset abnormal classification result label is obtained by encoding the preset abnormal classification result through one-hot encoding.
7. A data link anomaly locating device, characterized in that: include: An abnormality location data acquisition module is used to determine an abnormality location index according to the transmission mode of the data link, and acquire abnormality location data corresponding to each of the abnormality location indexes; A data feature determination module is used to process the abnormality location data corresponding to each of the abnormality location indicators to determine the data features corresponding to each of the abnormality location indicators; wherein the data feature determination module includes a standardization processing unit and a data feature determination unit, the standardization processing unit is used to perform standardization processing on the abnormality location data corresponding to each of the abnormality location indicators to obtain standard abnormality location data, and the data feature determination unit is used to pre-train the standard abnormality location data through an autoencoder to obtain data features that minimize reconstruction errors; An abnormal classification result determination module is used to input the data features into a preset machine learning classification model to obtain an abnormal classification result; The preset machine learning classification model determination module is used to obtain a training set corresponding to the anomaly location indicator before inputting the data features into the preset machine learning classification model to obtain the anomaly classification result, and use the machine learning classification algorithm to train the model to obtain the preset machine learning classification model; wherein the data in the training set includes anomaly location data corresponding to each anomaly location indicator, data features, and preset anomaly classification result labels.
8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data link anomaly locating method according to any one of claims 1-6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the data link anomaly locating method according to any one of claims 1 to 6 is implemented.
