A multi-source data integration method for IT operation and maintenance big data

Abstract

The present application relates to the technical field of multi-source data integration, and specifically to a multi-source data integration method for IT operation and maintenance big data, the method comprising: determining the motion vector between each acquisition moment and each reference moment to determine the motion direction vector of each acquisition moment; determining the system state similarity credibility between any two acquisition moments; based on the similarity between the motion direction vectors at any two acquisition moments, and the system state similarity credibility, determining the system state similarity value between any two acquisition moments, and combining the difference between the historical monitoring vectors at any two acquisition moments, determining the system state motion difference between any two acquisition moments, and combining the dimensionality reduction algorithm to integrate the operation and maintenance big data at the current moment and detect operation and maintenance faults. The present application improves the quality of multi-source data integration by calculating the system state motion difference through motion trajectory.

Description

Technical Field

[0001] The present application relates to the technical field of multi-source data integration, and in particular to a multi-source data integration method for IT operation and maintenance big data. Background Art

[0002] Distributed cloud is a cloud computing model based on distributed deployment and unified operation and management. It is a set of cloud technology product systems built on technologies such as containers, microservices, and DevOps. While distributed cloud brings convenience to enterprises, it also brings great challenges to platform operation and maintenance. Distributed cloud has many and complex management objects, generating a large amount of data of different scenarios at all times. However, current monitoring and operation tools usually only focus on a certain type of specific data or a certain scenario, which leads to the neglect of the overall data relevance of distributed cloud, forming data islands, making it difficult for users to discover, locate, and solve faults, and bringing operational difficulties to distributed cloud.

[0003] In order to eliminate the difficulties in operation and maintenance management caused by data silos, existing methods mostly use data dimensionality reduction to process data, and then obtain the classification features of the data to complete fault location during the operation and maintenance process. Since the structure of operation and maintenance data is complex and the operation and maintenance data often exhibits a nonlinear structure, manifold learning is generally used for data dimensionality reduction. Existing manifold learning data dimensionality reduction methods, such as the Isomap algorithm, usually only use simple statistical methods such as the shortest path as an indicator for obtaining geodesics, without considering the time series information of the cloud platform data itself, resulting in unreasonable data dimensionality reduction and reducing the quality of multi-source data integration. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a multi-source data integration method for IT operation and maintenance big data to solve the existing problems.

[0005] The multi-source data integration method of IT operation and maintenance big data in this application adopts the following technical solution:

[0006] An embodiment of the present application provides a multi-source data integration method for IT operation and maintenance big data, the method comprising the following steps:

[0007] In the distributed cloud monitoring system, multiple monitoring data at the current moment and at each collection moment within a preset time period before the current moment are obtained, and all the monitoring data at the current moment are combined into a monitoring vector at the current moment, and multiple monitoring data at each collection moment are combined into a historical monitoring vector at each collection moment;

[0008] All the acquisition moments in the neighborhood of each acquisition moment are used as reference moments of each acquisition moment. Based on the time sequence and historical monitoring vector of each acquisition moment and its reference moments, the interval vector between each acquisition moment and its reference moments is determined. In combination with the time interval between each acquisition moment and its reference moments, the motion vector between each acquisition moment and its reference moments is determined to determine the motion direction vector of each acquisition moment.

[0009] Based on the distance between each collection moment and the historical monitoring vector of all other collection moments, the mean value of the system state distance at each collection moment is determined, and combined with the distance between the historical monitoring vectors of any two collection moments, the similarity credibility of the system state between any two collection moments is determined;

[0010] Based on the similarity between the motion direction vectors at any two acquisition moments and the system state similarity credibility, determine the system state similarity value between any two acquisition moments, and combine the difference between the historical monitoring vectors at any two acquisition moments to determine the system state motion difference between any two acquisition moments;

[0011] Based on the system state movement difference between the monitoring vector at the current moment and any two vectors in the historical monitoring vectors of a preset number of collection moments before the current moment, a distance matrix at the current moment is constructed, and combined with a dimensionality reduction algorithm, the operation and maintenance big data at the current moment is integrated and operation and maintenance faults are detected.

[0012] Preferably, the expression of the interval vector between each acquisition moment and each reference moment is: ; In the formula, represents the interval vector between the acquisition time m and its nth reference time; , They respectively represent the historical monitoring vector at the collection time m and the historical monitoring vector at the nth reference time of the collection time m.

[0013] Preferably, the method for determining the motion direction vector at each acquisition moment is:

[0014] The result of multiplying the normalized value of the time interval between each acquisition moment and each reference moment by the unit vector number of the interval vector is used as the motion vector between each acquisition moment and each reference moment.

[0015] Preferably, the motion direction vector of each acquisition moment is the sum vector of the motion vectors between each acquisition moment and all its reference moments.

[0016] Preferably, the method for determining the mean value of the system state distance at each acquisition moment is:

[0017] Calculate the distance between the historical monitoring vector at each collection moment and the historical monitoring vector at all other collection moments, and take the average of the first preset value of distances in the results of ascending distance arrangement as the average system state distance at each collection moment.

[0018] Preferably, the system state similarity credibility between any two collection moments is the ratio of the mean of the system state distance between any two collection moments to the distance between the historical monitoring vectors corresponding to the two collection moments.

[0019] Preferably, the system state similarity value between any two acquisition moments is the product of the similarity between the motion direction vectors at any two acquisition moments and the system state similarity credibility.

[0020] Preferably, the expression of the system state motion difference between any two acquisition moments is: ; In the formula, represents the difference in system state motion between acquisition time k and acquisition time r; represents the distance between the historical monitoring vector at the collection time k and the historical monitoring vector at the collection time r; Represents the system state similarity value between the acquisition time k and the acquisition time r.

[0021] Preferably, the distance matrix at the current moment is constructed as follows:

[0022] Set fault labels for all historical monitoring vectors collected at all times before the current time, select a preset number of historical monitoring vectors under each fault label, and record them and the monitoring vector at the current time as fault detection vectors;

[0023] The distance matrix D at the current moment is expressed as: ; In the formula, b, c, d and all elements outside the diagonal represent the system state motion difference between any two fault detection vectors; the elements on the main diagonal are 0.

[0024] Preferably, the step of integrating the current operation and maintenance big data and detecting operation and maintenance failures includes:

[0025] The distance matrix at the current moment is used as the input of the dimension reduction algorithm, and a two-dimensional coordinate point set is output, where each coordinate point corresponds to a fault detection vector;

[0026] Assign the fault labels corresponding to all the fault detection vectors except the fault detection vector at the current moment to the corresponding coordinate points in the two-dimensional coordinate point set, and use the corresponding coordinate points and their fault labels as inputs of the support vector machine, and output a classification model of the two-dimensional coordinate point set;

[0027] The coordinate points corresponding to the fault detection vector at the current moment in the two-dimensional data point set are used as the input of the classification model, and the fault label at the current moment is output.

[0028] This application has at least the following beneficial effects:

[0029] This application constructs an interval vector by analyzing the temporal sequence and historical monitoring vector of each acquisition moment and all acquisition moments in its neighborhood. The beneficial effect is that it helps to identify the relative position and change trend between different acquisition moments, and provides a basis for the subsequent calculation of the motion direction vector;

[0030] The time interval between each acquisition moment and each reference moment of the present application is combined with the interval vector to determine the motion direction vector of each acquisition moment. The beneficial effect is that it can reflect the changing trend of the system state over time and help understand the changing law of the system state;

[0031] The present application determines the similarity credibility of the system state between any two collection moments by analyzing the distance of the historical monitoring vector between each collection moment and other collection moments, and combining the distance between the historical monitoring vectors of any two collection moments. The beneficial effect is that the similarity of the system state between two moments can be more accurately evaluated by combining the mean value of the system state distance and the distance between the historical monitoring vectors.

[0032] The present application determines the system state motion difference between any two acquisition moments by analyzing the similarity between the motion direction vectors at any two acquisition moments and the difference between the historical monitoring vectors at any two acquisition moments, and combines the system state similarity credibility. The beneficial effect is that the combination of the similarity and the difference between the historical monitoring vectors can more accurately evaluate the motion difference of the system state at two moments.

[0033] This application constructs a distance matrix at the current moment based on the system state motion difference between any two vectors in the historical monitoring vectors of a preset number of collection moments before the current moment, and combines the dimensionality reduction algorithm to integrate the operation and maintenance big data at the current moment and detect operation and maintenance failures. The beneficial effect is that the system state motion difference is calculated through motion trajectory, thereby improving the quality of data integration.

[0034] When performing data integration, this application obtains the motion trajectory of the distributed cloud system state by extracting the time series information in the historical data, calculates the system state motion difference through the motion trajectory, and then constructs a distance matrix based on this. Compared with the distance matrix obtained by the traditional manifold learning method, it can better distinguish the different working states of the distributed cloud and improve the quality of multi-source data integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0036] Figure 1 A flowchart of a multi-source data integration method for IT operation and maintenance big data provided by an embodiment of the present application;

[0037] Figure 2 A schematic diagram of a motion vector acquisition process provided by an embodiment of the present application;

[0038] Figure 3 A schematic diagram of a system state similarity value extraction process provided by an embodiment of the present application;

[0039] Figure 4 A distance matrix construction process framework diagram provided for one embodiment of the present application;

[0040] Figure 5 A flowchart of multi-source data integration and fault identification provided for one embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to further explain the technical means and effects adopted by the present application to achieve the predetermined invention purpose, the following is a detailed description of the multi-source data integration method of IT operation and maintenance big data proposed in the present application, its specific implementation method, structure, features and effects, in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0042] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0043] The following is a detailed description of a method for integrating multi-source data of IT operation and maintenance big data provided by the present application with reference to the accompanying drawings.

[0044] An embodiment of the present application provides a multi-source data integration method for IT operation and maintenance big data. Specifically, the following multi-source data integration method for IT operation and maintenance big data is provided. Figure 1 , the method comprises the following steps:

[0045] Step S1: In the distributed cloud monitoring system, obtain multiple monitoring data at the current moment and at each collection moment within a preset time period before the current moment, and combine all the monitoring data at the current moment into a monitoring vector at the current moment, and combine the multiple monitoring data at each collection moment into a historical monitoring vector at each collection moment.

[0046] For distributed cloud, a large amount of monitoring data is generated at every moment, and there is no fixed monitoring model to centrally process these monitoring data. This embodiment obtains monitoring data in real time through the monitoring system of distributed cloud, specifically: obtains multiple monitoring data at the current moment and at each collection moment within a preset time length t before the current moment, wherein the data sampling interval is set to T, that is, data is collected once every interval T.

[0047] It should be noted that the types of monitoring data and historical monitoring data in this embodiment are the same, specifically including 10 monitoring indicators: CPU usage, memory usage, disk IO, network traffic, server status, fault information, network traffic, bandwidth occupancy, delay, and packet loss rate. Implementers can also choose data collection objects based on specific circumstances, and this embodiment does not impose any special restrictions.

[0048] In addition, it should be understood that the values ​​of the preset duration t and the sampling interval T are artificially set. In this embodiment, the value of the preset duration t is 24h, and the value of the sampling interval T is 1min. The implementer can also set it according to the specific situation. This embodiment does not impose any special restrictions. Further, in order to simplify the calculation and eliminate the influence of the data dimension, all kinds of data at each collection time within the preset duration are normalized. There are many commonly used normalization methods. In this embodiment, the maximum and minimum value normalization method is used to process the collected data. The implementer can also use other normalization methods such as z-score normalization method. This embodiment does not impose any special restrictions on the selection of normalization methods.

[0049] Among them, the maximum and minimum value normalization method is a well-known technology in the field of data processing, and the specific process of normalizing the data will not be repeated here.

[0050] Furthermore, based on the result after normalization, all kinds of monitoring data at the current moment are combined into a monitoring vector at the current moment, and multiple kinds of monitoring data at each collection moment are combined into a historical monitoring vector at each collection moment.

[0051] Step S2: Take all the collection moments in the neighborhood of each collection moment as the reference moments of each collection moment, determine the interval vector between each collection moment and its reference moments based on the time sequence and historical monitoring vector between each collection moment and its reference moments, and determine the motion vector between each collection moment and its reference moments in combination with the time interval between each collection moment and its reference moments, so as to determine the motion direction vector of each collection moment.

[0052] Manifold learning is a data dimensionality reduction method. In the field of multi-source data integration, data dimensionality reduction can fuse the features of multivariate data, thereby avoiding the generation of data islands. In the IT operation and maintenance management of distributed clouds, the equipment structure of the entire distributed cloud is complex. When one device fails, it will affect other devices. Therefore, a large amount of multi-source data needs to be fused to complete accurate fault location. Traditional manifold learning methods use simple statistical methods (such as the shortest path) as indicators for obtaining geodesics. When used in the IT operation and maintenance management of distributed clouds, they ignore the time series information of the entire distributed cloud.

[0053] In this embodiment, the monitoring vector and historical monitoring vector of the distributed cloud are regarded as the system state of the distributed cloud. The monitoring vector and historical monitoring vector have their corresponding multi-dimensional space, and the system state is a coordinate point in the multi-dimensional space. The change of the coordinate point in the multi-dimensional space over time can be regarded as the change of the system state of the distributed cloud.

[0054] Since various devices in the distributed cloud influence each other, monitoring indicators of different dimensions in the multidimensional space usually show a trend of coordinated changes, which is reflected in the multidimensional space as follows: the coordinate points in the multidimensional space that represent the system state of the distributed cloud have motion trajectories. Therefore, for a system state, other system states that are close to it along its motion trajectory are system states similar to the working state of the system.

[0055] When manifold learning performs data dimensionality reduction, it needs to calculate the distances between different coordinate points in the multidimensional space. Traditional manifold learning ignores the motion trajectories in the multidimensional space when performing data dimensionality reduction, that is, it ignores the coordinated change rules between different monitoring indicators, which makes it easy to misjudge the working status differences between different system states when performing data dimensionality reduction, thereby reducing the quality of data dimensionality reduction.

[0056] Therefore, this embodiment uses all the collection moments in the neighborhood of each collection moment as the reference moment of each collection moment, and determines the interval vector between each collection moment and its reference moment based on the time sequence and historical monitoring vector of each collection moment and its reference moment, and determines the motion direction vector of each collection moment in combination with the time interval between each collection moment and its reference moment, so as to improve the quality of data integration. Specifically:

[0057] (1) Based on the time sequence of each collection moment and its reference moment and the historical monitoring vector, the interval vector between each collection moment and its reference moment is determined, specifically:

[0058] The interval vector between the acquisition time m and its nth reference time The expression is:

[0059] ; In the formula, , Represents the historical monitoring vector at the collection time m, and the historical monitoring vector at the nth reference time of the collection time m.

[0060] (2) Based on the interval vector between each acquisition moment and each reference moment, and in combination with the time interval between each acquisition moment and each reference moment, the motion vector between each acquisition moment and each reference moment is determined, specifically:

[0061] The reciprocal of the normalized value of the time interval between each acquisition moment and each reference moment and the multiplication result of the unit vector of the interval vector are used as the motion vector between each acquisition moment and each reference moment.

[0062] (3) Based on the motion vector between each acquisition moment and its reference moment, the motion direction vector of each acquisition moment is determined, specifically:

[0063] The motion direction vector at each acquisition moment is the sum vector of the motion vectors between each acquisition moment and all its reference moments.

[0064] Furthermore, according to the motion direction vector of each acquisition moment, it can be understood that the larger the inverse of the normalized value of the time interval between the acquisition moment and its reference moment, the smaller the time difference between the acquisition moment and its reference moment, and the motion vector corresponding to the acquisition moment and its reference moment can better represent the motion direction of the system state at the acquisition moment. Therefore, for system states with the same motion direction and the more similar the motion states are to the acquisition moment and its reference moment, when data dimensionality reduction is performed through manifold learning, the working state endpoint distance between these acquisition moments should be set smaller; conversely, the smaller the inverse of the normalized value of the time interval between the acquisition moment and its reference moment, the larger the time difference between the acquisition moment and its reference moment, and the motion vector corresponding to the acquisition moment and its reference moment can less represent the motion direction of the system state at the acquisition moment.

[0065] Preferably, the motion vector acquisition process diagram provided in this embodiment is as follows: Figure 2 shown.

[0066] Step S3: Based on the distance between the historical monitoring vectors of each collection moment and all other collection moments, the mean value of the system state distance at each collection moment is determined, and combined with the distance between the historical monitoring vectors of any two collection moments, the system state similarity credibility between any two collection moments is determined; based on the similarity between the motion direction vectors at any two collection moments and the system state similarity credibility, the system state similarity value between any two collection moments is determined, and combined with the difference between the historical monitoring vectors at any two collection moments, the system state motion difference between any two collection moments is determined.

[0067] S301: Based on the distance between each collection moment and the historical monitoring vector of all other collection moments, determine the mean value of the system state distance at each collection moment, and determine the system state similarity credibility between any two collection moments in combination with the distance between the historical monitoring vectors of any two collection moments.

[0068] For any two acquisition moments, the more similar their motion direction vectors are, the more likely the system states at the corresponding acquisition moments are to be two system states on the same motion trajectory, and the closer the distance between the two system states should be. Two system states that are far apart may also have the same or opposite motion directions. At this time, the two are not on the same motion trajectory, and the greater the trajectory similarity, the greater the distance between the historical monitoring vectors between each acquisition moment and all other acquisition moments. Therefore, based on the distance between the historical monitoring vectors of any two acquisition moments, the mean distance of the system state at each acquisition moment is determined, and combined with the distance between the historical monitoring vectors of any two acquisition moments, the similarity credibility of the system state between any two acquisition moments is determined, specifically:

[0069] (1) Calculate the distance between the historical monitoring vector at each collection time and the historical monitoring vectors at all other collection times, and take the average of the distances before the preset value in the results of ascending order as the average system state distance at each collection time;

[0070] (2) The system state similarity credibility between any two acquisition times is the ratio of the mean of the system state distance between any two acquisition times to the distance between the historical monitoring vectors at the corresponding two acquisition times.

[0071] It should be noted that there are many methods for measuring the distance between two vectors. In this embodiment, the difference between two historical monitoring vectors is measured by calculating the Euclidean distance between the historical monitoring vector at each collection moment and the historical monitoring vector at other collection moments. The implementer may also use other methods for measuring the distance between vectors, such as the Mahalanobis distance and the Manhattan distance. This embodiment does not impose any special restrictions on the selection of the method for measuring the distance between vectors.

[0072] The calculation process of the Euclidean distance is a well-known technique, and the specific calculation steps are not repeated here.

[0073] Furthermore, according to the system state similarity credibility between any two acquisition moments, it can be understood that the greater the distance between the historical monitoring vectors at any two acquisition moments and the smaller the mean value of the system state distance, the smaller the obtained system state similarity credibility, which means that the greater the distance between the two system states, the less likely it is that the system states are located on the same motion trajectory; conversely, the smaller the distance between the historical monitoring vectors at any two acquisition moments and the greater the mean value of the system state distance, the greater the obtained system state similarity credibility, which means that the closer the distance between the two system states, the more likely it is that the system states are located on the same motion trajectory.

[0074] S302: Based on the similarity between the motion direction vectors at any two acquisition moments and the system state similarity credibility, determine the system state similarity value between any two acquisition moments, and combine the difference between the historical monitoring vectors at any two acquisition moments to determine the system state motion difference between any two acquisition moments.

[0075] In order to monitor the dynamic changes of the system and predict future state changes, thereby preventing maintenance, optimizing resource allocation, and improving the accuracy of system fault detection, the system state similarity value between any two acquisition moments is determined based on the similarity between the motion direction vectors at any two acquisition moments and the system state similarity credibility, and the system state motion difference between any two acquisition moments is determined in combination with the difference between the historical monitoring vectors at any two acquisition moments, so as to improve the recognition accuracy of system state changes, specifically:

[0076] (1) The product of the similarity between the motion direction vectors at any two acquisition moments and the system state similarity credibility is taken as the system state similarity value between any two acquisition moments;

[0077] It should be noted that there are many methods for measuring the similarity between vectors. In this embodiment, the cosine similarity between the motion direction vectors at any two acquisition moments is calculated to measure the similarity between the motion direction vectors at different acquisition moments. The implementer may also use other methods for measuring the similarity between vectors, such as the inverse of the Euclidean distance. This embodiment does not impose any special restrictions on the selection of the method for measuring the similarity between vectors.

[0078] The calculation process of cosine similarity is a well-known technology, and its specific calculation steps are not repeated here.

[0079] According to the system state similarity value between any two acquisition moments, it can be understood that the closer the similarity between the motion direction vectors at any two acquisition moments is to 0, and the smaller the system state similarity credibility is, it means that the directions of the system state motion at the two acquisition moments do not belong to the same trajectory; conversely, the greater the similarity between the motion direction vectors at any two acquisition moments, and the greater the system state similarity credibility is, it means that the directions of the system state motion at the two acquisition moments are more likely to belong to the same trajectory.

[0080] Preferably, the schematic diagram of the system state similarity value extraction process provided in this embodiment is as follows: Figure 3 shown.

[0081] (2) Further, based on the system state similarity value between any two acquisition moments and combined with the difference between the historical monitoring vectors at any two acquisition moments, the system state motion difference between any two acquisition moments is determined, specifically:

[0082] The difference in system state motion between acquisition time k and acquisition time r The expression is: ; In the formula, represents the distance between the historical monitoring vector at the collection time k and the historical monitoring vector at the collection time r; Represents the system state similarity value between the acquisition time k and the acquisition time r.

[0083] According to the system state motion difference between any two acquisition moments, it can be understood that the greater the distance between the historical monitoring vectors at any two acquisition moments, the smaller the system state similarity value, and the greater the obtained system state motion difference, which means that the system states at the two acquisition moments are less likely to be on the same motion trajectory; conversely, the smaller the distance between the historical monitoring vectors at any two acquisition moments, the greater the system state similarity value, and the smaller the obtained system state motion difference, which means that the system states at the two acquisition moments are more likely to be on the same motion trajectory.

[0084] Step S4: Based on the monitoring vector at the current moment and the system state movement difference between any two vectors in the historical monitoring vectors of a preset number of collection moments before the current moment, a distance matrix at the current moment is constructed, and combined with a dimensionality reduction algorithm, the operation and maintenance big data at the current moment is integrated and operation and maintenance faults are detected.

[0085] (1) Obtain the fault labels of the historical monitoring vectors of all collection times before the current time through the operation and maintenance system log;

[0086] It should be noted that the fault types in this embodiment include: no fault, server downtime, storage device damage, network connection interruption, routing error, application crash, database error, unauthorized access, data leakage, distributed denial of service attack, resource overload, and system configuration error. The fault label value is set to y=1 to y=Y=12, and y=1 is a special case where the system has no fault.

[0087] (2) Furthermore, in order to reduce the amount of data in the process of data dimensionality reduction and make the samples uniform so that they contain data of various fault types to improve the fault identification effect, a preset number of historical monitoring vectors are selected under each fault label, and they and the monitoring vector at the current moment are recorded as fault detection vectors.

[0088] It should be understood that the value of the preset number is artificially set. In this embodiment, the value of the preset number is 50. The implementer can also set it by himself according to the specific situation. This embodiment does not impose any special restrictions.

[0089] (3) Further, based on the monitoring vector at the current moment and the system state motion difference between any two vectors in the historical monitoring vectors at a preset number of acquisition moments before the current moment, a distance matrix at the current moment is constructed, specifically:

[0090] The distance matrix D at the current moment is expressed as: ; In the formula, b, c, d and all elements outside the diagonal represent the system state motion difference between any two fault detection vectors; the elements on the main diagonal are 0, and the values ​​of the elements symmetrical about the main diagonal are the same.

[0091] Among them, constructing a distance matrix is ​​a necessary step in data dimensionality reduction through manifold learning and is a commonly used technology in the field of data processing.

[0092] Preferably, the distance matrix construction process framework diagram provided in this embodiment is as follows: Figure 4 shown.

[0093] (4) Further, the distance matrix at the current moment is used as the input of the dimensionality reduction algorithm, and a two-dimensional coordinate point set is output, where each coordinate point corresponds to a fault detection vector;

[0094] It should be noted that there are many commonly used dimensionality reduction algorithms. This embodiment uses the multidimensional scaling MDS algorithm to reduce the dimensionality of multidimensional data. Implementers can also use other dimensionality reduction methods such as principal component analysis PCA dimensionality reduction algorithm and linear judgment analysis LDA algorithm. This embodiment does not make any special restrictions on the selection of dimensionality reduction algorithm.

[0095] Among them, the multi-dimensional scaling MDS algorithm is a well-known technology in the field of dimensionality reduction, and its specific process of reducing the dimensionality of multi-dimensional data will not be described in detail.

[0096] (5) Further, the fault labels corresponding to all the fault detection vectors except the fault detection vector at the current moment are assigned to the corresponding coordinate points in the two-dimensional coordinate point set, and the corresponding coordinate points and their fault labels are used as inputs of the support vector machine (SVM), and a classification model of the two-dimensional coordinate point set is output;

[0097] The working principle of the support vector machine is a well-known technology, and its specific principle will not be described in detail.

[0098] (6) Furthermore, the coordinate point corresponding to the fault detection vector at the current moment in the two-dimensional data point set is used as the input of the classification model, and the fault label at the current moment is output.

[0099] Preferably, the multi-source data integration and fault identification flow chart provided in this embodiment is as follows: Figure 5 shown.

[0100] It should be noted that the above sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above is a description of a specific embodiment of this specification. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0101] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0102] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Modifications to the technical solutions recorded in the aforementioned embodiments, or equivalent replacement of some of the technical features therein, do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A multi-source data integration method for IT operation and maintenance big data, characterized in that: The method comprises the following steps: In the distributed cloud monitoring system, there are 10 monitoring indicators, including CPU usage, memory usage, disk IO, network traffic, server status, fault information, network traffic, bandwidth occupancy, delay, and packet loss rate. Multiple monitoring data at the current moment and each collection moment within the preset time before are obtained. All the monitoring data at the current moment are combined into the monitoring vector of the current moment, and the multiple monitoring data at each collection moment are combined into the historical monitoring vector of each collection moment. All the acquisition moments in the neighborhood of each acquisition moment are used as reference moments of each acquisition moment. Based on the time sequence and historical monitoring vector of each acquisition moment and its reference moments, the interval vector between each acquisition moment and its reference moments is determined. In combination with the time interval between each acquisition moment and its reference moments, the motion vector between each acquisition moment and its reference moments is determined to determine the motion direction vector of each acquisition moment. Based on the distance between each collection moment and the historical monitoring vector of all other collection moments, the mean value of the system state distance at each collection moment is determined, and combined with the distance between the historical monitoring vectors of any two collection moments, the similarity credibility of the system state between any two collection moments is determined; Based on the similarity between the motion direction vectors at any two acquisition moments and the system state similarity credibility, determine the system state similarity value between any two acquisition moments, and combine the difference between the historical monitoring vectors at any two acquisition moments to determine the system state motion difference between any two acquisition moments; Based on the system state movement difference between the monitoring vector at the current moment and any two vectors in the historical monitoring vectors of a preset number of collection moments before the current moment, a distance matrix at the current moment is constructed, and combined with a dimensionality reduction algorithm, the operation and maintenance big data at the current moment is integrated and operation and maintenance faults are detected.

2. A multi-source data integration method for IT operation and maintenance big data as claimed in claim 1, characterized in that: The expression of the interval vector between each acquisition moment and each reference moment is: ; In the formula, represents the interval vector between the acquisition time m and its nth reference time; , They respectively represent the historical monitoring vector at the collection time m and the historical monitoring vector at the nth reference time of the collection time m.

3. The multi-source data integration method of IT operation and maintenance big data according to claim 1, characterized in that: The method for determining the motion direction vector at each acquisition moment is: The result of multiplying the normalized value of the time interval between each acquisition moment and each reference moment by the unit vector number of the interval vector is taken as the motion vector between each acquisition moment and each reference moment.

4. The multi-source data integration method of IT operation and maintenance big data according to claim 1, characterized in that: The motion direction vector at each acquisition moment is the sum vector of the motion vectors between each acquisition moment and all its reference moments.

5. The multi-source data integration method of IT operation and maintenance big data according to claim 1, characterized in that: The method for determining the mean value of the system state distance at each acquisition time is: Calculate the distance between the historical monitoring vector at each collection moment and the historical monitoring vector at all other collection moments, and take the average of the first preset value of distances in the results of ascending distance arrangement as the average system state distance at each collection moment.

6. The multi-source data integration method of IT operation and maintenance big data according to claim 1, characterized in that: The system state similarity credibility between any two collection moments is the ratio of the mean of the system state distance mean values ​​at any two collection moments to the distance between the historical monitoring vectors at the corresponding two collection moments.

7. The multi-source data integration method of IT operation and maintenance big data according to claim 1, characterized in that: The system state similarity value between any two acquisition moments is the product of the similarity between the motion direction vectors at any two acquisition moments and the system state similarity credibility.

8. The multi-source data integration method of IT operation and maintenance big data according to claim 1, characterized in that: The expression of the system state motion difference between any two acquisition moments is: ; In the formula, represents the difference in system state motion between acquisition time k and acquisition time r; represents the distance between the historical monitoring vector at the collection time k and the historical monitoring vector at the collection time r; Represents the system state similarity value between the acquisition time k and the acquisition time r.

9. The multi-source data integration method of IT operation and maintenance big data according to claim 1, characterized in that: The construction process of the distance matrix at the current moment is: Set fault labels for all historical monitoring vectors collected at all times before the current time, select a preset number of historical monitoring vectors under each fault label, and record them and the monitoring vector at the current time as fault detection vectors; The distance matrix D at the current moment is expressed as: ; In the formula, b, c, d and all elements outside the diagonal represent the system state motion difference between any two fault detection vectors; the elements on the main diagonal are 0.

10. A multi-source data integration method for IT operation and maintenance big data as claimed in claim 9, characterized in that: The integration of the current operation and maintenance big data and the detection of operation and maintenance failures include: The distance matrix at the current moment is used as the input of the dimension reduction algorithm, and a two-dimensional coordinate point set is output, where each coordinate point corresponds to a fault detection vector; Assign the fault labels corresponding to all the fault detection vectors except the fault detection vector at the current moment to the corresponding coordinate points in the two-dimensional coordinate point set, and use the corresponding coordinate points and their fault labels as inputs of the support vector machine, and output a classification model of the two-dimensional coordinate point set; The coordinate points corresponding to the fault detection vector at the current moment in the two-dimensional data point set are used as the input of the classification model, and the fault label at the current moment is output.

Images