A method for analyzing alarm data based on large models

Abstract

The present invention relates to the field of data security technology, and in particular to a method for analyzing alarm data based on a large model. First, the alarm data is collected and converted into vectors, and an initial covariance matrix is ​​calculated. Due to the similarity of the alarm data, the feature vectors extracted during principal component analysis have high correlation, which affects the extraction of key features. Therefore, based on local characteristics, the initial covariance matrix is ​​divided into regions to identify high-retention and low-retention regional blocks. For the highly repetitive and highly sparse element values ​​in the low-retention regional blocks, a de-repetitiveness score is calculated to adjust the element value and obtain an updated value, thereby reducing its impact on the subsequent dimensionality reduction process. Finally, the updated covariance matrix is ​​used to perform principal component dimensionality reduction to obtain a dimensionality reduction result that is more representative of the key features of the alarm data, thereby improving the accuracy of model training.

Description

Technical Field

[0001] The present invention relates to the field of data security technology, and in particular to an alarm data analysis method based on a large model. Background Art

[0002] A large model usually refers to a complex machine learning model with a large number of parameters. It can be used to process large-scale data sets. Therefore, it shows great potential in the identification and analysis of alarm data and is crucial to improving the management efficiency of data security. Alarm data is usually diverse and complex, covering a variety of information from network traffic to system logs. When analyzing alarm data, due to the diversity and complexity of alarm data, in order to improve the accurate classification and prediction capabilities of large models for alarm data, it is necessary to extract key information from a large amount of alarm data. Therefore, it is necessary to perform dimensionality reduction analysis on a large amount of alarm data.

[0003] The existing technology usually uses the principal component analysis method to reduce the dimension of the data and obtain the dimension reduction result. However, due to the different types of alarm data, the information contained in them is also different, and there will be a large amount of similar and unrepresentative information. Therefore, when the principal component analysis method is used for dimensionality reduction, it will affect the extraction of key features, and thus affect the training accuracy of the model. Summary of the invention

[0004] In order to solve the technical problem that the types of alarm data are different, the information contained in them is also different, and there will be a large amount of similar and unrepresentative information, so when the principal component analysis method is used for dimensionality reduction, it will affect the extraction of key features, and then affect the training accuracy of the model, the purpose of the present invention is to provide an alarm data analysis method based on a large model, and the technical scheme adopted is as follows:

[0005] Acquire multiple pieces of alarm data, each piece of alarm data contains multiple types of data; perform vector conversion on each type of data in all the alarm data to obtain an initial covariance matrix corresponding to all the alarm data;

[0006] Performing regional division on the initial covariance matrix to obtain all initial regional blocks; for any initial regional block, analyzing the relationship between the difference, randomness and position distribution of element values ​​in the initial regional block to obtain the retained eigenvalue of the initial regional block; distinguishing low-retention regional blocks from high-retention regional blocks according to the retained eigenvalues ​​of all initial regional blocks;

[0007] Determine the neighborhood blocks of each element value in each low-retention area block; for any element value in any low-retention area block, obtain a de-repetitiveness score of the element value according to the difference between element values ​​in the neighborhood blocks corresponding to the element value and the numerical fluctuation between the element values ​​in the corresponding neighborhood blocks; adjust the element value according to the de-repetitiveness score of the element value to obtain an updated value;

[0008] According to the update values ​​corresponding to the element values ​​in all low-retention area blocks and the element values ​​in the high-retention area blocks, an updated covariance matrix is ​​obtained; principal component analysis is performed on the updated covariance matrix to obtain a dimensionality reduction result for model training.

[0009] Furthermore, the method for obtaining the retained characteristic value includes:

[0010] For any initial region block, the entropy value of all element values ​​in the initial region block is used as the randomness characteristic value;

[0011] All element values ​​in the initial region block are combined in pairs to obtain all non-repeating element value combinations. For any element value combination, the difference between the two element values ​​is calculated as the difference factor, and the Euclidean distance between the positions of the two element values ​​is calculated as the distance factor.

[0012] Obtaining importance scores of the two element values ​​according to the difference factor and the distance factor, wherein the importance score is positively correlated with the difference factor, and negatively correlated with the distance factor, and the importance score is a normalized value;

[0013] The average of the importance scores corresponding to all element value combinations in the initial area block is used as the retention factor of the initial area block; the retention characteristic value of the initial area block is obtained according to the retention factor and the random characteristic value of the initial area block, the retention factor and the random characteristic value are positively correlated with the retention characteristic value, and the value of the retention characteristic value is a normalized value.

[0014] Furthermore, the step of distinguishing low-retention-degree regional blocks from high-retention-degree regional blocks according to the retention feature values ​​of all initial regional blocks includes:

[0015] The initial region block corresponding to the retention feature value greater than or equal to the preset retention threshold is used as a high retention region block;

[0016] The initial region blocks corresponding to the retention feature values ​​less than the preset retention threshold are regarded as low retention region blocks.

[0017] Furthermore, the determining of the neighborhood block of each element value in each low-retention area block includes:

[0018] Taking each element value as the center, a preset second number of element values ​​that are adjacent and closest in a preset direction are taken as neighborhood element values, and an area formed by the central element value and the corresponding neighborhood element values ​​is taken as a neighborhood block corresponding to the central element value.

[0019] Furthermore, the method for obtaining the de-duplication score includes:

[0020] For any element value in any low-retention area block, a first repetition factor corresponding to the element value is obtained according to the difference between the element value and all neighboring element values ​​in the corresponding neighboring block;

[0021] Calculate the mean of all element values ​​in a neighborhood block centered on the element value as the mean eigenvalue, and take the difference between the element value and the mean eigenvalue as the deviation value; obtain a second repetition factor corresponding to the element value according to the element value and the corresponding deviation value, wherein both the deviation value and the element value are negatively correlated with the second repetition factor;

[0022] A de-repetitiveness score of the element value is obtained according to a first repetitiveness factor and a second repetitiveness factor of the element value, wherein both the first repetitiveness factor and the second repetitiveness factor are positively correlated with the de-repetitiveness score, and the value of the de-repetitiveness score is a normalized value.

[0023] Furthermore, the method for obtaining the first repetition factor includes:

[0024] In a neighborhood block centered on the element value, the numerical difference between the element value and each corresponding neighborhood element value is taken as the difference coefficient of the element value;

[0025] The mean values ​​of all difference coefficients corresponding to the element value are negatively correlated and normalized to obtain a value which is used as the first repetition factor.

[0026] Furthermore, the method for obtaining the update value includes:

[0027] For any element value in any low-retention area block, the sum of the de-repetitiveness score of the element value and a preset constant is used as an adjustment weight;

[0028] The element values ​​are weighted adjusted based on the adjustment weights corresponding to each element value to obtain an updated value corresponding to each element value.

[0029] Furthermore, the method for obtaining the updated covariance matrix includes:

[0030] The element values ​​in all low-retention area blocks are replaced with the corresponding updated values, and the element values ​​in all high-retention area blocks remain unchanged, thereby obtaining an updated covariance matrix.

[0031] Furthermore, the method for obtaining the initial region block includes:

[0032] The initial covariance matrix is ​​evenly divided into a preset first number of initial area blocks.

[0033] Furthermore, the preset retention threshold is set to 0.85.

[0034] The present invention has the following beneficial effects:

[0035] The present invention is directed to the problem that when a large amount of alarm data is subjected to dimensionality reduction analysis and main components are extracted, due to the presence of repeatability and other characteristics in the alarm data, the credibility of the dimensionality reduction result is low. First, a large amount of alarm data is obtained, and each alarm data contains multiple types of data, such as numerical type, text type, etc., and then different vector conversion methods are selected according to the type of data, so that the initial covariance matrix in the principal component analysis process can be calculated. Since there are similar features in a large amount of alarm data, a large amount of similar element values ​​will exist in the initial covariance matrix at this time, so that the eigenvector finally extracted will have a strong correlation, so the key data features in the alarm data cannot be completely extracted, so it is necessary to adjust the element values ​​in the initial covariance matrix. The initial covariance matrix is ​​divided into regions to obtain initial regional blocks. Since the data with significant differences from other data should have higher research value, that is, it should be feature data, for each initial regional block, the difference between the element values ​​and the position distribution are analyzed, and the reserved eigenvalues ​​are obtained to distinguish between low-retention regional blocks and high-retention regional blocks, wherein the data in the low-retention regional blocks are non-feature data. For the data in the low retention area block, because the data repetition is high, it can be regarded that the sparsity of the data will be relatively enhanced. In order to reduce the impact of this part of the data in the subsequent dimensionality reduction process, it is necessary to adjust the value: for any element value in any low retention area block, according to the difference between the element values ​​in the neighborhood blocks corresponding to the element value and the numerical fluctuation between the element values ​​in the corresponding neighborhood blocks, the de-repetitiveness score of the element value is obtained; and the element value is adjusted according to the de-repetitiveness score of the element value to obtain the updated value. At this time, the data repetitiveness in the adjusted low retention area will be reduced, so the impact of this part of non-feature data on the subsequent dimensionality reduction process can be reduced. Therefore, in the initial covariance matrix, the corresponding element value is replaced by the updated value, and then the subsequent principal component dimensionality reduction process is continued to obtain the dimensionality reduction result. At this time, the dimensionality reduction result is obtained on the basis of eliminating the repetitive characteristics of the data, so it can better represent the key features of the alarm data, so when the dimensionality reduction result is used to train the model, the accuracy will also be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A method flow chart of an alarm data analysis method based on a large model provided by an embodiment of the present invention;

[0038] Figure 2 A method flow chart of a method for obtaining a retained feature value provided by an embodiment of the present invention;

[0039] Figure 3 A schematic diagram of constructing a two-dimensional coordinate system provided by an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of constructing a neighborhood block provided by an embodiment of the present invention;

[0041] Figure 5 A method flow chart of a method for obtaining a de-repetitiveness score provided by one embodiment of the present invention;

[0042] Figure 6 A schematic diagram of a computer-readable storage medium provided for one embodiment of the present invention. DETAILED DESCRIPTION

[0043] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the following is a detailed description of the specific implementation method, structure, features and effects of an alarm data analysis method based on a large model proposed by the present invention in combination with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0044] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0045] The specific scheme of the alarm data analysis method based on a large model provided by the present invention is described in detail below with reference to the accompanying drawings.

[0046] See also Figure 1 , which shows a method flow chart of an alarm data analysis method based on a large model provided by an embodiment of the present invention, the method comprising the following steps:

[0047] Step S1: Acquire multiple pieces of alarm data, each piece of alarm data contains multiple types of data; perform vector conversion on each type of data in all the alarm data to obtain an initial covariance matrix corresponding to all the alarm data.

[0048] In enterprise IT infrastructure, when monitoring the operating status of various devices and services, alarm data will be generated once a failure or anomaly occurs. In the field of network security, alarm data will also be generated when potential network attacks, malicious behaviors or abnormal traffic occur. In the field of industrial automation, alarm data will also be generated when abnormalities occur in equipment, sensors and control systems on the production line.

[0049] As enterprises expand in size and become more digitalized, the amount of alarm data is growing exponentially. A large model usually refers to a complex machine learning model with a large number of parameters that can be used to process large data sets, so it can be applied to the identification and analysis of alarm data. Due to the diversity and complexity of alarm data, there will be redundant information in it. In order to improve the large model's ability to accurately classify and predict alarm data and improve the level of data security, it is necessary to perform dimensionality reduction on a large amount of alarm data to extract the main data features.

[0050] First, multiple alarm data are obtained, and each alarm data usually contains multiple types of data, such as: alarm start time, alarm end time, alarm duration, alarm number, device ID, alarm level, error code, alarm content, related users, alarm ID, etc.

[0051] It should be noted that the above alarm data can be obtained through network monitoring equipment, database monitoring tools, security monitoring equipment, etc. The specific acquisition method can be adjusted according to the implementation scenario and is not limited here; and the collection and acquisition of alarm data have been authorized by relevant users, and the process does not violate relevant laws and regulations and does not violate public order and good customs.

[0052] After obtaining multiple alarm data, since each alarm data contains multiple dimensions, and the subsequent process in this embodiment of the present invention uses the principal component analysis method to reduce the dimension for extracting feature data, and the data has different formats, for example, the alarm start time, alarm end time, alarm duration, and alarm number are numerical data; the device ID, alarm level, and alarm ID are classified data; the error code, alarm content, and related users are text data, so it is necessary to perform vector conversion on the data of different types in all alarm data to obtain the initial covariance matrix, which is a necessary step and preparatory work in the principal component analysis method.

[0053] In this embodiment of the present invention, 10 common types of alarm data are listed as examples, which can be regarded as 10 dimensions. If there are also 10 alarm data, there are 10 groups of data under each dimension. For numerical data, the normalization method can be used to specify a fixed range for the numerical data. Then, for the numerical data, each dimension corresponds to a column vector, and the element value in the column vector is the converted value; for the classified data, it can be converted into binary vector data using unique hot encoding, and for the text data, it can be converted into vector data using word embedding vector. It should be noted that since the length of text data and classified data may not be limited to one element value when they are converted into vectors, the vectors corresponding to the text data dimension and the classified data dimension may have multiple column vectors. For example, the converted vector of the alarm content of a certain alarm data is [1,2,1,1], then the converted vector of the alarm content dimension in all the alarm data has four column vectors. All the obtained column vectors are standardized, so that the initial covariance matrix corresponding to the standardized vector can be calculated.

[0054] It should be noted that the normalization method, unique hot encoding, and word embedding vector used in data processing are all technical means well known to those skilled in the art, and will not be elaborated on here; calculating the initial covariance matrix is ​​a part of the principal component analysis method, which is also a technical means well known to those skilled in the art, and will not be elaborated on here; the 10 common data types mentioned above are only used as examples. In other embodiments of the present invention, they may also be increased or decreased according to actual conditions, and are not limited here.

[0055] Step S2: Divide the initial covariance matrix into regions to obtain all initial region blocks; for any initial region block, analyze the relationship between the difference, randomness and position distribution of the element values ​​in the initial region block to obtain the retained eigenvalue of the initial region block; distinguish low-retention region blocks from high-retention region blocks based on the retained eigenvalues ​​of all initial region blocks.

[0056] The eigenvectors corresponding to different eigenvalues ​​of the matrix are linearly independent, which ensures that the data features in the matrix can be fully extracted. When there are many similar values ​​in the matrix, the linear correlation of the element values ​​in the matrix itself is relatively high, so most of the extracted eigenvalues ​​are equal, and the matrix eigenvectors obtained at this time are basically linearly related, so the data features in the matrix cannot be fully extracted. Therefore, in order to ensure or reduce the correlation between elements, and then increase unequal eigenvalues, and extract more features of the data in the matrix, the data needs to be adjusted.

[0057] First, it is necessary to find the data that does not have data characteristics. Data that does not have data characteristics should have a high similarity with other data, so this kind of data is often easier to be replaced. Therefore, in this embodiment of the present invention, starting from the local area of ​​the initial covariance matrix, the differences between element values ​​are analyzed to evaluate the degree of retention of the element values, that is, whether it has data characteristics.

[0058] Preferably, in one embodiment of the present invention, the initial covariance matrix is ​​divided into regions to obtain all initial region blocks, including:

[0059] The initial covariance matrix is ​​evenly divided into a preset first number of initial region blocks, and each element value in the initial covariance matrix is ​​in an initial region block. It should be noted that in this embodiment of the present invention, the preset first number is set to 4, and the specific value is not limited and can be adjusted according to actual conditions.

[0060] Based on the above method, the initial covariance matrix can be divided to obtain several initial area blocks, and then the element values ​​in each initial area block can be analyzed to evaluate the degree of retention, mainly starting from the differences, randomness and position distribution between the element values ​​in the initial area blocks to obtain the retained characteristic values ​​of the initial area blocks.

[0061] Preferably, in one embodiment of the present invention, the method for obtaining the retained characteristic value includes:

[0062] See also Figure 2 , which shows a method flow chart of a method for obtaining a retained feature value provided by an embodiment of the present invention, the method comprising the following steps:

[0063] Step S201: for any initial region block, analyze the distribution randomness of all element values ​​in the initial region block to obtain the randomness characteristic value corresponding to the initial region block.

[0064] In step S1, the data of each dimension is a column vector, so the element values ​​in each initial area block should come from different dimensions. Even if they come from the same dimension, such as text data, the element values ​​in different columns also represent different characters in the text. Therefore, the numerical distribution of the element values ​​in the initial area block should show a large randomness, and entropy can be used to measure the uncertainty or randomness of random variables. By calculating the entropy of all element values ​​in the initial area block, the distribution randomness of the element values ​​in the initial area block can be quantitatively evaluated. Therefore, for any initial area block, the entropy value of all element values ​​in the initial area block is used as the randomness characteristic value, denoted as SJ.

[0065] Among them, the larger the entropy value, the more uniform the distribution of element values, the closer the probability of element values ​​appearing, the higher the randomness of the elements in the initial area block, which can be regarded as the low repeatability between the element values ​​in the initial area block; conversely, the smaller the entropy value, the more concentrated the distribution of element values, the greater the difference in the probability of element values ​​appearing, and the higher the similarity of the element values ​​in the initial area block.

[0066] Step S202: Analyze the differences between element values ​​and position distribution characteristics in the initial region block, and calculate the retention factor of the initial region block.

[0067] In order to analyze the similarity between the element values ​​in the initial region block in more detail, all element values ​​in the initial region block can be combined in pairs to obtain all non-repeating element value combinations. For example, there are 4 element values, which are recorded as element value 1, element value 2, element value 3 and element value 4 respectively. Then all element value combinations include: (element value 1, element value 2), (element value 1, element value 3), (element value 1, element value 4), (element value 2, element value 3), (element value 2, element value 4), (element value 3, element value 4). For any element value combination, the difference between the two element values ​​is calculated as a difference factor. The difference factor reflects the actual difference between the element values ​​and is used to characterize the similarity. The Euclidean distance between the positions of the two element values ​​is calculated as a distance factor. The distance factor reflects the relative position of the element value in the initial region block.

[0068] Combining the difference factor and distance factor between element values ​​can more comprehensively evaluate the similarity between two element values. Therefore, the importance scores of the two element values ​​are obtained based on the difference factor and the distance factor. The importance score is positively correlated with the difference factor, and negatively correlated with the distance factor. The importance score is taken as the normalized value.

[0069] Finally, the average of the importance scores corresponding to all element value combinations in the initial region block is used as the retention factor of the initial region block.

[0070] Among them, a positive correlation means that the dependent variable will increase as the independent variable increases, and the dependent variable will decrease as the independent variable decreases. The specific relationship can be a multiplication relationship, an addition relationship, or the power of an exponential function, which is determined by actual applications; a negative correlation means that the dependent variable will decrease as the independent variable increases, and the dependent variable will increase as the independent variable decreases. It can be a subtraction relationship, a division relationship, etc., which is determined by actual applications. The formula model of the retention factor can be specifically, for example:

[0071]

[0072] Among them, BY irepresents the retention factor of the i-th initial region block; R i represents the number of element value combinations in the i-th initial region block; Represents the first element value in the jth element value combination; Indicates the second element value in the jth element value combination; d j represents the distance factor between two element values ​​in the jth element value combination; norm() represents the normalization function; γ1 represents the preset first parameter.

[0073] In the formula model of the retention factor, for any combination of element values, the difference between the two element values ​​is calculated as the difference factor The larger the value, the greater the difference between the two element values, and the lower the similarity, so this type of data should be retained because it is more irreplaceable in a large amount of data and can be used as feature data. On the contrary, the smaller the value, the smaller the difference between the two element values, and the higher the similarity, which means that this type of data is easier to replace, has less value, and is less likely to be feature data. Then calculate the distance factor d between the two element values j , the distance factor is calculated by the Euclidean distance between the two element values. Specifically, a two-dimensional coordinate system can be established with the lower left corner of the initial region block as the origin, the horizontal right direction as the horizontal axis, and the vertical upward direction as the vertical axis. At this time, each element value in the initial region block has a two-dimensional coordinate in the form of (x, y). Please refer to Figure 3 , which shows a schematic diagram of a two-dimensional coordinate system, where the coordinates of the element value p are (3,3). When the distance factor between two element values ​​is smaller, the credibility of the calculated difference factor will be higher, otherwise the credibility will be lower. Therefore, the distance factor is used as the denominator, the difference factor is used as the numerator, and the ratio is normalized to obtain the importance score of the two element values. The larger the value, the higher the retention degree of the two element values ​​should be. Then the importance scores corresponding to all element value combinations are accumulated and averaged, and the obtained average is used as the retention factor of the initial region block. The larger the retention factor, the lower the similarity between the element values ​​in the initial region block, which is more likely to be feature data and should be retained.

[0074] It should be noted that the purpose of presetting the first parameter γ1 is to prevent the denominator from being 0, and the value here can be 0.001. The specific value can be adjusted according to the actual situation and is not limited here.

[0075] Step S203: obtaining a retained characteristic value of the initial region block according to the random characteristic value of the initial region block and the retained factor.

[0076] The randomness eigenvalue calculated in step S201 can be used to measure the distribution randomness of the element values ​​in the initial region block, and the retention factor calculated in step S202 evaluates the similarity and position distribution characteristics between the element values ​​in the initial region block. Therefore, the two can be combined here to quantify the degree of retention of the element values ​​in the initial region block and obtain the retained eigenvalue of the initial region block; wherein the retention factor and the randomness eigenvalue are both positively correlated with the retained eigenvalue, and the retained eigenvalue is a normalized value. The formula model for the retained eigenvalue can be, for example, as follows:

[0077] BT i =norm(SJ i ×BY i )

[0078] Among them, BT i represents the retained feature value of the i-th initial region block; SJ i Represents the random characteristic value of the i-th initial area block; BY i represents the retention factor of the i-th initial region block; norm() represents the normalization function.

[0079] In the formula model for retaining eigenvalues, the larger the random eigenvalue of the initial area block, the higher the randomness of the element value in the initial area block, which can be regarded as a lower repeatability between the element values ​​in the initial area block; the larger the retention factor, the lower the similarity between the element values ​​in the initial area block, which is more likely to be feature data and should be retained, so the two are multiplied and the product is normalized to obtain the retained eigenvalue of the initial area block.

[0080] Since the retention factor and the randomness eigenvalue are both positively correlated with the retention eigenvalue, in other embodiments of the present invention, the following formula model for the retention eigenvalue is also constructed:

[0081] BT i =norm(SJ i +BY i )

[0082] Among them, BT i represents the retained feature value of the i-th initial region block; SJ i Represents the random characteristic value of the i-th initial area block; BY i represents the retention factor of the i-th initial region block; norm() represents the normalization function.

[0083] In this formula model, similarly, the larger the randomness characteristic value of the initial area block, the higher the randomness of the element value in the initial area block, which can be regarded as the lower the repeatability between the element values ​​in the initial area block; the larger the retention factor, the lower the similarity between the element values ​​in the initial area block, which is more likely to be feature data and should be retained, so the two are added and the sum is normalized to obtain the retained characteristic value of the initial area block.

[0084] At this point, the retained eigenvalues ​​of each initial region block can be obtained through the above process. The larger the retained eigenvalues, the lower the similarity between the element values ​​in this initial region block, and the more implicit feature data therein, which should be retained as the main part of the subsequent dimensionality reduction. For the element values ​​in the initial region blocks with a lower degree of retention, because of their high degree of similarity and large information redundancy, they can be optimized in the subsequent process to reduce the influence of these element values ​​on the dimensionality reduction results.

[0085] Preferably, in one embodiment of the present invention, all initial region blocks are distinguished based on the retention feature values ​​to obtain low retention region blocks and high retention region blocks, including:

[0086] The initial region blocks corresponding to the retention feature values ​​greater than or equal to the preset retention threshold are regarded as high retention region blocks; the initial region blocks corresponding to the retention feature values ​​less than the preset retention threshold are regarded as low retention region blocks.

[0087] It should be noted that since the value of the retained feature value is a normalized value, and the retention threshold is a threshold set to distinguish the degree of retention of the initial area block and screen out the low-retention area block, so as to facilitate the subsequent adjustment and optimization of the element value therein, its value range can be between 0-1, and the larger the value, the stricter the distinction standard between the low-retention area block and the high-retention area block. Therefore, in this embodiment, the preset retention threshold is preferably set to 0.85. In other embodiments of the present invention, it can be adjusted according to the implementation scenario and is not limited here.

[0088] Step S3: determine the neighborhood blocks of each element value in each low-retention area block; for any element value in any low-retention area block, obtain a de-duplication score for the element value based on the difference between the element values ​​in the neighborhood blocks corresponding to the element value and the numerical fluctuation between the element values ​​in the corresponding neighborhood blocks; adjust the element value according to the de-duplication score of the element value to obtain an updated value.

[0089] In the above steps, low-retention area blocks can be screened out. The similarity of the element values ​​in the low-retention area blocks will be higher, and their substitutability will be greater. Since the element values ​​are highly repetitive, they have fewer data features, so this type of data can be optimized to reduce the repetitiveness of the element values ​​in the initial covariance matrix, thereby obtaining more feature data in the subsequent dimensionality reduction process.

[0090] First, the neighborhood blocks of each element value in each low-retention area block are determined. The purpose of determining the neighborhood blocks is to further analyze the relationship between the element value and its surrounding element values, so as to analyze the repeatability between the element values ​​for subsequent value adjustment.

[0091] Preferably, in one embodiment of the present invention, determining the neighborhood block of each element value in each low-retention region block includes:

[0092] With each element value as the center, the preset second number of element values ​​that are closest to each other in the preset direction are taken as the neighborhood element values, and the area formed by the central element value and the corresponding neighborhood element values ​​is taken as the neighborhood block corresponding to the central element value. Since each dimension has a column vector, when analyzing the repeatability between elements, the repeatability of the elements on the column vector is given priority. Therefore, here, the preset direction is set to the vertical direction, and the preset second number is set to 4. The specific numerical settings can be adjusted according to the implementation scenario and are not limited here. Taking the element value q as an example, q1, q2, q3, and q4 are all neighborhood element values ​​of the element value q. Please refer to Figure 4 , which shows a schematic diagram of a neighborhood block provided by an embodiment of the present invention.

[0093] At this time, each element value in each low-retention area block has a corresponding neighborhood block, and then the repeatability between element values ​​can be analyzed in the neighborhood block. The repeatability is mainly characterized by the difference between element values ​​and the numerical fluctuation of element values. The higher the repeatability, the higher the de-repetitiveness score will be, and the more it is necessary to adjust the element value to avoid affecting the subsequent dimensionality reduction results.

[0094] Preferably, in one embodiment of the present invention, the method for obtaining the de-duplication score includes:

[0095] See also Figure 5 , which shows a method flow chart of a method for obtaining a de-repetitiveness score provided by an embodiment of the present invention, the method comprising the following steps:

[0096] Step S301: based on the differences between element values ​​in the neighborhood blocks, obtain a first repetition factor of the central element value.

[0097] For any element value in any low-retention area block, the numerical difference between the element value and each corresponding neighborhood element value is used as the difference coefficient of the element value.

[0098] Then, the average values ​​of all the difference coefficients corresponding to the element value are negatively correlated and normalized to form the first repetition factor. Taking the element value a as an example, the formula model of the first repetition factor can be specifically as follows:

[0099]

[0100] Among them, CF a Indicates the first repetition factor of element value a; N a Indicates the total number of neighborhood element values ​​of element value a; g a Represents element value a; Represents the kth neighboring element value of element value a; exp() represents the exponential function with the natural constant e as the base.

[0101] In the formula model of the first repeated factor, the difference between the element value and the corresponding neighboring element value is calculated as the difference coefficient When the value is larger, it means that the repeatability between elements is low. On the contrary, when the value is smaller and equal to 0, it means that the elements are repeated. All the difference coefficients corresponding to the element values ​​are averaged, negatively correlated and normalized to obtain the first repetition factor. When the repeatability between the element value and its neighboring element values ​​is higher, that is, the more cases where the difference coefficient is 0, the larger the value of the first repetition factor will be, and the more it is necessary to adjust the element value, thereby reducing the repeatability between the elements in the covariance matrix and improving the accuracy of subsequent feature data extraction.

[0102] Step S302: Determine a second repetition factor of the central element value based on fluctuations between element values ​​in the neighborhood block.

[0103] In the low retention area block, the mean of all element values ​​in the neighborhood block where the element value is located is calculated as the mean eigenvalue, and the difference between the element value and the mean eigenvalue is used as the deviation value; the second repetition factor corresponding to the element value is obtained according to the element value and the corresponding deviation value, and the deviation value and the element value are negatively correlated with the second repetition factor. The formula model of the second repetition factor can be specifically, for example:

[0104]

[0105] Among them, CF a ′ Indicates the second repetition factor of element value a; g a Represents element value a; represents the mean eigenvalue corresponding to the element value a; γ2 represents the preset second parameter.

[0106] In the formula model of the second repeated factor, the mean of all element values ​​in the neighborhood block is taken as the mean eigenvalue, which represents the average level of all element values ​​in the neighborhood block. Then the difference between the central element value and the mean eigenvalue is calculated as the deviation value. The larger the deviation value, the greater the difference between the central element value and the overall level of the element value in the corresponding neighborhood block. To a certain extent, this indicates that the repeatability is lower. Conversely, the smaller the deviation value, the closer the central element value is to the overall level of the element value. This indicates that the probability of repetition is higher. Therefore, the deviation value is used as the denominator to adjust the logical relationship. When the element value is closer to 0, it is considered to contain fewer features and requires subsequent adjustments. Therefore, the element value is also placed in the denominator, and the second repetition factor is finally obtained. The larger the second repetition factor, the more the element value needs to be adjusted to reduce the repeatability between element values.

[0107] It should be noted that the purpose of presetting the second parameter γ2 is to prevent the denominator from being 0, and the value here can be 0.001. The specific value can be adjusted according to the actual situation and is not limited here.

[0108] Step S303: The first repetition factor and the second repetition factor of the central element value are merged to obtain a de-repetition score of the central element value.

[0109] The product of the first repetition factor and the second repetition factor of the element value is normalized to obtain the de-repetitiveness score of the element value. The formula model of the de-repetitiveness score can be, for example:

[0110] QC a =norm(CF a ×CF a ′ )

[0111] Among them, QC a Indicates the de-duplication score of element value a; CF a Indicates the first repetition factor of element value a; CF a ′ Represents the second repetition factor of element value a; norm() represents the normalization function.

[0112] In the formula model of the de-repetitiveness score, the larger the first repetition factor is, the more it is necessary to adjust the element value; similarly, the larger the second repetition factor is, the more it is necessary to adjust the element value, so as to reduce the repetitiveness between element values ​​and thus reduce the repetitiveness between elements in the covariance matrix, and improve the accuracy of subsequent feature data extraction. Therefore, the first repetition factor is multiplied by the second repetition factor, and the obtained product is normalized to obtain the value as the de-repetitiveness score of the element value.

[0113] In other embodiments of the present invention, the following formula model for removing repetitiveness scores may also be constructed:

[0114] QC a =norm(CF a +CF a ′ )

[0115] Among them, QC a Indicates the de-duplication score of element value a; CF a Indicates the first repetition factor of element value a; CF a ′ Represents the second repetition factor of element value a; norm() represents the normalization function.

[0116] In this formula model, the larger the first repetition factor is, the more it is necessary to adjust the element value; similarly, the larger the second repetition factor is, the more it is necessary to adjust the element value, so as to reduce the repeatability between element values ​​and thus reduce the repeatability between elements in the covariance matrix, and improve the accuracy of subsequent feature data extraction. Therefore, the first repetition factor is added to the second repetition factor, and the normalized value of the sum is used as the de-repeatability score of the element value.

[0117] Based on the above process, the de-duplication score of any element value in any low-retention region block can be obtained, and then the element value can be adjusted based on the de-duplication score of the element value to obtain an updated value.

[0118] Preferably, in one embodiment of the present invention, the method for obtaining the update value includes:

[0119] For any element value in any low-retention region block, the sum of the de-repetitiveness score of the element value and a preset constant is used as the adjustment weight.

[0120] The element values ​​are weighted adjusted based on the adjustment weights corresponding to each element value to obtain the update value corresponding to each element value. Taking element value a as an example, the formula model of the update value can be specifically as follows:

[0121] GX a =g a×*α+QC a +

[0122] Among them, GX a Indicates the updated value of element value a; g a Indicates element value a; QC a represents the de-duplication score of element value a; α represents a preset constant.

[0123] In the formula model of the update value, in the calculation process of the de-repetitiveness score, the proximity of the element value to 0 is taken into account, and the closer the element value is to 0, the greater the de-repetitiveness score of the element value will be. Therefore, when adjusting the element value, the degree of increase should be correspondingly greater. Conversely, if the element value is farther away from 0, the de-repetitiveness score of the element value will be smaller, and the degree of adjustment will be correspondingly smaller. Therefore, the sum of the de-repetitiveness score and the preset constant is used as the adjustment weight α+QC a In this embodiment of the present invention, in order to ensure that the adjusted update value is not too different from the element value before adjustment, the preset constant α is set to 1, and finally the product of the adjustment weight and the element value is used as the update value.

[0124] At this point, based on the above method, the update value corresponding to each element value in all low-retention area blocks can be obtained.

[0125] Step S4: Obtain an updated covariance matrix according to the updated values ​​corresponding to each element value in all low-retention area blocks and the element values ​​in the high-retention area blocks; perform principal component analysis on the updated covariance matrix to obtain a dimensionality reduction result for model training.

[0126] After the element values ​​in the low-retention area blocks are adjusted, the updated values ​​obtained will have lower repeatability and will imply more feature information. Therefore, the initial covariance matrix can be updated according to the updated values ​​corresponding to the element values ​​in the low-retention area blocks and the element values ​​in the high-retention area blocks that originally contain more feature information to obtain an updated covariance matrix.

[0127] Preferably, in one embodiment of the present invention, the method for obtaining the updated covariance matrix includes:

[0128] The element values ​​in all low-retention area blocks are replaced with the corresponding updated values, and the element values ​​in all high-retention area blocks remain unchanged, thereby obtaining an updated covariance matrix.

[0129] After obtaining the updated covariance matrix, the updated covariance matrix at this time can extract more characteristic data, so the updated covariance matrix is ​​subjected to subsequent principal component analysis process to obtain the dimension reduction result. Principal component analysis is a commonly used dimensionality reduction technique, which can project high-dimensional data into low-dimensional space while retaining the main direction of change in the data. In this process, the updated covariance matrix is ​​subjected to eigenvalue decomposition to obtain a series of principal components (i.e., eigenvectors) and corresponding variances (i.e., eigenvalues). It should be noted that the principal component analysis method is a technical means well known to those skilled in the art, and will not be described in detail herein.

[0130] The reduced-dimensionality alarm data not only reduces the dimensionality, but also retains the main change information in the original data, which enables the large model to process data more efficiently. The reduced-dimensional data is used as the input of the large model, and the model is trained to classify, cluster, or detect anomalies on the alarm data. The reduced-dimensional data can better meet the training needs of the large model, allowing the model to converge faster and achieve better performance. At the same time, dimensionality reduction also helps reduce the complexity and overfitting risk of the model and improve the generalization ability of the model.

[0131] It should be noted that, in order to facilitate calculation, all indicator data involved in the calculation in the embodiment of the present invention are preprocessed to eliminate the dimension effect. The specific means of eliminating the dimension effect are technical means well known to those skilled in the art and are not limited here.

[0132] The present invention also provides a computer-readable storage medium corresponding to the method provided in the above-mentioned embodiment. Figure 6 , the storage medium shown is a CD, on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, it will execute the method provided by any of the aforementioned embodiments.

[0133] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), and other optical and magnetic storage media, which are not listed here one by one.

[0134] In summary, the embodiment of the present invention aims at the problem that when a large amount of alarm data is subjected to dimensionality reduction analysis and the main components are extracted, the credibility of the dimensionality reduction result is low due to the presence of repeatability and other characteristics in the alarm data. First, a large amount of alarm data is obtained, and each alarm data contains multiple types of data, such as numerical type, text type, etc., and then different vector conversion methods are selected according to the type of data, so that the initial covariance matrix in the principal component analysis process can be obtained. Since a large amount of alarm data has similar characteristics, a large number of similar element values ​​will exist in the initial covariance matrix at this time, so that the feature vector finally extracted will have a strong correlation, so the key data features in the alarm data cannot be fully proposed, so it is necessary to adjust the element values ​​in the initial covariance matrix. The initial covariance matrix is ​​divided into regions to obtain initial regional blocks. Since data with significant differences from other data should have higher research value, that is, it should be feature data, for each initial regional block, the difference between the element values ​​and the position distribution are analyzed, and the reserved feature values ​​are obtained to distinguish low retention regional blocks from high retention regional blocks, wherein the data in the low retention regional blocks are non-feature data. For the data in the low-retention area block, because the data repetition is high, the sparsity of the data will be relatively enhanced. In order to reduce the impact of this part of the data in the subsequent dimensionality reduction process, it is necessary to adjust the value: for any element value in any low-retention area block, according to the difference between the element values ​​in the neighborhood blocks corresponding to the element value and the numerical fluctuation between the element values ​​in the corresponding neighborhood blocks, the de-repetitiveness score of the element value is obtained; and the element value is adjusted according to the de-repetitiveness score of the element value to obtain the updated value. At this time, the data repetitiveness in the adjusted low-retention area will be reduced, so the impact of this part of non-feature data on the subsequent dimensionality reduction process can be reduced. Therefore, in the initial covariance matrix, the corresponding element value is replaced by the updated value, and then the subsequent principal component dimensionality reduction process is continued to obtain the dimensionality reduction result. At this time, the dimensionality reduction result eliminates the data, so it can better represent the key features of the alarm data, so when the dimensionality reduction result is used for model training, the accuracy will also be improved.

[0135] It should be noted that the sequence of the above embodiments of the present invention is only for description and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0136] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

Claims

1. A method for analyzing alarm data based on a large model, characterized in that: The method comprises: Acquire multiple pieces of alarm data, each piece of alarm data contains multiple types of data; perform vector conversion on each type of data in all the alarm data to obtain an initial covariance matrix corresponding to all the alarm data; Performing regional division on the initial covariance matrix to obtain all initial regional blocks; for any initial regional block, analyzing the relationship between the difference, randomness and position distribution of element values ​​in the initial regional block to obtain the retained eigenvalue of the initial regional block; distinguishing low-retention regional blocks from high-retention regional blocks according to the retained eigenvalues ​​of all initial regional blocks; Determine the neighborhood blocks of each element value in each low-retention area block; for any element value in any low-retention area block, obtain a de-repetitiveness score of the element value according to the difference between element values ​​in the neighborhood blocks corresponding to the element value and the numerical fluctuation between the element values ​​in the corresponding neighborhood blocks; adjust the element value according to the de-repetitiveness score of the element value to obtain an updated value; According to the update values ​​corresponding to the element values ​​in all low-retention area blocks and the element values ​​in the high-retention area blocks, an updated covariance matrix is ​​obtained; principal component analysis is performed on the updated covariance matrix to obtain a dimension reduction result for model training; The method for obtaining the retained characteristic value comprises: For any initial region block, the entropy value of all element values ​​in the initial region block is used as the randomness characteristic value; All element values ​​in the initial region block are combined in pairs to obtain all non-repeating element value combinations. For any element value combination, the difference between the two element values ​​is calculated as the difference factor, and the Euclidean distance between the positions of the two element values ​​is calculated as the distance factor. Obtaining importance scores of the two element values ​​according to the difference factor and the distance factor, wherein the importance score is positively correlated with the difference factor, and negatively correlated with the distance factor, and the importance score is a normalized value; The average of the importance scores corresponding to all element value combinations in the initial area block is used as the retention factor of the initial area block; the retention characteristic value of the initial area block is obtained according to the retention factor and the random characteristic value of the initial area block, the retention factor and the random characteristic value are positively correlated with the retention characteristic value, and the value of the retention characteristic value is a normalized value.

2. The method for analyzing alarm data based on a large model according to claim 1, characterized in that: The step of distinguishing low-retention-degree regional blocks from high-retention-degree regional blocks according to the retention feature values ​​of all initial regional blocks includes: The initial region block corresponding to the retention feature value greater than or equal to the preset retention threshold is used as a high retention region block; The initial region blocks corresponding to the retention feature values ​​less than the preset retention threshold are regarded as low retention region blocks.

3. The method for analyzing alarm data based on a large model according to claim 1, characterized in that: The step of determining a neighborhood block of each element value in each low retention area block comprises: Taking each element value as the center, a preset second number of element values ​​that are adjacent and closest in a preset direction are taken as neighborhood element values, and an area formed by the central element value and the corresponding neighborhood element values ​​is taken as a neighborhood block corresponding to the central element value.

4. The method for analyzing alarm data based on a large model according to claim 3 is characterized in that: The method for obtaining the de-duplication score includes: For any element value in any low-retention area block, a first repetition factor corresponding to the element value is obtained according to the difference between the element value and all neighboring element values ​​in the corresponding neighboring block; Calculate the mean of all element values ​​in a neighborhood block centered on the element value as the mean eigenvalue, and take the difference between the element value and the mean eigenvalue as the deviation value; obtain a second repetition factor corresponding to the element value according to the element value and the corresponding deviation value, wherein both the deviation value and the element value are negatively correlated with the second repetition factor; A de-repetitiveness score of the element value is obtained according to a first repetitiveness factor and a second repetitiveness factor of the element value, wherein both the first repetitiveness factor and the second repetitiveness factor are positively correlated with the de-repetitiveness score, and the value of the de-repetitiveness score is a normalized value.

5. The method for analyzing alarm data based on a large model according to claim 4 is characterized in that: The method for obtaining the first repetition factor includes: In a neighborhood block centered on the element value, the numerical difference between the element value and each corresponding neighborhood element value is taken as the difference coefficient of the element value; The means of all the difference coefficients corresponding to the element value are negatively correlated and normalized to obtain a value as the first repetition factor; Taking element value a as an example, the formula for the first repetition factor is: Among them, CF a Indicates the first repetition factor of element value a; N a Indicates the total number of neighborhood element values ​​of element value a; g a Represents element value a; Represents the kth neighboring element value of element value a; exp() represents the exponential function with the natural constant e as the base.

6. The method for analyzing alarm data based on a large model according to claim 1, characterized in that: The method for obtaining the update value includes: For any element value in any low-retention area block, the sum of the de-repetitiveness score of the element value and a preset constant is used as an adjustment weight; The element values ​​are weighted adjusted based on the adjustment weights corresponding to each element value to obtain an updated value corresponding to each element value.

7. The method for analyzing alarm data based on a large model according to claim 1, characterized in that: The method for obtaining the updated covariance matrix includes: The element values ​​in all low-retention area blocks are replaced with the corresponding updated values, and the element values ​​in all high-retention area blocks remain unchanged, thereby obtaining an updated covariance matrix.

8. The method for analyzing alarm data based on a large model according to claim 1, characterized in that: The method for obtaining the initial region block includes: The initial covariance matrix is ​​evenly divided into a preset first number of initial area blocks.

9. The method for analyzing alarm data based on a large model according to claim 2, characterized in that: The preset retention threshold is set to 0.85.

Images