Intelligent information processing system based on big data operation and maintenance

Abstract

The invention relates to the technical field of information supervision, and discloses an intelligent information processing system based on big data operation and maintenance, which comprises a data acquisition module, a feature extraction module, a data analysis module and a decision analysis module, the data acquisition module is used for acquiring abnormal data which is the same as to-be-processed information from internet big data through an API (Application Program Interface) and a web crawler, and feeding back the acquired abnormal data to the feature extraction module; and the feature extraction module is used for selecting features with the maximum correlation from the abnormal data collected from the Internet and the information data to be processed for extraction. According to the method, a double-loop judgment logic design is adopted, and the inner loop is used for judging whether the to-be-processed information is substantially the same as the data marked as abnormal in the big data or not, so that access and calling of the system or other third parties to the to-be-processed information are limited; and the external loop is used for secondarily judging whether the information is the same or not and actively modifying the security state of the processing information by combining the judgment of an administrator.

Description

Technical Field

[0001] The present invention relates to the field of information monitoring technology, and in particular to an intelligent information processing system based on big data operation and maintenance. Background Art

[0002] Intelligent information processing systems based on big data operation and maintenance are becoming an increasingly important part of modern enterprises and organizations. They use advanced technologies such as big data technology, artificial intelligence and cloud computing to optimize information processing and operation and maintenance management.

[0003] The existing patent discloses a big data detection system and method based on information security (publication number CN114840851A), including: a data acquisition module, a manual operation status acquisition module, a device information self-check module, a device risk feature extraction module and a device information security analysis module. The data acquisition module is used to obtain the status data of the user's device being attacked by viruses in the first unit time; the manual operation status acquisition module obtains all operation records of the user on the device in the first unit time. In the technology disclosed in this patent, during the monitoring process of information that needs to be processed and security judged, there is often a lack of the ability to make accurate and comprehensive judgments on various indicators of the data information to be processed and judged. Summary of the invention

[0004] The main technical problem solved by the present invention is to provide an intelligent information processing system based on big data operation and maintenance, which solves the problems in the above-mentioned background technology.

[0005] To solve the above technical problems, according to one aspect of the present invention, more specifically, a smart information processing system based on big data operation and maintenance includes a data acquisition module, a feature extraction module, a data analysis module, a decision analysis module, a data storage module and an access control module;

[0006] The data collection module collects abnormal data of the same type as the information to be processed from the Internet big data through API and web crawlers, and feeds the collected abnormal data back to the feature extraction module;

[0007] The feature extraction module is used to extract the most relevant feature from the abnormal data collected from the Internet and the information data to be processed, and feed the extracted feature data back to the data analysis module;

[0008] The data analysis module applies machine learning algorithms to calculate key statistical indicators to summarize data features, then uses regression analysis to predict future trends, mine potential trends in data features, and feeds back the predicted data trends and summarized data features to the decision analysis module;

[0009] The decision analysis module sends an instruction on whether to restrict permissions to the access control module based on the result of data analysis;

[0010] The access control module limits the access and calling authority of the data acquisition module and other third parties to the information data to be processed according to the feedback from the decision analysis module.

[0011] Furthermore, the data storage module is used to extract feature information from historical data, store decision information made according to the corresponding feature information, and feed back the stored historical data to the data analysis module for trend prediction.

[0012] Furthermore, the decision analysis module determines whether the data to be processed is classified as abnormal data based on the statistical indicators provided by the data analysis module, the correlation indicators between the abnormal data and the information data to be processed, and the frequency indicators with the characteristics of the correlation data, including:

[0013]

[0014] In the formula, W represents the probability coefficient of the data to be processed being classified as abnormal data, h represents the probability coefficient of the correlation between the abnormal data and the information data to be processed, c represents the frequency index coefficient with the characteristics of the correlation data, m represents the probability coefficient of the statistical correlation between the abnormal data and the information data to be processed, and there are:

[0015]

[0016] Where m represents the probability coefficient of the statistical correlation between the abnormal data and the information data to be processed, a represents the absolute value of the difference between the mean of the abnormal data and the mean of the information data to be processed, and s represents the standard deviation between the abnormal data and the data to be processed.

[0017] Furthermore, the decision analysis module determines the correlation index between the abnormal data and the information data to be processed according to the strength of the linear relationship between the abnormal data and the information data to be processed, and the relationship of the continuous variables between the abnormal data and the information data to be processed, which are:

[0018]

[0019] Wherein, h represents the correlation probability coefficient between abnormal data and the information data to be processed, u represents the Pearson correlation coefficient between the abnormal data and the information data to be processed, and k represents the absolute value of the maximum continuous variable difference between the abnormal data and the information data to be processed.

[0020] Furthermore, the decision analysis module determines the frequency index of the relevant data features according to the number of times the relevant data features appear in the data set and the proportion of the relevant data features in the total data, which are:

[0021]

[0022] In the formula, c represents the frequency index coefficient of the relevant data feature, n represents the number of times the relevant data feature appears in the data set, and d represents the proportion of the relevant data feature in the total data.

[0023] Furthermore, when W < 85%, it means that the data to be processed is not abnormal data;

[0024] When W ≥ 85%, it means that the data to be processed is classified as abnormal data.

[0025] Furthermore, it also includes an operation and maintenance monitoring module and a data visualization unit.

[0026] Furthermore, the operation and maintenance monitoring module receives the maximum correlation feature fed back by the feature extraction module and the decision instruction of the decision analysis module to monitor and re-determine whether the decision instruction of the decision analysis module is abnormal;

[0027] The data visualization unit is used to receive the abnormal judgment information and integrated data information from the operation and maintenance monitoring module, and display key indicators and data analysis results through dashboards and visualization tools.

[0028] Beneficial effects:

[0029] 1. The present invention adopts a double-loop judgment logic design. The inner loop is used to determine whether the information to be processed is substantially the same as the data marked as abnormal in the big data, thereby limiting the access and call of the system or other third parties to the information to be processed; and the outer loop is used to secondarily determine whether the data to be processed is classified as abnormal data, and actively modify the security status of the processed information in combination with the judgment of the administrator.

[0030] 2. The present invention can accurately and efficiently determine whether the data to be processed is classified as abnormal data based on the statistical indicators provided by the data analysis module, the correlation indicators between the abnormal data and the information data to be processed, and the frequency indicators with the characteristics of the correlation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 It is a structural diagram of the intelligent information processing system in the present invention. DETAILED DESCRIPTION

[0032] In order to make the technical solution of the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] Example 1

[0034] like Figure 1 As shown, according to one aspect of the present invention, there is provided a smart information processing system based on big data operation and maintenance, including a data acquisition module, a feature extraction module, a data analysis module, a decision analysis module, a data storage module and an access control module; the data acquisition module, through an API and a web crawler, collects abnormal data that is classified as the same type as the information to be processed from the Internet big data, and feeds the collected abnormal data back to the feature extraction module; the feature extraction module is used to select the most relevant features from the abnormal data collected from the Internet and the information data to be processed for extraction, and feed the extracted feature data back to the data analysis module; the data analysis module applies a machine learning algorithm to calculate key statistical indicators to summarize data features, and then uses regression analysis to predict future trends, mine potential trends in data features, and feed back the predicted data trends and summarized data features to the decision analysis module; the decision analysis module sends an instruction to the access control module on whether to restrict permissions based on the results of the data analysis; the access control module limits the access and calling permissions of the data acquisition module and other third parties to the information data to be processed according to the feedback from the decision analysis module. A double-loop judgment logic design is adopted. The inner loop is used to determine whether the information to be processed is essentially the same as the data marked as abnormal in the big data, thereby limiting the access and call of the system or other third parties to the information to be processed; and the outer loop is used for a second judgment of whether they are the same, and combined with the administrator's judgment to actively modify the security status of the processed information.

[0035] In this embodiment, the data storage module is used to extract feature information from historical data, store decision information made according to the corresponding feature information, and feed back the stored historical data to the data analysis module for trend prediction.

[0036] Example 2

[0037] like Figure 1 As shown, the decision analysis module determines the correlation index between the abnormal data and the information data to be processed according to the strength of the linear relationship between the abnormal data and the information data to be processed, and the relationship of the continuous variables between the abnormal data and the information data to be processed, which are:

[0038]

[0039] Wherein, h represents the correlation probability coefficient between abnormal data and the information data to be processed, u represents the Pearson correlation coefficient between the abnormal data and the information data to be processed, and k represents the absolute value of the maximum continuous variable difference between the abnormal data and the information data to be processed.

[0040] The Pearson correlation coefficient between the information data X to be processed and the abnormal data Y is u=0.4 (the Pearson correlation coefficient is a statistical indicator that measures the strength and direction of the linear relationship between two variables, and its value range is from 0 to 1. The larger the value of the Pearson correlation coefficient, the stronger the linear relationship between the two variables). The absolute value of the maximum continuous variable difference between the information data X to be processed and the abnormal data Y is k=4.2 (the maximum continuous variable represents the data value with the most consecutive times in the information data or abnormal data to be processed). Then:

[0041]

[0042] From the above calculation, it can be known that the correlation probability coefficient between the information data X to be processed and the abnormal data Y is h=66.1%.

[0043] In this embodiment, the decision analysis module determines the frequency index of the data feature with correlation according to the number of times the data feature with correlation appears in the data set and the proportion of the data feature with correlation to the total data, which are:

[0044]

[0045] In the formula, c represents the frequency index coefficient of the relevant data feature, n represents the number of times the relevant data feature appears in the data set, and d represents the proportion of the relevant data feature in the total data.

[0046] For the information data X and the abnormal data Y to be processed, the number of times the relevant data features appear in the data set is n=14 (the number of times in the data set generally does not exceed), and the proportion of relevant data features to the total data is d=0.13, then:

[0047]

[0048] From the above calculation, it can be known that the frequency index coefficient of the correlation data characteristics between the information data X to be processed and the abnormal data Y is c=0.72.

[0049] The decision analysis module determines whether the data to be processed is classified as abnormal data based on the statistical indicators provided by the data analysis module, the correlation indicators between the abnormal data and the information data to be processed, and the frequency indicators with the characteristics of the correlation data, which are:

[0050]

[0051] In the formula, W represents the probability coefficient of the data to be processed being classified as abnormal data, h represents the probability coefficient of the correlation between the abnormal data and the information data to be processed, c represents the frequency index coefficient with the characteristics of the correlation data, m represents the probability coefficient of the statistical correlation between the abnormal data and the information data to be processed, and there are:

[0052]

[0053] Where m represents the probability coefficient of the statistical correlation between the abnormal data and the information data to be processed, a represents the absolute value of the difference between the mean of the abnormal data and the mean of the information data to be processed, and s represents the standard deviation between the abnormal data and the data to be processed.

[0054] When W < 85%, it means that the data to be processed is not abnormal data;

[0055] When W ≥ 85%, it means that the data to be processed is classified as abnormal data.

[0056] For the information data to be processed X and the abnormal data Y, the absolute value of the difference between the mean of the abnormal data X and the mean of the information data to be processed Y is a=6.3, and the standard deviation between the abnormal data X and the data to be processed Y is s=2.3, then:

[0057]

[0058] From the above calculation, it can be known that the probability coefficient of the statistical correlation between the information data X to be processed and the abnormal data Y is m=81%.

[0059] Then, from the calculation of the above values, we can know that the probability coefficient of the data to be processed X being classified as abnormal data is:

[0060]

[0061] From the above calculation, it can be known that the probability coefficient of the data to be processed X being classified as abnormal data is W=85%, so the data to be processed X is classified as abnormal data.

[0062] Example 3

[0063] In this embodiment, the operation and maintenance monitoring module receives the maximum correlation feature fed back by the feature extraction module and the decision instruction of the decision analysis module to monitor and re-determine whether the decision instruction of the decision analysis module is abnormal; the data visualization unit is used to receive the abnormal judgment information of the operation and maintenance monitoring module and the integrated data information, and display the key indicators and data analysis results (such as Figure 1 As shown), the outer loop is used to secondarily determine whether the data to be processed is classified as abnormal data, and actively modify the security status of the processed information based on the judgment of the administrator.

[0064] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.

Claims

1. A smart information processing system based on big data operation and maintenance, characterized in that: It includes data acquisition module, feature extraction module, data analysis module, decision analysis module, data storage module and access control module; The data collection module collects abnormal data of the same type as the information to be processed from the Internet big data through API and web crawlers, and feeds the collected abnormal data back to the feature extraction module; The feature extraction module is used to extract the most relevant feature from the abnormal data collected from the Internet and the information data to be processed, and feed the extracted feature data back to the data analysis module; The data analysis module applies machine learning algorithms to calculate key statistical indicators to summarize data features, then uses regression analysis to predict future trends, mine potential trends in data features, and feeds back the predicted data trends and summarized data features to the decision analysis module; The decision analysis module sends an instruction on whether to restrict permissions to the access control module based on the result of data analysis; The access control module limits the access and calling authority of the data acquisition module and other third parties to the information data to be processed according to the feedback from the decision analysis module.

2. The intelligent information processing system based on big data operation and maintenance according to claim 1 is characterized by: The data storage module is used to extract feature information from historical data, store decision information made according to the corresponding feature information, and feed back the stored historical data to the data analysis module for trend prediction.

3. The intelligent information processing system based on big data operation and maintenance according to claim 1 is characterized by: The decision analysis module determines whether the data to be processed is classified as abnormal data based on the statistical indicators provided by the data analysis module, the correlation indicators between the abnormal data and the information data to be processed, and the frequency indicators with the characteristics of the correlation data, including: In the formula, W represents the probability coefficient of the data to be processed being classified as abnormal data, h represents the probability coefficient of the correlation between the abnormal data and the information data to be processed, c represents the frequency index coefficient with the characteristics of the correlation data, m represents the probability coefficient of the statistical correlation between the abnormal data and the information data to be processed, and there are: Where m represents the probability coefficient of the statistical correlation between the abnormal data and the information data to be processed, a represents the absolute value of the difference between the mean of the abnormal data and the mean of the information data to be processed, and s represents the standard deviation between the abnormal data and the data to be processed.

4. The intelligent information processing system based on big data operation and maintenance according to claim 3 is characterized by: The decision analysis module determines the correlation index between the abnormal data and the information data to be processed according to the strength of the linear relationship between the abnormal data and the information data to be processed, as well as the relationship between the continuous variables between the abnormal data and the information data to be processed, which are: Wherein, h represents the correlation probability coefficient between abnormal data and the information data to be processed, u represents the Pearson correlation coefficient between the abnormal data and the information data to be processed, and k represents the absolute value of the maximum continuous variable difference between the abnormal data and the information data to be processed.

5. The intelligent information processing system based on big data operation and maintenance according to claim 4 is characterized in that: The decision analysis module determines the frequency index of the relevant data features according to the number of times the relevant data features appear in the data set and the proportion of the relevant data features in the total data, which are: In the formula, c represents the frequency index coefficient of the relevant data feature, n represents the number of times the relevant data feature appears in the data set, and d represents the proportion of the relevant data feature in the total data.

6. The intelligent information processing system based on big data operation and maintenance according to claim 5 is characterized by: When W < 85%, it means that the data to be processed is not abnormal data; When W ≥ 85%, it means that the data to be processed is classified as abnormal data.

7. The intelligent information processing system based on big data operation and maintenance according to claim 1 is characterized in that: Also included is a data visualization unit; The data visualization unit is used to receive the abnormal judgment information and integrated data information from the operation and maintenance monitoring module, and display key indicators and data analysis results through dashboards and visualization tools.

Images