Network abnormal data processing method and system based on big data

Abstract

The invention relates to the field of data processing, and particularly discloses a network abnormal data processing method and system based on big data, and the method comprises the steps: obtaining network data, calculating the difference change rate of the network data, and determining normal data and preliminary abnormal data in the network data; determining the ratio of the abnormal data to the normal data, the type of the abnormal data and source equipment of the abnormal data; determining a processing strategy of the abnormal data; and executing the processing strategy, and repairing the source equipment. According to the method, the network data are preliminarily determined and finally determined, the final abnormal data are obtained through two-time determination, the accuracy is improved, the load on the server each time is reduced, then the processing strategy is determined by comprehensively considering the quantity proportion and the type of the abnormal data, the processing strategy is made to better conform to the actual situation, and the processing efficiency is improved. According to the method, the influence of the abnormal data can be eliminated, the source equipment of the abnormal data is repaired, and generation of the abnormal data is prevented.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a method and system for processing abnormal network data based on big data. Background Art

[0002] Oil is one of the main objects of geological exploration. It is a viscous, dark brown liquid and is called the "blood of industry". Oil is stored in some areas of the upper crust. The main component is a mixture of various alkanes, cycloalkanes, and aromatic hydrocarbons. It is also an important energy source. The data such as oil reserves, mining conditions, and location also need to be protected. The equipment at the oil mining site usually sends the relevant network data to the server in the confidential room through the network, and the server in the confidential room processes and stores the network data.

[0003] These data are sent to the server through the network by the detection equipment at the oil production site. However, due to the on-site working environment, network transmission and other reasons, abnormal data will appear in these data and must be cleared.

[0004] Currently, a one-time determination method is often used to clear abnormal data. Although this method can clear some abnormal data, the workload is too large at one time, which puts a heavy load on the server. In addition, one-time determination may mistake some normal data for abnormal data due to chance, and the accuracy is not high. Summary of the invention

[0005] The purpose of the present invention is to provide a method and system for processing network abnormal data based on big data to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions: A method for processing network abnormal data based on big data, the method comprising: Obtain network data, which includes real-time network data and historical network data, calculate the difference change rate between the real-time network data and the network data of the previous moment in the historical data, the difference change rate is (real-time network data - network data of the previous moment) / network data of the previous moment, and determine normal data and preliminary abnormal data in the network data based on the difference change rate; Perform feature extraction and feature vector calculation on the preliminary abnormal data to obtain the abnormal feature vector of each preliminary abnormal data, average all the abnormal feature vectors to obtain the mean feature vector, calculate the Manhattan distance between each abnormal feature vector and the mean feature vector and compare it with twice the standard deviation, determine the ratio of abnormal data to normal data, the type of abnormal data and the source device of the abnormal data; Determine the strategy for handling abnormal data based on the ratio of abnormal data to normal data and the type of abnormal data; Execute the processing strategy and analyze the reasons why the source device of the abnormal data generates the abnormal data, and repair the source device.

[0007] As a further solution of the present invention: the step of calculating the difference change rate between the real-time network data and the network data at the previous moment in the historical data, and determining the normal data and preliminary abnormal data in the network data according to the difference change rate comprises: Calculate the difference change rate, difference change rate = (real-time network data - network data of the previous moment) / network data of the previous moment; The difference change rate is compared with a preset threshold value. When the difference change rate is greater than the preset threshold value, the real-time network data is preliminary abnormal data.

[0008] As a further solution of the present invention: the step of calculating the Manhattan distance between each abnormal feature vector and the mean feature vector and comparing it with twice the standard deviation to determine the ratio of abnormal data to normal data, the type of abnormal data and the source device of the abnormal data includes: Calculate the Manhattan distance between each abnormal feature vector and the mean feature vector, and compare the result with twice the standard deviation. When the result is greater than twice the standard deviation, the preliminary abnormal data corresponding to the abnormal feature vector is abnormal data; Analyze the types and causes of all abnormal data, divide the number of abnormal data by the number of normal data, and obtain the ratio of abnormal data to normal data, the type of abnormal data, and the source device of the abnormal data.

[0009] As a further solution of the present invention: the step of determining the processing strategy of abnormal data according to the ratio of abnormal data to normal data and the type of abnormal data includes: The expert weight method is used in conjunction with the questionnaire to obtain the type score and type weight of each abnormal data; The impact score of each type of abnormal data is obtained by multiplying the percentage of the number of abnormal data types by the type score of the abnormal data and the type weight of the abnormal data. The sum of the impact scores of all types of abnormal data is the abnormal score. The category of the abnormality is determined based on the abnormal score, and the processing strategy is determined based on the category of the abnormality.

[0010] The technical solution of the present invention also provides a system for processing network abnormal data based on big data, the system comprising: A preliminary determination module is used to obtain network data, including real-time network data and historical network data, calculate the difference change rate between the real-time network data and the network data of the previous moment in the historical data, the difference change rate is (real-time network data - network data of the previous moment) / network data of the previous moment, and determine normal data and preliminary abnormal data in the network data based on the difference change rate; A final determination module is used to perform feature extraction and feature vector calculation on the preliminary abnormal data to obtain an abnormal feature vector of each preliminary abnormal data, average all abnormal feature vectors to obtain a mean feature vector, calculate the Manhattan distance between each abnormal feature vector and the mean feature vector and compare it with twice the standard deviation, determine the ratio of abnormal data to normal data, the type of abnormal data and the source device of the abnormal data; A strategy determination module, used to determine a processing strategy for abnormal data based on the ratio of abnormal data to normal data and the type of abnormal data; The processing module is used to execute the processing strategy and analyze the reasons why the source device of the abnormal data generates the abnormal data, and repair the source device.

[0011] As a further solution of the present invention: the preliminary determination module includes: An acquisition unit, used for acquiring network data, the network data including real-time network data and historical network data; The first calculation unit is used to calculate the difference change rate, where the difference change rate = (real-time network data - network data at the last moment) / network data at the last moment; The preliminary determination unit is used to compare the difference change rate with a preset threshold value. When the difference change rate is greater than the preset threshold value, the real-time network data is preliminary abnormal data.

[0012] As a further solution of the present invention: the final determination module includes: A feature vector unit is used to perform feature extraction and feature vector calculation on the preliminary abnormal data to obtain an abnormal feature vector of each preliminary abnormal data, and to average all the abnormal feature vectors to obtain a mean feature vector; The second calculation unit is used to calculate the Manhattan distance between each abnormal feature vector and the mean feature vector, and compare the result with twice the standard deviation. When the result is greater than twice the standard deviation, the preliminary abnormal data corresponding to the abnormal feature vector is abnormal data; The final determination unit is used to analyze the types and causes of all abnormal data, divide the number of abnormal data by the number of normal data, and obtain the ratio of abnormal data to normal data, the type of abnormal data, and the source device of the abnormal data.

[0013] As a further solution of the present invention: the strategy determination module includes: A parameter determination unit, used to obtain the type score and type weight of each abnormal data by using an expert weight method in conjunction with a questionnaire; The strategy determination unit is used to multiply the quantity percentage of each abnormal data type by the type score of the abnormal data and the type weight of the abnormal data to obtain the impact score of the abnormal data. The sum of the impact scores of all types of abnormal data is the abnormal score. The category of the abnormality is determined based on the abnormal score, and the processing strategy is determined based on the category of the abnormality.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention performs preliminary determination and final determination on the network data, and obtains the final abnormal data by means of two determinations, which not only improves the accuracy rate, but also reduces the load on the server each time, and then determines the processing strategy by comprehensively considering the quantity, proportion and type of the abnormal data, so that the processing strategy is more in line with the actual situation, and can eliminate the impact of the abnormal data. The source device of the abnormal data is also repaired to ensure that the network data subsequently generated by the source device is normal data, thereby preventing the generation of abnormal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention.

[0016] Figure 1 The flowchart is a method for processing abnormal network data based on big data.

[0017] Figure 2 This is a block diagram of the first sub-process of a method for processing abnormal network data based on big data.

[0018] Figure 3 This is a block diagram of the second sub-process of part of the method for processing network abnormal data based on big data.

[0019] Figure 4 This is a block diagram of the third sub-process of the method for processing network abnormal data based on big data.

[0020] Figure 5 This is a structural block diagram of the network anomaly data processing system based on big data.

[0021] Figure 6 A structural block diagram of the modules that are preliminarily determined in the network anomaly data processing system based on big data.

[0022] Figure 7 This is a structural block diagram of the final determination module in the network anomaly data processing system based on big data.

[0023] Figure 8 This is a structural block diagram of the composition of the strategy determination module in the network anomaly data processing system based on big data. DETAILED DESCRIPTION

[0024] At present, it is still necessary for humans to judge the situation captured by the camera before taking measures. Monitoring personnel monitor multiple cameras at the same time and rely on their experience to make judgments. Unsafe behaviors may be judged as safe behaviors due to their experience, or monitoring personnel may not be able to detect unsafe behaviors in time.

[0025] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0026] Embodiment 1: Figure 1 The flowchart of the method for processing network abnormal data based on big data is as follows. In an embodiment of the present invention, a method for processing network abnormal data based on big data includes: Step S100: Acquire network data, which includes real-time network data and historical network data, calculate the difference change rate between the real-time network data and the network data of the previous moment in the historical data, the difference change rate is (real-time network data - network data of the previous moment) / network data of the previous moment, and determine normal data and preliminary abnormal data in the network data according to the difference change rate; The network data is not abnormal data at the beginning, but it mutates at a certain moment, and the degree of this mutation is very large. Therefore, the difference change rate is used to describe the degree of mutation of the network data. The difference between the real-time network data and the network data of the previous moment divided by the network data of the previous moment is the difference change rate. The difference change rate has positive and negative values. Here, the absolute value of the difference change rate is taken, and the absolute value of the difference change rate is compared with the preset preliminary abnormal difference rate. When the absolute value of the difference change rate is greater than the preset preliminary abnormal difference rate, the network data is identified as preliminary abnormal data, and the rest are normal data. The preset preliminary abnormal difference rate is obtained by relevant personnel based on the experience of the oil production site. For example, if a component of the sensor fails, the mutation degree of the data generated at this time is about 20%. The preset preliminary abnormal difference rate is 15%, that is, when the absolute value of the difference rate between a real-time network data and the data of the previous moment is greater than 15%, the real-time network data is identified as preliminary abnormal data. In order to reduce the workload and ensure the accuracy of the results, the network data can be cleaned, supplemented and preprocessed before calculating the difference change rate. Cleaning is to remove obviously erroneous data, supplementation is to repair missing network data to ensure sufficient samples and complete data, and preprocessing is to normalize the format of network data to reduce subsequent work time.

[0027] Step S200: extracting features and performing feature vector calculation on the preliminary abnormal data to obtain an abnormal feature vector for each preliminary abnormal data, averaging all abnormal feature vectors to obtain a mean feature vector, calculating the Manhattan distance between each abnormal feature vector and the mean feature vector and comparing it with twice the standard deviation, and determining the ratio of abnormal data to normal data, the type of abnormal data, and the source device of the abnormal data; Feature extraction is to summarize the universal law of preliminary abnormal data. Feature vector operation usually includes eigenvalue calculation, matrix transformation and space mapping, which is to improve the accuracy of abnormal feature vector. Matrix transformation uses symmetric matrix, and space mapping is to reduce the dimension of features. After these operations, the abnormal feature vector of each preliminary abnormal data can be obtained. The average value of all abnormal feature vectors can realize the fusion of abnormal feature vectors. Then, the Manhattan distance can be used to see the distance between the abnormal feature vector and the mean feature vector of each preliminary abnormal data. Then, it is compared with the preset threshold (twice the standard deviation) to determine whether the preliminary abnormal data is abnormal data. The reason why twice the standard deviation is compared with the Manhattan distance is that twice the standard deviation is a dynamic value, which can better utilize the characteristics of all preliminary abnormal data and can change with the change of preliminary abnormal data. After determining whether each preliminary abnormal data is abnormal data, the number of normal data and abnormal data in all network data can be known, and the ratio of abnormal data to normal data can be determined. After analyzing and tracing the abnormal data, the type of abnormal data and the source device of the abnormal data can be known.

[0028] Step S300: Determine a processing strategy for abnormal data based on the ratio of abnormal data to normal data and the type of abnormal data; The type of abnormal data indicates the cause of the abnormal data. The larger the ratio of abnormal data to normal data, the greater the abnormality of the entire data. The server usually stores the process of how to handle abnormal data in the past. Taking all the above factors into consideration, the degree of harm of the current abnormal data to the overall security can be determined, and the processing strategy similar to the current abnormal situation can be queried from the server, which is the processing strategy for abnormal data.

[0029] Step S400: executing the processing strategy and analyzing the reasons why the source device of the abnormal data generates the abnormal data, and repairing the source device.

[0030] After executing the processing strategy, it is also necessary to generate a processing evaluation report and the management personnel's evaluation information on the processing strategy. These will be fed back to the server to provide a basis for subsequent improvement of the processing strategy. The processing strategy can only process the abnormal data that has been generated, but it is also necessary to repair the source device that generated the abnormal data, so that the network data subsequently generated by the source device can be normal data, which is convenient for subsequent processing. The strategy for repairing the source device usually includes installing firewalls, replacing data keys, replacing parts, etc., which can make the data sent to the server by the equipment at the mining site normal, and can know the true and accurate situation of oil resources.

[0031] Figure 2 The first sub-flow diagram of the method for processing abnormal network data based on big data is shown in FIG. 1 , wherein the step of calculating the difference change rate between the real-time network data and the network data at the previous moment in the historical data, and determining the normal data and the preliminary abnormal data in the network data according to the difference change rate includes: Calculate the difference change rate, difference change rate = (real-time network data - network data of the previous moment) / network data of the previous moment; Subtract the network data of the previous moment from the real-time network data, and divide the difference into the network data of the previous moment. The result is the rate of change of the difference, that is, the degree of mutation of the real-time network data relative to the network data of the previous moment.

[0032] The difference change rate is compared with a preset threshold value. When the difference change rate is greater than the preset threshold value, the real-time network data is preliminary abnormal data.

[0033] Relevant personnel determine the preset preliminary abnormal difference rate based on on-site experience. When the difference change rate is greater than the preset preliminary abnormal difference rate, it means that the real-time network data has a large mutation degree and needs to be further tested. Therefore, the real-time network data is determined as preliminary abnormal data. Only preliminary abnormal data needs to be screened later, which greatly reduces the subsequent workload. Here, the relevant personnel determine the preset preliminary abnormal difference rate to be 15%, that is, when the difference between a real-time network data and the network data of the previous moment is 16%, the real-time network data will be identified as preliminary abnormal data; when the difference between another real-time network data and the network data of the previous moment is 14%, the real-time network data will be identified as normal data.

[0034] Figure 3 This is a block diagram of a second sub-process of a method for processing abnormal network data based on big data, wherein the steps of calculating the Manhattan distance between each abnormal feature vector and the mean feature vector and comparing it with twice the standard deviation, and determining the ratio of abnormal data to normal data, the type of abnormal data, and the source device of the abnormal data include: Calculate the Manhattan distance between each abnormal feature vector and the mean feature vector, and compare the result with twice the standard deviation. When the result is greater than twice the standard deviation, the preliminary abnormal data corresponding to the abnormal feature vector is abnormal data; Manhattan distance is the sum of the absolute axis distances between the abnormal feature vector and the mean feature vector in the coordinate system, that is, h=|xi-xj|+|yi-yj|+…, h is the Manhattan distance, xi and xj are the coordinates of the abnormal feature vector and the mean feature vector on the x-axis respectively, and the coordinates on the other coordinate axes are deduced accordingly. The value of twice the standard deviation is used as the basis for judging the gap between the abnormal feature vector and the mean feature vector. It is derived from historical abnormal data. Each time abnormal data occurs, corresponding records will be made. These historical abnormal data are sorted and processed to obtain the characteristics of the abnormal data, and then the preset threshold of the Manhattan distance is determined to be twice the standard deviation. The Manhattan distance between each abnormal feature vector and the mean feature vector is compared with twice the standard deviation. When the Manhattan distance is greater than twice the standard deviation, it indicates that the difference between the two is large. At this time, the abnormal feature vector can be determined as an abnormal vector, and the preliminary abnormal data corresponding to the abnormal vector is the abnormal data.

[0035] Analyze the types and causes of all abnormal data, divide the number of abnormal data by the number of normal data, and obtain the ratio of abnormal data to normal data, the type of abnormal data, and the source device of the abnormal data.

[0036] After all abnormal data are determined, each abnormal data is analyzed to analyze the type and cause of abnormal data. The ratio of abnormal data to normal data can show the degree of data abnormality. The fewer abnormal data, the smaller the ratio of abnormal data to normal data, the lower the abnormality of the data, and the smaller the harm of data abnormality. The type of abnormal data can be determined by similar matching method, that is, matching with historical abnormal data to see which abnormal data is most similar to the previous one. Different types of abnormal data have different harms. Common causes of abnormal data include component failure, abnormality in call, leakage in transmission, and equipment being attacked by hackers. After determining that a certain real-time network data is abnormal data, reverse tracing can be used to query which device sent the abnormal data, and the source device of the abnormal data can be known.

[0037] Figure 4 The third sub-flow chart of the method for processing abnormal network data based on big data, wherein the step of determining the processing strategy of abnormal data according to the ratio of abnormal data to normal data and the type of abnormal data comprises: The expert weight method is used in conjunction with the questionnaire to obtain the type score and type weight of each abnormal data; An expert weight method is used to conduct a questionnaire survey on the degree of harm of abnormal data, so as to determine the type score of each abnormal data and the type weight of the abnormal data. In the present invention, it can be known through the questionnaire that the type score of abnormal data caused by component failure is a1, the type score of abnormal data caused by abnormality in call is a2, the type score of abnormal data caused by leakage in transmission is a3, and the type score of abnormal data caused by equipment being attacked by hackers is a4; the type weight of abnormal data caused by component failure is w1, the type weight of abnormal data caused by abnormality in call is w2, the type weight of abnormal data caused by leakage in transmission is w3, and the type weight of abnormal data caused by equipment being attacked by hackers is w4.

[0038] The impact score of each type of abnormal data is obtained by multiplying the percentage of the number of abnormal data types by the type score of the abnormal data and the type weight of the abnormal data. The sum of the impact scores of all types of abnormal data is the abnormal score. The category of the abnormality is determined based on the abnormal score, and the processing strategy is determined based on the category of the abnormality.

[0039] By knowing the type of abnormal data to which each abnormal data belongs, you can know the number of abnormal data of each type. The sum of the number of abnormal data plus the number of normal data is the number of total data. The number of abnormal data of each type divided by the number of total data is the percentage of the abnormal data type. The percentage of abnormal data types caused by component failure is q1, the percentage of abnormal data types caused by exceptions in calls is q2, the percentage of abnormal data types caused by leaks in transmission is q3, and the percentage of abnormal data types caused by hacker attacks on the device is q4. The impact score of the abnormal data caused by component failure is a1*w1*q1, and the total abnormal score s=a1*w1*q1+a2*w2*q2+a3*w3*q3+a4*w4*q4. Find the processing strategy with an abnormal score close to the current abnormal score in the historical processing record, and the server then detects and adjusts the processing strategy to obtain the improved processing strategy. The improved processing strategy is the final processing strategy.

[0040] Embodiment 2: Figure 5 The structure diagram of the network abnormal data processing system based on big data is shown in FIG. 1 . In an embodiment of the present invention, a network abnormal data processing system based on big data includes: A preliminary determination module is used to obtain network data, including real-time network data and historical network data, calculate the difference change rate between the real-time network data and the network data of the previous moment in the historical data, the difference change rate is (real-time network data - network data of the previous moment) / network data of the previous moment, and determine normal data and preliminary abnormal data in the network data based on the difference change rate; A final determination module is used to perform feature extraction and feature vector calculation on the preliminary abnormal data to obtain an abnormal feature vector of each preliminary abnormal data, average all abnormal feature vectors to obtain a mean feature vector, calculate the Manhattan distance between each abnormal feature vector and the mean feature vector and compare it with twice the standard deviation, determine the ratio of abnormal data to normal data, the type of abnormal data and the source device of the abnormal data; A strategy determination module, used to determine a processing strategy for abnormal data based on the ratio of abnormal data to normal data and the type of abnormal data; The processing module is used to execute the processing strategy and analyze the reasons why the source device of the abnormal data generates the abnormal data, and repair the source device.

[0041] Figure 6 This is a structural block diagram of the composition of a preliminary determination module in a network abnormal data processing system based on big data, and the preliminary determination module includes: An acquisition unit, used for acquiring network data, the network data including real-time network data and historical network data; The first calculation unit is used to calculate the difference change rate, where the difference change rate = (real-time network data - network data at the last moment) / network data at the last moment; The preliminary determination unit is used to compare the difference change rate with a preset threshold value. When the difference change rate is greater than the preset threshold value, the real-time network data is preliminary abnormal data.

[0042] Figure 7 The present invention is a structural block diagram of the composition of the final determination module in the network abnormal data processing system based on big data, and the final determination module includes: A feature vector unit is used to perform feature extraction and feature vector calculation on the preliminary abnormal data to obtain an abnormal feature vector of each preliminary abnormal data, and to average all the abnormal feature vectors to obtain a mean feature vector; The second calculation unit is used to calculate the Manhattan distance between each abnormal feature vector and the mean feature vector, and compare the result with twice the standard deviation. When the result is greater than twice the standard deviation, the preliminary abnormal data corresponding to the abnormal feature vector is abnormal data; The final determination unit is used to analyze the types and causes of all abnormal data, divide the number of abnormal data by the number of normal data, and obtain the ratio of abnormal data to normal data, the type of abnormal data, and the source device of the abnormal data.

[0043] Figure 8 The present invention is a structural block diagram of a strategy determination module in a network abnormal data processing system based on big data, wherein the strategy determination module includes: A parameter determination unit, used to obtain the type score and type weight of each abnormal data by using an expert weight method in conjunction with a questionnaire; The strategy determination unit is used to multiply the quantity percentage of each abnormal data type by the type score of the abnormal data and the type weight of the abnormal data to obtain the impact score of the abnormal data. The sum of the impact scores of all types of abnormal data is the abnormal score. The category of the abnormality is determined based on the abnormal score, and the processing strategy is determined based on the category of the abnormality.

[0044] The functions that can be achieved by the method for processing abnormal network data based on big data are all completed by a computer device, which includes one or more processors and one or more memories, and at least one program code is stored in the one or more memories. The program code is loaded and executed by the one or more processors to achieve the functions of the user behavior prediction method based on big data.

[0045] The processor takes out instructions from the memory one by one, analyzes the instructions, and then completes the corresponding operations according to the instruction requirements, generating a series of control commands, so that the various parts of the computer can automatically, continuously and coordinately move to become an organic whole, realize the input of programs, the input of data, and the calculation and output of results. The arithmetic operations or logical operations generated in this process are all completed by the operator; the memory includes a read-only memory (ROM), which is used to store computer programs, and a protection device is provided outside the memory.

[0046] Exemplarily, the computer program may be divided into one or more modules, one or more modules are stored in a memory and executed by a processor to implement the present invention. One or more modules may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in a terminal device.

[0047] Those skilled in the art will understand that the description of the above service equipment is merely an example and does not constitute a limitation on the terminal equipment. It may include more or fewer components than described above, or a combination of certain components, or different components, for example, it may include input and output devices, network access devices, buses, etc.

[0048] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire user terminal.

[0049] The above-mentioned memory can be used to store computer programs and / or modules. The above-mentioned processor realizes various functions of the above-mentioned terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as an information collection template display function, a product information release function, etc.), etc.; the data storage area can store data created according to the use of the berth status display system (such as product information collection templates corresponding to different product types, product information that different product providers need to release, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0050] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the modules / units in the above-mentioned embodiment system, and can also be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can realize the functions of the above-mentioned various system embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Computer-readable media can include: any entity or device that can carry computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0051] It should be noted that, in this article, the term "comprises" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of more restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0052] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for processing network abnormal data based on big data, characterized in that: The method comprises: Acquire network data, including real-time network data and historical network data, calculate the difference change rate between the real-time network data and the network data of the previous moment in the historical data, and determine normal data and preliminary abnormal data in the network data based on the difference change rate; Perform feature extraction and feature vector calculation on the preliminary abnormal data to obtain the abnormal feature vector of each preliminary abnormal data, average all the abnormal feature vectors to obtain the mean feature vector, calculate the Manhattan distance between each abnormal feature vector and the mean feature vector and compare it with twice the standard deviation, determine the ratio of abnormal data to normal data, the type of abnormal data and the source device of the abnormal data; Determine the strategy for handling abnormal data based on the ratio of abnormal data to normal data and the type of abnormal data; Execute the processing strategy and analyze the reasons why the source device of the abnormal data generates the abnormal data, and repair the source device.

2. The method for processing network abnormal data based on big data according to claim 1 is characterized in that: The step of calculating the difference change rate between the real-time network data and the network data at the previous moment in the historical data, and determining the normal data and preliminary abnormal data in the network data according to the difference change rate comprises: Calculate the difference change rate, difference change rate = (real-time network data - network data of the previous moment) / network data of the previous moment; The difference change rate is compared with a preset threshold value. When the difference change rate is greater than the preset threshold value, the real-time network data is preliminary abnormal data.

3. The method for processing network abnormal data based on big data according to claim 1 or 2, characterized in that: The steps of calculating the Manhattan distance between each abnormal feature vector and the mean feature vector and comparing it with twice the standard deviation to determine the ratio of abnormal data to normal data, the type of abnormal data and the source device of the abnormal data include: Calculate the Manhattan distance between each abnormal feature vector and the mean feature vector, and compare the result with twice the standard deviation. When the result is greater than twice the standard deviation, the preliminary abnormal data corresponding to the abnormal feature vector is abnormal data; Analyze the types and causes of all abnormal data, divide the number of abnormal data by the number of normal data, and obtain the ratio of abnormal data to normal data, the type of abnormal data, and the source device of the abnormal data.

4. The method for processing network abnormal data based on big data according to claim 1 is characterized in that: The step of determining a processing strategy for abnormal data based on the ratio of abnormal data to normal data and the type of abnormal data comprises: The expert weight method is used in conjunction with the questionnaire to obtain the type score and type weight of each abnormal data; The impact score of each type of abnormal data is obtained by multiplying the percentage of the number of abnormal data types by the type score of the abnormal data and the type weight of the abnormal data. The sum of the impact scores of all types of abnormal data is the abnormal score. The category of the abnormality is determined based on the abnormal score, and the processing strategy is determined based on the category of the abnormality.

5. A network abnormal data processing system based on big data, characterized in that: The system comprises: A preliminary determination module is used to obtain network data, including real-time network data and historical network data, calculate the difference change rate between the real-time network data and the network data of the previous moment in the historical data, the difference change rate is (real-time network data - network data of the previous moment) / network data of the previous moment, and determine normal data and preliminary abnormal data in the network data based on the difference change rate; A final determination module is used to perform feature extraction and feature vector calculation on the preliminary abnormal data to obtain an abnormal feature vector of each preliminary abnormal data, average all abnormal feature vectors to obtain a mean feature vector, calculate the Manhattan distance between each abnormal feature vector and the mean feature vector and compare it with twice the standard deviation, determine the ratio of abnormal data to normal data, the type of abnormal data and the source device of the abnormal data; A strategy determination module, used to determine a processing strategy for abnormal data based on the ratio of abnormal data to normal data and the type of abnormal data; The processing module is used to execute the processing strategy and analyze the reasons why the source device of the abnormal data generates the abnormal data, and repair the source device.

6. The system for processing network abnormal data based on big data according to claim 5 is characterized in that: The preliminary determination module comprises: An acquisition unit, used for acquiring network data, the network data including real-time network data and historical network data; The first calculation unit is used to calculate the difference change rate, where the difference change rate = (real-time network data - network data at the last moment) / network data at the last moment; The preliminary determination unit is used to compare the difference change rate with a preset threshold value. When the difference change rate is greater than the preset threshold value, the real-time network data is preliminary abnormal data.

7. The network abnormal data processing system based on big data according to claim 5 or 6, characterized in that: The final determination module comprises: A feature vector unit is used to perform feature extraction and feature vector calculation on the preliminary abnormal data to obtain an abnormal feature vector of each preliminary abnormal data, and to average all abnormal feature vectors to obtain a mean feature vector; The second calculation unit is used to calculate the Manhattan distance between each abnormal feature vector and the mean feature vector, and compare the result with twice the standard deviation. When the result is greater than twice the standard deviation, the preliminary abnormal data corresponding to the abnormal feature vector is abnormal data; The final determination unit is used to analyze the types and causes of all abnormal data, divide the number of abnormal data by the number of normal data, and obtain the ratio of abnormal data to normal data, the type of abnormal data, and the source device of the abnormal data.

8. The network abnormal data processing system based on big data according to claim 5 is characterized in that: The strategy determination module includes: A parameter determination unit, used to obtain the type score and type weight of each abnormal data by using an expert weight method in conjunction with a questionnaire; The strategy determination unit is used to multiply the quantity percentage of each abnormal data type by the type score of the abnormal data and the type weight of the abnormal data to obtain the impact score of the abnormal data. The sum of the impact scores of all types of abnormal data is the abnormal score. The category of the abnormality is determined based on the abnormal score, and the processing strategy is determined based on the category of the abnormality.

Images