A large-flow network information monitoring method and system

Abstract

The application discloses a large-flow network information monitoring method and system, and relates to the technical field of network security.The application comprises the following steps: receiving a data packet; analyzing the data packet to obtain a sending position of the data packet; performing feature extraction on the data packet to obtain data features under multiple feature types; arranging the data features of the data packet under each feature type in a set order to obtain flow features of the data packet; continuously acquiring and storing the sending position and the flow features of the data packet; inputting the flow features of the data packet into an abnormal feature comparison library to compare and judge whether the flow packet is abnormal; if yes, the data packet is rejected to be executed or transmitted; if no, whether the data packet is abnormal is judged according to the sending position and the flow features of the data packet; if yes, the data packet is rejected to be executed or transmitted, and the flow features of the abnormal data packet are input into the abnormal feature comparison library; if no, the data packet is continuously executed or transmitted.The application performs safety monitoring on large-flow network information.

Description

Technical Field

[0001] This invention belongs to the field of network security technology, and in particular relates to a method and system for monitoring large-volume network information. Background Technology

[0002] With the rapid development and widespread adoption of internet technology, network traffic has experienced explosive growth. Against this backdrop, network information monitoring has become crucial for ensuring network security, optimizing network resource allocation, and improving network service quality. Network information monitoring methods and systems need to be able to handle massive amounts of network traffic and identify and process various network behaviors and events.

[0003] However, existing network information monitoring methods and systems typically face various challenges when processing large volumes of network data. First, the scale and complexity of network data make the efficiency and accuracy of data processing critical issues. Traditional data processing methods are often inefficient when handling large-scale network data and cannot respond to network events in real time. Furthermore, existing methods often fall short in detecting and handling complex network behaviors and events, such as network attacks and abnormal traffic. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for monitoring high-volume network information. By parsing and analyzing the characteristics of data packets, the security judgment speed of data packets is improved, enabling security monitoring of high-volume network information.

[0005] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0006] This invention provides a method for monitoring high-volume network information, including:

[0007] Receive data packets;

[0008] The sending location of the data packet is obtained by parsing the data packet;

[0009] Feature extraction is performed on the data packet to obtain data features under multiple feature types;

[0010] The data characteristics of the data packets under each characteristic type are arranged in a set order to obtain the traffic characteristics of the data packets;

[0011] Continuously acquire and store the sending location and traffic characteristics of the data packets;

[0012] The traffic characteristics of the data packet are input into the anomaly feature comparison database for comparison to determine whether the traffic packet is abnormal;

[0013] If so, then refuse to execute or transmit the data packet;

[0014] If not, then determine whether the data packet is abnormal based on the sending location and traffic characteristics of the data packet;

[0015] If so, then the execution or transmission of the data packet is refused, and the traffic characteristics of the abnormal data packet are entered into the abnormal characteristic comparison library;

[0016] If not, continue executing or transmitting the data packet.

[0017] This invention also discloses a method for monitoring high-volume network information, including:

[0018] Receive the data packets that are determined to be abnormal in the above-mentioned high-volume network information monitoring method;

[0019] The data instructions of the data packet are obtained by parsing;

[0020] Determine whether the data instruction is a data query instruction;

[0021] If so, then execute the data packet;

[0022] If not, then the packet will be rejected.

[0023] This invention also discloses a high-traffic network information monitoring system, comprising,

[0024] The data receiving unit is used to receive data packets;

[0025] A parsing unit is used to parse the data packet to obtain the sending location of the data packet;

[0026] Feature extraction is performed on the data packet to obtain data features under multiple feature types;

[0027] The data characteristics of the data packets under each characteristic type are arranged in a set order to obtain the traffic characteristics of the data packets;

[0028] Continuously acquire and store the sending location and traffic characteristics of the data packets;

[0029] The first judgment unit is used to input the traffic characteristics of the data packet into the abnormal feature comparison database for comparison and to determine whether the traffic packet is abnormal.

[0030] If so, then refuse to execute or transmit the data packet;

[0031] The second judgment unit is used to determine whether the data packet is abnormal based on the sending location and traffic characteristics of the data packet if the condition is not met.

[0032] If so, then the execution or transmission of the data packet is refused, and the traffic characteristics of the abnormal data packet are entered into the abnormal characteristic comparison library;

[0033] If not, continue executing or transmitting the data packet;

[0034] A supplementary judgment unit is used to receive the data packet that is judged to be abnormal;

[0035] The data instructions of the data packet are obtained by parsing;

[0036] Determine whether the data instruction is a data query instruction;

[0037] If so, then execute the data packet;

[0038] If not, then the packet will be rejected.

[0039] This invention improves the speed of security judgment for data packets by performing feature analysis, thereby enabling security monitoring of high-volume network information. The system mainly includes a data receiving unit, a parsing unit, and a judgment unit. The data receiving unit receives data packets, and the parsing unit extracts features from the data packets to obtain data characteristics and transmission locations. These data characteristics are arranged in a predetermined order to form traffic characteristics. The system continuously acquires and stores the transmission locations and traffic characteristics of data packets. The system then compares the traffic characteristics of the data packets with an anomaly characteristic comparison database to determine if the data packets are abnormal. If a data packet is abnormal, execution or transmission is refused, and the traffic characteristics of the abnormal data packet are also entered into the anomaly characteristic comparison database. A supplementary judgment unit is also provided to process data packets judged to be abnormal. If the data instruction is a query instruction, it will be executed to prevent instructions that do not affect system operation from being mistakenly discarded.

[0040] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the functional modules and information flow of a high-volume network information monitoring system according to an embodiment of the present invention;

[0043] Figure 2 This is a flowchart illustrating the steps of an embodiment of the high-volume network information monitoring method described in this invention.

[0044] Figure 3 This is a flowchart illustrating step S6 of the present invention in one embodiment;

[0045] Figure 4 This is a flowchart illustrating step S63 of the present invention in one embodiment;

[0046] Figure 5 This is a flowchart illustrating step S8 of the present invention in one embodiment;

[0047] Figure 6 This is a flowchart illustrating step S81 of the present invention in one embodiment.

[0048] Figure 7 This is a flowchart illustrating step S814 of the present invention in one embodiment.

[0049] Figure 8 This is a flowchart illustrating step S82 of the present invention in one embodiment;

[0050] Figure 9 This is a flowchart illustrating step S821 of the present invention in one embodiment;

[0051] The attached diagram lists the components represented by each number as follows:

[0052] 1-Data receiving unit, 2-parse unit, 3-first judgment unit, 4-second judgment unit, 5-supplementary judgment unit. Detailed Implementation

[0053] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0054] To enable rapid monitoring of high-volume data packets, this invention provides the following solution.

[0055] Please see Figures 1 to 2 As shown, this invention provides a high-traffic network information monitoring system, which, in terms of functional units, may include a data receiving unit 1, a parsing unit 2, a first judgment unit 3, a second judgment unit 4, and a supplementary judgment unit 5. In specific implementations, the data receiving unit 1 may be an interface for reading network data, and the parsing unit 2, the first judgment unit 3, the second judgment unit 4, and the supplementary judgment unit 5 may be computing functional units integrated on the server side.

[0056] In the specific implementation process, the data receiving unit 1 first executes step S1 to receive the data packet. Next, the parsing unit 2 executes step S2 to parse the data packet and obtain its transmission location. Next, step S3 is executed to extract features from the data packet, thereby obtaining data features under multiple feature types. Next, step S4 is executed to arrange the data features of the data packet under each feature type in a set order to obtain the data packet's traffic features. Next, step S5 is executed to continuously acquire and store the data packet's transmission location and traffic features.

[0057] To determine the security of data packets, the first judgment unit 3 first executes step S6 to input the traffic characteristics of the data packet into the anomaly characteristic comparison database for comparison to determine whether the traffic packet is abnormal. If so, step S7 can be executed to refuse to execute or transmit the data packet; otherwise, the second judgment unit 4 executes step S8 to determine whether the data packet is abnormal based on the data packet's sending location and traffic characteristics. If so, step S9 can be executed to refuse to execute or transmit the data packet and input the traffic characteristics of the abnormal data packet into the anomaly characteristic comparison database; otherwise, step S10 can be executed to continue executing or transmitting the data packet.

[0058] To avoid misjudgment, supplementary judgment unit 5 can continue to execute step S11 to receive data packets judged as abnormal. Next, step S12 can be executed to parse the data instruction of the data packet. Next, step S13 can be executed to determine whether the data instruction is a query data instruction. If so, step S14 can be executed to process the data packet; otherwise, step S15 can be executed to reject the data packet.

[0059] In the above steps, the data receiving unit receives data packets, and the parsing unit extracts the data packet characteristics and sending location to form traffic characteristics. The system continuously acquires and stores the sending location and traffic characteristics of data packets. Subsequently, the system compares the traffic characteristics of the data packets with an anomaly characteristic comparison database to determine whether they are abnormal. Abnormal data packets will be rejected for execution or transmission, and their traffic characteristics will be entered into the anomaly characteristic comparison database. In addition, there is a supplementary judgment unit that processes query instructions in data packets judged to be abnormal to prevent misjudgment and discarding instructions that have no impact on system operation. By performing feature judgment on data packets, the speed of data packet security judgment is improved.

[0060] To further illustrate the above steps, source code for some functional modules is provided, with explanations in the comments. Due to legal restrictions preventing the disclosure of original parameters and user data, data that does not affect the implementation of the solution has been anonymized; the same applies below.

[0061] public class PacketMonitor {

[0062] public void monitor() {

[0063] / / Receive data packets

[0064] Packet packet = receivePacket();

[0065] / / Parse the data packet to obtain the data packet's sending location.

[0066] String source = parseSource(packet);

[0067] / / Perform feature extraction on the data packet to obtain data features under multiple feature types.

[0068] Feature feature = extractFeature(packet);

[0069] / / Arrange the data characteristics of the data packets under each characteristic type in a set order to obtain the traffic characteristics of the data packets.

[0070] Feature flowFeature = arrangeFeature(feature);

[0071] while (true) { / / Continuously acquire and store the sending location and traffic characteristics of data packets.

[0072] / / Input the traffic characteristics of the data packets into the anomaly characteristic comparison database for comparison to determine whether the traffic packets are abnormal.

[0073] boolean isAnomaly = compareWithDatabase(flowFeature);

[0074] / / If so, refuse to execute or transmit data packets;

[0075] if (isAnomaly) {

[0076] rejectPacket(packet);

[0077] } else {

[0078] / / If not, determine whether the data packet is abnormal based on the data packet's sending location and traffic characteristics.

[0079] boolean isAnomaly2 = detectAnomaly(source,flowFeature);

[0080] / / If so, refuse to execute or transmit the data packet, and input the traffic characteristics of the abnormal data packet into the anomaly characteristic comparison database.

[0081] if (isAnomaly2) {

[0082] rejectPacket(packet);

[0083] addToDatabase(flowFeature);

[0084] } else {

[0085] / / If not, continue execution or transfer data packets.

[0086] executePacket(packet);

[0087] }

[0088] }

[0089] / / Receive data packets that are determined to be abnormal

[0090] Packet anomalyPacket = receiveAnomalyPacket();

[0091] / / Data instructions for parsing the data packet

[0092] String command = parseCommand(anomalyPacket);

[0093] / / Determine if the data command is a query data command

[0094] if (isQueryCommand(command)) {

[0095] / / If so, execute the data packet

[0096] executePacket(anomalyPacket);

[0097] } else {

[0098] / / If not, refuse to execute the data packet.

[0099] rejectPacket(anomalyPacket);

[0100] }

[0101] }

[0102] }

[0103] }

[0104] Please see Figure 3 As shown, to improve the speed of comparing the traffic features of data packets with the anomaly feature comparison database, in the specific implementation of step S6 above, firstly, step S61 can be executed to obtain the hit count of each traffic feature in the anomaly feature comparison database. Next, step S62 can be executed according to the hit count of each traffic feature. Next, step S63 can be executed to sort the traffic features in the anomaly feature comparison database according to the hit count of each traffic feature. Next, step S64 can be executed to search and compare the traffic features of the data packets one by one according to the sorting of traffic features in the anomaly feature comparison database, and determine whether the search and comparison are consistent. If yes, then step S65 can be executed to determine that the data packet is abnormal; otherwise, step S66 can be executed to determine that the data packet is normal.

[0105] To further illustrate the above steps, source code for some functional modules is provided, with explanations in the comments.

[0106] import java.util.*;

[0107] public class PacketMonitor {

[0108] public void monitor() {

[0109] / / Get the number of hits for each traffic feature in the anomaly feature comparison database.

[0110] Map<Feature, Integer> hitCounts = getHitCountsFromDatabase();

[0111] / / Sort the traffic features in the anomaly feature comparison database according to the number of times each traffic feature is matched.

[0112] List <feature>sortedFeatures = sortFeaturesByHitCounts(hitCounts);

[0113] / / Obtain the traffic characteristics of the data packets to be detected

[0114] Packet packet = receivePacket();

[0115] Feature flowFeature = extractFeature(packet);

[0116] / / Search and compare the traffic characteristics of data packets one by one according to the order of traffic characteristics in the anomaly feature comparison database, and determine whether the search and comparison are consistent.

[0117] for (Feature feature : sortedFeatures) {

[0118] if (compareFeature(flowFeature, feature)) { / / If so, then determine if the data packet is abnormal.

[0119] System.out.println("Packet is abnormal.");

[0120] break

[0121] } else { / / If not, then determine that the data packet is normal

[0122] System.out.println("Packet is normal.");

[0123] break

[0124] }

[0125] }

[0126] }

[0127] }

[0128] Please see Figure 4 As shown, to prioritize traffic features with higher retrieval probability in the anomaly feature comparison database, step S63 can be implemented as follows: First, step S631 divides traffic features with similarity in the anomaly feature comparison database into several feature groups based on the similarity of data features of each traffic feature across multiple feature types. Next, step S632 obtains the retrieval hit count of traffic features within each feature group. Next, step S633 sorts each feature group according to the retrieval hit count of traffic features within the feature group, from largest to smallest. Next, step S634 sorts each traffic feature within each feature group according to the retrieval hit count of each traffic feature, from largest to smallest. Finally, step S635 obtains the ranking result of traffic features in the anomaly feature comparison database based on the ranking results of each feature group and the ranking results of traffic features within each feature group.

[0129] To further illustrate the above steps, source code for some functional modules is provided, with explanations in the comments.

[0130] import java.util.*;

[0131] public class FeatureComparator {

[0132] / / ...This section requires defining the implementation of specific operations such as feature comparison library, feature retrieval, and sorting. Due to the involvement of domain-specific knowledge, it is not provided at this time...

[0133] public List <featuregroup>sortFeatures() {

[0134] / / Based on the similarity of data features of each traffic feature across multiple feature types, traffic features with similarity in the anomaly feature comparison database are divided into several feature groups.

[0135] List <featuregroup>featureGroups = divideFeaturesIntoGroups();

[0136] / / Get the number of search hits for traffic features within each feature group

[0137] for (FeatureGroup featureGroup : featureGroups){

[0138] featureGroup.calculateHitCounts();

[0139] }

[0140] / / Sort each feature group by the number of times the traffic features within the feature group are retrieved, from largest to smallest.

[0141] featureGroups.sort(Comparator.comparing(FeatureGroup::getTotalHitCount).reversed());

[0142] / / Within each feature group, sort each traffic feature in descending order of the number of search hits for each traffic feature.

[0143] for (FeatureGroup featureGroup : featureGroups){

[0144] featureGroup.sortFeaturesByHitCounts();

[0145] }

[0146] / / Based on the ranking results of each feature group and the ranking results of the traffic features within each feature group, the ranking results of the traffic features in the anomaly feature comparison database are obtained.

[0147] return featureGroups;

[0148] }

[0149] }

[0150] Please see Figure 5 As shown, for data packets that show normal results in the anomaly feature comparison database, the problem might be due to incomplete database entries. Therefore, security can also be assessed based on the data packet's traffic characteristics and sending location. Specifically, in the implementation of step S8, step S81 can first be executed to obtain a warning data pool composed of potentially dangerous data packets based on the traffic characteristics of multiple data packets. Finally, step S82 can be executed to determine whether the data packets in the warning data pool are normal based on their sending location.

[0151] To further illustrate the above steps, source code for some functional modules is provided, with explanations in the comments.

[0152] import java.util.*;

[0153] public class PacketAnalyzer {

[0154] public void analyze(Packet packet) {

[0155] / / Based on the traffic characteristics of multiple data packets, a warning data pool consisting of dangerous data packets is obtained.

[0156] List <packet>alertPool = getAlertPool(packet);

[0157] / / Determine if the data packets in the alert data pool are normal based on their sending location.

[0158] for (Packet p : alertPool) {

[0159] if (isNormal(p)) {

[0160] System.out.println("The packet isnormal.");

[0161] } else {

[0162] System.out.println("The packet isabnormal.");

[0163] }

[0164] }

[0165] }

[0166] }

[0167] Please see Figure 6 As shown, in order to select dangerous data packets from a large number of data packets, step S81 can be implemented by first selecting a data feature as a baseline data feature under each feature type in step S811. Next, step S812 can be executed to obtain the similarity between the data feature in the traffic features of each data packet and the baseline data feature under the corresponding feature type. Next, step S813 can be executed to arrange the similarity values ​​between the data feature in the traffic features and the baseline data feature under the corresponding feature type in a predetermined order to obtain the feature vector of the corresponding data packet. Next, step S814 can be executed to obtain the dangerous data packets based on the feature vectors of the data packets. Finally, step S815 can be executed to construct a warning data pool from the dangerous data packets.

[0168] To further illustrate the above steps, source code for some functional modules is provided, with explanations in the comments.

[0169] import java.util.*;

[0170] public class PacketAnalyzer {

[0171] / / Baseline data characteristics

[0172] private HashMap<String, Feature> benchmarkFeatures = new HashMap<>();

[0173] / / Alert Data Pool

[0174] private List <packet>alertPool = new ArrayList<>();

[0175] public void analyze(List <packet>packets) {

[0176] / / Select one data feature as the baseline data feature under each feature type.

[0177] initializeBenchmarkFeatures();

[0178] for (Packet packet : packets) {

[0179] / / Obtain the similarity between the data features in the traffic features of each data packet and the baseline data features under the corresponding feature type.

[0180] Vector similarityVector = calculateSimilarity(packet);

[0181] / / Identify dangerous data packets based on their feature vectors

[0182] if (isDangerous(similarityVector)) {

[0183] / / Create a warning data pool from dangerous data packets.

[0184] alertPool.add(packet);

[0185] }

[0186] }

[0187] }

[0188] }

[0189] Please see Figure 7 As shown, regular user traffic exhibits a dispersed characteristic. Therefore, potentially dangerous data packets show a high degree of concentration in their data characteristics. Thus, in the specific implementation of step S814, the first step is to select several baseline feature vectors from the feature vectors of the data packets. Next, step S8142 is to obtain the vector difference magnitude between the baseline feature vectors and other feature vectors. Next, step S8143 is to group each other feature vector with the baseline feature vector having the lowest vector difference magnitude into the same feature vector group. Next, step S8144 is to obtain the mean vector of each feature vector group as the updated baseline feature vector. Next, step S8145 is to update the feature vector group based on the updated baseline feature vector. Next, step S8146 is to determine whether the number of feature vectors in the updated feature vector group has changed. If yes, then step S8147 is to continuously update the baseline feature vector and the feature vector group; otherwise, step S8148 is to finally identify the data packets corresponding to the feature vectors in the feature vector group as potentially dangerous data packets.

[0190] To further illustrate the above steps, source code for some functional modules is provided, with explanations in the comments.

[0191] import java.util.*;

[0192] public class PacketAnalyzer {

[0193] / / Data packet feature vector

[0194] private HashMap<Packet, Vector> packetFeatureVectors = new HashMap<>();

[0195] public List <packet>identifyDangerousPackets() {

[0196] / / Select several reference feature vectors from the feature vectors of the data packet

[0197] List <vector>benchmarkVectors = initializeBenchmarkVectors();

[0198] boolean groupUpdated;

[0199] do {

[0200] groupUpdated = false;

[0201] / / Baseline feature vector and corresponding feature vector set

[0202] HashMap <Vector, List <vector>>featureVectorGroups = newHashMap<>();

[0203] for (Vector vector : packetFeatureVectors.values()){

[0204] / / Obtain the magnitude of the vector difference between the baseline feature vector and other feature vectors

[0205] Vector benchmarkVector = findClosestBenchmarkVector(vector,benchmarkVectors);

[0206] / / Group each other eigenvector into the same eigenvector group as the baseline eigenvector with the lowest magnitude of the difference between the eigenvectors.

[0207] featureVectorGroups.get(benchmarkVector).add(vector);

[0208] }

[0209] / / Update baseline feature vector

[0210] for (Vector benchmarkVector : benchmarkVectors){

[0211] / / Obtain the mean vector of each feature vector group as the updated baseline feature vector.

[0212] Vector updatedBenchmarkVector = calculateMeanVector(featureVectorGroups.get(benchmarkVector));

[0213] / / Determine if the number of feature vectors in the updated feature vector group has changed.

[0214] if (!updatedBenchmarkVector.equals(benchmarkVector)){

[0215] groupUpdated = true;

[0216] benchmarkVectors.remove(benchmarkVector);

[0217] benchmarkVectors.add(updatedBenchmarkVector);

[0218] }

[0219] }

[0220] / / If the feature vector set changes, continuously update the baseline feature vector and feature vector set.

[0221] } while (groupUpdated);

[0222] / / If the feature vector group remains unchanged, then the data packets corresponding to the feature vectors within the feature vector group are considered dangerous data packets.

[0223] List <packet>dangerousPackets = getDangerousPackets(benchmarkVectors);

[0224] return dangerous Packets;

[0225] }

[0226] }

[0227] Please see Figure 8 As shown, if data packets within the same alert data pool have concentrated sending locations, it indicates they are likely attack data packets sent by botnets controlled by hackers. To detect these data packets, step S82 can first be implemented by executing step S821 to determine if there are any abnormal sending locations. If so, step S822 can be executed to determine if data packets sent from abnormal locations are abnormal, while other data packets in the alert data pool are normal. Otherwise, step S823 can be executed to determine if all data packets in the alert data pool are normal.

[0228] To further illustrate the above steps, source code for some functional modules is provided, with explanations in the comments.

[0229] import java.util.*;

[0230] public class PacketAnalyzer {

[0231] / / Data packets stored in the vigilance data pool

[0232] private List <packet>alertPool = new ArrayList<>();

[0233] public void checkAlertPoolPackets() {

[0234] boolean hasAbnormalLocation = false;

[0235] / / Traverse the data packets in the alert data pool

[0236] for(Packet packet : alertPool) {

[0237] / / Determine if there is an abnormal location based on the sending position of the data packet.

[0238] if(isLocationAbnormal(packet.getLocation())) {

[0239] hasAbnormalLocation = true;

[0240] System.out.println("A packet was found at an abnormal location. Packet ID: " + packet.getId());

[0241] / / This section handles abnormal data packets.

[0242] }

[0243] }

[0244] / / If an abnormal location is detected, the data packet sent from that location is considered abnormal, while other data packets in the alert data pool are considered normal.

[0245] if(hasAbnormalLocation) {

[0246] System.out.println("A data packet sent from an abnormal location was found in the alert data pool.");

[0247] } else {

[0248] / / If no abnormal location is found, then all data packets in the alert data pool are considered normal.

[0249] System.out.println("The sending positions of all data packets in the alert data pool are normal.");

[0250] }

[0251] }

[0252] / / Determine if the sending location is abnormal. The specific implementation may vary depending on the actual definition of the abnormal location.

[0253] private boolean isLocationAbnormal(String location) {

[0254] / / Example; the actual decision logic will be defined according to your needs.

[0255] return "Abnormal location".equals(location);

[0256] }

[0257] }

[0258] Please see Figure 9 As shown, to locate the compromised computer (the "zombie computer"), i.e., the abnormal location, step S821 can be implemented as follows: First, step S8211 selects several reference locations from the data packet transmission locations. Next, step S8212 obtains the distance between the reference locations and other transmission locations. Next, step S8213 groups the other transmission locations with the nearest reference location into the same location group. Next, step S8214 obtains the transmission location within each location group closest to the geometric center as the updated reference location. Next, step S8215 determines whether the updated reference location has changed. If so, step S8216 updates the location group based on the updated reference location and continues to update the reference location. Otherwise, step S8217 obtains the location group. Finally, step S8218 identifies the transmission locations within the location group as abnormal locations.

[0259] To further illustrate the above steps, source code for some functional modules is provided, with explanations in the comments.

[0260] import java.util.*;

[0261] public class LocationAnalyzer {

[0262] / / Store the location where the data packet was sent

[0263] private List <location>packetLocations = new ArrayList<>();

[0264] public void analyzeLocations() {

[0265] / / Select several reference locations

[0266] List <location>baseLocations = selectBaseLocations();

[0267] while (true) {

[0268] Map<Location, List <location>>locationGroups = new HashMap<>();

[0269] / / Group other sending locations into the same location group as the nearest reference location.

[0270] for (Location location : packetLocations){

[0271] Location closestBaseLocation =findClosestBaseLocation(location,baseLocations);

[0272] locationGroups.computeIfAbsent(closestBaseLocation, k->new ArrayList<>()).add(location);

[0273] }

[0274] / / Obtain the transmission position closest to the geometric center within each location group as the updated reference position.

[0275] List <location>newBaseLocations = new ArrayList<>();

[0276] for (List <location>group : locationGroups.values()) {

[0277] newBaseLocations.add(findGroupCenter(group));

[0278] }

[0279] / / Determine if the updated baseline position has changed

[0280] if (baseLocations.equals(newBaseLocations)) {

[0281] / / If there is no change, exit the loop.

[0282] break

[0283] }

[0284] / / If there are any changes, update the position group according to the updated reference position, and continue to update the reference position.

[0285] baseLocations = newBaseLocations;

[0286] }

[0287] / / Identify the sending location within the location group as an abnormal location.

[0288] }

[0289] private List <location>selectBaseLocations() {

[0290] / / Example: The actual method for selecting the reference location may differ.

[0291] return new ArrayList<>(packetLocations.subList(0, Math.min(3,packetLocations.size())));

[0292] }

[0293] private Location findClosestBaseLocation(Location location, List <location>baseLocations) {

[0294] Location closest = null;

[0295] double closestDistance = Double.MAX_VALUE;

[0296] for (Location baseLocation : baseLocations) {

[0297] double distance = location.distanceTo(baseLocation);

[0298] if (distance<closestDistance) {

[0299] closestDistance = distance;

[0300] closest = baseLocation;

[0301] }

[0302] }

[0303] return closest;

[0304] }

[0305] private Location findGroupCenter(List <location>group) {

[0306] This method needs to determine how to find the geometric center of a set of locations, which involves calculating the average or finding the location closest to all other locations.

[0307] / /

[0308] return null;

[0309] }

[0310] }

[0311] In summary, this solution improves the efficiency of security monitoring for high-volume network information by performing feature analysis on data packets. The data receiving unit in this solution collects data packets, and the parsing unit extracts features and obtains their transmission locations. These features are organized in a preset order to form traffic characteristics. The system continuously collects and saves the transmission locations and traffic characteristics of data packets. Then, the system compares the traffic characteristics of the data packets with an anomaly feature database to assess whether the data packets are abnormal. If an anomaly is determined, the system will refuse to execute or transmit the data packet and simultaneously record its traffic characteristics in the anomaly feature database. Furthermore, a supplementary judgment unit is used to process data packets determined to be abnormal. If the data instruction is a query type, it will be executed to avoid misjudgments that could lead to the unwarranted rejection of instructions that do not affect system operation.

[0312] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0313] It should also be noted that each block in the block diagram and / or flowchart, as well as combinations of blocks in the block diagram and / or flowchart, can be implemented using hardware that performs the corresponding function or action, such as circuits or ASICs (Application Specific Integrated Circuits), or using a combination of hardware and software, such as firmware.

[0314] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings, disclosure, and appended claims in carrying out the claimed invention. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0315] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.< / location> < / location> < / location> < / location> < / location> < / location> < / location> < / location> < / packet> < / packet> < / vector> < / vector> < / packet> < / packet> < / packet> < / packet> < / featuregroup> < / featuregroup> < / feature>

Claims

1. A large flow network information monitoring method, characterized in that, comprising, receiving a data packet; parsing the data packet to obtain a sending location of the data packet; extracting features of the data packet to obtain data features under multiple feature types; arranging the data features of the data packet under each feature type in a set order to obtain a traffic feature of the data packet; continuously obtaining and storing the sending location and the traffic feature of the data packet; inputting the traffic feature of the data packet into an abnormal feature comparison library to determine whether the data packet is abnormal; if yes, rejecting execution or transmission of the data packet; if no, selecting one data feature under each feature type as a reference data feature; obtaining a similarity between the data feature in the traffic feature of each data packet and the reference data feature under the corresponding feature type; arranging the values of the similarity between the data feature in the traffic feature and the reference data feature under the corresponding feature type in a set order to obtain a feature vector corresponding to the data packet; selecting several reference feature vectors from the feature vector of the data packet; obtaining a vector difference modulus length between the reference feature vector and other feature vectors; grouping each other feature vector with the reference feature vector having the minimum vector difference modulus length into a same feature vector group; obtaining a mean vector of each feature vector group as an updated reference feature vector; updating the feature vector group according to the updated reference feature vector; determining whether the number of feature vectors in the updated feature vector group changes; if yes, continuously updating the reference feature vector and the feature vector group; if no, regarding the data packet corresponding to the feature vector in the feature vector group as a dangerous data packet; constructing a warning data pool from the dangerous data packets; determining whether the data packets in the warning data pool are normal according to the sending locations of the data packets in the warning data pool; if yes, rejecting execution or transmission of the data packet and inputting the traffic feature of the abnormal data packet into the abnormal feature comparison library; if no, continuing to execute or transmit the data packet.

2. The method of claim 1, wherein, The step of inputting the traffic feature of the data packet into the abnormal feature comparison library to determine whether the data packet is abnormal, comprising, obtaining a hit number of each traffic feature in the abnormal feature comparison library; arranging the traffic features in the abnormal feature comparison library according to the hit number of each traffic feature; searching and comparing the traffic feature of the data packet according to the arrangement of the traffic features in the abnormal feature comparison library to determine whether the searching and comparing is consistent; if yes, determining that the data packet is abnormal; if no, determining that the data packet is normal. The step of arranging the traffic features in the abnormal feature comparison library according to the hit number of each traffic feature, 3. The method of claim 2, wherein, according to the similarity of the data features of each traffic feature under multiple feature types, grouping the traffic features having the similarity in the abnormal feature comparison library into several feature groups; ​ Obtaining the number of search hits of each traffic feature in each feature group; Sorting each feature group according to the number of search hits of the traffic features in the feature group from large to small; In each feature group, sorting each traffic feature according to the number of search hits of each traffic feature from large to small; According to the sorting results of each feature group and the sorting results of the traffic features in each feature group, obtaining the sorting results of the traffic features in the abnormal feature comparison library.

4. The method of claim 1, wherein, The step of determining whether the data packets in the warning data pool are normal according to the sending positions of the data packets, comprises, For the same warning data pool, Determining whether there is an abnormal position according to the sending position of the data packet; If yes, determining that the data packet sent by the abnormal position is abnormal, and other data packets in the warning data pool are normal; If no, determining that all data packets in the warning data pool are normal.

5. The method of claim 4, wherein, The step of determining whether there is an abnormal position according to the sending position of the data packet, Comprises, Selecting several reference positions in the sending positions of the data packets; Obtaining the distance between the reference positions and other sending positions; Grouping other sending positions with the reference position closest to them into the same position group; Obtaining the sending position closest to the geometric center in each position group as the updated reference position; Determining whether the updated reference position has changed; If yes, updating the position group according to the updated reference position, and continuously updating the reference position; If no, obtaining the position group; Determining the sending position in the position group as the abnormal position.

6. A large flow network information monitoring method characterized by, Comprises, Receiving the data packet determined as abnormal in the large-flow network information monitoring method of any one of claims 1 to 5; Analyzing to obtain the data instruction of the data packet; Determining whether the data instruction is a query data instruction; If yes, executing the data packet; If no, refusing to execute the data packet.

7. A large flow network information monitoring system characterized by, Comprises, A data receiving unit for receiving data packets; An analysis unit for analyzing the data packets to obtain the sending positions of the data packets; Extracting features of the data packets to obtain data features under multiple feature types; Arranging the data features of the data packets under each feature type in a set order to obtain traffic features of the data packets; Continuously obtaining and storing the sending positions and traffic features of the data packets; A first determination unit for inputting the traffic features of the data packets into an abnormal feature comparison library for comparison to determine whether the data packets are abnormal; If yes, refusing to execute or transmit the data packets; A second determination unit for selecting one data feature under each feature type as a reference data feature if no; Obtaining the similarity between the data features in the traffic features of each data packet and the reference data feature under the corresponding feature type; arranging the data features in the flow feature and the numerical values of the similarity of the reference data features corresponding to the feature types in a set order to obtain a feature vector corresponding to the data packet; selecting several reference feature vectors from the feature vector of the data packet; obtaining the vector difference module length of the reference feature vectors and other feature vectors; grouping each other feature vector and the reference feature vector with the minimum vector difference module length into the same feature vector group; obtaining the mean vector of each feature vector group as an updated reference feature vector; updating the feature vector group according to the updated reference feature vector; judging whether the number of feature vectors in the updated feature vector group changes; if yes, continuously updating the reference feature vector and the feature vector group; if no, regarding the data packet corresponding to the feature vector in the feature vector group as a dangerous data packet; constructing the dangerous data packet into a warning data pool; judging whether the data packet in the warning data pool is normal according to the sending position of the data packet in the warning data pool; if yes, rejecting to execute or transmit the data packet, and inputting the flow feature of the abnormal data packet into the abnormal feature comparison library; if no, continuing to execute or transmit the data packet; a supplementary judgment unit for receiving the data packet judged as abnormal; analyzing the data instruction of the data packet; judging whether the data instruction is a query data instruction; if yes, executing the data packet; if no, rejecting to execute the data packet.

Images