Data processing method, data flow correlation analysis method, device and apparatus

Abstract

Embodiments of the present application provide a data processing method, a data stream correlation analysis method, a device and equipment. The data processing method comprises: obtaining a first data stream, setting a first identification field for the first data stream; according to a data type of the first data stream, assigning a value to the first identification field to obtain a field value of the first data stream; the field value of the first identification field is used to indicate a type of a data table storing the first data stream; the type of the data table storing the first data stream corresponds to the data type of the first data stream; and storing the first data stream in the data table corresponding to the data type. In the above scheme, the type of the data table storing the data stream is determined based on the data type of the data stream, thereby improving the efficiency of storing the data stream.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing method, a data flow correlation analysis method, an apparatus, and a device. Background Technology

[0002] With the development of technology and the spread of knowledge, there is a need to store and process massive amounts of data, such as storing and analyzing large amounts of network behavior data to reduce security risks. For example, data is typically stored in databases in the form of tables for easy querying and use. Therefore, how to improve the efficiency of data storage for large amounts of data is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] This invention provides a data processing method, a data flow correlation analysis method, an apparatus, and a device to improve data storage efficiency.

[0004] Specifically, the embodiments of the present invention provide the following technical solutions:

[0005] In a first aspect, embodiments of the present invention provide a data processing method, including:

[0006] Obtain the first data stream and set the first identifier field for the first data stream;

[0007] Based on the data type of the first data stream, the first identifier field is assigned a value to obtain the field value of the first data stream; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream; the type of the data table storing the first data stream corresponds to the data type of the first data stream.

[0008] The first data stream is stored in the data table corresponding to the data type.

[0009] Secondly, embodiments of the present invention provide a data flow correlation analysis method, including:

[0010] Obtain the first data stream and the first identifier field in the first data stream;

[0011] When the type of the data table storing the first data stream is determined to be a first type based on the field value of the first identifier field, the first data stream is associated with at least one second data stream in the data table of the second type to obtain at least one first association result; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream.

[0012] Based on the first association result, an association analysis is performed.

[0013] Thirdly, embodiments of the present invention provide a data processing apparatus, including:

[0014] The acquisition module is used to acquire the first data stream and set a first identifier field for the first data stream;

[0015] The processing module is used to assign a value to the first identifier field according to the data type of the first data stream to obtain the field value of the first data stream; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream; the type of the data table storing the first data stream corresponds to the data type of the first data stream.

[0016] A storage module is used to store the first data stream in a data table corresponding to the data type.

[0017] Fourthly, embodiments of the present invention provide a data flow correlation analysis apparatus, comprising:

[0018] The acquisition module is used to acquire the first data stream and the first identifier field in the first data stream;

[0019] The processing module is configured to, when determining that the type of the data table storing the first data stream is a first type based on the field value of the first identifier field, perform association calculations between the first data stream and at least one second data stream in a data table of a second type to obtain at least one first association result; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream;

[0020] The processing module is also used to perform association analysis based on the first association result.

[0021] Fifthly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the data processing method as described in the first aspect or the data flow correlation analysis method as described in the second aspect.

[0022] In a sixth aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the data processing method as described in the first aspect or the data flow correlation analysis method as described in the second aspect.

[0023] In a seventh aspect, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the steps of the data processing method as described in the first aspect or the data flow correlation analysis method as described in the second aspect.

[0024] The data processing method, data stream correlation analysis method, apparatus, and device provided in this invention embodiment acquire a first data stream and set a first identifier field for the first data stream. Since the type of the data table storing the first data stream corresponds to the data type of the first data stream, the first identifier field is assigned a value according to the data type of the first data stream to obtain the field value of the first data stream. The field value of the first identifier field is used to indicate the type of the data table storing the first data stream. Furthermore, the first data stream is stored in the data table corresponding to the data type. In the above scheme, determining the type of the data table storing the data stream based on the data type of the data stream can improve the efficiency of storing the data stream. Moreover, when processing the data stream subsequently, the corresponding data table can be quickly queried through the field value of the first identifier field, thereby improving the data processing efficiency. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0026] Figure 1 This is one of the flowcharts illustrating the data processing method provided in this embodiment of the invention;

[0027] Figure 2 This is a second schematic flowchart of the data processing method provided in the embodiments of the present invention;

[0028] Figure 3 This is the third flowchart illustrating the data processing method provided in this embodiment of the invention;

[0029] Figure 4 This is the fourth flowchart of the data processing method provided in the embodiments of the present invention;

[0030] Figure 5 This is the fifth flowchart illustrating the data processing method provided in this embodiment of the invention;

[0031] Figure 6 This is one of the flowcharts illustrating the data flow correlation analysis method provided in this embodiment of the invention;

[0032] Figure 7 This is a second flowchart illustrating the data flow correlation analysis method provided in this embodiment of the invention;

[0033] Figure 8 This is the third flowchart illustrating the data flow correlation analysis method provided in this embodiment of the invention;

[0034] Figure 9This is the fourth flowchart of the data flow correlation analysis method provided in this embodiment of the invention;

[0035] Figure 10 This is the fifth flowchart illustrating the data flow correlation analysis method provided in this embodiment of the invention;

[0036] Figure 11 This is a schematic diagram of the data processing device provided by the present invention;

[0037] Figure 12 This is a schematic diagram of the data flow correlation analysis device provided by the present invention;

[0038] Figure 13 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0040] The following is combined Figures 1-13 The technical solutions of the embodiments of the present invention will be described in detail with reference to specific examples. The following specific examples can be combined with each other, and the same or similar concepts or processes may not be described again in some examples.

[0041] Figure 1 This is one of the flowcharts illustrating the data processing method provided in this embodiment of the invention. For example... Figure 1 As shown, the method provided in this embodiment includes:

[0042] Step 101: Obtain the first data stream and set the first identifier field for the first data stream;

[0043] Specifically, the first data stream is represented by the data structure Data, which stores the data stream data corresponding to this data. The first data stream Data includes one or more data, for example, each data is represented by RowData. For the convenience of data storage and use, a first identifier field can be set for the first data stream. The first identifier field can be represented by isLeft. Subsequently, by identifying the first identifier field in the data stream, the type of data table storing the first data stream can be determined.

[0044] Optionally, the first data stream may be network behavior data, system log data, etc.

[0045] Step 102: Assign a value to the first identifier field according to the data type of the first data stream to obtain the field value of the first data stream; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream; the type of the data table storing the first data stream corresponds to the data type of the first data stream.

[0046] Specifically, the data types of the first data stream can be classified based on the data source, such as different source servers; or based on security risks, such as classification based on the degree of security risk, such as data types with security risk, no security risk, high security risk, low security risk, etc.

[0047] Optionally, the first data stream may include a second identification field, which can be used to indicate the data type of the first data stream. Therefore, the data type of the first data stream can be determined based on the second identification field in the first data stream.

[0048] To improve the efficiency of subsequent data processing, the acquired data streams can be categorized and stored in data tables corresponding to the data types of the data streams. That is, they can be categorized and stored in different types of data tables. First, the first identifier field of the first data stream can be assigned a value. Different field values ​​correspond to different data table types. For example, if the first data stream is data categorized into data table A, then the field value of the first identifier field isLeft is field value a (such as true). If the first data stream is data categorized into data table B, then the field value of isLeft is field value b (such as false).

[0049] Step 103: Store the first data stream in the data table corresponding to the data type.

[0050] Specifically, after assigning a value to the first identifier field, since the field value of the first identifier field is used to indicate the type of the data table storing the first data stream, the first data stream is stored in the data table corresponding to the data type, that is, in the data table corresponding to the type indicated by the field value.

[0051] The method in this embodiment involves acquiring a first data stream and setting a first identifier field for the first data stream. Since the type of the data table storing the first data stream corresponds to the data type of the first data stream, the first identifier field is assigned a value according to the data type of the first data stream to obtain the field value of the first data stream. The field value of the first identifier field is used to indicate the type of the data table storing the first data stream. Furthermore, the first data stream is stored in the data table corresponding to the data type. In the above scheme, determining the type of the data table storing the data stream based on the data type of the data stream can improve the efficiency of storing the data stream. Moreover, when processing the data stream subsequently, the corresponding data table can be quickly queried through the field value of the first identifier field, thereby improving data processing efficiency.

[0052] Optionally, step 102 can be implemented in the following way:

[0053] When the data type of the first data stream is determined to be a first data type, the first identifier field is assigned a first field value; the first field value is used to indicate that the data table storing the first data stream is of the first type; or,

[0054] When it is determined that the data type of the first data stream is the second data type, the first identifier field is assigned the value of the second field; the value of the second field is used to indicate that the data table storing the first data stream is of the second type;

[0055] Specifically, the first data type is different from the second data type. The first data type corresponds to a data table of the first type, that is, it corresponds to the value of the first field; the second data type corresponds to a data table of the second type, that is, it corresponds to the value of the second field. For example, if the first data type is an abnormal data type, the data table of the first type is used to store the data stream of this abnormal data type; if the second data type is a data type without security risks, the data table of the second type is used to store the data stream of this data type without security risks.

[0056] Optionally, after storing the first data stream in the first data table, a first index corresponding to the first data stream can be set in the first data table.

[0057] Specifically, a first index is set in the first data table corresponding to the first data stream. In subsequent processes such as querying or deleting the first data stream, the first data stream can be quickly retrieved from the first data table based on the first index, which can improve data processing efficiency.

[0058] For example, if the first data stream is stored in the 10th row of the first data table, then the first index can be 10.

[0059] Optionally, the first data stream is stored in a second data table corresponding to the second type, and a second index corresponding to the first data stream is set in the second data table.

[0060] Specifically, a second index is set in the second data table corresponding to the first data stream. In subsequent processes such as querying or deleting the first data stream, the first data stream can be quickly retrieved from the second data table based on the second index, which can improve data processing efficiency.

[0061] In practical applications, the first data stream can also be divided into multiple rows of data storage. For example, the first index and the second index can be the starting index and / or ending index of the first data stream.

[0062] For example, set the first index to index, and set the row data storage index to rowData.rowDataIndex = index.

[0063] In the above implementation, by setting an index for the data stream, it is convenient to query when using the data, which can improve data processing efficiency.

[0064] Optionally, storing the first data stream in a first data table corresponding to the first type includes:

[0065] Determine the first key value of the first data stream, and store the first data stream in the first data table based on the first key value.

[0066] Specifically, the first key value may be, for example, the source IP address, destination IP address, source server name, destination server name, etc. included in the data stream.

[0067] Use a first key value generator (such as leftKeyBuilder) to calculate the first key value (e.g., represented by keyValue), set rowData.keyValue = keyValue, and save the row data rowData of the first data stream to the first data table (e.g., the first data table includes the key value table for storing the left row data) based on the key value keyValue.

[0068] For example, if the first data stream contains multiple key values, one of them can be used as the first key value. Based on actual needs, if it is necessary to analyze the destination IP address of the data stream to determine whether multiple data streams have the same destination IP address, then the destination IP address can be used as the first key value.

[0069] Optionally, the first data stream is stored in a second data table corresponding to the second type, including:

[0070] Determine the second key value of the first data stream, and store the first data stream in the second data table based on the second key value.

[0071] Specifically, the leftKeyBuilder is used to calculate the second key value, and rowData.keyValue is set to keyValue. Based on the second key value, rowData is saved to the second data table (e.g., the second data table includes the right row data storage key value table).

[0072] Optionally, different types of data tables may have different key values.

[0073] In the above implementation, the key value of the data stream is determined and stored based on the key value, which facilitates subsequent data querying and data processing.

[0074] Optionally, a listener is set for the first data stream, and the information of the first data stream for which the listener is set is added to the listener list; the listener is used to instruct the deletion of the first data stream when a timeout notification of the first data stream is heard.

[0075] Specifically, create a new row data named rowData with the data type RowData, set rowData.data = data, set a row data listener, set rowData.isLeft = isLeft, and add the information of the first data stream (which may include the information of the row data rowData) to the listener list data.listenerList, such as adding the identifier and storage location of the first data stream to the listener list.

[0076] In the above implementation, by setting a listener, the deletion of data streams can be monitored, thereby improving the real-time performance of data processing.

[0077] For example, the parameters involved in the foregoing embodiments are shown in Tables 1 and 2 below:

[0078] Table 1

[0079]

[0080] Table 2

[0081]

[0082]

[0083] Optionally, the method further includes:

[0084] Retrieve the hit data pairs, store them in the hit data table, and set a third index for each hit data pair; the hit data pair is obtained by performing a correlation calculation on the first data stream and the second data stream, and when the correlation result is successful; the second data stream and the first data stream come from different data tables;

[0085] Store the third index of the hit data pair in the hit data table in the first data table and the second data table respectively;

[0086] Set a fourth index in the hit data table to associate the hit data pairs. The fourth index is the index of the third index in the first data table.

[0087] Set a fifth index in the hit data table to associate the hit data pairs. The fifth index is the index of the third index in the second data table.

[0088] Specifically, after performing correlation analysis on the first and second data streams, if the correlation is successful, a matching data pair is obtained. The matching data pair is stored in the matching data table (such as matchedDataTable), and a third index is set for the matching data pair. For example, if the matching data pair is stored in the 15th row of the matching data table, then the third index can be 15.

[0089] Furthermore, to facilitate the querying of the hit data pairs associated with the data streams, the third index of the hit data pairs can be stored in the first data table and the second data table respectively. In the first data table, the third index is associated with the data of the first data stream included in the hit data pair. For example, the third index can be stored in the row where the first data stream is located in the first data table. Furthermore, a fourth index can be set for the third index in the first data table. For example, if the third index is stored in the 10th row of the first data table, then the fourth index is 10. In the hit data table, the hit data table is associated with the fourth index. For example, the fourth index can be stored in the row where the hit data pair is located in the hit data table.

[0090] In the second data table, the third index is associated with the data of the second data stream included in the hit data pair. For example, the third index is stored in the row of the first data stream in the first data table. Furthermore, the third index can be set as a fifth index in the second data table. For example, if the third index is stored in the 12th row of the second data table, then the fifth index is 12. In the hit data table, the hit data table is associated with the fifth index. For example, the fifth index is stored in the row of the hit data pair in the hit data table.

[0091] For example, a matched data pair is represented by the data structure matchedData, and matchedData.leftRowData = leftRowData (leftRowData represents the first data stream) is set, and matchedData.rightRowData = rightRowData (rightRowData represents the second data stream);

[0092] Add the third index, index1, to the first data table. For example, you can set up a sub-table in the first data table to store the third index, such as using leftRowData.matchedDataIndexList. Set up the fourth index, leftIndex, in the first data table to store the third index. Add index1 to the second data table. For example, you can set up a sub-table in the first data table to store the third index, such as using rightRowData.matchedDataIndexList. Set up the fifth index, rightIndex, in the first data table to store the third index.

[0093] In the above embodiments, by storing the successfully associated data streams, setting a third index for the associated hit data pairs, and associating the hit data pairs with the data streams included in the hit data pairs through the third index and the index of the third index in the data tables of different data streams, the efficiency of subsequent data query and processing can be improved.

[0094] For example, the parameters involved in the above embodiments are shown in Table 3 below:

[0095] Table 3

[0096]

[0097]

[0098] Optionally, the method further includes:

[0099] When the first data stream is a data stream to be deleted, and the data type of the first data stream is the first data type, query the third index corresponding to the first data stream in the first data table;

[0100] Retrieve the matching data pair corresponding to the third index from the matching data table, and delete the matching data pair.

[0101] Query the fifth index in the hit data table, and delete the third index corresponding to the fifth index in the second data table;

[0102] Based on the first index, delete the first data stream and related data in the first data table.

[0103] Specifically, for the deletion process of the first data stream, if the first data stream is stored in the first data table, the third index corresponding to the first data stream is queried in the first data table; the third index is the index of the hit data pair including the first data stream in the hit data table; then, the hit data pair corresponding to the third index is obtained in the hit data table and deleted; the fifth index is queried in the hit data table; where the fifth index is the index of the third index in the second data table, the third index corresponding to the fifth index in the second data table is deleted, that is, the corresponding third index is queried in the second data table based on the fifth index and the third index is deleted; furthermore, the first data stream and related data, such as the third index, are queried in the first data table based on the first index and deleted. The third index can be found based on the fourth index in the hit data table and deleted. The fourth index is the index of the third index in the first data table. Furthermore, the fourth and fifth indexes associated with the hit data pairs in the hit data table can also be deleted.

[0104] Optionally, when the first data stream is a data stream to be deleted and the data type of the first data stream is the second data type, the third index corresponding to the first data stream is queried in the second data table;

[0105] Retrieve the hit data pair corresponding to the third index from the hit data table, and delete the hit data pair;

[0106] Query the fourth index associated with the hit data pair in the hit data table, and delete the third index corresponding to the fourth index in the first data table;

[0107] Based on the second index, delete the second data stream and related data in the second data table.

[0108] Specifically, when the first data stream is a data stream to be deleted and the data type of the first data stream is the second data type, the principle and technical effect are similar to those of the first data stream being the first data type, and will not be repeated here.

[0109] In the above implementation, when deleting a data stream, the data stream and the associated hit data pairs can be quickly queried through the index of the data stream and the third index of the hit data pairs, resulting in high data processing efficiency.

[0110] Optionally, the method further includes:

[0111] If the first data stream is deleted, retrieve and delete the information about the first data stream from the listener list.

[0112] Specifically, the first data stream can be monitored through a listener. After a timeout notification and / or deletion notification of the data stream is detected, the information of the first data stream can be deleted from the listener list. For example, the information of all data streams in the listener list (data.listenerList) can be traversed to find the information corresponding to the first data stream and then the information of the first data stream can be deleted.

[0113] Optionally, the method further includes:

[0114] Receive timeout notifications and / or deletion notifications for data streams sent by the listener; the timeout notifications and / or deletion notifications for the data streams carry information about the data streams to be deleted.

[0115] Specifically, the lifecycle of a data stream can be timed using a timer. For example, if a data stream is valid for one hour or one day, a timeout notification is sent to the listener after the timer expires, and the data stream and related data are deleted based on this timeout notification.

[0116] Optionally, the deletion notification may be a deletion instruction triggered by the user or a deletion instruction from other modules.

[0117] The parameters involved in the following embodiments are shown in Tables 4 and 5 below, wherein the JoinComputingUnit is the execution subject for performing any method of the embodiments of the present invention:

[0118] Table 4

[0119]

[0120]

[0121] Table 5

[0122]

[0123]

[0124] For example, such as Figure 2 As shown, adding a data stream involves the following steps:

[0125] Step 110: Input the first data stream data into the joinComputingUnit;

[0126] Step 111: Use the data classifier (joinComputingUnit.dataClassifier) ​​to classify the first data stream data into the first data table or the second data table, and set the first field identifier isLeft. If it is the first data table, the first field identifier is the first field value; if it is the second data table, the first field identifier is the second field value.

[0127] For example, if it is the first data table, then isLeft = true; if it is the second data table, then isLeft = false.

[0128] Step 112: Set a listener for the first data stream, rowData.listener, and add the information of the first data stream to the listener list, data.listenerList;

[0129] Specifically, create a new row data type named RowData, the first data stream includes multiple rows of data, set rowData.data = data, and set rowData.isLeft = isLeft;

[0130] Step 113: Determine whether the first identifier field isLeft is equal to the value of the first field. If yes, proceed to step 114; otherwise, proceed to step 116.

[0131] Step 114: Use the left key value generator `leftKeyBuilder` to calculate the first key value `keyValue`; set `rowData.keyValue = keyValue`;

[0132] Step 115: Save the row data rowData of the first data stream to the first data table (e.g., the left row data storage key-value table joinComputingUnit.leftKeyedRowDataTable) according to the first key value keyValue, and set the saving index to the first index index, and set the index of the row data rowData.rowDataIndex = index. After completion, the process ends.

[0133] Step 116: Use the right-key value generator rightKeyBuilder to calculate the second key value keyValue; set rowData.keyValue = keyValue;

[0134] Step 117: Save the row data rowData of the first data stream to the second data table (e.g., the right row data storage key value table joinComputingUnit.rightKeyedRowDataTable) according to the second key value keyValue, and set the saving index to the second index index, and set the index of the row data rowData.rowDataIndex = index. After completion, the process ends.

[0135] For example, such as Figure 3 As shown, the data deletion process for a data stream includes the following steps:

[0136] Step 120: Input the first data stream data to be deleted into the joinComputingUnit;

[0137] Step 121: Retrieve information about the associated first data stream data (such as the row data rowData of the first data stream) from the listener list data.listenerList, and remove the information about the first data stream from data.listenerList;

[0138] Step 122: Iterate through the list of storage indices of the matched data pairs, rowData.matchedDataIndexList;

[0139] Step 123: Determine if there are any unprocessed matching data pairs in rowData.matchedDataIndexList and their storage index (i.e., the third index, matchedDataIndex). If there are, proceed to step 124; otherwise, proceed to step 125.

[0140] Step 124: Call the process to delete matched data: deleteMatchedData(matchedDataIndex, rowData) (see...) Figure 4 (See the flowchart shown), after completion, proceed to steps 123;

[0141] Step 125: Determine whether the first identifier field rowData.isLeft of the row data is equal to the value of the first field. If it is equal to the value of the first field, proceed to step 126; otherwise, proceed to step 127.

[0142] Step 126: Delete the row data rowData from the first data table (e.g., the left row data key-value table joinComputingUnit.leftKeyedRowDataTable included in the first data table) using the first key value rowData.keyValue and the first index rowData.rowDataIndex of the row data. Then end the process.

[0143] Step 127: Delete the row data rowData from the second data table (e.g., the right row data key-value table joinComputingUnit.rightKeyedRowDataTable) using the second key value rowData.keyValue and the second index rowData.rowDataIndex of the row data. Then end the process.

[0144] like Figure 4 As shown, the process for deleting hit data is as follows:

[0145] Step 130: Call the joinComputingUnit function to delete the matched data (deleteMatchedData(matchedDataIndex, fromRowData). The input parameters include the storage index of the matched data pair and the input data stream information.

[0146] Step 131: Use the storage index (i.e., the third index) of the matched data pair, matchedDataIndex, to retrieve the corresponding matched data pair, matchedData, from the matched data table joinComputingUnit.matchedDataTable;

[0147] Step 132: Determine whether the first data stream matchedData.leftRowData of the matched data pair is the input data stream fromRowData. If it is, proceed to step 133; otherwise, proceed to step 134.

[0148] Step 133: Use the storage index of the matched data pair in the second data table to delete the corresponding data from the second data table (such as the right index list of the matched data storage indexes included in the second data table, matchedData.rowDataIndex.matchedDataIndexList). After completion, proceed to step 135.

[0149] Step 134: Use the storage index of the matched data pair in the first data table (such as the left index list of matched data storage indexes included in the first data table, matchedData.leftRowData.matchedDataIndexList) to delete the corresponding data. After completion, proceed to step 135.

[0150] Step 135: Use the storage index matchedDataIndex of the matched data pair to delete the corresponding matched data pair from the matched data table joinComputingUnit.matchedDataTable. After completion, the process ends.

[0151] For example, such as Figure 5 As shown, the data stream timeout handling process is as follows:

[0152] Step 140: The first data stream receives a timer timeout notification;

[0153] Step 141: Iterate through the rowData information of all data streams in the listener list data.listenerList, and notify each rowData of the first data stream of the timeout.

[0154] Step 142: Notify the row data listener rowData.listener that the row data corresponding to the joinComputingUnit has timed out, and input the information of the timed-out row data rowData;

[0155] Step 143: Iterate through the list of indexes for the matched data pairs, rowData.matchedDataIndexList;

[0156] Step 144: Determine if there are any unprocessed matching data pairs in rowData.matchedDataIndexList and their storage index (i.e., the third index, matchedDataIndex). If there are, proceed to step 145; otherwise, proceed to step 146.

[0157] Step 145: Call the process to delete matched data: deleteMatchedData(matchedDataIndex, rowData) (see...) Figure 4 (See the flowchart shown), after completion, proceed to step 144;

[0158] Step 146: Determine whether the first identifier field rowData.isLeft of the row data is equal to the value of the first field. If it is equal to the value of the first field (true), proceed to step 147; otherwise, proceed to step 148.

[0159] Step 147: Delete the row data rowData from the first data table (e.g., the left row data key-value table joinComputingUnit.leftKeyedRowDataTable included in the first data table) using the first key value rowData.keyValue and the first index rowData.rowDataIndex of the row data. Then end the process.

[0160] Step 148: Delete the row data rowData from the second data table (e.g., the right row data key-value table joinComputingUnit.rightKeyedRowDataTable included in the first data table) using the second key value rowData.keyValue and the second index rowData.rowDataIndex of the row data. Then end the process.

[0161] Figure 6 This is one of the flowcharts illustrating the data flow association method provided in this embodiment of the invention. For example... Figure 6 As shown, the method provided in this embodiment includes:

[0162] Step 201: Obtain the first data stream and the first identifier field in the first data stream;

[0163] Specifically, the type of the data table storing the first data stream can be determined by the value of the first identifier field. For example, the type of the data table storing the first data stream corresponds to the data type of the first data stream. Optionally, the first data stream can be network behavior data, system log data, etc.

[0164] Step 202: When the type of the data table storing the first data stream is determined to be a first type based on the field value of the first identifier field, the first data stream is associated with at least one second data stream in the data table of the second type to obtain at least one first association result; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream.

[0165] Specifically, when the type of the data table storing the first data stream is determined to be the first type, that is, when the data type of the first data stream is determined to be the first data type, the first data type corresponds to the data table of the first type; at least one second data stream in the data table of the second type is obtained, and the first data stream is associated with at least one second data stream in the data table of the second type respectively to obtain at least one first association result.

[0166] The second type of data table corresponds to the second data type. For example, if the first data type is an abnormal data type, the first type of data table is used to store the data stream of the abnormal data type; if the second data type is a data type without security risks, the second type of data table is used to store the data stream of the data type without security risks.

[0167] For example, a data table of the first type can be called a first data table, and a data table of the second type can be called a second data table.

[0168] Step 203: Based on the first association result, perform association analysis.

[0169] Specifically, based on the first association result, it is determined whether the first data stream and the second data stream are successfully associated. For example, it is determined whether the first data stream and the second data stream meet the preset association conditions. The preset association conditions can be an association expression, such as the IP of the first data stream == the IP of the second data stream.

[0170] Optionally, if the association is successful, and the first and second data streams contain data with security risks, further security analysis can be performed, such as determining the security risk level or whether the data is abnormal.

[0171] Optionally, if the first association result is successful, network security analysis can also be performed based on the first association result. If the first data stream is a network behavior data stream that needs to be detected for security, and the second data stream is abnormal network behavior data, then when the association result is successful, the first data stream is determined to be abnormal network behavior data.

[0172] If the first data stream is an abnormal network behavior data stream; and the second data stream is network behavior data that needs to be detected for security; and the association result is successful, then the second data stream is determined to be abnormal network behavior data.

[0173] The method of this embodiment obtains a first data stream and a first identifier field in the first data stream; when the type of the data table storing the first data stream is determined to be a first type based on the field value of the first identifier field, the first data stream is associated with at least one second data stream in a data table of the second type to obtain at least one first association result; the type of the data table storing the first data stream can be quickly determined by the field value of the first identifier field, and then the data stream in the data table with a different type than the data table storing the first data stream is selected for association calculation during association calculation, and association analysis is performed based on the first association result, thereby improving the efficiency of data stream query and association processing.

[0174] Alternatively, step 202 can be implemented in the following way:

[0175] For any second data stream, determine whether the first data stream and the second data stream satisfy a preset association condition; the preset association condition is determined based on the attribute parameters of the first data stream and the attribute parameters of the second data stream.

[0176] When it is determined that the first data stream and the second data stream meet the preset association conditions, the first association result is determined to be a successful association;

[0177] If it is determined that the first data stream and the second data stream do not meet the preset association conditions, the first association result is determined to be an association failure.

[0178] Specifically, it is determined whether the first data stream and the second data stream meet the preset association conditions. The preset association conditions can be represented by an association expression, such as an association expression generated based on the attribute parameters of the first data stream and the second data stream, such as the source IP of the first data stream == the source IP of the second data stream, and the destination IP of the first data stream == the destination IP of the second data stream.

[0179] If the preset association conditions are met, the first association result is determined to be a successful association, that is, the first data stream and the second data stream are successfully associated. If the preset association conditions are not met, the first association result is determined to be a failed association, that is, the first data stream and the second data stream are not successfully associated.

[0180] If the association is successful, further network security analysis can be performed based on the first and second data streams, which can improve the efficiency of network security analysis.

[0181] Optionally, when the first association result is successful, the first data stream and the second data stream are stored as a hit data pair in the hit data table.

[0182] Optionally, when the data type of the first data stream is determined to be the first data type based on the second identifier field of the first data stream, the first data stream is stored in the first data table, a first index and / or a first key value corresponding to the first data stream is set in the first data table, and a corresponding listener is set for the first data stream. The listener is used to instruct the deletion of the first data stream when a timeout notification of the first data stream is heard.

[0183] Specifically, since the data type corresponds to the type of the data table storing the data stream, if the data type of the first data stream is determined to be a first data type, it can be stored in the first data table. Furthermore, a first index and / or first key value can be set for the first data stream, thereby improving the efficiency of subsequent data queries and processing. Moreover, a corresponding listener can be set for the first data stream to monitor whether the data stream needs to be deleted, improving the real-time performance of data processing.

[0184] Optionally, the method includes:

[0185] When it is determined that the type of the data table used to store the first data stream is the second type, the first data stream is associated with at least one third data stream in the data table of the first type to obtain at least one second association result.

[0186] Based on the second association result, an association analysis is performed.

[0187] Specifically, if the first data stream is of the second data type, that is, the type of the stored data table is of the second type, at least one third data stream from the first type of data table is obtained for association calculation. The implementation scheme and effect are similar to those in the previous embodiments, and will not be repeated here.

[0188] The parameter definitions involved in the following embodiments are shown in Table 6 below:

[0189] Table 6

[0190]

[0191]

[0192] The following data flow correlation analysis method embodiment includes four processes, the specific processes of which are shown in Table 7 below:

[0193] Table 7

[0194]

[0195] The insertion of data stream data includes two key processes, as shown in Table 8 below:

[0196] Table 8

[0197]

[0198]

[0199] For example, such as Figure 7 As shown, inserting data into a data stream involves calling the `insertStreamData(data)` function, which follows this process:

[0200] Step 210: Input the first data stream data <data>;

[0201] Step 211, Data Stream Classifier <streamclassifier>For the first input data stream <data>Perform classification calculations;

[0202] Step 212: If the result is left data, then proceed to step 213;

[0203] Step 213: Call the insertLeftTable(data) procedure to insert the left table. <data>Perform storage and calculations related to the left table, and then end the process.

[0204] Alternatively, call insertLeftKeyTable(data) to... <data>Perform storage and calculations related to the left key value table, and then end the process.

[0205] Step 214: If the result is the right data, proceed to step 215; otherwise, proceed to step 216: Discard this data.

[0206] Step 215: Call the insertRightTable(data) procedure to insert into the right table. <data>Perform storage and calculations related to the right table, and then end the process.

[0207] Alternatively, call insertRightKeyTable(data) to... <data>Perform storage and calculations related to the right-click value table, and then end the process.

[0208] like Figure 8 As shown, inserting data into the left table involves calling insertLeftTable(data), and the process is as follows:

[0209] Step 220: Input the first data stream data <data>;

[0210] Step 221: Traverse the right table <righttable>For each row of data <row>And the input first data stream data (the row data of the first data stream) <data>Perform correlation calculations;

[0211] Step 222, right table <righttable>Is there still data in China? <row>If yes, proceed to step 223; otherwise, proceed to step 226.

[0212] Step 223, for <row>and <data>Perform left correlation calculation, i.e. <data>As an lvalue <row>As an rvalue, the parameter<data,row> Input correlation filter <joinfilter>Perform correlation calculations;

[0213] Step 224, if <joinfilter>If the calculation result is true, proceed to step 225; if it is false, proceed to step 222. A true calculation result indicates that the association was successful.

[0214] Step 225,<data,row> Insert into the hit data table <matchedtable>After completion, proceed to step 222;

[0215] Step 226, for the first data stream <data>Registering and deleting listeners;

[0216] Step 227, <data>Store it in the left table (leftTable).

[0217] Inserting into the right table (insertRightTable) is similar to inserting into the left table, so it will not be described in detail here.

[0218] like Figure 9 As shown, the process for inserting the right-click value table (calling insertRightKeyTable(data)) is as follows:

[0219] Step 230: Input the first data stream data <data>;

[0220] Step 231: Use the right-click value generator <rightkeybuilder>Calculate the second key value, i.e., the right key value (rightKeyValue);

[0221] Step 232, Use the second key value <rightkeyvalue>From the first data table (such as the left key value table) <leftkeyedtable>Query the corresponding dataset in ); set the query result as leftRowEvents;

[0222] Step 233: Traverse the retrieved dataset <leftrowevents>For each row of data <row>and the first input data stream <data>Perform correlation calculations on the row data;

[0223] Steps 234 <leftrowevents>Is there still data in China? <row>If yes, proceed to step 235; otherwise, proceed to step 238.

[0224] Step 235, for <row>and <data>Perform left correlation calculation, i.e. <data>As an lvalue <row>As an rvalue, the parameter <lvalue, rvalue> (<data,row> Input correlation filter <joinfilter>Perform correlation calculations;

[0225] Step 236, if <joinfilter>If the calculation result is true, proceed to step 237; if it is false, proceed to step 234. A true calculation result indicates that the data association was successful.

[0226] Step 237,<data,row> Insert into the hit data table <matchedtable>After completion, proceed to step 234;

[0227] Step 238, for the first data stream <data>Registering and deleting listeners;

[0228] Step 239, based on the second key value <rightkeyvalue>The first data stream <data>Store in a second data table (such as a right-key value table) <rightkeyedtable>)middle.

[0229] For example, a sub-table corresponding to the second key value is stored in the second data table based on the second key value.

[0230] like Figure 10 As shown, the process of deleting a data stream (calling deleteStreamData(data)) is as follows:

[0231] Step 240: Input the first data stream data to be deleted. <data>;

[0232] Step 241: Cancel as the first data stream <data>Registered listeners can be deleted.

[0233] Step 242: From the hit data table <matchedtable>The query checks if there are any related hit pairs; if so, the corresponding hit pairs are deleted.

[0234] Step 243, Data Stream Classifier <streamclassifier>For the first input data stream <data>Perform classification calculations. If the calculation result is left data, proceed to step 244; if it is right data, proceed to step 246.

[0235] Step 244: Use the left key value generator <leftkeybuilder>Calculate the first key value, which is the left key value (leftKeyValue);

[0236] Step 245: Based on the first key value <leftkeyvalue>From the first data table (such as the left key value table) <leftkeyedtable>) Delete the first data stream <data>Once the corresponding data is complete, proceed to step 248;

[0237] Step 246: Use the right-click value generator <rightkeybuilder>Calculate the second key value, i.e., the right key value (rightKeyValue);

[0238] Step 247: Based on the second key value <rightkeyvalue>From the second data table (such as the right-click value table) <rightkeyedtable>) Delete the first data stream <data>Once the corresponding data is complete, proceed to step 248;

[0239] Step 248: If both the first and second data tables are empty, delete the current calculation unit.

[0240] For example, the process of updating data stream data using updateStreamData(oldData, newData) is as follows:

[0241] Call updateStreamData(oldData, newData) and input the data stream to be updated.<oldData,newData> ;

[0242] Call the deleteStreamData(oldData) method to delete the data stream. <olddata>;

[0243] Call the insertStreamData(NewData) method to insert data. <newdata>.

[0244] Optionally, notify the data stream to delete data after timeout by calling onStreamDataExpire(data), the process is the same as above. Figure 10 The deletion process shown is similar and will not be repeated here. In onStreamDataExpire(data), the input parameter data is the timed-out data stream data.

[0245] The data processing apparatus and data flow correlation analysis apparatus provided by the present invention are described below. The data processing apparatus described below and the data processing method described above can be referred to in correspondence with each other. The data flow correlation analysis apparatus described below and the data flow correlation analysis method described above can be referred to in correspondence with each other.

[0246] Figure 11 This is a schematic diagram of the structure of the data processing apparatus provided by the present invention. The data processing apparatus provided in this embodiment includes:

[0247] The acquisition module 1101 is used to acquire the first data stream and set a first identifier field for the first data stream;

[0248] The processing module 1102 is used to assign a value to the first identifier field according to the data type of the first data stream to obtain the field value of the first data stream; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream; the type of the data table storing the first data stream corresponds to the data type of the first data stream.

[0249] The storage module 1103 is used to store the first data stream in a data table corresponding to the data type.

[0250] Optionally, the processing module 1102 is specifically used for:

[0251] When the data type of the first data stream is determined to be a first data type, the first identifier field is assigned a first field value; the first field value is used to indicate that the data table storing the first data stream is of the first type; or,

[0252] When it is determined that the data type of the first data stream is the second data type, the first identifier field is assigned the value of the second field; the value of the second field is used to indicate that the data table storing the first data stream is of the second type;

[0253] Optionally, the storage module 1103 is specifically used for:

[0254] Store the first data stream in a first data table corresponding to the first type, and set a first index corresponding to the first data stream in the first data table; or,

[0255] The first data stream is stored in the second data table corresponding to the second type, and a second index corresponding to the first data stream is set in the second data table.

[0256] Optionally, the storage module 1103 is specifically used for:

[0257] Determine a first key value for the first data stream, and store the first data stream in the first data table based on the first key value;

[0258] Determine a second key value for the first data stream, and store the first data stream in the second data table based on the second key value.

[0259] Optionally, the acquisition module 1101 is also used for:

[0260] Obtain the hit data pair, store the hit data pair in the hit data table, and set a third index for the hit data pair; the hit data pair is obtained by performing an association calculation on the first data stream and the second data stream, and when the association result is successful; the second data stream and the first data stream come from different data tables;

[0261] Optionally, the storage module 1103 is also used for:

[0262] The third index of the hit data pair in the hit data table is stored in the first data table and the second data table respectively;

[0263] Optionally, the processing module 1102 is also used for:

[0264] A fourth index is set in the hit data table to associate the hit data pairs, and the fourth index is the index of the third index in the first data table;

[0265] A fifth index is set in the hit data table to associate the hit data pairs, and the fifth index is the index of the third index in the second data table.

[0266] Optionally, the processing module 1102 is also used for:

[0267] When the first data stream is a data stream to be deleted and the data type of the first data stream is a first data type, query the third index corresponding to the first data stream in the first data table;

[0268] Obtain the hit data pair corresponding to the third index from the hit data table, and delete the hit data pair;

[0269] Query the fifth index in the hit data table, and delete the third index corresponding to the fifth index in the second data table;

[0270] Based on the first index, delete the first data stream and related data in the first data table.

[0271] Optionally, the processing module 1102 is also used for:

[0272] When the first data stream is a data stream to be deleted and the data type of the first data stream is the second data type, query the third index corresponding to the first data stream in the second data table;

[0273] Retrieve the hit data pair corresponding to the third index from the hit data table, and delete the hit data pair;

[0274] Query the fourth index in the hit data table, and delete the third index corresponding to the fourth index in the first data table;

[0275] Based on the second index, delete the second data stream and related data in the second data table.

[0276] Optionally, the processing module 1102 is also used for:

[0277] A listener is set up for the first data stream, and the information of the first data stream for which the listener is set is added to the listener list; the listener is used to instruct the deletion of the first data stream when a timeout notification and / or deletion notification of the first data stream is detected.

[0278] Optionally, the processing module 1102 is also used for:

[0279] In the event that the first data stream is deleted, the information for deleting the first data stream is retrieved from the listener list.

[0280] Optionally, the processing module 1102 is also used for:

[0281] Receive timeout notifications and / or deletion notifications for data streams sent by the listener; the timeout notifications and / or deletion notifications for the data streams carry information about the data streams to be deleted.

[0282] Optionally, the processing module 1102 is also used for:

[0283] The data type of the first data stream is determined based on the second identifier field in the first data stream; the second identifier field is used to indicate the data type of the first data stream.

[0284] The apparatus of this invention is used to execute the method in any of the foregoing data processing method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0285] Figure 12 This is a schematic diagram of the data flow correlation analysis device provided by the present invention. The data flow correlation analysis device provided in this embodiment includes:

[0286] The acquisition module 1201 is used to acquire the first data stream and the first identifier field in the first data stream;

[0287] Processing module 1202 is used to perform association calculations on the first data stream and at least one second data stream in a second type data table respectively when the type of the data table storing the first data stream is determined to be a first type based on the field value of the first identifier field, to obtain at least one first association result; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream;

[0288] The processing module 1202 is also used to perform association analysis based on the first association result.

[0289] Optionally, the processing module 1202 is specifically used for:

[0290] For any second data stream, determine whether the first data stream and the second data stream satisfy a preset association condition; the preset association condition is determined based on the attribute parameters of the first data stream and the attribute parameters of the second data stream.

[0291] When it is determined that the first data stream and the second data stream meet the preset association conditions, the first association result is determined to be a successful association;

[0292] If it is determined that the first data stream and the second data stream do not meet the preset association conditions, the first association result is determined to be an association failure.

[0293] Optionally, the processing module 1202 is further configured to:

[0294] When the first association result is successful, the first data stream and the second data stream are stored as a hit data pair in the hit data table.

[0295] Optionally, the first data stream is a network behavior data stream for which security needs to be detected; the second data stream is abnormal network behavior data; the processing module 1202 is specifically used for:

[0296] When the association result is successful, the first data stream is determined to be abnormal network behavior data.

[0297] Optionally, the processing module 1202 is further configured to:

[0298] When the data type of the first data stream is determined to be the first data type based on the second identifier field of the first data stream, the first data stream is stored in the first data table, a first index and / or a first key value corresponding to the first data stream is set in the first data table, and a listener is set for the first data stream. The listener is used to instruct the deletion of the first data stream when a timeout notification of the first data stream is heard.

[0299] Optionally, the processing module 1202 is further configured to:

[0300] When it is determined that the type of the data table used to store the first data stream is the second type, the first data stream is associated with at least one third data stream in the data table of the first type to obtain at least one second association result.

[0301] Based on the second association result, an association analysis is performed.

[0302] Optionally, the processing module 1202 is further configured to:

[0303] If the first data stream is a data stream to be deleted or a data stream that has timed out, and the data type of the first data stream is a first data type, then based on the fourth index, the third index of the hit data pair including the first data stream is queried in the first data table; the fourth index is the index of the third index in the first data table.

[0304] Based on the third index, query the hit data table for hit data pairs that include the first data stream, and delete the hit data pairs in the hit data table;

[0305] The first data stream is deleted from the first data table based on the first index and / or the first key value corresponding to the first data stream.

[0306] Optionally, the processing module 1202 is further configured to:

[0307] If the first data stream is a data stream to be deleted or a data stream that has timed out, and the data type of the first data stream is a second data type, then based on the fifth index, the third index of the hit data pair including the first data stream is queried in the second data table; the fifth index is the index of the third index in the second data table.

[0308] Based on the third index, query the hit data table for hit data pairs that include the first data stream, and delete the hit data pairs in the hit data table;

[0309] Based on the first index and / or the first key value corresponding to the first data stream, delete the first data stream from the second data table.

[0310] The apparatus of this invention is used to execute the method in any of the aforementioned data stream association method embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.

[0311] Figure 13 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 13 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a data processing method, which includes:

[0312] Obtain the first data stream and set the first identifier field for the first data stream;

[0313] Based on the data type of the first data stream, the first identifier field is assigned a value to obtain the field value of the first data stream; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream; the type of the data table storing the first data stream corresponds to the data type of the first data stream.

[0314] The first data stream is stored in the data table corresponding to the data type.

[0315] or,

[0316] Data flow correlation analysis method, which includes:

[0317] Obtain the first data stream and the first identifier field in the first data stream;

[0318] When the type of the data table storing the first data stream is determined to be a first type based on the field value of the first identifier field, the first data stream is associated with at least one second data stream in the data table of the second type to obtain at least one first association result; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream.

[0319] Based on the first association result, an association analysis is performed.

[0320] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0321] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the data processing method provided by the above methods, the method comprising:

[0322] Obtain the first data stream and set the first identifier field for the first data stream;

[0323] Based on the data type of the first data stream, the first identifier field is assigned a value to obtain the field value of the first data stream; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream; the type of the data table storing the first data stream corresponds to the data type of the first data stream.

[0324] The first data stream is stored in the data table corresponding to the data type.

[0325] or,

[0326] Data flow correlation analysis method, which includes:

[0327] Obtain the first data stream and the first identifier field in the first data stream;

[0328] When the type of the data table storing the first data stream is determined to be a first type based on the field value of the first identifier field, the first data stream is associated with at least one second data stream in the data table of the second type to obtain at least one first association result; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream.

[0329] Based on the first association result, an association analysis is performed.

[0330] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the data processing methods provided above, the method comprising:

[0331] Obtain the first data stream and set the first identifier field for the first data stream;

[0332] Based on the data type of the first data stream, the first identifier field is assigned a value to obtain the field value of the first data stream; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream; the type of the data table storing the first data stream corresponds to the data type of the first data stream.

[0333] The first data stream is stored in the data table corresponding to the data type.

[0334] or,

[0335] Data flow correlation analysis method, which includes:

[0336] Obtain the first data stream and the first identifier field in the first data stream;

[0337] When the type of the data table storing the first data stream is determined to be a first type based on the field value of the first identifier field, the first data stream is associated with at least one second data stream in the data table of the second type to obtain at least one first association result; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream.

[0338] Based on the first association result, an association analysis is performed.

[0339] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0340] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0341] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / newdata> < / olddata> < / data> < / rightkeyedtable> < / rightkeyvalue> < / rightkeybuilder> < / data> < / leftkeyedtable> < / leftkeyvalue> < / leftkeybuilder> < / data> < / streamclassifier> < / matchedtable> < / data> < / data> < / rightkeyedtable> < / data> < / rightkeyvalue> < / data> < / matchedtable> < / joinfilter> < / joinfilter> < / row> < / data> < / data> < / row> < / row> < / leftrowevents> < / data> < / row> < / leftrowevents> < / leftkeyedtable> < / rightkeyvalue> < / rightkeybuilder> < / data> < / data> < / data> < / matchedtable> < / joinfilter> < / joinfilter> < / row> < / data> < / data> < / row> < / row> < / righttable> < / data> < / row> < / righttable> < / data> < / data> < / data> < / data> < / data> < / data> < / streamclassifier> < / data>

Claims

1. A data processing method, characterized in that, include: Obtain the first data stream and set the first identifier field for the first data stream; Based on the data type of the first data stream, the first identifier field is assigned a value to obtain the field value of the first data stream; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream; the type of the data table storing the first data stream corresponds to the data type of the first data stream. Store the first data stream in the data table corresponding to the data type; The method further includes: obtaining a hit data pair, storing the hit data pair in a hit data table, and setting a third index for the hit data pair; the hit data pair is obtained by performing an association calculation on the first data stream and the second data stream, and when the association result is successful; the second data stream and the first data stream come from different data tables; The hit data pairs are stored in the first data table and the second data table respectively using the third index in the hit data table; A fourth index is set in the hit data table to associate the hit data pairs, and the fourth index is the index of the third index in the first data table; A fifth index is set in the hit data table to associate the hit data pairs, and the fifth index is the index of the third index in the second data table; When the first data stream is a data stream to be deleted, the hit data pair is deleted from the hit data table based on the third index, and the associated index in the associated data table is cleared based on the fourth or fifth index.

2. The data processing method according to claim 1, characterized in that, Assigning a value to the first identifier field according to the data type of the first data stream includes: When the data type of the first data stream is determined to be a first data type, the first identifier field is assigned a first field value; the first field value is used to indicate that the data table storing the first data stream is of the first type; or, When it is determined that the data type of the first data stream is the second data type, the first identifier field is assigned the value of the second field; the value of the second field is used to indicate that the data table storing the first data stream is of the second type; The step of storing the first data stream in the data table corresponding to the data type includes: Store the first data stream in a first data table corresponding to the first type, and set a first index corresponding to the first data stream in the first data table; or, The first data stream is stored in the second data table corresponding to the second type, and a second index corresponding to the first data stream is set in the second data table.

3. The data processing method according to claim 2, characterized in that, The step of storing the first data stream in the first data table corresponding to the first type includes: Determine a first key value for the first data stream, and store the first data stream in the first data table based on the first key value; The step of storing the first data stream in the second data table corresponding to the second type includes: Determine a second key value for the first data stream, and store the first data stream in the second data table based on the second key value.

4. The data processing method according to claim 2, characterized in that, The method further includes: When the first data stream is a data stream to be deleted and the data type of the first data stream is a first data type, query the third index corresponding to the first data stream in the first data table; Obtain the hit data pair corresponding to the third index from the hit data table, and delete the hit data pair; Query the fifth index in the hit data table, and delete the third index corresponding to the fifth index in the second data table; Based on the first index, delete the first data stream and related data in the first data table.

5. The data processing method according to claim 2, characterized in that, The method further includes: When the first data stream is a data stream to be deleted and the data type of the first data stream is the second data type, query the third index corresponding to the first data stream in the second data table; Retrieve the hit data pair corresponding to the third index from the hit data table, and delete the hit data pair; Query the fourth index in the hit data table, and delete the third index corresponding to the fourth index in the first data table; Based on the second index, delete the second data stream and related data in the second data table.

6. The data processing method according to any one of claims 1-5, characterized in that, The method further includes: A listener is set up for the first data stream, and the information of the first data stream for which the listener is set is added to the listener list; the listener is used to instruct the deletion of the first data stream when a timeout notification and / or deletion notification of the first data stream is detected.

7. The data processing method according to claim 6, characterized in that, The method further includes: In the event that the first data stream is deleted, the information for deleting the first data stream is retrieved from the listener list.

8. The data processing method according to claim 4 or 5, characterized in that, Before querying the third index corresponding to the first data stream in the first data table, the method further includes: Receive timeout notifications and / or deletion notifications for data streams sent by the listener; the timeout notifications and / or deletion notifications for the data streams carry information about the data streams to be deleted.

9. The data processing method according to any one of claims 1-5, characterized in that, The method further includes: The data type of the first data stream is determined based on the second identifier field in the first data stream; the second identifier field is used to indicate the data type of the first data stream.

10. A data flow correlation analysis method, characterized in that, include: Obtain the first data stream and the first identifier field in the first data stream; When the type of the data table storing the first data stream is determined to be a first type based on the field value of the first identifier field, the first data stream is associated with at least one second data stream in the data table of the second type to obtain at least one first association result; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream. Based on the first association result, perform association analysis; The method further includes: when the first data stream is a data stream to be deleted or a timed-out data stream, and the data type of the first data stream is a first data type, querying the third index in the hit data table of the first data table based on the fourth index; the fourth index is the index of the third index in the first data table; Based on the third index, query the hit data table for hit data pairs that include the first data stream, and delete the hit data pairs in the hit data table; Based on the first index and / or first key value corresponding to the first data stream, the first data stream is deleted from the first data table; if the first data stream is a data stream to be deleted or a timed-out data stream, and the data type of the first data stream is a second data type, based on the fifth index, a third index is queried in the second data table for the hit data pairs including the first data stream in the hit data table; the fifth index is the index of the third index in the second data table; Based on the third index, query the hit data table for hit data pairs that include the first data stream, and delete the hit data pairs in the hit data table; Based on the first index and / or the first key value corresponding to the first data stream, delete the first data stream from the second data table.

11. The data flow correlation analysis method according to claim 10, characterized in that, The step of performing association calculations between the first data stream and at least one second data stream in a second type of data table to obtain at least one first association result includes: For any second data stream, determine whether the first data stream and the second data stream satisfy a preset association condition; the preset association condition is determined based on the attribute parameters of the first data stream and the attribute parameters of the second data stream. When it is determined that the first data stream and the second data stream meet the preset association conditions, the first association result is determined to be a successful association; If it is determined that the first data stream and the second data stream do not meet the preset association conditions, the first association result is determined to be an association failure.

12. The data flow correlation analysis method according to claim 11, characterized in that, The method further includes: When the first association result is successful, the first data stream and the second data stream are stored as a hit data pair in the hit data table.

13. The data flow correlation analysis method according to claim 12, characterized in that, The first data stream is a network behavior data stream for which security needs to be detected; The second data stream is abnormal network behavior data; The association analysis based on the first association result includes: When the association result is successful, the first data stream is determined to be abnormal network behavior data.

14. The data flow correlation analysis method according to claim 12, characterized in that, The method further includes: When the data type of the first data stream is determined to be the first data type based on the second identifier field of the first data stream, the first data stream is stored in the first data table, a first index and / or a first key value corresponding to the first data stream is set in the first data table, and a listener is set for the first data stream. The listener is used to instruct the deletion of the first data stream when a timeout notification of the first data stream is heard.

15. The data flow correlation analysis method according to any one of claims 10-14, characterized in that, The method further includes: When it is determined that the type of the data table used to store the first data stream is the second type, the first data stream is associated with at least one third data stream in the data table of the first type to obtain at least one second association result. Based on the second association result, an association analysis is performed.

16. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire the first data stream and set a first identifier field for the first data stream; The processing module is used to assign a value to the first identifier field according to the data type of the first data stream, so as to obtain the field value of the first data stream; The value of the first identifier field is used to indicate the type of the data table storing the first data stream; the type of the data table storing the first data stream corresponds to the data type of the first data stream. A storage module is used to store the first data stream in a data table corresponding to the data type; The storage module is further configured to obtain hit data pairs, store the hit data pairs in a hit data table, and set a third index for the hit data pairs; the hit data pairs are obtained by performing an association calculation on the first data stream and the second data stream, and when the association result is successful; the second data stream and the first data stream come from different data tables; The hit data pairs are stored in the first data table and the second data table respectively using the third index in the hit data table; A fourth index is set in the hit data table to associate the hit data pair, and the fourth index is the index of the third index in the first data table; a fifth index is set in the hit data table to associate the hit data pair, and the fifth index is the index of the third index in the second data table; when the first data stream is a data stream to be deleted, the hit data pair is deleted from the hit data table based on the third index, and the associated index in the associated data table is cleared based on the fourth index or the fifth index.

17. A data flow correlation analysis device, characterized in that, include: The acquisition module is used to acquire the first data stream and the first identifier field in the first data stream; The processing module is configured to, when determining that the type of the data table storing the first data stream is a first type based on the field value of the first identifier field, perform association calculations between the first data stream and at least one second data stream in a data table of a second type to obtain at least one first association result; the field value of the first identifier field is used to indicate the type of the data table storing the first data stream; The processing module is further configured to perform association analysis based on the first association result; the processing module is further configured to, when the first data stream is a data stream to be deleted or a timed-out data stream, and the data type of the first data stream is a first data type, query the third index of the hit data pair including the first data stream in the hit data table based on the fourth index in the first data table; the fourth index is the index of the third index in the first data table; query the hit data pair including the first data stream in the hit data table based on the third index, and delete the hit data pair in the hit data table; based on the first index and / or the first key value corresponding to the first data stream, from The first data stream is deleted from the first data table; if the first data stream is a data stream to be deleted or a timed-out data stream, and the data type of the first data stream is a second data type, based on the fifth index, the third index of the hit data pair including the first data stream is queried in the second data table; the fifth index is the index of the third index in the second data table; based on the third index, the hit data pair including the first data stream is queried in the hit data table, and the hit data pair is deleted in the hit data table; based on the first index and / or the first key value corresponding to the first data stream, the first data stream is deleted from the second data table.

18. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data processing method as described in any one of claims 1 to 9, or the data flow correlation analysis method as described in any one of claims 10 to 15.

19. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 9, or the data flow correlation analysis method as described in any one of claims 10 to 15.

20. A computer program product having executable instructions stored thereon, characterized in that, When executed by the processor, this instruction causes the processor to implement the data processing method as described in any one of claims 1 to 9, or the data flow correlation analysis method as described in any one of claims 10 to 15.

Images