A data processing method and apparatus

Abstract

This invention provides a data processing method, apparatus, device, and storage medium. The method includes: determining a first key value and a second key value corresponding to a first data stream; the first key value is used to store a first portion of data in the first data stream, and the second key value is used to store a second portion of data in the first data stream; based on the first key value, storing the first data stream in a preset first data storage table, and generating a first index corresponding to the first data stream; wherein the first index corresponds to the first key value; based on the second key value, storing the first data stream in a preset second data storage table, and generating a second index corresponding to the first data stream; wherein the second index corresponds to the first key value; storing the first index in a first index table, and storing the second index in a second index table. This solution enables rapid construction of an index corresponding to a stored single data stream, improving the efficiency of single data stream storage and management.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a data processing method and apparatus. Background Technology

[0002] With the rapid development of network technology, the methods and number of network attacks have also increased significantly, and various new attack methods are emerging one after another, bringing great pressure and challenges to management and maintenance personnel.

[0003] When managing single data streams, standard single data streams are typically stored directly in the database. This makes subsequent data association processing difficult. In reality, some data stored in the database is unrelated to the single data stream to be processed. Directly storing standard single data streams in the database results in a large volume of data to be associated with all single data streams in the database, making it difficult to quickly retrieve valid single data streams. Therefore, establishing appropriate identifiers for data streams to quickly retrieve valid single data streams and improve subsequent association processing capabilities has become an urgent technical problem to be solved. Summary of the Invention

[0004] This invention provides a data processing method, apparatus, device, and storage medium to solve the technical problem in the prior art that it is difficult to quickly build an index for a data stream, resulting in low data processing efficiency and the inability to quickly obtain a valid single data stream.

[0005] Specifically, the embodiments of the present invention provide the following technical solutions:

[0006] In a first aspect, embodiments of the present invention provide a data processing method, including:

[0007] Determine a first key value and a second key value corresponding to the first data stream; the first key value is used to store a first part of the data in the first data stream, and the second key value is used to store a second part of the data in the first data stream;

[0008] Based on the first key value, the first part of the data in the first data stream is stored in a preset first data storage table, and a first index corresponding to the first data stream is generated; wherein, the first index corresponds to the first key value;

[0009] Based on the second key value, the second part of the data in the first data stream is stored in a preset second data storage table, and a second index corresponding to the first data stream is generated; wherein, the second index corresponds to the first key value;

[0010] Store the first index and the second index into a preset index table.

[0011] Furthermore, after determining the first key value and the second key value corresponding to the first data stream, the method further includes: determining the row data listener corresponding to the first data stream; listening to the first data stream based on the row data listener, and determining whether the first data stream has timed out based on the listener information obtained from the listening; if the first data stream has timed out, triggering a data stream deletion process to delete the timed-out first data stream; wherein, the row data listener is used to instruct the deletion of the first data stream upon receiving a timeout notification from the first data stream.

[0012] Furthermore, the data processing method further includes:

[0013] If the first data stream is a data stream to be deleted, the indexes in the index table are traversed to find the corresponding first index and second index; based on the corresponding first index and second index and the preset data stream deletion process, the first data stream to be deleted in the first data storage table and the second data storage table is deleted.

[0014] Furthermore, the step of storing the first portion of data in the first data stream in a preset first data storage table based on the first key value, and generating a first index corresponding to the first data stream, specifically includes:

[0015] If it is determined that the first key value is not null, the first data stream is stored in the first data storage table based on the first key value, and a first index corresponding to the first data stream is generated.

[0016] Furthermore, the step of storing the second part of the data in the first data stream in a preset second data storage table based on the second key value and generating a second index corresponding to the first data stream specifically includes: when it is determined that the second key value is not null and the second key value is the same as the first key value, storing the first data stream in the second data storage table based on the second key value and generating a second index corresponding to the first data stream.

[0017] Secondly, embodiments of the present invention also provide a data flow correlation analysis method, which, after obtaining an index table storing a first index and a second index based on the method described above, further includes:

[0018] A first target key and a second target key are determined for the target data stream to be analyzed. The first target key corresponds to a first index, and the second target key corresponds to a second index. The target data stream is network behavior data obtained from network devices.

[0019] Based on the first target key value, the first data stream corresponding to the first index is found in the index table. The first data stream corresponding to the first index is performed with the target data stream to obtain the first association calculation data. The first association calculation data is then input into a preset association filter for association analysis to obtain the first association result.

[0020] Based on the second target key value, the first data stream corresponding to the second index is found in the index table. The first data stream corresponding to the second index is then subjected to a second association calculation with the target data stream to obtain second association calculation data. The second association calculation data is then input into a preset association filter for association analysis to obtain a second association result.

[0021] If the first association result is an association failure and the second association result is an association success, network security analysis is performed on the target data stream.

[0022] Furthermore, the data flow correlation analysis method, after obtaining the second correlation result, further includes:

[0023] If the first target key value and the second target key value are different, based on the first target key value, the second target key value, and a preset same-index data association analysis process, the target data stream is subjected to same-index data association analysis with the first data stream pre-stored in the data storage table to obtain a third association result; if the first target key value and the second target key value are the same, the target data stream is inserted into the hit data storage table, and the first target key value and the second target key value are stored in a preset index table as the first index value and the second index value, respectively.

[0024] The process of analyzing data associated with the same index includes: inputting data streams with the same index value into a preset association filter for association calculation; the index value includes a first index value and a second index value; and the hit data storage table is a storage table used to store data that has been successfully associated.

[0025] Furthermore, the data flow association analysis method, after obtaining the second association result, further includes: if the second association result is a successful association, storing the target data flow and the first data flow as a hit data pair in a hit data storage table.

[0026] Furthermore, the data flow correlation analysis method, after obtaining the first correlation result, further includes:

[0027] Update the version information corresponding to the version field in the target data stream to obtain the target version information;

[0028] If the first association result is successful, the target data stream and the first data stream are stored as a hit data pair in the hit data storage table, and the target version information is added to the first data stream.

[0029] Furthermore, the data stream association analysis method, after determining the first target key value and the second target key value of the target data stream, further includes:

[0030] If the first target index value and the second target index value are the same, the target data stream is stored in the first data storage table based on the second target index value;

[0031] The first target index value and the second target index value are stored in the preset index table, and a listener field is set in the target data stream. The listener is used to instruct the deletion of the target data stream upon receiving a timeout notification from the target data stream.

[0032] Furthermore, the data flow correlation analysis method further includes:

[0033] If the target data stream is a data stream to be deleted or a data stream that has timed out, and if a hit data pair including the target data stream is found in the hit data storage table, the hit data pair is deleted from the hit data storage table.

[0034] If it is determined that the first target index value and the second target index value stored in the target data stream are the same, the target data stream is deleted from the first index table based on the second target index value stored in the target data stream, and the second target index value in the first index table is also deleted.

[0035] If it is determined that the first target index value and the second target index value stored in the target data stream are different, the target data stream is deleted from the first index table based on the second target index value, and the second target index value in the first index table is also deleted; and the target data stream is deleted from the second index table based on the first target index value, and the first target index value in the second index table is also deleted.

[0036] Thirdly, embodiments of the present invention also provide a data processing apparatus, comprising:

[0037] A key value determination unit is used to determine a first key value and a second key value corresponding to a first data stream; the first key value is used to store a first part of the data in the first data stream, and the second key value is used to store a second part of the data in the first data stream.

[0038] The first data and index storage unit is used to store a first portion of the data in the first data stream in a preset first data storage table based on the first key value, and generate a first index corresponding to the first data stream; wherein the first index corresponds to the first key value.

[0039] The second data and index storage unit is used to store the second part of the data in the first data stream in a preset second data storage table based on the second key value, and generate a second index corresponding to the first data stream; wherein the second index corresponds to the first key value;

[0040] A data indexing processing unit is used to store the first index and the second index into a preset index table.

[0041] Furthermore, after determining the first key value and the second key value corresponding to the first data stream, the system further includes: a timeout data stream determination unit, used to determine the row data listener corresponding to the first data stream; to listen to the first data stream based on the row data listener, and to determine whether the first data stream has timed out based on the listener information obtained from the listening; if the first data stream has timed out, to trigger a data stream deletion process to delete the timed-out first data stream; wherein, the row data listener is used to instruct the deletion of the first data stream upon receiving a timeout notification from the first data stream.

[0042] Furthermore, the data processing device further includes a data stream deletion unit, configured to: when the first data stream is a data stream to be deleted, traverse the indexes in the index table to find the corresponding first index and second index; and delete the first data stream to be deleted from the first data storage table and the second data storage table based on the corresponding first index and second index and a preset data stream deletion process.

[0043] Furthermore, the first data and index storage unit is specifically used for:

[0044] If it is determined that the first key value is not null, the first data stream is stored in the first data storage table based on the first key value, and a first index corresponding to the first data stream is generated.

[0045] Furthermore, the second data and index storage unit is specifically used to: when it is determined that the second key value is not null and the second key value is the same as the first key value, store the first data stream in the second data storage table based on the second key value, and generate a second index corresponding to the first data stream.

[0046] Fourthly, embodiments of the present invention also provide a data association analysis apparatus, which, after obtaining an index table storing a first index and a second index based on the apparatus described above, further includes:

[0047] The target key value determination unit is used to determine a first target key value and a second target key value of the target data stream to be analyzed. The first target key value corresponds to a first index, and the second target key value corresponds to a second index. The target data stream is network behavior data obtained from a network device.

[0048] The first association analysis unit is used to find the first data stream corresponding to the first index from the index table based on the first target key value, perform a first association calculation on the first data stream corresponding to the first index and the target data stream to obtain first association calculation data, and input the first association calculation data into a preset association filter for association analysis to obtain a first association result;

[0049] The second association analysis unit is used to find the first data stream corresponding to the second index from the index table based on the second target key value, perform a second association calculation on the first data stream corresponding to the second index and the target data stream to obtain second association calculation data, and input the second association calculation data into a preset association filter for association analysis to obtain a second association result;

[0050] The security analysis unit is used to perform network security analysis on the target data stream when the first association result is an association failure and the second association result is an association success.

[0051] Furthermore, the data stream correlation analysis device, after obtaining the second correlation result, further includes: a data stream insertion unit, used for:

[0052] If the first target key value and the second target key value are different, based on the first target key value, the second target key value, and a preset same-index data association analysis process, the target data stream is subjected to same-index data association analysis with the first data stream pre-stored in the data storage table to obtain a third association result; if the first target key value and the second target key value are the same, the target data stream is inserted into the hit data storage table, and the first target key value and the second target key value are stored in a preset index table as the first index value and the second index value, respectively.

[0053] The process of analyzing data associated with the same index includes: inputting data streams with the same index value into a preset association filter for association calculation; the index value includes a first index value and a second index value; and the hit data storage table is a storage table used to store data that has been successfully associated.

[0054] Furthermore, the data stream association analysis device, after obtaining the second association result, further includes: a data stream storage unit, used to: store the target data stream and the first data stream as a hit data storage table when the second association result is a successful association.

[0055] Furthermore, the data stream correlation analysis device, after obtaining the first correlation result, further includes:

[0056] The data stream storage unit is further configured to: update the version information corresponding to the version field in the target data stream to obtain target version information;

[0057] If the first association result is successful, the target data stream and the first data stream are stored as a hit data pair in the hit data storage table, and the target version information is added to the first data stream.

[0058] Furthermore, the data stream association analysis device, after determining the first target key value and the second target key value of the target data stream, further includes: a data stream timeout deletion unit, used to: store the target data stream in the first data storage table based on the second target index value when the first target index value and the second target index value are the same;

[0059] The first target index value and the second target index value are stored in the preset index table, and a listener field is set in the target data stream. The listener is used to instruct the deletion of the target data stream upon receiving a timeout notification from the target data stream.

[0060] Furthermore, the data stream correlation analysis device further includes: a data stream deletion control unit, used for:

[0061] If the target data stream is a data stream to be deleted or a data stream that has timed out, and if a hit data pair including the target data stream is found in the hit data storage table, the hit data pair is deleted from the hit data storage table.

[0062] If it is determined that the first target index value and the second target index value stored in the target data stream are the same, the target data stream is deleted from the first index table based on the second target index value stored in the target data stream, and the second target index value in the first index table is also deleted.

[0063] If it is determined that the first target index value and the second target index value stored in the target data stream are different, the target data stream is deleted from the first index table based on the second target index value, and the second target index value in the first index table is also deleted; and the target data stream is deleted from the second index table based on the first target index value, and the first target index value in the second index table is also deleted.

[0064] Fifthly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the data processing method as described in any of the preceding claims, or implements the data flow correlation analysis method as described in any of the preceding claims.

[0065] Sixthly, embodiments of the present invention also provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described in any of the preceding claims, or implements the data flow correlation analysis method as described in any of the preceding claims.

[0066] Seventhly, embodiments of the present invention also provide a computer program product having executable instructions stored thereon, characterized in that, when executed by a processor, the instructions cause the processor to implement the data processing method as described in any of the preceding claims, or to implement the data flow correlation analysis method as described in any of the preceding claims.

[0067] The data processing method and apparatus provided in this invention determine a first key value and a second key value corresponding to a first data stream, store the first data stream in a preset first data storage table based on the first key value, and generate a first index corresponding to the first data stream; store the first data stream in a preset second data storage table based on the second key value, and generate a second index corresponding to the first data stream; store the first index and the second index in corresponding index tables for subsequent association lookup. This method can quickly construct the index corresponding to the stored single data stream, improving the efficiency of single data stream storage and management, so as to quickly obtain a valid single data stream in subsequent association processing.

[0068] The data association analysis method and apparatus provided in this invention, after obtaining a first index and a second index of a first data stream, determines a first target key value and a second target key value of the target data stream to be analyzed; based on the first target key value, searches for the first data stream corresponding to the first index in the index table, performs a first association calculation with the target data stream to obtain first association calculation data, and inputs the first association calculation data into a preset association filter for association analysis to obtain a first association result; based on the second target key value, searches for the first data stream corresponding to the second index in the index table, performs a second association calculation with the target data stream to obtain second association calculation data, and inputs the second association calculation data into a preset association filter for association analysis to obtain a second association result; if the first association result is an association failure and the second association result is an association success, performs network security analysis on the target data stream. This method can quickly perform association analysis using the index corresponding to a single data stream, effectively improving the efficiency of network security analysis of target data streams. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0070] Figure 1 This is a flowchart illustrating the data processing method provided in an embodiment of the present invention;

[0071] Figure 2 This is a data flow chart for adding data provided in an embodiment of the present invention;

[0072] Figure 3 This is a flowchart of the successful addition of association analysis data provided in an embodiment of the present invention;

[0073] Figure 4 This is a flowchart of the data deletion process provided in an embodiment of the present invention;

[0074] Figure 5 This is a flowchart of the data flow timeout process provided in an embodiment of the present invention;

[0075] Figure 6 This is a flowchart of the deletion of hit data provided in an embodiment of the present invention;

[0076] Figure 7 This is a schematic diagram of the structure of the data processing device provided in an embodiment of the present invention;

[0077] Figure 8 This is a flowchart illustrating the data association analysis method provided in an embodiment of the present invention;

[0078] Figure 9 This is a data flow diagram of the insertion data stream provided in an embodiment of the present invention;

[0079] Figure 10 This is a flowchart of the right index table association calculation provided in an embodiment of the present invention;

[0080] Figure 11 This is a flowchart of the left index table association calculation provided in an embodiment of the present invention;

[0081] Figure 12 This is a flowchart of the calculation process for data association with the same index provided in an embodiment of the present invention;

[0082] Figure 13 This is a data flow diagram of the storage data stream provided in an embodiment of the present invention;

[0083] Figure 14 This is a data flow diagram for deleting a data stream provided in an embodiment of the present invention;

[0084] Figure 15 This is a data flow diagram of the update data stream provided in an embodiment of the present invention;

[0085] Figure 16 This is a flowchart of the notification data stream deletion process provided in an embodiment of the present invention;

[0086] Figure 17 This is a schematic diagram of the data correlation analysis device provided in an embodiment of the present invention;

[0087] Figure 18 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0088] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0089] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0090] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0091] The following is combined Figures 1-18 The technical solutions of the embodiments of the present invention will be described in detail with reference to specific examples.

[0092] The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0093] Figure 1 This is a flowchart illustrating the data processing method provided in an embodiment of the present invention. Figure 1 As shown, the method provided in this embodiment includes:

[0094] Step 101: Determine the first key value and the second key value corresponding to the first data stream; the first key value is used to store the first part of the data in the first data stream, and the second key value is used to store the second part of the data in the first data stream.

[0095] The first data stream can be data stream data pointing to the input of the joinComputingUnit (association analysis computing unit). Specifically, for example... Figure 2As shown, the left row data storage key value, i.e., the leftKeyValue value (the second key value), can be generated based on the preset left key value generator `joinComputingUnit.leftKeyBuilder`, and the right row data storage key value, i.e., the rightKeyValue value (the first key value), can be generated using the preset right key value generator `joinComputingUnit.rightKeyBuilder`. The left row data is the second part of the data in the first data stream; the right row data is the first part of the data in the first data stream. A new row data `rowData` of data type `RowData` is created. Specifically, `rowData.data = data` can be set, and the row data listener `rowData.listener = joinComputingUnit` can be set, along with `rowData.leftKeyValue = leftKeyValue` and `rowData.rightKeyValue = rightKeyValue`. The row data `rowData` is then added to the listener list `data.listenerList`. The first data stream contains right row data and left row data. The first key value and the second key value described in this invention each contain corresponding key-value pairs, wherein the "key" in the key-value pair is used as the name of the key-value pair for storing and retrieving the corresponding value, and it can correspond to the index in the first index or the second index; the "value" in the key-value pair is used to represent the data stored and read.

[0096] If the first data stream is a data stream to be deleted, the indexes in the index table are traversed to find the corresponding first index and second index; based on the corresponding first index and second index and the preset data stream deletion process, the first data stream to be deleted from the first data storage table and the second data storage table is deleted. Figure 4The diagram shows the flowchart for deleting data stream data. (1) Input the data stream data to be deleted (i.e., the first data stream or the data stream to be deleted) into the joinComputingUnit. (2) Obtain the associated row data rowData from data.listenerList and delete rowData from data.listenerList. (3) Traverse the list of matched table storage indexes (preset index table or row data matched table storage index list) rowData.matchedDataIndexList. (4) Determine whether there are any unprocessed matched table storage indexes matchedDataIndex in rowData.matchedDataIndexList. If there are, execute step (5); otherwise, execute step (6). (5) Call the deleteMatchedData(matchedDataIndex, rowData) flowchart (see deleteMatchedData), and execute step (4) after completion. (6) If the key value rowData.rightKeyValue stored in the right row data storage key value table is not null, then the row data rowData is deleted from the right row data storage key value table joinComputingUnit.rightKeyedRowDataTable using the right row data storage key value table index rowData.rightRowDataIndex (i.e., the first index). (7) If rowData.isSameKeyValue is false and the key value rowData.leftKeyValue stored in the left row data storage key value table is not null, then the row data rowData is deleted from the left row data storage key value table joinComputingUnit.leftKeyedRowDataTable using the left row data storage key value table index rowData.leftRowDataIndex (i.e., the second index), and the process ends.

[0097] like Figure 6The flowchart shown is for deleting matched data. The process of deleting matched data deleteMatchedData includes: (1) calling the joinComputingUnit to deleteMatchedData(matchedDataIndex, fromRowData). (2) using the index matchedDataIndex in the index list of the matched data table to retrieve the corresponding matched data matchedData from the matched data storage table joinComputingUnit.matchedDataTable. (3) determining whether matchedData.leftRowData is equal to fromRowData. If they are equal, proceed to step (4); otherwise, proceed to step (5). (4) using the index matchedData.rightRowDataMatchedDataIndex to delete the corresponding data from the right row matched data matched data in the index list matchedData.rowDataIndex.matchedDataIndexList. After completion, proceed to step (6). (5) Use the index matchedData.leftRowDataMatchedDataIndex to delete the corresponding data from the left row data match table index list matchedData.leftRowData.matchedDataIndexList, and then execute step (6). (6) Use the index MatchedDataIndex to delete the corresponding matched data from the matched data storage table joinComputingUnit.matchedDataTable, and then end the process.

[0098] After determining the first key value and the second key value corresponding to the first data stream, the method further includes: determining the row data listener corresponding to the first data stream; listening to the first data stream based on the row data listener, and determining whether the first data stream has timed out based on the listener information obtained from the listening; if the first data stream has timed out, triggering a data stream deletion process to delete the timed-out first data stream; wherein, the row data listener is used to instruct the deletion of the first data stream when a timeout notification of the first data stream is received.

[0099] Step 102: Based on the first key value, store the first part of the data in the first data stream in a preset first data storage table, and generate a first index corresponding to the first data stream; wherein, the first index corresponds to the first key value.

[0100] Specifically, if the first key value is determined to be non-null, the first data stream is stored in the first data storage table based on the first key value, and a first index corresponding to the first data stream is generated. For example, it checks whether the rightKeyValue value is null. If not, the row data rowData is saved to the right row data storage key-value table joinComputingUnit.rightRowData, and the saving index is set to index. The right row data storage key-value table storage index rowData.rightRowDataIndex is set to index. If it is null, no operation is performed. The row data rowData includes left row data and right row data.

[0101] Step 103: Based on the second key value, store the second part of the data in the first data stream in a preset second data storage table, and generate a second index corresponding to the first data stream; wherein the second index corresponds to the first key value.

[0102] Specifically, if it is determined that the second key value is not null and is the same as the first key value, the first data stream is stored in the second data storage table based on the second key value, and a second index corresponding to the first data stream is generated. For example, it checks whether the leftKeyValue value is null. If it is, no operation is performed. If not, it continues to check whether the leftKeyValue value is equal to the rightKeyValue value. If they are equal, the row data rowData is saved to the left row data storage key value table joinComputingUnit.leftKeyedRowDataTable, and the saving index is set to index. The left row data storage key value table storage index rowData.leftRowDataIndex = index is also set. If they are not equal, rowData.isSameKeyValue = true is set, and the process ends after completion.

[0103] Step 104: Store the first index and the second index into a preset index table.

[0104] In addition, such as Figure 3The flowchart shown is a process for adding data that has been successfully added through association analysis. Specifically, it includes: (1) inputting a pair of rows of data that have been successfully added through association analysis into the joinComputingUnit.<leftRowData,rightRowData> (2) Create a new matched data pair named MatchedData, set matchedData.leftRowData = leftRowData, and set matchedData.rightRowData = rightRowData. (3) Save the matched data pairs to the matched data storage table joinComputingUnit.matchedDataTable, and set the save index to index. (4) Add the index to the left row matched data table storage index list (preset index table) leftRowData.matchedDataIndexList, set the save index to leftIndex, add the index to the right row matched data table storage index list (preset index table) rightRowData.matchedDataIndexList, and set the save index to rightIndex. (5) Set matchedData.leftRowDataMatchedDataIndex = leftIndex, set matchedData.rightRowDataMatchedDataIndex = rightIndex, and end the process after completion.

[0105] like Figure 5As shown, this is a flowchart of the data flow timeout process. Specifically, it includes: (1) The data flow data receives a timeout notification. (2) Iterate through all rows of data rowData in the listener list data.listenerList and notify each rowData that the data flow has timed out. (3) The rowData notifies the joinComputingUnit stored in the row data listener rowData.listener of the timeout and inputs the timed-out rowData. (4) Iterate through the list of matched table storage indexes rowData.matchedDataIndexList. (5) Determine whether there are any unprocessed matched table storage indexes matchedDataIndex in rowData.matchedDataIndexList. If there are, execute step (6); otherwise, execute step (7). (6) Call the process deleteMatchedData(matchedDataIndex, rowData) to delete the matched table data (see 2.2.5). After completion, execute step (5). (7) If the key value rowData.rightKeyValue stored in the right row data storage key value table is not null, then delete the row data rowData from the right row data storage key value table joinComputingUnit.rightKeyedRowDataTable using the right row data storage key value table index rowData.rightRowDataIndex. (8) If rowData.isSameKeyValue is false and the key value rowData.leftKeyValue stored in the left row data storage key value table is not null, then delete the row data rowData from the left row data storage key value table joinComputingUnit.leftKeyedRowDataTable using the left row data storage key value table index rowData.leftRowDataIndex. The process ends after completion.

[0106] It should be noted that in the data stream definition, the parameter `fieldValueMap` is defined as a field value table with a multi-level key-value structure, allowing you to retrieve the corresponding field value by inputting the field name. The parameter `listenerList` is defined as a list of listeners, storing the listeners registered on `Data` (row data listeners) that listen for data timeout deletion notifications.

[0107] In the row data definition, the parameter `data` is defined as data stream data, with data type `Data`, storing the data stream data corresponding to this row of data. One `RowData` corresponds to one data stream `Data`. The parameter `listener` is defined as the row data listener, storing the association analysis calculation unit associated with this row of data. This listener is notified when this row of data is deleted. The parameter `leftKeyValue` is defined as the key value stored in the left row data storage key-value table, recording the key value stored in the left row data storage key-value table (`JoinComputingUnit.leftKeyedRowDataTable`). The parameter `rightKeyValue` is defined as the key value stored in the right row data storage key-value table, recording the key value stored in the right row data storage key-value table (`JoinComputingUnit.rightKeyedRowDataTable`). The parameter `leftRowDataIndex` is defined as the index stored in the left row data storage key-value table, recording the position index of the current row of data in the left row data storage key-value table (`JoinComputingUnit.leftKeyedRowDataTable`), used for quickly deleting this row of data. The parameter `rightRowDataIndex` is defined as the index stored in the right row data storage key-value table. It records the position index of the current row data in the right row data storage key-value table (JoinComputingUnit.rightKeyedRowDataTable), used for quickly deleting data in the current row. The parameter `matchedDataIndexList` is defined as the list of indexes stored in the match table, recording the indexes of all matched data related to the current row in the match data storage table (JoinComputingUnit.matchedDataTable). It is used for quickly deleting match table data related to the current row. The parameter `isSameKeyValue` is defined as the key value stored in the row data storage key-value table to determine if the key and value are the same. It is a boolean value, true when `leftKeyValue` equals `rightKeyValue`, and false when `leftKeyValue` does not equal `rightKeyValue`. The default value is false.

[0108] In the definition of the hit data, the parameter `leftRowData` is defined as the left row data, with a data type of `RowData`, storing the left-joined row data from a pair of rows that have successfully undergone association analysis. The parameter `rightRowData` is defined as the right row data, with a data type of `RowData`, storing the right-joined row data from a pair of rows that have successfully undergone association analysis. The parameter `leftRowDataMatchedDataIndex` is defined as the index of the current hit data stored in the hit data storage table (`JoinComputingUnit.matchedDataTable`) within the hit table corresponding to the left row data, used for quickly deleting the left row data corresponding to the hit data. The parameter `rightRowDataMatchedDataIndex` is defined as the index of the current hit data stored in the hit data storage table (`rightRowData.matchedDataIndexList`) within the hit table corresponding to the right row data, used for quickly deleting the left row data corresponding to the hit data.

[0109] In the JoinComputingUnit definition, the parameter `leftKeyedRowDataTable` is defined as the key-value table for storing left-row data, where row data is stored by key value and the data type is `RowData`. The parameter `rightKeyedRowDataTable` is defined as the key-value table for storing right-row data, where row data is stored by key value and the data type is `RowData`. The parameter `leftKeyBuilder` is defined as the left key-value generator, which generates the corresponding key-value pairs for the left-row data based on the input data; the parameter `rightKeyBuilder` is defined as the right key-value generator, which generates the corresponding key-value pairs for the right-row data based on the input data. The parameter `matchedDataTable` is defined as the match data table, which stores the matched data, and the data type is `MatchedData`.

[0110] The use of real-time single data stream association analysis data is divided into 5 processes: (1) Adding data stream data process, the corresponding process includes saving the data stream data to the row data storage table (JoinComputingUnit.leftKeyedRowDataTable / rightKeyedRowDataTable). (2) Adding data stream data that has been successfully associated, including storing the pair of rows that have been successfully associated to the hit data storage table (JoinComputingUnit.matchedDataTable). (3) Deleting data stream data process, the corresponding process includes deleting the corresponding row data from the row data storage table (JoinComputingUnit.leftKeyedRowDataTable / rightKeyedRowDataTable), and deleting the hit data associated with the row data from the hit data storage table (JoinComputingUnit.matchedDataTable). The row data storage table includes the right row data storage key-value table and the left row data storage key-value table. (4) Data timeout process: The corresponding process includes deleting the corresponding timed-out row data from the row data storage table (JoinComputingUnit.leftKeyedRowDataTable / rightKeyedRowDataTable) and deleting the associated hit data from the hit data storage table (JoinComputingUnit.matchedDataTable). (5) Delete hit data process: The corresponding process includes deleting hit data. The calling method is deleteMatchedData(matchedDataIndex, fromRowData), where matchedDataIndex is the index of the hit table to be deleted, and fromRowData is the row data that triggers the deletion of hit data.

[0111] The data processing method described in this embodiment of the invention determines a first key value and a second key value corresponding to a first data stream. Based on the first key value, the first data stream is stored in a preset first data storage table, and a first index corresponding to the first data stream is generated. Based on the second key value, the first data stream is stored in a preset second data storage table, and a second index corresponding to the first data stream is generated. The first and second indexes are stored in corresponding index tables for subsequent association lookups. This method can quickly construct the index corresponding to the stored single data stream, improving the efficiency of single data stream storage and management, and facilitating the rapid retrieval of valid single data streams during subsequent association processing.

[0112] Corresponding to the data processing method provided above, the present invention also provides a data processing apparatus. Since the embodiments of this apparatus are similar to the method embodiments described above, the description is relatively simple. For relevant details, please refer to the description in the method embodiment section above. The embodiments of the data processing apparatus described below are merely illustrative. Please refer to... Figure 7 As shown, it is a structural schematic diagram of a data processing device provided in an embodiment of the present invention.

[0113] The data processing apparatus of the present invention specifically includes the following parts:

[0114] The key value determination unit 701 is used to determine the first key value and the second key value corresponding to the first data stream;

[0115] The first data and index storage unit 702 is used to store a first portion of the data in the first data stream in a preset first data storage table based on the first key value, and generate a first index corresponding to the first data stream; wherein the first index corresponds to the first key value;

[0116] The second data and index storage unit 703 is used to store the second part of the data in the first data stream in a preset second data storage table based on the second key value, and generate a second index corresponding to the first data stream; wherein the second index corresponds to the first key value;

[0117] The data indexing processing unit 704 is used to store the first index and the second index into a preset index table.

[0118] Furthermore, after determining the first key value and the second key value corresponding to the first data stream, the system further includes: a timeout data stream determination unit, used to determine the row data listener corresponding to the first data stream; to listen to the first data stream based on the row data listener, and to determine whether the first data stream has timed out based on the listener information obtained from the listening; if the first data stream has timed out, to trigger a data stream deletion process to delete the timed-out first data stream; wherein, the row data listener is used to instruct the deletion of the first data stream upon receiving a timeout notification from the first data stream.

[0119] Furthermore, the data processing device further includes a data stream deletion unit, configured to: when the first data stream is a data stream to be deleted, traverse the indexes in the index table to find the corresponding first index and second index; and delete the first data stream to be deleted from the first data storage table and the second data storage table based on the corresponding first index and second index and a preset data stream deletion process.

[0120] Furthermore, the first data and index storage unit is specifically used for:

[0121] If it is determined that the first key value is not null, the first data stream is stored in the first data storage table based on the first key value, and a first index corresponding to the first data stream is generated.

[0122] Furthermore, the second data and index storage unit is specifically used to: when it is determined that the second key value is not null and the second key value is the same as the first key value, store the first data stream in the second data storage table based on the second key value, and generate a second index corresponding to the first data stream.

[0123] The data processing apparatus described in this embodiment of the invention determines a first key value and a second key value corresponding to a first data stream, stores the first data stream in a preset first data storage table based on the first key value, and generates a first index corresponding to the first data stream; stores the first data stream in a preset second data storage table based on the second key value, and generates a second index corresponding to the first data stream; and stores the first index and the second index in corresponding index tables for subsequent association lookups. This method can quickly construct the index corresponding to the stored single data stream, improving the efficiency of single data stream storage and management, so as to quickly obtain a valid single data stream in subsequent association processing.

[0124] Corresponding to the data processing method and apparatus provided above, this invention also provides a data flow correlation analysis method and apparatus. For relevant details, please refer to the description in the embodiments section of the above method and apparatus. Embodiments of the data flow correlation analysis method and apparatus are described below. Please refer to... Figure 8 As shown, it is a flowchart illustrating a data flow correlation analysis method provided by an embodiment of the present invention.

[0125] Step 801: Determine the first target key value and the second target key value of the target data stream to be analyzed. The first target key value corresponds to the first index, and the second target key value corresponds to the second index. The target data stream is network behavior data obtained from network devices.

[0126] The target data stream can refer to the input data stream. <data>This refers to network behavior data obtained from network devices. The first target key value can refer to data based on... <rightkeybuilder>Calculate the right index value rightKeyValue, where the second target key value can refer to a key based on... <leftkeybuilder>The calculated left index value, leftKeyValue.

[0127] If the first target index value and the second target index value are the same, the target data stream is stored in the first data storage table based on the second target index value; the first target index value and the second target index value are stored in the preset index table, and a listener field is set in the target data stream. The listener is used to instruct the deletion of the target data stream when a timeout notification is received.

[0128] Step 802: Based on the first target key value, find the first data stream corresponding to the first index in the index table, perform a first association calculation between the first data stream corresponding to the first index and the target data stream to obtain first association calculation data, and input the first association calculation data into a preset association filter for association analysis to obtain a first association result. The first index dataset contains the first index.

[0129] For example, such as Figure 11 As shown, it is a flowchart of the left index table association calculation. Specifically, it includes the following steps: (1) Based on <rightkeyvalue>Values ​​from the left index table <leftkeyedtable>Query the corresponding index dataset <leftrowevents>(i.e., the first index dataset), the query result is set to leftRowEvents, the left index table <leftkeyedtable>This is the first index table. (2) Traverse the index dataset. <leftrowevents>For each row of data <row>(i.e., the corresponding first data stream) and input data <data>(i.e., the target data stream) is used for correlation calculation. (3) Index dataset <leftrowevents>Is there still data in the middle? <row>If yes, proceed to step (4); otherwise, exit the entire process. (4) Row data <row>The stored version number (version) and <insertversion>Check if they are equal. If they are equal, proceed to step (3); otherwise, proceed to step (5). (5) For <row>and <data>Perform right correlation calculation, i.e. <row>As an lvalue <data>As an rvalue, the parameter<row,data> (i.e., the first associated calculation data) is input into the preset associated filter. <joinfilter>Perform association calculations if <joinfilter>If the calculation result is true, then execute step (6); if it is false, then execute step (3). (6) Set the parameters<row,data> Insert into the hit table <matchedtable>After completion, proceed to step (3). The parameters mentioned above...<row,data> This refers to the first associated calculation data. Specifically, the version information (e.g., version number) corresponding to the version field in the target data stream can be updated to obtain the target version information. If the first association result is successful, the target data stream and the first data stream are stored as a hit data pair in the hit data storage table, and the target version information is added to the first data stream.

[0130] Step 803: Based on the second target key value, find the first data stream corresponding to the second index in the index table, perform a second association calculation between the first data stream corresponding to the second index and the target data stream to obtain second association calculation data, and input the second association calculation data into a preset association filter for association analysis to obtain a second association result. The second index dataset contains the second index.

[0131] For example, such as Figure 10 As shown, it is a flowchart of the right index table association calculation, which specifically includes the following steps: (1) Using <leftkeyvalue>The value (i.e., the second target key value) is from the right index table. <rightkeyedtable>(i.e., the second index table) queries the corresponding index dataset (i.e., the second index dataset), and sets the query result to rightRowEvents. (2) Traverse the index dataset <rightrowevents>For each row of data <row>(i.e., the first data stream corresponding to the second index dataset) and input data <data>(i.e., the target data stream) is used for correlation calculation. (3) <rightrowevents>Is there still data in the dataset? <row>If yes, proceed to step (4); otherwise, exit the entire process. (4) For <row>and <data>Perform left correlation calculation, i.e. <data>As an lvalue <row>As an rvalue, the parameter<data,row> Input correlation filter <joinfilter>Perform association calculation. If the result of joinFilter> is true, proceed to step (5); otherwise, proceed to step (6). (5)<data,row> Insert into the hit table <matchedtable>and set <row>version value version= <insertversion>After completion, proceed to step (3). (6) Judgment <row>Stored in<leftKeyValue,rightKeyValue> The values ​​are equal and the input is<leftKeyValue,rightKeyValue> If the values ​​are equal, proceed to step (7) if the condition is met; otherwise, proceed to step (3). (7) For <row>and <data>Perform right correlation calculation, i.e. <row>As an lvalue <data>As an rvalue, the parameter<row,data> (i.e., the second related calculation data) is input into the related filter. <joinfilter>Perform association calculation. If the result of joinFilter> is true, proceed to step (8); otherwise, proceed to step (3). (8)<row,data> Insert into the hit table <matchedtable>After completion, proceed to step (3).

[0132] After obtaining the second association result, the method further includes: if the second association result indicates a successful association, storing the target data stream and the first data stream as a hit data pair in a hit data storage table. Additionally, if the first target key value and the second target key value are different, based on the first target key value, the second target key value, and a preset same-index data association analysis process, performing same-index data association analysis on the target data stream and the first data stream pre-stored in the data storage table to obtain a third association result; if the first target key value and the second target key value are the same, inserting the target data stream into the hit data storage table, and storing the first target key value and the second target key value as the first index value and the second index value in a preset index table, respectively. The same-index data association analysis process includes: inputting data streams with corresponding index values ​​to a preset association filter for association calculation; the index values ​​include a first index value and a second index value; and the hit data storage table is a storage table used to store successfully associated data.

[0133] like Figure 12 As shown, it is a flowchart of the data association analysis process with the same index. Specifically, it includes the following steps: (1) Call joinSameKeyTable(data, rightKeyValue, insertVersion), and input the process parameters.<data,rightKeyValue,insertVersion> (2) Use <rightkeyvalue>Values ​​from the right index table <rightkeyedtable>(3) Traverse the index dataset. <leftsamekeyrowevents>For each row of data <row>(i.e., the first data stream pre-stored in the data storage table) and input data <data>(i.e., the target data stream) is used for correlation calculation. (4) <leftsamekeyrowevents>Is there still data in the dataset? <row>If the condition is met, proceed to step (5); otherwise, exit the entire process. (5) <row>Stored in<leftKeyValue,rightKeyValue> Check if the values ​​(i.e., the second target key value and the first target key value) are equal. If they are equal, proceed to step (6); otherwise, proceed to step (4). (6) For <row>and <data>Perform right correlation calculation, i.e. <row>As an lvalue <data>As an rvalue, the parameter<row,data> (i.e., data streams with the same corresponding index value) are input to the correlation filter. <joinfilter>Perform association calculation. If the result of joinFilter> is true, proceed to step (7); otherwise, proceed to step (4). (7)<row,data> Insert into the hit table <matchedtable>(Hit the data storage table), and then proceed to step (4).

[0134] like Figure 13 As shown, it is a flowchart for storing data stream data. (1) Call insertTable(data, leftKeyValue, rightKeyValue) and input the process parameters.<data,leftKeyValue,rightKeyValue> (2) with <rightkeyvalue>The value is the index value, <data>Store in the right index table <rightkeyedtable>(3) <leftkeyvalue>Check if the value of rightKeyValue is equal. If they are equal, proceed to step (5); otherwise, proceed to step (4). (4) <leftkeyvalue>The value is the index value, <data>Store in the left index table <leftkeyedtable>After completing the middle, proceed to step (5). (5)<leftKeyValue,rightKeyValue> Store to <data>(6) Towards <data>Register and delete listeners.

[0135] Step 804: If the first association result is association failure and the second association result is association success, perform network security analysis on the target data stream.

[0136] If the target data stream is a data stream to be deleted or a timed-out data stream, and a matching data pair including the target data stream is found in the matching data storage table, the matching data pair is deleted from the matching data storage table. If it is determined that the first target index value and the second target index value stored in the target data stream are the same, the target data stream is deleted from the first index table based on the second target index value stored in the target data stream, and the second target index value in the first index table is also deleted. If it is determined that the first target index value and the second target index value stored in the target data stream are different, the target data stream is deleted from the first index table based on the second target index value, and the second target index value in the first index table is also deleted. Furthermore, the target data stream is deleted from the second index table based on the first target index value, and the first target index value in the second index table is also deleted.

[0137] Specifically, such as Figure 14 As shown, the flowchart of the data deletion process is as follows: (1) Call deleteStreamData(data) and input the data stream to be deleted. <data>(2) Cancel the process <data>Registered listener deletion. (3) From <matchedtable>Check if there are any related hit rows; if so, delete the corresponding hit data. (4) Based on <data>Stored index value <rightkeyvalue>From the right index table <rightkeyedtable>Delete <data>。(5) <data>Stored <leftkeyvalue>Check if the value of rightKeyValue is equal. If they are equal, proceed to step (7); otherwise, proceed to step (6). (6) According to <data>Stored index value <leftkeyvalue>From the left index table <leftkeyedtable>Delete <data>After completion, proceed to step (7). If the left index table <leftkeyedtable>and right index table <rightkeyedtable>If all values ​​are empty, then delete the current calculation unit.

[0138] like Figure 15 As shown, it is a flowchart of the data update process. (1) Call updateStreamData(oldData, newData) and input the data stream to be updated.<oldData,newData> (2) Call deleteStreamData(oldData) to delete the data stream. <olddata>(3) Call the insertStreamData(NewData) function to insert data. <newdata>.

[0139] like Figure 16 The diagram shows the flowchart for the notification data stream deletion process. It calls onStreamExpire(data) and inputs the timed-out data stream data. <data>(1) From <matchedtable>Query whether there are any related hit rows; if so, delete the corresponding hit data. (2) Based on <data>Stored index value <rightkeyvalue>From the right index table <rightkeyedtable>Delete <data>。(3) <data>Stored <leftkeyvalue>Check if the value of rightKeyValue is equal. If they are equal, proceed to step (6); otherwise, proceed to step (5). (4) According to <data>Stored index value <leftkeyvalue>From the left index table <leftkeyedtable>Delete <data>After completion, proceed to step (6). If the left index table <leftkeyedtable>and right index table <rightkeyedtable>If all values ​​are empty, then delete the current calculation unit.

[0140] It should be noted that in the parameter definition of indexed real-time single-stream association analysis, the parameter `streamFilter` is defined as a stream filter, which filters the input stream data and removes data that does not meet the filtering conditions. The parameter `leftKeyedTable` is defined as the left index table, storing the left-joined data index values ​​and their corresponding datasets. The parameter `rightKeyedTable` is defined as the right index table, storing the right-joined data index values ​​and their corresponding datasets. The parameter `leftKeyBuilder` is defined as the left index value generator, generating the corresponding left index value based on the input data. The parameter `rightKeyBuilder` is defined as the right index value generator, generating the corresponding right index value based on the input data. The parameter `matchedTable` is defined as the match table, storing successfully joined data, with each row corresponding to a pair of successfully joined data. The parameter `joinFilter` is defined as the association filter, calculating the values ​​for the pairs of data to be joined; a value of `true` indicates a successful association, while `false` indicates a failed association. The parameter `sequenceID` is defined as the sequence number used for association calculation.

[0141] The indexed real-time single data stream correlation analysis method includes four processes: the insertStreamData(data) process, which inserts the corresponding data stream data. <data>The `deleteStreamData(data)` method deletes the corresponding data stream. <data>The updateStreamData(oldData, newData) function updates the corresponding data stream data and includes two steps: 1. Deleting data stream data. <olddata>;

[0142] 2. Insert data stream data <newdata>The notification process for data stream timeout is `onStreamDataExpire(data)`, which involves receiving data from the data stream. <data>Upon receiving a timeout notification, the corresponding data stream data will be deleted. <data>.

[0143] The `insertStreamData(data)` process for inserting data into a data stream comprises four key steps: 1. `joinRightTable`, which performs a right index table join calculation (called `joinRightTable(data, leftKeyValue, rightKeyValue, insertVersion)`). 2. `joinLeftTable`, which performs a left index table join calculation (called `joinLeftTable(data, rightKeyValue, insertVersion)`). 3. `joinSameKeyTable`, which performs a same-index join calculation (called `joinSameKeyTable(data, rightKeyValue, insertVersion)`). 4. `insertTable`, which stores the data stream data (called `insertTable(data, leftKeyValue, rightKeyValue)`).

[0144] The data stream association analysis method described in this embodiment of the invention obtains a first index and a second index of a first data stream, determines a first target key value and a second target key value of the target data stream to be analyzed, searches for the first data stream corresponding to the first index in the index table based on the first target key value, performs a first association calculation with the target data stream to obtain first association calculation data, and inputs the first association calculation data into a preset association filter for association analysis to obtain a first association result; searches for the first data stream corresponding to the second index in the index table based on the second target key value, performs a second association calculation with the target data stream to obtain second association calculation data, and inputs the second association calculation data into a preset association filter for association analysis to obtain a second association result; if the first association result is an association failure and the second association result is an association success, network security analysis is performed on the target data stream. This method can quickly perform association analysis using the index corresponding to a single data stream, effectively improving the efficiency of network security analysis of target data streams.

[0145] Please refer to Figure 17 As shown, it is a flowchart illustrating a data flow correlation analysis method provided by an embodiment of the present invention.

[0146] This invention also provides a data correlation analysis apparatus, characterized in that, after obtaining the first index and the second index of the first data stream based on the apparatus described above, it further includes:

[0147] The target key value determination unit 1701 is used to determine a first target key value and a second target key value of the target data stream to be analyzed. The first target key value corresponds to a first index, and the second target key value corresponds to a second index. The target data stream is network behavior data obtained from a network device.

[0148] The first association analysis unit 1702 is used to search for the first data stream corresponding to the first index in the index table based on the first target key value, perform a first association calculation on the first data stream corresponding to the first index and the target data stream to obtain first association calculation data, and input the first association calculation data into a preset association filter for association analysis to obtain a first association result;

[0149] The second association analysis unit 1703 is used to search for the first data stream corresponding to the second index in the index table based on the second target key value, perform a second association calculation on the first data stream corresponding to the second index and the target data stream to obtain second association calculation data, and input the second association calculation data into a preset association filter for association analysis to obtain a second association result;

[0150] The third association analysis unit 1704 is used to perform same-index data association analysis on the target data stream and the first data stream pre-stored in the data storage table based on the first target key value, the second target key value and the preset same-index data association analysis process, and obtain the third association result;

[0151] The security analysis unit is used to perform network security analysis on the target data stream when the first association result is an association failure and the second association result is an association success.

[0152] Furthermore, the data stream correlation analysis device, after obtaining the second correlation result, further includes: a data stream insertion unit, used for:

[0153] If the first target key value and the second target key value are different, based on the first target key value, the second target key value, and a preset same-index data association analysis process, the target data stream is subjected to same-index data association analysis with the first data stream pre-stored in the data storage table to obtain a third association result; if the first target key value and the second target key value are the same, the target data stream is inserted into the hit data storage table, and the first target key value and the second target key value are stored in a preset index table as the first index value and the second index value, respectively.

[0154] The process of analyzing data associated with the same index includes: inputting data streams with the same index value into a preset association filter for association calculation; the index value includes a first index value and a second index value; and the hit data storage table is a storage table used to store data that has been successfully associated.

[0155] Furthermore, the data stream association analysis device, after obtaining the second association result, further includes: a data stream storage unit, used to: store the target data stream and the first data stream as a hit data storage table when the second association result is a successful association.

[0156] Furthermore, the data stream correlation analysis device, after obtaining the first correlation result, further includes:

[0157] The data stream storage unit is further configured to: update the version information corresponding to the version field in the target data stream to obtain target version information;

[0158] If the first association result is successful, the target data stream and the first data stream are stored as a hit data pair in the hit data storage table, and the target version information is added to the first data stream.

[0159] Furthermore, the data stream association analysis method, after determining the first target key value and the second target key value of the target data stream, further includes: a data stream timeout deletion unit, used to: store the target data stream in the first data storage table based on the second target index value when the first target index value and the second target index value are the same;

[0160] The first target index value and the second target index value are stored in the preset index table, and a listener field is set in the target data stream. The listener is used to instruct the deletion of the target data stream upon receiving a timeout notification from the target data stream.

[0161] Furthermore, the data stream correlation analysis device further includes: a data stream deletion control unit, used for:

[0162] If the target data stream is a data stream to be deleted or a data stream that has timed out, and if a hit data pair including the target data stream is found in the hit data storage table, the hit data pair is deleted from the hit data storage table.

[0163] If it is determined that the first target index value and the second target index value stored in the target data stream are the same, the target data stream is deleted from the first index table based on the second target index value stored in the target data stream, and the second target index value in the first index table is also deleted.

[0164] If it is determined that the first target index value and the second target index value stored in the target data stream are different, the target data stream is deleted from the first index table based on the second target index value, and the second target index value in the first index table is also deleted; and the target data stream is deleted from the second index table based on the first target index value, and the first target index value in the second index table is also deleted.

[0165] The data stream correlation analysis device described in this embodiment of the invention, after obtaining a first index and a second index of a first data stream, determines a first target key value and a second target key value of the target data stream to be analyzed; based on the first target key value, it searches for the first data stream corresponding to the first index in the index table, performs a first correlation calculation with the target data stream to obtain first correlation calculation data, and inputs the first correlation calculation data into a preset correlation filter for correlation analysis to obtain a first correlation result; based on the second target key value, it searches for the first data stream corresponding to the second index in the index table, performs a second correlation calculation with the target data stream to obtain second correlation calculation data, and inputs the second correlation calculation data into a preset correlation filter for correlation analysis to obtain a second correlation result; if the first correlation result is a failure and the second correlation result is a success, it performs network security analysis on the target data stream. This method can quickly perform correlation analysis using the index corresponding to a single data stream, effectively improving the efficiency of network security analysis of target data streams.

[0166] Corresponding to the data processing method provided above, the present invention also provides an electronic device. Since the embodiment of this electronic device is similar to the method embodiment described above, it is described simply. For relevant details, please refer to the description in the method embodiment section above. The electronic device described below is merely illustrative. Figure 18 The diagram shown is a schematic representation of the physical structure of an electronic device disclosed in an embodiment of the present invention. The electronic device may include a processor 1801, a memory 1802, and a communication bus 1803. The processor 1801 and the memory 1802 communicate with each other via the communication bus 1803 and communicate with external devices via a communication interface 1804. The processor 1801 can invoke logical instructions in the memory 1802 to execute a data processing method, the method comprising: determining a first key value and a second key value corresponding to a first data stream; the first key value being a key value used to store a first portion of data in the first data stream, and the second key value being a key value used to store a second portion of data in the first data stream; based on the first key value, storing the first portion of data in the first data stream in a preset first data storage table, and generating a first index corresponding to the first data stream; wherein the first index corresponds to the first key value; based on the second key value, storing the second portion of data in the first data stream in a preset second data storage table, and generating a second index corresponding to the first data stream; wherein the second index corresponds to the first key value; storing the first index in a first index table, and storing the second index in a second index table. Alternatively, a data flow correlation analysis method can be executed, which includes: determining a first target key value and a second target key value for a target data flow to be analyzed, wherein the first target key value corresponds to a first index and the second target key value corresponds to a second index; the target data flow is network behavior data obtained from a network device; based on the first target key value, searching for the first data flow corresponding to the first index in the index table, performing a first correlation calculation between the first data flow corresponding to the first index and the target data flow to obtain first correlation calculation data, and inputting the first correlation calculation data into a preset correlation filter for correlation analysis to obtain a first correlation result; based on the second target key value, searching for the first data flow corresponding to the second index in the index table, performing a second correlation calculation between the first data flow corresponding to the second index and the target data flow to obtain second correlation calculation data, and inputting the second correlation calculation data into a preset correlation filter for correlation analysis to obtain a second correlation result; and performing network security analysis on the target data flow if the first correlation result is a failure and the second correlation result is a success.

[0167] Furthermore, the logical instructions in the aforementioned memory 1802 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as memory chips, USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0168] On the other hand, embodiments of the present invention also provide a computer program product, the computer program product including a computer program stored on a processor-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the data processing methods provided in the above-described method embodiments. The method includes: determining a first key value and a second key value corresponding to a first data stream; the first key value is a key value used to store a first portion of data in the first data stream, and the second key value is a key value used to store a second portion of data in the first data stream; based on the first key value, storing the first portion of data in the first data stream in a preset first data storage table, and generating a first index corresponding to the first data stream; wherein the first index corresponds to the first key value; based on the second key value, storing the second portion of data in the first data stream in a preset second data storage table, and generating a second index corresponding to the first data stream; wherein the second index corresponds to the first key value; storing the first index and the second index in a preset index table; or, a data stream association analysis method may also be executed, the method including: determining a first target key value and a second target key value of a target data stream to be analyzed, the first target key value and the first... The indexes correspond, with the second target key value corresponding to the second index; the target data stream is network behavior data obtained from network devices; based on the first target key value, the first data stream corresponding to the first index is searched in the index table, and a first association calculation is performed between the first data stream corresponding to the first index and the target data stream to obtain first association calculation data, which is then input into a preset association filter for association analysis to obtain a first association result; based on the second target key value, the first data stream corresponding to the second index is searched in the index table, and a second association calculation is performed between the first data stream corresponding to the second index and the target data stream to obtain second association calculation data, which is then input into a preset association filter for association analysis to obtain a second association result; if the first association result is an association failure and the second association result is an association success, network security analysis is performed on the target data stream.

[0169] In another aspect, embodiments of the present invention also provide a processor-readable storage medium storing a computer program, which, when executed by a processor, is implemented to perform the data processing methods provided in the above embodiments. The method includes: determining a first key value and a second key value corresponding to a first data stream; the first key value being a key value used to store a first portion of data in the first data stream, and the second key value being a key value used to store a second portion of data in the first data stream; based on the first key value, storing the first portion of data in the first data stream in a preset first data storage table, and generating a first index corresponding to the first data stream; wherein the first index corresponds to the first key value; based on the second key value, storing the second portion of data in the first data stream in a preset second data storage table, and generating a second index corresponding to the first data stream; wherein the second index corresponds to the first key value; and storing the first index and the second index in a preset index table. Alternatively, a data flow correlation analysis method can be performed, which includes: determining a first target key value and a second target key value for the target data flow to be analyzed, wherein the first target key value corresponds to a first index and the second target key value corresponds to a second index; the target data flow is network behavior data obtained from a network device; based on the first target key value, searching for the first data flow corresponding to the first index in the index table, performing a first correlation calculation between the first data flow corresponding to the first index and the target data flow to obtain first correlation calculation data, and inputting the first correlation calculation data into a preset correlation filter for correlation analysis to obtain a first correlation result; based on the second target key value, searching for the first data flow corresponding to the second index in the index table, performing a second correlation calculation between the first data flow corresponding to the second index and the target data flow to obtain second correlation calculation data, and inputting the second correlation calculation data into a preset correlation filter for correlation analysis to obtain a second correlation result; and performing network security analysis on the target data flow if the first correlation result is a failure and the second correlation result is a success.

[0170] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0171] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0172] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / data> < / data> < / newdata> < / olddata> < / data> < / data> < / rightkeyedtable> < / leftkeyedtable> < / data> < / leftkeyedtable> < / leftkeyvalue> < / data> < / leftkeyvalue> < / data> < / data> < / rightkeyedtable> < / rightkeyvalue> < / data> < / matchedtable> < / data> < / newdata> < / olddata> < / rightkeyedtable> < / leftkeyedtable> < / data> < / leftkeyedtable> < / leftkeyvalue> < / data> < / leftkeyvalue> < / data> < / data> < / rightkeyedtable> < / rightkeyvalue> < / data> < / matchedtable> < / data> < / data> < / data> < / data> < / leftkeyedtable> < / data> < / leftkeyvalue> < / leftkeyvalue> < / rightkeyedtable> < / data> < / rightkeyvalue> < / matchedtable> < / joinfilter> < / data> < / row> < / data> < / row> < / row> < / row> < / leftsamekeyrowevents> < / data> < / row> < / leftsamekeyrowevents> < / rightkeyedtable> < / rightkeyvalue> < / matchedtable> < / joinfilter> < / data> < / row> < / data> < / row> < / row> < / insertversion> < / row> < / matchedtable> < / joinfilter> < / row> < / data> < / data> < / row> < / row> < / rightrowevents> < / data> < / row> < / rightrowevents> < / rightkeyedtable> < / leftkeyvalue> < / matchedtable> < / joinfilter> < / joinfilter> < / data> < / row> < / data> < / row> < / insertversion> < / row> < / row> < / leftrowevents> < / data> < / row> < / leftrowevents> < / leftkeyedtable> < / leftrowevents> < / leftkeyedtable> < / rightkeyvalue> < / leftkeybuilder> < / rightkeybuilder> < / data>

Claims

1. A data processing method, characterized in that, include: Determine the first key value and the second key value corresponding to the first data stream; The first key value is used to store the first part of the data in the first data stream, and the second key value is used to store the second part of the data in the first data stream; Based on the first key value, the first part of the data in the first data stream is stored in a preset first data storage table, and a first index corresponding to the first data stream is generated; wherein, the first index corresponds to the first key value; Based on the second key value, the second part of the data in the first data stream is stored in a preset second data storage table, and a second index corresponding to the first data stream is generated; wherein, the second index corresponds to the first key value; Store the first index and the second index into a preset index table; The index table is used to find the first data stream corresponding to the first index based on the first target key value, and to find the first data stream corresponding to the second index based on the second target key value; the first target key value and the second target key value are determined based on the target data stream to be analyzed, the first target key value corresponds to the first index, and the second target key value corresponds to the second index.

2. The data processing method according to claim 1, characterized in that, After determining the first key value and the second key value corresponding to the first data stream, the process also includes: A row data listener corresponding to the first data stream is determined; the first data stream is monitored based on the row data listener, and the listener information obtained from the monitoring is used to determine whether the first data stream has timed out. If the first data stream times out, a data stream deletion process is triggered to delete the timed-out first data stream; wherein, the row data listener is used to instruct the deletion of the first data stream when a timeout notification of the first data stream is received.

3. The data processing method according to claim 1, characterized in that, Also includes: If the first data stream is a data stream to be deleted, traverse the indexes in the index table to find the corresponding first index and second index; Based on the corresponding first index and second index and the preset data stream deletion process, the first data stream to be deleted in the first data storage table and the second data storage table is deleted.

4. The data processing method according to claim 1, characterized in that, The step of storing a first portion of the data in the first data stream in a preset first data storage table based on the first key value, and generating a first index corresponding to the first data stream, specifically includes: If it is determined that the first key value is not null, the first data stream is stored in the first data storage table based on the first key value, and a first index corresponding to the first data stream is generated.

5. The data processing method according to claim 1, characterized in that, The step of storing the second part of the data in the first data stream in a preset second data storage table based on the second key value and generating a second index corresponding to the first data stream specifically includes: when it is determined that the second key value is not null and the second key value is the same as the first key value, storing the first data stream in the second data storage table based on the second key value and generating a second index corresponding to the first data stream.

6. A data flow correlation analysis method, characterized in that, After obtaining the index table storing the first index and the second index using the method according to any one of claims 1 to 5, the method further includes: A first target key and a second target key are determined for the target data stream to be analyzed. The first target key corresponds to a first index, and the second target key corresponds to a second index. The target data stream is network behavior data obtained from network devices. Based on the first target key value, the first data stream corresponding to the first index is found in the index table. The first data stream corresponding to the first index is performed with the target data stream to obtain the first association calculation data. The first association calculation data is then input into a preset association filter for association analysis to obtain the first association result. Based on the second target key value, the first data stream corresponding to the second index is found in the index table. The first data stream corresponding to the second index is then subjected to a second association calculation with the target data stream to obtain second association calculation data. The second association calculation data is then input into a preset association filter for association analysis to obtain a second association result. If the first association result is an association failure and the second association result is an association success, network security analysis is performed on the target data stream.

7. The data flow correlation analysis method according to claim 6, characterized in that, After obtaining the second association result, it also includes: If the first target key value and the second target key value are different, based on the first target key value, the second target key value, and a preset same-index data association analysis process, the target data stream is subjected to same-index data association analysis with the first data stream pre-stored in the data storage table to obtain a third association result; if the first target key value and the second target key value are the same, the target data stream is inserted into the hit data storage table, and the first target key value and the second target key value are stored in a preset index table as the first index value and the second index value, respectively. The data association analysis process includes: inputting data streams with the same index value into a preset association filter for association calculation; the index value includes a first index value and a second index value; and the hit data storage table is a storage table used to store successfully associated data.

8. The data flow correlation analysis method according to claim 6, characterized in that, After obtaining the second association result, the method further includes: if the second association result is a successful association, storing the target data stream and the first data stream as a hit data pair in the hit data storage table.

9. The data flow correlation analysis method according to claim 6, characterized in that, After obtaining the first association result, it also includes: Update the version information corresponding to the version field in the target data stream to obtain the target version information; If the first association result is successful, the target data stream and the first data stream are stored as a hit data pair in the hit data storage table, and the target version information is added to the first data stream.

10. The data flow correlation analysis method according to claim 6, characterized in that, After determining the first and second target key values ​​of the target data stream, the process also includes: If the first target index value and the second target index value are the same, the target data stream is stored in the first data storage table based on the second target index value; The first target index value and the second target index value are stored in the preset index table, and a listener field is set in the target data stream. The listener is used to instruct the deletion of the target data stream upon receiving a timeout notification from the target data stream.

11. The data flow correlation analysis method according to claim 8, characterized in that, The method further includes: If the target data stream is a data stream to be deleted or a data stream that has timed out, and if a hit data pair including the target data stream is found in the hit data storage table, the hit data pair is deleted from the hit data storage table. If it is determined that the first target index value and the second target index value stored in the target data stream are the same, the target data stream is deleted from the first index table based on the second target index value stored in the target data stream, and the second target index value in the first index table is also deleted. If it is determined that the first target index value and the second target index value stored in the target data stream are different, the target data stream is deleted from the first index table based on the second target index value, and the second target index value in the first index table is also deleted; and the target data stream is deleted from the second index table based on the first target index value, and the first target index value in the second index table is also deleted.

12. A data processing apparatus, characterized in that, include: A key value determination unit is used to determine the first key value and the second key value corresponding to the first data stream; The first key value is used to store the first part of the data in the first data stream, and the second key value is used to store the second part of the data in the first data stream; The first data and index storage unit is used to store a first portion of the data in the first data stream in a preset first data storage table based on the first key value, and generate a first index corresponding to the first data stream; wherein the first index corresponds to the first key value. The second data and index storage unit is used to store the second part of the data in the first data stream in a preset second data storage table based on the second key value, and generate a second index corresponding to the first data stream; wherein the second index corresponds to the first key value; A data indexing processing unit is used to store the first index and the second index into a preset index table; The index table is used to find the first data stream corresponding to the first index based on the first target key value, and to find the first data stream corresponding to the second index based on the second target key value; the first target key value and the second target key value are determined based on the target data stream to be analyzed, the first target key value corresponds to the first index, and the second target key value corresponds to the second index.

13. A data correlation analysis device, characterized in that, After obtaining the index table storing the first index and the second index based on the apparatus as described in claim 12, the method further includes: The target key value determination unit is used to determine a first target key value and a second target key value of the target data stream to be analyzed. The first target key value corresponds to a first index, and the second target key value corresponds to a second index. The target data stream is network behavior data obtained from a network device. The first association analysis unit is used to find the first data stream corresponding to the first index from the index table based on the first target key value, perform a first association calculation on the first data stream corresponding to the first index and the target data stream to obtain first association calculation data, and input the first association calculation data into a preset association filter for association analysis to obtain a first association result; The second association analysis unit is used to find the first data stream corresponding to the second index from the index table based on the second target key value, perform a second association calculation on the first data stream corresponding to the second index and the target data stream to obtain second association calculation data, and input the second association calculation data into a preset association filter for association analysis to obtain a second association result; The security analysis unit is used to perform network security analysis on the target data stream when the first association result is an association failure and the second association result is an association success.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data processing method as described in any one of claims 1 to 5, or the data flow correlation analysis method as described in any one of claims 6 to 11.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the data processing method as described in any one of claims 1 to 5, or the data flow correlation analysis method as described in any one of claims 6 to 11.

16. A computer program product having executable instructions stored thereon, characterized in that, When executed by the processor, this instruction causes the processor to implement the data processing method as described in any one of claims 1 to 5, or the data flow correlation analysis method as described in any one of claims 6 to 11.

Images