Multi-datastream data processing methods, devices, and electronic equipment

Abstract

This invention provides a data processing method, apparatus, and electronic device based on multiple data streams, relating to the field of network security technology. The method includes: acquiring associated data; the associated data includes data to be processed and N-1 reference data, where the N data streams originate from N different servers; generating first hit data based on the associated data, storing the first hit data in a hit dataset, and determining a first index of the first hit data as its storage index in the hit dataset; the first hit data is successfully associated data; the first hit data includes at least one first data; storing the first index of the first hit data in a first index list corresponding to each first data, and determining the storage index of the first index in the first index list as a second index of the first data. This invention can improve data storage efficiency.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to a data processing method, apparatus, and electronic device based on multiple data streams. Background Technology

[0002] The rapid development of internet technology and the deepening digital transformation of various industries have brought tremendous convenience to people's lives and work. However, with the development of technology and the spread of knowledge, the methods and number of cyberattacks have also increased significantly, with various new attack methods emerging one after another, posing great challenges and pressure to security analysts and products.

[0003] Current data processing requires the use of large amounts of data and databases, and how to improve data storage efficiency is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] To address the problems in the prior art, embodiments of the present invention provide a data processing method, apparatus, and electronic device based on multiple data streams.

[0005] Specifically, the embodiments of the present invention provide the following technical solutions:

[0006] In a first aspect, embodiments of the present invention provide a data processing method based on multiple data streams, including:

[0007] Obtain associated data; the associated data includes data to be processed and N-1 reference data, where N is an integer greater than or equal to 3. The data to be processed comes from the first data stream among N data streams, and the N-1 reference data come from N-1 data streams other than the first data stream among the N data streams. The N data streams come from N different servers.

[0008] First hit data is generated based on the associated data, and the first hit data is stored in the hit dataset. The first index of the first hit data is determined as the storage index of the first hit data in the hit dataset. The first hit data is associated data that has been successfully associated. The first hit data includes at least one first data.

[0009] The first index of the first hit data is stored in a first index list corresponding to each of the first data, and the storage index of the first index in the first index list is determined as the second index of the first data; the first index list is used to record the storage index of all hit data related to the corresponding first data in the hit dataset.

[0010] Furthermore, the method also includes:

[0011] Determine the tag of the first data stream corresponding to the data to be processed;

[0012] The data to be processed is stored in the first reference dataset corresponding to the label of the first data stream, as the first reference data; the third index of the first reference data is the storage index of the first reference data in the first reference dataset.

[0013] Furthermore, the method also includes:

[0014] Determine the listener associated with the first reference data, the listener being used to monitor whether the first reference data has been deleted;

[0015] Add the listener to the listener list associated with the data to be processed.

[0016] Furthermore, the method also includes:

[0017] Retrieve data to be deleted;

[0018] From the list of listeners associated with the data to be deleted, obtain the listeners associated with the data to be deleted;

[0019] The second index list corresponding to the second reference data associated with the listener is traversed. For the fourth index in the traversed second index list, the second hit data corresponding to the fourth index is deleted from the hit dataset. The second index list is used to record the storage index of all hit data related to the second reference data in the hit dataset.

[0020] Determine the tag of the second data stream corresponding to the data to be deleted; based on the fifth index of the second reference data, delete the second reference data in the second reference dataset corresponding to the second data stream.

[0021] Furthermore, the method also includes:

[0022] In the list of listeners associated with the data to be deleted, delete the listener associated with the second reference data.

[0023] Furthermore, the method also includes:

[0024] Receive a timer timeout notification for the data to be deleted;

[0025] Iterate through the listeners in the list of listeners associated with the data to be deleted, and notify the listeners that the data to be deleted has timed out.

[0026] Further, deleting the second hit data corresponding to the fourth index from the hit dataset includes:

[0027] The fourth index is used to retrieve the second hit data from the hit dataset; the second hit data includes N second data items;

[0028] Determine whether the data to be processed among the N second data is not equal to the second reference data;

[0029] If the data to be processed is not equal to the second reference data, the fifth index of the data to be processed is used to delete the fifth index from the third index list corresponding to the data to be processed; the third index list is used to record the storage index of all hit data related to the data to be processed in the hit dataset.

[0030] Alternatively, if the data to be processed is equal to the second reference data, the second hit data is deleted from the hit dataset based on the fourth index.

[0031] Secondly, embodiments of the present invention provide a data processing method based on multiple data streams, including:

[0032] When data to be processed is detected, the tag of the first data stream corresponding to the data to be processed is determined;

[0033] From N data streams, obtain N-1 reference datasets corresponding to N-1 data streams other than the first data stream; N is an integer greater than or equal to 3, each reference dataset includes at least one piece of data, and the first data stream and the N-1 data streams come from N different servers respectively;

[0034] Traverse each reference data in the N-1 reference datasets. For each of the N-1 reference data in the N-1 reference datasets, perform association calculations between the data to be processed and the N-1 reference data to obtain the association calculation results.

[0035] If the association calculation result is successful, the data to be processed and the N-1 reference data are determined to be associated data.

[0036] Further, the step of performing association calculations on the data to be processed and the N-1 reference data to obtain association calculation results includes:

[0037] Obtain association conditions, which include the relationship between attribute information of at least two reference data;

[0038] Determine whether the data to be processed and the N-1 reference data meet the association conditions;

[0039] If the data to be processed and the N-1 reference data meet the association conditions, the association calculation result is determined to be a successful association.

[0040] Alternatively, if the data to be processed does not meet the association conditions with the N-1 reference data, the association calculation result is determined to be an association failure.

[0041] Furthermore, the method also includes:

[0042] The data to be processed is stored in the first reference dataset corresponding to the tag of the first data stream, and is used as the first reference data;

[0043] Identify the listener associated with the first reference data, the listener being used to monitor whether the first reference data has been deleted; add the listener to the listener list associated with the data to be processed.

[0044] Furthermore, the method also includes:

[0045] If the association calculation result is successful, first hit data is generated based on the data to be processed and the N-1 reference data, and the first hit data is stored in the hit dataset.

[0046] Furthermore, the method also includes:

[0047] Retrieve data to be deleted or data that has timed out;

[0048] From the list of listeners associated with the data to be deleted or the timed-out data, obtain the listeners associated with the data to be deleted or the timed-out data;

[0049] If, in the hit dataset, hit data related to the second reference data associated with the listener is found, the hit data related to the second reference data is deleted from the hit dataset.

[0050] Determine the tag of the second data stream corresponding to the data to be deleted or the timed-out data; delete the second reference data in the second reference dataset corresponding to the second data stream.

[0051] Furthermore, the method also includes:

[0052] In the list of listeners associated with the data to be deleted, delete the listener associated with the second reference data.

[0053] Furthermore, before determining the tag of the first data stream corresponding to the data to be processed, the method further includes:

[0054] Determine whether the data to be processed meets the filtering conditions; the filtering conditions include at least one preset data attribute information;

[0055] The step of determining the marker of the first data stream corresponding to the data to be processed includes:

[0056] If the data to be processed meets the filtering conditions, the label of the first data stream corresponding to the data to be processed is determined.

[0057] Furthermore, the method also includes:

[0058] If the association calculation result is successful, an alarm message is generated; the alarm message is used to indicate that there is a network security risk between the data to be processed and the N-1 reference data.

[0059] Thirdly, embodiments of the present invention also provide a data processing apparatus based on multiple data streams, comprising:

[0060] The first acquisition module is used to acquire associated data; the associated data includes data to be processed and N-1 reference data, where N is an integer greater than or equal to 3. The data to be processed comes from the first data stream among N data streams, and the N-1 reference data come from the N-1 data streams other than the first data stream among the N data streams. The N data streams come from N different servers.

[0061] A first processing module is configured to generate first hit data based on the associated data, store the first hit data in a hit dataset, and determine a first index of the first hit data as the storage index of the first hit data in the hit dataset; the first hit data is associated data that has been successfully associated; the first hit data includes at least one first data.

[0062] The first processing module is further configured to store the first index of the first hit data into a first index list corresponding to each of the first data, and determine the storage index of the first index in the first index list as the second index of the first data; the first index list is used to record the storage index of all hit data related to the corresponding first data in the hit dataset.

[0063] Fourthly, embodiments of the present invention also provide a data processing apparatus based on multiple data streams, comprising:

[0064] The detection module is used to determine the marker of the first data stream corresponding to the data to be processed when the data to be processed is detected.

[0065] The second acquisition module is used to acquire N-1 reference datasets corresponding to N-1 data streams other than the first data stream from N data streams; N is an integer greater than or equal to 3, each reference dataset includes at least one piece of data, and the first data stream and the N-1 data streams are respectively from N different servers;

[0066] The second processing module is used to traverse each reference data in the N-1 reference datasets, and for each of the N-1 reference data in the traversed N-1 reference datasets, to perform an association calculation between the data to be processed and the N-1 reference data to obtain an association calculation result; if the association calculation result is successful, the data to be processed and the N-1 reference data are determined to be associated data.

[0067] Fifthly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the data processing method based on multiple data streams as described in the first aspect, or implements the data processing method based on multiple data streams as described in the second aspect.

[0068] In a sixth aspect, embodiments of the present invention also provide a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method based on multiple data streams as described in the first aspect, or implements the data processing method based on multiple data streams as described in the second aspect.

[0069] In a seventh aspect, embodiments of the present invention also provide a computer program product having executable instructions stored thereon, which, when executed by a processor, cause the processor to implement the data processing method based on multiple data streams as described in the first aspect, or to implement the data processing method based on multiple data streams as described in the second aspect.

[0070] The data processing method, apparatus, and electronic device based on multiple data streams provided in this invention acquires associated data from different data streams, generates first hit data based on the associated data and stores it in the hit dataset, determines the first index of the first hit data as the storage index of the first hit data in the hit dataset, stores the first index of the first hit data in a first index list corresponding to each first data, and determines the storage index of the first index in the first index list as the second index of the first data, thereby realizing the storage of associated data. Based on the second index of the first data, the first data can be quickly searched and manipulated, thus improving data storage efficiency. Attached Figure Description

[0071] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0072] Figure 1 This is one of the flowcharts of a data processing method based on multiple data streams provided in an embodiment of the present invention;

[0073] Figure 2 This is a schematic diagram of the process for adding data stream data provided in an embodiment of the present invention;

[0074] Figure 3 This is a flowchart illustrating the process of adding successful correlation analysis data provided in an embodiment of the present invention;

[0075] Figure 4 This is one of the flowcharts for deleting data stream data provided in the embodiments of the present invention;

[0076] Figure 5 This is a schematic diagram of the data stream timeout processing provided in an embodiment of the present invention;

[0077] Figure 6 This is a schematic diagram of the process for deleting hit data provided in an embodiment of the present invention;

[0078] Figure 7 This is a second flowchart illustrating the data processing method based on multiple data streams provided in this embodiment of the invention.

[0079] Figure 8 This is a schematic diagram of the process for inserting data stream data provided in an embodiment of the present invention;

[0080] Figure 9 This is a flowchart illustrating the data flow of the associated table provided in an embodiment of the present invention;

[0081] Figure 10 This is the second schematic diagram of the process for deleting data stream data provided in the embodiments of the present invention;

[0082] Figure 11 This is a schematic diagram of the process for updating data stream data provided in an embodiment of the present invention;

[0083] Figure 12 This is a schematic diagram of the notification data stream timeout process provided in an embodiment of the present invention;

[0084] Figure 13 This is one of the structural schematic diagrams of a data processing device based on multiple data streams provided in an embodiment of the present invention;

[0085] Figure 14 This is a second schematic diagram of the structure of the data processing device based on multiple data streams provided in the embodiments of the present invention;

[0086] Figure 15 This is a schematic diagram of the physical structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0088] Figure 1 This is one of the flowcharts illustrating a data processing method based on multiple data streams provided in an embodiment of the present invention, such as... Figure 1 As shown, this multi-data-stream-based data processing method includes the following steps:

[0089] Step 101: Obtain associated data; the associated data includes data to be processed and N-1 reference data, where N is an integer greater than or equal to 3. The data to be processed comes from the first data stream among N data streams, and the N-1 reference data come from the N-1 data streams other than the first data stream. The N data streams come from N different servers.

[0090] It should be noted that the execution subject of the multi-data-stream-based data processing method provided in this embodiment of the invention can be a multi-data-stream-based data processing device, such as a network security detection device, a network security analysis device, or a network security analysis engine, etc., which are users of network data. The multi-data-stream-based data processing method provided in this embodiment of the invention is used for security analysis and detection of multiple data streams.

[0091] Optionally, the data to be processed may include, for example, logs, emails, programs, or files. In a network security detection scenario, the data to be processed constituting the associated data is related to the N-1 reference data, meaning the associated data poses a network security risk. If any one or more of the data to be processed and the N-1 reference data are marked as anomalous data, then the data to be processed and other data in the N-1 reference data may also be anomalous.

[0092] Optionally, the data to be processed comes from a first data stream among N data streams, and the N-1 reference data come from the N-1 data streams other than the first data stream, that is, the data to be processed and the N-1 reference data come from different data streams. These different data streams come from different servers.

[0093] Step 102: Generate first hit data based on the associated data, store the first hit data in the hit dataset, and determine the first index of the first hit data as the storage index of the first hit data in the hit dataset; the first hit data is successfully associated associated data; the first hit data includes at least one first data.

[0094] Optionally, the hit dataset (e.g., the hit data storage table matchedDataTable) stores multiple hit data (matchedData), each hit data corresponding to successfully associated data, and each hit data includes at least one piece of data.

[0095] Optionally, the first hit data is stored in the hit dataset, the storage index of the first hit data in the hit dataset is obtained, and the storage index is recorded as the first index of the first hit data.

[0096] Step 103: Store the first index of the first hit data into a first index list corresponding to each of the first data, and determine the storage index of the first index in the first index list as the second index of the first data; the first index list is used to record the storage index of all hit data related to the corresponding first data in the hit dataset.

[0097] Optionally, the first index list corresponding to the first data may be denoted as the matched data index list.

[0098] It should be noted that the storage index of the first hit data in the hit dataset is denoted as the first index of the first hit data.

[0099] The first hit data packet includes at least one first data; each first data corresponds to its own index list, which is used to record the storage index of all hit data related to the corresponding first data in the hit data packet.

[0100] For each piece of first data, the first index of the first hit data is stored in the first index list corresponding to the first data, and the storage index of the first index of the first hit data in the first index list is denoted as the second index of the first data.

[0101] Optionally, by querying the second index of the first data, the first index of the first hit data can be found. Based on the first index of the first hit data, the first hit data can be found in the hit data set. This enables the first data to be used to find the hit data that includes the first data, and then network security detection, network security analysis and other operations can be performed on the found hit data, thereby improving the efficiency of data storage, retrieval and processing.

[0102] The data processing method based on multiple data streams provided in this invention obtains associated data from different data streams, generates first hit data based on the associated data and stores it in the hit dataset, determines the first index of the first hit data as the storage index of the first hit data in the hit dataset, stores the first index of the first hit data in a first index list corresponding to each first data, and determines the storage index of the first index in the first index list as the second index of the first data, thereby realizing the storage of associated data. Based on the second index of the first data, the first data can be quickly searched and manipulated, thus improving data storage efficiency.

[0103] Optionally, Table 1 shows the definition and description of parameter N in the embodiments of the present invention, as shown in Table 1:

[0104] Table 1 Definition of parameter N

[0105] parameter definition N Number of data streams or flow tables, N>=3

[0106] Optionally, Table 2 shows the definition and description of the data to be processed (also called data stream data) in the embodiments of the present invention, as shown in Table 2:

[0107] Table 2 Data Flow Data Definitions

[0108]

[0109] The data to be processed is represented as data stream data. Data includes two parameters: a field value table (fieldValueMap) and a listener list (listenerList). The field value table (fieldValueMap) has a multi-level key-value structure, and the corresponding field value can be obtained by inputting the field name. The listener list (listenerList) stores the listeners registered on Data, which listen for data timeout deletion notifications.

[0110] Optionally, after receiving the data to be detected, the data is stored. The methods for storing the data to be detected may include:

[0111] Step 1: Determine the tag tableID of the first data stream corresponding to the data to be processed (Data); the tag of the first data stream is, for example, the flow table number of the first data stream.

[0112] Step 2: Store the data to be processed in the first reference dataset corresponding to the label of the first data stream, as the first reference data; the third index of the first reference data is the storage index of the first reference data in the first reference dataset.

[0113] Optionally, determine the listener associated with the first reference data, the listener is used to listen for whether the first reference data has been deleted; add the listener to the listener list associated with the data to be processed.

[0114] Optionally, in scenarios where data deletion is required, the methods for deleting data may include:

[0115] Step a: Obtain the data to be deleted;

[0116] Step b: Obtain the listeners associated with the data to be deleted from the list of listeners associated with the data to be deleted;

[0117] Step c: Traverse the second index list corresponding to the second reference data associated with the listener. For the fourth index in the traversed second index list, delete the second hit data corresponding to the fourth index in the hit dataset. The second index list is used to record the storage index of all hit data related to the second reference data in the hit dataset.

[0118] Step d: Determine the tag of the second data stream corresponding to the data to be deleted; based on the fifth index of the second reference data, delete the second reference data in the second reference dataset corresponding to the second data stream.

[0119] Optionally, the listener associated with the second reference data is deleted from the list of listeners associated with the data to be deleted.

[0120] Optionally, in step c, the method for deleting the second hit data corresponding to the fourth index in the hit dataset may include:

[0121] Step 1) Use the fourth index to obtain the second hit data from the hit dataset; the second hit data includes N second data points;

[0122] Step 2) Determine whether the data to be processed among the N second data is not equal to the second reference data;

[0123] Step 3) If the data to be processed is not equal to the second reference data, use the fifth index of the data to be processed to delete the fifth index from the third index list corresponding to the data to be processed; the third index list is used to record the storage index of all hit data related to the data to be processed in the hit dataset;

[0124] If the data to be processed is equal to the second reference data, the second hit data is deleted from the hit dataset based on the fourth index.

[0125] Optionally, upon receiving a timer timeout notification for the data, the data deletion can be implemented in the following ways:

[0126] Step 1: Receive a timer timeout notification for the data to be deleted;

[0127] Step 2: Iterate through the listeners in the list of listeners associated with the data to be deleted, and notify the listeners that the data to be deleted has timed out.

[0128] Optionally, Table 3 shows the definition and description of the first reference data (also called row data) in the embodiments of the present invention, Table 4 shows the definition and description of the hit data in the embodiments of the present invention, and Table 5 shows the definition and description of the association analysis calculation unit in the embodiments of the present invention. See Tables 3-5 for details.

[0129] Table 3 Row Data Definitions

[0130]

[0131] The first reference data is represented as RowData. RowData includes 5 parameters: data stream data (Data), row data listener, rowDataIndex (index of the row data storage table), matchedDataIndexList (list of indexes of the matched table storage table), and tableID (data stream number).

[0132] Table 4. Definition of Hit Data

[0133]

[0134] Table 5 Definition of Correlation Analysis Calculation Unit

[0135]

[0136] Optionally, Table 6 shows the definitions and descriptions of the data processing operations involved in the embodiments of the present invention:

[0137] Table 6 Data Processing Operations

[0138]

[0139] Figure 2 It is a schematic flow chart of adding data stream data provided by an embodiment of the present invention. As Figure 2 shown, the process of adding data stream data includes the following steps:

[0140] Step 201: Input data stream data data into the association analysis computing unit joinComputingUnit;

[0141] Step 202: Use the data classifier joinComputingUnit.dataClassifier of the input data to calculate the corresponding flow table number, and set it as tableID;

[0142] Step 203: Create a row data rowData of the data type RowData, set rowData.data = data, set the row data listener rowData.listener = joinComputingUnit, set rowData.tableID = tableID, and add the row data rowData to the listener list data.listenerList;

[0143] Step 204: In the row data storage table slot joinComputingUnit.rowDataTableSlot, obtain the corresponding row data storage table joinComputingUnit.rowDataTableSlot[tableID] through the flow table number tableID, set rowDataTable = joinComputingUnit.rowDataTableSlot[tableID], save the row data rowData to the row data storage table rowDataTable, set the save index as index, and set the row data storage table index rowData.rowDataIndex = index. After completion, the process ends.

[0144] Figure 3 It is a schematic flow chart of adding successfully associated analysis data provided by an embodiment of the present invention. As Figure 3 shown, the process of adding successfully associated analysis data includes the following steps:

[0145] Step 301: Input a set of row data <rowDataSlot[N](rowDataSlot[n](0 <= n < N)) that are successfully associated analysis into the association analysis computing unit joinComputingUnit, where each row dataSlot[n] is a row data with a flow table number of n>;

[0146] Step 302: Create a new matched data file named `matchedData` with the data type `MatchedData`, and set the row data slots `matchedData.rowDataSlot[0] = rowDataSlot[0]`, `matchedData.rowDataSlot[1] = rowDataSlot[1]`, ..., `matchedData.rowDataSlot[n] = rowDataSlot[n]`, where 0 <= n <N;

[0147] Step 303: Save the matched data (matchedData) to the matched data storage table (joinComputingUnit.matchedDataTable) and set the save index to index;

[0148] Step 304: Iterate through the row data slots matchedData.rowDataSlot;

[0149] Step 305: Determine if there are any untraversed row data rows in the row data slot matchedData.rowDataSlot; if so, proceed to step 306; otherwise, end the process.

[0150] Step 306: Add the index to the row data matched index list rowData.matchedDataIndexList, set the saved index to rowDataMatchedDataIndex, and set the associated matched row data index.

[0151] matchedData.rowDataMatchedDataIndexSlot[rowData.tableID]=

[0152] rowDataMatchedDataInde; after completion, proceed to step 305.

[0153] Figure 4 This is one of the flowcharts illustrating the deletion of data stream data provided in this embodiment of the invention, such as... Figure 4 As shown, the process for deleting data from a data stream includes the following steps:

[0154] Step 401: Input the data stream data to be deleted into the joinComputingUnit;

[0155] Step 402: Retrieve the associated row data rowData from data.listenerList and remove rowData from data.listenerList;

[0156] Step 403: Traverse the list of matched data indexes stored in the table: rowData.matchedDataIndexList;

[0157] Step 404: Determine if there are any unprocessed matched table storage indexes in rowData.matchedDataIndexList; if so, proceed to step 405; otherwise, proceed to step 406.

[0158] Step 405: Call the process to delete the matched data in the table, deleteMatchedData(matchedDataIndex, rowData). After completion, execute step 404.

[0159] Step 406: Obtain the corresponding row data storage table joinComputingUnit.rowDataTableSlot[rowData.tableID] from the row data storage table slot joinComputingUnit.rowDataTableSlot using the flow table number rowData.tableID, set rowDataTable = joinComputingUnit.rowDataTableSlot[rowData.tableID], and delete the row data rowData from the row data storage table rowDataTable using the row data storage table index rowData.rowDataIndex. After completion, the process ends.

[0160] Figure 5 This is a schematic diagram of the data stream timeout processing provided in an embodiment of the present invention, such as... Figure 5 As shown, the data timeout handling process for this data stream includes the following steps:

[0161] Step 501: The data stream receives a timer timeout notification;

[0162] Step 502: Iterate through all rows of data rowData in the listener list data.listenerList and notify each rowData data stream that the data timeout has occurred;

[0163] Step 503: The rowData notification in the rowData.listener indicates that the row data of the joinComputingUnit stored in the rowData listener has timed out, and the timed-out row data is entered as rowData.

[0164] Step 504: Traverse the list of matched data indexes stored in the table: rowData.matchedDataIndexList;

[0165] Step 505: Determine if there are any unprocessed matched table storage indexes in rowData.matchedDataIndexList; if so, proceed to step 506; otherwise, proceed to step 507.

[0166] Step 506: Call the process to delete the matched data in the table, deleteMatchedData(matchedDataIndex, rowData). After completion, execute step 505.

[0167] Step 507: Obtain the corresponding row data storage table joinComputingUnit.rowDataTableSlot[rowData.tableID] from the row data storage table slot joinComputingUnit.rowDataTableSlot using the flow table number rowData.tableID, set rowDataTable = joinComputingUnit.rowDataTableSlot[rowData.tableID], and delete the row data rowData from the row data storage table rowDataTable using the row data storage table index rowData.rowDataIndex. After completion, the process ends.

[0168] Figure 6 This is a schematic diagram of the process for deleting hit data provided in an embodiment of the present invention, such as... Figure 6 As shown, the process of deleting matched data (deleteMatchedData) includes the following steps:

[0169] Step 601: Call the joinComputingUnit to delete matched data using the deleteMatchedData(matchedDataIndex,fromRowData);

[0170] Step 602: Use the matchedDataIndex stored in the hit table to retrieve the corresponding matched data matchedData from the matched data storage table joinComputingUnit.matchedDataTable;

[0171] Step 603: Traverse the row data slots matchedData.rowDataSlot;

[0172] Step 604: Determine whether there are any untraversed row data rowData in matchedData.rowDataSlot and whether rowData is not equal to fromRowData. If yes, proceed to step 605; otherwise, proceed to step 606.

[0173] Step 605: Use the index matchedData.rowDataMatchedDataIndexSlot[rowData.tableID] to delete the corresponding data from the row data match table storage index list rowData.matchedDataIndexList. After completion, proceed to step 604.

[0174] Step 606: Use the index matchedDataIndex to delete the corresponding matched data from the matched data storage table joinComputingUnit.matchedDataTable. Once completed, the process ends.

[0175] Figure 7 This is a second flowchart illustrating the data processing method based on multiple data streams provided in this embodiment of the invention, as shown below. Figure 7 As shown, this multi-data-stream-based data processing method includes the following steps:

[0176] Step 701: When data to be processed is detected, determine the tag of the first data stream corresponding to the data to be processed;

[0177] It should be noted that the execution subject of the multi-data-stream-based data processing method provided in this embodiment of the invention can be a multi-data-stream-based data processing device, such as a network security detection device, a network security analysis device, or a network security analysis engine, etc., which are users of network data. The multi-data-stream-based data processing method provided in this embodiment of the invention is used for security analysis and detection of multiple data streams.

[0178] Optionally, the data to be processed may include, for example, logs, emails, programs, or files. The data to be processed comes from a first data stream. The first data stream is labeled, for example, by its flow table number.

[0179] Optionally, before determining the label of the first data stream corresponding to the data to be processed, it is determined whether the data to be processed meets the filtering conditions; the filtering conditions include at least one preset data attribute information; if the data to be processed meets the filtering conditions, the label of the first data stream corresponding to the data to be processed is determined to improve the utilization rate of processing resources.

[0180] Step 702: From the N data streams, obtain N-1 reference datasets corresponding to the N-1 data streams other than the first data stream; N is an integer greater than or equal to 3, each reference dataset includes at least one piece of data, and the first data stream and the N-1 data streams come from N different servers respectively.

[0181] Optionally, the data to be processed comes from the first data stream of N data streams. The N data streams each come from a different server. N-1 reference datasets are obtained from each of the N-1 data streams, with one reference dataset obtained from each data stream.

[0182] Step 703: Traverse each reference data in the N-1 reference datasets. For each of the N-1 reference data in the N-1 reference datasets, perform association calculations between the data to be processed and the N-1 reference data to obtain the association calculation results.

[0183] Optionally, the implementation of the association calculation may include: obtaining association conditions, the association conditions including the relationship between attribute information of at least two reference data; determining whether the data to be processed and the N-1 reference data meet the association conditions; if the data to be processed and the N-1 reference data meet the association conditions, determining the association calculation result as successful; or, if the data to be processed and the N-1 reference data do not meet the association conditions, determining the association calculation result as unsuccessful. The association calculation result is used to indicate whether the data to be processed and the N-1 reference data meet the association conditions.

[0184] Step 704: If the association calculation result is successful, determine that the data to be processed and the N-1 reference data are associated data.

[0185] It should be noted that in a network security detection scenario, the data to be processed constituting the associated data is related to the N-1 reference data, meaning that the associated data poses a network security risk. If any one or more data points among the data to be processed and the N-1 reference data are marked as anomalous, then the other data among the data to be processed and the N-1 reference data may also be anomalous.

[0186] Optionally, if the association calculation result is successful, an alarm message is generated; the alarm message is used to indicate that there is a network security risk between the data to be processed and the N-1 reference data.

[0187] The data processing method based on multiple data streams provided in this invention determines the first data stream containing the data to be processed and obtains N-1 reference datasets corresponding to N-1 data streams different from the first data stream. It iterates through each reference data in the N-1 reference datasets. For each of the N-1 reference data in the traversed datasets, it performs correlation analysis between the data to be processed and the N-1 reference data. If the correlation calculation result is successful, it determines that the data to be processed and the N-1 reference data are associated data. In a network security detection scenario, associated data may pose network security risks. If any one or more data in the data to be processed and the N-1 reference data are marked as abnormal data, then the data to be processed and other data in the N-1 reference data may also be abnormal. This enables security analysis and risk detection of the data to be detected, allowing for timely discovery of various potential security risks and attacks, and reducing network security risks.

[0188] Optionally, when data to be processed is detected, the data to be processed is stored in a first reference dataset corresponding to the tag of the first data stream, as the first reference data; a listener associated with the first reference data is determined, the listener being used to monitor whether the first reference data has been deleted; and the listener is added to the listener list associated with the data to be processed.

[0189] Optionally, if the association calculation result is successful, first hit data is generated based on the data to be processed and the N-1 reference data, and the first hit data is stored in the hit dataset.

[0190] Optionally, when it is necessary to delete data to be deleted or timed out data, the following steps are taken: Obtain the data to be deleted or timed out data; obtain the listener associated with the data to be deleted or timed out data from the list of listeners associated with the data to be deleted or timed out data; if a hit data related to the second reference data associated with the listener is found in the hit dataset, delete the hit data related to the second reference data in the hit dataset; determine the tag of the second data stream corresponding to the data to be deleted or timed out data; and delete the second reference data in the second reference dataset corresponding to the second data stream.

[0191] Optionally, the listener associated with the second reference data is deleted from the list of listeners associated with the data to be deleted.

[0192] Optionally, Table 7 shows the definitions and descriptions of the parameters, Table 8 shows the process involved in real-time multi-data stream correlation analysis, and Table 9 shows the process of inserting data into the data stream (insertStreamData(data)). See Tables 7-9 for details.

[0193] Table 7 Parameter Definitions

[0194]

[0195] Table 8. Processes Involved in Real-Time Multi-Data Stream Correlation Analysis

[0196]

[0197] Table 9 Insert Data Stream Data Flow

[0198]

[0199] Figure 8 This is a schematic diagram of the data insertion process provided in an embodiment of the present invention, such as... Figure 8 As shown, the process of inserting data into the data stream using insertStreamData(data) includes the following steps:

[0200] Step 801: Call insertStreamData(data) to input the data stream. <data>;

[0201] Step 802, Data Stream Classifier <streamclassifier>calculate <data>The corresponding table number is index;

[0202] Step 803: Define the joinTableSlot[N] as an array, and set joinTableSlot[index] = ... <data>, define the subscript n, where 0 <= n < N, and set the items in the array joinTableSlot[n] where n != index to joinTableSlot[n] = tableSlot[n];

[0203] Step 804: Define the associated table data slot joinRowSlot[N], with the data type being an array, define the table serial number tableIndex, and set tableIndex = 0;

[0204] Step 805: Call joinTableRows(joinTableSlot, joinRowSlot, tableIndex) to perform the calculation of table data association;

[0205] Step 806: The <data>Store in table slot <tableslot>The Middle <index>The index is in tableSlot[index];

[0206] Step 807, To <data>Register and delete listeners.

[0207] Figure 9 This is a flowchart illustrating the data flow of the associated table provided in an embodiment of the present invention, such as... Figure 9 As shown, the process of joining TableRows for this related table data includes the following steps:

[0208] Step 901: Call insertLeftKeyTable(data), input parameters<joinTableSlot,joinRowSlot,tableIndex> ;

[0209] Step 902: Define the current table `currentTable` and set `currentTable = joinTableSlot[tableIndex]`.

[0210] Step 903: Traverse the table <currenttable>For each row of data <row>Perform calculations;

[0211] Step 904 <currenttable>Is there any more data in the table? <row>If yes, proceed to step 905; otherwise, proceed to step 911.

[0212] Step 905: Set joinRowSlot[tableIndex] = <row>;

[0213] Step 906: Determine if tableIndex+1 is equal to N; if it is equal to N, proceed to step 907; if it is not equal to N, proceed to step 910.

[0214] Step 907: Transfer the data slots of the associated table <joinrowslot>All table data<joinRowSlot[0],joinRowSlot[1],…,joinRowSlot[N-1]> Input correlation filter <joinfilter>Perform correlation calculations;

[0215] Step 908, Judgment <joinfilter>Is the calculation result true? If <joinfilter>If the calculation result is true, proceed to step 909; if it is false, proceed to step 904.

[0216] Step 909,<joinRowSlot[0],joinRowSlot[1],…,joinRowSlot[N-1]> Insert into the hit table <matchedtable>After completion, proceed to step 904;

[0217] Step 910: Recursively call joinTableRows(joinTableSlot, joinRowSlot, tableIndex+1), and execute step 904 after the call is completed;

[0218] Step 911: This function call returns.

[0219] Figure 10 This is a second schematic diagram of the process for deleting data stream data provided in an embodiment of the present invention, such as... Figure 10 As shown, the process of deleting data from a data stream using deleteStreamData(data) includes the following steps:

[0220] Step 1001: Call deleteStreamData(data), and input the data stream to be deleted. <data>;

[0221] Step 1002, Cancel the process <data>Registered listeners can be deleted.

[0222] Step 1003, from <matchedtable>Check if there are any related hit rows; if so, delete the corresponding hit data.

[0223] Step 1004, Data Stream Classifier <streamclassifier>calculate <data>The corresponding table number is index;

[0224] Step 1005, from the slot <tableslot>The Middle <index>Delete from tableSlot[index] <data>The corresponding data;

[0225] Step 1006, if the table slot <tableslot>All tables<tableSlot[0],tableSlot[1],…,tableSlot[N-1]> If all values ​​are empty, then delete the current calculation unit.

[0226] Figure 11 This is a schematic diagram of the process for updating data stream data provided in an embodiment of the present invention, such as... Figure 11 As shown, the process of updating the data stream using updateStreamData(oldData, newData) includes:

[0227] Step 1101: Call updateStreamData(oldData, newData) and input the data stream to be updated.<oldData,newData> ;

[0228] Step 1102: Call the deleteStreamData(oldData) function to delete the data stream. <olddata>;

[0229] Step 1103: Call the insertStreamData(NewData) method to insert data. <newdata>.

[0230] Figure 12 This is a schematic diagram of the notification data stream timeout process provided in an embodiment of the present invention, such as... Figure 12 As shown, the process of the notification data stream timeout onStreamDataExpire(data) includes:

[0231] Step 1201: Call onStreamExpire(data) and input the timed-out data stream. <data>;

[0232] Step 1202, from <matchedtable>Check if there are any related hit rows; if so, delete the corresponding hit data.

[0233] Step 1203, Data Stream Classifier <streamclassifier>calculate <data>The corresponding table number is index;

[0234] Step 1204, from the slot <tableslot>The Middle <index>Delete from tableSlot[index] <data>The corresponding data;

[0235] Step 1205, if the table slot <tableslot>All tables<tableSlot[0],tableSlot[1],…,tableSlot[N-1]> If all values ​​are empty, then delete the current calculation unit.

[0236] The data processing apparatus based on multiple data streams provided by the present invention is described below. The data processing apparatus based on multiple data streams described below and the data processing method based on multiple data streams described above can be referred to in correspondence.

[0237] Figure 13 This is one of the structural schematic diagrams of a data processing device based on multiple data streams provided in an embodiment of the present invention, such as... Figure 13 As shown, the multi-data-stream-based data processing device 1300 includes: a first acquisition module 1301 and a first processing module 1302; wherein:

[0238] The first acquisition module 1301 is used to acquire associated data; the associated data includes data to be processed and N-1 reference data, where N is an integer greater than or equal to 3, the data to be processed comes from the first data stream among N data streams, and the N-1 reference data come from the N-1 data streams other than the first data stream among the N data streams, and the N data streams come from N different servers.

[0239] The first processing module 1302 is configured to generate first hit data based on the associated data, store the first hit data in the hit dataset, and determine the first index of the first hit data as the storage index of the first hit data in the hit dataset; the first hit data is associated data that has been successfully associated; the first hit data includes at least one first data.

[0240] The first processing module 1302 is further configured to store the first index of the first hit data into a first index list corresponding to each of the first data, and determine the storage index of the first index in the first index list as the second index of the first data; the first index list is used to record the storage index of all hit data related to the corresponding first data in the hit dataset.

[0241] The data processing apparatus based on multiple data streams provided in this embodiment of the invention acquires associated data from different data streams, generates first hit data based on the associated data and stores it in the hit dataset, determines the first index of the first hit data as the storage index of the first hit data in the hit dataset, stores the first index of the first hit data in a first index list corresponding to each first data, and determines the storage index of the first index in the first index list as the second index of the first data, thereby realizing the storage of associated data. Based on the second index of the first data, the first data can be quickly searched and manipulated, thus improving data storage efficiency.

[0242] Based on any of the above embodiments, the first processing module 1302 is further configured to: determine the tag of the first data stream corresponding to the data to be processed; store the data to be processed in a first reference dataset corresponding to the tag of the first data stream, as first reference data; and the third index of the first reference data is the storage index of the first reference data in the first reference dataset.

[0243] Based on any of the above embodiments, the first processing module 1302 is further configured to:

[0244] Determine the listener associated with the first reference data, the listener being used to monitor whether the first reference data has been deleted;

[0245] Add the listener to the listener list associated with the data to be processed.

[0246] Based on any of the above embodiments, the first processing module 1302 is further configured to:

[0247] Retrieve data to be deleted;

[0248] From the list of listeners associated with the data to be deleted, obtain the listeners associated with the data to be deleted;

[0249] The second index list corresponding to the second reference data associated with the listener is traversed. For the fourth index in the traversed second index list, the second hit data corresponding to the fourth index is deleted from the hit dataset. The second index list is used to record the storage index of all hit data related to the second reference data in the hit dataset.

[0250] Determine the tag of the second data stream corresponding to the data to be deleted; based on the fifth index of the second reference data, delete the second reference data in the second reference dataset corresponding to the second data stream.

[0251] Based on any of the above embodiments, the first processing module 1302 is further configured to: delete the listener associated with the second reference data from the listener list associated with the data to be deleted.

[0252] Based on any of the above embodiments, the first processing module 1302 is further configured to:

[0253] Receive a timer timeout notification for the data to be deleted;

[0254] Iterate through the listeners in the list of listeners associated with the data to be deleted, and notify the listeners that the data to be deleted has timed out.

[0255] Based on any of the above embodiments, the first processing module 1302 is further configured to:

[0256] The fourth index is used to retrieve the second hit data from the hit dataset; the second hit data includes N second data items;

[0257] Determine whether the data to be processed among the N second data is not equal to the second reference data;

[0258] If the data to be processed is not equal to the second reference data, the fifth index of the data to be processed is used to delete the fifth index from the third index list corresponding to the data to be processed; the third index list is used to record the storage index of all hit data related to the data to be processed in the hit dataset.

[0259] Alternatively, if the data to be processed is equal to the second reference data, the second hit data is deleted from the hit dataset based on the fourth index.

[0260] Figure 14 This is a second schematic diagram of the structure of a data processing device based on multiple data streams provided in an embodiment of the present invention, as shown below. Figure 14 As shown, the multi-data-stream-based data processing device 1400 includes: a detection module 1401, a second acquisition module 1402, and a second processing module 1403; wherein:

[0261] The detection module 1401 is used to determine the marker of the first data stream corresponding to the data to be processed when the data to be processed is detected;

[0262] The second acquisition module 1402 is used to acquire N-1 reference datasets corresponding to N-1 data streams other than the first data stream from N data streams; N is an integer greater than or equal to 3, each reference dataset includes at least one piece of data, and the first data stream and the N-1 data streams are respectively from N different servers;

[0263] The second processing module 1403 is used to traverse each reference data in the N-1 reference datasets, and for each of the N-1 reference data in the traversed N-1 reference datasets, to perform an association calculation between the data to be processed and the N-1 reference data to obtain an association calculation result; if the association calculation result is successful, the data to be processed and the N-1 reference data are determined to be associated data.

[0264] The data processing device based on multiple data streams provided in this invention determines the first data stream containing the data to be processed and obtains N-1 reference datasets corresponding to N-1 data streams different from the first data stream. It then iterates through each reference data in the N-1 reference datasets. For each of the N-1 reference data in the traversed datasets, it performs correlation analysis between the data to be processed and the N-1 reference data. If the correlation calculation result is successful, it determines that the data to be processed and the N-1 reference data are associated data. In a network security detection scenario, associated data may pose network security risks. If any one or more data in the data to be processed and the N-1 reference data are marked as abnormal data, then the data to be processed and other data in the N-1 reference data may also be abnormal. This enables security analysis and risk detection of the data to be detected, allowing for timely discovery of various potential security risks and attacks, and reducing network security risks.

[0265] Based on any of the above embodiments, the second processing module 1403 is specifically used for:

[0266] Obtain association conditions, which include the relationship between attribute information of at least two reference data;

[0267] Determine whether the data to be processed and the N-1 reference data meet the association conditions;

[0268] If the data to be processed and the N-1 reference data meet the association conditions, the association calculation result is determined to be a successful association.

[0269] Alternatively, if the data to be processed does not meet the association conditions with the N-1 reference data, the association calculation result is determined to be an association failure.

[0270] Based on any of the above embodiments, the second processing module 1403 is further configured to:

[0271] The data to be processed is stored in the first reference dataset corresponding to the tag of the first data stream, and is used as the first reference data;

[0272] Identify the listener associated with the first reference data, the listener being used to monitor whether the first reference data has been deleted; add the listener to the listener list associated with the data to be processed.

[0273] Based on any of the above embodiments, the second processing module 1403 is further configured to: generate first hit data based on the data to be processed and the N-1 reference data when the association calculation result is a successful association, and store the first hit data in the hit dataset.

[0274] Based on any of the above embodiments, the second processing module 1403 is further configured to:

[0275] Retrieve data to be deleted or data that has timed out;

[0276] From the list of listeners associated with the data to be deleted or the timed-out data, obtain the listeners associated with the data to be deleted or the timed-out data;

[0277] If, in the hit dataset, hit data related to the second reference data associated with the listener is found, the hit data related to the second reference data is deleted from the hit dataset.

[0278] Determine the tag of the second data stream corresponding to the data to be deleted or the timed-out data; delete the second reference data in the second reference dataset corresponding to the second data stream.

[0279] Based on any of the above embodiments, the second processing module 1403 is further configured to: delete the listener associated with the second reference data from the listener list associated with the data to be deleted.

[0280] Based on any of the above embodiments, the second processing module 1403 is further configured to:

[0281] Determine whether the data to be processed meets the filtering conditions; the filtering conditions include at least one preset data attribute information;

[0282] The step of determining the marker of the first data stream corresponding to the data to be processed includes:

[0283] If the data to be processed meets the filtering conditions, the label of the first data stream corresponding to the data to be processed is determined.

[0284] Based on any of the above embodiments, the second processing module 1403 is further configured to: generate alarm information when the association calculation result is successful; the alarm information is used to indicate that the data to be processed and the N-1 reference data have network security risks.

[0285] Figure 15 This is a schematic diagram of the physical structure of the electronic device provided in the embodiments of the present invention, such as... Figure 15 As shown, the electronic device 1500 includes a processor 1510, a communications interface 1520, a memory 1530, and a communication bus 1540. The processor 1510, communications interface 1520, and memory 1530 communicate with each other via the communication bus 1540. The processor 1510 can call logical instructions from the memory 1530 to execute the aforementioned multi-data-stream-based data processing method.

[0286] Furthermore, the logical instructions in the aforementioned memory 1530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0287] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the above-described data processing method based on multiple data streams.

[0288] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the above-described data processing method based on multiple data streams.

[0289] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0290] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0291] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / tableslot> < / data> < / index> < / tableslot> < / data> < / streamclassifier> < / matchedtable> < / data> < / newdata> < / olddata> < / tableslot> < / data> < / index> < / tableslot> < / data> < / streamclassifier> < / matchedtable> < / data> < / data> < / matchedtable> < / joinfilter> < / joinfilter> < / joinfilter> < / joinrowslot> < / row> < / row> < / currenttable> < / row> < / currenttable> < / data> < / index> < / tableslot> < / data> < / data> < / data> < / streamclassifier> < / data>

Claims

1. A data processing method based on multiple data streams, characterized in that, include: Obtain associated data; the associated data includes data to be processed and N-1 reference data, where N is an integer greater than or equal to 3. The data to be processed comes from the first data stream among N data streams, and the N-1 reference data come from the N-1 data streams other than the first data stream. The N data streams come from N different servers. The data to be processed is represented as data stream data, which includes a field value table and a listener list. The field value table is used to obtain the corresponding field value by inputting the field name. The listener list is used to store listeners registered to the data stream and to notify listeners of data timeout deletion. First hit data is generated based on the associated data, and the first hit data is stored in the hit dataset. The first index of the first hit data is determined as the storage index of the first hit data in the hit dataset. The first hit data is associated data that has been successfully associated. The first hit data includes at least one first data. The first index of the first hit data is stored in the first index list corresponding to each of the first data, and the storage index of the first index in the first index list is determined as the second index of the first data. The first index list is used to record the storage indexes of all hit data related to the corresponding first data in the hit dataset.

2. The data processing method based on multiple data streams according to claim 1, characterized in that, The method further includes: Determine the tag of the first data stream corresponding to the data to be processed; The data to be processed is stored in the first reference dataset corresponding to the label of the first data stream, as the first reference data; the third index of the first reference data is the storage index of the first reference data in the first reference dataset.

3. The data processing method based on multiple data streams according to claim 2, characterized in that, The method further includes: Determine the listener associated with the first reference data, the listener being used to monitor whether the first reference data has been deleted; Add the listener to the listener list associated with the data to be processed.

4. The data processing method based on multiple data streams according to claim 1, characterized in that, The method further includes: Retrieve data to be deleted; From the list of listeners associated with the data to be deleted, obtain the listeners associated with the data to be deleted; The second index list corresponding to the second reference data associated with the listener is traversed. For the fourth index in the traversed second index list, the second hit data corresponding to the fourth index is deleted from the hit dataset. The second index list is used to record the storage index of all hit data related to the second reference data in the hit dataset. Determine the tag of the second data stream corresponding to the data to be deleted; based on the fifth index of the second reference data, delete the second reference data in the second reference dataset corresponding to the second data stream.

5. The data processing method based on multiple data streams according to claim 4, characterized in that, The method further includes: In the list of listeners associated with the data to be deleted, delete the listener associated with the second reference data.

6. The data processing method based on multiple data streams according to claim 4, characterized in that, The method further includes: Receive a timer timeout notification for the data to be deleted; Iterate through the listeners in the list of listeners associated with the data to be deleted, and notify the listeners that the data to be deleted has timed out.

7. The data processing method based on multiple data streams according to any one of claims 4 to 6, characterized in that, Deleting the second hit data corresponding to the fourth index from the hit dataset includes: The fourth index is used to retrieve the second hit data from the hit dataset; the second hit data includes N second data items; Determine whether the data to be processed among the N second data is not equal to the second reference data; If the data to be processed is not equal to the second reference data, the fifth index of the data to be processed is used to delete the fifth index from the third index list corresponding to the data to be processed; the third index list is used to record the storage index of all hit data related to the data to be processed in the hit dataset. Alternatively, if the data to be processed is equal to the second reference data, the second hit data is deleted from the hit dataset based on the fourth index.

8. A data processing method based on multiple data streams, characterized in that, include: When data to be processed is detected, the tag of the first data stream corresponding to the data to be processed is determined; The data to be processed is represented as data stream data, which includes a field value table and a listener list; the field value table is used to obtain the corresponding field value by inputting the field name; The listener list is used to store listeners registered to the data stream and to notify listeners of data timeout deletion. From N data streams, obtain N-1 reference datasets corresponding to N-1 data streams other than the first data stream; N is an integer greater than or equal to 3, each reference dataset includes at least one piece of data, and the first data stream and the N-1 data streams come from N different servers respectively; Traverse each reference data in the N-1 reference datasets. For each of the N-1 reference data in the N-1 reference datasets, perform association calculations between the data to be processed and the N-1 reference data to obtain the association calculation results. If the association calculation result is successful, the data to be processed and the N-1 reference data are determined to be associated data.

9. The data processing method based on multiple data streams according to claim 8, characterized in that, The step of performing association calculations between the data to be processed and the N-1 reference data to obtain association calculation results includes: Obtain association conditions, which include the relationship between attribute information of at least two reference data; Determine whether the data to be processed and the N-1 reference data meet the association conditions; If the data to be processed and the N-1 reference data meet the association conditions, the association calculation result is determined to be a successful association. Alternatively, if the data to be processed does not meet the association conditions with the N-1 reference data, the association calculation result is determined to be an association failure.

10. The data processing method based on multiple data streams according to claim 8, characterized in that, The method further includes: The data to be processed is stored in the first reference dataset corresponding to the tag of the first data stream, and is used as the first reference data; Identify the listener associated with the first reference data, the listener being used to monitor whether the first reference data has been deleted; add the listener to the listener list associated with the data to be processed.

11. The data processing method based on multiple data streams according to claim 8, characterized in that, The method further includes: If the association calculation result is successful, first hit data is generated based on the data to be processed and the N-1 reference data, and the first hit data is stored in the hit dataset.

12. The data processing method based on multiple data streams according to claim 8, characterized in that, Before determining the tag of the first data stream corresponding to the data to be processed, the method further includes: Determine whether the data to be processed meets the filtering conditions; the filtering conditions include at least one preset data attribute information; The step of determining the marker of the first data stream corresponding to the data to be processed includes: If the data to be processed meets the filtering conditions, the label of the first data stream corresponding to the data to be processed is determined.

13. The data processing method based on multiple data streams according to claim 8, characterized in that, The method further includes: If the association calculation result is successful, an alarm message is generated; the alarm message is used to indicate that there is a network security risk between the data to be processed and the N-1 reference data.

14. A data processing device based on multiple data streams, characterized in that, include: The first acquisition module is used to acquire associated data; the associated data includes data to be processed and N-1 reference data, where N is an integer greater than or equal to 3. The data to be processed comes from the first data stream among N data streams, and the N-1 reference data come from the N-1 data streams other than the first data stream. The N data streams come from N different servers. The data to be processed is represented as data stream data, which includes a field value table and a listener list. The field value table is used to obtain the corresponding field value by inputting the field name. The listener list is used to store listeners registered to the data stream and to notify listeners of data timeout deletion. A first processing module is configured to generate first hit data based on the associated data, store the first hit data in a hit dataset, and determine a first index of the first hit data as the storage index of the first hit data in the hit dataset; the first hit data is associated data that has been successfully associated; the first hit data includes at least one first data. The first processing module is further configured to store the first index of the first hit data into a first index list corresponding to each of the first data, and determine the storage index of the first index in the first index list as the second index of the first data; The first index list is used to record the storage indexes of all hit data related to the corresponding first data in the hit dataset.

15. A data processing device based on multiple data streams, characterized in that, include: The detection module is used to determine the marker of the first data stream corresponding to the data to be processed when the data to be processed is detected. The data to be processed is represented as data stream data, which includes a field value table and a listener list; the field value table is used to obtain the corresponding field value by inputting the field name; The listener list is used to store listeners registered to the data stream and to notify listeners of data timeout deletion. The second acquisition module is used to acquire N-1 reference datasets corresponding to N-1 data streams other than the first data stream from N data streams; N is an integer greater than or equal to 3, each reference dataset includes at least one piece of data, and the first data stream and the N-1 data streams are respectively from N different servers; The second processing module is used to traverse each reference data in the N-1 reference datasets, and for each of the N-1 reference data in the traversed N-1 reference datasets, to perform association calculation between the data to be processed and the N-1 reference data to obtain the association calculation result; If the association calculation result is successful, the data to be processed and the N-1 reference data are determined to be associated data.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the data processing method based on multiple data streams as described in any one of claims 1 to 7, or implements the data processing method based on multiple data streams as described in any one of claims 8 to 13.

17. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the data processing method based on multiple data streams as described in any one of claims 1 to 7, or implements the data processing method based on multiple data streams as described in any one of claims 8 to 13.

18. A computer program product having executable instructions stored thereon, characterized in that, When executed by the processor, this instruction causes the processor to implement the data processing method based on multiple data streams as described in any one of claims 1 to 7, or to implement the data processing method based on multiple data streams as described in any one of claims 8 to 13.

Images