Data association analysis method, electronic equipment, readable medium and program product

Abstract

The invention provides a data association analysis method, which comprises the following steps of: designing an association analysis model according to a business demand; determining a minimum data set definition according to the correlation analysis model; according to the minimum dataset definition, extracting a plurality of datasets from each data center and transmitting the plurality of datasets to the master data center, the plurality of datasets including data entries and data elements thereof conforming to the minimum dataset definition; based on a correlation analysis model, performing time sequence correlation analysis on the transmitted data sets to generate alarm data containing data numbers; and according to the data number in the alarm data, re-extracting the data entry corresponding to the data number and all data elements thereof, and carrying out data fusion on the data entry and the data elements to generate complete alarm output. The invention further provides electronic equipment, a readable medium and a computer program product. Through simplified data transmission and efficient time sequence correlation analysis, the efficiency, flexibility and real-time performance of cross-data-center large-scale data analysis are remarkably improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of big data analysis, and in particular to a data association analysis method, an electronic device, a computer-readable medium, and a computer program product. Background Art

[0002] With the rapid development of enterprise informatization and digitalization, business data is experiencing explosive growth. Large conglomerates, in particular, need to store and manage massive amounts of data. To cope with this demand, enterprises often build multiple regional big data centers. These data centers ensure efficient storage and computation of business data. To improve data reliability, a "two-site, three-center" or "multi-site, multi-center" architecture is often adopted. These architectures, by deploying multiple data centers in different locations, enhance data backup and system disaster recovery capabilities.

[0003] Existing big data solutions typically address data storage and computing within a single data center, but they are limited in their ability to comprehensively analyze and compute business data across multiple data centers. While centralizing data from multiple data centers to process data in a single data center can address cross-center data analysis needs, it can also lead to the following issues: wasted data storage resources; and the need to consume significant dedicated network bandwidth, particularly during peak network times, potentially impacting normal business operations.

[0004] To more efficiently utilize the resources of multiple data centers and reduce bandwidth pressure for data transmission, a method and system are needed to minimize data transmission between multiple data centers while enabling comprehensive analysis and calculation of business data from each data center. Therefore, new solutions are urgently needed to more effectively process and analyze data across data centers. Summary of the Invention

[0005] Embodiments of the present disclosure provide a data association analysis method, an electronic device, a computer-readable medium, and a computer program product.

[0006] In a first aspect, an embodiment of the present disclosure provides a data association analysis method, comprising: designing an association analysis model according to business needs; determining a minimized data set definition according to the association analysis model; extracting multiple data sets from each data center and transmitting them to a master data center according to the minimized data set definition, wherein the multiple data sets contain data entries and data elements thereof that conform to the minimized data set definition; based on the association analysis model, performing time series association analysis on the transmitted multiple data sets to generate alarm data containing data numbers; and re-extracting the data entries and all data elements thereof corresponding to the data numbers in the alarm data, and performing data fusion on them to generate a complete alarm output.

[0007] In some embodiments, an association analysis model is designed based on business needs, including: setting data set selection conditions that meet the business needs; based on the business needs, setting business logic for multiple data sets that meet the data set selection conditions; based on the set data set selection conditions, business logic and their execution order, generating a directed acyclic graph model description language for the association analysis model.

[0008] In some embodiments, based on the business needs, business logic is set for multiple data sets that meet the data set selection conditions, including: based on the business needs, setting the screening conditions for the multiple data sets; based on the business needs, setting the association conditions between the multiple data sets; based on the business needs, selecting the time window that needs to be counted and setting the statistical function.

[0009] In some embodiments, before designing an association analysis model according to business needs, the process includes: performing standardization processing on the multiple data sets to generate a standardized data set.

[0010] In some embodiments, determining a minimized dataset definition based on the association analysis model includes: selecting multiple datasets that meet the dataset selection criteria; determining the data entries and data elements required for the business logic from the selected datasets; and generating a minimized dataset definition based on the determined data entries and data elements.

[0011] In some embodiments, determining the data entries and data elements required for the business logic from the selected data set includes: determining the filtered data entries that meet the filtering conditions from the selected data set; determining the associated data elements required for the associated conditions from the filtered data entries; and determining the statistical data elements required for the statistical function from the filtered data entries.

[0012] In some embodiments, according to the minimized dataset definition, multiple datasets are extracted from each data center and transmitted to the master data center, including: sending the minimized dataset definition to the data proxy node of each data center; the data proxy node extracts the multiple datasets from the local data bus according to the minimized dataset definition; the data proxy node transmits the extracted multiple datasets to the message bus of the master data center in real time.

[0013] In some embodiments, based on the correlation analysis model, a time series correlation analysis is performed on the transmitted multiple data sets to generate alarm data containing data numbers, including: receiving the transmitted multiple data sets from the message bus; in the master control data center, based on the business logic set by the correlation analysis model, a time series correlation analysis is performed on the received multiple data sets; and based on the results of the time series correlation analysis, alarm data containing data numbers is generated.

[0014] In some embodiments, in the master control data center, based on the business logic set by the association analysis model, a time series association analysis is performed on the received multiple data sets, including: associating the data entries in the received multiple data sets according to the association conditions; and within the time window, performing a time series association analysis on the associated data entries based on the statistical function.

[0015] In some embodiments, based on the results of the time series correlation analysis, alarm data containing data numbers are generated, including: treating the data entries in the time series correlation analysis results that meet the set range of the statistical function as a group of alarm events; generating an alarm correlation identifier for each group of alarm events; outputting the data numbers of the data entries in each group of alarm events as alarm data, and recording all alarm data to form an alarm data log.

[0016] In some embodiments, based on the data number in the alarm data, the data entry corresponding to the data number and all its data elements are re-extracted, and data fusion is performed on them to generate a complete alarm output, including: based on the data number in the alarm data log, the corresponding data entry and all its related data elements are transmitted to the master data center through the data proxy node of each data center; in the master data center, based on the alarm association identifier, data fusion is performed on the received data entries to generate a complete alarm output.

[0017] In a second aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a memory on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the data association analysis method described in the first aspect of the embodiment of the present disclosure.

[0018] In a third aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon. When the computer program is executed by a processor, the data association analysis method described in the first aspect of the embodiment of the present disclosure is implemented.

[0019] In a fourth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the data association analysis method described in the first aspect of the embodiment of the present disclosure.

[0020] This disclosure specifically relates to a cross-data center data association analysis method. This method predefines data screening and association conditions through an association analysis model, dynamically generates a minimized data set, and implements efficient alarm generation and data fusion based on time-series association analysis. This method reduces the amount of data transmitted across data centers, improves analysis timeliness and flexibility, and enables real-time alarms and rapid response to complex events. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flow chart of a data association analysis method according to an embodiment of the present disclosure;

[0022] Figure 2 is a schematic diagram of the composition of an electronic device according to an embodiment of the present disclosure;

[0023] Figure 3 is a block diagram of a data association analysis system according to an embodiment of the present disclosure;

[0024] Figure 4 An exemplary directed acyclic graph of a correlation analysis model between VPN login logs and server operation logs is shown. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the technical solution of the present disclosure, the technical solution of the present disclosure is described in detail below with reference to the accompanying drawings.

[0026] Example embodiments will be described more fully hereinafter with reference to the accompanying drawings, but the example embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope of this disclosure to those skilled in the art.

[0027] In the absence of conflict, the various embodiments of the present disclosure and the various features therein may be combined with each other.

[0028] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0029] The terms used herein are used only to describe specific embodiments and are not intended to limit the present disclosure. As used herein, the singular forms "a," "an," and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that when the terms "comprising" and / or "made of" are used in this specification, the presence of the features, wholes, steps, operations, elements, and / or components is specified, but the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or groups thereof is not excluded.

[0030] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and will not be interpreted as having an idealized or overly formal meaning unless expressly defined as such herein.

[0031] In this disclosure, unless otherwise specified, the following technical terms should be understood as follows:

[0032] Data proxy nodes are nodes deployed in various data centers to perform data filtering, extraction, and transmission functions.

[0033] The message bus is a core component of the master data center, used for caching and transmitting data. Data from each data center is transmitted to the message bus via data proxy nodes for subsequent analysis and computation. As a bridge for data exchange, the message bus supports real-time data management and processing.

[0034] Time series correlation analysis refers to a technology that analyzes data based on its temporal sequence and relationship. It is particularly suitable for detecting the correlation between a series of specific events within a certain time window, thereby generating alarms or warnings.

[0035] JSON (JavaScript Object Notation) is a lightweight data exchange format used to represent data in a structured manner. It uses key-value pairs, supports nested structures, and is easy for humans to read and machines to parse.

[0036] This disclosure relates to a real-time data correlation analysis method across data centers. By minimizing dataset extraction, dynamic data transmission, and time-series correlation analysis, this method enables efficient correlation and alert generation for data from diverse sources in a large-scale distributed environment. The method is suitable for scenarios requiring large-scale data processing and real-time analysis, such as network security monitoring and system log analysis.

[0037] Figure 1 This is a flow chart of a data association analysis method according to an embodiment of the present disclosure.

[0038] First, refer to Figure 1 , the present disclosure provides a data association analysis method, including:

[0039] S11. Design association analysis model based on business needs;

[0040] S12. Determine a minimum data set definition based on the association analysis model;

[0041] S13. Extracting multiple data sets from each data center and transmitting them to a master data center according to the minimized data set definition, wherein the multiple data sets include data entries and data elements that conform to the minimized data set definition;

[0042] S14. Based on the correlation analysis model, perform time series correlation analysis on the transmitted multiple data sets to generate alarm data including data numbers;

[0043] S15. Re-extract the data entry corresponding to the data number and all its data elements according to the data number in the alarm data, and perform data fusion on them to generate a complete alarm output.

[0044] In some embodiments, based on business needs, a correlation analysis model is designed, including:

[0045] Set the data set selection criteria that meet the business requirements;

[0046] Based on the business requirements, setting business logic for multiple data sets that meet the data set selection conditions;

[0047] Based on the set data set selection conditions, business logic and execution order thereof, a directed acyclic graph model description language of the association analysis model is generated.

[0048] In some embodiments, based on the business requirements, setting business logic for multiple data sets that meet the data set selection criteria includes:

[0049] Based on the business requirements, setting screening conditions for the multiple data sets;

[0050] Based on the business requirements, setting association conditions between the multiple data sets;

[0051] Based on the business requirements, select the time window that needs statistics and set the statistical function.

[0052] In the disclosed embodiments, statistical functions may include: counting a field within a specified time range; performing deduplication counting on a field within a specified time range; summing, averaging, maximizing, or minimizing a numeric field within a specified time range; determining whether the statistical result falls within the range set by the statistical function, and generating an alert based on the determination result. However, those skilled in the art will appreciate that statistical functions can be flexibly configured based on the business needs of different scenarios.

[0053] In an embodiment of the present disclosure, the directed acyclic graph model description language of the association analysis model can be expressed based on JSON, wherein operator is used to represent node objects (for example, task type and operation content, such as filtering, association, statistics, etc.), and connection is used to represent data flow and dependency relationships between nodes (for example, a sequential relationship or data transfer relationship from one node to another).

[0054] Those skilled in the art will appreciate that JSON isn't the only model description language. It's well-suited for describing DAGs because it provides a clear and structured representation of nodes and dependencies. Besides JSON, DAGs can also be represented using languages ​​like YAML, GraphML, and DOT, each with its own specific applications. JSON's strengths lie in its simplicity and widespread support, making it particularly well-suited for integration with various programming languages ​​and systems.

[0055] In some embodiments, before designing an association analysis model based on business requirements, the following steps are included:

[0056] The multiple data sets are standardized to generate a standardized data set.

[0057] In the embodiment of the present disclosure, in order to ensure that data across data centers can be effectively integrated and analyzed, the data needs to be uniformly standardized before performing cross-data center correlation analysis. A unified name and data type are set for each data element in each data type involved in the analysis to ensure that the data structure of different data sources is consistent. The data number uses a combination of "data type number", "data center number" and "data hash value" to uniquely identify each data entry. The data time identifier uses a unified format of "YYYY-MM-dd HH:mm:ss.S" to identify the data time to ensure that the timestamp format is consistent in different data centers. Business data of different formats, different sources and different structures are cleaned and preprocessed in real time according to a unified data specification to generate standardized data with a unified format. The standardized data is then transmitted to the message bus for subsequent real-time correlation analysis.

[0058] In some embodiments, determining a minimized data set definition based on the association analysis model includes:

[0059] selecting a plurality of data sets that meet the data set selection criteria;

[0060] From the selected data set, determine the data entries and data elements required for the business logic;

[0061] Based on the determined data items and data elements, a minimized data set definition is generated.

[0062] In some embodiments, determining data entries and data elements required for the business logic from the selected data set includes:

[0063] Determining, from the selected data set, filtered data entries that meet the filtering criteria;

[0064] Determining, from the filtered data entries, associated data elements required by the associated condition;

[0065] determining statistical data elements required by the statistical function from the filtered data entries;

[0066] In an embodiment of the present disclosure, generating a minimized data set definition based on the determined data entries and data elements includes:

[0067] A minimized data set definition is generated based on the filtered data items, the associated data elements, and the statistical data elements.

[0068] In an embodiment of the present disclosure, the definition of a minimized dynamic data set is determined based on the data elements to be applied in the definition of the association analysis model. Based on the screening conditions, association conditions, and statistical conditions set in the model, it is ensured that only the necessary data elements required for analysis are included, and data entries that do not meet the conditions are eliminated based on the screening conditions. When the association analysis model is modified, the data set definition is automatically regenerated and updated based on the new model to ensure that the data set always remains minimized and dynamic. The minimized data set definition does not perform actual data screening operations, but rather provides a rule definition for specific data screening and analysis in subsequent steps.

[0069] The minimized data set only contains the necessary data elements required for the association analysis model and excludes redundant information to reduce data transmission volume, reduce network bandwidth usage and improve processing efficiency.

[0070] In some embodiments, extracting multiple data sets from each data center and transmitting them to a master data center according to the minimized data set definition includes:

[0071] Sending the minimized data set definition to the data proxy nodes in each data center;

[0072] The data proxy node extracts the multiple data sets from the local data bus according to the minimized data set definition;

[0073] The data proxy node transmits the extracted multiple data sets to the message bus of the master control data center in real time.

[0074] In the embodiment of the present disclosure, the data proxy node monitors and extracts data entries and data elements that meet the definition of the minimized data set in real time, and transmits them to the message bus of the master data center in real time. In this way, the transmitted data only contains necessary fields, avoids redundant information, and reduces network bandwidth usage. The message bus of the master data center is used as a cache to temporarily store data sets from each data center to provide support for subsequent data analysis and alarm generation. In this way, each data center can quickly transmit the extracted qualified data set to the master data center for subsequent analysis and alarm generation. The data proxy node monitors and extracts qualified log data in real time and transmits it to the message bus. The message bus caches and temporarily stores data to support further processing and analysis by the master data center.

[0075] In summary, the data proxy nodes in each data center filter the data in real time according to the minimum data set definition, extract only the necessary fields, and transmit the qualified data sets to the message bus of the master data center to ensure the real-time and efficient data processing.

[0076] In some embodiments, based on the correlation analysis model, performing time series correlation analysis on the transmitted multiple data sets to generate alarm data including data numbers includes:

[0077] receiving the transmitted plurality of data sets from the message bus;

[0078] In the master data center, based on the business logic set by the correlation analysis model, a time series correlation analysis is performed on the received multiple data sets;

[0079] Based on the results of the time series correlation analysis, alarm data including data numbers is generated.

[0080] In some embodiments, in the master data center, based on the business logic set by the correlation analysis model, a time series correlation analysis is performed on the received multiple data sets, including:

[0081] Associating the data items in the received multiple data sets according to the association condition;

[0082] Within the time window, a time series correlation analysis is performed on the correlated data entries based on the statistical function.

[0083] In some embodiments, generating alarm data including a data number based on the result of the time series correlation analysis includes:

[0084] Taking the data entries in the time series correlation analysis result that are within the set range of the statistical function as a group of alarm events;

[0085] Generate an alarm correlation identifier for each group of alarm events;

[0086] The data numbers of the data items in each group of alarm events are output as alarm data, and all alarm data are recorded to form an alarm data log.

[0087] In the embodiment of the present disclosure, the alarm correlation identifier marks a group of related data entries that meet the conditions, and different groups of events generate different alarm correlation identifiers. The composition of a group of events depends on the matching of the correlation conditions, statistical functions and time windows.

[0088] In the disclosed embodiment, the time series correlation analysis calculation is mainly performed based on the business logic such as filtering, correlation, and statistics set in the correlation analysis model to generate alarm data and record the data numbers that meet the alarm conditions. It should be noted that the calculation of the time series correlation analysis depends on the design of the correlation analysis model. Among the multiple business logics designed, if further filtering is still required after correlation, it needs to be executed in this step; if the filtering condition is pre-set, the filtering is completed by each data proxy node in the data extraction stage, and there is no need to repeat the filtering after correlation.

[0089] In the embodiment of the present disclosure, according to actual needs, the alarm data may further include other data elements of the data entry.

[0090] In some embodiments, based on the data number in the alarm data, re-extracting the data entry corresponding to the data number and all its data elements, and performing data fusion on them to generate a complete alarm output, includes:

[0091] According to the data number in the alarm data log, the corresponding data entry and all related data elements thereof are transmitted to the master control data center through the data proxy node of each data center;

[0092] In the master control data center, data fusion is performed on each received data item according to the alarm association identifier to generate a complete alarm output.

[0093] In the embodiment of the present disclosure, according to the data number generated by analysis and calculation, the corresponding data is filtered out from each data center to the master control data center, and data fusion is performed to output alarm data.

[0094] Those skilled in the art will appreciate that while the amount of data transmitted during this phase is relatively large, it is limited to the data items associated with the alarm data number generated in the previous step, significantly reducing the need for full data transmission. While the original transmission of tens of thousands or even hundreds of millions of items of data is reduced to hundreds or fewer, this process simplifies the transmission of complete data elements, enabling the system to present comprehensive alarm details, facilitating subsequent security analysis and response.

[0095] Figure 2 It is a schematic diagram of the composition of an electronic device according to an embodiment of the present disclosure.

[0096] Secondly, refer to Figure 2 , an embodiment of the present disclosure provides an electronic device, comprising:

[0097] One or more processors 201;

[0098] A memory 202 storing one or more programs, which, when executed by one or more processors, enables the one or more processors to implement the data association analysis method according to the first aspect of the embodiment of the present disclosure;

[0099] One or more I / O interfaces 203 are connected between the processor and the memory and are configured to implement information exchange between the processor and the memory.

[0100] Among them, the processor 201 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 202 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) 203 is connected between the processor 201 and the memory 202, and can realize information interaction between the processor 201 and the memory 202, including but not limited to a data bus (Bus), etc.

[0101] In some embodiments, the processor 201 , the memory 202 , and the I / O interface 203 are connected to each other via a bus 204 , and further connected to other components of the computing device.

[0102] In a third aspect, an embodiment of the present disclosure provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data association analysis method described in the first aspect of the embodiment of the present disclosure.

[0103] In a fourth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the data association analysis method described in the first aspect of the embodiment of the present disclosure.

[0104] Figure 3 4 is a block diagram of a data association analysis system according to an embodiment of the present disclosure.

[0105] Fifth, refer to Figure 3 , an embodiment of the present disclosure provides a data association analysis system, including:

[0106] The association analysis model design module 301 is configured to design an association analysis model according to business requirements;

[0107] A minimized data set definition module 302 is configured to determine a minimized data set definition according to the association analysis model;

[0108] a data acquisition module 303 configured to extract a plurality of data sets from each data center according to the minimized data set definition and transmit the data sets to a master data center, wherein the plurality of data sets include data entries and data elements that conform to the minimized data set definition;

[0109] The time series correlation analysis module 304 is configured to perform time series correlation analysis on the transmitted multiple data sets based on the correlation analysis model to generate alarm data including data numbers;

[0110] The data fusion module 305 is configured to re-extract the data entry and all its data elements corresponding to the data number according to the data number in the alarm data, and perform data fusion on them to generate a complete alarm output.

[0111] In some embodiments, the association analysis model design module 301 further includes:

[0112] A data set selection condition setting module, configured to set data set selection conditions that meet the business requirements;

[0113] a business logic setting module configured to set business logic for a plurality of data sets that meet the data set selection conditions based on the business requirements;

[0114] The association analysis model generation module is configured to generate a directed acyclic graph model description language of the association analysis model based on the set data set selection conditions, business logic and execution order.

[0115] In some embodiments, the business logic setting module includes:

[0116] A screening condition setting module, configured to set screening conditions for the plurality of data sets based on the business requirements;

[0117] an association condition setting module, configured to set association conditions between the plurality of data sets based on the business requirements;

[0118] The statistical function setting module is configured to select a time window that needs to be counted and set a statistical function based on the business needs.

[0119] In some embodiments, the data association analysis system further includes:

[0120] The data set standardization module is configured to perform standardization processing on the multiple data sets to generate a standardized data set.

[0121] In some embodiments, the minimized data set definition module 302 includes:

[0122] a data set selection module configured to select a plurality of data sets that meet the data set selection condition;

[0123] a data determination module configured to determine data entries and data elements required by the business logic from the selected data set;

[0124] The data set definition generation module is configured to generate a minimized data set definition based on the determined data items and data elements.

[0125] In some embodiments, the data determination module includes:

[0126] a screening data determination module configured to determine, from the selected data set, screened data entries that meet the screening conditions;

[0127] an associated data determination module configured to determine associated data elements required by the association condition from the filtered data entries;

[0128] The statistical data determination module is configured to determine the statistical data elements required by the statistical function from the filtered data entries.

[0129] In some embodiments, the data acquisition module 303 includes:

[0130] A module below the minimized data set definition is configured to send the minimized data set definition to the data proxy nodes of each data center;

[0131] A data extraction module, configured for the data proxy node to extract the multiple data sets from a local data bus according to the minimized data set definition;

[0132] The data transmission module is configured so that the data proxy node transmits the extracted multiple data sets to the message bus of the master control data center in real time.

[0133] In some embodiments, the time series correlation analysis module 304 includes:

[0134] a data receiving module configured to receive the transmitted multiple data sets from the message bus;

[0135] an analysis module configured to perform, in the master control data center, a time series correlation analysis on the received multiple data sets based on the business logic set by the correlation analysis model;

[0136] The alarm data generation module is configured to generate alarm data including data numbers according to the results of the time series correlation analysis.

[0137] In some embodiments, the analysis module comprises:

[0138] Associating the data items in the received multiple data sets according to the association condition;

[0139] Within the time window, a time series correlation analysis is performed on the correlated data entries based on the statistical function.

[0140] In some embodiments, the alarm data generation module includes:

[0141] Taking the data entries in the time series correlation analysis result that are within the set range of the statistical function as a group of alarm events;

[0142] Generate an alarm correlation identifier for each group of alarm events;

[0143] The data numbers of the data items in each group of alarm events are output as alarm data, and all alarm data are recorded to form an alarm data log.

[0144] In some embodiments, the data fusion module 305 includes:

[0145] a data retransmission module configured to transmit the corresponding data entry and all related data elements thereof to the master control data center through the data proxy node of each data center according to the data number in the alarm data log;

[0146] The data aggregation module is configured to perform data fusion on each received data item in the master control data center according to the alarm association identifier to generate a complete alarm output.

[0147] In order to enable those skilled in the art to more clearly understand the technical solutions provided by the embodiments of the present disclosure, the data association analysis method provided by the embodiments of the present disclosure is described in detail below through specific examples:

[0148] Example 1

[0149] VPN login logs and server operation logs are collected in Data Centers A and B. Analysis reveals that a user attempted to log in to the VPN more than five times, successfully logged in, and then performed high-risk server operations in Data Center B.

[0150] Step 1: Business data standardization: To conduct effective data analysis, these two types of log data need to be standardized to ensure that the data in each data set has a unified format and structure.

[0151] Define VPN login log data specifications, including data elements such as "data number, data time, client IP, user account, VPN service IP, VPN service port, login status, and source IP." Define server operation log specifications, including data elements such as "data number, data time, client IP, server IP, server name, user name, connection method, executed command, operation content, operation type, and operation result." Business data standardization ensures that data from different sources is formatted uniformly, facilitating subsequent correlation analysis.

[0152] In the embodiments of the present disclosure, a "dataset" refers to a collection of data with the same data structure and data elements. For example, the aforementioned VPN login log table can be called a VPN login log dataset, and the server operation log table can be called a server operation log dataset. Each row in the dataset represents a "data entry." Each data entry corresponds to an operation record, such as each VPN login attempt or each server operation. Each column in a data entry is called a "data element," which represents an attribute of a record. For example, "data number," "data time," "client IP," etc. are all data elements that describe the specific information in the data entry.

[0153] For example, the VPN login log dataset is as follows:

[0154]

[0155] An example of a server operation log dataset is as follows:

[0156]

[0157] In practice, a VPN login attempt is first made, recorded in the VPN login log. If the login attempt succeeds (possibly after multiple failures), the user may perform high-risk operations on the server, which are recorded in the server operation log. Correlation analysis can link these two types of logs to identify potentially risky user behavior. For example, a user may attempt to log in to the VPN multiple times and finally succeed, then connect to the server and execute high-risk commands within a short period of time.

[0158] Step 2: Design the association analysis model. In this step, design the association analysis model based on specific business needs. This model defines the relationship between business data through data selection, screening, association, and statistics, and generates a directed acyclic graph model description language (JSON format) for subsequent real-time analysis. The specific logic is as follows:

[0159] 1) Select a standardized dataset. Based on your business needs, select standardized datasets from different data centers. The datasets have already been preprocessed in step 1 and have uniform specifications. For example, in this example, we select VPN login logs from data center A and server operation logs from data center B.

[0160] 2) Set filtering criteria for each selected dataset. Set filtering criteria for the selected dataset to extract the necessary data entries that meet business requirements. For this example, the following filtering example filters out data entries with "Login Status = Failed" from the VPN login log, focusing only on unsuccessful login attempts. Also, filter out data entries with "Operation Type = High-Risk Operation" from the server operation log, focusing on potentially risky operations.

[0161] 3) Set association conditions between datasets. Set association conditions for the filtered data entries to logically associate records between different datasets. Association conditions are typically based on common fields in the business data, for example, the client IP address in a VPN login log equals the client IP address in a server operation log. The association operation does not merge entries from the two datasets; instead, it maintains the separate datasets and matches corresponding entries based on the association logic.

[0162] 4) Set statistical conditions based on business needs. These conditions are used to perform statistical analysis within a specific time window. For example, consider a 5-minute window and a statistical function that generates an alert identifying a potentially risky behavior if the same client IP address has three or more VPN login failures and at least one high-risk operation within the 5-minute window.

[0163] 5) Generate a model description language for the corresponding directed acyclic graph. Based on the above selection conditions, screening conditions, association conditions and statistical conditions, a directed acyclic graph (DAG) model is generated, and its logical process is described in JSON format. The model description language fully presents the data flow and processing logic from data selection, screening, association and statistics, providing a clear rule basis for subsequent real-time analysis. The exemplary directed acyclic graph generated according to this example is as follows Figure 4 shown.

[0164] This clear logical description and JSON model definition visualizes the complex association analysis model in an intuitive way, facilitating further implementation and understanding while providing an operational model foundation for the system.

[0165] Figure 4 An exemplary directed acyclic graph (DAG) of the correlation analysis model between VPN login logs and server operation logs is shown. The DAG illustrates the correlation analysis logic between VPN login logs and server operation logs, including key steps such as data screening, matching, and alarm triggering. For example,

[0166] Extract data entries from all VPN login logs and filter out entries that meet the conditions, that is, records with login status = failed;

[0167] Extract data entries from all server operation logs and filter out records where the operation type = high-risk operation;

[0168] The matching between the filtered VPN login logs and server operation logs is based on the client IP data element. That is, only when the client IP in the VPN login log matches the client IP in the server operation log will these entries be linked together for further analysis.

[0169] Within a set time window (e.g., 5 minutes), if the matching client IP meets the following conditions, an alert is generated: the IP address has three or more failed login events in the VPN login log; within the same time window, the IP address has at least one high-risk operation recorded in the server operation log.

[0170] Step 3: Define the minimum dataset. This step focuses on identifying the necessary data elements for analysis, rather than performing specific data filtering operations. By properly defining the minimum dataset, you can effectively reduce redundancy in data transmission and processing, improving the overall analysis efficiency of the system.

[0171] Minimizing the data set means retaining only the necessary data elements directly related to alarm analysis during the correlation analysis process, while eliminating unnecessary data entries. This ensures that each retained data element plays a clear role in the analysis process, reducing the storage and transmission of invalid data.

[0172] For example, the minimized data set of a VPN login log includes the data elements "data number, data time, client IP, login status", and uses the filter condition "login status = failure". The filtered minimized data set only focuses on unsuccessful VPN login attempts, providing input for alarm analysis.

[0173] The minimized data set of the server operation log includes the data elements "data number, data time, client IP, and operation type". The filtering condition is "operation type = high-risk operation". The filtered minimized data set only contains records of high-risk operations, so that correlation analysis can be used to identify potential risky behaviors.

[0174] By defining a minimal data set, only necessary data elements and entries are transmitted and stored, avoiding redundant data. This ensures that each data entry and element has a clear role in the analytical model, improving data processing and analysis efficiency. In this step, only the required data elements are defined, while the actual data screening will be performed in subsequent data processing stages.

[0175] Assume that the VPN login log and server operation log are filtered and the following results are obtained:

[0176]

[0177]

[0178] Step 4: Data Set Extraction. In this step, the data agent nodes in each data center filter and extract qualified log data from the data bus based on the minimized data set defined in the previous step, and transmit it to the message bus of the master data center to support subsequent analysis.

[0179] The data proxy node monitors and filters log data from the data bus in real time. The filtering process strictly adheres to the filtering criteria. For example, from VPN login logs, it extracts the data elements "data number, data time, client IP address, and login status," retaining entries where the login status equals "failed." From server operation logs, it extracts the data elements "data number, data time, client IP address, and operation type," retaining entries where the operation type equals "high-risk operation." After initial processing by the proxy node, the filtered log data is transmitted in real time to the message bus of the master control data center. The message bus serves as a cache layer, temporarily storing the extracted data and providing data support for subsequent correlation analysis and alarm generation.

[0180] Only log entries and key data elements that meet business needs are extracted to avoid redundant data transmission and improve transmission efficiency; the data proxy node completes screening locally to ensure that the main control data center only receives necessary data, providing more efficient data support for correlation analysis and alarm generation.

[0181] Step 5: Time series correlation analysis and calculation. In the main control data center, based on the correlation analysis model and the data obtained from the message bus, time series correlation analysis and calculation are performed to detect potential risks and generate alarm data.

[0182] Minimized VPN login log data and server operation log data are read from the message bus of the master data center. This data includes "data number, data time, client IP address, login status (VPN login log)" and "data number, data time, client IP address, operation type (server operation log)".

[0183] Within the timeframe, VPN login logs are first analyzed to identify failed login attempts (at least three failed logins) from the same client IP. Next, server operation logs are analyzed to check for any high-risk operations (at least one) from the client IP. Both types of events must occur within five minutes of each other, meaning the time difference between the failed VPN login and the high-risk operation must be no more than five minutes. When these conditions are met, an alert is generated. The alert includes the log data number for subsequent tracking and display.

[0184] The output of this step may also include a log of alarm data.

[0185] For example:

[0186] Alarm data log A001-vpn004 A001-vpn005 B001-op006 B001-op007

[0187] Through the time series correlation analysis, specific behavior patterns can be automatically identified, such as events where high-risk operations are performed after consecutive VPN login failures, and corresponding alarms can be generated to ensure security monitoring of the system.

[0188] Step 6: Aggregate the result data. In this step, according to the associated log data number recorded in the alarm data, extract the complete log data from each data center, and perform data fusion to generate a complete alarm record.

[0189] Based on the data numbers of VPN login logs and server operation logs in the alarm data, the master data center initiates a data extraction request through the data proxy nodes in each data center to obtain the corresponding complete log data, including the "data number, data time, client IP address, user account, VPN service IP address, VPN service port, login status, and source IP address" for VPN login logs and the "data number, data time, client IP address, server IP address, server name, user name, connection method, executed command, operation content, operation type, and operation result" for server operation logs. The data proxy nodes in each data center extract the complete log data from the data source and transmit it in real time to the master data center's message bus. At the master data center, the received VPN login log data and server operation log data are fused. The resulting alarm data log includes all associated data elements and records the specific alarm details. The current model aggregates and displays the extracted log data based on high-risk operations as key criteria.

[0190] For example:

[0191]

[0192] Example 1 illustrates a specific application of the disclosed data association analysis method. However, those skilled in the art will appreciate that this method offers a high degree of flexibility, allowing for the selection of different datasets based on diverse business needs and the design of association analysis models tailored to specific scenarios. The dataset type, screening criteria, association criteria, and statistical functions can be flexibly adjusted to meet diverse analysis objectives and business logic, depending on actual needs.

[0193] The key technical points involved in the data association analysis method disclosed in this disclosure are as follows:

[0194] 1) Extract a minimized dataset based on the association analysis model. This disclosure uses the association analysis model to filter out the core data elements required for analysis and generate a minimized dataset. This minimized dataset contains only the necessary data elements related to association analysis, such as the ID, time, IP address, and status in log data, avoiding the transmission and processing of redundant fields. This streamlined dataset reduces data transmission and processing overhead, enabling more efficient association analysis in a cross-data center environment.

[0195] 2) Dynamically generate a minimized dataset based on changes in the association analysis model. This method offers flexibility, meaning that the system can dynamically adjust the definition of the minimized dataset as the association analysis model changes. As analysis requirements evolve with business needs, rules, or models, the system automatically updates the filter criteria and data element selections to generate a minimized dataset that meets the latest requirements. This dynamic adjustment capability ensures the system's scalability and adaptability, enabling it to flexibly respond to different analysis scenarios and changes in business logic.

[0196] Predefined model-driven analysis predefines analysis conditions and data elements by associating analysis models, providing rule-based guidance for subsequent analysis; on-demand extraction of minimized data sets, extracting only necessary data elements based on model requirements, reducing the burden of transmitting and processing irrelevant data; automatic adjustment when the model changes, automatically adjusting the data extraction strategy when the model is updated to generate a new minimized data set; improving transmission efficiency and analysis flexibility, achieving efficient cross-data center analysis with minimal bandwidth consumption through streamlined data sets and dynamic adjustment strategies.

[0197] The data association analysis method disclosed in the present disclosure can bring the following beneficial effects:

[0198] 1) Effectively reduce the amount of data transmitted across the network. This approach significantly reduces the amount of data transmitted across data centers by pre-defined, precise data filtering and minimal dataset extraction. Compared to traditional full data transmission methods, this solution only transmits the core data elements directly relevant to analysis, reducing bandwidth usage and enabling efficient distributed data analysis without increasing network burden.

[0199] 2) Improve the timeliness of data analysis. Due to the reduction in data transmission volume, transmission efficiency is significantly improved, keeping data transmission delays within the network to within seconds. The system can complete real-time correlation analysis and generate alarm data with minimal latency, enabling rapid response to potential security threats or abnormal behavior.

[0200] 3) Support for time-series correlation statistical analysis. Leveraging the master data center's message bus, transmitted minimized data sets are stored in real time, supporting correlation statistical analysis based on time series analysis methods. By setting time windows and correlation conditions, time-series correlation calculations can be effectively performed on distributed log data, generating timely alert data. This approach not only provides powerful data integration capabilities but also enables analysis of complex behaviors across time periods, improving the system's insight into potential security risks.

[0201] This disclosure significantly improves the efficiency of large-scale data analysis across data centers by streamlining data transmission, improving analysis timeliness, and supporting complex time-series correlation analysis. This solution combines practicality and efficiency, providing a strong foundation for real-time security monitoring and data insights.

[0202] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable, and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically embodies computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0203] Example embodiments have been disclosed herein, and although specific terms are employed, they are used and should be interpreted only in a general illustrative sense and not for purposes of limitation. In some instances, it will be apparent to those skilled in the art that, unless otherwise expressly indicated, features, characteristics, and / or elements described in conjunction with a particular embodiment may be used alone or in combination with features, characteristics, and / or elements described in conjunction with other embodiments. Therefore, it will be understood by those skilled in the art that various changes in form and detail may be made without departing from the scope of the present disclosure as set forth in the appended claims.

Claims

1. A data association analysis method, comprising: Design association analysis models based on business needs; Determine a minimized data set definition according to the association analysis model; According to the minimized data set definition, extract multiple data sets from each data center and transmit them to the master data center, wherein the multiple data sets contain data entries and data elements thereof that conform to the minimized data set definition; Based on the association analysis model, performing time series association analysis on the transmitted multiple data sets to generate alarm data including data numbers; According to the data number in the alarm data, the data entry corresponding to the data number and all its data elements are re-extracted, and data fusion is performed on them to generate a complete alarm output.

2. The data association analysis method according to claim 1, wherein: Design an association analysis model based on business needs, including: Set data set selection criteria that meet the business requirements; Based on the business requirements, setting business logic for multiple data sets that meet the data set selection conditions; Based on the set data set selection conditions, business logic and execution order thereof, a directed acyclic graph model description language of the association analysis model is generated.

3. The data association analysis method according to claim 2, based on the business requirements, setting business logic for multiple data sets that meet the data set selection conditions, including: Based on the business requirements, setting screening conditions for the multiple data sets; Based on the business requirements, setting association conditions between the multiple data sets; Based on the business requirements, select the time window that needs statistics and set the statistical function.

4. The data association analysis method according to claim 3, wherein: Before designing an association analysis model based on business needs, include: The multiple data sets are standardized to generate a standardized data set.

5. The data association analysis method according to claim 4, wherein: According to the association analysis model, a minimum data set definition is determined, including: Selecting multiple data sets that meet the data set selection condition; From the selected data set, determine the data entries and data elements required for the business logic; Based on the determined data items and data elements, a minimized data set definition is generated.

6. The data association analysis method according to claim 5, wherein: From the selected data set, determine the data entries and data elements required for the business logic, including: Determining, from the selected data set, filtered data entries that meet the filtering conditions; Determining, from the filtered data items, the associated data elements required by the associated condition; From the filtered data items, statistical data elements required by the statistical function are determined.

7. The data association analysis method according to claim 5 or 6, wherein: According to the minimized data set definition, extracting multiple data sets from each data center and transmitting them to the master data center includes: Sending the minimized data set definition to the data proxy nodes of each data center; The data proxy node extracts the multiple data sets from a local data bus according to the minimized data set definition; The data proxy node transmits the extracted multiple data sets to the message bus of the master control data center in real time.

8. The data association analysis method according to claim 7, wherein: Based on the correlation analysis model, a time series correlation analysis is performed on the transmitted multiple data sets to generate alarm data containing data numbers, including: receiving the transmitted plurality of data sets from the message bus; In the master control data center, based on the business logic set by the association analysis model, a time series association analysis is performed on the received multiple data sets; According to the result of time series correlation analysis, alarm data including data number is generated.

9. The data association analysis method according to claim 8, wherein: In the master control data center, based on the business logic set by the association analysis model, a time series association analysis is performed on the received multiple data sets, including: Associating data items in the received multiple data sets according to the association condition; In the time window, a time series correlation analysis is performed on the correlated data items based on the statistical function.

10. The data association analysis method according to claim 9, wherein: According to the results of time series correlation analysis, alarm data containing data numbers are generated, including: Taking the data items in the time series correlation analysis result that are within the set range of the statistical function as a group of alarm events; Generate an alarm correlation identifier for each group of alarm events; The data numbers of the data items in each group of alarm events are output as alarm data, and all alarm data are recorded to form an alarm data log.

11. The data association analysis method according to any one of claims 8 to 10, wherein: According to the data number in the alarm data, re-extract the data entry corresponding to the data number and all its data elements, and perform data fusion on them to generate a complete alarm output, including: According to the data number in the alarm data log, the corresponding data entry and all its related data elements are transmitted to the master control data center through the data proxy node of each data center; In the master control data center, data fusion is performed on each received data item according to the alarm association identifier to generate a complete alarm output.

12. An electronic device comprising: one or more processors; A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data association analysis method according to any one of claims 1 to 11.

13. A computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the data association analysis method according to any one of claims 1 to 11.

14. A computer program product, comprising a computer program or instructions, wherein when the computer program or instructions are executed by a processor, the data association analysis method according to any one of claims 1 to 11 is implemented.

Images