A data analysis method, device, equipment and computer storage medium

Abstract

The application discloses a data analysis method and device, equipment and a computer storage medium. The method comprises the following steps: acquiring time window information corresponding to a current time window; determining target label information and target rule information according to the time window information; acquiring to-be-processed data corresponding to the target label information from a preset database; and performing data analysis on the to-be-processed data according to the target rule information in a time period corresponding to the current time window to obtain an analysis result. In this way, the real-time data in the preset database and the time window information have an association relationship, so that the to-be-processed data is selected based on the time window information, and the to-be-processed data is analyzed by using the rule corresponding to the time window information, which can realize asynchronous processing of data receiving and data analysis, thereby effectively improving the system data analysis performance.

Description

Technical Field

[0001] This application relates to the field of secure data analysis technology, and in particular to a data analysis method, apparatus, device, and computer storage medium. Background Technology

[0002] Data analysis is the process of analyzing large amounts of collected data using appropriate statistical analysis methods, extracting useful information, and drawing conclusions through detailed study and summarization. In scenarios such as security device log analysis and other real-time data analysis, data analysis can help people make judgments so that appropriate actions can be taken.

[0003] In related technologies, there are currently some open-source rule engines. These engines load data into memory and run scripts in memory for data matching. The entire matching process is linear and serial, which places high demands on the host machine's performance and cannot adapt to flexible and ever-changing needs. They are not suitable for processing large amounts of data and lack good horizontal scalability, making them unsuitable for flexible and ever-changing usage scenarios. In addition, current big data components offer a wealth of real-time analysis components, but these components rely on the big data system itself and require the deployment of many supporting components, making them too heavyweight and costly, resulting in poor practicality. They also place high demands on server clusters and inevitably require more investment, which to some extent limits the use cases of such components. Summary of the Invention

[0004] This application proposes a data analysis method, apparatus, device, and computer storage medium that enables asynchronous data reception and data analysis, thereby effectively improving the system's data analysis performance. Moreover, it can adapt to various data analysis scenarios and has high practicality.

[0005] To achieve the above objectives, the technical solution of this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a data analysis method, the method comprising:

[0007] Get the time window information corresponding to the current time window;

[0008] Based on the time window information, determine the target label information and target rule information;

[0009] The system retrieves the data to be processed corresponding to the target tag information from the preset database, and performs data analysis on the data to be processed according to the target rule information within the time period corresponding to the current time window to obtain the analysis results.

[0010] Secondly, embodiments of this application provide a data analysis device, which includes: an acquisition unit, a determination unit, and an analysis unit, wherein...

[0011] The acquisition unit is configured to acquire time window information corresponding to the current time window;

[0012] The determining unit is configured to determine target label information and target rule information based on the time window information;

[0013] The analysis unit is configured to retrieve the data to be processed corresponding to the target tag information from a preset database, and perform data analysis on the data to be processed according to the target rule information within the time period corresponding to the current time window to obtain the analysis results.

[0014] Thirdly, embodiments of this application provide a data analysis device, which includes: a memory and a processor; wherein,

[0015] The memory is used to store computer programs that can run on the processor;

[0016] The processor is configured to execute the method as described in the first aspect when running the computer program.

[0017] Fourthly, embodiments of this application provide a computer storage medium, characterized in that the computer storage medium stores a data analysis program, which, when executed by at least one processor, implements the method described in the first aspect.

[0018] This application provides a data analysis method, apparatus, device, and computer storage medium that acquires time window information corresponding to the current time window; determines target label information and target rule information based on the time window information; retrieves the data to be processed corresponding to the target label information from a preset database; and performs data analysis on the data to be processed according to the target rule information within the time period corresponding to the current time window to obtain analysis results. Thus, because there is a correlation between real-time data in the preset database and time window information, selecting the data to be processed based on the time window information and analyzing it using the rules corresponding to that time window information enables asynchronous data reception and data analysis. Furthermore, it has good scalability and is suitable for flexible and varied scenarios, thereby effectively improving the system's data analysis performance and possessing high practicality. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a data analysis method provided in an embodiment of this application;

[0020] Figure 2A flowchart illustrating another data analysis method provided in an embodiment of this application;

[0021] Figure 3 A flowchart illustrating yet another data analysis method provided in an embodiment of this application;

[0022] Figure 4 A schematic diagram of the system architecture of a data analysis method provided in this application embodiment;

[0023] Figure 5 A schematic diagram illustrating the creation process of a data analysis rule provided in an embodiment of this application;

[0024] Figure 6 This is a schematic diagram illustrating the working process of a data receiving procedure provided in an embodiment of this application.

[0025] Figure 7 This application provides a schematic diagram of the working process of a data analysis workflow.

[0026] Figure 8 This is a schematic diagram of the composition structure of a data analysis device provided in an embodiment of this application;

[0027] Figure 9 A schematic diagram of the specific hardware structure of a data analysis device provided in this application embodiment;

[0028] Figure 10 This is a schematic diagram of the composition structure of a data analysis device provided in an embodiment of this application. Detailed Implementation

[0029] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0030] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0031] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first, second, third" used in the embodiments of this application are merely for distinguishing similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0032] In practical applications, existing data analysis engines are mainly divided into lightweight script-based rule engines and real-time analysis engines based on big data. However, both have some drawbacks, which to some extent limit the application scenarios of data analysis engines. For example, these drawbacks are as follows:

[0033] The main drawbacks of script-based Drools-like rule engines are: 1) They are unsuitable for large-scale data processing scenarios. In massive processing scenarios, it's impractical to process all data in memory. Due to the extremely large concurrent data volume, memory consumption is enormous, and once memory overflows, data will be permanently lost and unrecoverable, which would be an irreparable disaster for data-sensitive systems. 2) They lack good horizontal scalability. To address the risk of data loss, subsequent operations must have data persistence capabilities. However, horizontal scaling makes it difficult to guarantee that data is consumed only once, rendering such horizontal scaling meaningless and rendering the generated data worthless. 3) They cannot adapt to flexible and changing scenarios. These rule engines only support single-item processing, making it difficult to leverage their advantages when batch processing data is required within time windows.

[0034] The main drawbacks of real-time analytics engines based on big data are: 1) They are too heavyweight. While the advantages of big data analytics are obvious, running an analytics engine requires numerous supporting components, and the preparation work is extensive. Applications without the capability to build a large database platform are simply unable to participate. 2) They are costly. Deploying a big data platform requires more machine resources and incurs higher maintenance costs.

[0035] Based on this, this application provides a data analysis method. The basic idea of ​​this method is: to obtain the time window information corresponding to the current time window; to determine the target label information and target rule information based on the time window information; to obtain the data to be processed corresponding to the target label information from a preset database; and to perform data analysis on the data to be processed according to the target rule information within the time period corresponding to the current time window, thereby obtaining the analysis results. In this way, since there is a correlation between the real-time data in the preset database and the time window information, selecting the data to be processed based on the time window information and analyzing the data using the rules corresponding to that time window information enables asynchronous processing of data reception and data analysis. Furthermore, it has good scalability and is suitable for flexible and varied scenarios, thereby effectively improving the system's data analysis performance and possessing high practicality.

[0036] Example 1

[0037] In one embodiment of this application, see [link to embodiment]. Figure 1 This illustrates a flowchart of a data analysis method provided in an embodiment of this application. Figure 1 As shown, the method may include:

[0038] S101: Get the time window information corresponding to the current time window.

[0039] It should be noted that, in this embodiment, the time window information may include information such as window identifier (winId), rule identifier (ruleId), data type (dataType), and analysis identifier (isAya). The time window information is determined when the time window (timeWin) is created; that is, each time window corresponds to a set of time window information.

[0040] It should also be noted that, in this embodiment, the time window can be created by calling the Task Service's scheduled task template program to create and start a scheduled task. The scheduled task then calls the time window creation interface to achieve the time window's scheduled creation. In practical applications, the length of the time window can be specifically set according to the actual situation, such as adjusting it based on the characteristics of the data to be analyzed or the device's own data processing capabilities. No limitations are imposed here.

[0041] In some embodiments, obtaining the current time window information may include:

[0042] Retrieve the current time window from the preset message queue;

[0043] Based on the current time window, the time window information corresponding to the current time window is obtained from the preset database.

[0044] It should be noted that the preset message queue here may include a time window message queue and an initial data message queue. The time window message queue stores the time window information that is written sequentially, while the initial data message queue can contain any data that needs to be analyzed in real time. For example, the data in the initial data message queue may be data from external input that needs to be analyzed, such as log data.

[0045] It should also be noted that, taking the time-window message queue as an example, the determination of the preset message queue is, in some embodiments, such as... Figure 2 As shown, the method may further include the following during the creation of a time window:

[0046] S201: After the scheduled task starts, a first time window is created according to a preset time interval, and the information of the first time window corresponding to the first time window is written into the preset database.

[0047] In the implementation process, the TaskService's scheduled task template program can be called to create and start a scheduled task. This provides an interface for creating time window information and parameters such as the time window (timeWin). The interface's Uniform Resource Locator (URL) format is / v1 / timeWindow / add / 1399357342040354818 / 1. The URL format uses " / " as separators, with the second-to-last parameter being the rule identifier (ruleId) and the last parameter being the data type (dataType). When the scheduled task starts, it will call the URL in the scheduled task template to create the time window. Furthermore, after the time window is created, the corresponding time window information needs to be stored in a preset database for later use in data analysis.

[0048] S202: Obtain the second time window information from the preset cache, and replace the second time window information with the first time window information and store it in the preset cache.

[0049] S203: Write the second time window into the preset message queue.

[0050] It should be noted that, in this embodiment, the preset cache can be represented by a Reids cache. This preset cache stores multiple time window information, and the data type corresponding to each time window information is entirely derived from the data types present in the original data. In other words, all data types in the original data can find time window information of the same data type in the preset cache. Thus, when the original data successfully matches the second time window information in the preset cache during processing, that is, when the second time window information, which was previously used, is replaced by the first time window information, the second time window information is simultaneously written into a preset message queue.

[0051] It should also be noted that, in this embodiment, the first time window information can be the latest time window information created based on the scheduled task, and the second time window information can be the time window information from the last use. Specifically, the latest created time window information refers to the time window information corresponding to the latest time window created according to the time window created by the scheduled task, and the last used time window information refers to the time window information from the most recent time when the corresponding window identifier was added as a label to the intermediate data. In short, the above embodiment uses the latest created time window information to continuously update the time window information that has already been matched.

[0052] Thus, for the thread creating the time window, the timing expression of the scheduled task can be automatically generated based on the time window (timeWin). For example, if timeWin=30, the expression would be * / 30 * * * * ?. At this time, the time window creation interface provided by Data Analysis Service, / v1 / timeWindow / add / 1399357342040354818 / 1, is called periodically. After receiving the request, Data Analysis Service first creates the time window information and writes it to the database. The time window information mainly includes window identifier (winId), rule identifier (ruleId), data type (dataType), analysis flag (isAya), etc. Among them, the window identifier (winId) is a unique long data type automatically generated by the system on the time series, the rule identifier (ruleId) is the second to last segment of data after being separated by " / " in the URL request, and the data type (dataType) is the last segment of data after being separated by " / " in the URL request. Then, access the preset cache area and use dataType#ruleId as the key to retrieve the information of the previous time window. Write the latest time window information, i.e. the first time window information, into the preset cache area with dataType#ruleId as the key and the window identifier (winId) as the value. Then, write the previous time window information, i.e. the second time window information, into the preset message queue.

[0053] In this way, after obtaining the current time window information for data analysis from the preset message queue, the corresponding data analysis can be performed based on the time window information.

[0054] S102: Determine the target label information and target rule information based on the time window information.

[0055] It should be noted that, in the embodiments of this application, the time window information may include label information and rule information. The label information may include data type (dataType) and window identifier (winId), while the rule information may include rule identifier (ruleId).

[0056] In this way, when using the current time window as the carrier for data analysis, the corresponding time window information can be directly obtained. Furthermore, the data type and window identifier in the time window information can be obtained as target label information, and the rule identifier in the time window information can be obtained as target rule information.

[0057] S103: Obtain the data to be processed corresponding to the target label information from the preset database, and perform data analysis on the data to be processed according to the target rule information within the time period corresponding to the current time window to obtain the analysis results.

[0058] It should be noted that before performing data analysis on the data to be processed, a data receiving process from a preset database needs to be implemented first. This process can specifically involve: retrieving data from a preset message queue, then retrieving the latest time window information from a preset cache, adding a time window tag to the data, and storing it in the preset database. In some embodiments, such as... Figure 3 As shown, the method may further include:

[0059] S301: Receive the initial data to be analyzed;

[0060] S302: Perform array transformation on the initial data to obtain a node array.

[0061] S303: Obtain the data type from the node array, perform deserialization operation according to the data type, and obtain at least one data table; wherein each data table includes several intermediate data.

[0062] S304: Update the at least one data table according to the data type corresponding to each data table to determine at least one candidate data table; wherein, each candidate data table includes the plurality of intermediate data and the window identifier corresponding to the intermediate data.

[0063] S305: Store the at least one candidate data table into the preset database.

[0064] It should be noted that in practical applications, initial data can be obtained from a preset message queue. This initial data comes from externally input data that needs to be analyzed. Taking security log data as an example, security log data contains data type fields. First, the security log data is converted into a node (JsonNode) array. The data corresponding to the data type attributes is obtained from the node array. Based on the data type, the security log data is deserialized into specific data objects. At the same time, time window information with the same data type is selected from the preset cache based on the data type. The window identifier (winId) in the time window information is added as a label to the data object to obtain intermediate data. The marking process of security log data is completed in this way, and the marked data is stored in the preset database.

[0065] It should also be noted that when the amount of data is large, the initial data can be divided into multiple parts and marked. The criteria for dividing the initial data can be set according to the actual situation. For example, it can be determined based on the length of the time window and the data processing capability of the device. No restrictions are imposed here.

[0066] Furthermore, after receiving the data, for data analysis, a target data table can be selected first based on the data type, and then a data list can be selected based on the window identifier, i.e., the selected data to be processed, for subsequent analysis. Here, the target label information may include the target data type and the target window identifier; therefore, in some embodiments, obtaining the data to be processed corresponding to the target label information from a preset database includes:

[0067] Select the target data table corresponding to the target data type from the preset database;

[0068] Select the data to be processed corresponding to the target window identifier from the target data table.

[0069] It should be noted that the data table is a space in the pre-defined database used to store the target data. During the data receiving process, the initial data can be converted into an array and deserialized before being stored in the corresponding data table, based on the data type. In this way, during the data analysis process, the corresponding target data table can be selected from the pre-defined database by data type. Then, with the target window identifier determined, the target data with the target window identifier is selected from the target data table and identified as the data to be processed.

[0070] Furthermore, in embodiments of this application, the target rule information may include a rule identifier. Therefore, in some embodiments, the step of performing data analysis on the data to be processed based on the target rule information to obtain analysis results may include:

[0071] Based on the rule identifier, load the first rule information corresponding to the rule identifier from the preset rule table;

[0072] The data to be processed is analyzed based on the first rule information to obtain the analysis results.

[0073] In other words, time window information is obtained from a preset message queue, and then the first rule information is loaded from a preset rule table according to the rule identifier, so that data analysis (such as deduplication and matching processing) can be performed on the data to be processed according to the first rule information, thereby obtaining the analysis results.

[0074] In one specific embodiment, the step of performing data analysis on the data to be processed based on the first rule information to obtain the analysis result may include:

[0075] The data to be processed is matched according to the rules in the first rule information;

[0076] If there is a rule in the first rule information that matches the data to be processed, then the matching result is determined as the analysis result.

[0077] It should be noted that the rule matching process for the data to be processed based on the first rule information can specifically involve matching the data type of the data to be processed with the rules in the first rule information during the data analysis process. Furthermore, when a match is successful, the data to be processed can be analyzed according to the matched rules, and the analysis results can be output; when a match fails, an alarm message can be issued to remind manual intervention to check the labels of the target data and determine if there are any label errors, and to retrieve new data to be processed from the preset database.

[0078] In some embodiments, after performing data analysis on the data to be processed based on the target rule information to obtain the analysis results, the method may further include:

[0079] The analysis results shall be presented in the form of a graph or table; and / or,

[0080] When the analysis results trigger a preset event, an alarm message is issued.

[0081] It should be noted that after obtaining the analysis results, they can be displayed in the form of graphs or tables, or the analysis results can be queried through the query interface; in addition, when the analysis results trigger preset events, alarm messages can be issued to remind manual intervention or to automate the processing according to preset methods.

[0082] Furthermore, to provide rich extensibility and support assistive function plugins that can be configured and activated, the data to be processed can be preprocessed. Therefore, in some embodiments, the method may further include: preprocessing the data to be processed.

[0083] In one specific embodiment, the preprocessing of the data to be processed may include:

[0084] The data to be processed is deduplicated; and / or,

[0085] Real-time traffic statistics processing is performed on the data to be processed.

[0086] It should be noted that, in the embodiments of this application, deduplication and real-time traffic statistics are two possible options in the preprocessing process. Preprocessing the data to be processed is to adapt to rich business needs, that is, to provide rich scalability, and also to enable the data to better match with rule information, thereby further improving the accuracy of analysis.

[0087] In addition, after performing data analysis on the data to be processed according to the first rule information and obtaining the analysis results, in some embodiments, the method may further include:

[0088] Update the analysis identifier in the time window information to 1, and delete the data to be processed.

[0089] In other words, for the selected data to be processed, deduplication and other operations can be performed based on the hash. Then, the data to be processed is matched according to the first rule information. If there is data that matches the rule, the matching result is output. Finally, the analysis identifier (isAya) in the time window information is updated to 1, and the data to be processed with the winId label is deleted.

[0090] It should also be noted that, in the embodiments of this application, the preprocessing rules can be stored in a preset database in the form of time window information. In some embodiments, they can also be connected to the data processing device in the form of a data processing plugin. Specifically, the data processing device needs to reserve a preprocessing interface, and a corresponding data processing plugin can be developed and connected according to the interface rules.

[0091] In this way, target tag information or data to be processed corresponding to the target data type is obtained from the preset database, and data analysis is performed on the data to be processed according to the target rule information to obtain the analysis results; then, the above process is repeated as needed to complete real-time data analysis based on time windows.

[0092] In other words, this application proposes a data analysis engine suitable for batch processing of continuous data. On one hand, it provides a real-time analysis option for scenarios where big data analysis engines cannot be used due to cost constraints. On the other hand, it solves the performance problems that conventional lightweight open-source rule engines cannot address in massive data analysis scenarios, such as single-machine processing bottlenecks, lack of horizontal scalability, and limited rule richness. Furthermore, it also solves the problem that conventional lightweight open-source rule engines cannot adapt to changing data analysis scenarios. Thus, this application proposes a design method for a real-time data analysis engine based on continuous time windows. This analysis engine is used for security-level analysis and can respond to security attacks based on real-time analysis results. The continuous time window approach is used to improve real-time performance and response speed.

[0093] This embodiment provides a data analysis method. It involves acquiring time window information corresponding to the current time window; determining target label information and target rule information based on the time window information; retrieving the data to be processed corresponding to the target label information from a preset database; and performing data analysis on the data to be processed within the time period corresponding to the current time window, based on the target rule information, to obtain analysis results. Because there is a correlation between real-time data in the preset database and time window information, selecting data to be processed based on time window information and analyzing it using the rules corresponding to that time window information enables asynchronous data reception and analysis. Furthermore, it has good scalability and is suitable for flexible and varied scenarios, thereby effectively improving the system's data analysis performance and demonstrating high practicality.

[0094] Example 2

[0095] Based on the same inventive concept as the foregoing embodiments, this application proposes a data analysis method for a real-time data analysis engine based on continuous time windows. This method primarily decouples real-time data persistence and data analysis by maintaining time windows, transforming serial data processing into asynchronous parallel data processing. By adding configurable auxiliary function plugins, such as data deduplication and real-time data traffic statistics, it can adapt to diverse business needs. In scenarios supported by Kubernetes (K8S), K8S can be used to support horizontal scaling and service governance. For scenarios where K8S cannot be used, a simple microservice architecture can be used to achieve horizontal scaling. See also... Figure 4 This illustrates a system architecture diagram of a data analysis method provided in an embodiment of this application. Figure 4 As shown, it can include data analysis equipment and a task server. The data analysis equipment mainly consists of four parts:

[0096] 401: Rule Creation Thread. This thread creates data analysis rules and defines time windows, then creates scheduled task templates based on those time windows. The rule content depends on the selected data storage tool; for relational databases, it should be a WHERE clause, while for search engines (like Elasticsearch), it should be the JSON parameters of the query interface, etc. Here, the time window can be created in the Task Service within the rule creation thread.

[0097] 402: Time Window Creation Thread. The scheduled task periodically initiates the time window creation interface. First, it creates the time window information and writes it to the database. Then, it accesses the cache to retrieve the previously used time window information, overwrites the previously saved information with the latest time window information, and finally writes the previous time window information to the message queue.

[0098] Specifically, in the time window creation thread, firstly, a time window is created and saved to the database; secondly, the cache is read, the original time window information is retrieved, and the new time window information is put into the cache; finally, the original time window information is put into the message queue.

[0099] 403: Data receiving thread group. Retrieves data from the message queue, retrieves the latest time window information from the cache, tags the data with time window information (i.e., window identifier), and stores it in the database.

[0100] Specifically, multiple data receiving threads can run concurrently, forming a data receiving thread group. This allows for the rapid tagging of data with time window information, facilitating subsequent data analysis and processing. For a single data receiving thread, the process involves: first, retrieving data from the message queue; second, retrieving time window information from the cache; and finally, tagged the data with the time window information and storing it in the database.

[0101] 404: Data Analysis Thread Group. This group retrieves time window information from the message queue, loads rule information created in rule creation thread 401 from the database, analyzes and processes the data based on the rule information, and outputs the analysis results.

[0102] Specifically, multiple data analysis threads can run simultaneously in a data analysis thread group. For a single data analysis thread, firstly, time window information is obtained from the message queue; secondly, rule information is loaded from the database, and then the data is deduplicated and matched; finally, the analysis results are output.

[0103] In some embodiments, the message queue can be RabbitMQ, the database can be MySQL, and the cache can be Redis.

[0104] In one specific implementation, asynchronous analysis of real-time data is performed based on continuous time windows, as follows:

[0105] 1) Rule Creation Thread. Define data analysis rules, including rule identifier, data type, time window, and rule information. After saving to the database, call the TaskService's scheduled task creation template program to create and start a scheduled task. The parameters provide the interface for creating the time window and other parameters such as the time window. The interface URL format is / v1 / timeWindow / add / 1399357342040354818 / 1. The interface URL is separated by " / ", where the second to last parameter is the rule identifier and the last parameter is the data type. After the scheduled task starts, it will call the URL in the scheduled task template to create the time window.

[0106] 2) Time Window Creation Thread. The timing expression for the scheduled task is automatically generated based on the time window. For example, if timeWin=30, the expression is * / 30 * * * * ?. This periodically initiates a call to the Data Analysis Service's time window creation interface / v1 / timeWindow / add / 1399357342040354818 / 1. Upon receiving the request, Data Analysis Service first creates the time window information and writes it to the database. The time window information mainly includes the window identifier, rule identifier, data type, and analysis flag. The window identifier is a unique Long type data automatically generated by the system on the time series. The rule identifier is the second-to-last segment of data separated by " / " in the URL request, and the data type is the last segment of data separated by " / " in the URL request. Then, it accesses the Redis cache using dataType#ruleId as the key to retrieve the previously used time window information. The latest time window information, i.e., dataType#ruleId as the key and winId as the value, is written to the Redis cache. Finally, the previous time window information (i.e., the old time window) is written to the message queue. See details below. Figure 5 .

[0107] 3) Data Receiving Thread Group. This thread retrieves data from the message queue. This data comes from external input and needs to be analyzed; log data is used as an example here. The data contains data type fields. First, the data is converted into a JsonNode array. The dataType attribute data is retrieved from the JsonNode array. Based on the data type, the data is explicitly deserialized into specific data objects. Simultaneously, using dataType as a condition, a solution provided by Jedis (which encapsulates the Redis caching API) is used to perform a fuzzy match on all key values ​​using `keys(dataType+"*")`. The returned result is a Set collection containing the key values ​​that meet the condition. Then, the `multiGet()` method is used to retrieve a list of winIds. Finally, the data to be analyzed is tagged with the winId field and stored in the database. For the same data, multiple copies need to be stored depending on the number of winIds in the list, with each winId requiring a copy containing the winId information. This thread can be integrated with scalable analysis scenarios, such as data deduplication (calculating the data's hash value) and merging. See details below. Figure 6 .

[0108] 4) Data Analysis Thread Group. This group retrieves time window information from the message queue, loads rule information from the rule table based on the ruleId, selects the data table based on the data type, and selects the data list based on the winId. For the selected data, it performs deduplication operations such as hashing. Then, it matches the selected data list against the selection criteria in the rule information. If any data matches the rule, it outputs the matching result. Finally, it updates the time window information analysis flag to 1 and deletes the data corresponding to the winId from the data table. See details in [link to details]. Figure 7 .

[0109] like Figure 7 As shown, the specific implementation steps for the data analysis thread may include:

[0110] S701: Retrieve time window information from the message queue.

[0111] S702: Load rule information from the rule table based on the rule identifier (ruleId).

[0112] S703: Select a data table based on the data type (dataType) and select data based on the window identifier (winId).

[0113] S704: For the selected data, perform deduplication based on the hash.

[0114] S705: Perform query processing on the selected data according to the rules (selection).

[0115] S706: Based on the deduplication and query results, convert them into event objects.

[0116] S707: Update the analysis identifier (isAya) in the time window information to 1, and delete the data with the corresponding winId in the data table.

[0117] In short, this application proposes a microservice-based solution for massive data analysis without altering the device architecture. Specifically, it proposes a real-time data analysis engine based on continuous time windows. In real-time data analysis scenarios, an asynchronous separation model for data reception and analysis is used to improve efficiency. This application utilizes Redis caching and message queues (MQ) flexibly, combined with the multi-threaded mode of the Netty network programming framework, to achieve asynchronous processing of data reception and analysis, significantly improving the system's real-time analysis performance. Furthermore, this application provides scalable data preprocessing scenarios. Custom data processing plugins can effectively expand the use of data preprocessing scenarios. Thus, compared to open-source rule engines (such as Drools and Aviator) in related technologies, this approach has lower requirements for device resources and offers better performance with the same resources. It also provides richer usage scenarios, implements the functionality of open-source rule engines, and offers extensive extensibility for data preprocessing. Compared to real-time analysis components provided by big data components in related technologies (such as Spark Streaming and Flink), it has relatively lower requirements for equipment resources, and its software, hardware, and operation and maintenance costs are relatively lower.

[0118] In summary, the above embodiments have provided a detailed explanation of the specific implementation of the aforementioned embodiments. It can be seen that, due to the correlation between real-time data in the preset database and time window information, selecting data to be processed based on time window information and analyzing the data using the rules corresponding to that time window information can achieve asynchronous processing of data reception and data analysis. Moreover, it has good scalability and is suitable for flexible and varied scenarios, thereby effectively improving the system's data analysis performance and having high practicality.

[0119] Example 3

[0120] Based on the same inventive concept as the foregoing embodiments, see [link to previous document]. Figure 8 This illustrates a schematic diagram of the composition of a data analysis device 80 provided in an embodiment of this application. For example... Figure 8 As shown, the data analysis device 80 may include: an acquisition unit 801, a determination unit 802, and an analysis unit 803; wherein,

[0121] The acquisition unit 801 is configured to acquire the time window information corresponding to the current time window.

[0122] The determining unit 802 is configured to determine target label information and target rule information based on the time window information;

[0123] The analysis unit 803 is configured to retrieve the data to be processed corresponding to the target tag information from a preset database, and perform data analysis on the data to be processed according to the target rule information within the time period corresponding to the current time window to obtain the analysis results.

[0124] In some embodiments, the acquisition unit is specifically configured to acquire the current time window from a preset message queue; and, based on the current time window, acquire the time window information corresponding to the current time window from the preset database.

[0125] In some embodiments, see Figure 8 The data analysis device 80 further includes a creation unit 804, configured to create a first time window at a preset time interval after the scheduled task starts, and write the first time window information corresponding to the first time window into the preset database; and to obtain second time window information from a preset cache, and replace the second time window information with the first time window information and store it in the preset cache; and to write the second time window into the preset message queue; wherein the first time window information is the latest time window information created according to the scheduled task, and the second time window information is the time window information used in the last time.

[0126] In some embodiments, see Figure 8 The data analysis device 80 further includes an identification unit 805, configured to receive initial data to be analyzed; and to perform array conversion on the initial data to obtain a node array; and to obtain a data type from the node array, and perform deserialization operation according to the data type to obtain at least one data table; wherein each data table includes a plurality of intermediate data; and to update the at least one data table according to the data type corresponding to each data table to determine at least one candidate data table; wherein each candidate data table includes the plurality of intermediate data and a window identifier corresponding to the intermediate data; and to store the at least one candidate data table in the preset database.

[0127] In some embodiments, the target rule information includes a rule identifier; correspondingly, the analysis unit 803 is specifically configured to load first rule information corresponding to the rule identifier from a preset rule table according to the rule identifier; and to perform data analysis on the data to be processed according to the first rule information to obtain the analysis result.

[0128] In some embodiments, the analysis unit 803 is specifically configured to perform rule matching processing on the data to be processed according to the rules in the first rule information; and if there is a rule in the first rule information that matches the data to be processed, then the matching result is determined as the analysis result.

[0129] In some embodiments, the analysis unit 803 is further configured to display the analysis results in the form of a graph or table; and / or to issue an alarm message when the analysis results trigger a preset event.

[0130] In some embodiments, the analysis unit 803 is further configured to perform deduplication processing on the data to be processed; and / or to perform real-time traffic statistics processing on the data to be processed.

[0131] Understandably, in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular component. Furthermore, the components in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0132] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0133] Therefore, this embodiment provides a computer storage medium storing a data analysis program that, when executed by at least one processor, implements the steps of the method described in any of the foregoing embodiments.

[0134] Based on the composition of the aforementioned data analysis device 80 and the computer storage medium, see [link to relevant documentation]. Figure 9 This illustrates a schematic diagram of the specific hardware structure of the data analysis device 90 provided in an embodiment of this application. For example... Figure 9 As shown, it may include: a communication interface 901, a memory 902, and a processor 903; the various components are coupled together through a bus system 904. It is understood that the bus system 904 is used to implement communication between these components. In addition to a data bus, the bus system 904 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 9 The various buses are all labeled as bus system 904. Among them, the communication interface 901 is used for receiving and sending signals during the process of sending and receiving information with other external network elements;

[0135] Memory 902 is used to store computer programs that can run on processor 903;

[0136] Processor 903, when running the computer program, performs the following:

[0137] Get the time window information corresponding to the current time window;

[0138] Based on the time window information, determine the target label information and target rule information;

[0139] The system retrieves the data to be processed corresponding to the target tag information from the preset database, and performs data analysis on the data to be processed according to the target rule information within the time period corresponding to the current time window to obtain the analysis results.

[0140] It is understood that the memory 902 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 902 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0141] The processor 903 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 903 or by software instructions. The processor 903 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 902, and the processor 903 reads the information in memory 902 and, in conjunction with its hardware, completes the steps of the above method.

[0142] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0143] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.

[0144] Alternatively, as another embodiment, the processor 903 is further configured to perform the steps of the method described in any of the foregoing embodiments when running the computer program.

[0145] Based on the composition of the aforementioned data analysis device 80 and the computer storage medium, see [link to relevant documentation]. Figure 10 This illustrates a schematic diagram of the composition of a data analysis device 90 provided in an embodiment of this application. For example... Figure 10 As shown, the data analysis device 90 may include the data analysis device 80 as described in any of the foregoing embodiments.

[0146] In this embodiment, the data analysis device 90 acquires the time window information corresponding to the current time window; determines the target label information and target rule information based on the time window information; retrieves the data to be processed corresponding to the target label information from a preset database; and performs data analysis on the data to be processed according to the target rule information within the time period corresponding to the current time window to obtain the analysis results. Thus, because there is a correlation between the real-time data in the preset database and the time window information, selecting the data to be processed based on the time window information and analyzing it using the rules corresponding to that time window information enables asynchronous processing of data reception and data analysis. Furthermore, it has good scalability and is suitable for flexible and varied scenarios, thereby effectively improving the system's data analysis performance and demonstrating high practicality.

[0147] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0148] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0149] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0150] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0151] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0152] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data analysis method, characterized in that, The method includes: Get the time window information corresponding to the current time window; Based on the time window information, target label information and target rule information are determined; wherein, the target label information includes target data type and target window identifier; The system retrieves the data to be processed corresponding to the target tag information from a preset database, and performs data analysis on the data to be processed according to the target rule information within the time period corresponding to the current time window to obtain the analysis results. The method further includes: After the scheduled task starts, a first time window is created at a preset time interval, and the information of the first time window corresponding to the first time window is written into the preset database. Obtain the second time window information from the preset cache, and replace the second time window information with the first time window information and store it in the preset cache; Write the second time window into a preset message queue; The first time window information is the latest time window information created based on the scheduled task, and the second time window information is the time window information from the last use.

2. The method according to claim 1, characterized in that, The step of obtaining the time window information corresponding to the current time window includes: Retrieve the current time window from the preset message queue; Based on the current time window, the time window information corresponding to the current time window is obtained from the preset database.

3. The method according to claim 1, characterized in that, The step of obtaining the data to be processed corresponding to the target tag information from the preset database includes: Select the target data table corresponding to the target data type from the preset database; Select the data to be processed corresponding to the target window identifier from the target data table.

4. The method according to claim 1, characterized in that, The method further includes: Receive the initial data to be analyzed; The initial data is transformed into an array to obtain a node array; The data type is obtained from the node array, and deserialization is performed according to the data type to obtain at least one data table; wherein each data table includes several intermediate data; The at least one data table is updated according to the data type corresponding to each data table to determine at least one candidate data table; wherein, each candidate data table includes the plurality of intermediate data and the window identifier corresponding to the intermediate data; Store the at least one candidate data table into the preset database.

5. The method according to claim 1, characterized in that, The target rule information includes a rule identifier; The step of performing data analysis on the data to be processed based on the target rule information to obtain analysis results includes: Based on the rule identifier, load the first rule information corresponding to the rule identifier from the preset rule table; The data to be processed is analyzed based on the first rule information to obtain the analysis results.

6. The method according to claim 5, characterized in that, The step of performing data analysis on the data to be processed based on the first rule information to obtain the analysis result includes: The data to be processed is subjected to rule matching processing based on the first rule information; If there is a rule in the first rule information that matches the data to be processed, then the matching result is determined as the analysis result.

7. The method according to any one of claims 1-6, characterized in that, After performing data analysis on the data to be processed based on the target rule information to obtain the analysis results, the method further includes: The analysis results shall be presented in the form of a graph or table; and / or, When the analysis results trigger a preset event, an alarm message is issued.

8. The method according to claim 1, characterized in that, Before performing data analysis on the data to be processed based on the target rule information, the method further includes: The data to be processed is deduplicated; and / or, Real-time traffic statistics processing is performed on the data to be processed.

9. A data analysis device, characterized in that, The data analysis device includes: an acquisition unit, a determination unit, and an analysis unit, wherein, The acquisition unit is configured to acquire time window information corresponding to the current time window; The determining unit is configured to determine target label information and target rule information based on the time window information; wherein, the target label information includes target data type and target window identifier; The analysis unit is configured to retrieve the data to be processed corresponding to the target tag information from a preset database, and perform data analysis on the data to be processed according to the target rule information within the time period corresponding to the current time window to obtain the analysis results; The data analysis device further includes a creation unit configured to, after the scheduled task starts, create a first time window at a preset time interval and write the first time window information corresponding to the first time window into the preset database; obtain second time window information from a preset cache and replace the second time window information with the first time window information and store it in the preset cache; and write the second time window into a preset message queue; wherein the first time window information is the latest time window information created according to the scheduled task, and the second time window information is the time window information used in the last time.

10. A data analysis device, characterized in that, The data analysis device includes: a memory and a processor; wherein... The memory is used to store computer programs that can run on the processor; The processor is configured to perform the method as described in any one of claims 1 to 8 when running the computer program.

11. A computer storage medium, characterized in that, The computer storage medium stores a data analysis program, which, when executed by at least one processor, implements the method as described in any one of claims 1 to 8.

Images