Data time sequence association method and device, equipment and storage medium
By storing data of the same event type and association fields on the same node, the efficiency improvement and resource optimization of timing correlation analysis are achieved, and the performance bottlenecks and resource waste problems in the timing correlation analysis process in Flink are solved.
Patent Information
- Application Number
- CN202411524595.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-29
- Publication Date
- 2025-01-17
AI Technical Summary
When using Apache Flink for timing correlation analysis, multiple correlation conditions, association methods and time windows cause multiple windows to consume data multiple times, causing performance bottlenecks and waste of computing resources.
By storing the target data of the same event type and associated fields on the same target node, data matching is required only once, avoiding repeated judgments, improving computing efficiency, and releasing resources through regular data cleaning.
The efficiency of timing correlation analysis is improved, resource waste is avoided, and performance bottlenecks and computational resource waste problems in the timing correlation analysis process are solved.
Smart Images
Figure CN119322780A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data technology, and in particular to a data time series association method, device, equipment and storage medium. Background Art
[0002] Apache Flink (a real-time processing engine) is a common distributed real-time processing engine in the big data field. Its high throughput and low latency characteristics are widely suitable for the development of some large-scale stream processing applications.
[0003] In the prior art, when using Flink for time series correlation analysis, if there are many correlation models with different correlation conditions, different correlation methods, and different time windows, such as more than 20, there will be a problem of multiple windows consuming data multiple times for time series correlation analysis, resulting in a large performance bottleneck and waste of computing resources in the time series correlation analysis process under certain resources. Summary of the invention
[0004] In view of this, the purpose of the present invention is to provide a data time series association method, device, equipment and storage medium, which can store target data corresponding to the same event type and the same associated field data in the same target node. After acquiring new data, for the same type of target data, the new data only needs to be matched with each node once, thereby improving the computational efficiency of data time series association. The specific scheme is as follows:
[0005] In a first aspect, the present application provides a data time series association method, which is applied to the Flink distributed real-time processing engine, including:
[0006] Extracting target association conditions that meet preset conditions from a preset association model, and generating target keys corresponding to target data based on the target association conditions;
[0007] The target data are grouped based on the target key, and the grouped target data are respectively stored in corresponding target nodes; wherein the target data stored in the same target node have the same event type and associated field data;
[0008] When the data to be matched is obtained, the data to be matched is matched with the target data in any target node based on the event type and the associated field data, and whether to associate the data to be matched with the target data in any target node is determined based on the matching result.
[0009] Optionally, before extracting the target association condition satisfying the preset condition from the preset association model, the method further includes:
[0010] Process the real-time middleware data through the Flink distributed real-time processing engine to access the initial data;
[0011] Performing data analysis on the initial data after access using a preset data analysis method, so as to screen the initial data based on corresponding analysis results;
[0012] If the initial data meets the preset screening condition, the initial data is determined as the target data.
[0013] Optionally, before extracting the target association condition satisfying the preset condition from the preset association model, the method further includes:
[0014] A label of the event type corresponding to the target data is added to the target data, so as to extract the target association condition satisfying the preset condition from the preset association model according to the label.
[0015] Optionally, extracting a target association condition satisfying a preset condition from a preset association model includes:
[0016] Splitting the preset association model containing the target data according to the preset association judgment condition in the preset association model;
[0017] The target association condition satisfying the preset condition is extracted according to the corresponding splitting result and the label.
[0018] Optionally, extracting a target association condition satisfying a preset condition from a preset association model includes:
[0019] Splitting the preset association model containing the target data according to the preset association judgment condition in the preset association model;
[0020] The target association condition satisfying the preset condition is extracted according to the corresponding splitting result and the label.
[0021] Optionally, grouping the target data based on the target key and storing the grouped target data in corresponding target nodes respectively includes:
[0022] A preset Flink selector is set according to the target key, and the target key is grouped by using the preset Flink selector and the preset grouping operator to obtain a corresponding grouping result;
[0023] storing the target data satisfying the target association condition in a preset state storage operator according to the grouping result;
[0024] Using a preset logical calculation operator, the target data in the preset state storage operator is secondary grouped according to the event type and the associated field data, so as to store the target data with the same event type and the associated field data in the same target node in a preset hierarchical tree structure.
[0025] Optionally, matching the to-be-matched data with the target data in any target node based on the event type and the associated field data includes:
[0026] Determining a condition type corresponding to the preset association judgment condition in the preset association model;
[0027] Based on the event type and the associated field data, and using a matching method corresponding to the condition type, the data to be matched is matched with the target data in any target node in a matching method corresponding to the condition type.
[0028] Optionally, after determining whether to associate the to-be-matched data with the target data in any target node based on the matching result, the method further includes:
[0029] Recording the data cleaning time point corresponding to the to-be-matched data in the target node;
[0030] Acquire the current system working time point, and perform timeout judgment on the to-be-matched data and the target data in the target node based on the current system working time point and the data cleaning time point;
[0031] If the data to be matched and the target data have timed out, the data to be matched and the target data are deleted from the target node.
[0032] In a second aspect, the present application discloses a data time series association device, which is applied to the Flink distributed real-time processing engine, including:
[0033] A target key generation module, used to extract target association conditions that meet preset conditions from a preset association model, and generate a target key corresponding to the target data based on the target association conditions;
[0034] A data storage module, used for grouping the target data based on the target key, and storing the grouped target data in corresponding target nodes respectively; wherein the target data stored in the same target node have the same event type and associated field data;
[0035] The association judgment module is used to match the data to be matched with the target data in any target node based on the event type and the associated field data when the data to be matched is obtained, and determine whether to associate the data to be matched with the target data in any target node based on the matching result.
[0036] In a third aspect, the present application discloses an electronic device, comprising:
[0037] Memory, used to store computer programs;
[0038] A processor is used to execute the computer program to implement the aforementioned data timing association method.
[0039] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned data timing association method when executed by a processor.
[0040] This application first extracts the target association conditions that meet the preset conditions from the preset association model, and generates the target key corresponding to the target data based on the target association conditions, and then groups the target data based on the target key, and stores the grouped target data in the corresponding target nodes respectively, wherein the target data stored in the same target node have the same event type and associated field data, and finally, when the data to be matched is obtained, the data to be matched is matched with the target data in any target node based on the event type and the associated field data, and it is determined whether to associate the data to be matched with the target data in any target node based on the matching result. It can be seen that this application realizes the classification of target data of the same type by storing target data with the same event type and associated field data in the same target node. When new data enters, it only needs to match once for the target data of the same type, avoiding the problem of multiple windows consuming data multiple times for time series association analysis, thereby solving the problem of large performance bottlenecks and waste of computing resources in the time series association analysis process under certain resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.
[0042] Figure 1 A flow chart of a data time series association method disclosed in this application;
[0043] Figure 2 A flowchart of a data time series association method disclosed in this application;
[0044] Figure 3 A schematic diagram of a tree structure disclosed in this application;
[0045] Figure 4 A data time series association and data cleaning flow chart disclosed in this application;
[0046] Figure 5 A specific data time series association and data cleaning flow chart disclosed in this application;
[0047] Figure 6 A specific data time series association and data cleaning flow chart disclosed in this application;
[0048] Figure 7 A specific data time series association and data cleaning flow chart disclosed in this application;
[0049] Figure 8 A specific data time series association and data cleaning flow chart disclosed in this application;
[0050] Fig. 9 This is a schematic diagram of the structure of a data timing association device disclosed in this application;
[0051] Fig.10 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION
[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0053] In the prior art, when using the Flink distributed real-time processing engine to perform data time series association analysis, if there are many association models with different association modes and time windows, multiple windows will consume data multiple times for time series association analysis, resulting in a waste of computing resources and reduced data association efficiency. To this end, the present application provides a data time series association method, which avoids repeated data judgment by storing target data with the same association fields and data types in the same node, thereby improving the efficiency of data time series association.
[0054] See also Figure 1As shown, an embodiment of the present invention discloses a data time series association method, which is applied to the Flink distributed real-time processing engine, including:
[0055] Step S11: extracting target association conditions that meet preset conditions from a preset association model, and generating target keys corresponding to target data based on the target association conditions.
[0056] In this embodiment, the overall process of data timing association is as follows: Figure 2 As shown, before extracting the target association condition that meets the preset condition from the preset association model, it specifically includes: processing the real-time middleware data through the Flink distributed real-time processing engine to access the initial data; using the preset data parsing method to parse the initial data after access, so as to filter the initial data based on the corresponding parsing results; if the initial data meets the preset screening condition, the initial data that meets the preset screening condition is determined as the target data, and correspondingly, the data that does not meet the preset screening condition can be directly discarded; it can be understood that the data is accessed by consuming the real-time middleware data set through the Flink distributed real-time processing engine. Once the data access is completed, the data can be parsed by operator definition, parser development, etc. In a specific implementation, a custom Map operator can be written, which receives the input data stream, parses and converts it according to specific rules, and maps each element in the input data stream to the parsed data object. In another specific implementation, a parser can be defined to parse the original data into the required fields and attributes according to the format and structure of the data. The parser can be a function or a class, which is appropriately designed and implemented as needed.
[0057] In this embodiment, before extracting the target association condition that meets the preset condition from the preset association model, it also includes: adding a label of the event type corresponding to the target data to the target data, so as to extract the target association condition that meets the preset condition from the preset association model according to the label, and the above operation of adding the label of the event type corresponding to the target data to the target data is completed by the filtering operator; in a specific implementation, firstly, the filtering configuration of all association models and the mapping relationship between the model ID () and the filtering identifier are obtained through the database. The filtering identifier can be obtained by hash calculation from the filtering configuration. Then, the expression engine in the filtering operator is used to calculate and determine whether a single piece of data meets the filtering configuration of a single or multiple association models. If none of them are satisfied, the data is discarded, indicating that the data does not participate in the calculation of the current association model. If a single or multiple filtering configurations are satisfied, the data adds a filtering result field, and assigns a value to the corresponding single or multiple event types eventType, and after the assignment, it is output to the downstream for the next calculation.
[0058] In this embodiment, extracting target association conditions that meet preset conditions from the preset association model can specifically include: splitting the preset association model containing target data according to the preset association judgment conditions in the preset association model; extracting target association conditions that meet the preset conditions according to the corresponding splitting results and labels; after obtaining the target association conditions, generating a corresponding target key, i.e., correKey, according to the target association conditions. In a specific implementation, if the preset association model contains an association condition containing ==, that is, a target association condition, the association condition containing == and the remaining association conditions are first split according to the association judgment condition, the association condition containing == is extracted, and the correKey (ruleId+equalFiledValue) is generated. For example, for the association model with the following association conditions: eventA.src==eventB.dest ANDeventA.requestBody contains eventB.requestDomain, the AiQL expression (Artificial Intelligence Query Language) is first split according to AND into == and non-== expressions, where the == expression is eventA.src==eventB.dest, and the field for generating the correKey for event A is the ruleId of the model + the src field value of the eventA event, and the field for generating the correKey for event B is the ruleId of the model + the dest field value of the eventB event. The remaining association condition eventA.requestBodycontains eventB.requestDomain is used as the association judgment condition after the preset grouping operator keyBy is distributed; in another specific implementation, if the association model does not contain the association condition of ==, the generated correKey only contains ruleId to ensure that the data to be matched that conforms to the association model are in the same concurrent operator after the preset grouping operator keyBy; for example, the association model with the following association condition, eventA.requestBody containseventB.requestDomain, then the fields corresponding to event A and event B that generate correKey are both the ruleId of the model, to ensure that the event data of eventA and eventB that conform to the model are in the same concurrent operator after keyBy. In this embodiment, by generating correKey in different ways according to the association conditions in the association model, it can be ensured that the data to be matched that conforms to the association model are in the same concurrent operator after the preset grouping operator keyBy.
[0059] Step S12: grouping the target data based on the target key, and storing the grouped target data in corresponding target nodes respectively; wherein the target data stored in the same target node have the same event type and associated field data.
[0060] In this embodiment, the target data is grouped according to the target key, and the grouped target data is stored in the corresponding target node. Specifically, it may include: setting a preset Flink selector according to the target key, and using the preset Flink selector and the preset grouping operator to group the target key to obtain the corresponding grouping result; storing the target data that meets the target association condition in the same preset state storage operator keyedStat according to the grouping result; using the preset logical calculation operator keyedProcess, the target data in the preset state storage operator is secondary grouped according to the event type and the associated field data, so that the target data with the same event type and associated field data are stored in the same target node in the preset hierarchical tree structure. In a specific implementation, the flink keyselector selector, that is, the preset Flink selector, is used to group correKey according to keyby, and the target data that meets the same event type and the same associated field is stored in the same keyedStat, and then the keyedProcess operator is used to group the target data in keyedStat again, so that the data with the same eventType and the same associated field valueMap is stored in Figure 3 The same target node shown. In the above keyedProcess operator, all data with the same associated model Id that meets a certain type of event can be summarized, and association judgments can be made according to different association methods corresponding to the model. For example, if the model has a corresponding == field, then the event data that meets the == association condition is summarized in the current operator, and the remaining association conditions are supplemented and judged later. In this embodiment, by storing data with the same eventType and the same associated field valueMap in the same target node, when the data is judged for association, it only needs to be judged once for the target data with the same data type and associated field data, rather than repeating the judgment for all data, thereby reducing the number of data matches.
[0061] Step S13: When the data to be matched is obtained, the data to be matched is matched with the target data in any target node based on the event type and the associated field data, and based on the matching result, it is determined whether to associate the data to be matched with the target data in any target node.
[0062] In this embodiment, after obtaining the data to be matched corresponding to the target data, the data to be matched is matched with the target data in the target node based on the time type and associated field data of the target data and the data to be matched, which can specifically include: determining the condition type corresponding to the preset association judgment condition in the preset association model; based on the event type and the associated field data, and using the matching method corresponding to the condition type, matching the data to be matched with the target data in any target node in a matching method corresponding to the condition type. In a specific implementation, if the preset association judgment condition is of the follow_by type, the association judgment logic corresponding to the follow_by type is used to perform data timing association matching; in another specific implementation, if the preset association judgment condition is of the AND type, the association judgment logic corresponding to the AND type is used to perform data timing association matching; in a third specific implementation, if the preset association judgment condition is of the not_follow_by type, the association judgment logic corresponding to the not_follow_by type is used to perform data timing association matching; in a fourth specific implementation, if the preset association judgment condition is of the reapeat_until type, the association judgment logic corresponding to the reapeat_until type is used to perform data timing association matching; accordingly, after associating the data to be matched with the target data in any target node based on the matching result, it specifically also includes: recording the data cleaning time point corresponding to the data to be matched in the target node; obtaining the current system working time point, and performing timeout judgment on the data to be matched and the target data in the target node based on the current system working time point and the data cleaning time point; if the data to be matched and the target data have timed out, deleting the timed out data to be matched and the target data from the target node. By regularly cleaning up the data in the target node, the timeliness of the data in the node is guaranteed, and storage resources are released to a certain extent, avoiding the waste of computer-related resources.
[0063] It can be seen that the present application realizes the classification of target data of the same type by storing target data with the same event type and associated field data in the same target node. When new data enters, it only needs to be matched once for the target data of the same type, avoiding the problem of multiple windows consuming data multiple times for time series correlation analysis. By regularly cleaning the data in the node after the data time series is correlated, the timeliness of the data in the node is guaranteed, and storage resources are released to a certain extent, avoiding the waste of computer-related resources, and solving the problem of large performance bottlenecks and waste of computing resources in the time series correlation analysis process under certain resources.
[0064] To further explain the matching method according to the condition type of the association judgment condition, and the detailed process of data temporal association matching between the matching data and the target data, see Figure 4 As shown, the present application also discloses a method for data time series association based on the condition type of the association judgment condition, which specifically includes:
[0065] Step S21: Determine a condition type corresponding to a preset association judgment condition in a preset association model.
[0066] In this embodiment, it is necessary to first determine the condition type corresponding to the preset association judgment condition from the preset association model, so as to perform data time series association between the target data and the data to be matched according to the data processing method corresponding to the condition type, and perform data cleaning after the data association.
[0067] Step S22: using a processing method corresponding to the condition type to perform data time series association on the data to be matched and the target data in any target node using a processing method corresponding to the condition type, and performing data cleaning after the data association.
[0068] It should be noted that different types of association judgment conditions have data processing methods corresponding to the condition types. In this embodiment, it is necessary to use the data processing method corresponding to the condition type of the association judgment condition to match the target data and the data to be matched and perform data cleaning operations.
[0069] In a specific embodiment, Figure 5As shown, if the condition type of the association judgment condition is follow_by, after obtaining the data to be matched, first determine whether the data to be matched belongs to eventA. If the data to be matched belongs to eventA, find out whether there is a target node with the same event type and associated field data as the target data stored internally. If not, add a node corresponding to the data to be matched, set the event type and matching expression corresponding to the node, and record the corresponding cleanup time of the data in the node; if it exists, add the data to be matched to the above target node, and continue to process the next data to be matched; if the data to be matched does not belong to eventA, calculate the previous data type of the data to be matched, and if Search the target node list that matches the data type of the previous time in the target node. If the target node list does not exist, discard the data to be matched. If the target node list exists, traverse the target node list, obtain the valueMap corresponding to any target node in the target node list, add the valueMap corresponding to the current data to be matched, and use the aviator expression to match the valueMap corresponding to the above target node with the valueMap corresponding to the data to be matched. If the match is successful, add the node corresponding to the data to be matched under the target node, set the event type and matching expression to be matched corresponding to the node, and record the data cleanup time in the node; if the match fails, discard the data to be matched. It should be noted that the follow_by condition type can support the association of two or more data. During the data cleaning process, if the data to be matched does not belong to eventA, the bottom-level node that meets the follow_by condition type is searched. If the bottom-level node does not exist, the next data to be matched is processed. If the bottom-level node exists, the parent node is searched upwards, the target data in the parent node is collected, and stored in the trigger result variable. The bottom-level node and the parent node are marked, and the marked nodes are deleted uniformly. If the current system working time point meets the corresponding data cleaning time point of the target node, the timeout data in the target node is deleted starting from the first level of the hierarchical tree. If the target node is empty, the target node is marked, and the marked nodes are deleted uniformly. Correspondingly, if Figure 6 , Figure 7 and Figure 8 As shown, when the condition type of the association judgment condition is AND, not_follow_by, and reapeat_until, the data timing association and data cleaning operations have a similar processing flow to those when the condition type of the association judgment condition is follow_by, and will not be repeated here.
[0070] It can be seen that in this application, when the data to be matched enters, the same type of target data only needs to be matched once, avoiding the problem of multiple windows consuming data multiple times for time series correlation analysis. By regularly cleaning the data in the node after the data time series is correlated, the timeliness of the data in the node is guaranteed, and storage resources are released to a certain extent, avoiding the waste of computer-related resources, and solving the problem of large performance bottlenecks and waste of computing resources in the time series correlation analysis process under certain resources.
[0071] See also Fig. 9 As shown, the embodiment of the present invention further discloses a data time series association device, which is applied to the Flink distributed real-time processing engine, including:
[0072] A target key generating module 11 is used to extract target association conditions that meet preset conditions from a preset association model, and generate target keys corresponding to target data based on the target association conditions;
[0073] A data storage module 12, used for grouping the target data based on the target key, and storing the grouped target data in corresponding target nodes respectively; wherein the target data stored in the same target node have the same event type and associated field data;
[0074] The association judgment module 13 is used to match the data to be matched with the target data in any target node based on the event type and the associated field data when the data to be matched is obtained, and determine whether to associate the data to be matched with the target data in any target node based on the matching result.
[0075] It can be seen that the present application realizes the classification of target data of the same type by storing target data with the same event type and associated field data in the same target node. When new data enters, it only needs to be matched once for the target data of the same type, avoiding the problem of multiple windows consuming data multiple times for time series correlation analysis. By regularly cleaning the data in the node after the data time series is correlated, the timeliness of the data in the node is guaranteed, and storage resources are released to a certain extent, avoiding the waste of computer-related resources, and solving the problem of large performance bottlenecks and waste of computing resources in the time series correlation analysis process under certain resources.
[0076] In a specific implementation, the target key generation module 11 further includes:
[0077] The data access unit is used to process the real-time middleware data through the Flink distributed real-time processing engine to access the initial data;
[0078] A data screening unit, used to perform data analysis on the initial data after access using a preset data analysis method, so as to screen the initial data based on the corresponding analysis result;
[0079] a data determination unit, configured to determine the initial data as the target data if the initial data meets a preset screening condition;
[0080] A label adding unit is used to add a label of the event type corresponding to the target data to the target data, so as to extract the target association condition that meets the preset condition from the preset association model according to the label.
[0081] In a specific implementation, the target key generation module 11 specifically includes:
[0082] A model splitting unit, used for splitting the preset association model containing the target data according to the preset association judgment condition in the preset association model;
[0083] The condition extraction unit is used to extract the target association condition that meets the preset condition according to the corresponding split result and the label.
[0084] In a specific implementation, the data storage module 12 specifically includes:
[0085] A target key grouping unit, used to set a preset Flink selector according to the target key, and group the target key using the preset Flink selector and the preset grouping operator to obtain a corresponding grouping result;
[0086] A data storage unit, configured to store the target data satisfying the target association condition in a preset state storage operator according to the grouping result;
[0087] A data grouping unit is used to use a preset logical calculation operator to perform secondary grouping of the target data in the preset state storage operator according to the event type and the associated field data, so as to store the target data with the same event type and the associated field data in the same target node in a preset hierarchical tree structure.
[0088] In a specific implementation, the association determination module 13 specifically includes:
[0089] A condition type determination unit, used to determine a condition type corresponding to the preset association judgment condition in the preset association model;
[0090] A data matching unit is used to match the to-be-matched data with the target data in any target node based on the event type and the associated field data and using a matching method corresponding to the condition type.
[0091] In a specific implementation, the association determination module 13 further includes:
[0092] A time point recording unit, used for recording the data cleaning time point corresponding to the to-be-matched data in the target node;
[0093] A timeout judgment unit, used for obtaining a current system working time point, and performing a timeout judgment on the to-be-matched data and the target data in the target node based on the current system working time point and the data cleaning time point;
[0094] A data deleting unit is used to delete the data to be matched and the target data from the target node if the data to be matched and the target data have timed out.
[0095] Furthermore, the present application also discloses an electronic device. Fig.10 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.
[0096] Fig.10 A schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the data timing association method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0097] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.
[0098] In addition, the memory 22 as a carrier for storing resources may be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon may include an operating system 221, a computer program 222, etc., and the storage method may be temporary storage or permanent storage.
[0099] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the data timing association method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.
[0100] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed data timing association method. For the specific steps of the method, reference may be made to the corresponding contents disclosed in the aforementioned embodiments, and no further description will be given here.
[0101] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.
[0102] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0103] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
[0104] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0105] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A data time series association method, characterized in that: Applied to the Flink distributed real-time processing engine, including: Extracting target association conditions that meet preset conditions from a preset association model, and generating target keys corresponding to target data based on the target association conditions; The target data are grouped based on the target key, and the grouped target data are respectively stored in corresponding target nodes; wherein the target data stored in the same target node have the same event type and associated field data; When the data to be matched is obtained, the data to be matched is matched with the target data in any target node based on the event type and the associated field data, and it is determined whether to associate the data to be matched with the target data in any target node based on the matching result.
2. The data time series association method according to claim 1, characterized in that: Before extracting the target association condition that meets the preset condition from the preset association model, the method further includes: Process the real-time middleware data through the Flink distributed real-time processing engine to access the initial data; Performing data analysis on the initial data after access using a preset data analysis method, so as to screen the initial data based on corresponding analysis results; If the initial data meets the preset screening condition, the initial data is determined as the target data.
3. The data time series association method according to claim 1, characterized in that: Before extracting the target association condition that meets the preset condition from the preset association model, the method further includes: A label of the event type corresponding to the target data is added to the target data, so as to extract the target association condition satisfying the preset condition from the preset association model according to the label.
4. The data time series association method according to claim 3, characterized in that: The step of extracting target association conditions that meet preset conditions from the preset association model includes: Splitting the preset association model containing the target data according to the preset association judgment condition in the preset association model; The target association condition satisfying the preset condition is extracted according to the corresponding splitting result and the label.
5. The data time series association method according to claim 1, characterized in that: The grouping the target data based on the target key and storing the grouped target data in corresponding target nodes respectively includes: A preset Flink selector is set according to the target key, and the target key is grouped by using the preset Flink selector and the preset grouping operator to obtain a corresponding grouping result; storing the target data satisfying the target association condition in a preset state storage operator according to the grouping result; Using a preset logical calculation operator, the target data in the preset state storage operator is secondary grouped according to the event type and the associated field data, so as to store the target data with the same event type and the associated field data in the same target node in a preset hierarchical tree structure.
6. The data time series association method according to claim 4, characterized in that: The matching the to-be-matched data with the target data in any target node based on the event type and the associated field data includes: Determining a condition type corresponding to the preset association judgment condition in the preset association model; Based on the event type and the associated field data, and using a matching method corresponding to the condition type, the data to be matched is matched with the target data in any target node in a matching method corresponding to the condition type.
7. The data time series association method according to any one of claims 1 to 6, characterized in that: After determining whether to associate the to-be-matched data with the target data in any target node based on the matching result, the method further includes: Recording the data cleaning time point corresponding to the to-be-matched data in the target node; Acquire the current system working time point, and perform timeout judgment on the to-be-matched data and the target data in the target node based on the current system working time point and the data cleaning time point; If the data to be matched and the target data have timed out, the data to be matched and the target data are deleted from the target node.
8. A data timing association device, characterized in that: Applied to the Flink distributed real-time processing engine, including: A target key generation module, used to extract target association conditions that meet preset conditions from a preset association model, and generate a target key corresponding to the target data based on the target association conditions; A data storage module, used for grouping the target data based on the target key, and storing the grouped target data in corresponding target nodes respectively; wherein the target data stored in the same target node have the same event type and associated field data; The association judgment module is used to match the data to be matched with the target data in any target node based on the event type and the associated field data when the data to be matched is obtained, and determine whether to associate the data to be matched with the target data in any target node based on the matching result.
9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the data timing association method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, implements the data timing association method as described in any one of claims 1 to 7.