Real-time multi-stream association method, system and device and storage medium
By monitoring and updating local status, judging and processing out-of-order data, and loading dimensional stream data from external storage in a timely manner for correlation, the problems of inaccurate data association and redundant calculation in the prior art are solved, and the efficiency of data processing is improved and accuracy.
Patent Information
- Application Number
- CN202410944868.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-12
AI Technical Summary
Existing real-time multi-stream association schemes, such as Flink, are difficult to ensure data accuracy, especially when processing duplicate data, which can generate redundant calculations, reduce efficiency, and if there is a delay in the dimensional stream data or is not the latest data, it will lead to association results. Inaccurate, affects the quality of data analysis.
By monitoring the arrival of real-time mainstream data and dimensional stream data, update the local status, and determine whether the mainstream data is out of order. If it is not out of order, query the local dimensional stream status. If it is missing, load the dimensional stream data from external storage for correlation. , ensure data integrity and accuracy, and preload target dimension stream data if necessary to optimize efficiency.
It improves the accuracy and overall performance of data associations, reduces dependence on external storage, ensures fast response and efficiency of data processing, and is especially suitable for scenarios with high data timeliness.
Smart Images

Figure CN118939692A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing, and in particular to a method, system, device and storage medium for real-time multi-stream association. Background Art
[0002] Multi-stream association refers to a technology that associates and synchronously processes multiple data streams or information streams in the field of data processing. Multi-stream association can synchronously process and analyze multiple data streams to ensure data consistency and accuracy. Currently, multi-stream association technology is widely used in finance, the Internet of Things and other fields.
[0003] Existing real-time association solutions, such as Flink, have some obvious defects: facing complex data association requirements, Flink is difficult to guarantee data accuracy, and redundant calculations will be generated when processing duplicate data, reducing overall efficiency. In addition, if the dimension stream data is stored in an external storage system, although the storage capacity can be expanded, when the real-time data stream arrives, if the dimension stream data is delayed or the stored data is not the latest data, the association results will be inaccurate, affecting the quality of the final data analysis. Summary of the invention
[0004] The present application provides a real-time multi-stream association method, system, device and storage medium for ensuring the accuracy of data association and improving the overall performance of data association.
[0005] The first aspect of the present application provides a method for real-time multi-stream association, comprising:
[0006] Monitor the arrival of real-time mainstream data and real-time dimensional stream data, and update the local mainstream state and local dimensional stream state according to the arrival;
[0007] When new mainstream data arrives, judging whether the mainstream data is out-of-order data according to the local mainstream state;
[0008] If the mainstream data is not disordered data, query the local dimension stream state to see whether there is dimension stream data matching the mainstream data;
[0009] If the local dimensional stream state does not contain the dimensional stream data, loading the dimensional stream data into the local dimensional stream state from an external storage, wherein the external storage contains the dimensional stream data written in real time;
[0010] If the dimensional stream data exists in the local dimensional stream state, the mainstream data is associated with the dimensional stream data, and the association result is output.
[0011] Optionally, judging whether the mainstream data is out-of-order data according to the local mainstream state includes:
[0012] Compare a first update time of cached data in the local mainstream state with a second update time of the mainstream data;
[0013] If the second update time of the mainstream data is later than the first update time of the cache data, it is determined that the mainstream data is not out-of-order data;
[0014] If the second update time of the mainstream data is earlier than the first update time of the cache data, it is determined that the mainstream data is out-of-order data.
[0015] Optionally, after determining whether the mainstream data is out-of-order data according to the local mainstream state, the method further includes:
[0016] If the mainstream data is out-of-order data, the mainstream data is discarded.
[0017] Optionally, after monitoring the arrival of real-time mainstream data and real-time dimensional stream data, the method further includes:
[0018] Filtering target dimension stream data in external storage according to preset conditions, wherein the preset conditions include that the data volume is less than a preset data volume, the data growth rate is less than a preset growth rate, and / or the association frequency is higher than a preset frequency;
[0019] Preload the target dimension stream data into the local dimension stream state.
[0020] Optionally, after associating the mainstream data with the dimensional stream data and outputting the association result, the method further includes:
[0021] Determine whether the mainstream data is factual data, where the factual data is static mainstream data that cannot be changed;
[0022] If so, the memory of the local mainstream state is released.
[0023] Optionally, associating the mainstream data with the dimensional stream data and outputting the association result includes:
[0024] When the primary keys of the mainstream data and the dimensional stream data are the same, the mainstream data is associated with the dimensional stream data, and the association result is output.
[0025] A second aspect of the present application provides a system for real-time multi-stream association, including:
[0026] A monitoring unit, used to monitor the arrival of real-time mainstream data and real-time dimensional stream data, and update the local mainstream state and local dimensional stream state according to the arrival;
[0027] a judging unit, configured to judge whether new mainstream data is out-of-order data according to the local mainstream state when new mainstream data arrives;
[0028] A query unit, configured to query the local dimensional stream state whether there is dimensional stream data matching the mainstream data when the judgment unit determines that the mainstream data is not out-of-order data;
[0029] An external loading unit, used for loading the dimensional stream data from an external storage into the local dimensional stream state when the dimensional stream data does not exist in the local dimensional stream state, wherein the external storage contains the dimensional stream data written in real time;
[0030] An associating unit is used to associate the mainstream data with the dimensional stream data when the dimensional stream data exists in the local dimensional stream state, and output an association result.
[0031] Optionally, the judging unit is specifically configured to:
[0032] Compare a first update time of cached data in the local mainstream state with a second update time of the mainstream data;
[0033] If the second update time of the mainstream data is later than the first update time of the cache data, it is determined that the mainstream data is not out-of-order data;
[0034] If the second update time of the mainstream data is earlier than the first update time of the cache data, it is determined that the mainstream data is out-of-order data.
[0035] Optionally, the judging unit is further configured to:
[0036] When it is determined that the mainstream data is out-of-order data, the mainstream data is discarded.
[0037] Optionally, the system further comprises: a preloading unit;
[0038] The preloading unit is used for:
[0039] Filtering target dimension stream data in external storage according to preset conditions, wherein the preset conditions include that the data volume is less than a preset data volume, the data growth rate is less than a preset growth rate, and / or the association frequency is higher than a preset frequency;
[0040] Preload the target dimension stream data into the local dimension stream state.
[0041] Optionally, the system further includes: a memory release unit;
[0042] The memory release unit is used for:
[0043] Determine whether the mainstream data is factual data, where the factual data is static mainstream data that cannot be changed;
[0044] If so, the memory of the local mainstream state is released.
[0045] Optionally, the association unit is specifically used for:
[0046] When the primary keys of the mainstream data and the dimensional stream data are the same, the mainstream data is associated with the dimensional stream data, and the association result is output.
[0047] A third aspect of the present application provides a device for real-time multi-stream association, the device comprising:
[0048] Processor, memory, input-output unit, and bus;
[0049] The processor is connected to the memory, the input and output unit, and the bus;
[0050] The memory stores a program, and the processor calls the program to execute the first aspect and any optional method of real-time multi-stream association in the first aspect.
[0051] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the method for real-time multi-stream association according to the first aspect and any optional method in the first aspect is performed.
[0052] It can be seen from the above technical solutions that this application has the following advantages:
[0053] Through the real-time processing program, both real-time mainstream data and real-time dimension stream data are obtained and placed in their respective local state memories. When new mainstream data arrives, it is first determined whether the mainstream data is out-of-order data. If it is not out-of-order, the dimension stream data is first queried from the local dimension stream state for association to ensure a fast response. If the required dimension stream data is missing in the local dimension stream state, the dimension stream data in the external storage is queried for association, thereby ensuring the integrity and accuracy of data association, optimizing data access efficiency, reducing dependence on external storage, and improving the overall performance of data processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0055] Figure 1 A flowchart of an embodiment of a method for real-time multi-stream association provided by the present application;
[0056] Figure 2 A schematic diagram of the processing flow of the real-time processing program and the data writing program in the real-time multi-stream association method provided by the present application;
[0057] Figure 3 A schematic diagram of an embodiment of out-of-order data judgment in the real-time multi-stream association method provided by the present application;
[0058] Figure 4 A schematic flow chart of an embodiment of a preloading mechanism in the real-time multi-stream association method provided by the present application;
[0059] Figure 5 A flowchart of an embodiment of a memory release mechanism in the real-time multi-stream association method provided by the present application;
[0060] Figure 6 A schematic diagram of the structure of an embodiment of a system for real-time multi-stream association provided by the present application;
[0061] Figure 7 A schematic diagram of the structure of an embodiment of a device for real-time multi-stream association provided by the present application. DETAILED DESCRIPTION
[0062] The present application provides a real-time multi-stream association method, system, device and storage medium for ensuring the accuracy of data association and improving the overall performance of data association.
[0063] It should be noted that the real-time multi-stream association method provided in this application can be applied to a terminal or a server. For example, the terminal can be a smart phone or a computer, a tablet computer, a smart TV, a smart watch, a portable computer terminal, or a fixed terminal such as a desktop computer. For the convenience of explanation, this application uses a real-time processing program running on a terminal as an example for execution.
[0064] See also Figure 1 , Figure 1 An embodiment of a real-time multi-stream association method provided by the present application includes:
[0065] S101, monitor the arrival of real-time mainstream data and real-time dimensional stream data, and update the local mainstream state and local dimensional stream state according to the arrival;
[0066] The real-time processing program connects the captured data stream to a stream processing framework such as Flink to realize real-time data processing. In the stream processing framework, local mainstream state and local dimensional stream state are maintained for real-time mainstream data and real-time dimensional stream data respectively. The local mainstream state refers to a set of latest data states maintained in memory for real-time mainstream data (usually core business data, such as transaction records, event logs, etc.), which are usually indexed by the unique identifier of the data (such as primary key or event ID). The local dimensional stream state refers to the state maintained in memory for real-time dimensional stream data (usually used to describe the characteristics and attributes of data, such as time, location, product details, user information, etc., usually as context or supplementary information associated with the data). These dimensional stream data are crucial to understanding mainstream data.
[0067] Specifically, when the real-time processing program starts, the local dimension stream state is initialized, and it is prepared to receive and store real-time mainstream data and real-time dimension stream data, and the input channels of real-time mainstream data and real-time dimension stream data are monitored to capture newly arrived data. When the real-time processing program monitors the arrival of new real-time mainstream data or real-time dimension stream data, the real-time processing program checks whether it exists in the corresponding local state. If it does not exist, the data is written to the local state as a new record. If it already exists, it is determined whether it is an update record based on the timestamp or version number of the data, and the corresponding update operation is performed. By monitoring real-time mainstream data and real-time dimension stream data and updating the corresponding local state, it can be ensured that the data can reflect the latest situation.
[0068] S102: When new mainstream data arrives, determine whether the mainstream data is out-of-order data based on the local mainstream status;
[0069] Out-of-order data refers to the situation in which the order in which data packets arrive in a data stream is inconsistent with the logical order in which they are generated or should be processed. In a distributed system, due to network delays or instability, data packets may be transmitted at different speeds, causing later generated data to arrive before earlier generated data, resulting in out-of-order data. In addition, different data items may have different processing times in the system, and some data items may be delayed due to more complex processing logic, which can also lead to the generation of out-of-order data. The association between mainstream data and dimensional stream data depends on time order or a specific order. Out-of-order data may lead to incorrect data matching, seriously affecting the accuracy of data association.
[0070] Based on this, the real-time processing program retrieves all records associated with the newly arrived mainstream data from the local mainstream state, and determines whether the newly arrived mainstream data is out-of-order data based on these records. If it is not out-of-order data, the subsequent step S103 is executed, thereby maintaining the timing accuracy of the mainstream data and the logical consistency of subsequent data associations.
[0071] It should be noted that for newly arrived dimension stream data, the real-time processing program also needs to determine whether the dimension stream data is out-of-order data. If it is not out-of-order data, the local dimension stream state is updated according to the newly arrived dimension stream data. If it is out-of-order data, the data is directly discarded. The specific judgment method for out-of-order dimension stream data is the same as that for mainstream data, and will not be explained in detail here.
[0072] S103, querying whether there is dimension stream data matching the mainstream data in the local dimension stream state;
[0073] If the real-time processing program determines that the newly arrived mainstream data is not out-of-order data, it is necessary to associate the mainstream data. The real-time processing program first queries the local dimension stream state to check whether there is dimension stream data associated with the mainstream data. Specifically, the real-time processing program can use the user ID, product ID and other information in the mainstream data as query conditions to find matching dimension stream data in the local dimension stream state, which is not limited here.
[0074] If the real-time processing program determines that the local dimension stream state does not have dimension stream data that matches the mainstream data, step 104 is executed to perform external writing. If the real-time processing program determines that the local dimension stream state already has dimension stream data that matches the mainstream data, step 105 is executed to associate the mainstream data with the dimension stream data.
[0075] 104. If the local dimension stream state does not contain dimension stream data, load the dimension stream data from the external storage into the local dimension stream state, where the external storage contains the dimension stream data written in real time;
[0076] In this embodiment, a data writing program is also provided to access a stream processing framework such as Flink, which is responsible for writing the dimensional stream data into external storage in real time. The external storage may be a storage medium such as Doris and HBase that supports real-time reading and writing of massive data. It is suitable for long-term storage of large amounts of data and supports complex queries and analysis, so as to maintain the real-time nature of data writing and the reliability of data query even in the case of huge amounts of data.
[0077] The real-time processing program establishes a connection with the external storage. When the dimensional stream data that matches the mainstream data is not found in the local dimensional stream state, the required dimensional stream data is queried and loaded from the external storage. The real-time processing program loading the dimensional stream data specifically refers to loading the dimensional stream data transmitted by the external storage into the local dimensional stream state, and updating the local dimensional stream state information. Furthermore, before loading the dimensional stream data from the external storage, data deduplication and verification can also be performed to ensure the consistency and accuracy of the loaded dimensional stream data. After the dimensional stream data is loaded from the external storage into the local dimensional stream state, the local dimensional stream state exists in the dimensional stream data that matches the mainstream data, and step 105 is then executed to perform data association.
[0078] S105. If there is dimensional stream data in the local dimensional stream state, the mainstream data is associated with the dimensional stream data, and the association result is output.
[0079] By matching the dimension stream data queried in the local dimension stream state with the mainstream data, the association logic is implemented in the real-time processing program, and the matched dimension stream data is associated with the mainstream data to form a complete data record, and then the association result is formatted into the required output format, such as JSON, XML or insert statements of a specific database, and output to the predetermined target, such as a database, data warehouse, message queue or file system for downstream use, such as bi reports, interface calls, etc. Through this association operation, information from different data streams can be integrated to provide more comprehensive data services, including but not limited to report display, providing data for business systems to obtain risk control strategies, marketing plans, etc.
[0080] Since the real-time processing program obtains mainstream data and dimensional stream data at the same time, the real-time performance of the association can be improved. No matter which side has data delay or disorder, as long as the mainstream data / dimensional stream data arrives, the corresponding local dimensional stream state / local mainstream state has data, it can be immediately associated and output. This is particularly suitable for scenarios with high requirements for data timeliness, such as financial transactions, real-time monitoring and other systems.
[0081] See also Figure 2 , Figure 2 The processing flow diagram of the real-time processing program and the data writing program provided for this embodiment takes the Flink stream processing framework as an example. The data writing program writes the dimension stream data from the external dimension stream real-time data source to the external storage through the Flink stream processing framework. The real-time processing program writes the mainstream data from the external mainstream real-time data source to the local mainstream state, and also writes the dimension stream data from the external dimension stream real-time data source to the local dimension stream state, and then associates the mainstream data with the dimension stream data in real time through the Flink stream processing framework to obtain the association result.
[0082] In some specific embodiments, mainstream data and dimensional stream data can be associated based on primary key information. Specifically, when the primary keys of mainstream data and dimensional stream data are the same, mainstream data and dimensional stream data are associated and the association result is output. The primary key specifically refers to a field, such as user ID, product ID, etc., or it can be a composite primary key formed by a combination of multiple fields, which is not limited here. In mainstream data, the primary key may be a transaction record ID, event ID, etc., while in dimensional stream data, the primary key may be a user ID, product ID, etc. The primary key is used to associate mainstream data with dimensional stream data. By matching the corresponding primary keys in the two data, the dimensional stream information can be associated with specific business events or records. Using the primary key as the basis for association can simplify the association logic, making the code easier to implement and maintain.
[0083] In this embodiment, both real-time mainstream data and real-time dimension stream data are obtained through the real-time processing program and placed in their respective local state memories. When new mainstream data arrives, it is first determined whether the mainstream data is out-of-order data. If it is not out-of-order, the dimension stream data is first queried from the local dimension stream state for association to ensure a fast response. If the required dimension stream data is missing in the local dimension stream state, the dimension stream data in the external storage is queried for association, thereby ensuring the integrity and accuracy of the data association, optimizing data access efficiency, reducing dependence on external storage, and improving the overall performance of data processing.
[0084] In step S102 of the above embodiment, it is specifically possible to determine whether the mainstream data is out-of-order data based on the first update time of the cached data in the local mainstream state and the second update time of the mainstream data. Figure 3 , Figure 3 An embodiment for determining whether mainstream data is out-of-order data includes:
[0085] S301, comparing a first update time of cached data in a local mainstream state with a second update time of mainstream data;
[0086] When new mainstream data arrives, the real-time processing program first determines whether the new mainstream data exists in the local mainstream state. If not, the local mainstream state is updated according to the new mainstream data. If there is cached data in the local mainstream state that is consistent with the newly arrived mainstream data, the update time of the cached data and the mainstream data in the local mainstream state is compared, that is, the timestamp of the newly arrived mainstream data (second update time) is compared with the timestamp of the local cached data (first update time).
[0087] S302: If the second update time of the mainstream data is later than the first update time of the cache data, it is determined that the mainstream data is not out-of-order data;
[0088] If the timestamp (second update time) of the newly arrived mainstream data is updated, that is, later, it means that the newly arrived mainstream data is not out-of-order data. At this time, the old record in the local mainstream state can be replaced, and the local mainstream state can be updated with the newly arrived mainstream data. At this time, if there is dimension stream data matching the newly arrived mainstream data in the local dimension stream state, the association operation with the dimension stream data may also be triggered.
[0089] S303: If the second update time of the mainstream data is earlier than the first update time of the cache data, it is determined that the mainstream data is out-of-order data;
[0090] If the timestamp (second update time) of the newly arrived mainstream data is older, that is, earlier, it is considered to be a disordered data. The disordered data can be processed according to the preset business rules, and step S304 can be executed to directly discard it, or it can be cached for a period of time and then judged after the subsequent data arrives.
[0091] For example:
[0092] Mainstream data 1: primary key '1', data 1, data update time: 2024-06-12 08:08:08, Flink processing time: 2024-06-12 08:08:09;
[0093] Mainstream data 2: primary key '1', data 2, data update time: 2024-6-12 08:07:07, Flink processing time: 2024--06-12 08:08:10;
[0094] When mainstream data 2 reaches Flink for processing, its second update time (2024-6-12 08:07:07) is earlier than the first update time of mainstream data 1 (2024-06-12 08:08:08), indicating that mainstream data 2 is no longer the latest data. Mainstream data 1 is the latest data, but it reaches Flink for processing earlier than mainstream data 2, indicating that mainstream data 2 is data that arrived out of order and is invalid.
[0095] S304: discard mainstream data.
[0096] In this embodiment, if the newly arrived mainstream data is out-of-order data, the mainstream data is directly discarded, thereby avoiding the use of invalid or outdated mainstream data for association, reducing erroneous analysis and decision-making caused by out-of-order data, and alleviating the system's load cost for further processing of these out-of-order data.
[0097] In this embodiment, by comparing the first update time of the cached data in the local mainstream state with the second update time of the mainstream data, it is determined whether it is out-of-order data, thereby effectively identifying out-of-order data and ensuring the timing accuracy and logical consistency of the data processing flow. In addition, the out-of-order data is discarded in a timely manner to avoid erroneous data association and analysis results, thereby improving data quality.
[0098] It should be noted that the processing method for dimensional stream data is similar to the above steps S301 to S304. When new dimensional stream data arrives, if the local dimensional stream state has no data, the local dimensional stream state is updated. Otherwise, the update time of the newly arrived dimensional stream data and the local dimensional stream state are compared. If the update time of the newly arrived dimensional stream data is later, the local dimensional stream state is updated. Otherwise, it means that this dimensional stream state is out-of-order data and has expired, and it is directly discarded.
[0099] In the above embodiment, when the real-time processing program is started, the dimension stream data that meets the preset conditions can also be preloaded from the external storage to the local dimension stream state to improve the association efficiency. Figure 4 , Figure 4 An embodiment of preloading dimension stream data, the embodiment comprising:
[0100] S401, filtering target dimension stream data in external storage according to preset conditions, where the preset conditions include that the data volume is less than a preset data volume, the data growth rate is less than a preset growth rate, and / or the correlation frequency is higher than a preset frequency;
[0101] S402. Preload the target dimension stream data into the local dimension stream state.
[0102] In this embodiment, when the real-time processing program is started, the target dimension stream data that meets the preset conditions can be preloaded into the local memory, thereby reducing external storage queries and improving association efficiency.
[0103] Specifically, target dimension stream data that meets the preset conditions usually includes:
[0104] 1. The data volume is smaller than the preset data volume, that is, the dimensional stream data with a small data volume (within thousands);
[0105] 2. The data growth rate is lower than the preset growth rate, that is, the dimensional stream data with a slower data growth rate, such as product configuration table and regional information table;
[0106] 3. The correlation frequency is higher than the preset frequency, that is, the dimensional stream data with a higher correlation frequency.
[0107] Such screening ensures that only the most relevant and frequently accessed target dimension stream data is preloaded into the local dimension stream state, thereby reducing the query demand for external storage and network latency, reducing the overall response time of the system, and further improving the real-time and efficiency of data association. In addition, this preloading mechanism also helps to maintain stable operation and high-performance output of the system when data reaches a peak or external storage access is limited.
[0108] In the above embodiment, after the mainstream data is associated with the dimension stream data and the associated result is output, resources can be saved by releasing the local mainstream state memory. Figure 5 , Figure 5 An embodiment of releasing local mainstream state memory includes:
[0109] S501, determining whether the mainstream data is factual data, where factual data is static mainstream data that cannot be changed;
[0110] S502: Release the memory of the local mainstream state.
[0111] In this embodiment, considering that there are some factual data in the business scenario, this type of factual data is static data that cannot be changed, such as point-of-use log data, repayment flow, etc. Once these factual data are generated in the database, they will not change again. Therefore, once the association process is completed, these data no longer need to be retained in the memory. Based on this, after associating the mainstream data with the dimension stream data and outputting the association result, this embodiment will also execute step S501 to determine whether the mainstream data is factual data. If the mainstream data is factual data, step S502 is executed to release the memory of the local mainstream state. The purpose of this is to effectively manage resource usage and avoid memory waste, thereby improving the overall resource utilization and efficiency of the system, making the system more lightweight and responsive.
[0112] The following is a detailed description of the real-time multi-stream association system provided by this application. Figure 6 , Figure 6 Another embodiment of the system for real-time multi-stream association provided by the present application includes:
[0113] The monitoring unit 601 is used to monitor the arrival of real-time mainstream data and real-time dimensional stream data, and update the local mainstream state and local dimensional stream state according to the arrival;
[0114] A judging unit 602 is used to judge whether the mainstream data is out-of-order data according to the local mainstream state when new mainstream data arrives;
[0115] A query unit 603 is used to query whether there is dimension stream data matching the mainstream data in the local dimension stream state when the judgment unit determines that the mainstream data is not disordered data;
[0116] The external loading unit 604 is used to load the dimension stream data from the external storage into the local dimension stream state when the dimension stream data does not exist in the local dimension stream state, and the external storage contains the dimension stream data written in real time;
[0117] The associating unit 605 is used to associate the mainstream data with the dimensional stream data when there is dimensional stream data in the local dimensional stream state, and output the association result.
[0118] Optionally, the determining unit 602 is specifically configured to:
[0119] Compare a first update time of the cached data in the local mainstream state with a second update time of the mainstream data;
[0120] If the second update time of the mainstream data is later than the first update time of the cache data, it is determined that the mainstream data is not out-of-order data;
[0121] If the second update time of the mainstream data is earlier than the first update time of the cache data, it is determined that the mainstream data is out-of-order data.
[0122] Optionally, the determining unit 602 is further configured to:
[0123] When it is determined that the mainstream data is out-of-order data, the mainstream data is discarded.
[0124] Optionally, the system further includes: a preloading unit 606;
[0125] The preloading unit 606 is used for:
[0126] Filter target dimension stream data in external storage according to preset conditions, where the preset conditions include that the data volume is less than a preset data volume, the data growth rate is less than a preset growth rate, and / or the association frequency is higher than a preset frequency;
[0127] Preload the target dimension stream data into the local dimension stream state.
[0128] Optionally, the system further includes: a memory release unit 607;
[0129] The memory release unit 607 is used for:
[0130] Determine whether mainstream data is factual data. Factual data is static mainstream data that cannot be changed.
[0131] If so, release the memory of the local mainstream state.
[0132] Optionally, the association unit 605 is specifically configured to:
[0133] When the primary keys of the mainstream data and the dimension stream data are the same, the mainstream data is associated with the dimension stream data and the associated result is output.
[0134] In this embodiment, the functions of each unit are the same as those described above. Figure 1 , Figures 3 to 5 The steps in the method embodiment shown correspond to each other and will not be repeated here.
[0135] This application also provides a device for real-time multi-stream association, see Figure 7 , Figure 7 An embodiment of a device for real-time multi-stream association provided by the present application includes:
[0136] Processor 701, memory 702, input and output unit 703, bus 704;
[0137] The processor 701 is connected to the memory 702, the input and output unit 703 and the bus 704;
[0138] The memory 702 stores a program, and the processor 701 calls the program to execute any of the above methods for real-time multi-stream association.
[0139] The present application also relates to a computer-readable storage medium, on which a program is stored. When the program is run on a computer, the computer executes any of the above methods for real-time multi-stream association.
[0140] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0141] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0142] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0143] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0144] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.
Claims
1. A method for real-time multi-stream association, characterized in that: The method comprises: Monitor the arrival of real-time mainstream data and real-time dimensional stream data, and update the local mainstream state and local dimensional stream state according to the arrival; When new mainstream data arrives, judging whether the mainstream data is out-of-order data according to the local mainstream state; If the mainstream data is not disordered data, query the local dimension stream state to see whether there is dimension stream data matching the mainstream data; If the local dimensional stream state does not contain the dimensional stream data, loading the dimensional stream data into the local dimensional stream state from an external storage, wherein the external storage contains the dimensional stream data written in real time; If the dimensional stream data exists in the local dimensional stream state, the mainstream data is associated with the dimensional stream data, and the association result is output.
2. The method according to claim 1, characterized in that The determining whether the mainstream data is out-of-order data according to the local mainstream state includes: Compare a first update time of cached data in the local mainstream state with a second update time of the mainstream data; If the second update time of the mainstream data is later than the first update time of the cache data, it is determined that the mainstream data is not out-of-order data; If the second update time of the mainstream data is earlier than the first update time of the cache data, it is determined that the mainstream data is out-of-order data.
3. The method according to claim 1, characterized in that After determining whether the mainstream data is out-of-order data according to the local mainstream state, the method further includes: If the mainstream data is out-of-order data, the mainstream data is discarded.
4. The method according to claim 1, characterized in that: After monitoring the arrival of the real-time mainstream data and the real-time dimensional stream data, the method further includes: Filtering target dimension stream data in external storage according to preset conditions, wherein the preset conditions include that the data volume is less than a preset data volume, the data growth rate is less than a preset growth rate, and / or the association frequency is higher than a preset frequency; Preload the target dimension stream data into the local dimension stream state.
5. The method according to any one of claims 1 to 4, characterized in that After associating the mainstream data with the dimensional stream data and outputting the association result, the method further includes: Determine whether the mainstream data is factual data, where the factual data is static mainstream data that cannot be changed; If so, the memory of the local mainstream state is released.
6. The method according to any one of claims 1 to 4, characterized in that The associating the mainstream data with the dimensional stream data and outputting the associating result includes: When the primary keys of the mainstream data and the dimensional stream data are the same, the mainstream data is associated with the dimensional stream data, and the association result is output.
7. A real-time multi-stream association system, characterized in that: The system comprises: A monitoring unit, used to monitor the arrival of real-time mainstream data and real-time dimensional stream data, and update the local mainstream state and local dimensional stream state according to the arrival; a judging unit, configured to judge whether new mainstream data is out-of-order data according to the local mainstream state when new mainstream data arrives; A query unit, configured to query the local dimensional stream state whether there is dimensional stream data matching the mainstream data when the judgment unit determines that the mainstream data is not out-of-order data; An external loading unit, used for loading the dimensional stream data from an external storage into the local dimensional stream state when the dimensional stream data does not exist in the local dimensional stream state, wherein the external storage contains the dimensional stream data written in real time; An associating unit is used to associate the mainstream data with the dimensional stream data when the dimensional stream data exists in the local dimensional stream state, and output an association result.
8. The system according to claim 7, characterized in that The judging unit is specifically used for: Compare a first update time of cached data in the local mainstream state with a second update time of the mainstream data; If the second update time of the mainstream data is later than the first update time of the cache data, it is determined that the mainstream data is not out-of-order data; If the second update time of the mainstream data is earlier than the first update time of the cache data, it is determined that the mainstream data is out-of-order data.
9. A device for real-time multi-stream association, characterized in that: The device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the method according to any one of claims 1 to 6.
10. A computer-readable storage medium having a program stored thereon, wherein the program, when executed on a computer, performs the method according to any one of claims 1 to 6.
