A method, device, equipment and storage medium for processing unbounded stream data association

Abstract

The embodiment of the present invention discloses a method, device, equipment and storage medium for unbounded stream data association processing, the method comprising: obtaining a main data stream and an associated data stream associated with the main data stream; determining a main data acquisition path of the main data stream and an associated data acquisition path of the associated data stream, the main data acquisition path or the associated data acquisition path comprising a local cache, the local cache being used to store business data with a change frequency greater than a set frequency threshold; acquiring target main data based on the main data acquisition path, acquiring target associated data based on the associated data acquisition path; performing association processing on the target main data and the target associated data to obtain an association processing result. The method provided by the embodiment of the present invention ensures the accuracy of association of rapidly changing data and data sensitive to state changes, avoids data loss during data stream delays or network jitters, and improves the accuracy of data processing.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular to a method, device, equipment and storage medium for unbounded stream data association processing. Background Art

[0002] In stream processing applications, there are correlation relationships between data streams. In order to realize correlation analysis of mutually correlated data streams, a common processing method is to use a stream processing window to collect and receive data of data streams with correlation relationships, and perform data analysis between correlated streams based on the data received by the stream processing window.

[0003] In the process of implementing the present invention, the inventors found that there are at least the following technical problems in the prior art: when any data stream is delayed or the network jitters, it is impossible to obtain complete associated data, resulting in data loss and poor data processing effect. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, equipment and storage medium for processing unbounded stream data association, so as to ensure the integrity of associated data stream data and improve data processing effect.

[0005] In a first aspect, an embodiment of the present invention provides a method for associating unbounded stream data, comprising:

[0006] Acquire a main data stream and an associated data stream associated with the main data stream;

[0007] Determine a main data acquisition path of a main data stream and an associated data acquisition path of an associated data stream, wherein the main data acquisition path or the associated data acquisition path includes a local cache, and the local cache is used to store business data whose change frequency is greater than a set frequency threshold;

[0008] Acquire target master data based on the master data acquisition path, and acquire target associated data based on the associated data acquisition path;

[0009] The target master data and the target associated data are associated with each other to obtain an associated processing result.

[0010] In a second aspect, an embodiment of the present invention further provides an unbounded stream data association processing device, including:

[0011] A data stream acquisition module, used to acquire a main data stream and associated data streams associated with the main data stream;

[0012] An acquisition path determination module is used to determine a main data acquisition path of a main data stream and an associated data acquisition path of an associated data stream, wherein the main data acquisition path or the associated data acquisition path includes a local cache, and the local cache is used to store business data whose change frequency is greater than a set frequency threshold;

[0013] A data acquisition module, used to acquire target master data based on a master data acquisition path, and to acquire target associated data based on an associated data acquisition path;

[0014] The data processing module is used to perform association processing on the target master data and the target associated data to obtain an association processing result.

[0015] In a third aspect, an embodiment of the present invention further provides a computer device, the device comprising:

[0016] one or more processors;

[0017] A storage device for storing one or more programs;

[0018] When one or more programs are executed by one or more processors, the one or more processors implement the unbounded stream data association processing method provided by any embodiment of the present invention.

[0019] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the unbounded stream data association processing method provided by any embodiment of the present invention.

[0020] The embodiment of the present invention obtains a main data stream and an associated data stream associated with the main data stream; determines a main data acquisition path of the main data stream and an associated data acquisition path of the associated data stream, wherein the main data acquisition path or the associated data acquisition path includes a local cache, and the local cache is used to store business data whose change frequency is greater than a set frequency threshold, thereby ensuring the accuracy of association of rapidly changing data and data that is sensitive to state changes; obtains target main data based on the main data acquisition path, and obtains target associated data based on the associated data acquisition path; performs association processing on the target main data and the target associated data to obtain an association processing result, thereby avoiding data loss due to data stream delay or network jitter, and improving the accuracy of data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a flow chart of an unbounded stream data association processing method provided by Embodiment 1 of the present invention;

[0022] Figure 2a It is a principle diagram of an unbounded stream data association processing method provided by Embodiment 2 of the present invention;

[0023] Figure 2b This is a schematic diagram of a process of persistent storage of associated streams provided by Embodiment 2 of the present invention;

[0024] Figure 2c It is a schematic diagram of a flow chart of data stream association calculation provided by the second embodiment of the present invention;

[0025] Figure 3 It is a structural diagram of an unbounded stream data association processing device provided by Embodiment 3 of the present invention;

[0026] Figure 4 It is a structural diagram of a computer device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0027] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0028] Embodiment 1

[0029] Figure 1 1 is a flowchart of an unbounded stream data association processing method provided by the first embodiment of the present invention. This embodiment is applicable to the case where unbounded stream data is associated and processed. The method can be executed by an unbounded stream data association processing device, which can be implemented in software and / or hardware. For example, the unbounded stream data association processing device can be configured in a computer device. Figure 1 As shown, the method includes:

[0030] S110: Acquire a main data stream and an associated data stream associated with the main data stream.

[0031] In this embodiment, the main data stream and the associated data stream are data streams for performing association analysis. Both the main data stream and the associated data stream can be determined according to the data processing requirements. Taking the e-commerce platform as an example, if the data processing requirement is the analysis of the successful characteristics of the order transaction, considering that the browsing, collection, delivery speed, etc. of the items in the order may all have an associated relationship with the success of the order transaction, the order stream can be used as the main data stream, and the browsing stream, collection stream, logistics information stream, etc. can be used as the associated data stream. It can be understood that the association keys between the main data stream and different associated data streams may be different or the same. Taking the main data stream as the order stream and the associated data stream as the browsing stream, collection stream and logistics information stream as an example, the association key between the order stream and the browsing stream and the collection stream can be the item identifier, and the association key between the order stream and the logistics information stream can be the logistics identifier.

[0032] In one embodiment, obtaining a main data stream and an associated data stream associated with the main data stream includes: in response to a detected data processing request, determining the main data stream and an associated condition according to the data processing type of the data processing request; determining the associated data stream associated with the main data stream, and the associated key between the main data stream and the associated data stream according to the associated condition. Optionally, when a data processing request is detected, the main data stream and the associated data stream are determined according to the data processing request. Exemplarily, a user can manually set the main data stream and the associated condition, and initiate a data processing request with the above information. After receiving the data processing request, the unbounded stream data association processing device parses the data processing request to obtain the main data stream and the associated condition. The main data stream and the associated condition corresponding to the data processing type can also be pre-set. After the user initiates the data processing request, the unbounded stream data association processing device determines the main data stream and the associated condition of the data processing request based on the pre-set corresponding relationship according to the data processing type of the data processing request. Among them, the associated condition can be one or more. For example, the associated condition can be an item identification association, or the associated condition can be a user identification or an item identification association. After the association condition is obtained, at least one associated data stream can be determined according to the information carried in each data stream, and finally the association key between the main data stream and each associated data stream can be determined according to the association relationship between the main data stream and the associated data stream.

[0033] On the basis of the above scheme, the method also includes: determining the connection type between the main data stream and the associated data stream; selecting a data stream from the main data stream and the associated data stream as the storage data stream according to the connection type; using the association key as the storage primary key of the storage data stream, and storing the storage data stream based on the storage primary key. . When performing association analysis on the data in the main data stream and the associated data stream, it is necessary to aggregate and connect the data in the main data stream and the data in the associated data stream for processing. Different connection types correspond to different connection methods. In order to ensure that any connection method can obtain complete associated data, in this embodiment, it is determined whether the main data stream is persistently stored or the associated data stream is persistently stored according to the connection type. When performing persistent storage of the storage data stream, the table name, primary key and other information can be written in the file in the key-value format.

[0034] S120, determining a main data acquisition path of a main data stream and an associated data acquisition path of an associated data stream, wherein the main data acquisition path or the associated data acquisition path includes a local cache, and the local cache is used to store business data whose change frequency is greater than a set frequency threshold.

[0035] In this embodiment, in order to avoid data loss during data flow delay or network jitter and reduce the accuracy of data processing, the data of a certain data flow can be stored in the storage space. When processing data, data is obtained from the real-time data flow and the storage space for association processing. When storing data, it is considered that storing change-sensitive data in the passive cache will cause inconsistency in data status, such as the order status in the order table, which is constantly changing. If it is placed in the passive cache, assuming that the passive cache time is 30 minutes, the order status will not change within 30 minutes. Therefore, in order to ensure the accuracy of data storage, it is necessary to actively update the cache from the original data flow, that is, add an active cache layer in the local cache, and put data that is sensitive to changes (that is, business data with a storage change frequency greater than a set frequency threshold, or pre-set change-sensitive data), such as constantly changing and updated data, into the cache, and use the original data flow to continuously and actively update the active cache layer in the local cache. It should be noted that when updating the active cache with the original data flow, the same data table needs to be cached in the same instance, and the same shared cache is used for both reading and writing.

[0036] Optionally, change-sensitive data can be determined based on the change frequency of business data types or business scenarios, where the change frequency of business data types or business scenarios can be calculated based on historical data; change-sensitive data can also be set manually, such as according to the characteristics of business data scenarios. For example, in order status scenarios, changes are relatively fast, and downstream calculations need to quickly understand order changes, so order status data can be set as change-sensitive data; for example, product information, such as product name, weight, length, width, height, etc., the downstream can accept delays in these changes and can be set as change-insensitive data.

[0037] In one embodiment of the present invention, determining the main data acquisition path of the main data stream and the associated data acquisition path of the associated data stream includes: determining the main data acquisition path of the main data stream and the associated data acquisition path of the associated data stream according to the connection type of the main data stream and the associated data stream. Optionally, in order to ensure that different connection types can reasonably obtain complete associated data, different data stream processing methods are set for different connection types in this embodiment. Therefore, when acquiring data, different connection types and different data streams have different data acquisition paths.

[0038] Generally, the connection modes of the main data in the main data stream and the associated data in the associated data stream include inner join, left outer join, right outer join and full outer join. When the connection type is inner join, only the matching data in the main data stream and the associated data stream are aggregated into a combination; when the connection type is left outer join, all the data in the main data stream are aggregated into a combination; when the connection type is right outer join, all the data in the associated data stream are aggregated into a combination; when the connection type is full outer join, all the data in the main data stream and the associated data stream are aggregated into a combination. In order to ensure that different connection types can obtain complete data, the connection mode corresponding to the connection type selects a data stream from the main data stream and the associated data stream as a storage data stream for persistent storage. Specifically, when the connection type is inner join and right outer join, the associated data stream is used as the storage data stream, and the real-time data of the associated data stream is persistently stored. When the connection type is left outer join and full outer join, the main data stream is used as the storage data stream, and the real-time data of the main data stream is persistently stored.

[0039] On the basis of the above scheme, when the connection type is inner join and right outer join, the associated data stream is used as the storage data stream. Therefore, when the connection type is inner join and right outer join, the main data acquisition path of the main data stream and the associated data acquisition path of the associated data stream are determined according to the connection type of the main data stream and the associated data stream, including: determining that the main data acquisition path is a real-time data stream, the associated data acquisition path is a real-time data stream and a storage space, and the storage space includes a local cache and a persistent storage unit. Specifically, the data of the main data stream is obtained from the real-time data stream, and the data of the associated data stream is obtained from the real-time data stream and the storage space. Optionally, obtaining the data of the associated data stream from the storage space includes: obtaining the active cache data of the associated data stream in the active cache layer from the local cache and the persistent storage data of the associated data stream in the persistent storage unit.

[0040] When the connection type is a left outer join and a full outer join, the main data stream is used as the storage data stream, and the real-time data of the main data stream is persistently stored. Therefore, when the connection type is a left outer join and a full outer join, the main data acquisition path of the main data stream and the associated data acquisition path of the associated data stream are determined according to the connection type of the main data stream and the associated data stream, including: determining that the associated data acquisition path is a real-time data stream, the main data acquisition path is a real-time data stream and a storage space, and the storage space includes a local cache and a persistent storage unit. Specifically, the data of the associated data stream is obtained from the real-time data stream, and the data of the main data stream is obtained from the real-time data stream and the storage space. Optionally, obtaining the data of the main data stream from the storage space includes: obtaining the active cache data of the main data stream in the active cache layer from the local cache and the persistent storage data of the main data stream in the persistent storage unit.

[0041] S130 . Acquire target master data based on the master data acquisition path, and acquire target associated data based on the associated data acquisition path.

[0042] After the data acquisition path of each data stream is determined, the data of each data stream is acquired based on the determined data acquisition path.

[0043] When the acquisition path of the main data stream is the real-time data stream, and the acquisition path of the associated data stream is the real-time data stream and the storage space, the target associated data is acquired based on the associated data acquisition path, including: acquiring the association key between the main data stream and the associated data stream; using the association key as the primary key to acquire the real-time associated data from the real-time data stream, and determining the unmatched master data in the target master data that is not matched according to the target master data and the real-time associated data; using the association key of the unmatched master data as the primary key to acquire the storage associated data corresponding to the unmatched master data from the storage space; merging the real-time associated data and the storage associated data to remove duplicates, and obtaining the target associated data. Specifically, the target master data is acquired from the real-time data stream. The real-time associated data is acquired from the real-time data stream, and the unmatched master data in the target master data that is not matched with the real-time associated data is determined, and then the association key of the unmatched master data is used as the primary key to acquire the storage associated data corresponding to the unmatched master data from the storage space, and the real-time associated data and the storage associated data are merged to remove duplicates, and the target associated data is obtained. Optionally, the storage space may include a local cache and a persistent storage unit, wherein the local cache has a small space but a fast data processing speed, and the persistent storage unit has a large space but a relatively slow data processing speed. Accordingly, using the association key of the unmatched primary data as the primary key to obtain the storage-related data corresponding to the unmatched primary data from the storage space may be specifically as follows: traversing whether there is the association data corresponding to the association key in the local cache, and if so, using the association data as the storage-related data; if not, traversing whether there is the association data corresponding to the association key in the persistent storage unit, and if so, using the association data as the storage-related data, and storing the storage-related data in the local cache corresponding to the association key.

[0044] When the acquisition path of the associated data stream is the real-time data stream, and the acquisition path of the main data stream is the real-time data stream and the storage space, the target master data is acquired based on the main data acquisition path, including: acquiring the association key between the main data stream and the associated data stream; using the association key as the primary key to acquire the real-time master data from the real-time data stream, and determining the unmatched associated data in the target associated data according to the target associated data and the real-time master data; using the association key of the unmatched associated data as the primary key to acquire the storage master data corresponding to the unmatched associated data from the storage space; merging the real-time master data and the storage master data to remove duplicates, and obtaining the target master data. Specifically, the target associated data is acquired from the real-time data stream. The real-time master data is acquired from the real-time data stream, and the unmatched associated data in the target associated data that is not matched with the real-time master data is determined, and then the association key of the unmatched associated data is used as the primary key to acquire the storage master data corresponding to the unmatched associated data from the storage space, and the real-time master data and the storage master data are merged to remove duplicates, and the target associated data is obtained. Optionally, using the association key of the unmatched associated data as the primary key to obtain the storage master data corresponding to the unmatched associated data from the storage space can be specifically as follows: traversing whether there is data corresponding to the association key in the local cache, if so, using the data as the storage master data; if not, traversing whether there is data corresponding to the association key in the persistent storage unit, if so, using the data as the storage master data, and storing the storage master data and the association key in the local cache correspondingly.

[0045] S140: Perform association processing on the target master data and the target associated data to obtain an association processing result.

[0046] After obtaining the target master data and target associated data, the target master data and target associated data are stored in the memory database for associated calculation, and the calculation results are sent to the result stream. The specific associated calculation method can be determined based on the data processing requirements and will not be described here.

[0047] The embodiment of the present invention obtains a main data stream and an associated data stream associated with the main data stream; determines a main data acquisition path of the main data stream and an associated data acquisition path of the associated data stream, wherein the main data acquisition path or the associated data acquisition path includes a local cache, and the local cache is used to store business data whose change frequency is greater than a set frequency threshold, thereby ensuring the accuracy of association of rapidly changing data and data that is sensitive to state changes; obtains target main data based on the main data acquisition path, and obtains target associated data based on the associated data acquisition path; performs association processing on the target main data and the target associated data to obtain an association processing result, thereby avoiding data loss due to data stream delay or network jitter, and improving the accuracy of data processing.

[0048] Embodiment 2

[0049] This embodiment provides a preferred embodiment based on the above solution. This embodiment takes the associated data stream as a storage data stream as an example to illustrate the unbounded stream data association processing method.

[0050] Figure 2a FIG. 1 is a schematic diagram of a method for processing unbounded stream data association provided by Embodiment 2 of the present invention. Figure 2a As shown, when the main data stream 1 and the associated data stream 2 are associated in real time, they are first preprocessed. A persistent storage unit and an active cache layer are added to the processing flow of the associated data stream 2. The persistent storage unit stores the data in a persistent storage device, such as hbase. The active cache layer can be a local memory cache (such as guava cache), which puts data that is sensitive to changes, such as constantly changing and updated data, into the cache, and actively updates the cache with the original data flow. When the two streams are associated for calculation, the data loader is used to retrieve the unassociated data from the active cache and persistent storage. In this way, even if the data of the data stream is delayed, the data can still be obtained from the storage space after the data is sent for calculation, which ensures that all existing data in the two streams will be associated for calculation, so that data will not be lost, and by adding active cache, the associated calculation is accelerated and the reading of persistent storage is reduced.

[0051] Figure 2b This is a schematic diagram of a process for persistent storage of associated streams provided by the second embodiment of the present invention. Figure 2b As shown in the figure, the persistent storage of associated streams includes four parts: determining the primary key, updating the configuration file, deduplicating data within the window, and updating the persistent storage. Determine the primary key: First, according to the association conditions of the two stream association calculations, determine the association key used in each process as the primary key of the persistent storage. Update the configuration file: In order to facilitate the later task management, add a configuration file to record the table name, primary key and other information of the persistent storage corresponding to each real-time stream. After determining the primary key, you need to update the real-time stream name, persistent storage table name, primary key, used field information, etc. to the configuration file. Deduplication of data within the window: In order to reduce the interaction with the persistent storage, you can set a stream processing window (for example, a window of 2 seconds) and perform deduplication operations within the stream processing window. For the data with the same primary key, only the latest state of the data is retained, which can reduce the write operation to the persistent storage. Determine the table that can be actively cached: Write the deduplicated data to the active cache in the local cache and the persistent storage system.

[0052] Figure 2c FIG. 1 is a flow chart of a data stream association calculation provided by Embodiment 2 of the present invention. Figure 2c As shown in the figure, the data flow association calculation process mainly includes:

[0053] 1. Load the data of main data stream 1 and extract the associated key. With main data stream 1 as the mainstream, the data in the current window is organized into a data set format, such as Map1<key,data> The key in Map1 is the association key and also the primary key in the persistent storage.

[0054] 2. Load the associated data from persistent storage. Stream 2 needs associated data.

[0055] (1) Query the local cache by primary key. If the cache hits, store the associated data DataLoadStream2<key,data> Return and proceed to step 3.

[0056] If the cache does not hit, the missed data is marked as unassociated data UnJoinStream1 and the process goes to step (2).

[0057] (2) Traverse the associated key values ​​in UnJoinStream1, query the persistent storage, and obtain the persistent associated data of the hit data DataLoadStream2<key,data> .

[0058] (3) Load persistent associated data into DataLoadStream2<key,data> Write to the local cache so that the data can be hit in the local cache later, speeding up the processing.

[0059] 3. Data merging

[0060] Map1 of main data stream 1<key,data> , DataLoadStream2, the storage-related data loaded from cache or persistent storage<key,data> , merge to get the data set ToBeCalculatedMap1 that needs to be calculated<key,data> .

[0061] 4. Load the merged data set into the in-memory database.

[0062] 5. Perform association calculations in the in-memory database.

[0063] 6. Send the calculation results to the result stream.

[0064] The unbounded stream data association processing method provided in this embodiment has been tested and the active cache hit rate is more than 60%, which greatly reduces the access pressure of the HBase cluster.

[0065] In the process of real-time calculation of massive data, the embodiment of the present invention associates two or more streams through the stream processing window for calculation. When one of the streams is delayed or jittered, persistent storage is introduced to automatically load the unassociated data, thereby cleverly solving the problem that the data cannot be associated and is not sent down, resulting in less data in the stream. Active cache (local cache) is added on top of the persistent cache to ensure the accuracy of change-sensitive data, thereby improving the accuracy of data processing, effectively enhancing the robustness, stability and fault tolerance of real-time association calculation of massive data and multiple streams, and providing automation and intelligent convenience for the operation and maintenance of massive real-time calculation.

[0066] Embodiment 3

[0067] Figure 3 1 is a schematic diagram of the structure of an unbounded stream data association processing device provided by Embodiment 3 of the present invention. The unbounded stream data association processing device can be implemented in software and / or hardware, for example, the unbounded stream data association processing device can be configured in a computer device. Figure 3 As shown, the device includes a data flow acquisition module 310, an acquisition path determination module 320, a data acquisition module 330 and a data processing module 340, wherein:

[0068] A data stream acquisition module 310, configured to acquire a main data stream and associated data streams associated with the main data stream;

[0069] An acquisition path determination module 320 is used to determine a main data acquisition path of a main data stream and an associated data acquisition path of an associated data stream, wherein the main data acquisition path or the associated data acquisition path includes a local cache, and the local cache is used to store business data whose change frequency is greater than a set frequency threshold;

[0070] The data acquisition module 330 is used to acquire target master data based on the master data acquisition path and acquire target associated data based on the associated data acquisition path;

[0071] The data processing module 340 is used to perform association processing on the target master data and the target associated data to obtain an association processing result.

[0072] The embodiment of the present invention obtains a main data stream and an associated data stream associated with the main data stream; determines a main data acquisition path of the main data stream and an associated data acquisition path of the associated data stream, wherein the main data acquisition path or the associated data acquisition path includes a local cache, and the local cache is used to store business data whose change frequency is greater than a set frequency threshold, thereby ensuring the accuracy of association of rapidly changing data and data that is sensitive to state changes; obtains target main data based on the main data acquisition path, and obtains target associated data based on the associated data acquisition path; performs association processing on the target main data and the target associated data to obtain an association processing result, thereby avoiding data loss due to data stream delay or network jitter, and improving the accuracy of data processing.

[0073] Optionally, based on the above solution, the acquisition path determination module 320 is specifically used for:

[0074] A main data acquisition path of the main data stream and an associated data acquisition path of the associated data stream are determined according to the connection type of the main data stream and the associated data stream.

[0075] Optionally, based on the above solution, the connection type includes an inner connection and a right outer connection, and the acquisition path determination module 320 is specifically used for:

[0076] It is determined that the main data acquisition path is the real-time data stream, the associated data acquisition path is the real-time data stream and the storage space, and the storage space includes a local cache and a persistent storage unit.

[0077] Optionally, based on the above solution, the data acquisition module 330 is specifically used for:

[0078] Get the association key between the main data stream and the associated data stream;

[0079] The association key is used as the primary key to obtain the real-time association data from the real-time data stream, and the unmatched master data that is not matched in the target master data is determined according to the target master data and the real-time association data;

[0080] Using the association key of the unmatched master data as the primary key, the storage association data corresponding to the unmatched master data is obtained from the storage space;

[0081] The real-time associated data and the stored associated data are combined and deduplicated to obtain the target associated data.

[0082] Optionally, based on the above solution, the connection type includes a left outer connection and a full outer connection, and the acquisition path determination module 320 is specifically used for:

[0083] It is determined that the main data acquisition path is a real-time data stream and a storage space, the associated data acquisition path is a real-time data stream, and the storage space includes a local cache and a persistent storage unit.

[0084] Optionally, based on the above solution, the data acquisition module 330 is specifically used for:

[0085] Get the association key between the main data stream and the associated data stream;

[0086] The association key is used as the primary key to obtain the real-time master data from the real-time data stream, and the unmatched association data that is not matched in the target association data is determined according to the target association data and the real-time master data;

[0087] Using the association key of the unmatched associated data as the primary key, the storage primary data corresponding to the unmatched associated data is obtained from the storage space;

[0088] The real-time master data and the stored master data are combined and deduplicated to obtain the target master data.

[0089] Optionally, based on the above solution, the data stream acquisition module 310 is specifically used for:

[0090] In response to the detected data processing request, determining a main data flow and an associated condition according to a data processing type of the data processing request;

[0091] The associated data stream associated with the main data stream is determined according to the associated condition, and the associated key between the main data stream and the associated data stream is determined based on the associated condition.

[0092] Optionally, based on the above solution, the device further includes a storage data flow determination module, which is used to:

[0093] Determine the connection type between the main data stream and the associated data stream;

[0094] Selecting a data stream from the main data stream and the associated data stream as the storage data stream according to the connection type;

[0095] The association key is used as a storage primary key of the storage data stream, and the storage data stream is stored based on the storage primary key.

[0096] The unbounded stream data association processing device provided in the embodiment of the present invention can execute the unbounded stream data association processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0097] Embodiment 4

[0098] Figure 4 It is a structural diagram of a computer device provided in Embodiment 4 of the present invention. Figure 4 It is a structural diagram of a computer device provided in Embodiment 6 of the present invention. Figure 4 A block diagram of an exemplary computer device 412 suitable for use in implementing embodiments of the present invention is shown. Figure 4The computer device 412 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0099] like Figure 4 As shown, the computer device 412 is in the form of a general-purpose computing device. The components of the computer device 412 may include, but are not limited to: one or more processors 414, a system memory 428, and a bus 418 that connects different system components (including the system memory 428 and the processor 414).

[0100] Bus 418 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a local bus to processor 414, or a bus structure using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0101] The computer device 412 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 412, including volatile and non-volatile media, removable and non-removable media.

[0102] System memory 428 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 430 and / or cache memory 432. Computer device 412 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage device 434 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 not shown, usually called a "hard drive"). Although Figure 4 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 418 via one or more data medium interfaces. The memory 428 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0103] A program / utility 440 having a set (at least one) of program modules 442 may be stored, for example, in memory 428, such program modules 442 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 442 generally perform the functions and / or methods of the embodiments described herein.

[0104] The computer device 412 may also communicate with one or more external devices 414 (e.g., keyboard, pointing device, display 424, etc.), may also communicate with one or more devices that enable a user to interact with the computer device 412, and / or communicate with any device that enables the computer device 412 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed via input / output (I / O) interface 422. Furthermore, the computer device 412 may also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN) and / or public network, such as the Internet) via network adapter 420. As shown, the network adapter 420 communicates with other modules of the computer device 412 via bus 418. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the computer device 412, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0105] The processor 414 executes various functional applications and data processing by running the program stored in the system memory 428, for example, implementing the unbounded stream data association processing method provided in an embodiment of the present invention, the method comprising:

[0106] Acquire a main data stream and an associated data stream associated with the main data stream;

[0107] Determine a main data acquisition path of a main data stream and an associated data acquisition path of an associated data stream, wherein the main data acquisition path or the associated data acquisition path includes a local cache, and the local cache is used to store business data whose change frequency is greater than a set frequency threshold;

[0108] Acquire target master data based on the master data acquisition path, and acquire target associated data based on the associated data acquisition path;

[0109] The target master data and the target associated data are associated with each other to obtain an associated processing result.

[0110] Of course, those skilled in the art can understand that the processor can also implement the technical solution of the unbounded stream data association processing method provided by any embodiment of the present invention.

[0111] Embodiment 5

[0112] Embodiment 5 of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the unbounded stream data association processing method provided in the embodiment of the present invention is implemented. The method includes:

[0113] Acquire a main data stream and an associated data stream associated with the main data stream;

[0114] Determine a main data acquisition path of a main data stream and an associated data acquisition path of an associated data stream, wherein the main data acquisition path or the associated data acquisition path includes a local cache, and the local cache is used to store business data whose change frequency is greater than a set frequency threshold;

[0115] Acquire target master data based on the master data acquisition path, and acquire target associated data based on the associated data acquisition path;

[0116] The target master data and the target associated data are associated with each other to obtain an associated processing result.

[0117] Of course, the computer-readable storage medium provided by the embodiment of the present invention stores a computer program which is not limited to the above method operations, but can also execute related operations of the unbounded stream data association processing method provided by any embodiment of the present invention.

[0118] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0119] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0120] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0121] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0122] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for processing unbounded stream data association, characterized in that: include: Acquire a main data stream and an associated data stream associated with the main data stream; Determine a main data acquisition path for the main data stream and an associated data acquisition path for the associated data stream, wherein the main data acquisition path or the associated data acquisition path includes a local cache, and the local cache is used to store business data whose change frequency is greater than a set frequency threshold; Acquire target master data based on the master data acquisition path, and acquire target associated data based on the associated data acquisition path; Performing association processing on the target master data and the target associated data to obtain an association processing result; The step of acquiring a main data stream and an associated data stream associated with the main data stream includes: In response to the detected data processing request, determining a main data flow and an associated condition according to a data processing type of the data processing request; The associated data stream associated with the main data stream and the associated key between the main data stream and the associated data stream are determined according to the associated condition.

2. The method according to claim 1, characterized in that: The determining of the main data acquisition path of the main data stream and the associated data acquisition path of the associated data stream includes: A main data acquisition path of the main data stream and an associated data acquisition path of the associated data stream are determined according to the connection type of the main data stream and the associated data stream.

3. The method according to claim 2, characterized in that The connection type includes an inner connection and a right outer connection, and the determining, according to the connection type of the main data stream and the associated data stream, a main data acquisition path of the main data stream and an associated data acquisition path of the associated data stream includes: It is determined that the main data acquisition path is a real-time data stream, the associated data acquisition path is a real-time data stream and a storage space, and the storage space includes a local cache and a persistent storage unit.

4. The method according to claim 3, characterized in that The acquiring target associated data based on the associated data acquisition path includes: Obtaining an association key between the main data stream and the associated data stream; Using the association key as a primary key, acquiring real-time association data from a real-time data stream, and determining unmatched master data that is not matched in the target master data according to the target master data and the real-time association data; Using the association key of the unmatched master data as the primary key, acquiring the storage association data corresponding to the unmatched master data from the storage space; The real-time associated data and the stored associated data are combined and deduplicated to obtain the target associated data.

5. The method according to claim 2, characterized in that: The connection types include left outer join and full outer join, and the determining of the main data acquisition path of the main data stream and the associated data acquisition path of the associated data stream according to the connection types of the main data stream and the associated data stream includes: It is determined that the main data acquisition path is a real-time data stream and a storage space, the associated data acquisition path is a real-time data stream, and the storage space includes a local cache and a persistent storage unit.

6. The method according to claim 5, characterized in that The acquiring target master data based on the master data acquisition path includes: Obtaining an association key between the main data stream and the associated data stream; Using the association key as a primary key, acquiring real-time master data from a real-time data stream, and determining unmatched association data that is not matched in the target association data according to the target association data and the real-time master data; Using the association key of the unmatched associated data as a primary key, acquiring the stored primary data corresponding to the unmatched associated data from the storage space; The real-time master data and the stored master data are combined and deduplicated to obtain the target master data.

7. The method according to claim 1, characterized in that The method further comprises: Determining a connection type between the main data stream and the associated data stream; Selecting a data stream from the main data stream and the associated data stream as a storage data stream according to the connection type; The association key is used as a storage primary key of the storage data stream, and the storage data stream is stored based on the storage primary key.

8. An unbounded stream data association processing device, characterized in that: include: A data stream acquisition module, used to acquire a main data stream and an associated data stream associated with the main data stream; an acquisition path determination module, used to determine a main data acquisition path of the main data stream and an associated data acquisition path of the associated data stream, wherein the main data acquisition path or the associated data acquisition path includes a local cache, and the local cache is used to store business data whose change frequency is greater than a set frequency threshold; A data acquisition module, used to acquire target master data based on the master data acquisition path, and acquire target associated data based on the associated data acquisition path; A data processing module, used for performing association processing on the target master data and the target associated data to obtain an association processing result; Wherein, the data stream acquisition module is specifically used for: In response to the detected data processing request, determining a main data flow and an associated condition according to a data processing type of the data processing request; The associated data stream associated with the main data stream and the associated key between the main data stream and the associated data stream are determined according to the associated condition.

9. A computer device, characterized in that: The device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the unbounded stream data association processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the unbounded stream data association processing method as described in any one of claims 1 to 7 is implemented.

Images