Method, device and data processing system for determining unassociated data in data stream

Abstract

The present application provides a method, device, and data processing system for determining unassociated data in a data stream. The method includes: obtaining the data time of a first data object and the data time of a second data object; data-associating a second data object whose data time is within a first predetermined range with the corresponding first data object, thereby updating the association status of the associated first data object from unassociated to associated; data-associating a first data object whose data time is within a second predetermined range with the corresponding second data object, thereby updating the association status of the associated second data object from unassociated to associated; recording the first data object when the processing time of the first data object reaches a first predetermined processing time and the association status of the first data object is unassociated; and recording the second data object when the processing time of the second data object reaches a second predetermined processing time and the association status of the second data object is unassociated.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular, to a method, device, computer-readable storage medium, processor, and data processing system for determining unassociated data in a data stream. Background Art

[0002] When processing real-time data streams, due to the infinite nature of stream messages and the impact of message disorder, the time when messages that should be associated enter the processing system may vary greatly. At the same time, a message in one stream may need to be associated with multiple messages in another stream.

[0003] In this scenario, Flink natively provides a method called IntervalJoin to join two grouped data streams. This join method can connect two keyed streams (hereinafter referred to as the left stream and the right stream) based on the same key within a time period based on data time. Each record in the left or right stream is only associated with the data with the same key value in the same time period in the other stream. After the join, the time column of the input stream is retained, and time-based operations can continue.

[0004] The current mainstream technology for data processing scenarios where two data streams are associated based on a key and a time period mainly uses the IntervalJoin operator natively provided by Flink. Its implementation mainly involves the following steps:

[0005] 1) Connect two data streams so that they can share state and access data with the same key.

[0006] 2) Perform an associated operator on the connected data stream. The operator defines the actual data processing logic and two state objects (of type MapState) to cache the data of the two streams. The key of the state object corresponds to the timestamp of the data, and the value corresponds to the data with the same timestamp.

[0007] 3) Determine the delay of data records in the data stream. If the data time is less than the current watermark value, the data is considered delayed and not processed. Otherwise, the data is added to the corresponding cache status object with the key being the data time.

[0008] 4) Loop through the data objects in another state object (that is, the data elements in another data stream). If the data time of the data object is within a certain time interval before or after the data time of this data object (the actual value of the time interval can be passed in through the operator parameter), then the two data objects are considered to be associated, and the data processing logic is called for processing.

[0009] 5) Data objects that are not associated with any data in another stream after the previous traversal are not processed.

[0010] In summary, Flink's native intervaljoin operator directly discards data that does not meet the join conditions. The application that calls the operator cannot discover and further process this data.

[0011] The above information disclosed in the background technology section is only used to enhance the understanding of the background technology of the technology described in this article. Therefore, the background technology may contain certain information that does not form the prior art known in this country to those skilled in the art. Summary of the Invention

[0012] The main purpose of this application is to provide a method, device, computer-readable storage medium, processor and data processing system for determining unrelated data in a data stream, so as to solve the problem that unrelated data cannot be discovered in the prior art.

[0013] According to one aspect of an embodiment of the present invention, a method for determining unassociated data in a data stream is provided, comprising: obtaining a data time of a first data object and a data time of a second data object, wherein the first data object is a data object of a first key data stream, the second data object is a data object of a second key data stream, and the data time is a time point when the first data object or the second data object is generated; data-associating the second data object whose data time is within a first predetermined range with the corresponding first data object, so that the association status of the associated first data object is updated from unassociated to associated, and the first predetermined range has a one-to-one correspondence with the first data object; data-associating the first data object whose data time is within a second predetermined range with the corresponding second data object, so that the association status of the associated second data object is updated from unassociated to associated, and the second predetermined range has a one-to-one correspondence with the second data object; when the processing time of the first data object reaches a first predetermined processing time and the association status of the first data object is unassociated, recording the first data object, and the first predetermined processing time has a one-to-one correspondence with the first data object; when the processing time of the second data object reaches a second predetermined processing time and the association status of the second data object is unassociated, recording the second data object, and the second predetermined processing time has a one-to-one correspondence with the second data object.

[0014] Optionally, before associating the second data object whose data time is within the first predetermined range with the corresponding first data object, the method further includes: when the data time of the first data object is less than a first watermark time, locking the first data object so that the first data object cannot be data associated; when the data time of the second data object is less than a second watermark time, locking the second data object so that the second data object cannot be data associated.

[0015] Optionally, after data associating the second data object whose data time is within a first predetermined range with the corresponding first data object, so that the association status of the associated first data object is updated from unassociated to associated, the method further includes: storing the first data object and the first associated data object in a first associated data set of the first data object, the first associated data object being the second data object associated with the first data object; and performing data processing on the first associated data set.

[0016] Optionally, after data associating the first data object whose data time is within a second predetermined range with the corresponding second data object, so that the association status of the associated second data object is updated from unassociated to associated, the method further includes: storing the second data object and the second associated data object in a second associated data set of the second data object, the second associated data object being the first data object associated with the second data object; and performing data processing on the second associated data set.

[0017] Optionally, before obtaining the data time of the first data object and the data time of the second data object, the method further includes: grouping the data records of the first data stream and the data records of the second data stream according to the data time to obtain the first key data stream and the second key data stream, and the data object is a group of data records with the same data time.

[0018] Optionally, the minimum value of the first predetermined range is the difference between the data time of the first data object and the first time interval, the maximum value of the first predetermined range is the sum of the data time of the first data object and the second time interval, the minimum value of the second predetermined range is the difference between the data time of the second data object and the third time interval, and the maximum value of the second predetermined range is the sum of the data time of the second data object and the fourth time interval.

[0019] According to another aspect of an embodiment of the present invention, a device for determining unassociated data of a data stream is further provided, comprising: an acquisition unit, configured to acquire a data time of a first data object and a data time of a second data object, wherein the first data object is a data object of a first key data stream, the second data object is a data object of a second key data stream, and the data time is a time point at which the first data object or the second data object is generated; a first processing unit, configured to perform data association on the second data object whose data time is within a first predetermined range with the corresponding first data object, so that the association state of the associated first data object is updated from unassociated to associated, and the first predetermined range corresponds one-to-one to the first data object; and a second processing unit, configured to perform data association on the second data object whose data time is within a first predetermined range with the corresponding first data object. The first data object within a predetermined range is data-associated with the corresponding second data object, so that the association state of the associated second data object is updated from unassociated to associated, and the second predetermined range has a one-to-one correspondence with the second data object; a first recording unit is configured to record the first data object when the processing time of the first data object reaches a first predetermined processing time and the association state of the first data object is the unassociated, and the first predetermined processing time has a one-to-one correspondence with the first data object; and a second recording unit is configured to record the second data object when the processing time of the second data object reaches a second predetermined processing time and the association state of the second data object is the unassociated, and the second predetermined processing time has a one-to-one correspondence with the second data object.

[0020] According to yet another aspect of the embodiments of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium includes a stored program, wherein the program executes any one of the methods described above.

[0021] According to yet another aspect of the embodiments of the present invention, a processor is provided, wherein the processor is configured to run a program, wherein any one of the methods is executed when the program is run.

[0022] According to another aspect of an embodiment of the present invention, a data processing system is also provided, comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the methods described.

[0023] In an embodiment of the present invention, in the method for determining the unassociated data of the data stream, first, the data time of the first data object and the data time of the second data object are obtained, the first data object is a data object of the first key data stream, the second data object is a data object of the second key data stream, and the data time is the time point when the first data object or the second data object is generated; then, the second data object within the first predetermined range of the data time is data-associated with the corresponding first data object, so that the association status of the associated first data object is updated from unassociated to associated, and the first predetermined range corresponds to the first data object one by one; then, the second data object within the second predetermined range of the data time is data-associated with the corresponding first data object. The first data object is data-associated with the corresponding second data object, so that the association status of the associated second data object is updated from unassociated to associated, and the second predetermined range corresponds one-to-one with the second data object; then, when the processing time of the first data object reaches the first predetermined processing time and the association status of the first data object is unassociated, the first data object is recorded, and the first predetermined processing time corresponds one-to-one with the first data object; finally, when the processing time of the second data object reaches the second predetermined processing time and the association status of the second data object is unassociated, the second data object is recorded, and the second predetermined processing time corresponds one-to-one with the second data object. This method data-associates the data objects of the first key data stream and the second key data stream, and records the unassociated data objects when the corresponding predetermined processing time is reached, so as to facilitate subsequent further processing of the unassociated data objects, thereby avoiding direct discarding of the unassociated data objects during data processing, and solving the problem of the inability to discover unassociated data in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings that constitute part of this application are used to provide a further understanding of this application. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0025] Figure 1 A schematic flow chart of a method for determining unassociated data in a data stream according to an embodiment of the present application is shown;

[0026] Figure 2 A schematic structural diagram of a device for determining unassociated data in a data stream according to an embodiment of the present application is shown;

[0027] Figure 3 A schematic flow chart of a method for determining unassociated data in a data stream according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0029] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] It should be understood that when an element (such as a layer, film, region, or substrate) is described as being "on" another element, the element may be directly on the other element or intervening elements may be present. Moreover, in the specification and claims, when it is described that an element is "connected to" another element, the element may be "directly connected to" the other element or "connected to" the other element through a third element.

[0032] For ease of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0033] Flink: Apache Flink is an open-source stream processing framework developed by the Apache Software Foundation. Its core is a distributed streaming data flow engine written in Java and Scala. Flink executes arbitrary streaming programs in a data-parallel, pipelined manner. Flink's pipeline runtime system can execute both batch and stream processing programs. Furthermore, Flink's runtime itself supports iterative algorithms.

[0034] Keyed data streams, also known as KeyedStreams, represent data streams grouped by key values ​​generated according to specified rules. A data stream is a collection of data records that are infinite in both time distribution and number, and serves as an abstraction for unbounded datasets. A data record is the smallest unit of a data stream and corresponds to event information generated by various real-world data sources, such as transaction records, website page views / user visits, user-accessed content, and search results.

[0035] As mentioned in the background technology, the existing technology is unable to discover unrelated data. In order to solve the above problem, a typical embodiment of the present application provides a method, device, computer-readable storage medium, processor and data processing system for determining unrelated data in a data stream.

[0036] According to an embodiment of the present application, a method for determining unassociated data in a data stream is provided.

[0037] Figure 1 FIG. 1 is a flow chart of a method for determining unassociated data in a data stream according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0038] Step S101, obtaining a data time of a first data object and a data time of a second data object, wherein the first data object is a data object of a first key data stream, the second data object is a data object of a second key data stream, and the data time is a time point when the first data object or the second data object is generated;

[0039] Step S102: Associating the second data objects whose data time is within a first predetermined range with the corresponding first data objects, so that the association status of the associated first data objects is updated from unassociated to associated, and the first predetermined range corresponds to the first data objects one-to-one;

[0040] Step S103: Associating the first data object whose data time is within the second predetermined range with the corresponding second data object, so that the association status of the associated second data object is updated from unassociated to associated, and the second predetermined range corresponds to the second data object in a one-to-one manner;

[0041] Step S104: When the processing time of the first data object reaches the first predetermined processing time and the association status of the first data object is the unassociated state, record the first data object, and the first predetermined processing time corresponds to the first data object in a one-to-one manner;

[0042] Step S105 : When the processing time of the second data object reaches the second predetermined processing time and the association status of the second data object is the unassociated state, record the second data object, and the second predetermined processing time corresponds to the second data object in a one-to-one manner.

[0043] In the method for determining the unassociated data of the above-mentioned data stream, first, the data time of the first data object and the data time of the second data object are obtained, the above-mentioned first data object is a data object of the first key data stream, the above-mentioned second data object is a data object of the second key data stream, and the above-mentioned data time is the time point when the above-mentioned first data object or the above-mentioned second data object is generated; then, the above-mentioned second data object within the above-mentioned data time within a first predetermined range is data-associated with the corresponding above-mentioned first data object, so that the association status of the associated above-mentioned first data object is updated from unassociated to associated, and the above-mentioned first predetermined range corresponds to the above-mentioned first data object one by one; then, the above-mentioned first data object within the above-mentioned data time within the second predetermined range is data-associated with the corresponding above-mentioned first data object The method associates the data objects of the first key data stream with the corresponding second data object, so that the association status of the associated second data object is updated from unassociated to associated, and the second predetermined range corresponds one-to-one with the second data object; then, when the processing time of the first data object reaches the first predetermined processing time and the association status of the first data object is unassociated, the first data object is recorded, and the first predetermined processing time corresponds one-to-one with the first data object; finally, when the processing time of the second data object reaches the second predetermined processing time and the association status of the second data object is unassociated, the second data object is recorded, and the second predetermined processing time corresponds one-to-one with the second data object. The method associates the data objects of the first key data stream and the second key data stream, and records the unassociated data objects when the corresponding predetermined processing time is reached, so as to facilitate further processing of the unassociated data objects in the future, thereby avoiding direct discarding of the unassociated data objects during data processing, and solving the problem of the inability to discover unassociated data in the prior art.

[0044] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0045] Specifically, the first data object and the second data are correspondingly stored in a cache.

[0046] In a specific embodiment of the present application, when the association status of the above-mentioned data object is unassociated, after recording the above-mentioned first data object or the above-mentioned second data object, or after the above-mentioned data objects are associated, the above-mentioned first data object or the above-mentioned second data object is removed from the cache.

[0047] In one embodiment of the present application, before associating the second data object whose data time is within the first predetermined range with the corresponding first data object, the method further includes: if the data time of the first data object is less than the first watermark time, locking the first data object so that the first data object cannot be associated with the data; and if the data time of the second data object is less than the second watermark time, locking the second data object so that the second data object cannot be associated with the data. In actual applications, data objects may be delayed. In order to prevent data object delays from leading to misjudgment of data object status, in this embodiment, when the data time of the first data object is less than the first watermark time or when the data time of the second data object is less than the second watermark time, it is determined that the data object is delayed, and the data object is locked and no further processing is performed. In this case, the current data will not be associated with any data object.

[0048] In another embodiment of the present application, after associating the second data object within the first predetermined data time range with the corresponding first data object, thereby updating the association status of the associated first data object from unassociated to associated, the method further includes: storing the first data object and a first associated data object in a first associated data set of the first data object, the first associated data object being the second data object associated with the first data object; and performing data processing on the first associated data set. In this embodiment, after the data association, the first data object and the first associated data object are also stored in the first associated data set to facilitate subsequent data processing of the associated data by a user.

[0049] In another embodiment of the present application, after data associating the first data object within the second predetermined data time range with the corresponding second data object, thereby updating the association status of the associated second data object from unassociated to associated, the method further includes: storing the second data object and a second associated data object in a second associated data set for the second data object, the second associated data object being the first data object associated with the second data object; and performing data processing on the second associated data set. In this embodiment, after the data association, the second data object and the second associated data object are also stored in the second associated data set to facilitate subsequent data processing of the associated data by a user.

[0050] In another embodiment of the present application, before obtaining the data time of the first data object and the data time of the second data object, the method further includes: grouping the data records of the first data stream and the data records of the second data stream according to the data time to obtain the first key data stream and the second key data stream, and the data object is a group of the data records with the same data time. In order to achieve state sharing between the two data streams, in this embodiment, the first data stream and the second data stream are grouped according to the data time to obtain the first key data stream and the second key data stream, and the data object is a group of the data records with the same data time. The same data time means that the key value of the data object is the same, and data objects with the same key value can access each other, thereby achieving state sharing between the two data streams.

[0051] In another embodiment of the present application, the minimum value of the first predetermined range is the difference between the data time of the first data object and the first time interval, the maximum value of the first predetermined range is the sum of the data time of the first data object and the second time interval, the minimum value of the second predetermined range is the difference between the data time of the second data object and the third time interval, and the maximum value of the second predetermined range is the sum of the data time of the second data object and the fourth time interval. In this embodiment, if the data time of a data object is within a time interval before or after the data time of the current data object, then the two data objects are considered to be associated.

[0052] Specifically, the values ​​of the above time intervals are all greater than 0, and the values ​​of the above time intervals can be set according to the actual needs of the user.

[0053] The present embodiment also provides a device for determining unassociated data in a data stream. It should be noted that the device for determining unassociated data in a data stream according to the present embodiment can be used to execute the method for determining unassociated data in a data stream according to the present embodiment. The following describes the device for determining unassociated data in a data stream according to the present embodiment.

[0054] Figure 2 Schematic diagram of a device for determining unassociated data in a data stream according to an embodiment of the present application. Figure 2 As shown, the device includes:

[0055] An acquiring unit 10 is configured to acquire a data time of a first data object and a data time of a second data object, wherein the first data object is a data object of a first key data stream, the second data object is a data object of a second key data stream, and the data time is a time point at which the first data object or the second data object is generated;

[0056] a first processing unit 20 configured to associate the second data objects whose data time is within a first predetermined range with the corresponding first data objects, so that the association status of the associated first data objects is updated from unassociated to associated, and the first predetermined range corresponds one-to-one to the first data objects;

[0057] a second processing unit 30 configured to associate the first data objects whose data time is within a second predetermined range with the corresponding second data objects, so that the association status of the associated second data objects is updated from unassociated to associated, and the second predetermined range corresponds one-to-one to the second data objects;

[0058] A first recording unit 40 is configured to record the first data object when the processing time of the first data object reaches a first predetermined processing time and the association status of the first data object is the unassociated state, wherein the first predetermined processing time corresponds to the first data object in a one-to-one manner;

[0059] The second recording unit 50 is configured to record the second data object when the processing time of the second data object reaches a second predetermined processing time and the association status of the second data object is the unassociated state, wherein the second predetermined processing time corresponds to the second data object in a one-to-one manner.

[0060] The device for determining the unassociated data of the above-mentioned data stream includes an acquisition unit, a first processing unit, a second processing unit, a first recording unit, and a second recording unit, wherein the acquisition unit is used to acquire the data time of the first data object and the data time of the second data object, the above-mentioned first data object is a data object of the first key data stream, the above-mentioned second data object is a data object of the second key data stream, and the above-mentioned data time is the time point when the above-mentioned first data object or the above-mentioned second data object is generated; the first processing unit is used to data associate the above-mentioned second data object within the above-mentioned data time within a first predetermined range with the corresponding above-mentioned first data object, so that the association status of the associated above-mentioned first data object is updated from unassociated to associated, and the above-mentioned first predetermined range corresponds one-to-one with the above-mentioned first data object; the second processing unit is used to record the above-mentioned The first data object whose data time is within a second predetermined range is data-associated with the corresponding second data object, so that the association status of the associated second data object is updated from unassociated to associated, and the second predetermined range corresponds one-to-one with the second data object. The first recording unit is used to record the first data object when the processing time of the first data object reaches the first predetermined processing time and the association status of the first data object is unassociated, and the first predetermined processing time corresponds one-to-one with the first data object. The second recording unit is used to record the second data object when the processing time of the second data object reaches the second predetermined processing time and the association status of the second data object is unassociated, and the second predetermined processing time corresponds one-to-one with the second data object. The device data-associates data objects of the first key data stream and the second key data stream, and records the unassociated data objects when the corresponding predetermined processing time is reached, so as to facilitate subsequent processing of the unassociated data objects and avoid directly discarding the unassociated data objects during data processing, thereby solving the problem of the inability to discover unassociated data in the prior art.

[0061] Specifically, the first data object and the second data are correspondingly stored in a cache.

[0062] In a specific embodiment of the present application, when the association status of the data object is unassociated, after recording the first data object or the second data object, the first data object or the second data object is removed from the cache.

[0063] In one embodiment of the present application, the apparatus further comprises a first locking unit and a second locking unit, wherein the first locking unit is configured to, before associating the second data object whose data time is within the first predetermined range with the corresponding first data object, lock the first data object if the data time of the first data object is less than the first watermark time, so that the first data object cannot be data associated; and the second locking unit is configured to, before associating the second data object whose data time is within the first predetermined range with the corresponding first data object, lock the second data object if the data time of the second data object is less than the second watermark time, so that the second data object cannot be data associated. In actual applications, data objects may be delayed. In order to prevent data object delays from leading to misjudgment of data object status, in this embodiment, when the data time of the first data object is less than the first watermark time or when the data time of the second data object is less than the second watermark time, it is determined that the data object is delayed, and the data object is locked and no further processing is performed. In this case, the current data will not be associated with any data object.

[0064] In another embodiment of the present application, the apparatus further includes a first storage unit and a third processing unit, wherein the first storage unit is configured to, after associating the second data object within the first predetermined data time range with the corresponding first data object so that the association status of the associated first data object is updated from unassociated to associated, store the first data object and the first associated data object in a first associated data set of the first data object, wherein the first associated data object is the second data object associated with the first data object; and the third processing unit is configured to perform data processing on the first associated data set. In this embodiment, after the data association, the first data object and the first associated data object are also stored in the first associated data set to facilitate subsequent data processing of the associated data by the user.

[0065] In another embodiment of the present application, the apparatus further includes a second storage unit and a fourth processing unit, wherein the second storage unit is configured to, after associating the second data object within the first predetermined data time range with the corresponding first data object so that the association status of the associated first data object is updated from unassociated to associated, store the second data object and the second associated data object in a second associated data set of the second data object, wherein the second associated data object is the first data object associated with the second data object; and the fourth processing unit is configured to perform data processing on the second associated data set. In this embodiment, after the data association, the second data object and the second associated data object are also stored in the second associated data set to facilitate subsequent data processing of the associated data by the user.

[0066] In another embodiment of the present application, the device further includes a fifth processing unit, wherein the fifth processing unit is used to group the data records of the first data stream and the data records of the second data stream according to the data time before obtaining the data time of the first data object and the data time of the second data object, to obtain the first key data stream and the second key data stream, and the data object is a group of the data records with the same data time. In order to achieve state sharing between the two data streams, in this embodiment, the first data stream and the second data stream are grouped according to the data time to obtain the first key data stream and the second key data stream, and the data object is a group of the data records with the same data time. The same data time means that the key value of the data object is the same, and data objects with the same key value can access each other, thereby achieving state sharing between the two data streams.

[0067] In another embodiment of the present application, the minimum value of the first predetermined range is the difference between the data time of the first data object and the first time interval, the maximum value of the first predetermined range is the sum of the data time of the first data object and the second time interval, the minimum value of the second predetermined range is the difference between the data time of the second data object and the third time interval, and the maximum value of the second predetermined range is the sum of the data time of the second data object and the fourth time interval. In this embodiment, if the data time of a data object is within a time interval before or after the data time of the current data object, then the two data objects are considered to be associated.

[0068] Specifically, the values ​​of the above time intervals are all greater than 0, and the values ​​of the above time intervals can be set according to the actual needs of the user.

[0069] The device for determining the unassociated data of the above-mentioned data stream includes a processor and a memory. The above-mentioned acquisition unit, first processing unit, second processing unit, first recording unit, second recording unit, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0070] The processor includes a kernel, which retrieves the corresponding program unit from the memory. One or more kernels can be set, and the problem of being unable to discover unrelated data in the existing technology can be solved by adjusting the kernel parameters.

[0071] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0072] An embodiment of the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the method for determining unassociated data in the above-mentioned data stream is implemented.

[0073] An embodiment of the present invention provides a processor, which is used to run a program, wherein the method for determining unassociated data in the data stream is executed when the program is run.

[0074] An embodiment of the present invention provides a data processing system, comprising one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of the above methods.

[0075] The above-mentioned data processing system includes one or more processors, a memory and one or more programs, wherein the above-mentioned one or more programs are stored in the above-mentioned memory and are configured to be executed by the above-mentioned one or more processors. The above-mentioned one or more programs include a method for executing any one of the above-mentioned methods, which associates data objects of the first key data stream and the second key data stream, and records unassociated data objects when the corresponding predetermined processing time is reached, so as to facilitate subsequent further processing of the unassociated data objects, so as to avoid directly discarding the unassociated data objects during the data processing process, thereby solving the problem of the inability to discover unassociated data in the prior art.

[0076] An embodiment of the present invention provides a device, comprising a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, at least the following steps are performed:

[0077] Step S101, obtaining a data time of a first data object and a data time of a second data object, wherein the first data object is a data object of a first key data stream, the second data object is a data object of a second key data stream, and the data time is a time point when the first data object or the second data object is generated;

[0078] Step S102: Associating the second data objects whose data time is within a first predetermined range with the corresponding first data objects, so that the association status of the associated first data objects is updated from unassociated to associated, and the first predetermined range corresponds to the first data objects one-to-one;

[0079] Step S103: Associating the first data object whose data time is within the second predetermined range with the corresponding second data object, so that the association status of the associated second data object is updated from unassociated to associated, and the second predetermined range corresponds to the second data object in a one-to-one manner;

[0080] Step S104: When the processing time of the first data object reaches the first predetermined processing time and the association status of the first data object is the unassociated state, record the first data object, and the first predetermined processing time corresponds to the first data object in a one-to-one manner;

[0081] Step S105 : When the processing time of the second data object reaches the second predetermined processing time and the association status of the second data object is the unassociated state, record the second data object, and the second predetermined processing time corresponds to the second data object in a one-to-one manner.

[0082] The devices in this article can be servers, PCs, PADs, mobile phones, etc.

[0083] The present application also provides a computer program product, which, when executed on a data processing device, is adapted to execute a program for initializing at least the following method steps:

[0084] Step S101, obtaining a data time of a first data object and a data time of a second data object, wherein the first data object is a data object of a first key data stream, the second data object is a data object of a second key data stream, and the data time is a time point when the first data object or the second data object is generated;

[0085] Step S102: Associating the second data objects whose data time is within a first predetermined range with the corresponding first data objects, so that the association status of the associated first data objects is updated from unassociated to associated, and the first predetermined range corresponds to the first data objects one-to-one;

[0086] Step S103: Associating the first data object whose data time is within the second predetermined range with the corresponding second data object, so that the association status of the associated second data object is updated from unassociated to associated, and the second predetermined range corresponds to the second data object in a one-to-one manner;

[0087] Step S104: When the processing time of the first data object reaches the first predetermined processing time and the association status of the first data object is the unassociated state, record the first data object, and the first predetermined processing time corresponds to the first data object in a one-to-one manner;

[0088] Step S105 : When the processing time of the second data object reaches the second predetermined processing time and the association status of the second data object is the unassociated state, record the second data object, and the second predetermined processing time corresponds to the second data object in a one-to-one manner.

[0089] In order to enable those skilled in the art to more clearly understand the technical solution of the present disclosure, the technical solution of the present disclosure will be described in detail below with reference to specific embodiments and comparative examples.

[0090] Example

[0091] The flowchart of the method for determining the unassociated data of the data stream is as follows: Figure 3 As shown, the method includes the following steps:

[0092] Grouping the data records of the first data stream and the data records of the second data stream according to the data time to obtain the first key data stream and the second key data stream, and then associating the first key data stream and the second key data stream;

[0093] The association process is as follows:

[0094] Writing the first key data stream and the second key data stream into the buffer respectively, then determining whether the data object is delayed based on the data time, locking it if delayed, and associating the data object if not delayed, determining whether the data time of the data object is within a predetermined range, and if so, updating the state to associated, and then associating the data object. After associating the data object, removing the data object from the buffer;

[0095] When the processing time of the first data object reaches the first predetermined processing time or the processing time of the second data object reaches the second predetermined processing time, if there is an unassociated state in the associated state, the data object is recorded and then removed from the cache.

[0096] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0097] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the above-mentioned units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0098] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0099] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0100] If the above-mentioned integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a computer-readable storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the above-mentioned methods of each embodiment of the present invention. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0101] From the above description, it can be seen that the above embodiments of the present application achieve the following technical effects:

[0102] 1) The method for determining unassociated data of a data stream of the present application comprises: first, obtaining the data time of a first data object and the data time of a second data object, wherein the first data object is a data object of a first key data stream, the second data object is a data object of a second key data stream, and the data time is the time point when the first data object or the second data object is generated; then, the second data object within a first predetermined range of the data time is data-associated with the corresponding first data object, so that the association state of the associated first data object is updated from unassociated to associated, and the first predetermined range corresponds to the first data object one-to-one; then, the first data object within a second predetermined range of the data time is data-associated with the corresponding first data object. The method associates the data objects of the first key data stream with the corresponding second data object, so that the association status of the associated second data object is updated from unassociated to associated, and the second predetermined range corresponds one-to-one with the second data object; then, when the processing time of the first data object reaches the first predetermined processing time and the association status of the first data object is unassociated, the first data object is recorded, and the first predetermined processing time corresponds one-to-one with the first data object; finally, when the processing time of the second data object reaches the second predetermined processing time and the association status of the second data object is unassociated, the second data object is recorded, and the second predetermined processing time corresponds one-to-one with the second data object. The method associates the data objects of the first key data stream and the second key data stream, and records the unassociated data objects when the corresponding predetermined processing time is reached, so as to facilitate the subsequent further processing of the unassociated data objects, so as to avoid directly discarding the unassociated data objects during the data processing process, thereby solving the problem of the inability to discover unassociated data in the prior art.

[0103] 2) The device for determining unassociated data of a data stream of the present application includes an acquisition unit, a first processing unit, a second processing unit, a first recording unit, and a second recording unit, wherein the acquisition unit is used to acquire the data time of a first data object and the data time of a second data object, the first data object being a data object of a first key data stream, the second data object being a data object of a second key data stream, and the data time being the time point at which the first data object or the second data object is generated; the first processing unit is used to data associate the second data object within a first predetermined range of the data time with the corresponding first data object, so that the association state of the associated first data object is updated from unassociated to associated, and the first predetermined range corresponds one-to-one to the first data object; the second processing unit is used to The first data object whose data time is within a second predetermined range is data-associated with the corresponding second data object, so that the association status of the associated second data object is updated from unassociated to associated, and the second predetermined range corresponds one-to-one with the second data object; a first recording unit is used to record the first data object when the processing time of the first data object reaches a first predetermined processing time and the association status of the first data object is unassociated, and the first predetermined processing time corresponds one-to-one with the first data object; a second recording unit is used to record the second data object when the processing time of the second data object reaches a second predetermined processing time and the association status of the second data object is unassociated, and the second predetermined processing time corresponds one-to-one with the second data object. The device data-associates data objects of the first key data stream and the second key data stream, and records the unassociated data objects when the corresponding predetermined processing time is reached, so as to facilitate subsequent processing of the unassociated data objects and avoid directly discarding the unassociated data objects during data processing, thereby solving the problem of the inability to discover unassociated data in the prior art.

[0104] 3) The data processing system of the present application includes one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors. The one or more programs include a method for executing any one of the above methods, which associates data objects of the first key data stream and the second key data stream, and records unassociated data objects when the corresponding predetermined processing time is reached, so as to facilitate subsequent further processing of the unassociated data objects, so as to avoid directly discarding the unassociated data objects during the data processing process, thereby solving the problem of the inability to discover unassociated data in the prior art.

[0105] The above description is merely a preferred embodiment of the present application and is not intended to limit the present application. Persons skilled in the art will readily appreciate that various modifications and variations are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A method for determining unassociated data in a data stream, characterized in that: include: Obtaining a data time of a first data object and a data time of a second data object, where the first data object is a data object of a first key data stream, the second data object is a data object of a second key data stream, and the data time is a time point at which the first data object or the second data object is generated; Associating the second data object whose data time is within a first predetermined range with the corresponding first data object, so that the association status of the associated first data object is updated from unassociated to associated, and the first predetermined range corresponds to the first data object in a one-to-one manner; Associating the first data objects whose data time is within a second predetermined range with the corresponding second data objects, so that the association status of the associated second data objects is updated from unassociated to associated, and the second predetermined range corresponds to the second data objects in a one-to-one manner; When the processing time of the first data object reaches a first predetermined processing time and the association state of the first data object is the unassociated state, recording the first data object, wherein the first predetermined processing time corresponds to the first data object in a one-to-one manner; When the processing time of the second data object reaches the second predetermined processing time and the association state of the second data object is the unassociated state, the second data object is recorded, and the second predetermined processing time corresponds to the second data object in a one-to-one manner. The minimum value of the first predetermined range is the difference between the data time of the first data object and the first time interval, the maximum value of the first predetermined range is the sum of the data time of the first data object and the second time interval, the minimum value of the second predetermined range is the difference between the data time of the second data object and the third time interval, and the maximum value of the second predetermined range is the sum of the data time of the second data object and the fourth time interval.

2. The method according to claim 1, characterized in that Before associating the second data object within the first predetermined range of data time with the corresponding first data object, the method further includes: When the data time of the first data object is less than the first watermark time, locking the first data object so that data association with the first data object cannot be performed; In the case that the data time of the second data object is less than the second watermark time, the second data object is locked so that data association cannot be performed on the second data object.

3. The method according to claim 1, characterized in that After associating the second data object within the first predetermined data time range with the corresponding first data object so that the association status of the associated first data object is updated from unassociated to associated, the method further includes: storing the first data object and a first associated data object in a first associated data set of the first data object, wherein the first associated data object is the second data object associated with the first data object; Data processing is performed on the first associated data set.

4. The method according to claim 1, wherein After associating the first data object within the second predetermined data time range with the corresponding second data object so that the association status of the associated second data object is updated from unassociated to associated, the method further includes: storing the second data object and a second associated data object in a second associated data set of the second data object, the second associated data object being the first data object associated with the second data object; Data processing is performed on the second associated data set.

5. The method according to any one of claims 1 to 4, characterized in that Before acquiring the data time of the first data object and the data time of the second data object, the method further includes: The data records of the first data stream and the data records of the second data stream are grouped according to the data time to obtain the first key data stream and the second key data stream, and the data object is a group of the data records with the same data time.

6. A device for determining unassociated data in a data stream, characterized in that: include: An acquiring unit, configured to acquire a data time of a first data object and a data time of a second data object, wherein the first data object is a data object of a first key data stream, the second data object is a data object of a second key data stream, and the data time is a time point at which the first data object or the second data object is generated; a first processing unit, configured to associate data of the second data objects whose data time is within a first predetermined range with the corresponding first data objects, so that an association state of the associated first data objects is updated from unassociated to associated, and the first predetermined range corresponds to the first data objects in a one-to-one manner; a second processing unit, configured to associate data between the first data objects whose data time is within a second predetermined range and the corresponding second data objects, so that an association state of the associated second data objects is updated from unassociated to associated, and the second predetermined range corresponds to the second data objects in a one-to-one manner; a first recording unit, configured to record the first data object when the processing time of the first data object reaches a first predetermined processing time and the association status of the first data object is the unassociated state, wherein the first predetermined processing time corresponds to the first data object in a one-to-one manner; a second recording unit configured to record the second data object when the processing time of the second data object reaches a second predetermined processing time and the association state of the second data object is the unassociated state, wherein the second predetermined processing time corresponds to the second data object in a one-to-one manner; The minimum value of the first predetermined range is the difference between the data time of the first data object and the first time interval, the maximum value of the first predetermined range is the sum of the data time of the first data object and the second time interval, the minimum value of the second predetermined range is the difference between the data time of the second data object and the third time interval, and the maximum value of the second predetermined range is the sum of the data time of the second data object and the fourth time interval.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 5.

8. A processor, characterized in that: The processor is configured to run a program, wherein the program executes the method according to any one of claims 1 to 5 when running.

9. A data processing system, characterized in that: The method comprises one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include methods for executing any one of claims 1 to 5.

Images