A method for real-time determination of the repeatability of a given delay of stream data itself

Abstract

The autocorrelation of a given delay can be used to determine the repeatability of a given delay of a time series or stream data itself. The present invention discloses a method, system and computing system program product that can determine the repeatability of a given delay of a time series or stream data itself in real time by incrementally calculating the autocorrelation of a specified delay of a calculation window of a time series or stream data. The implementation scheme of the present invention includes incrementally calculating two or more components of the autocorrelation of the specified delay of the calculation window after adjustment based on two or more components of the autocorrelation of the specified delay of the calculation window before adjustment, and then generating the autocorrelation of the specified delay of the calculation window after adjustment based on the two or more components of the incremental calculation as needed. Incremental calculation of the autocorrelation avoids accessing all data elements in the calculation window after adjustment and performing repeated calculations, thereby improving computing efficiency, saving computing resources and reducing computing system energy consumption, making it efficient and low-cost to determine the repeatability of a given delay of the stream data itself in real time, and some scenarios of real-time determination of the repeatability of a given delay of the stream data itself become possible from impossible.

Description

Technical Field

[0001] Big data or streaming data analytics. Background Art

[0002] The Internet, mobile communications, navigation, online games, sensor technology and large-scale computing infrastructure generate massive amounts of data every day. Big data is data that exceeds the processing capabilities of traditional database systems and the analytical capabilities of traditional analytical methods due to its huge size, rapid changes and growth rate.

[0003] Streaming data is data that is continuously transmitted by at least one provider and continuously received. Streaming data can be real-time data collected from sensors and continuously transmitted to a computing device or electronic device. Usually this includes receiving data elements of the same format that are continuously divided by time intervals. Streaming data can also be data that is continuously read from a storage device, that is, a large data set stored on one or more storage devices.

[0004] Autocorrelation, also known as delay correlation or serial correlation, is a measure of the degree of correlation between a specific time series and the time series itself delayed by l time points. It can be obtained by dividing the co-correlation of observations of a time series separated by l time points by its standard deviation. If the autocorrelation value of a certain delay is 1 or close to 1, it can be considered that the stream data or streaming big data has a self-repetitive pattern after the delay. Therefore, it is obvious to judge the repeatability of the stream data itself for a given delay based on the autocorrelation of a given delay, but the difficulty and challenge lies in how to calculate the autocorrelation on the stream data in real time.

[0005] The autocorrelation may need to be recalculated after receiving new stream data to reflect the latest data status. Each time a data element is received, the data element is added to the calculation window, and the n+1 data elements in the calculation window are accessed to recalculate the autocorrelation. Recalculating the autocorrelation using all the data elements in the calculation window involves repeated data access and calculation, which is time-consuming and wastes resources.

[0006] Depending on the needs, the size of the calculation window may be very large. For example, the data elements in the calculation window may be distributed on thousands of computing / storage devices on the cloud platform. Recalculating the autocorrelation on some stream data after some data changes using traditional methods cannot be done in real time and occupies and wastes a lot of computing resources. It also makes it impossible to achieve some real-time judgment of the repeatability of the given delay of the stream data itself. Summary of the invention

[0007] The present invention extends to methods, systems and computing system program products that incrementally calculate the autocorrelation of a given delay of streaming data so that the repeatability of the given delay of the streaming data itself can be determined in real time after adjusting the calculation window size. The computing system includes one or more computing devices. Each computing device includes one or more processors. The computing system can access a data stream. The computing system includes one or more storage media. The computing system maintains a computing subset size counter. The computing subset size counter maintains the number of data elements in the calculation window of the data set. Incrementally calculating the autocorrelation of a specified delay of l (l>0) for an adjusted calculation window includes incrementally calculating two or more components of the autocorrelation of the specified delay of the calculation window before adjustment based on two or more (p (p>1)) components of the autocorrelation of the specified delay of the calculation window before adjustment, and then generating the autocorrelation of the specified delay of the adjusted calculation window based on the two or more components of the incremental calculation as needed. Incremental calculation of autocorrelation only needs to access and use the components of incremental calculation, the newly added data elements, and the l data elements adjacent to the newly added data elements in the calculation window and the l data elements on the other side of the calculation window, while avoiding accessing all data elements in the adjusted calculation window and performing repeated calculations, thereby reducing data access delay, improving computing efficiency, saving computing resources and reducing computing system energy consumption, making real-time judgment of the repeatability of a given delay of the streaming data itself efficient and low-cost, and some scenarios of real-time judgment of the repeatability of a given delay of the streaming data itself possible.

[0008] The computing system includes a buffer to store stream data elements. This buffer can be in memory or other computer-readable media, such as a hard disk or other media, or even multiple distributed files distributed on multiple computing devices, which are logically interconnected end-to-end to form a "circular buffer".

[0009] The computing system initializes two or more (p (p>1)) components of the autocorrelation of a given delay l (l≥1) of a pre-adjustment calculation window of a given size n (n>1). The initialization of the two or more components includes calculating the components based on the data elements in the pre-adjustment calculation window according to their definitions or accessing or receiving the already calculated components from one or more computing device readable media.

[0010] The computing system receives a new data element.

[0011] The computing system saves the received data elements into the input buffer.

[0012] The computing system adjusts the pre-adjusted computing window by adding the received data element to the pre-adjusted computing window and does the adjustment by increasing the computing window size counter by 1.

[0013] The computing system directly and incrementally computes one or more (assuming v (1≤v≤p)) components of the autocorrelation of a given delay l for the adjusted computing window. Directly and incrementally computing v (1≤v≤p) components at a given delay l includes: accessing l data elements on both sides of the pre-adjusted computing window and the data elements newly added to the pre-adjusted computing window; accessing v components of the autocorrelation with a delay of l calculated for the pre-adjusted computing window; and mathematically adding any contribution of the added data element to each accessed component.

[0014] The computing system indirectly incrementally computes w=pv components of the autocorrelation of the specified delay of the adjusted computing window as needed. Indirectly incrementally computing the w components of the specified delay includes indirectly incrementally computing each of the w components one by one. Indirectly incrementally computing a component of the specified delay includes accessing and using one or more components of the specified delay other than the component to compute the component. The one or more components may be initialized, directly incrementally computed, indirectly incrementally computed, or computed in any other manner.

[0015] The computing system generates an adjusted autocorrelation of the specified delay of the computing window based on a component of the autocorrelation of the specified delay of the adjusted computing window computed by one or more incremental calculations.

[0016] The computing system can continuously receive a new data element, save the new data element to the input buffer, adjust the computing window and the computing window size, incrementally calculate a sum or an average value or a sum and an average value of the adjusted computing window, directly incrementally calculate v components of a specified delay, and indirectly incrementally calculate w=pv components of a specified delay and calculate the autocorrelation of the specified delay as needed. The computing system can repeat this process as many times as needed.

[0017] This summary is to introduce some selected concepts in a simplified manner, which will be further described below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0018] Other features and advantages of the present invention will be apparent from the following description, will be apparent in part from the description, or may be learned from the practice of the present invention. The features and advantages of the present invention may be realized and obtained from the methods, apparatus, and combinations thereof particularly pointed out in the appended claims. These and other features of the present invention will become more fully apparent from the following description and the appended claims or from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] To describe the manner in which the above and other advantages and features of the present invention can be obtained, a more specific description of the present invention briefly described above will be presented by reference to specific embodiments shown in the attached figures. These figures only describe typical embodiments of the present invention, and therefore they should not be construed as limiting the scope of the present invention:

[0020] Figure 1 A high-level overview of an example computational system that supports incremental computation of autocorrelations is illustrated.

[0021] Figure 1-1 An example computing system architecture is shown that supports incremental computation of autocorrelations of streaming data and all components are computed in a straightforward incremental manner.

[0022] Figure 1-2 An example computing system architecture that supports incremental computing of autocorrelations of streaming data with some components computed in a direct incremental manner and some components computed in an indirect incremental manner is shown.

[0023] Figure 2 A flow chart showing an example method for incrementally computing the autocorrelation of streaming data.

[0024] Figure 3-1 The data elements to the left of the join calculation window 300A are shown.

[0025] Figure 3-2 The data elements accessed within the computation window 300A are shown when incrementally computing the autocorrelation for a given delay on the streaming data.

[0026] Figure 3-3 The data elements to the right of the join calculation window 300B are shown.

[0027] Figure 3-4 The data elements accessed within computation window 300B are shown when incrementally computing the autocorrelation for a given delay on the streaming data.

[0028] Figure 4-1 The definition of autocorrelation and the traditional equation for calculating autocorrelation are shown.

[0029] Figure 4-2 The first autocorrelation incremental calculation algorithm (Incremental Algorithm 1) is shown.

[0030] Figure 4-3 A second autocorrelation incremental calculation algorithm (Incremental Algorithm 2) is shown.

[0031] Figure 4-4 A third autocorrelation incremental calculation algorithm (Incremental Algorithm 3) is shown.

[0032] Figure 5-1Displays the first calculation window for a calculation instance.

[0033] Figure 5-2 Displays the second calculation window for a calculation instance.

[0034] Figure 5-3 Displays the third calculation window for a calculation instance.

[0035] Figure 6-1 Displays the comparison of the computational amounts of the traditional and incremental autocorrelation algorithms when the calculation window size is 6 and the delay is 1.

[0036] Figure 6-2 Displays the comparison of the computational amounts of the traditional and incremental autocorrelation algorithms when the calculation window size is 1000000 and the delay is 1. Detailed implementation

[0037] Calculating autocorrelation is an effective method for judging the self-given delay repeatability of a time series or streaming big data. The present invention extends to a method, system, and computing system program product for judging the self-given delay repeatability of streaming data in real time by incrementally calculating the autocorrelation of a specified delay l (1 ≤ l < n) of a calculation window of size n (n > 1). A computing system includes one or more processor-based computing devices and one or more storage media. Each computing device includes one or more processors. The computing system includes an input buffer. The input buffer stores streaming data elements. This buffer can be in memory or other computer-readable media, such as a hard disk or other media, or even multiple distributed files allocated on multiple computing devices that are logically interconnected end-to-end to form a "circular buffer". Multiple data elements from the data stream that are involved in the autocorrelation calculation form a pre-adjustment calculation window. The computing system maintains a calculation window size counter n (n > l) indicating the number of data elements in a calculation window of the buffer. The delay l indicates the delay used for the autocorrelation calculation. Embodiments of the present invention include incrementally calculating the autocorrelation of a specified delay of more than two (p (p > 1)) components of the adjusted calculation window based on the autocorrelation of the specified delay of the pre-adjustment calculation window, and then generating the autocorrelation of the specified delay of the adjusted calculation window based on more than two components of the incremental calculation as needed. Incrementally calculating the autocorrelation avoids accessing all data elements in the adjusted calculation window and performing repeated calculations, thereby improving the calculation efficiency, saving computing resources, and reducing the energy consumption of the computing system, making it possible to efficiently and with low power consumption judge the self-given delay repeatability of streaming data in real time and making some scenarios of judging the self-given delay repeatability of streaming data in real time possible that were previously impossible.

[0038] Autocorrelation, also known as lag correlation or serial correlation, is a measure of the degree of correlation between a particular time series and itself with a lag of l time points. It can be obtained by dividing the covariance of the observations of a time series separated by l time points by its standard variance. If the autocorrelations for all different lag values of a time series are calculated, the autocorrelation function of that time series is obtained. For a time series that does not change over time, its autocorrelation values will exponentially decay to 0. The values of autocorrelation range between -1 and +1. A value of +1 indicates a perfect positive linear relationship between past and future values of the time series, while a value of -1 indicates a perfect negative linear relationship between past and future values of the time series.

[0039] In this article, a calculation window contains the data involved in autocorrelation calculation. The data elements in the calculation window have an order relationship, that is, changing the order of the data elements in the calculation window can affect the autocorrelation calculation result.

[0040] In this article, a component of autocorrelation is a quantity or expression that appears in the autocorrelation definition formula or any transformation of its definition formula. Autocorrelation is its own largest component. The following are some examples of components of autocorrelation.

[0041]

[0042] Autocorrelation can be calculated based on one or more components or their combinations, so multiple algorithms support incremental autocorrelation calculation.

[0043] A component can be calculated incrementally directly or indirectly. The difference is that when a component is calculated incrementally directly, the component is calculated based on the value of the component in the previous round of calculation, while when the component is calculated incrementally indirectly, the component is calculated using other components outside of the component.

[0044] For a given component, it may be calculated incrementally directly in one algorithm but incrementally indirectly in another algorithm.

[0045] For any algorithm, at least two components will be calculated incrementally, one component is calculated incrementally directly and the other component is calculated incrementally directly or indirectly. For a given algorithm, assuming the total number of different components used is p (p > 1), if the number of components calculated incrementally directly is v (1 ≤ v ≤ p), then the number of components calculated incrementally indirectly is w = p - v (0 ≤ w < p). It is possible that all components are calculated incrementally directly (in this case v = p > 1 and w = 0). However, regardless of whether the result of autocorrelation is needed and accessed in a particular round, the components calculated incrementally directly must be calculated.

[0046] For a given algorithm, if a component is directly incrementally computed, then the component must be computed (i.e., every time a new data element is added to the computation window). However, if a component is indirectly incrementally computed, then the component can be computed on demand by using one or more other components in addition to the component being computed, i.e., only when the autocorrelation needs to be computed and accessed. In this way, when the autocorrelation is not accessed during a particular computation round, only a small number of components need to be incrementally computed. An indirectly incrementally computed component may be used in a directly incrementally computed component, in which case the computation of the indirectly incrementally computed component cannot be omitted.

[0047] An implementation of the present invention includes incrementally computing two or more (p(p>1)) components of the autocorrelation of an adjusted calculation window based on two or more (p(p>1)) components computed for a previous calculation window.

[0048] The computing system starts to incrementally compute autocorrelation of two or more components at a specified delay l from an empty pre-adjustment calculation window or a non-empty pre-adjustment calculation window whose calculation window size and two or more components have been initialized. When the incremental autocorrelation calculation starts from an empty pre-adjustment calculation window, the values ​​of the initial calculation window size and the two or more components are zero. When the incremental autocorrelation calculation starts from a non-empty pre-adjustment calculation window, the calculation window size and the two or more components are initialized. Initializing the calculation window size includes setting it to the number of data elements included in the pre-adjustment calculation window or accessing or receiving a specified calculation window size. Initializing the two or more components includes computing two or more components based on the data elements in the pre-adjustment calculation window through the definition of the components or receiving or accessing the two or more components that have been computed from a computing device readable medium.

[0049] The computing system includes an input buffer. The input buffer stores the stream data elements. The computing window size n (n>l) specifies the number of data elements in a computing window of the buffer.

[0050] The computing system initializes two or more (p (p>1)) components of the autocorrelation of a given delay l (l≥1) of a pre-adjustment calculation window of a given size n (n>1). The initialization of the two or more components includes calculating the components based on the data elements in the pre-adjustment calculation window according to their definitions or accessing or receiving the already calculated components from one or more computing device readable media.

[0051] The computation system receives a new stream data element.

[0052] The computing system saves the received data elements into the input buffer.

[0053] The computing system adjusts the pre-adjustment computing window by: adding new data elements to the pre-adjustment computing window; and adjusting the computing window scale counter by: incrementing its value by 1.

[0054] The computing system incrementally computes a sum or an average or a sum and an average of the post-adjustment computing window.

[0055] The computing system directly incrementally computes one or more (v (1 ≤ v < p) ones) components other than the sum and the average of the autocorrelation of a specified delay l of the post-adjustment computing window. Directly incrementally computing these one or more components includes: accessing l data elements on each side of the pre-adjustment computing window and the newly added data elements; accessing v components of the autocorrelation of the pre-adjustment computing window with a delay of l; and mathematically adding any contribution of the added data elements to each accessed component.

[0056] The computing system indirectly incrementally computes w = p - v components of the autocorrelation of a given delay l for the post-adjustment computing window as needed. Indirectly incrementally computing the w components of the autocorrelation of a given delay l includes incrementally computing each of the w components of the autocorrelation of a given delay l indirectly one by one. Indirectly incrementally computing a component of a given delay l includes: accessing one or more components of the autocorrelation of a given delay l other than this component and computing this component based on the accessed components. These one or more components of the autocorrelation of a given delay l may be initialized, directly incrementally computed, indirectly incrementally computed, or computed in any other way.

[0057] The computing system generates the autocorrelation of a given delay l for the post-adjustment computing window as needed based on one or more components of the autocorrelation of a given delay l incrementally computed for the post-adjustment computing window.

[0058] The computing system may continuously receive new data elements, save the data elements into the input buffer, adjust the pre-adjustment computing window, incrementally compute a sum or an average or a sum and an average of the post-adjustment computing window, directly incrementally compute one or more i.e., v components of the autocorrelation of a specified delay, indirectly incrementally compute w = p - v components of the autocorrelation of a specified delay as needed, generate the autocorrelation of a given delay based on one or more incrementally computed components as needed, and repeat this process multiple times as needed.

[0059] Embodiments of the present invention may include or utilize computing device hardware, such as one or more processors and storage devices described in more detail below, dedicated or general computing devices. The scope of embodiments of the present invention also includes physical and other computing device readable media for carrying or storing computing device executable instructions and / or data structures. These computing device readable media can be any media accessible to general or dedicated computing devices. The computing device readable media that stores computing device executable instructions is storage media (device). The computing device readable media that carries computing device executable instructions is transmission media. Therefore, by way of example and not limitation, embodiments of the present invention may include at least two different types of computing device readable media: storage media (device) and transmission media.

[0060] Storage media (devices) include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk-read-only memory (CD-ROM), solid-state drive (SSD), flash memory, phase change memory (PCM), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store the required program code in the form of executable instructions or data structures for a computing device and which can be accessed by a general or special computing device.

[0061] A "network" is defined as one or more data links that enable computing devices and / or modules and / or other electronic devices to transmit electronic data. When information is transmitted or provided to a computing device via a network or other communication connection (wired, wireless, or a combination of wired and wireless), the computing device considers the connection to be a transmission medium. The transmission medium may include a network and / or data link for carrying the required program code in the form of computing device executable instructions or data structures and which can be accessed by general or special computing devices. Combinations of the above should also be included in the scope of computing device readable media.

[0062] In addition, when using different computing device components, program code in the form of computing device executable instructions or data structures can be automatically transferred from the transmission medium to the storage medium (device) (or vice versa). For example, computing device executable instructions or data structures received from a network or data link can be temporarily stored in a random access memory in a network interface module (e.g., NIC) and then ultimately transferred to the computing device's random access memory and / or to a smaller, volatile storage medium (device) of the computing device. Therefore, it should be understood that the storage medium (device) can be included in a computing device component that also (or even primarily) uses a transmission medium.

[0063] Computing device executable instructions include, for example, instructions and data that, when executed by a processor, cause a general-purpose computing device or a special-purpose computing device to perform a specific function or a set of functions. Computing device executable instructions can be, for example, binary, intermediate format instructions such as assembly code, or even source code. Although the described object is described in specific language of structural features and / or method actions, it should be understood that the object defined in the attached claims is not necessarily limited to the features or actions described above. Instead, the described features or actions are only disclosed in the form of examples of implementing the claims.

[0064] The embodiments of the present invention can be implemented in a network computing environment configured by various types of computing devices, including personal computers, desktop computers, notebook computers, information processors, handheld devices, multiprocessing systems, microprocessor-based or programmable consumer electronics, network computers, minicomputers, mainframe computers, supercomputers, mobile phones, PDAs, tablet computers, pagers, routers, switches and similar products. The embodiments of the present invention can also be applied to a distributed system environment consisting of local or remote computing devices that perform tasks interconnected by a network (i.e., by wired data links, wireless data links, or a combination of wired data links and wireless data links). In a distributed system environment, program modules can be stored on local or remote storage devices.

[0065] Embodiments of the present invention may also be implemented in a cloud computing environment. In this description and in the claims that follow, "cloud computing" is defined as a model that enables on-demand access to a shared pool of configurable computing resources over a network. For example, cloud computing may be utilized by the marketplace to provide pervasive and convenient on-demand access to a shared pool of configurable computing resources. A shared pool of configurable computing resources may be quickly provisioned through virtualization and provided with low management overhead or low service provider interaction, and then adjusted accordingly.

[0066] The cloud computing model can include various features such as on-demand self-service, broadband network access, resource collection, rapid scaling, metered services, etc. The cloud computing model can also be embodied in various service models, such as software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). The cloud computing model can also be deployed through different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, etc.

[0067] Since the present invention effectively reduces the requirements for computing power, its implementation scheme can also be applied to edge computing.

[0068] In the description and claims herein, a "circular buffer" is a data structure that is a single buffer of fixed length that appears to be connected end to end. A circular buffer can be a commonly used space allocated in memory, or it can be a "virtual circular buffer", which is not necessarily in memory, but a file on the hard disk, or even a distributed file physically distributed on multiple storage devices, as long as they are logically connected end to end to form a "circular buffer".

[0069] Several examples are given in the following sections.

[0070] Figure 1 A high-level overview of an example computing system 100 for incrementally computing autocorrelations for streaming data is illustrated. Figure 1 The computing system 100 includes multiple devices connected by different networks, such as a local area network 1021, a wireless network 1022, and the Internet 1023, etc. The multiple devices include, for example, a data analysis engine 1007, a storage system 1011, a real-time data stream 1006, and multiple distributed computing devices that can schedule data analysis tasks and / or query data analysis results, such as a personal computer 1016, a handheld device 1017, and a desktop computer 1018, etc.

[0071] The data analysis engine 1007 may include one or more processors, such as CPU 1009 and CPU 1010, one or more system memories, such as system memory 1008, and component calculation module 131 and autocorrelation calculation module 192. The details of module 131 will be illustrated in more detail in other diagrams (e.g., Figure 1-1 and Figure 1-2 ). Storage system 1011 may include one or more storage media, such as storage media 1012 and storage media 1014, which may be used to store large data sets. For example, 1012 and / or 1014 may include data sets. The data sets in storage system 1011 may be accessed by data analysis engine 1007.

[0072] In general, data stream 1006 may include streaming data from different data sources, such as stock prices, audio data, video data, geospatial data, Internet data, mobile communication data, online game data, bank transaction data, sensor data, and / or closed caption data, etc. Here, a few examples are depicted, and real-time data 1000 may include data collected in real time from sensors 1001, stocks 1002, communications 1003, and banks 1004, etc. Data analysis engine 1007 may receive data elements from data stream 1006. Data from different data sources may be stored in storage system 1011 and accessed by big data analysis, for example, data sets may come from different data sources and be accessed by big data analysis.

[0073] Please understand Figure 1 Some concepts are introduced in a very simplified form. For example, the distribution devices 1016 and 1017 may be connected to the data analysis engine 1007 through a firewall, and the data accessed or received by the data analysis engine 1007 from the data stream 1006 and / or the storage system 1011 may be filtered by a data filter, and so on.

[0074] Figure 1-1 The diagram shows an example computing system architecture 100A for incrementally calculating autocorrelation for streaming data, where all (v=p>1) components are directly incrementally calculated. Regarding the computing system architecture 100A, only the functions and interrelationships of the main components in the architecture will be introduced here, and the process of how these components work together to complete the incremental autocorrelation calculation will be combined later. Figure 2 The process described in Figure 1 Let me introduce you. Figure 1-1 Pictured Figure 1 1006 and 1007 are shown. Reference Figure 1-1 , the computing system architecture 100A includes a component computing module 131 and an autocorrelation computing module 192. The component computing module 131 can be tightly coupled to one or more storage media via a high-speed data bus or loosely coupled to one or more storage media managed by the storage system via a network, such as a local area network, a wide area network, or even the Internet. Accordingly, the component computing module 131 and any other connected computing devices and their components can send and receive message-related data (e.g., Internet Protocol ("IP") datagrams and other high-level protocols that use IP datagrams, such as User Datagram Protocol ("UDP"), Real Time Streaming Protocol ("RTSP"), Real Time Transport Protocol ("RTP"), Microsoft Media Server ("MMS"), Transmission Control Protocol ("TCP"), Hypertext Transfer Protocol ("HTTP"), Simple Mail Transfer Protocol ("SMTP"), etc.) on the network. The output of the component computing module 131 will be used as an input to the autocorrelation computing module 192, and the autocorrelation computing module 192 can generate an autocorrelation 193.

[0075] Generally, data stream 190 may be a sequence of digitally encoded signals (i.e., packets of data or data packets) for transmitting or receiving information during transmission, respectively. Data stream 190 may include data from different sources, such as stock prices, audio data, video data, geospatial data, Internet data, mobile communication data, online game data, bank transaction data, sensor data, closed caption data, and real-time text, etc. Data stream 190 may be a real-time stream or a streamed stored data.

[0076] As stream data elements are received, they may be placed in data buffer 121. For example, data element 101 is placed at position 121C.

[0077] Then, data element 102 may be received. Data element 102 may be placed at position 121D.

[0078] As shown, data buffer 121 has two data elements at positions 121A and 121B before receiving data element 101. As new data elements are placed into data buffer 121, the size of the computation window increases and the computation window becomes a new computation window.

[0079] As stream data elements are received, they pass through a computation window size counter 118 stored on storage device 119. Computation window size counter 118 maintains the computation window size. Whenever a new data element is received, the computation system makes adjustments by adding the new data element to the computation window and incrementing the computation window size counter 118 by 1. The computation window size counter 118 may be reset to 0 when the incremental autocorrelation computation is reset. The computation window size counter 118 may be reset to a specific value when the incremental autocorrelation computation begins from a non-empty computation window. For example, in 100A, before receiving data element 101, the pre-adjusted computation window 122 contains two data elements at positions 121A and 121B, so the computation window size is initialized to 2. When data element 101 is moved to position 121C, data element 101 is added to the pre-adjusted computation window 122, the pre-adjusted computation window 122 becomes the post-adjusted computation window 122A, and the computation window size counter 118 is incremented by 1 to 3. The computation window size counter 118 is stored on the storage device 119. The computation window size counter 118 and the data element 101 may be accessed by the component computation module 131.

[0080] Then, data element 102 may be received. Data element 102 may be placed at position 121D. Adjusted computation window 122A may become adjusted computation window 122B. Computation window size counter 118 may be incremented by 1 to become 4. Adjusted computation window size counter 118 and data element 102 may be accessed by component computation module 131.

[0081] Referring to the computing system architecture 100A, the component computing module 131 generally includes v component computing modules for v (v=p>1) components of a set of n data elements of a computation window for direct incremental computation. v is the number of components directly incrementally computed in a given algorithm for a given delayed incremental computation of autocorrelation, and it varies with the incremental algorithm used. Figure 1-1As shown in FIG. 1 , the component calculation module 131 includes a component Cd1 calculation module 161 and a component Cd v The computing module 162 has v-2 other component computing modules between them, which can be the component Cd2 computing module, the component Cd3 computing module, ..., and the component Cd v-1 Calculation modules. Each component calculation module calculates a specific component for a given delay. Each component calculation module includes an initialization module for initializing a component for the first calculation window and an algorithm for directly incrementally calculating the component for the adjusted calculation window. For example, component Cd1 calculation module 161 includes initialization module 132 to initialize component Cd1 for a given delay and incremental algorithm 133 to incrementally calculate component Cd1 for a given delay, component Cd v The calculation module 162 includes an initialization module 138 to initialize the component Cd of a given delay v and incremental algorithm 139 to incrementally calculate the component Cd for a given delay v .

[0082] Initialization module 132 may be used when initializing component Cd1 or when the autocorrelation calculation is reset. Similarly, initialization module 138 may be used when initializing component Cd v 141 is the initial value of component Cd1. Initialization module 132 may be used to initialize component Cd1 or when autocorrelation calculation is reset. If the calculation window is empty, initialization module 132 may initialize component Cd1 141 to 0. If the calculation window is not empty, initialization module 132 may initialize component Cd1 141 to a specific value contribution 151 transmitted by reset module 123. Initialization module 138 may be used to initialize component Cd1. v Or when the autocorrelation calculation is reset. If the calculation window is empty, the initialization module 138 can initialize the component Cd v 145 is 0. If the calculation window is not empty, the initialization module 138 can initialize the component Cd v 145 contributes 181 to the specific value transmitted by the reset module 123 .

[0083] The incremental algorithm includes directly incrementally calculating v components of the autocorrelation with a given delay of l for the adjusted calculation window. The incremental algorithm 133 accesses or receives the component Cd1 with a delay of l calculated in the previous round, l data elements on both sides of the calculation window before adjustment, and the data elements added to the calculation window before adjustment as input. The incremental algorithm 133 directly incrementally calculates a new component Cd1 with a delay of l using the component Cd1 with a delay of l calculated in the previous round, l data elements on both sides of the calculation window before adjustment, and the data elements added to the calculation window before adjustment. The contribution addition module 133A can mathematically add any contribution of the added data element to the component Cd1 with a delay of l calculated in the previous round for the calculation window before adjustment, thereby obtaining a new component Cd1 with a delay of l for the adjusted calculation window. The incremental algorithm 139 works in a similar manner to the incremental algorithm 133. The incremental algorithm 139 accesses or receives the component Cd1 with a delay of l calculated in the previous round v , l data elements on both sides of the calculation window before adjustment and the data elements added to the calculation window before adjustment are used as input. The incremental algorithm 139 uses the component Cd with a delay of l in the previous round of calculation v , the l data elements on both sides of the calculation window before adjustment and the data elements added to the calculation window before adjustment are directly incrementally calculated to form a new component Cd with a delay of l v The contribution adding module 139A may mathematically add any contribution of the added data element to the component Cd with a delay of l calculated in the previous round of calculation for the pre-adjustment calculation window. v , thus obtaining a new component Cd with a delay of l after the calculation window is adjusted v .

[0084] refer to Figure 1-1 The computing system architecture 100A further includes an autocorrelation calculation module 192. The autocorrelation calculation module 192 may generate an autocorrelation 193 of a given delay based on one or more components of the autocorrelation of a given delay calculated incrementally as needed.

[0085] The computing system can continue to receive new data elements, save the data elements to the data buffer, adjust the computing window and the computing window size, directly incrementally calculate v (1≤v≤p) components at a given delay l, calculate the autocorrelation at a given delay l as needed, and repeat this process as many times as needed.

[0086] Figure 1-2FIG. illustrates an example computing system architecture 100B for incrementally computing the autocorrelation for a streaming data and directly incrementally computing some (v (1≤v<p)) components and indirectly incrementally computing some (w=p-v) components. In some implementations, the difference between computing system architectures 100B and 100A is that architecture 100B includes a component computing module 135. Except for this, the parts with the same reference numerals as in 100A work in the same way. To avoid repeating what has been explained in the description of 100A before, only the different parts will be discussed here. The number v in 100B may be different from the number v in 100A because some components that are directly incrementally computed in 100A are indirectly incrementally computed in 100B. In 100A, v=p>1, but in 100B, 1≤v<p. Refer to Figure 1-2 , computing system architecture 100B includes a component computing module 135. The output of component computing module 131 can be used as the input of component computing module 135, and the outputs of computing modules 131 and 135 can be used as the input of autocorrelation computing module 192, and autocorrelation computing module 192 can generate an autocorrelation 193. Component computing module 135 generally includes w=p-v component computing modules for indirectly incrementally computing w components. For example, component computing module 135 includes a component computing module 163 for indirectly incrementally computing component Ci1, a component computing module 164 for indirectly incrementally computing component Ci w , and the other w-2 component computing modules therebetween. Indirectly incrementally computing w components includes indirectly incrementally computing each of the w components one by one. Indirectly incrementally computing a component includes accessing and using one or more components other than the component itself. The one or more components may be initialized, directly incrementally computed, or indirectly incrementally computed. Once the w=p-v components are indirectly incrementally computed, autocorrelation computing module 192 can compute the autocorrelation 193 for a given latency l based on one or more incrementally computed components.

[0087] Figure 2 FIG. illustrates a flowchart of an example method 200 for incrementally computing the autocorrelation for a streaming large dataset or data stream. Method 200 will be described in conjunction with the components and data of computing system architectures 100A and 100B respectively.

[0088] Method 200 includes initializing a computational window size counter n (n > 1), and p (p > 1) components (201) of autocorrelation of a specified pre-adjustment computational window of a specified size n (n > 1) of a streaming large dataset or data stream in a data buffer with a specified latency l (0 < l < n). For example, the computational window size counter 118 can be initialized and stored in the storage device 119. The initialization module 132 can initialize the autocorrelation component Cd1 141 with a given latency l using contribution 151. The contribution 151 can be the contribution of the data elements of 121A and 121B to the component Cd1 at the given latency l. The initialization module 138 can initialize the autocorrelation component Cd v 145 with contribution 181. The contribution 181 can be the contribution of the data elements of 121A and 121B to the component Cd v at the given latency l.

[0089] Method 200 includes when v < p, that is, when not all components are directly incrementally computed, indirectly incrementally computing each of the w = p - v components one by one as needed based on one or more components other than the component to be computed. These w components are only computed when the autocorrelation is accessed (209). For example, referring to Figure 1-2 where some of its components are directly incrementally computed and some are indirectly incrementally computed, the computation module 163 can indirectly incrementally compute the component Ci1 based on one or more components other than the component Ci1, and the computation module 164 can indirectly incrementally compute the component Ci w based on one or more components other than the component Ci w . These one or more components can be initialized, directly incrementally computed, indirectly incrementally computed, or computed in any other way.

[0090] Method 200 includes generating an autocorrelation with a given latency as needed. When the autocorrelation is accessed, the autocorrelation is generated based on one or more incrementally computed components (210), otherwise only those v components are incrementally computed.

[0091] Method 200 includes receiving a data element to be added to the pre-adjustment computational window (202). For example, the data element 101 can be received.

[0092] Method 200 includes saving the received data element to the data buffer (203). For example, after receiving the data element 101, the data element 101 can be saved in the data buffer 121.

[0093] Method 200 includes adding the received data element to the pre-adjustment calculation window and adjusting the calculation window size counter n (204). For example, data element 101 is added to the pre-adjustment calculation window 122, and then the pre-adjustment calculation window 122 is transformed into the post-adjustment calculation window 122A. The calculation window size counter 118 is incremented by 1 after receiving the data element 101.

[0094] Method 200 includes directly incrementally calculating v (1 ≤ v < p) components of autocorrelation with a delay of l for the post-adjustment calculation window (205), including: accessing l data elements on each side of the pre-adjustment calculation window and the newly added data element (206); accessing v components of autocorrelation with a delay of l of the pre-adjustment calculation window (207); and mathematically adding any contribution of the added data element to each of the accessed components (208). The details are described below.

[0095] Directly incrementally calculating v components of autocorrelation with a specified delay l for the post-adjustment calculation window includes accessing l data elements on each side of the pre-adjustment calculation window and the added data element (206). For example, if the specified delay l = 1, the incremental algorithm 133 can access the data elements at 121A and 121B and the added data element 101. Similarly, if the specified delay l = 1, the incremental algorithm 139 can access the data elements at 121A and 121B and the added data element 101.

[0096] Directly incrementally calculating v components of autocorrelation with a delay of l for the post-adjustment calculation window includes accessing v components of autocorrelation with a delay of l of the pre-adjustment calculation window (207). For example, if the specified delay l = 1, the incremental algorithm 133 can access the component Cd1 141 with a delay of 1. If the specified delay l = 2, the incremental algorithm 133 can access the component Cd1 141 with a delay of 2... Similarly, if the specified delay l = 1, the incremental algorithm 139 can access the component Cd v 145. If the specified delay l = 2, the incremental algorithm 139 can access the component Cd v 145...

[0097] Directly incrementally calculating v components of autocorrelation with a delay of l for the post-adjustment calculation window includes mathematically adding any contribution of the added data element to each of the accessed components (208). For example, if the specified delay l = 1, the component Cd1 143 with a directly incrementally calculated delay of 1 can include a contribution increase module 133A that mathematically adds the contribution 152 to the component Cd1 141 with an autocorrelation delay of 1. Similarly, the component Cd v 147 with a directly incrementally calculated delay of 1 can include a contribution increase module 139A that adds the contribution to the component Cd with an autocorrelation delay of 1 v182 is mathematically added to the contribution in 145. The contributions 152 and 182 come from the data element 101.

[0098] As Figure 1-1 As shown in FIGS. 1-2, the component Cd1 143 includes a contribution 151 (contributions from data elements of 121A and 121B) and a contribution 152 (contribution from the data element 101). Similarly, the component Cd v 147 includes a contribution 181 (contributions from data elements of 121A and 121B) and a contribution 182 (contribution from the data element 101).

[0099] When the autocorrelation is accessed and v < p (i.e., not all components are directly incrementally calculated), the method 200 includes indirectly incrementally calculating w = p - v components with a delay of l as needed (209). These w components are only calculated when the autocorrelation is accessed. For example, referring to Figure 1-2 where some of its components are directly incrementally calculated and some are indirectly incrementally calculated, the calculation module 163 can indirectly incrementally calculate the component Ci1 based on one or more components other than the component Ci1, and the calculation module 164 can be based on the component Ci w outside one or more components to indirectly incrementally calculate the component Ci w . These one or more components can be initialized, directly incrementally calculated, indirectly incrementally calculated, or calculated in any other way.

[0100] The method 200 includes generating an autocorrelation with a delay of l based on one or more initialized or incrementally calculated components of the autocorrelation with a delay of l as needed (210). When the autocorrelation is accessed, the autocorrelation is generated based on one or more incrementally calculated components; otherwise, only v components are directly incrementally calculated. When the autocorrelation is accessed, the method 200 includes indirectly incrementally calculating one by one the w components of the autocorrelation with a delay of l based on one or more components other than the components themselves as needed (209), and then generating the autocorrelation with a delay of l. For example, in the architecture 100A, the autocorrelation module 192 can generate an autocorrelation 193 with a given delay. In the architecture 100B, the calculation module 163 can indirectly incrementally calculate Ci1 based on one or more components other than the component Ci1, and the calculation module 164 can be based on the component Ci w outside one or more components to indirectly incrementally calculate Ci w , ……, the autocorrelation calculation module 192 can generate an autocorrelation 193 with a given delay (210). Once the autocorrelation with a given delay is calculated, the method 200 includes receiving the next stream data element to start the next round of incremental calculation. Each time a new round of incremental calculation starts, the adjusted calculation window of the previous round becomes the pre-adjusted calculation window of the new round of incremental calculation.

[0101] As more data elements are accessed, 202-208 may be repeated, and 209-210 may be repeated as needed. For example, after data element 101 is received and component Cd1 143 is accessed to component Cd v After the components within the range 147 are calculated, data element 102 can be received (202). Once a new data element is received, method 200 includes saving the received data element to a buffer (203). Method 200 includes adding the newly received data element to the pre-adjusted calculation window and adjusting the calculation window size counter (204). For example, data element 102 can be stored in data buffer 121 and added to calculation window 122A, and calculation window size counter 118 can be incremented by 1 after receiving data element 102.

[0102] Method 200 includes directly incrementally calculating v components of the autocorrelation with a delay of l for the adjusted calculation window based on the v components of the calculation window before adjustment (205), which includes accessing l data elements on both sides of the calculation window before adjustment and the added data element (206), accessing the v components (207), and mathematically adding any contribution of the added data element to each accessed component (208). For example, referring to 100A and 100B, at a specified delay such as l=1, algorithm 133 can be used to directly incrementally calculate the component Cd1 144 with a delay of 1 for the adjusted calculation window 122B based on the component Cd1 143 of the autocorrelation with a delay of 1 calculated for the adjusted calculation window 122A (205). The incremental algorithm 133 can access the data elements of 121A and the data elements of 121C (data element 101) and data element 102 (206). The incremental algorithm 133 can access the component Cd1 143 of the autocorrelation with a delay of 1 (207). Directly incrementally calculating the component Cd1 144 of the delay 1 includes the contribution adding module 133A mathematically adding the contribution 153, i.e., the contribution of the data element 102, to the component Cd1 143 of the autocorrelation of the delay 1 (208). Similarly, at a specified delay, such as l=1, the incremental algorithm 139 may be used to directly incrementally calculate the component Cd1 144 of the autocorrelation of the delay 1 for the adjusted calculation window 122B. v 148 is based on the component Cd of the autocorrelation with a delay of 1 calculated for the adjusted calculation window 122A v 147. Incremental algorithm 139 may access data element 121A and data element 121C (data element 101) and data element 102 (206). Incremental algorithm 139 may access component Cd of the autocorrelation with a delay of 1. v 147. Direct incremental calculation of the component Cd of the autocorrelation with a delay of 1 v 148 includes a contribution increasing module 139A to the component Cd with a delay of 1 v147 mathematically adds contribution 185, ie the contribution of data element 102.

[0103] As shown, the component Cd1 144 of the autocorrelation with a delay of l includes other contributions 151 (the contribution from data elements 121A and 121B), contribution 152 (the contribution from data element 101), and contribution 153 (the contribution from data element 102). v 148 includes other contributions 181 (contributions from data elements 121A and 121B), contribution 182 (contribution from data element 101), and contribution 183 (contribution from data element 102).

[0104] The method 200 includes computing w components and autocorrelations for a given delay incrementally as needed.

[0105] Method 200 includes, as needed, indirectly incrementally computing w components and autocorrelations for a given delay, if only the autocorrelations are accessed. If the autocorrelations are not accessed, method 200 includes continuing to receive the next data element to be added for the next calculation window (202). If the autocorrelations are accessed, method 200 includes indirectly incrementally computing w components for a given delay (209), generating an autocorrelation for the given delay based on one or more of the incrementally computed components for the given delay (210).

[0106] When the next stream data element is received, component Cd1 144 can be used to directly incrementally calculate the next component Cd1. Component Cd v 148 can be used to directly incrementally calculate the next component Cd v .

[0107] Method 200 includes reset 211. Reset 211 can be used to reset the incremental autocorrelation calculation. When reset 211 is called after 201 or 210, the calculation window size counter and the v (1≤v≤p) components of the autocorrelation at a given delay can be initialized. For example, component Cd1 141 can be initialized by definition with the data elements in the calculation window or initialized to a given calculated value. The latter can occur when combining incremental autocorrelation calculation with iterative autocorrelation calculation or decremental autocorrelation calculation. Component Cd v 145 can be initialized in a similar manner.

[0108] Figure 3-1 The diagram shows the data elements added to the left of the calculation window 300A when incrementally calculating the autocorrelation on the stream data. The calculation window 300A can be empty or non-empty. As time goes by, the old data elements, such as x m+n , then x m+n-1 , then x m+n-lIt can be added to the left of the calculation window 300A.

[0109] Figure 3-2 The diagram illustrates the data elements accessed in the computation window 300A when incrementally computing autocorrelations on streaming data. When 300A is empty, the computation system may continue to add data elements to 300A until the number of data elements reaches l+1, where l is a given delay, and then initialize two or more (v) components. When 300A is non-empty, the v components may have been computed. If not, the v components may be initialized based on the data elements of 300A. The v components of delay l may be directly incrementally computed using the l data elements on both sides of the computation window 300A, the added data elements, and the v components of the computation window before adjustment. If the specified delay is 1, the leftmost 1 data element and the rightmost 1 data element of the computation window 300A and the added data element are accessed. If the specified delay is 2, the leftmost 2 data elements and the rightmost 2 data elements of the computation window 300A and the added data element are accessed. If the specified delay is l, the leftmost l data elements and the rightmost l data elements of the computation window 300A and the added data elements will be accessed. For a given increment algorithm, v is a constant, and the number of operands of the indirect increment w=pv components is also a constant, so for a given delay, the amount of data access and the amount of computation are reduced and are constant. The larger the computation window size n, the more significant the reduction in the amount of data access and computation.

[0110] Figure 3-3 The diagram shows the data elements added to the right of the calculation window 300B when incrementally calculating the autocorrelation on the streaming data. The calculation window 300A can be empty or non-empty. As time goes by, new data elements, such as x m+1 , then x m+2 , then x m+l+1 The right side of the calculation window 300B may be added.

[0111] Figure 3-4The diagram shows the data accessed in the calculation window 300B when the autocorrelation is incrementally calculated on the streaming data. When 300B is empty, the calculation system can keep adding data elements to 300B until the number of data elements reaches l+1, where l is a given delay, and then initialize two or more (v) components. When 300B is non-empty, v components may have been calculated. If not, v components can be initialized based on the data elements of 300B. The v components with a delay of l can be directly incrementally calculated by calculating the l data elements on both sides of the calculation window 300B, the added data elements, and the v components of the calculation window before adjustment. If the specified delay is 1, the leftmost 1 data element and the rightmost 1 data element of the calculation window 300B and the added data element will be accessed. If the specified delay is 2, the leftmost 2 data elements and the rightmost 2 data elements of the calculation window 300B and the added data element will be accessed. If the specified delay is l, the leftmost l data elements and the rightmost l data elements of the calculation window 300B and the added data element will be accessed. For a given incremental algorithm, v is a constant, and the number of operations of the indirect incremental w=pv components is also a constant, so for a given delay, the amount of data access and computation is reduced and constant. The larger the computation window size n, the more significant the reduction in data access and computation.

[0112] Figure 4-1 The figure shows the definition of autocorrelation. Assume X = {x i |i=m+1,…,m+n} is a data stream containing a computation window of size n involving the data involved in the autocorrelation computation. Suppose that after a while a new data element x a Add to compute window X. Each time a data element is added, the computation window is considered a new computation window. A new round of computation is started whenever any component of the autocorrelation needs to be recomputed because of changes in the data in the computation window.

[0113] The calculation result of autocorrelation is not only related to the value of each data element in the calculation window, but also to the order of each data element. Therefore, different positions of data elements in the calculation window require different processing. There are three cases. First, adding a new data element x a Go to the far left of the calculation window. Second, add a new data element x a To the far right of the calculation window. Third, add a new data element x a To any position outside the calculation window. The third case rarely occurs in the data stream, so only the first two cases are discussed here. The equations used to calculate one or more components in these two cases are different. To distinguish, define the adjusted calculation window in the first case as X I , the second case is XII .

[0114] Calculate X I and X II There is no difference between the equations for the sum and the average. Equations 401 and 402 are respectively the sum S of all data elements in the calculation window X with a calculation size of n in the kth round. k and average Equation 403 is the autocorrelation ρ of the given delay l for the calculation window X for the kth round. (k,l) Equations 404 and 405 are the sum of all data elements in the adjusted calculation window with a calculation size of n in the k+1th round, respectively. k+1 and average Equation 406 is the traditional equation for calculating the adjusted calculation window X for the k+1th round I The autocorrelation ρ of a given delay l I (k+1,l) Equation 407 is the traditional equation for calculating the adjusted calculation window X for the k+1th round II The autocorrelation ρ of a given delay l II (k+1,l) The traditional equation.

[0115] To demonstrate how to use component increments to calculate autocorrelation, three different incremental autocorrelation algorithms are provided as examples. A new round of calculation begins each time there is a data change in the calculation window (e.g., 122 → 122A → 122B). A sum or average is the basic component for calculating autocorrelation. The equation for incrementally calculating a sum or average is the incremental component equation used by all incremental autocorrelation calculation algorithms.

[0116] Figure 4-2 The first example incremental autocorrelation calculation algorithm (incremental algorithm 1) is described. Equations 401 and 402 can be used to initialize the components S respectively. k and / or Equations 408, 409, and 410 may be used to initialize component SS, respectively. k , SX k , and covX (k,l) Equation 411 can be used to generate the autocorrelation ρ (k,l) When the data element is added to the leftmost part of the computation window, the incremental algorithm 1 includes the component S k+1 or SS k+1 ,SX k+1 , and covX I (k+1,l) The incremental calculation, once the component SX k+1 and covX I (k+1,l)is calculated, the autocorrelation ρ I (k+1,l) Once the component S k and / or Available, equations 412 and 413 can be used to incrementally calculate the adjusted calculation window X I Component S k+1 and Once the component SS k Available, Equation 414 can be used to directly incrementally calculate the adjusted calculation window X I Components of SS k+1 Once component S k+1 or and SS k+1 Available, equation 415 can be used to indirectly calculate the incremental adjustment calculation window X I Components of SS k+1 Once the component covX (k,l) , SS k+1 , S k or and S k+1 or Available, Equation 416 can be used to directly incrementally calculate the adjusted calculation window X I Components of covX I (k+1,l) 412, 413, 415, and 416 each contain multiple equations but only one of each is required depending on whether the sum or the average or both are available. Once the component covX I (k+1,l) and SX k+1 is calculated, equation 417 can be used to indirectly calculate the adjusted calculation window X I The autocorrelation ρ of a given delay l I (k+1,l) When the data element is added to the rightmost side of the computation window, the incremental algorithm 1 includes the component S k+1 or SS k+1 ,SX k+1 , and covX II (k+1,l) The incremental calculation, once the component SX k+1 and covX II (k+1,l) is calculated, the autocorrelation ρ II (k+1,l) Equations 418 and 419 can be used to incrementally calculate the adjusted calculation window X, respectively. II Component S k+1 and Once the component S k and / or Equation 420 can be used to directly incrementally calculate the adjusted calculation window X II Components of SS k+1 Once the component SS k Available. 421 can be used for indirect incremental calculation to adjust the calculation window X II Components of SX k+1 Once the component S k+1 or and SS k+1 Equation 422 can be used to directly incrementally calculate the adjusted calculation window X II Components of covX II (k+1,l) Once the component covX (k,l) , SS k+1 , S k or and S k+1 or 418, 419, 421, and 422 each contain multiple equations but only one of each is required depending on whether the sum or average or both are available. Equation 423 can be used to indirectly calculate the incremental adjustment calculation window X II The autocorrelation ρ of a given delay l II (k+1,l) Once the component covX II (l+1,l) and SX k+1 is calculated.

[0117] Figure 4-3 The second example incremental autocorrelation calculation algorithm (incremental algorithm 2) is described. Equations 401 and 402 can be used to initialize the component S respectively. k and / or Equations 424 and 425 can be used to initialize component SX, respectively. k and covX (k,l) Equation 426 can be used to calculate the autocorrelation ρ (k,l) When the data element is added to the leftmost part of the computation window, the incremental algorithm 2 includes the component S k+1 or SX k+1 , and covX I (k+1,l) The incremental calculation, once the component SX k+1 and covX I (k+1,l) is calculated, the autocorrelation ρ I (k+1,l) Once the component S k and / or Available, equations 427 and 428 can be used to incrementally calculate the adjusted calculation window X I Component Sk+1 and Once the component SX k ,S k+1 and / or Available, Equation 429 can be used to directly increment the adjusted calculation window X I Components of SX k+1 Equation 430 can be used to directly incrementally calculate the adjusted calculation window X I Components of covX I (k+1,l) Once the component covX (k,l) , S k or and S k+1 or 427, 428, 429, and 430 each contain multiple equations but only one of them is needed depending on whether the sum or the average or both are available. Once the component covX I (k+1,l) and SX k+1 is calculated, equation 431 can be used to indirectly calculate the adjusted calculation window X I The autocorrelation ρ of a given delay l I (k+1,l) When the data element is added to the rightmost side of the computation window, the incremental algorithm 2 includes the component S k+1 or SX k+1 , and covX II (k+1,l) The incremental calculation, once the component SX k+1 and covX II (k+1,l) is calculated, the autocorrelation ρ II (k+1,l) Equations 432 and 433 can be used to incrementally calculate the adjusted calculation window X, respectively. II Component S k+1 and Once the component S k and / or Equation 434 can be used to directly incrementally calculate the adjusted calculation window X II Components of SX k+1 Once the component SX k , S k+1 and / or Equation 435 can be used to directly incrementally calculate the adjusted calculation window X II Components of covX II (k+1,l) Once the component covX (k,l) , S k or and Sk+1 or 432, 433, 434, and 435 each contain multiple equations but only one of each is required depending on whether the sum or average or both are available. Equation 436 can be used to indirectly calculate the adjusted calculation window X II The autocorrelation ρ of a given delay l II (k+1,l) Once the component covX II (k+1,l) and SX k+1 is calculated.

[0118] Figure 4-4 The third example incremental autocorrelation calculation algorithm (incremental algorithm 3) is described. Equations 401 and 402 can be used to initialize the component S respectively. k and / or Equations 437 and 438 can be used to initialize component SX, respectively. k and covX (k,l) Equation 439 can be used to calculate the autocorrelation ρ (k,l) When the data element is added to the leftmost part of the computation window, the incremental algorithm 3 includes the component S k+1 or SX k+1 , and covX I (k+1,l) The incremental calculation, once the component SX k+1 and covX I (k+1,l) is calculated, the autocorrelation ρ I (k+1,l) Equations 440 and 441 can be used to incrementally calculate the adjusted calculation window X, respectively. I Component S k+1 and Once the component S k and / or Equation 442 can be used to directly incrementally calculate the adjusted calculation window X I Components of SX k+1 Once the component SX k , S k and / or and S k+1 and / or Equation 443 can be used to directly incrementally calculate the adjusted calculation window X I Components of covX I (k+1,l) Once the component covX (k,l) , S k or and S k+1 or Available. 440, 441, 442, and 443 each contain multiple equations but only one of each is required depending on whether the sum or average or both are available. Equation 444 can be used to indirectly calculate the incremental adjustment calculation window X I The autocorrelation ρ of a given delay l I (k+1,l) Once the component covX I (k+1,l) and SX k+1 When the data element is added to the rightmost side of the calculation window, the incremental algorithm 3 includes the component S k+1 or SX k+1 , and covX II (k+1,l) The incremental calculation, once the component SX k+1 and covX II (k+1,l) is calculated, the autocorrelation ρ II (k+1,l) Equations 445 and 446 can be used to incrementally calculate the adjusted calculation window X, respectively. II Component S k+1 and Once the component S k and / or Equation 447 can be used to directly increment the adjusted calculation window X II Components of SX k+1 Once the component SX k , S k and / or and S k+1 and / or Equation 448 can be used to directly increment the adjusted calculation window X II Components of covX II (k+1,l) Once the component covX (k,l) , S k or and S k+1 or 445, 446, 447, and 448 each contain multiple equations but only one of them is needed depending on whether the sum or the average or both are available. Once the component covX II (k+1,l) and SX k+1 is calculated, equation 449 can be used to indirectly calculate the adjusted calculation window X II The autocorrelation ρ of a given delay l II (k+1,l) .

[0119] To demonstrate the incremental autocorrelation algorithms and how they compare to traditional algorithms, three examples are given below. Data from three calculation windows are used. For the traditional algorithm, the calculation process for all three calculation windows is exactly the same. For the incremental algorithm, the first calculation window performs the initialization of two or more components, and the second and third calculation windows perform incremental calculations.

[0120] Figure 5-1 , Figure 5-2 , Figure 5-3 The first calculation window, the second calculation window, and the third calculation window for a calculation example are shown respectively. Calculation window 503 includes 4 data elements of data stream 501: 8, 3, 6, 1. Calculation window size 503 (n) is 4. Calculation window 504 includes 5 data elements of data stream 501: 8, 3, 6, 1, 9. Calculation window size 505 (n) is 5. Calculation window 505 includes 6 data elements of data stream 501: 8, 3, 6, 1, 9, 2. Calculation window size 507 (n) is 6. This calculation example assumes that the data elements are added to the rightmost side of the calculation window. Data stream 501 can be streamed big data or streaming data.

[0121] First, the autocorrelations of the windows 502, 504, and 506 with a delay of 1 are calculated using a conventional algorithm.

[0122] Compute the autocorrelation with a delay of 1 for computation window 502:

[0123]

[0124] Without any optimization, computing the autocorrelation with latency 1 for a computation window of size 4 requires 2 divisions, 7 multiplications, 8 additions, and 10 subtractions.

[0125] The same equations and procedures can be used to Figure 5-2 The calculation window 504 shown calculates the autocorrelation sum with a delay of 1 as Figure 5-3 The calculation window 505 shown calculates the autocorrelation with a delay of 1. The calculation window 504 with a size of 5 calculates the autocorrelation with a delay of 1. The calculation without optimization includes 2 divisions, 9 multiplications, 11 additions and 13 subtractions. The autocorrelation of the calculation window 506 with a size of 6 and a delay of 1 is This calculation includes 2 divisions, 11 multiplications, 14 additions and 16 subtractions without optimization. The traditional algorithm usually needs to complete 2 divisions, 2n-1 multiplications, 3n-(l+3) additions, and 3n-2l subtractions when calculating the autocorrelation with a window size of n and a given delay of l without optimization.

[0126] Next, the incremental algorithm 1 is used to calculate the autocorrelations of windows 502, 504, and 506 with a delay of 1.

[0127] Compute the autocorrelation with a delay of 1 for computation window 502:

[0128] 1. Initialize the components of round 1 using equations 402, 408, 409, and 410 respectively SS1, SX1, and covX (1,1) :

[0129]

[0130] 2. Calculate the autocorrelation ρ of the first round using equation 411 (1,1) :

[0131]

[0132] When calculating the autocorrelation with a delay of 1 for calculation window 502, there are 2 divisions, 9 multiplications, 8 additions and 7 subtractions.

[0133] Compute the autocorrelation with a delay of 1 for computation window 504:

[0134] 1. Use equations 419, 420, 421, and 422 to incrementally calculate the components of the second round. SS2, SX2, and covX II (2,1) :

[0135]

[0136]

[0137] 2. Calculate the autocorrelation ρ of the second round using equation 423 II (2,1) :

[0138]

[0139] When calculating the autocorrelation with a delay of 1 in the calculation window 504 increment, there are 2 divisions, 8 multiplications, 7 additions and 6 subtractions.

[0140] Compute the autocorrelation with a delay of 1 for computation window 506:

[0141] 1. Use equations 419, 420, 421, and 422 to incrementally calculate the components of the third round SS3, SX3, and covX II (3,1) :

[0142]

[0143] 2. Calculate the autocorrelation ρ of the third round using equation 423 II (3,1) :

[0144]

[0145] There are 2 divisions, 8 multiplications, 7 additions and 6 subtractions when calculating the autocorrelation with a delay of 1 for calculation window 506.

[0146] Next, the incremental algorithm 2 is used to calculate the autocorrelations of windows 502, 504, and 506 with a delay of 1.

[0147] Compute the autocorrelation with a delay of 1 for computation window 502:

[0148] 1. Initialize the components of round 1 using equations 402, 424, and 425 SX1, and covX (1,1) :

[0149]

[0150] 2. Calculate the autocorrelation ρ of the first round using equation 426 (1,1) :

[0151]

[0152] When calculating the autocorrelation with a delay of 1 for calculation window 502, there are 2 divisions, 9 multiplications, 8 additions and 7 subtractions.

[0153] Compute the autocorrelation with a delay of 1 for computation window 504:

[0154] 1. Use equations 433, 434, and 435 to incrementally calculate the components of the second round SX2, and covX II (2,1) :

[0155]

[0156] 2. Calculate the autocorrelation ρ of the second round using equation 436 II (2,1) :

[0157]

[0158] When calculating the autocorrelation with a delay of 1 in the calculation window 504 increment, there are 2 divisions, 6 multiplications, 7 additions and 7 subtractions.

[0159] Compute the autocorrelation with a delay of 1 for computation window 506:

[0160] 1. Use equations 433, 434, and 435 to incrementally calculate the components of round 3 SX3, and covX II (3,1) :

[0161]

[0162] 2. Calculate the autocorrelation ρ of the third round using equation 436 II (3,1) :

[0163]

[0164] When calculating the autocorrelation with a delay of 1 in the calculation window 506 increment, there are 2 divisions, 6 multiplications, 7 additions and 7 subtractions.

[0165] Next, the incremental algorithm 3 is used to calculate the autocorrelations of windows 502, 504, and 506 with a delay of 1.

[0166] Compute the autocorrelation with a delay of 1 for computation window 502:

[0167] 1. Initialize the first round of components using equations 402, 437, and 438 SX1, and covX (1,1) :

[0168]

[0169] 2. Calculate the autocorrelation ρ of the first round using equation 439 (1,1) :

[0170]

[0171] When calculating the autocorrelation with a delay of 1 for calculation window 502, there are 2 divisions, 9 multiplications, 8 additions and 7 subtractions.

[0172] Compute the autocorrelation with a delay of 1 for computation window 504:

[0173] 1. Use equations 446, 447, and 448 to incrementally calculate the components of the second round SX2, and covX II (2,1) :

[0174]

[0175] 2. Calculate the autocorrelation ρ of the second round using equation 449 II (2,1) :

[0176]

[0177] When calculating the autocorrelation with a delay of 1 in the calculation window 504 increment, there are 2 divisions, 5 multiplications, 7 additions and 7 subtractions.

[0178] Compute the autocorrelation with a delay of 1 for computation window 506:

[0179] 1. Use equations 446, 447, and 448 to incrementally calculate the components of round 3 SX3, and covX II (3,1) :

[0180]

[0181]

[0182] Calculate the autocorrelation ρ of the third round using Equation 449 II (3,1) :

[0183]

[0184] When calculating the autocorrelation with a delay of 1 in the calculation window 506 increment, there are 2 divisions, 5 multiplications, 7 additions and 7 subtractions.

[0185] In the three examples above, the mean is used for the incremental autocorrelation calculation. The sum can also be used for the incremental autocorrelation calculation, but the operands are different. In addition, in the three examples above, the data elements are added to the rightmost part of the calculation window. When the data elements are added to the leftmost part of the calculation window, the calculation process is similar, but a different set of equations are applied.

[0186] Figure 6-1 The figure shows the comparison of the computational complexity of the traditional autocorrelation algorithm and the incremental autocorrelation algorithm when n=6 and the delay is 1. As shown in the figure, the division operation, multiplication operation, addition operation and subtraction operation of any incremental algorithm and the traditional algorithm are similar.

[0187] Figure 6-2 The figure shows the comparison of the computational complexity of the traditional autocorrelation algorithm and the incremental autocorrelation algorithm when n=1,000,000 and the delay is 1. As shown in the figure, any incremental algorithm has much fewer multiplication, addition and subtraction operations than the traditional algorithm. The incremental autocorrelation algorithm can process data that needs to be processed on thousands of computers on a single machine. It greatly improves computing efficiency, reduces computing resources, and reduces computing system energy consumption, making it efficient and low-cost to judge the repeatability of the given delay of the streaming data itself in real time, and some scenarios of judging the repeatability of the given delay of the streaming data itself in real time have become possible.

[0188] The present invention can be implemented in other specific ways without departing from its ideas or essential features. The implementation scheme described in this application is only exemplary and not restrictive in all respects. Therefore, the scope of the present invention is indicated by the attached claims rather than the previous description. All changes equivalent to the meaning and scope of the claims in the claims are included in their scope.

Claims

1. A method implemented by a computing system based on one or more computing devices, characterized in that: A computing system based on a computing device initializes a computing window size counter n (n > 1), a latency l (0 < l < n), and more than two components of autocorrelation with latency l for an unadjusted computing window of a data stream. The computing window counter specifies the number of data elements included in the unadjusted computing window. The data elements of the unadjusted computing window are stored in a buffer of the computing system; The computing system based on the computing device receives a data element; The computing system based on the computing device stores the received data element into the buffer; The computing system based on the computing device adjusts the unadjusted computing window by: Adding the received data element to the unadjusted computing window; And Adjusting the computing window size counter; The computing system based on the computing device incrementally calculates more than two components of autocorrelation with latency l for an adjusted computing window at least based on more than two components of autocorrelation with latency l of the unadjusted computing window, and avoids accessing and using all data elements in the adjusted computing window during the incremental calculation of the more than two components to reduce data access latency, improve computing efficiency, save computing resources, and reduce the energy consumption of the computing system; And The computing system based on the computing device generates autocorrelation with latency l for the adjusted computing window based on one or more components incrementally calculated for the adjusted computing window.

2. The method implemented by a computing system according to claim 1, characterized in that: The receiving a data element to be added includes receiving multiple data elements to be added to the unadjusted computing window. The method further includes storing the received data element into the buffer for each of the multiple data elements to be added, adjusting the unadjusted computing window, incrementally calculating more than two components, and generating autocorrelation with latency l for the adjusted computing window.

3. The method implemented by a computing system according to claim 2, characterized in that: The generating autocorrelation with latency l for the adjusted computing window is performed if and only if the autocorrelation is accessed.

4. The method implemented by a computing system according to claim 3, characterized in that: The generating autocorrelation with latency l for the adjusted computing window further includes the computing system based on the computing device indirectly incrementally calculating one or more components of autocorrelation with latency l for the adjusted computing window. Indirectly incrementally calculating the one or more components includes calculating the one or more components separately one by one based on one or more components other than the components to be calculated.

5. A computing system, characterized in that: One or more processors; One or more storage media; And One or more computing modules, when executed by at least one of the one or more processors, execute a method that includes: a. Initializing a computing window size counter n (n > 1), a latency l (0 < l < n), and more than two components of autocorrelation with latency l for an unadjusted computing window stored in a buffer on one or more storage devices of the computing system for a data stream. The computing window counter specifies the number of data elements included in the unadjusted computing window; b. Receiving a data element to be added to the unadjusted computing window; c. Saving the data element to the buffer; d. Adjust the pre - adjustment calculation window, including: Adding the received data elements to the pre - adjustment calculation window; and Adjusting the calculation window size counter; e. Based on at least two components of the autocorrelation with delay l of the pre - adjustment calculation window, incrementally calculate at least two components of the autocorrelation with delay l for the post - adjustment calculation window, and avoid accessing and using all data elements in the post - adjustment calculation window during the incremental calculation of these at least two components to reduce data access latency, improve calculation efficiency, save calculation resources, and reduce the energy consumption of the calculation system; and f. Generate the autocorrelation with delay l for the post - adjustment calculation window based on one or more components incrementally calculated for the post - adjustment calculation window.

6. The computing system according to claim 5, characterized in that: The one or more calculation modules, when executed by at least one of one or more processors, execute b, c, d, e, and f multiple times.

7. The computing system according to claim 6, characterized in that: Execute f if and only if the autocorrelation with delay l of the post - adjustment calculation window is accessed.

8. The computing system according to claim 7, characterized in that: Executing f further includes the calculation system indirectly incrementally calculating one or more components of the autocorrelation with delay l for the post - adjustment calculation window, and indirectly incrementally calculating these one or more components includes calculating these one or more components separately one by one based on one or more components other than the components to be calculated.

9. A computing system program product, running on a computing system including one or more computing devices, the computing system including one or more processors and one or more storage media, the computing system program product including multiple computing - device - executable instructions, when these computing - device - executable instructions are run by at least one computing device in the computing system, causing the computing system to execute a method, characterized in that: Initialize a calculation window size counter n (n > 1), a delay l (0 < l < n), and at least two components of the autocorrelation with delay l for a pre - adjustment calculation window of a buffer stored on at least one storage media of the computing system for a data stream, the calculation window counter specifying the number of data elements included in the pre - adjustment calculation window; Receive a data element to be added to the pre - adjustment calculation window; Save the received data element to the buffer; Adjust the pre - adjustment calculation window by: Adding the data element to be added to the pre - adjustment calculation window; And Adjusting the calculation window size counter; Based on at least two components of the autocorrelation with delay l of the pre - adjustment calculation window, incrementally calculate at least two components of the autocorrelation with delay l for the post - adjustment calculation window, and avoid accessing and using all data elements in the post - adjustment calculation window during the incremental calculation of these at least two components to reduce data access latency, improve calculation efficiency, save calculation resources, and reduce the energy consumption of the calculation system; And Generate the autocorrelation with delay l for the post - adjustment calculation window based on one or more components incrementally calculated for the post - adjustment calculation window.

10. The computer system program product of claim 9, wherein: The method of generating an autocorrelation with a delay of 1 for the adjusted calculation window further includes indirectly incrementally calculating one or more components of the autocorrelation with a delay of 1 for the adjusted calculation window, and the indirect incremental calculation of the one or more components includes calculating the one or more components one by one based on one or more components other than the component to be calculated.

Images