A method for real-time determination of the repeatability of a given delay in big data itself

Abstract

The autocorrelation of a given delay can be used to determine the repeatability of a given delay of the big data itself. The present invention discloses a method, system and computing system program product that can determine the repeatability of a given delay of the streaming big data itself in real time by incrementally calculating the autocorrelation of a specified delay of a computing window of the big data. The implementation scheme of the present invention includes incrementally calculating two or more components of the autocorrelation of the specified delay of the computing window after adjustment based on two or more components of the autocorrelation of the specified delay of the computing window before adjustment, and then generating the autocorrelation of the specified delay of the adjusted computing window based on the two or more components of the incremental calculation as needed. Incremental calculation of the autocorrelation avoids accessing all data elements in the adjusted computing window and performing repeated calculations, thereby improving computing efficiency, saving computing resources and reducing computing system energy consumption, making it efficient and low-cost to determine the repeatability of a given delay of the streaming big data itself in real time, and some scenarios of real-time determination of the repeatability of a given delay of the big data itself become possible from impossible.

Description

Technical Field

[0001] Big data or streaming data analytics. Background Art

[0002] The Internet, mobile communications, navigation, online games, sensor technology and large-scale computing infrastructure generate massive amounts of data every day. Big data is data that exceeds the processing capabilities of traditional database systems and the analytical capabilities of traditional analytical methods due to its huge size, rapid changes and growth rate.

[0003] Autocorrelation, also known as delay correlation or serial correlation, is a measure of the degree of correlation between a specific time series and the time series itself delayed by l time points. It can be obtained by dividing the co-correlation of observations of a time series separated by l time points by its standard deviation. If the autocorrelation value of a certain delay is 1 or close to 1, it can be considered that the time series or streaming big data has a self-repetitive pattern after the delay. Therefore, it is obvious to judge the repeatability of the big data itself for a given delay based on the autocorrelation of a given delay, but the difficulty and challenge lies in how to calculate the autocorrelation on big data in real time.

[0004] In order to be able to discover and utilize the self-repetitiveness of the latest data in real time and reflect the latest data status, the autocorrelation may need to be recalculated after the large data set changes, such as after new data is added to the large data set or after the calculation window size is adjusted, and therefore many data elements may be repeatedly accessed and used. For example, the autocorrelation is calculated for a calculation window of size n that increases with the newly accessed data elements of the large data set. Each time a data element is accessed, the data element is added to the calculation window, and the n+1 data elements in the calculation window are accessed to recalculate the autocorrelation. Recalculating the autocorrelation using all the data elements in the calculation window involves repeated data access and calculation, which is time-consuming and wastes resources.

[0005] Depending on the needs, the size of the calculation window may be very large. For example, the data elements in the calculation window may be distributed on thousands of computing devices in the cloud platform. Using traditional methods to recalculate autocorrelations on big data after some data changes cannot be processed in real time and occupies and wastes a lot of computing resources. Therefore, using traditional methods to determine the repeatability of big data itself with a given delay in real time not only wastes a lot of computing resources but may also not be able to meet the needs. Summary of the invention

[0006] The present invention extends to methods, systems and computing system program products that incrementally calculate the autocorrelation of a given delay of large data so that the repeatability of the given delay of the large data itself can be determined in real time after adjusting the calculation window size. The computing system includes one or more computing devices. Each computing device includes one or more processors. The computing system includes one or more storage media, in which a data set is stored. The computing system maintains a calculation window size counter. The calculation window size counter maintains the number of data elements in the calculation window of the data set. Incrementally calculating the autocorrelation of a specified delay l (l>0) for an adjusted calculation window includes incrementally calculating two or more components of the autocorrelation of the specified delay of the calculation window before adjustment based on two or more (p (p>1)) components of the autocorrelation of the specified delay of the calculation window before adjustment, and then generating the autocorrelation of the specified delay of the adjusted calculation window based on the two or more components of the incremental calculation as needed. Incremental calculation of autocorrelation only needs to access and use the components of incremental calculation, the newly added data elements, and the l data elements adjacent to the newly added data elements in the calculation window and the l data elements on the other side of the calculation window, while avoiding accessing all data elements in the adjusted calculation window and performing repeated calculations, thereby reducing data access delay, improving computing efficiency, saving computing resources and reducing computing system energy consumption, making real-time judgment of the repeatability of a given delay of streaming big data itself efficient and low-cost, and some scenarios of real-time judgment of the repeatability of a given delay of big data itself possible.

[0007] A computing system is initialized with delay l (l>1), a computing window size n (n>1) and two or more (p (p>1)) components of the autocorrelation of a pre-adjusted computing window at delay l of a large data set of size n stored on one or more storage media.

[0008] The computation system accesses a data element to be added to the pre-adjusted computation window.

[0009] The computing system adjusts the pre-adjustment computing window by adding the data element to be added to the pre-adjustment computing window and makes the adjustment by adding 1 to the computing window size counter.

[0010] The computing system directly incrementally computes one or more v (1≤v≤p) components of the autocorrelation of a given delay l for the adjusted computing window. Directly incrementally computing v (1≤v≤p) components at a given delay l includes: accessing l data elements on both sides of the pre-adjusted computing window; accessing v components of the autocorrelation of the delay l computed for the pre-adjusted computing window; and mathematically adding any contribution of the added data element to each accessed component.

[0011] The computing system indirectly incrementally computes w=pv components of the autocorrelation of the specified delay of the adjusted computing window as needed. Indirectly incrementally computing the w components of the specified delay includes indirectly incrementally computing each of the w components one by one. Indirectly incrementally computing a component of the specified delay includes accessing and using one or more components of the specified delay other than the component to compute the component. The one or more components may be initialized, directly incrementally computed, indirectly incrementally computed, or computed in any other manner.

[0012] The computing system generates an adjusted autocorrelation of the specified delay of the computing window based on a component of the autocorrelation of the specified delay of the adjusted computing window computed by one or more incremental calculations.

[0013] The computing system can continuously access a data element to be added to the computing window, adjust the computing window and the computing window size, directly incrementally calculate v components of the specified delay, and indirectly incrementally calculate w=pv components of the specified delay and calculate the autocorrelation of the specified delay as needed. The computing system can repeat this process as many times as needed.

[0014] This summary is to introduce some selected concepts in a simplified manner, which will be further described in detail below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0015] Other features and advantages of the present invention will be apparent from the following description, will be apparent in part from the description, or may be learned from the practice of the present invention. The features and advantages of the present invention may be realized and obtained from the methods, apparatus, and combinations thereof particularly pointed out in the appended claims. These and other features of the present invention will become more fully apparent from the following description and the appended claims or from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] To describe the manner in which the above and other advantages and features of the present invention can be obtained, a more specific description of the present invention briefly described above will be presented by reference to specific embodiments shown in the attached figures. These figures only describe typical embodiments of the present invention, and therefore they should not be construed as limiting the scope of the present invention:

[0017] Figure 1 A high-level overview of an example computational system that supports incremental computation of autocorrelations is illustrated.

[0018] Figure 1-1 An example computing system architecture is shown that supports incremental computing of autocorrelations of big data and all components are computed in a straightforward incremental manner.

[0019] Figure 1-2 An example computing system architecture that supports incremental computing of autocorrelations of big data with some components computed in a direct incremental manner and some components computed in an indirect incremental manner is shown.

[0020] Figure 2 A flowchart showing an example method for incrementally computing autocorrelations of large data.

[0021] Figure 3-1 The data elements to the left of the join calculation window 300A are shown.

[0022] Figure 3-2 The data elements accessed within the computation window 300A when incrementally computing the autocorrelation for a given delay on large data are shown.

[0023] Figure 3-3 The data elements to the right of the join calculation window 300B are shown.

[0024] Figure 3-4 The data elements accessed within the computation window 300B when incrementally computing the autocorrelation for a given delay on large data are shown.

[0025] Figure 4-1 The definition of autocorrelation and the traditional equation for calculating autocorrelation are shown.

[0026] Figure 4-2 The first autocorrelation incremental calculation algorithm (Incremental Algorithm 1) is shown.

[0027] Figure 4-3 A second autocorrelation incremental calculation algorithm (Incremental Algorithm 2) is shown.

[0028] Figure 4-4 A third autocorrelation incremental calculation algorithm (Incremental Algorithm 3) is shown.

[0029] Figure 5-1 The first calculation window for a calculation instance is shown.

[0030] Figure 5-2 A second calculation window for a calculation instance is shown.

[0031] Figure 5-3 The third calculation window for a calculation instance is shown.

[0032] Figure 6-1 The figure shows the computational complexity comparison between the traditional and incremental autocorrelation algorithms when the computation window size is 6 and the delay is 1.

[0033] Figure 6-2 The figure shows the computational complexity comparison between the traditional and incremental autocorrelation algorithms when the computation window size is 1,000,000 and the delay is 1. DETAILED DESCRIPTION

[0034] Calculating autocorrelation is an effective method for judging the self-given delay repeatability of big data. The present invention extends to a method, system, and computing system program product for incrementally and real-time calculating the autocorrelation of a given delay on big data by incrementally calculating more than two components of the autocorrelation of a specified delay l (1 ≤ l < n) of a computing window of size n (n > 1). A computing system includes one or more processor-based computing devices. Each computing device includes one or more processors. The computing system includes one or more storage media. The computing system stores a data set on the storage media. The computing system has a computing window size counter. The computing window size counter holds the number of data elements in the computing window of the data set. The delay l indicates the delay used for autocorrelation calculation. Embodiments of the present invention include incrementally calculating more than two components of the autocorrelation of a specified delay of an adjusted computing window based on more than two components of the autocorrelation of a specified delay of a pre-adjusted computing window, and then generating the autocorrelation of a specified delay of the adjusted computing window based on the more than two components of the incremental calculation as needed. Incrementally calculating the autocorrelation avoids accessing all data elements in the adjusted computing window and performing repeated calculations, thereby improving computing efficiency, saving computing resources, and reducing the energy consumption of the computing system, making it possible to efficiently and with low power consumption judge the self-given delay repeatability of big data in real time and making some scenarios of judging the self-given delay repeatability of big data in real time possible that were previously impossible.

[0035] Autocorrelation, also known as lag correlation or serial correlation, is a measure of the degree of correlation between a particular time series and the time series itself delayed by l time points. It can be obtained by dividing the covariance of observations of a time series separated by l time points by its standard variance. If the autocorrelation for all different delay values of a time series is calculated, the autocorrelation function of that time series is obtained. For a time series that does not change over time, its autocorrelation values decay exponentially to 0. The values of autocorrelation range between -1 and +1. A value of +1 indicates a perfect positive linear relationship between past and future values of the time series, while a value of -1 indicates a perfect negative linear relationship between past and future values of the time series.

[0036] In this article, a computing window contains the data involved in autocorrelation calculation. The data elements in the computing window have an order relationship, that is, changing the order of the data elements in the computing window can affect the autocorrelation calculation result.

[0037] In this article, a component of autocorrelation is a quantity or expression that appears in the autocorrelation definition formula or any transformation of its definition formula. Autocorrelation is its own largest component. The following are some examples of components of autocorrelation.

[0038]

[0039]

[0040]

[0041]

[0042]

[0043] Autocorrelation can be calculated based on one or more components or a combination thereof, so multiple algorithms support incremental autocorrelation calculation.

[0044] A component can be incrementally calculated directly or indirectly. The difference between them is that when a component is incrementally calculated directly, the component is calculated based on the value of the component in the previous round of calculation, while when the component is incrementally calculated indirectly, the component is calculated using other components outside the component.

[0045] For a given component, it may be incrementally calculated directly in one algorithm but indirectly in another algorithm.

[0046] For any algorithm, at least two components will be incrementally calculated, one component is incrementally calculated directly, and the other component is incrementally calculated directly or indirectly. For a given algorithm, assuming the total number of different components used is p (p > 1), if the number of components incrementally calculated directly is v (1 ≤ v ≤ p), then the number of components incrementally calculated indirectly is w = p - v (0 ≤ w < p). It is possible that all components are incrementally calculated directly (in this case v = p > 1 and w = 0). However, regardless of whether the result of the autocorrelation is needed and accessed in a specific round, the components incrementally calculated directly must be calculated.

[0047] For a given algorithm, if a component is incrementally calculated directly, then the component must be calculated (i.e., whenever a data element is added to the calculation window). However, if a component is incrementally calculated indirectly, then the component can be calculated as needed by using one or more other components outside the component, that is, only when the autocorrelation needs to be calculated and accessed. In this way, when the autocorrelation is not accessed in a certain calculation round, only a small number of components need to be incrementally calculated. An indirectly incrementally calculated component may be used for the direct incremental calculation of a component, in which case the calculation of the indirectly incrementally calculated component cannot be omitted.

[0048] The implementation solution of the present invention includes initializing a delay l (l>1), calculating a window scale counter n (n>1), and two or more (p (p>1)) components for calculating the autocorrelation with a delay of l for a pre-adjustment calculation window of size n for a large data set existing on one or more storage media.

[0049] Initializing the calculation window scale counter includes setting its value with the number of data elements in the pre-adjustment calculation window or accessing or receiving a specified value of the calculation window scale counter. The initialization of the two or more components includes calculating based on the data elements in the pre-adjustment calculation window according to their definitions or accessing or receiving the components that have been calculated from one or more computer-readable media.

[0050] The computing system indirectly incrementally calculates one or more (w (w = p - v)) components as needed. Then, it generates the autocorrelation with a given delay l based on the initialized or incrementally calculated components.

[0051] The computing system accesses a data element to be added to the pre-adjustment calculation window.

[0052] The computing system adjusts the pre-adjustment calculation window by: adding the data element to be added to the pre-adjustment calculation window. The computing system adjusts the calculation window scale counter by: incrementing its value by 1.

[0053] The computing system incrementally calculates a sum or an average or a sum and an average of the post-adjustment calculation window.

[0054] The computing system directly incrementally calculates one or more (v (1 ≤ v < p)) components other than the sum and the average for the autocorrelation with a delay of l for the post-adjustment calculation window. Directly incrementally calculating these v components includes: accessing l data elements on each side of the pre-adjustment calculation window; accessing v components of the autocorrelation with a delay of l calculated for the pre-adjustment calculation window; and mathematically adding any contribution of the data element to be added to each accessed component.

[0055] The computing system indirectly incrementally calculates w = p - v components of the autocorrelation with a given delay l for the post-adjustment calculation window as needed. Indirectly incrementally calculating the w components of the autocorrelation with a given delay l includes incrementally calculating each of the w components of the autocorrelation with a given delay l indirectly one by one. Indirectly incrementally calculating a component with a given delay l includes: accessing one or more components with a given delay l other than the component and calculating the component based on the accessed components. These one or more components with a given delay l can be initialized, directly incrementally calculated, indirectly incrementally calculated, or calculated in any other way.

[0056] The computing system generates an autocorrelation for a given delay l for the adjusted computing window based on one or more components of the autocorrelation for the given delay l computed for the adjusted computing window increment, as needed.

[0057] The computing system can continuously access the data elements to be added to the pre-adjustment calculation window, adjust the pre-adjustment calculation window and the calculation window size counter, incrementally calculate a sum or an average value or a sum and an average value of the adjusted calculation window, directly incrementally calculate one or more components of v specified delays, indirectly incrementally calculate w=pv components of specified delays as needed, generate autocorrelation of a given delay based on one or more incrementally calculated components as needed, and repeat this process multiple times as needed.

[0058] Embodiments of the present invention may include or utilize computing device hardware, such as one or more processors and storage devices described in more detail below, dedicated or general computing devices. The scope of embodiments of the present invention also includes physical and other computing device readable media for carrying or storing computing device executable instructions and / or data structures. These computing device readable media can be any media accessible to general or dedicated computing devices. The computing device readable media that stores computing device executable instructions is storage media (device). The computing device readable media that carries computing device executable instructions is transmission media. Therefore, by way of example and not limitation, embodiments of the present invention may include at least two different types of computing device readable media: storage media (device) and transmission media.

[0059] Storage media (devices) include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disk-read-only memory (CD-ROM), solid-state drive (SSD), flash memory, phase change memory (PCM), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store the required program code in the form of executable instructions or data structures for a computing device and which can be accessed by a general or special computing device.

[0060] A "network" is defined as one or more data links that enable computing devices and / or modules and / or other electronic devices to transmit electronic data. When information is transmitted or provided to a computing device via a network or other communication connection (wired, wireless, or a combination of wired and wireless), the computing device considers the connection to be a transmission medium. The transmission medium may include a network and / or data link for carrying the required program code in the form of computing device executable instructions or data structures and which can be accessed by general or special computing devices. Combinations of the above should also be included in the scope of computing device readable media.

[0061] In addition, when using different computing device components, program code in the form of computing device executable instructions or data structures can be automatically transferred from the transmission medium to the storage medium (device) (or vice versa). For example, computing device executable instructions or data structures received from a network or data link can be temporarily stored in a random access memory in a network interface module (e.g., NIC) and then ultimately transferred to the computing device's random access memory and / or to a smaller, volatile storage medium (device) of the computing device. Therefore, it should be understood that the storage medium (device) can be included in a computing device component that also (or even primarily) uses a transmission medium.

[0062] Computing device executable instructions include, for example, instructions and data that, when executed by a processor, cause a general-purpose computing device or a special-purpose computing device to perform a specific function or a set of functions. Computing device executable instructions can be, for example, binary, intermediate format instructions such as assembly code, or even source code. Although the described object is described in specific language of structural features and / or method actions, it should be understood that the object defined in the attached claims is not necessarily limited to the features or actions described above. Instead, the described features or actions are only disclosed in the form of examples of implementing the claims.

[0063] The embodiments of the present invention can be implemented in a network computing environment configured by various types of computing devices, including personal computers, desktop computers, notebook computers, information processors, handheld devices, multiprocessing systems, microprocessor-based or programmable consumer electronics, network computers, minicomputers, mainframe computers, supercomputers, mobile phones, PDAs, tablet computers, pagers, routers, switches and similar products. The embodiments of the present invention can also be applied to a distributed system environment consisting of local or remote computing devices that perform tasks interconnected by a network (i.e., by wired data links, wireless data links, or a combination of wired data links and wireless data links). In a distributed system environment, program modules can be stored on local or remote storage devices.

[0064] Embodiments of the present invention may also be implemented in a cloud computing environment. In this description and in the claims that follow, "cloud computing" is defined as a model that enables on-demand access to a shared pool of configurable computing resources over a network. For example, cloud computing may be utilized by the marketplace to provide pervasive and convenient on-demand access to a shared pool of configurable computing resources. A shared pool of configurable computing resources may be quickly provisioned through virtualization and provided with low management overhead or low service provider interaction, and then adjusted accordingly.

[0065] The cloud computing model can include various features such as on-demand self-service, broadband network access, resource collection, rapid scaling, metered services, etc. The cloud computing model can also be embodied in various service models, such as software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). The cloud computing model can also be deployed through different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, etc.

[0066] Since the present invention effectively reduces the requirements for computing power, its implementation scheme can also be applied to edge computing.

[0067] Several examples are given in the following sections.

[0068] Figure 1 A high-level overview of an example computing system 100 for computing autocorrelations for large data increments is illustrated. Figure 1 The computing system 100 includes multiple devices connected by different networks, such as a local area network 1021, a wireless network 1022, and the Internet 1023, etc. The multiple devices include, for example, a data analysis engine 1007, a storage system 1011, a real-time data stream 1006, and multiple distributed computing devices that can schedule data analysis tasks and / or query data analysis results, such as a personal computer 1016, a handheld device 1017, and a desktop computer 1018, etc.

[0069] The data analysis engine 1007 may include one or more processors, such as CPU 1009 and CPU 1010, one or more system memories, such as system memory 1008, and component calculation module 131 and autocorrelation calculation module 192. The details of module 131 are illustrated in more detail in other diagrams (e.g., Figure 1-1 and Figure 1-2 ). Storage system 1011 may include one or more storage media, such as storage media 1012 and storage media 1014, which may be used to store large data sets. For example, 1012 and / or 1014 may include data set 123. The data sets in storage system 1011 may be accessed by data analysis engine 1007.

[0070] In general, data stream 1006 may include streaming data from different data sources, such as stock prices, audio data, video data, geospatial data, Internet data, mobile communication data, online game data, bank transaction data, sensor data, and / or closed caption data, etc. Here, a few examples are depicted, and real-time data 1000 may include data collected in real time from sensors 1001, stocks 1002, communications 1003, and banks 1004, etc. Data analysis engine 1007 may receive data elements from data stream 1006. Data from different data sources may be stored in storage system 1011 and accessed by big data analysis, for example, data set 123 may come from different data sources and be accessed by big data analysis.

[0071] Please understand Figure 1 Some concepts are introduced in a very simplified form. For example, the distribution devices 1016 and 1017 may be connected to the data analysis engine 1007 through a firewall, and the data accessed or received by the data analysis engine 1007 from the data stream 1006 and / or the storage system 1011 may be filtered by a data filter, and so on.

[0072] Figure 1-1 The diagram shows an example computing system architecture 100A for incrementally calculating autocorrelations for large data, where all (v=p>1) components are directly incrementally calculated. Regarding the computing system architecture 100A, only the functions and interrelationships of the main components in the architecture will be introduced here, and the process of how these components work together to complete the incremental autocorrelation calculation will be combined later. Figure 2 The process described in Figure 1 Let me introduce you. Figure 1-1 Pictured Figure 1 1006 and 1007 are shown. Reference Figure 1-1, the computing system architecture 100A includes a component computing module 131 and an autocorrelation computing module 192. The component computing module 131 can be tightly coupled to one or more storage media via a high-speed data bus or loosely coupled to one or more storage media managed by the storage system via a network, such as a local area network, a wide area network, or even the Internet. Accordingly, the component computing module 131 and any other connected computing devices and their components can send and receive message-related data (e.g., Internet Protocol ("IP") datagrams and other high-level protocols that use IP datagrams, such as User Datagram Protocol ("UDP"), Real Time Streaming Protocol ("RTSP"), Real Time Transport Protocol ("RTP"), Microsoft Media Server ("MMS"), Transmission Control Protocol ("TCP"), Hypertext Transfer Protocol ("HTTP"), Simple Mail Transfer Protocol ("SMTP"), etc.) on the network. The output of the component computing module 131 will be used as an input to the autocorrelation computing module 192, and the autocorrelation computing module 192 can generate an autocorrelation 193.

[0073] Typically, the storage medium 121 may be a single local storage medium or a complex storage system consisting of multiple physically distributed storage devices managed by a storage management system.

[0074] Storage medium 121 includes data set 124. Generally, data set 124 may include data from different categories, such as stock prices, audio data, video data, geospatial data, Internet data, mobile communication data, online game data, bank transaction data, sensor data, closed caption data, and real-time text.

[0075] As shown, data set 124 includes multiple data elements in multiple storage media 121. For example, data elements 101, 102, 103, 104, and 105 are respectively stored in locations 121A, 121B, 121C, 121D, and 121E of storage media 121. Other locations of storage media 121 store other data elements.

[0076] The calculation window size counter 118 can be implemented by hardware or software. When implemented by hardware, the counter 118 can be a small device that updates the calculation window size. When implemented by software, the counter 118 can be a variable stored in memory, hard disk or other computing device readable storage medium, and its content or value is the calculation window size. Each time a data element is accessed, the computing system adjusts the calculation window by removing the data element from the calculation window and subtracting 1 from the value of the calculation window size counter 118. The calculation window size counter 118 can be accessed or received by the component calculation module 131, and can also be set to a specific value by the reset module 129. In the description of this article, the calculation window size counter and the calculation window size are equivalent and interchangeable.

[0077] For example, Figure 1-1 As shown, after accessing data element 204 , computation window size counter 118 is incremented by 1. Computation window size counter 118 and data element 104 may be accessed by component computation module 131 .

[0078] Then, data element 105 may be accessed and computation window size counter 118 may be incremented by 1. Computation window size counter 118 and data element 105 may be accessed by component computation module 131 .

[0079] Referring to the computing system architecture 100A, the component computing module 131 generally includes v component computing modules for v (v=p>1) components of a set of n data elements of a computation window for direct incremental computation. v is the number of components directly incrementally computed in a given algorithm for a given delayed incremental computation of autocorrelation, and it varies with the incremental algorithm used. Figure 1-1 As shown in FIG. 1 , the component calculation module 131 includes a component Cd1 calculation module 161 and a component Cd v The computing module 162 has v-2 other component computing modules between them, which can be the component Cd2 computing module, the component Cd3 computing module, ..., and the component Cd v-1 Calculation modules. Each component calculation module calculates a specific component for a given delay. Each component calculation module includes an initialization module for initializing a component for the first calculation window and an algorithm for directly incrementally calculating the component for the adjusted calculation window. For example, component Cd1 calculation module 161 includes initialization module 132 to initialize component Cd1 for a given delay and incremental algorithm 133 to incrementally calculate component Cd1 for a given delay, component Cd v The calculation module 162 includes an initialization module 138 to initialize the component Cd of a given delay v and incremental algorithm 139 to incrementally calculate the component Cd for a given delay v .

[0080] Initialization module 132 may be used when initializing component Cd1 or when the autocorrelation calculation is reset. Similarly, initialization module 138 may be used when initializing component Cd v 141 is the initial value of component Cd1. Initialization module 132 may be used to initialize component Cd1 or reset autocorrelation calculation. Initialization module 132 may initialize component Cd1 141 according to the definition of component Cd1 based on the data elements of the calculation window or initialize component Cd1 to contribute a specific value 151 transmitted by reset module 123. Initialization module 138 may be used to initialize component Cd1. v Or reset the autocorrelation calculation. The initialization module 138 can calculate the data elements of the window according to the component Cd v Definition of initialization component Cd v 145 or initialization component Cd v The specific value contribution 181 is transmitted to the reset module 123 .

[0081] The incremental algorithm includes directly incrementally calculating v components of the autocorrelation with a given delay of l for the adjusted calculation window. The incremental algorithm 133 accesses or receives the component Cd1 with a delay of l calculated in the previous round, l data elements on both sides of the calculation window before adjustment, and the data elements added to the calculation window before adjustment as input. The incremental algorithm 133 directly incrementally calculates a new component Cd1 with a delay of l using the component Cd1 with a delay of l calculated in the previous round, l data elements on both sides of the calculation window before adjustment, and the data elements added to the calculation window before adjustment. The contribution addition module 133A can mathematically add any contribution of the added data element to the component Cd1 with a delay of l calculated in the previous round for the calculation window before adjustment, thereby obtaining a new component Cd1 with a delay of l for the adjusted calculation window. The incremental algorithm 139 works in a similar manner to the incremental algorithm 133. The incremental algorithm 139 accesses or receives the component Cd1 with a delay of l calculated in the previous round v , l data elements on both sides of the calculation window before adjustment and the data elements added to the calculation window before adjustment are used as input. The incremental algorithm 139 uses the component Cd with a delay of l in the previous round of calculation v , the l data elements on both sides of the calculation window before adjustment and the data elements added to the calculation window before adjustment are directly incrementally calculated to form a new component Cd with a delay of l v The contribution adding module 139A may mathematically add any contribution of the added data element to the component Cd with a delay of l calculated in the previous round of calculation for the pre-adjustment calculation window. v , thus obtaining a new component Cd with a delay of l after the calculation window is adjusted v .

[0082] refer to Figure 1-1, the computing system architecture 100A further includes an autocorrelation calculation module 192. The autocorrelation calculation module 192 can calculate the autocorrelation 193 of a given delay based on one or more components of a given delay calculated incrementally as needed.

[0083] The computing system can continue to access data elements to be added to the calculation window, adjust the calculation window and the calculation window size, directly incrementally calculate v (1 ≤ v ≤ p) components at a given delay l, calculate the autocorrelation at the given delay l as needed, and repeat this process multiple times as needed.

[0084] Figure 1-2 Illustrated is an example computing system architecture 100B for calculating the autocorrelation incrementally for a large amount of data and directly incrementally calculating part (v (1 ≤ v < p)) of the components and indirectly incrementally calculating part (w = p - v) of the components. In some implementations, the difference between the computing system architectures 100B and 100A is that the architecture 100B includes a component calculation module 135.除此之外, the parts with the same reference numerals as in 100A work in the same way. To avoid repeating what has been explained in the description of 100A before, only the different parts will be discussed here. The number v in 100B may be different from the number v in 100A because some components that are directly incrementally calculated in 100A will be indirectly incrementally calculated in 100B. In 100A, v = p > 1, but in 100B, 1 ≤ v < p. Refer to Figure 1-2 , the computing system architecture 100B includes a component calculation module 135. The output of the component calculation module 131 can be used as the input of the component calculation module 135, the outputs of the calculation modules 131 and 135 can be used as the input of the autocorrelation calculation module 192, and the autocorrelation calculation module 192 can generate the autocorrelation 193. The component calculation module 135 generally includes w = p - v component calculation modules to indirectly incrementally calculate w components. For example, the component calculation module 135 includes a component calculation module 163 for indirectly incrementally calculating the component Ci1 and a component calculation module 164 for indirectly incrementally calculating the component Ci w , and the other w - 2 component calculation modules between them. Indirectly incrementally calculating w components includes indirectly incrementally calculating each of the w components one by one. Indirectly incrementally calculating a component includes accessing and using one or more components other than the component itself. The one or more components can be components that have been initialized, directly incrementally calculated, or indirectly incrementally calculated. Once the w = p - v components are indirectly incrementally calculated, the autocorrelation calculation module 192 can calculate the autocorrelation 193 of a given delay l based on one or more incrementally calculated components.

[0085] Figure 2FIG. illustrates a flow chart of an example method 200 for incrementally calculating autocorrelation for a large data set. Method 200 will be described in conjunction with the components and data of computing system architectures 100A and 100B, respectively.

[0086] Method 200 includes initializing a computing window size counter n (n>1) and p (p>1) components (201) of autocorrelation of a pre-adjustment computing window of a specified size n of a large data set with a specified delay of l (0<l<n). For example, the computing window size counter 118 can initialize the computing window size. The initialization module 132 can initialize the component Cd1 141 with a given delay l using contribution 151. Contribution 151 can be the contribution of data elements 121B and 121C to component Cd1 at the given delay l. The initialization module 138 can initialize the component Cd v 145 with contribution 181. Contribution 181 can be the contribution of data elements 121B and 121C to component Cd v at the given delay l.

[0087] Method 200 includes, when v<p, i.e., not all components are directly incrementally calculated, indirectly incrementally calculating each of the w = p - v components one by one as needed based on one or more components other than the component to be calculated. These w components will only be calculated when the autocorrelation is accessed (208). For example, referring to Figure 1-2 where some of its components are directly incrementally calculated and some are indirectly incrementally calculated, the calculation module 163 can indirectly incrementally calculate component Ci1 based on one or more components other than component Ci1, and the calculation module 164 can indirectly incrementally calculate component Ci w based on one or more components other than component Ci w . These one or more components can be initialized, directly incrementally calculated, indirectly incrementally calculated, or calculated in any other way.

[0088] Method 200 includes generating an autocorrelation of a given delay. When the autocorrelation is accessed, the autocorrelation will be generated based on one or more initialized or incrementally calculated components (209).

[0089] Method 200 includes accessing a data element to be added to the pre-adjustment computing window (202). For example, data element 104 can be accessed.

[0090] Method 200 includes adding the data element to be added to the pre-adjustment computing window and adjusting the computing window size counter n (203). For example, data element 104 is added to the pre-adjustment computing window 122, and then the pre-adjustment computing window 122 is transformed into the adjusted computing window 122A, and the computing window size counter 118 is incremented by 1 after accessing data element 104.

[0091] The method 200 includes directly incrementally computing v (1≤v≤p) components of the autocorrelation with a delay of l for the adjusted calculation window (204), including: accessing l data elements on both sides of the calculation window before adjustment (205); accessing v components of the autocorrelation with a delay of l (206); and mathematically adding any contribution of the added data element to each accessed component (207). Details are described below.

[0092] Directly incrementally computing the v components of the autocorrelation for the specified delay l for the adjusted calculation window includes accessing l data elements on both sides of the pre-adjustment calculation window (205). For example, if the specified delay l=1, the incremental algorithm 133 may access data elements 102 and 103 at 121B and 121C. Similarly, if the specified delay l=1, the incremental algorithm 139 may access data elements 102 and 103 at 121B and 121C.

[0093] Directly incrementally calculating the v components of the autocorrelation with a delay of l for the adjusted calculation window includes accessing the v components of the autocorrelation with a delay of l for the calculation window before adjustment (206). For example, if the specified delay l=1, the incremental algorithm 133 can access the component Cd1 141 with a delay of 1, if the specified delay l=2, the incremental algorithm 133 can access the component Cd1 141 with a delay of 2, etc. Similarly, if the specified delay l=1, the incremental algorithm 139 can access the component Cd1 141 with a delay of 1. v 145, if the specified delay l = 2, the incremental algorithm 139 can access the component Cd with a delay of 2 v 145…….

[0094] Directly incrementally computing the v components of the autocorrelation with a delay of l for the adjusted computation window includes mathematically adding any contributions of the added data elements to each component accessed (207). For example, if the specified delay l=1, directly incrementally computing the component Cd1 143 with a delay of 1 may include the contribution adding module 133A mathematically adding the contribution 152 to the component Cd1 141 with a delay of 1. Similarly, directly incrementally computing the component Cd1 143 with a delay of 1 may include the contribution adding module 133A mathematically adding the contribution 152 to the component Cd1 141 with a delay of 1. v 147 may include contributing to the component Cd with a delay of 1 by the module 139A v 145 mathematically adds contribution 182. Contributions 152 and 182 come from data element 104. Figure 1-1 1-2, component Cd1 143 includes contribution 151 (the contribution from data elements 121B and 121C) and contribution 152 (the contribution from data element 104). Similarly, component Cd v147 includes contribution 181 (contribution from data elements of 121B and 121C) and contribution 182 (contribution from data element 104).

[0095] When autocorrelation is accessed and v < p (i.e., not all components are directly incrementally computed), method 200 includes indirectly incrementally computing w = p - v components with a latency of l as needed (208). These w components are only computed when autocorrelation is accessed. For example, referring to Figure 1-2 Some of its components are directly incrementally computed and some are indirectly incrementally computed. The computing module 163 can indirectly incrementally compute component Ci1 based on one or more components other than component Ci1, and the computing module 164 can be based on component Ci w outside one or more components to indirectly incrementally compute component Ci w . These one or more components can be initialized, directly incrementally computed,, indirectly incrementally computed, or computed in any other way.

[0096] Method 200 includes generating an autocorrelation with a latency of l based on one or more initialized or incrementally computed components of the autocorrelation with a latency of l as needed (209). When the autocorrelation is accessed, the autocorrelation is computed based on one or more incrementally computed components; otherwise only v components are directly incrementally computed. When the autocorrelation is accessed, method 200 includes indirectly incrementally computing w = p - v components with a latency of l one by one as needed using one or more components other than the components themselves, and then generating an autocorrelation with a latency of l. For example, in architecture 100A, the autocorrelation module 192 can generate an autocorrelation 193 with a given latency. In architecture 100B, the computing module 163 can indirectly incrementally compute Ci1 based on one or more components other than component Ci1, and the computing module 164 can be based on component Ci w outside one or more components to indirectly incrementally compute Ci w , ……, the autocorrelation computing module 192 can generate an autocorrelation 193 with a given latency (209). Once the autocorrelation with a given latency is computed, method 200 includes accessing the next data element to be added to the computing window to start the next round of incremental computation. Each time a new round of incremental computation starts, the adjusted computing window of the previous round becomes the pre-adjusted computing window of the new round of incremental computation.

[0097] As more data elements are accessed, 202 - 207 can be repeated, and 208 - 209 can be repeated as needed. For example, when data element 104 is accessed and components Cd1 143 to component Cd vAfter the components within the range 147 are calculated, data element 105 can be accessed (202). Method 200 includes adding the most recently accessed data element to the pre-adjustment calculation window and adjusting the calculation window size counter (203). For example, data element 105 can be added to calculation window 122A, and calculation window size counter 118 can be incremented by 1 after accessing data element 105.

[0098] The method 200 includes directly incrementally calculating v (1≤v≤p) components of the autocorrelation with a delay of l for the adjusted calculation window based on the v components of the pre-adjusted calculation window (204), which includes accessing l data elements on both sides of the pre-adjusted calculation window (205), accessing the v components (206), and mathematically adding any contribution of the added data elements to each accessed component (207). For example, referring to 100A and 100B, at a specified delay such as l=1, the algorithm 133 can be used to directly incrementally calculate the component Cd1 144 with a delay of 1 for the adjusted calculation window 122B based on the component Cd1 143 with a delay of 1 calculated for the adjusted calculation window 122A (204). The incremental algorithm 133 can access the data elements (data elements 102) of 121B and the data elements (data elements 104) of 121D (205). The incremental algorithm 133 can access the component Cd1 143 with a delay of 1 (206). Directly incrementally calculating the component Cd1 144 with a delay of 1 includes the contribution adding module 133A mathematically adding the contribution 153, i.e., the contribution of the data element 105, to the component Cd1 143 with a delay of 1 (207). Similarly, at a specified delay, such as l=1, the incremental algorithm 139 may be used to directly incrementally calculate the component Cd1 144 with a delay of 1 for the adjusted computation window 122B. v 148 is based on the component Cd with a delay of 1 calculated for the adjusted calculation window 122A v 147. Increment algorithm 139 can access the data element of 121B (data element 102) and the data element of 121D (data element 104). Increment algorithm 139 can access component Cd with a delay of 1 v 147. Direct incremental calculation of component Cd with delay 1 v 148 includes contributing module 139A to component Cd with delay 1 v 147 mathematically adds contribution 185, which is the contribution of data element 105.

[0099] As shown, component Cd1 144 with a delay of 1 includes other contributions 151 (contributions from data elements 121B and 121C), contribution 152 (contribution from data element 104), and contribution 153 (contribution from data element 105). v148 includes other contributions 181 (contributions from data elements 121B and 121C), contribution 182 (contribution from data element 104), and contribution 183 (contribution from data element 105).

[0100] The method 200 includes computing w components and autocorrelations for a given delay incrementally as needed.

[0101] Method 200 includes, as needed, indirectly incrementally computing w components and autocorrelations for a given delay, if only the autocorrelations are accessed. If the autocorrelations are not accessed, method 200 includes continuing to access the next data element to be added for the next calculation window (202). If the autocorrelations are accessed, method 200 includes indirectly incrementally computing w components for a given delay (208), computing the autocorrelations for the given delay based on one or more of the incrementally computed components for the given delay (209).

[0102] When the next data element to be added to the calculation window is accessed, component Cd1 144 can be used to directly incrementally calculate the next component Cd1. Component Cd v 148 can be used to directly incrementally calculate the next component Cd v .

[0103] Method 200 includes reset 210. Reset 210 can be used to reset the incremental autocorrelation calculation. When reset 210 is called after 201 or 209, the calculation window size counter and the v (1≤v≤p) components of the autocorrelation at a given delay can be initialized. For example, component Cd1 141 can be initialized by definition with the data elements in the calculation window or initialized to a given calculated value. The latter can occur when combining incremental autocorrelation calculation with iterative autocorrelation calculation or decremental autocorrelation calculation. Component Cd v 145 can be initialized in a similar manner.

[0104] Figure 3-1 The diagram shows the data elements added to the left of the calculation window 300A when incrementally calculating autocorrelations on large data. The calculation window 300A can be empty or non-empty. As time goes by, old data elements, such as x m+n , then x m+n-1 , then x m+n-l It can be added to the left of the calculation window 300A.

[0105] Figure 3-2The diagram illustrates data accessed from a computation window 300A when incrementally computing autocorrelations on large data. When 300A is empty, the computation system may continue to add data elements to 300A until the number of data elements reaches l+1, where l is a given delay, and then initialize two or more (v) components. When 300A is non-empty, the v components may have been computed. If not, the v components may be initialized based on the data elements of 300A. The v components of delay l may be directly incrementally computed using the l data elements on both sides of the computation window 300A (excluding the added data elements) and the v components of the computation window before adjustment. If the specified delay is 1, the leftmost 1 data element (excluding the added data elements) and the rightmost 1 data element of the computation window 300A are accessed. If the specified delay is 2, the leftmost 2 data elements (excluding the added data elements) and the rightmost 2 data elements of the computation window 300A are accessed. If the specified delay is l, the leftmost l data elements (excluding the added data elements) and the rightmost l data elements of the computation window 300A will be accessed. For a given increment algorithm, v is a constant, and the number of operands of the indirect increment w=pv components is also a constant, so for a given delay, the amount of data access and the amount of computation are reduced and are constant. The larger the computation window size n, the more significant the reduction in the amount of data access and computation.

[0106] Figure 3-3 The diagram shows the data elements added to the right of the calculation window 300B when incrementally calculating autocorrelations on large data. The calculation window 300B can be empty or non-empty. As time goes by, new data elements, such as x m+1 , then x m+2 , then x m+l+1 The right side of the calculation window 300B may be added.

[0107] Figure 3-4The diagram illustrates data accessed from the computation window 300B when incrementally computing autocorrelations on large data. When 300B is empty, the computation system may continue to add data elements to 300B until the number of data elements reaches l+1, where l is a given delay, and then initialize two or more (v) components. When 300B is non-empty, the v components may have been computed. If not, the v components may be initialized based on the data elements of 300B. The v components of delay l may be directly incrementally computed using the l data elements on both sides of the computation window 300B (excluding the added data elements) and the v components of the computation window before adjustment. If the specified delay is 1, the leftmost 1 data element and the rightmost 1 data element of the computation window 300B (excluding the added data elements) are accessed. If the specified delay is 2, the leftmost 2 data elements and the rightmost 2 data elements of the computation window 300B (excluding the added data elements) are accessed. If the specified delay is l, the leftmost l data elements and the rightmost l data elements (excluding the added data elements) of the computation window 300B will be accessed. For a given increment algorithm, v is a constant, and the number of operands of the indirect increment w=pv components is also a constant, so for a given delay, the amount of data access and the amount of computation are reduced and are constant. The larger the computation window size n, the more significant the reduction in the amount of data access and computation.

[0108] Figure 4-1 The figure shows the definition of autocorrelation. Assume X = {x i |i=m+1,…,m+n} is a computation window of size n containing data involved in the autocorrelation computation of a large dataset. Suppose that over time a data element is added to the computation window X. Whenever a data element is added, the computation window is considered a new computation window. A new round of computation will start whenever any component of the autocorrelation needs to be recomputed because of changes in the data in the computation window.

[0109] The calculation result of autocorrelation is not only related to the value of each data element in the calculation window, but also to the order of each data element. Therefore, different positions of data elements in the calculation window require different processing. There are three cases. First, adding data element x m Go to the far left of the calculation window. Second, add the data element x m+n+1 To the far right of the calculation window. Third, add the data element x a to any position outside the calculation window. The third case rarely occurs so only the first two cases are discussed here. The equations used to calculate one or more components in these two cases are different. To distinguish, define the adjusted calculation window in the first case as X I , the second case is X II .

[0110] Calculate X I and X II There is no difference between the equations for the sum and the average. Equations 401 and 402 are respectively the sum S of all data elements in the calculation window X with a calculation size of n in the kth round. k and average Equation 403 is the autocorrelation ρ of the given delay l for the calculation window X for the kth round. (k,l) Equations 404 and 405 are the sum of all data elements in the adjusted calculation window with a calculation size of n in the k+1th round, respectively. k+1 and average Equation 406 is the traditional equation for calculating the adjusted calculation window X for the k+1th round I The autocorrelation ρ of a given delay l I (k+1,l) Equation 407 is the traditional equation for calculating the adjusted calculation window X for the k+1th round II The autocorrelation ρ of a given delay l II (k+1,l) The traditional equation.

[0111] To demonstrate how to use component increments to calculate autocorrelation, three different incremental autocorrelation algorithms are provided as examples. A new round of calculations begins each time there is a data change in the calculation window (e.g., 122 → 122A → 122B). A sum or average is the basic component for calculating autocorrelation. The equation for incrementally calculating a sum or average is the incremental component equation used by all example incremental autocorrelation calculation algorithms.

[0112] Figure 4-2 The first example incremental autocorrelation calculation algorithm (incremental algorithm 1) is described. Equations 401 and 402 can be used to initialize the components S respectively. k and / or Equations 408, 409, and 410 may be used to initialize component SS, respectively. k , SX k , and covX (k,l) Equation 411 can be used to calculate the autocorrelation ρ (k,l) When the data element x m When adding the leftmost calculation window, incremental algorithm 1 includes component S k+1 or SS k+1 ,SX k+1 , and covX I (k+1,l) The incremental calculation, once the component SX k+1 and covX I (k+1,l)is calculated, the autocorrelation ρ I (k+1,l) can be calculated based on them. Once the component S k and / or Available, equations 412 and 413 can be used to incrementally calculate the adjusted calculation window X I Component S k+1 and Once the component SS k Available, Equation 414 can be used to directly incrementally calculate the adjusted calculation window X I Components of SS k+1 Once component S k+1 or and SS k+1 Available, equation 415 can be used to indirectly calculate the incremental adjustment calculation window X I Components of SX k+1 Once the component covX (k,l) , SS k+1 , S k or and S k+1 or Available, Equation 416 can be used to directly incrementally calculate the adjusted calculation window X I Components of covX I (k+1,l) 412, 413, 415, and 416 each contain multiple equations but only one of each is required depending on whether the sum or the mean or both are available. Once the component cov X (k+1,l) and SX k+1 is calculated, equation 417 can be used to indirectly calculate the adjusted calculation window X I The autocorrelation ρ of a given delay l I (k+1,l) When the data element x m+n+1 When adding the rightmost part of the calculation window, the incremental algorithm 1 includes the component S k+1 or SS k+1 ,SX k+1 , and covX II (k+1,l) The incremental calculation, once the component SX k+1 and covX II (k+1,l) is calculated, the autocorrelation ρ II (k+1,l) Equations 418 and 419 can be used to incrementally calculate the adjusted calculation window X, respectively. II Component S k+1 and Once the component S k and / or Equation 420 can be used to directly incrementally calculate the adjusted calculation window X II Components of SS k+1 Once the component SS k Available. 421 can be used for indirect incremental calculation to adjust the calculation window X II Components of SX k+1 Once the component S k+1 or and SS k+1 Equation 422 can be used to directly incrementally calculate the adjusted calculation window X II Components of covX II (k+1,l) Once the component covX (k,l) , SS k+1 , S k or and S k+1 or 418, 419, 421, and 422 each contain multiple equations but only one of each is required depending on whether the sum or average or both are available. Equation 423 can be used to indirectly calculate the incremental adjustment calculation window X II The autocorrelation ρ of a given delay l II (k+1,l) Once the component covX II (k+1,l) and SX k+1 is calculated.

[0113] Figure 4-3 The second example incremental autocorrelation calculation algorithm (incremental algorithm 2) is described. Equations 401 and 402 can be used to initialize the component S respectively. k and / or Equations 424 and 425 can be used to initialize component SX, respectively. k and covX (k,l) Equation 426 can be used to calculate the autocorrelation ρ (k,l) When the data element x m When added to the leftmost side of the computation window, incremental algorithm 2 includes component S k+1 or SX k+1 , and covX I (k+1,l) The incremental calculation, once the component SX k+1 and covX I (k+1,l) is calculated, the autocorrelation ρ I (k+1,l) Once the component S k and / or Available, equations 427 and 428 can be used to incrementally calculate the adjusted calculation window X I Component S k+1 and Once the component SX k ,S k+1 and / or Available, Equation 429 can be used to directly increment the adjusted calculation window X I Components of SX k+1 Equation 430 can be used to directly incrementally calculate the adjusted calculation window X I Components of covX I (k+1,l) Once the component covX (k,l) , S k or and S k+1 or 427, 428, 429, and 430 each contain multiple equations but only one of them is needed depending on whether the sum or the average or both are available. Once the component covX I (k+1,l) and SX k+1 is calculated, equation 431 can be used to indirectly calculate the adjusted calculation window X I The autocorrelation ρ of a given delay l I (k+1,l) When the data element x m+n+1 When adding the rightmost part of the calculation window, incremental algorithm 2 includes component S k+1 or SX k+1 , and covX II (k+1,l) The incremental calculation, once the component SX k+1 and covX II (k+1,l) is calculated, the autocorrelation ρ II (k+1,l) Equations 432 and 433 can be used to incrementally calculate the adjusted calculation window X, respectively. II Component S k+1 and Once the component S k and / or Equation 434 can be used to directly incrementally calculate the adjusted calculation window X II Components of SX k+1 Once the component SX k , S k+1 and / or Equation 435 can be used to directly incrementally calculate the adjusted calculation window X II Components of covX II(k+1,l) Once the component covX (k,l) , S k or and S k+1 or 432, 433, 434, and 435 each contain multiple equations but only one of each is required depending on whether the sum or average or both are available. Equation 436 can be used to indirectly calculate the adjusted calculation window X II The autocorrelation ρ of a given delay l II (k+1,l) Once the component covX II (k+1,l) and SX k+1 is calculated.

[0114] Figure 4-4 The third example incremental autocorrelation calculation algorithm (incremental algorithm 3) is described. Equations 401 and 402 can be used to initialize the component S respectively. k and / or Equations 437 and 438 can be used to initialize component SX, respectively. k and covX (k,l) Equation 439 can be used to calculate the autocorrelation ρ (k,l) When the data element x m When adding the leftmost calculation window, incremental algorithm 3 includes component S k+1 or SX k+1 , and covX I (k+1,l) The incremental calculation, once the component SX k+1 and covX I (k+1,l) is calculated, the autocorrelation ρ I (k+1,l) Equations 440 and 441 can be used to incrementally calculate the adjusted calculation window X, respectively. I Component S k+1 and Once the component S k and / or Equation 442 can be used to directly incrementally calculate the adjusted calculation window X I Components of SX k+1 Once the component SX k , S k and / or and S k+1 and / or Equation 443 can be used to directly incrementally calculate the adjusted calculation window X I Components of covX I (k+1,l)Once the component covX (k,l) , S k or and S k+1 or Available. 440, 441, 442, and 443 each contain multiple equations but only one of each is required depending on whether the sum or average or both are available. Equation 444 can be used to indirectly calculate the incremental adjustment calculation window X I The autocorrelation ρ of a given delay l I (k+1,l) Once the component covX I (k+1,l) and SX k+1 is calculated. When the data element x m+n+1 When adding the rightmost part of the calculation window, incremental algorithm 3 includes component S k+1 or SX k+1 , and covX II (k+1,l) The incremental calculation, once the component SX k+1 and covX II (k+1,l) is calculated, the autocorrelation ρ II (k+1,l) Equations 445 and 446 can be used to incrementally calculate the adjusted calculation window X, respectively. II Component S k+1 and Once the component S k and / or Equation 447 can be used to directly increment the adjusted calculation window X II Components of SX k+1 Once the component SX k , S k and / or and S k+1 and / or Equation 448 can be used to directly increment the adjusted calculation window X II Components of covX II (k+1,l) Once the component covX (k,l) , S k or and S k+1 or 445, 446, 447, and 448 each contain multiple equations but only one of them is needed depending on whether the sum or the average or both are available. Once the component covX II (k+1,l) and SX k+1 is calculated, equation 449 can be used to indirectly calculate the adjusted calculation window XII The autocorrelation ρ of a given delay l II (k+1,l) .

[0115] To demonstrate the incremental autocorrelation algorithms and how they compare to traditional algorithms, three examples are given below. Data from three calculation windows are used. For the traditional algorithm, the calculation process for all three calculation windows is exactly the same. For the incremental algorithm, the first calculation window performs the initialization of two or more components, and the second and third calculation windows perform incremental calculations.

[0116] Figure 5-1 , Figure 5-2 , Figure 5-3 The first calculation window, the second calculation window, and the third calculation window for a calculation example are shown respectively. Calculation window 503 includes 4 data elements of large data set 501: 8, 3, 6, 1. Calculation window size 503 (n) is 4. Calculation window 504 includes 5 data elements of large data set 501: 8, 3, 6, 1, 9. Calculation window size 505 (n) is 5. Calculation window 505 includes 6 data elements of large data set 501: 8, 3, 6, 1, 9, 2. Calculation window size 507 (n) is 6. This calculation example assumes that the data elements are added to the rightmost side of the calculation window.

[0117] First, the autocorrelations of the windows 502, 504, and 506 with a delay of 1 are calculated using a conventional algorithm.

[0118] Compute the autocorrelation with a delay of 1 for computation window 502:

[0119]

[0120]

[0121]

[0122]

[0123] Without any optimization, computing the autocorrelation with a latency of 1 for a computation window of size 4 requires 2 divisions, 7 multiplications, 8 additions, and 10 subtractions.

[0124] The same equations and procedures can be used to Figure 5-2 The calculation window 504 shown calculates the autocorrelation sum with a delay of 1 as Figure 5-3 The calculation window 505 shown calculates the autocorrelation with a delay of 1. The calculation window 504 with a size of 5 calculates the autocorrelation with a delay of 1. This calculation includes 2 divisions, 9 multiplications, 11 additions, and 13 subtractions without optimization. This calculation includes 2 divisions, 11 multiplications, 14 additions, and 16 subtractions without optimization. Conventional algorithms usually need to complete 2 divisions, 2n-1 multiplications, 3n-(l+3) additions, and 3n-2l subtractions when calculating the autocorrelation with a window size of n and a given delay of l without optimization.

[0125] Next, the incremental algorithm 1 is used to calculate the autocorrelations of windows 502, 504, and 506 with a delay of 1.

[0126] Compute the autocorrelation with a delay of 1 for computation window 502:

[0127] 1. Initialize the components of round 1 using equations 402, 408, 409, and 410 respectively SS1, SX1, and

[0128] xvX (1,1) :

[0129]

[0130]

[0131]

[0132]

[0133] 2. Calculate the autocorrelation ρ of the first round using equation 411 (1，1) :

[0134]

[0135] There are 2 divisions, 9 multiplications, 8 additions, and 7 subtractions when calculating the autocorrelation with a delay of 1 for calculation window 502.

[0136] Compute the autocorrelation with a delay of 1 for computation window 504:

[0137] 1. Incrementally calculate the components of the second round using equations 419, 420, 421, and 422 SS2, SX2, and

[0138] xvX II (2，1) :

[0139]

[0140] SS2=SS1+x a 2 =110+9 2 =110+81=191

[0141]

[0142]

[0143] 2. Calculate the autocorrelation ρ of the second round using equation 423 II (2，1) :

[0144]

[0145] There are 2 divisions, 8 multiplications, 7 additions, and 6 subtractions when calculating the autocorrelation with a delay of 1 by calculating window 504 increments.

[0146] Compute the autocorrelation with a delay of 1 for computation window 506:

[0147] 1. Incrementally calculate the components of round 3 using equations 419, 420, 421, and 422 SS3, SX3, and

[0148] xvX II (3，1) :

[0149]

[0150] SS3=SS2+x a 2 =191+2 2 =191+4=195

[0151]

[0152]

[0153] 2. Calculate the autocorrelation ρ of the third round using equation 423 II (3,1) :

[0154]

[0155] There are 2 divisions, 8 multiplications, 7 additions, and 6 subtractions when calculating the autocorrelation with a delay of 1 for calculation window 506.

[0156] Next, the incremental algorithm 2 is used to calculate the autocorrelations of windows 502, 504, and 506 with a delay of 1.

[0157] Compute the autocorrelation with a delay of 1 for computation window 502:

[0158] 1. Initialize the components of round 1 using equations 402, 424, and 425 SX1, and covX (1,1) :

[0159]

[0160]

[0161]

[0162] 2. Calculate the autocorrelation ρ of the first round using equation 426 (1,1) :

[0163]

[0164] There are 2 divisions, 9 multiplications, 8 additions, and 7 subtractions when calculating the autocorrelation with a delay of 1 for calculation window 502.

[0165] Compute the autocorrelation with a delay of 1 for computation window 504:

[0166] 1. Use equations 433, 434, and 435 to incrementally calculate the components of the second round SX2, and covX II (2,1) :

[0167]

[0168]

[0169]

[0170]

[0171] 2. Calculate the autocorrelation ρ of the second round using equation 436 II (2,1) :

[0172]

[0173] When calculating the autocorrelation with a delay of 1 by calculating window 504 increments, there are 2 divisions, 6 multiplications, 7 additions, and 7 subtractions.

[0174] Compute the autocorrelation with a delay of 1 for computation window 506:

[0175] 1. Use equations 433, 434, and 435 to incrementally calculate the components of round 3 SX3, and covX II (3,1) :

[0176]

[0177]

[0178]

[0179] 2. Calculate the autocorrelation ρ of the third round using equation 436 II (3,1) :

[0180]

[0181] When calculating the autocorrelation with a delay of 1 by calculating window 506 increments, there are 2 divisions, 6 multiplications, 7 additions, and 7 subtractions.

[0182] Next, the incremental algorithm 3 is used to calculate the autocorrelations of windows 502, 504, and 506 with a delay of 1.

[0183] Compute the autocorrelation with a delay of 1 for computation window 502:

[0184] 1. Initialize the first round of components using equations 402, 437, and 438 SX1, and covX (1,1) :

[0185]

[0186]

[0187]

[0188] 2. Calculate the autocorrelation ρ of the first round using equation 439 (1,1) :

[0189]

[0190] There are 2 divisions, 9 multiplications, 8 additions, and 7 subtractions when calculating the autocorrelation with a delay of 1 for calculation window 502.

[0191] Compute the autocorrelation with a delay of 1 for computation window 504:

[0192] 1. Use equations 446, 447, and 448 to incrementally calculate the components of the second round SX2, and covX II (2,1) :

[0193]

[0194]

[0195]

[0196] 2. Calculate the autocorrelation ρ of the second round using equation 449 II (2,1) :

[0197]

[0198] When calculating the autocorrelation with a delay of 1 by calculating the window 504 increment, there are 2 divisions, 5 multiplications, 7 additions, and 7 subtractions.

[0199] Compute the autocorrelation with a delay of 1 for computation window 506:

[0200] 1. Use equations 446, 447, and 448 to incrementally calculate the components of round 3 SX3, and covX II (3,1) :

[0201]

[0202]

[0203]

[0204] Calculate the autocorrelation ρ of the third round using Equation 449 II (3,1) :

[0205]

[0206] When calculating the autocorrelation with a delay of 1 by calculating window 506 increments, there are 2 divisions, 5 multiplications, 7 additions, and 7 subtractions.

[0207] In the three examples above, the mean is used for the incremental autocorrelation calculation. The sum can also be used for the incremental autocorrelation calculation, but the operands are different. In addition, in the three examples above, the data elements are added to the rightmost part of the calculation window. When the data elements are added to the leftmost part of the calculation window, the calculation process is similar, but a different set of equations are applied.

[0208] Figure 6-1 The figure shows the comparison of the computational complexity of the traditional autocorrelation algorithm and the incremental autocorrelation algorithm when n=6 and the delay is 1. As shown in the figure, the division operation, multiplication operation, addition operation, and subtraction operation of any incremental algorithm and the traditional algorithm are similar.

[0209] Figure 6-2The figure shows the comparison of the computational complexity of the traditional autocorrelation algorithm and the incremental autocorrelation algorithm when n=1,000,000 and the delay is 1. As shown in the figure, any incremental algorithm has much fewer multiplication, addition and subtraction operations than the traditional algorithm. The incremental autocorrelation algorithm can process data that needs to be processed on thousands of computers on a single machine. It greatly improves computing efficiency, reduces computing resources, and reduces the energy consumption of computing equipment, making it efficient and low-cost to judge the repeatability of a given delay of big data itself in real time, and some scenarios of real-time judgment of the repeatability of a given delay of big data itself possible from impossible.

[0210] The present invention can be implemented in other specific ways without departing from its ideas or essential features. The implementation scheme described in this application is only exemplary and not restrictive in all respects. Therefore, the scope of the present invention is indicated by the attached claims rather than the previous description. All changes equivalent to the meaning and scope of the claims in the claims are included in their scope.

Claims

1. A method implemented by a computing system based on one or more computing devices, characterized in that: A computing system based on a computing device initializes a computing window size counter n, where n > 1, a latency l, where 0 < l < n, and more than two components of autocorrelation with latency l for an unadjusted computing window of a data set stored on at least one storage medium in the computing system. The computing window counter specifies the number of data elements included in the unadjusted computing window; The computing system based on the computing device accesses a data element to be added to the unadjusted computing window; The computing system based on the computing device adjusts the unadjusted computing window by: Adding the accessed data element to the unadjusted computing window; And Adjusting the computing window size counter; The computing system based on the computing device, based on at least more than two components of autocorrelation with latency l of the unadjusted computing window, incrementally calculates more than two components of autocorrelation with latency l for the adjusted computing window, and avoids accessing and using all data elements in the adjusted computing window during the incremental calculation of the more than two components to reduce data access latency, improve computing efficiency, save computing resources, and reduce the energy consumption of the computing system; And The computing system based on the computing device generates autocorrelation with latency l for the adjusted computing window based on one or more components incrementally calculated for the adjusted computing window.

2. The method according to claim 1, characterized in that: The accessing a data element to be added includes accessing multiple data elements to be added to the unadjusted computing window. The method further includes adjusting the unadjusted computing window for each of the multiple data elements to be added, incrementally calculating more than two components, and generating autocorrelation with latency l for the adjusted computing window.

3. The method according to claim 2, characterized in that: The generating autocorrelation with latency l for the adjusted computing window is performed if and only if the autocorrelation is accessed.

4. The method according to claim 3, characterized in that: The generating autocorrelation with latency l for the adjusted computing window further includes the computing system based on the computing device indirectly incrementally calculating one or more components of autocorrelation with latency l for the adjusted computing window. Indirectly incrementally calculating the one or more components includes calculating the one or more components separately one by one based on one or more components other than the components to be calculated.

5. A computing system, characterized in that: One or more processors; One or more storage media, where at least one storage medium stores a data set; and One or more computing modules, when executed by at least one of the one or more processors, perform a method that includes: a. Initializing a computing window size counter n, where n > 1, a latency l, where 0 < l < n, and more than two components of autocorrelation with latency l for an unadjusted computing window of the data set. The computing window counter specifies the number of data elements included in the unadjusted computing window; b. Accessing a data element to be added to the unadjusted computing window; c. Adjusting the unadjusted computing window, including: Adding the accessed data element to the unadjusted computing window; and Adjusting the computing window size counter; d. Based on at least two or more components of the autocorrelation with a delay of l in the pre-adjustment calculation window, calculate two or more components of the autocorrelation with a delay of l for the post-adjustment calculation window increment, and avoid accessing and using all data elements in the post-adjustment calculation window during the process of incrementally calculating the two or more components to reduce data access latency, improve calculation efficiency, save calculation resources, and reduce the energy consumption of the calculation system; and e. Generate the autocorrelation with a delay of l for the post-adjustment calculation window based on one or more components calculated incrementally for the post-adjustment calculation window increment.

6. The computing system according to claim 5, characterized in that: The one or more calculation modules, when executed by at least one of the one or more processors, execute b, c, d, and e multiple times.

7. The computing system according to claim 6, characterized in that: Execute e if and only if the autocorrelation with a delay of l in the post-adjustment calculation window is accessed.

8. The computing system according to claim 7, characterized in that: The execution of e further includes indirectly incrementally calculating, by the calculation system, one or more components of the autocorrelation with a delay of l for the post-adjustment calculation window increment, and the indirect incremental calculation of the one or more components includes calculating the one or more components separately one by one based on one or more components other than the components to be calculated.

9. A calculation system program product, running on a calculation system including one or more calculation devices, the calculation system including one or more processors and one or more storage media, the calculation system program product including multiple calculation device executable instructions, when these calculation device executable instructions are run by at least one calculation device in the calculation system, causing the calculation system to execute a method, characterized in that: Initialize a calculation window size counter n, where n > 1, a delay l, where 0 < l < n, and two or more components of the autocorrelation with a delay of l for a pre-adjustment calculation window of a data set stored in at least one storage media in the calculation system, the calculation window counter specifying the number of data elements included in the pre-adjustment calculation window; Access a data element to be added to the pre-adjustment calculation window; Adjust the pre-adjustment calculation window by: Adding the data element to be added to the pre-adjustment calculation window; And Adjusting the calculation window size counter; Based on at least two or more components of the autocorrelation with a delay of l in the pre-adjustment calculation window, calculate two or more components of the autocorrelation with a delay of l for the post-adjustment calculation window increment, and avoid accessing and using all data elements in the post-adjustment calculation window during the process of incrementally calculating the two or more components to reduce data access latency, improve calculation efficiency, save calculation resources, and reduce the energy consumption of the calculation system; And Generate the autocorrelation with a delay of l for the post-adjustment calculation window based on one or more components calculated incrementally for the post-adjustment calculation window increment.

10. A computer system program product, characterized in that : The calculation system program product includes multiple calculation device executable instructions, when these calculation device executable instructions are run by at least one calculation device in a calculation system including one or more calculation devices and one or more storage devices, causing the calculation system to implement the method according to any one of claims 1-4.

Images