A method for determining the asymmetry of streaming data distribution in real time

Abstract

Skewness can be used to determine the asymmetry of data distribution. The present invention discloses a method, system and computing device readable medium for iteratively calculating the skewness of stream data in real time by iteratively calculating two or more components of the skewness of a calculation window of a specified size, so as to determine the asymmetry of the distribution of stream data in real time. The embodiment of the present invention includes iteratively calculating two or more components of the skewness of the calculation window after adjustment based on the two or more components of the skewness of the calculation window before adjustment, and then generating the skewness of the adjusted calculation window based on the two or more components of the iterative calculation as needed. Iterative calculation of skewness can update the calculation results in real time based on the latest data and avoid accessing all data elements in the calculation window after adjustment and performing repeated calculations, thereby improving calculation efficiency, saving calculation resources and reducing the energy consumption of the computing system, so that real-time judgment of the asymmetry of the distribution of stream data can be efficient and low-cost, and some application scenarios of real-time judgment of the asymmetry of the distribution of stream data can be changed from impossible to possible.

Description

Technical Field

[0001] Big data or streaming data analytics. Background Art

[0002] The Internet, mobile communications, navigation, online games, sensor technology and large-scale computing infrastructure generate massive amounts of data every day. Big data is data that exceeds the processing capabilities of traditional database systems and the analytical capabilities of traditional analytical methods due to its huge size, rapid changes and growth rate. How to process and analyze big data efficiently, in real time and with minimal resources poses a difficult challenge to data analysts and computer scientists.

[0003] Streaming data is data that is continuously transmitted from a data source and continuously received by a receiver. Streaming data can be real-time data collected by sensors and continuously transmitted to a computing device or electronic device. Usually this includes the continuous reception of data elements with similar formats at time intervals. Streaming data can also be data that is continuously read from a memory, such as a large data set stored in the memory of multiple computing devices. Therefore, streaming data processing algorithms can be extended to big data processing, because large data sets can be considered as data streams with irregular time intervals as they accumulate over time.

[0004] Some statistical calculations may need to be recalculated after new data elements are received to reflect the latest data conditions. For example, the skewness may need to be calculated for a calculation window containing n data elements that were recently added to a large data set on a storage medium, so that each time a data element is received or accessed, the data element is added to the calculation window and the nth data element is removed from the calculation window, and the n data elements in the calculation window are accessed to recalculate the skewness. Recalculating the skewness using all the data elements in the calculation window involves repeated data access and calculation, which is time-consuming and wastes resources.

[0005] Depending on the needs, the size of the computation window may be very large. For example, the data elements in the computation window may be distributed across thousands of computing devices on the cloud platform. Recalculating the skewness using traditional methods on streaming data after some data changes cannot be done in real time and occupies and wastes a lot of computing resources. Summary of the invention

[0006] Skewness can be used to determine data distribution asymmetry. The present invention discloses a method, system and computing device readable medium for iteratively calculating the skewness of stream data in real time by iteratively calculating two or more components of the skewness of a calculation window of a specified size, so as to determine the asymmetry of the distribution of stream data in real time. Iteratively calculating the skewness for a data element of an adjusted calculation window includes iteratively calculating the two or more components of the skewness of the data element of the calculation window before adjustment, and then generating the skewness of the data element of the adjusted calculation window based on the two or more components of the iterative calculation as needed. Iterative calculation of skewness only needs to access and use the components of the iterative calculation, the newly added and removed data elements, and avoids accessing all data elements in the adjusted calculation window and performing repeated calculations, thereby reducing data access delay, improving computing efficiency, saving computing resources and reducing computing system energy consumption, so that real-time judgment of the asymmetry of the distribution of stream data can be efficient and low-cost, and some real-time judgment of the asymmetry of the distribution of stream data becomes possible from impossible.

[0007] The sum or average value of the calculation window is a special component that must be calculated iteratively. Assume that the total number of all iteratively calculated components (including the sum or average value of the calculation window) in the same round of iterative calculation is p (p>1). The number of components directly iterated is v (1≤v≤p), and the number of components indirectly iterated is w=pv (w≥0). The sum or average value of the calculation window is a special component that must be calculated iteratively. The sum or average value can be calculated iteratively directly or indirectly.

[0008] The computing system initializes two or more components (a total of p (p>1), including a sum or an average value or a sum and an average value, and one or more other components of skewness) of a pre-adjustment computing window of a data stream stored in a buffer with a designated computing window size of n (n>1). The initialization of the two or more components includes calculating the two or more components through the definition of the components based on the data elements in the pre-adjustment computing window or receiving or accessing the two or more components that have been calculated from a computing device readable medium.

[0009] The computing system receives a new data element to be added to the pre-adjusted computing window.

[0010] The computing system saves the received data elements into a buffer.

[0011] The computing system adjusts the pre-adjustment calculation window by removing the oldest data element from the pre-adjustment calculation window and adding the received data element to the pre-adjustment calculation window.

[0012] The calculation system iteratively calculates a sum, an average, or a sum and an average for the adjusted calculation window.

[0013] The computing system directly iterates and calculates v (1≤v≤p) components of the skewness of the adjusted computing window other than the sum and the average. Directly iteratively calculating the v components includes: accessing the removed data elements and the added data elements; accessing the v components of the skewness of the computing window before adjustment; mathematically removing any contribution of the removed data elements from each accessed component; and mathematically adding any contribution of the added data elements to each of the v components.

[0014] The computing system indirectly iterates and calculates w=pv components of the skewness of the adjusted computing window as needed. Indirectly iteratively calculating the w components includes indirectly iteratively calculating each of the w components one by one. Indirectly iteratively calculating a component includes: accessing and using one or more components other than the component to calculate the component. The one or more components may be initialized, directly iteratively calculated, or indirectly iteratively calculated.

[0015] The computing system generates the skewness of the adjusted computing window based on the components of the skewness of the adjusted computing window of one or more iterative calculations as needed, and the adjusted computing window will become the pre-adjustment computing window of the next round of iterative calculations.

[0016] The computing system can continuously receive a new data element, save the received data element into a buffer, adjust the pre-adjustment computing window, iteratively calculate a sum, an average, or a sum and an average of the post-adjustment computing window, directly iteratively calculate v components, indirectly iteratively calculate w=pv components and calculate skewness as needed. The computing system can repeat this process as many times as needed.

[0017] This summary is to introduce some selected concepts in a simplified manner, which will be further described in detail below. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0018] Other features and advantages of the present invention will be apparent from the following description, will be apparent in part from the description, or may be learned from the practice of the present invention. The features and advantages of the present invention may be realized and obtained from the methods, apparatus, and combinations thereof particularly pointed out in the appended claims. These and other features of the present invention will become more fully apparent from the following description and the appended claims or from the practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] To describe the manner in which the above and other advantages and features of the present invention can be obtained, a more specific description of the present invention briefly described above will be presented by reference to specific embodiments shown in the attached diagrams. The present invention will be described and explained in more detail and detail through the following diagrams, please understand that these diagrams only describe typical embodiments of the present invention, and therefore they should not be understood as limiting the scope of the present invention:

[0020] Figure 1 The diagram shows a high-level overview of a computation system for an example of iteratively computing skewness for streaming data.

[0021] Figure 1-1 The figure shows an example of a computing system architecture that is skewed for iterative computing of streaming data and where all components perform direct iterative computing.

[0022] Figure 1-2 The diagram shows an example of a computing system architecture that performs skewed iterative computing for streaming data and in which some components perform direct iterative computing and some components perform indirect iterative computing.

[0023] Figure 2 An example of a flow chart of a method for iteratively calculating skewness for streaming data is shown.

[0024] Figure 3 The diagram illustrates data elements in a computation window accessed by a computation system for iterative computation of skewness on streaming data.

[0025] Figure 4-1 The diagram shows the definition of skewness and the traditional equation for calculating skewness over a calculation window.

[0026] Figure 4-2 The first iterative skewness calculation example algorithm (iterative algorithm 1) is shown.

[0027] Figure 4-3 A second iterative skewness calculation example algorithm (iterative algorithm 2) is shown.

[0028] Figure 4-4 The third iterative skewness calculation example algorithm (iterative algorithm 3) is shown.

[0029] Figure 5-1 The first calculation window for a calculation instance is shown.

[0030] Figure 5-2 The second calculation window for this calculation instance is displayed.

[0031] Figure 5-3 The third calculation window for this calculation instance is displayed.

[0032] Figure 6-1The figure shows the comparison of the computational workload of the traditional skew algorithm and the iterative skew algorithm when the computational window size is 4.

[0033] Figure 6-2 The figure shows the comparison of the computational workload of the traditional skew algorithm and the iterative skew algorithm when the computational window size is 1,000,000. Specific implementation methods

[0034] Skewness can be used to determine the asymmetry of data distribution. The present invention extends to a method, system, and computing device readable medium that iteratively calculates the skewness on stream data by iteratively calculating two or more components of the skewness of a calculation window of a specified size, thereby determining the asymmetry of the distribution of stream data in real time. A computing system includes one or more computing devices. Each computing device includes one or more processors. The computing system can access at least one data stream. The computing system includes a buffer on one or more storage media for retaining data elements in a calculation window composed of multiple data elements from the data stream that are involved in the skewness calculation. The computing system maintains a calculation window size counter n (n>1). The calculation window size counter n indicates the number of data elements in the calculation window of the buffer. An embodiment of the present invention includes iteratively calculating two or more components of the skewness of the calculation window after adjustment based on the two or more components of the skewness of the calculation window before adjustment, and then generating the skewness of the adjusted calculation window based on the two or more components of the iterative calculation as needed. Iterative calculation of skewness avoids accessing all data elements in the adjusted calculation window and performing repeated calculations, thereby improving calculation efficiency, saving computing resources and reducing the energy consumption of the computing system, making real-time judgment of the asymmetric distribution of streaming data efficient and low-cost, and making some real-time judgments of the asymmetric distribution of streaming data possible from impossible.

[0035] The computing system includes an input buffer for storing large data or streaming data elements. The buffer can be in memory or other computer-readable media, such as a hard disk or other media, or even multiple distributed files distributed on multiple computing devices, which are interconnected end-to-end to form one or two "circular buffers".

[0036] In this article, a component of skewness is a quantity or expression that appears in the skewness definition formula or in any transformation of its definition formula. Skewness is its own largest component. Skewness can be calculated based on one or more components or their combination, so multiple algorithms support iterative skewness calculations. Examples of components of skewness.

[0037] ·

[0038] ·

[0039] ·

[0040] ·

[0041] ·

[0042] ·

[0043] ·

[0044] ·

[0045] ·

[0046] ·

[0047] ·

[0048] A component can be iteratively calculated directly or indirectly. The difference between them is that when a component is iteratively calculated directly, the component is calculated based on the value of the component in the previous round of calculation, while when the component is iteratively calculated indirectly, the component is calculated based on other components outside the component.

[0049] For a given component, it may be iteratively calculated directly in one algorithm but indirectly in another algorithm.

[0050] For any algorithm, at least two components will be iteratively calculated, one component is iteratively calculated directly and the other component is iteratively calculated directly or indirectly. For a given algorithm, assume that the total number of different components used is p (p > 1). If the number of components iteratively calculated directly is v (1 ≤ v ≤ p), then the number of components iteratively calculated indirectly is w = p - v (0 ≤ w < p). It is possible that all components are iteratively calculated directly (in this case v = p > 1 and w = 0). However, regardless of whether the skewed result is needed and accessed in a specific round, the components iteratively calculated directly must be calculated.

[0051] For a given algorithm, if a component is directly iteratively computed, then the component must be computed (i.e., every time an existing data element is removed from the computation window and every time a data element is added to the computation window). However, if a component is indirectly iteratively computed, then the component can be computed on demand by using one or more other components in addition to the component, i.e., only when the skewness needs to be computed and accessed. In this way, when the skewness is not accessed in a particular computation round, only a small number of components need to be iteratively computed. An indirectly iteratively computed component may be used in a directly iteratively computed component, in which case the computation of the indirectly iterative component cannot be omitted.

[0052] The skewness can be calculated as needed. When the skewness does not need to be accessed each time the data in the calculation window changes, the computing system only needs to iteratively calculate one or more components for each data change. Iteratively calculating one or more components avoids accessing all previous inputs and doing repeated calculations, thereby improving computing efficiency. The skewness can be generated by the computing system based on the iterative calculation of one or more components when it needs to be accessed.

[0053] In this specification and claims, a "circular buffer" is a data structure that uses a single "buffer" of fixed size that seems to be connected end to end, sometimes also called a ring buffer. The "buffer" can be a common circular buffer, which is usually a space allocated in local memory, or a "virtual circular buffer", which is not necessarily in memory but a file on the hard disk or even multiple distributed files on multiple distributed computing devices as long as these distributed files are logically connected end to end to form a "circular buffer".

[0054] Typically, input data is added to a buffer of size n. When the buffer is not full of data, there are at least two approaches. One is to not perform skewness calculations until the buffer is full of data and then use the first n data to calculate two or more components according to the definition of the component. Another approach is to incrementally calculate the skewness from the beginning when necessary using the method described in another patent application of the inventor until the buffer is full. Once the buffer is full and two or more components of the skewness of the first n data elements are calculated, the iterative algorithm provided herein can be used to iteratively calculate two or more components of the skewness of the data element, and then the skewness can be calculated based on the iteratively calculated components.

[0055] In this specification and claims, an oldest data element refers to a data element that is nth closest in time to the most recently received data element in a calculation window.

[0056] The implementation scheme of the present invention includes iteratively calculating two or more components of the skewness of the calculation window after adjustment based on the two or more components of the skewness calculated for the calculation window before adjustment. The sum or average value of the calculation window is a special component that must be iteratively calculated. The sum or average value can be directly or indirectly iteratively calculated. For a given iterative skewness algorithm, it is assumed that the total number of all iteratively calculated components (including the sum or average value of the calculation window) in the same round of iterative calculation is p (p>1). The number of components of direct iteration is v (1≤v≤p), and the number of components of indirect iteration is w=pv (w≥0).

[0057] The computing system initializes two or more components (a total of p (p>1), including a sum or an average value or a sum and an average value, and one or more other components of skewness) of a pre-adjustment calculation window of a given size n (n>1) in a buffer. The initialization of the two or more components includes calculating the data elements in the pre-adjustment calculation window according to their definitions or accessing or receiving the already calculated components from one or more computing device readable media.

[0058] The computing system receives a new stream data element to be added to the pre-adjusted computing window.

[0059] The computing system saves the received data elements into a buffer.

[0060] The computing system adjusts the pre-adjustment calculation window by removing the oldest data element from the pre-adjustment calculation window and adding the received data element to the pre-adjustment calculation window.

[0061] The calculation system iteratively calculates a sum, an average, or a sum and an average for the adjusted calculation window.

[0062] The computing system directly iterates and calculates one or more v (1≤v≤p) components of the skewness other than the sum and the average for the adjusted computing window. Directly iteratively calculating the v components includes: accessing the removed data elements and the added data elements; accessing the v components of the skewness calculated for the pre-adjusted computing window; mathematically removing any contribution of the removed data elements from each of the v components; and mathematically adding any contribution of the added data elements to each of the v components.

[0063] The computing system indirectly iterates and calculates w=pv components of the skewness of the adjusted computing window as needed. Indirectly iteratively calculating the w components includes indirectly iteratively calculating each of the w components one by one. Indirectly iteratively calculating a component includes: accessing and using one or more components other than the component to calculate the component. The one or more components may be initialized, directly iteratively calculated, or indirectly iteratively calculated.

[0064] The computing system calculates the skewness for the adjusted computing window as needed based on one or more components for iteratively calculating the skewness for the adjusted computing window. The adjusted computing window will become the pre-adjustment computing window for the next round of iterative calculation.

[0065] The computing system can continuously receive data elements to be added to the pre-adjustment calculation window, save the received data elements to a buffer, adjust the pre-adjustment calculation window, iteratively calculate a sum, an average, or a sum and an average of the post-adjustment calculation window, directly iteratively calculate one or more v components, indirectly iteratively calculate w=pv components as needed, calculate the skewness based on one or more iteratively calculated components as needed, and repeat this process multiple times as needed.

[0066] Embodiments of the present invention may include or utilize a dedicated or general purpose computing device that includes computing device hardware, such as one or more processors and storage devices described in more detail below. The scope of embodiments of the present invention also includes physical and other computing device readable media for carrying or storing computing device executable instructions and / or data structures. These computing device readable media can be any media accessible to a general or dedicated computing device. A computing device readable medium that stores computing device executable instructions is a storage medium (device). A computing device readable medium that carries computing device executable instructions is a transmission medium. Therefore, by way of example and not limitation, embodiments of the present invention may include at least two different types of computing device readable media: a storage medium (device) and a transmission medium.

[0067] Storage media (devices) include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), read-only compact disk memory (CD-ROM), solid-state drive (SSD), flash memory, phase change memory (PCM), other types of memory, other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other media that can be used to store the required program code in the form of executable instructions or data structures of a computing device and which can be accessed by a general or special computing device.

[0068] A "network" is defined as one or more data links that enable computing devices and / or modules and / or other electronic devices to transmit electronic data. When information is transmitted or provided to a computing device by a network or other communication connection (wired, wireless, or a combination of wired and wireless), the computing device considers the connection to be a transmission medium. The transmission medium may include a network and / or data link for carrying the required program code in the form of computing device executable instructions or data structures and which can be accessed by general or special computing devices. Combinations of the above should also be included in the scope of computing device readable media.

[0069] In addition, when using different computing device components, program code in the form of computing device executable instructions or data structures can be automatically transferred from the transmission medium to the storage medium (device) (or vice versa). For example, the computing device executable instructions or data structures received from the network or data link can be temporarily stored in the random access memory in the network interface module (e.g., NIC), and then ultimately transferred to the computing device's random access memory and / or to a smaller volatile storage medium (device) of the computing device. Therefore, it should be understood that the storage medium (device) can be included in the computing device components that also (or even primarily) use the transmission medium.

[0070] Computing device executable instructions include, for example, instructions and data that, when executed by a processor, cause a general purpose computing device or a special purpose computing device to perform a specific function or a set of functions. The computing device executable instructions may be, for example, binary, intermediate format instructions such as assembly code, or even source code. Although the described subject matter is described in specific language of structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features or actions are disclosed only in the form of examples of implementing the claims.

[0071] The embodiments of the present invention can be implemented in a network computing environment configured by various types of computing devices, including personal computers, desktop computers, laptop computers, information processors, handheld devices, multi-processing systems, microprocessor-based or programmable consumer electronics, network computers, minicomputers, mainframe computers, supercomputers, mobile phones, PDAs, tablet computers, pagers, routers, switches and similar products. The embodiments of the present invention can also be applied to a distributed system environment consisting of local or remote computing devices that perform tasks interconnected by a network (that is, by wired data links, wireless data links, or a combination of wired data links and wireless data links). In a distributed system environment, program modules can be stored on local or remote storage devices.

[0072] Embodiments of the present invention may also be implemented in a cloud computing environment. In this description and in the claims that follow, "cloud computing" is defined as a model that enables on-demand access to a shared pool of configurable computing resources over a network. For example, cloud computing may be utilized by the marketplace to provide pervasive and convenient on-demand access to a shared pool of configurable computing resources. A shared pool of configurable computing resources may be quickly provisioned through virtualization and provided with low management overhead or low service provider interaction, and then adjusted accordingly.

[0073] The cloud computing model can include various features such as on-demand self-service, broadband network access, resource collection, rapid expansion and contraction, metered services, etc. The cloud computing model can also be embodied in various service models, such as software as a service ("SaaS"), platform as a service ("PaaS"), and infrastructure as a service ("IaaS"). The cloud computing model can also be deployed through different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, etc.

[0074] Since the present invention effectively reduces the requirements for computing power, its implementation scheme can also be applied to edge computing.

[0075] Several examples are given in the following sections.

[0076] Figure 1 A high-level overview of an example computing system 100 for iteratively computing skewness for streaming data is shown. Figure 1 The computing system 100 includes multiple devices connected by different networks, such as a local area network 1021, a wireless network 1022, and the Internet 1023. The multiple devices include, for example, a data analysis engine 1007, a storage system 1011, a real-time data stream 1006, and multiple distributed computing devices that can schedule data analysis tasks and / or query data analysis results, such as a personal computer 1016, a handheld device 1017, and a desktop computer 1018.

[0077] The data analysis engine 1007 may include one or more processors, such as CPU 1009 and CPU 1010, one or more system memories, such as system memory 1008, and component calculation module 131 and skewness calculation module 191. The details of module 131 are illustrated in more detail in other diagrams (e.g., Figure 1-1 and Figure 1-2 The storage system 1011 may include one or more storage media, such as storage media 1012 and storage media 1014, which may be used to store large data sets. The data sets in the storage system 1011 may be accessed by the data analysis engine 1007.

[0078] Typically, data stream 1006 may include streaming data from different data sources, such as stock prices, audio data, video data, geospatial data, Internet data, mobile communication data, online game data, bank transaction data, sensor data, and / or closed caption data, etc. Here, a few examples are depicted, and real-time data 1000 may include data collected in real time from sensors 1001, stocks 1002, communications 1003, and banks 1004, etc. Data analysis engine 1007 may receive data elements from data stream 1006. Data from different data sources may be stored in storage system 1011 and accessed for big data analysis.

[0079] Please understand Figure 1 Some concepts are introduced in a very simplified form. For example, the distribution devices 1016 and 1017 may be connected to the data analysis engine 1007 through a firewall, and the data accessed or received by the data analysis engine 1007 from the data stream 1006 and / or the storage system 1011 may be filtered by a data filter, and so on.

[0080] Figure 1-1 The figure shows an example computing system architecture 100A for iteratively calculating skewness for a data stream, where all (v=p>1) components are directly iteratively calculated. Regarding the computing system architecture 100A, only the functions and relationships of the main components in the architecture will be introduced here, and the process of how these components cooperate to jointly complete the iterative skewness calculation will be combined later. Figure 2 The process described in Figure 1 Let me introduce you. Figure 1-1 Pictured Figure 1 1006 and 1007 are shown. Reference Figure 1-1 The computing system architecture 100A includes a component computing module 131 and a skew computing module 191. The component computing module 131 may be tightly coupled to one or more storage media via a high-speed data bus or loosely coupled to one or more storage media managed by the storage system via a network, such as a local area network, a wide area network, or even the Internet. Accordingly, the component computing module 131 and any other connected computing devices and their components may send and receive message-related data (e.g., Internet Protocol ("IP") datagrams and other high-level protocols that use IP datagrams, such as User Datagram Protocol ("UDP"), Real Time Streaming Protocol ("RTSP"), Real Time Transport Protocol ("RTP"), Microsoft Media Server ("MMS"), Transmission Control Protocol ("TCP"), Hypertext Transfer Protocol ("HTTP"), Simple Mail Transfer Protocol ("SMTP"), etc.) on the network. The output of the component computing module 131 may be used as an input to the skew computing module 191, and the skew computing module 191 may generate the skew 192.

[0081] Typically, data stream 190 may be a sequence of digitally encoded signals (ie, packets of data or data packets) used to transmit or receive information during transmission, respectively. Data stream 190 may be a real-time stream or a streamed stored data.

[0082] As stream data elements are received, they may be placed in circular buffer 121. For example, data elements 101, 102, 103, 104, 105, 106, 107, 108, and 109 may be placed at locations 121A, 121B, 121C, 121D, 121E, 121F, 121G, 121H, and 121I, respectively.

[0083] Then, data element 110 may be received. Data element 110 may be placed at position 121A (overlapping data element 101).

[0084] As shown, circular buffer 121 has 9 positions, 121A-121I. This buffer has a calculation window 122 containing 8 (n=8) data elements. The data elements in calculation window 122 can be switched as new data elements are placed in circular buffer 121. For example, when data element 109 is placed in position 121I, calculation window 122 will become calculation window 122A. When data element 110 is placed in position 121A, calculation window 122A will become calculation window 122B.

[0085] Referring to the computing system architecture 100A, the component computing module 131 generally includes v component computing modules for directly iterating and computing v (v=p>1) components of a set of n data elements of the computing window. v is the number of components directly iteratively computed in a given algorithm of iterative computing skewness, which varies with the iterative algorithm used. Figure 1-1 As shown in FIG. 1 , the component calculation module 131 includes a component Cd1 calculation module 161 and a component Cd v The computing module 162 has v-2 other component computing modules between them, which can be the component Cd2 computing module, the component Cd3 computing module, ..., and the component Cd v-1 Calculation modules. Each component calculation module calculates a specific component. Each component calculation module includes an initialization module for initializing a component for the first pre-adjustment calculation window and an algorithm for directly iteratively calculating the component for the post-adjustment calculation window. For example, component Cd1 calculation module 161 includes initialization module 132 for initializing component Cd1 and iterative algorithm 133 for iteratively calculating component Cd1. v The calculation module 162 includes an initialization module 138 to initialize the component Cd v and iterative algorithm 139 to iteratively calculate the component Cd v .

[0086] Initialization module 132 can be used when initializing component Cd1 or when the skew calculation is reset. Similarly, initialization module 138 can be used when initializing component Cd v or when the skewness calculation is reset.

[0087] refer to Figure 1-1 The computing system architecture 100A further includes a skewness calculation module 191. The skewness calculation module 191 can calculate the skewness 192 based on one or more initialization or iterative calculation components as needed.

[0088] Figure 1-2 Illustrated is an example computing system architecture 100B that skews for data stream iteration calculations and directly iteratively calculates some (v (1 ≤ v < p)) components and indirectly iteratively calculates some (w = p - v) components. In some implementations, the difference between computing system architectures 100B and 100A is that architecture 100B includes a component computing module 135.除此之外, parts with the same reference numerals as 100A work in the same way. To avoid repeating what was previously explained in the description of 100A, only the different parts will be discussed here. The number v in 100B may be different from the number v in 100A because some components that were directly iteratively calculated in 100A are indirectly iteratively calculated in 100B. In 100A, v = p > 1, but in 100B, 1 ≤ v < p. Refer to Figure 1-2 , the computing system architecture 100B includes a component computing module 135. The output of the component computing module 131 can be used as the input of the component computing module 135, and the outputs of the computing modules 131 and 135 can be used as the input of the skew computing module 191, which can generate a skew 192. The component computing module 135 generally includes w = p - v component computing modules for indirectly iteratively calculating w components. For example, the component computing module 135 includes a component computing module 163 for indirectly iteratively calculating the component Ci1 and a component computing module 164 for indirectly iteratively calculating the component Ci w , and the other w - 2 component computing modules between them. Indirectly iteratively calculating w components includes indirectly iteratively calculating each of the w components one by one. Indirectly iteratively calculating a component includes accessing and using one or more components other than the component itself. The one or more components can be initialized, directly iteratively calculated, or indirectly iteratively calculated.

[0089] Figure 2 Illustrated is a flowchart of an example method 200 for iteratively calculating skews for stream data. The method 200 will be described in conjunction with the components and data of the computing system architectures 100A and 100B respectively.

[0090] The method 200 includes initializing v (1 ≤ v ≤ p, p > 1) components of a skew for a pre - adjustment calculation window of a specified size n (n > 1) (201). For example, in the computing system architectures 100A and 100B, the initialization module 132 can initialize the value 141 of the component Cd1 with the values of contribution 151 (the contribution of data element 101), contribution 152 (the contribution of data element 102), and contribution 153 (the contributions of other data elements 103, 104, 105, 106, 107, and 108). Similarly, the initialization module 138 can access the component Cd vand initializes component Cd with the values ​​of contribution 181 (the contribution of data element 101), contribution 182 (the contribution of data element 102), and contribution 183 (the contributions of the other data elements 103, 104, 105, 106, 107, and 108). v The value is 145.

[0091] Method 200 includes receiving a data element to be added to the pre-adjustment calculation window (202). For example, data element 109 may be received after data elements 102-108 are received.

[0092] Method 200 includes storing the received data element in a buffer (203). For example, referring to 100A and 100B, data element 109 may be stored at location 121I in buffer 121.

[0093] Method 200 includes adjusting the pre-adjustment calculation window (204). Adjusting the pre-adjustment calculation window includes removing the oldest data element from the pre-adjustment calculation window (205) and adding the received data element to the pre-adjustment calculation window (206). For example, after data element 101 is removed from pre-adjustment calculation window 122 and data element 109 is added to pre-adjustment calculation window 122, pre-adjustment calculation window 122 becomes post-adjustment calculation window 122A.

[0094] Method 200 includes directly iteratively calculating v components of skewness for an adjusted calculation window (207), including: accessing data elements removed from the pre-adjustment calculation window and data elements added to the pre-adjustment calculation window (208); accessing v components of skewness initialized or calculated for the pre-adjustment calculation window (209); mathematically removing any contribution of the removed data elements from each accessed component (210); and mathematically adding any contribution of the added data elements to each accessed component (211). Details are described below.

[0095] Directly iteratively calculating the v components of the skewness for the adjusted calculation window includes accessing the data elements removed from the pre-adjustment calculation window and the data elements added to the pre-adjustment calculation window (208). For example, the iterative algorithm 133 can access the data elements 101 and 109, and the iterative algorithm 139 can also access the data elements 101 and 109.

[0096] Directly iteratively calculating the v components of the skewness for the adjusted calculation window includes accessing the v components of the skewness initialized or calculated for the pre-adjustment calculation window (209). For example, the iterative algorithm 133 may access the value 141 of the component Cd1, and the iterative algorithm 139 may access the component Cd v The value is 145.

[0097] The v components for directly iteratively calculating the skewness of the adjusted calculation window adjusted calculate by mathematically removing any contribution of the removed data element from each accessed component (210). For example, directly iteratively calculating the value 143 of component Cd1 may include contribution removal module 133A mathematically removing contribution 151 (the contribution of data element 101) from the value 141 of component Cd1, and directly iteratively calculating the value 147 of component Cd v may include contribution removal module 139A mathematically removing contribution 181 (the contribution of data element 101) from the value 145 of component Cd v The v components for directly iteratively calculating the skewness of the adjusted calculation window adjusted calculate by mathematically adding any contribution of the newly added data element to each accessed component (211). For example, directly iteratively calculating the value 143 of component Cd1 may include contribution addition module 133B mathematically adding contribution 154 to the value 141 of component Cd1, and directly iteratively calculating the value 147 of component Cd

[0098] may include contribution addition module 139B mathematically adding contribution 184 to the value 145 of component Cd v Contributions 154 and 184 are both contributions from data element 109. As shown in FIGS. 1-2, the value 143 of component Cd1 includes contribution 152 (the contribution of data element 102), other contributions 153 (the contributions of data elements 103-108), and contribution 154 (the contribution of data element 109). Similarly, the value 147 of component Cd v includes contribution 182 (the contribution of data element 102), other contributions 183 (the contributions of data elements 103-108), and contribution 184 (the contribution of data element 109).

[0099] As Figure 1-1 shown in FIGS. 1-2, the value 143 of component Cd1 includes contribution 152 (the contribution of data element 102), other contributions 153 (the contributions of data elements 103-108), and contribution 154 (the contribution of data element 109). Similarly, the value 147 of component Cd v includes contribution 182 (the contribution of data element 102), other contributions 183 (the contributions of data elements 103-108), and contribution 184 (the contribution of data element 109).

[0100] The skewness can be calculated as needed, i.e., only when accessed, but whenever a data element is removed and a data element is added to the pre-adjustment calculation window, the v components must be calculated.

[0101] When the skewness is accessed and v < p (i.e., not all components are directly iteratively calculated), method 200 includes indirectly iteratively calculating w = p - v components as needed (212). These w components are only calculated when the skewness is accessed. For example, referring to Figure 1-2 which has some components directly iteratively calculated and some components indirectly iteratively calculated, calculation module 163 can indirectly iteratively calculate component Ci1 based on one or more components other than component Ci1, and calculation module 164 can be based on component Ci wOne or more components outside of the calculation component Ci to indirectly iterate w The one or more components may be initialized, directly iteratively computed, or indirectly iteratively computed.

[0102] Method 200 includes calculating skewness (213) as needed using one or more initialized or iteratively calculated components. Figure 1-1 The skewness calculation module 191 can be based on the value 143 of the component Cd1 to the component Cd v The skewness 192 is calculated for one or more components of the value 147 .

[0103] 202-211 can be repeated as more data elements are received. 212-213 can be repeated as needed. A new round of iterative calculation begins each time a data element is received. The adjusted calculation window of the previous round becomes the adjusted calculation window of the new round of iterative calculation. For example, after calculating the value 143 of component Cd1 to component Cd v After the value of 147 is within the range of components, data element 110 can be received (202). 110 can be placed at position 121A to cover data element 101 (203). By removing the oldest data element 102 (205) and adding the data element 110 to be added (206), the adjusted calculation window 122A can become the adjusted calculation window 122B (204). The previous round of adjusted calculation window 122A becomes the pre-adjustment calculation window in the new round of iterative calculation.

[0104] The iterative algorithm 133 may directly iteratively calculate the value 144 of the component Cd1 (of the adjusted calculation window 122B) based on the value 143 of the component Cd1 (of the adjusted calculation window 122A) (207). The iterative algorithm 133 may access the removed data element 102 and the added data element 110 (208). The iterative algorithm 133 may access the value 143 of the component Cd1 (209). Directly iteratively calculating the value 144 of the component Cd1 may include the contribution removal module 133A mathematically removing the contribution 152 (i.e., the contribution of the removed data element 102) from the value 143 of the component Cd1 (210). Directly iteratively calculating the value 144 of the component Cd1 may include the contribution addition module 133B mathematically adding the contribution 155 (i.e., the contribution of the added data element 110) to the value 143 of the component Cd1 (211). Similarly, the iterative algorithm 139 may use the component Cd1 to calculate the value 144 of the component Cd1. v The value 147 (of the adjusted calculation window 122A) of the direct iterative calculation component Cd v The value 148 (of the adjusted calculation window 122B) (207). The iterative algorithm 139 can access the removed data element 102 and the added data element 110 (208). The iterative algorithm 139 can access the component Cdv The value of 147 (209). Directly iterate and calculate the component Cd v The value 148 may include the contribution removal module 139A from the component Cd v The value 147 of 147 mathematically removes the contribution 182 (ie, the contribution of the removed data element 102) (210). Direct iteration calculates the component Cd v The value 148 of the data element 148 may include the contribution adding module 139B mathematically adding the contribution 185 (ie, the contribution of the added data element 110) to the component Cd v The value is 147 (211).

[0105] like Figure 1-1 1-2, the value 144 of component Cd1 includes other contributions 153 (the contributions of data elements 103-108), contribution 154 (the contribution of data element 109), and contribution 155 (the contribution of data element 110). Similarly, component Cd v The value 148 includes other contributions 183 (the contributions of data elements 103 - 108 ), contribution 184 (the contribution of data element 109 ), and contribution 185 (the contribution of data element 110 ).

[0106] Method 200 includes, as needed, indirectly iteratively calculating w components and the skewness. If the skewness is not accessed, method 200 includes continuing to receive the next data element to be added for the next calculation window (202). If the skewness is accessed, method 200 includes indirectly iteratively calculating w components (212), and calculating the skewness based on one or more iteratively calculated components (213).

[0107] When the next data element to be added is received, the value 144 of component Cd1 can be used to directly iterate and calculate the new value of component Cd1. v The value 148 can be used to directly iterate and calculate the component Cd v The new value of .

[0108] Figure 3 The diagram shows the data elements accessed in a computation window 300 for iteratively computing skewness on streaming data. The difference between a computation window and a computation set is that the data elements on a computation window are ordered (e.g., data elements are always removed from the leftmost or rightmost of a computation window and data elements can be added to the leftmost or rightmost of a computation window). Figure 3 What is shown is the situation when the calculation window moves to the right (moving to the left is similar, but the direction is opposite, and the data is added from the leftmost side of the calculation window and removed from the rightmost side). Figure 3, the earliest received data element is removed from the leftmost side of the computation window 300 and a newly received data element is added to the rightmost side of the computation window 300. The first n data elements are accessed to compute one or more (p(p=v+w)) components of the computation window. As time goes by, the oldest data element is removed from the computation window and the oldest data element is added to the computation window, e.g. n+1 , then x n+2 ,x n+3 etc., will be added to the calculation window in order to directly iterate and calculate the v components of the skewness of the adjusted calculation window and indirectly iterate and calculate w=pv components. One or more of these iteratively calculated components will be used to calculate the skewness. These v components can be directly iteratively calculated by the data elements removed from the calculation window, the added data elements and the v components of the previous calculation window without accessing other data elements in the calculation window 300. For a given iterative algorithm, v is a constant, so the number of operations for directly iteratively calculating the v components is also a constant, and the number of operations for indirectly iteratively calculating the w=pv components is also a constant. Therefore, the computational workload of calculating all p components of a given calculation window of size n is reduced. The skewness can be calculated using one or more iteratively calculated components so that the overall computational workload is reduced. The larger n is, the more significant the reduction in computational workload is.

[0109] The following sections have some examples of skew components and examples of iterative skew calculation algorithms.

[0110] Figure 4-1 The figure shows the definition of skewness. Skewness includes population skewness and sample skewness. Assume that the calculation window X = {x i |i=1,…,n} contains the data elements involved in the skewness calculation, data elements x1, x2, x3, x4,…, x n are the observed data from the predictor variable X and the skewness is assumed to be calculated. Assume that the oldest data element x in the past period r To remove from the computation window X the data element x a To be added to the calculation window X. Whenever a data element is removed from the calculation window and a data element is added to the calculation window, the skewness needs to be recalculated, the calculation window will be considered as the adjusted calculation window X′, and a new round of iterative calculation will begin.

[0111] Equations 401 and 402 are respectively used to calculate the sum S of all data elements of X in the kth round. k and average Equation 403 is the traditional equation for calculating the population skewness of X for the kth round. Equation 404 is the traditional equation for calculating the sample skewness of X for the kth round. Equations 405 and 406 are the sums of all data elements of X′ after adjustment for the k+1th round, respectively.k+1 and average Equation 407 is the traditional equation for calculating the adjusted X for round k+1 ′ Equation 408 is the traditional equation for the population skewness of X for round k+1. ′ The traditional equation for the sample skewness of .

[0112] To show how to calculate skewness using component iterations, three different iterative skewness algorithms are provided as examples.

[0113] Figure 4-2 The first iterative skewness calculation example algorithm (iterative algorithm 1) is described. Equation 401 can be used to initialize the sum S of all data elements in X. k Equation 402 can be used to initialize the average value of all data elements in X Equation 409 can be used to initialize component SS k Equation 410 can be used to initialize component SX k Equation 411 can be used to initialize component CX k Equation 412 can calculate the overall skewness γ based on the initialized components as needed. k (1p) Equation 413 can calculate the sample skewness γ based on the initialized components as needed. k (1s) . Suppose that after a period of time, the data element x r To remove from the computation window X the data element x a To join the calculation window X. Based on the component S k , equation 414 can be used to iteratively calculate the sum S of the adjusted calculation window X′ k+1 Component-based Equation 415 can be used to iteratively calculate the average value of the adjusted calculation window X′ Component-based SS k , equation 416 can be iteratively calculated after the adjustment of the calculation window X ′ component SS k+1 . Component-based SS k+1 , equation 417 can be iteratively calculated after the adjustment of the calculation window X ′ component SX k+1 . Component-based CX k , equation 418 can be iteratively calculated after the adjustment of the calculation window X ′ component CX k+1 Based on the iterative calculation components, equation 419 can be iteratively calculated after adjusting the calculation window X ′ The overall skewness of γ k+1 (1p) , and equation 420 can iteratively calculate the adjusted calculation window X ′ The sample skewness of γ k+1(1s) 414, 415, 417 and 418 each contain multiple equations but only one of each is required depending on whether the sum or the average or both are available.

[0114] Figure 4-3 The second iterative skewness calculation example algorithm (iterative algorithm 2) is described. Equation 401 can be used to initialize the sum S of all data elements in X. k Equation 402 can be used to initialize the average value of all data elements in X Equation 421 can be used to initialize component SS k Equation 422 can be used to initialize component SX k Equation 423 can be used to initialize the component CS k Equation 424 can be used to initialize component CX k Equation 425 can generate the overall skewness γ based on the initialized components as needed. k (1p) , and Equation 426 can generate the sample skewness γ based on the initialized components k (1s) . Suppose that after a period of time, the data element x r To remove from the computation window X the data element x a To join the calculation window X. Based on the component S k , equation 427 can be used to iteratively calculate the adjusted calculation window X ′ S k+1 Component-based Equation 428 can be used to iteratively calculate the adjusted calculation window X ′ of Component-based SS k , equation 429 can be iteratively calculated after adjusting the calculation window X ′ Components of SS k+1 . Component-based SS k+1 , equation 430 can be iteratively calculated after adjusting the calculation window X ′ Components of SX k+1 . Component-based CS k , equation 431 can be iteratively calculated after adjusting the calculation window X ′ Components of CS k+1 . Component-based CS k+1 , equation 432 can be iteratively calculated after adjusting the calculation window X ′ Components of CX k+1 Based on the iterative calculation components, equation 433 can be iteratively calculated after adjusting the calculation window X ′ The overall skewness of γ k+1 (1p), and equation 434 can be iteratively calculated after adjusting the calculation window X ′ The sample skewness of γ k+1 (1s) 427, 428, 430 and 432 each contain multiple equations but only one of each is required depending on whether the sum or the average or both are available.

[0115] Figure 4-4 The third iterative skewness calculation example algorithm (iterative algorithm 3) is described. Equation 401 can be used to initialize the sum of all data elements in X. Equation 402 can be used to initialize the average value of all data elements in X. Equation 435 can be used to initialize component SS k Equation 436 can be used to initialize component SX k Equation 437 can be used to initialize the component CS k Equation 438 can be used to initialize component CX k If desired, Equation 439 can calculate the overall skewness γ based on the initialized components k (1p) , and Equation 440 can calculate the sample skewness γ based on the initialized components k (1s) . Suppose that after a period of time, the data element x r To remove from the computation window X the data element x a To join the calculation window X. Based on the component S k , equation 441 can be used to iteratively calculate the adjusted calculation window X ′ S k+1 Component-based Equation 442 can be used to iteratively calculate the adjusted calculation window X ′ of Component-based SS k , equation 443 can be iteratively calculated after adjusting the calculation window X ′ Components of SS k+1 . Based on component SX k , equation 444 can be iteratively calculated after adjusting the calculation window X ′ Components of SX k+1 . Component-based CS k , equation 445 can be iteratively calculated after adjusting the calculation window X ′ Components of CS k+1 . Component-based CS k+1 , equation 446 can be iteratively calculated after adjusting the calculation window X ′ Components of CX k+1 Based on the iterative calculation components, equation 447 can be iteratively calculated after adjusting the calculation window X ′ The overall skewness of γk+1 (1p) , and equation 448 can be iteratively calculated after adjusting the calculation window X ′ The sample skewness γ k+1 (1s) 441, 442, 444 and 446 each contain multiple equations but only one of each is required depending on whether the sum or the average or both are available.

[0116] To demonstrate the iterative skewness algorithms and how they compare to traditional algorithms, three examples are given below that use data from three calculation windows to calculate sample skewness. The calculation of the overall skewness is similar. For the traditional algorithm, the calculation process for all three calculation windows is exactly the same. For the iterative algorithm, the first calculation window performs the initialization of two or more components, and the second and third calculation windows perform the iterative calculations.

[0117] Figure 5-1 , Figure 5-2 , Figure 5-3 The first calculation window, the second calculation window, and the third calculation window for the calculation example are shown respectively. Calculation window 503 includes the first 4 data elements of data stream 501: 8, 3, 6, 1. Calculation window 504 includes 4 data elements of data stream 501: 3, 6, 1, 9. Calculation window 505 includes 4 data elements of data stream 501: 6, 1, 9, 2. The calculation window size 502 (n) is 4.

[0118] First, the skewness of the calculation windows 503, 504 and 505 are calculated respectively using a traditional algorithm.

[0119] Calculate the skewness for computation window 503:

[0120]

[0121] Without any optimization, computing the skewness for a computation window of size 4 involves 1 square root, 2 divisions, 18 multiplications, 9 additions, and 11 subtractions.

[0122] The same equations and procedures can be used to Figure 5-2 The calculation window 504 displayed calculates the sample skewness and is Figure 5-3 The calculation window 505 shown calculates the sample skewness. The sample skewness of the calculation window 504 This calculation includes 1 square root, 2 divisions, 18 multiplications, 9 additions and 11 subtractions without any optimization. Calculate the sample skewness of window 505 The calculation includes 1 square root, 2 divisions, 18 multiplications, 9 additions and 11 subtractions without optimization. The traditional algorithm usually needs to complete 1 square root, 2 divisions, 3n+6 multiplications, 3(n-1) additions, and 2n+3 subtractions when calculating the sample skewness of the selected data elements in the calculation window of size n without any optimization.

[0123] Next, iterative algorithm 1 is used to calculate the sample skewness of windows 503, 504, and 505 respectively.

[0124] Calculate the sample skewness for a computation window of size 4, 503:

[0125] Initialize the first round with equation 402

[0126]

[0127] Use equations 409, 410, and 411 to initialize the first round SS1, SX1, and CX1 respectively:

[0128]

[0129] Use Equation 413 to calculate the skewness of the first round of samples:

[0130]

[0131] There are 1 square root, 2 divisions, 14 multiplications, 12 additions and 6 subtractions in calculating the sample skewness for calculation window 503.

[0132] Calculate the sample skewness for a computation window of size 4, 504 iterations:

[0133] Equations 415, 416, 417 and 418 are used to iterate the components of the second round. SS2, SX2, CX2:

[0134]

[0135] SS2=SS1+x a 2 -x r 2 =110+81-8 2 =127

[0136]

[0137]

[0138] Use Equation 420 to calculate the second round sample skewness:

[0139]

[0140] There are 1 square root, 2 divisions, 18 multiplications, 9 additions and 7 subtractions in calculating the sample skewness for the calculation window 504 iterations.

[0141] Compute the sample skewness for a computation window of size 4 with 505 iterations:

[0142] Equations 415, 416, 417, and 418 are used to iterate the components of the third round. SS3, SX3, CX3:

[0143]

[0144] SS3=SS2+x a 2 -x r 2 =127+4-3 2 =122

[0145]

[0146]

[0147] Use Equation 420 to calculate the skewness of the third round of samples:

[0148]

[0149] There are 1 square root, 2 divisions, 18 multiplications, 9 additions and 7 subtractions in calculating the sample skewness for the calculation window 505 iterations.

[0150] Next, iterative algorithm 2 is used to calculate the sample skewness of windows 503, 504, and 505 respectively.

[0151] Calculate the sample skewness for a computation window of size 4, 503:

[0152] Initialize the first round with equation 402

[0153]

[0154] Use equations 421, 422, 423, and 424 to initialize the first round SS1, SX1, CS1, and CX1 respectively:

[0155]

[0156]

[0157] Use Equation 426 to calculate the sample skewness for round 1:

[0158]

[0159] There are 1 square root, 2 divisions, 21 multiplications, 15 additions and 6 subtractions in calculating the sample skewness for calculation window 503.

[0160] Calculate the sample skewness for a computation window of size 4, 504 iterations:

[0161] Use equations 428, 429, 430, 431, and 432 to iterate and calculate the components of the second round respectively. SS2, SX2, CS2, CX2:

[0162]

[0163] SS2=SS1+x a 2 -x r 2 =110+81-64=127

[0164]

[0165] CS2=CS1+x a 3 -x r 3 =756+9×9×9-64×8=973

[0166]

[0167] Use Equation 434 to calculate the sample skewness for round 2:

[0168]

[0169] There are 1 square root, 2 divisions, 16 multiplications, 4 additions and 7 subtractions in total when calculating the sample skewness for the calculation window 504 iterations.

[0170] Compute the sample skewness for a computation window of size 4 with 505 iterations:

[0171] Use equations 428, 429, 430, 431, and 432 to iterate and calculate the components of the third round respectively. SS3, SX3, CS3, CX3:

[0172]

[0173] SS3=SS2+x a 2 -x r 2 =127+4-9=122

[0174]

[0175] CS3=CS2+x a 3 -x r 3 =CS2+x a 2 x a -x r 2 x r =973+4×2-9×3=954

[0176]

[0177] Use Equation 434 to calculate the sample skewness for round 3:

[0178]

[0179] There are 1 square root, 2 divisions, 16 multiplications, 4 additions and 7 subtractions in calculating the sample skewness for the calculation window 505 iterations.

[0180] Next, iterative algorithm 3 is used to calculate the sample skewness in calculation windows 503, 504, and 505 respectively.

[0181] Calculate the sample skewness for a computation window of size 4, 503:

[0182] Initialize the first round using equations 402, 435, 436, 437, and 438 respectively SS1, SX1, CS1, CX1:

[0183]

[0184] Use Equation 440 to calculate the sample skewness for round 1:

[0185]

[0186] There are 1 square root, 2 divisions, 21 multiplications, 15 additions and 6 subtractions in calculating the sample skewness for calculation window 503.

[0187] Calculate the skewness for a computation window of size 4 with 504 iterations:

[0188] Use equations 442, 443, 444, 445, and 446 to iterate and calculate the components of the second round respectively. SS2, SX2, CS2, CX2:

[0189]

[0190] SS2=SS1+x a 2 -x r 2 =110+81-64=127

[0191]

[0192] CS2=CS1+x a 3 -x r 3 =CS1+x a 2 x a -x r 2 x r =756+81×9-64×8=973

[0193]

[0194] Use Equation 448 to calculate the sample skewness for round 2:

[0195]

[0196] There are 1 square root, 2 divisions, 15 multiplications, 7 additions and 8 subtractions in calculating the sample skewness for the calculation window 504 iterations.

[0197] Calculate the sample skewness for a computation window of size 4, 505:

[0198] Use equations 442, 443, 444, 445, and 446 to iterate and calculate the components of the third round respectively. SS3, SX3, CS3, CX3:

[0199]

[0200] SS3=SS2+x a 2 -x r 2 =127+4-9=122

[0201]

[0202] CS3=CS2+x a 3 -x r 3 =CS2+x a 2 x a -x r 2 xr =973+4×2-9×3=954

[0203]

[0204] Use Equation 448 to calculate the sample skewness for round 3:

[0205]

[0206] There are 1 square root, 2 divisions, 15 multiplications, 7 additions and 8 subtractions in calculating the sample skewness for the calculation window 505 iterations.

[0207] In the above three examples, the mean is used for iterative skewness calculation. The sum can also be used for iterative skewness calculation, but the operands are different.

[0208] Figure 6-1 The figure shows the comparison of the computational complexity of the traditional skew algorithm and the iterative skew algorithm when n = 4. As shown in the figure, any iterative algorithm has a similar computational complexity compared to the traditional algorithm and has slightly fewer multiplication, addition and subtraction operations.

[0209] Figure 6-2 The figure shows the comparison of the computational complexity of the traditional skew algorithm and the iterative skew algorithm when n=1,000,000. As shown in the figure, any iterative algorithm has much fewer multiplication, addition and subtraction operations than the traditional algorithm. The iterative skew algorithm can process data that may need to be processed on thousands of computers on a single machine. It greatly improves computing efficiency, reduces computing resources, and reduces the energy consumption of computing equipment, making some application scenarios of real-time judgment of asymmetric distribution of streaming data possible from impossible.

[0210] The present invention can be implemented in other specific ways without departing from its ideas or essential features. The implementation scheme described in this application is only exemplary and not restrictive in all aspects. Therefore, the scope of the present invention is indicated by the attached claims rather than the previous description. All changes equivalent to the meaning and scope of the claims in the claims are included in their scope.

Claims

1. A method for calculating the skewness of an adjusted calculation window, characterized in that: A computing system based on one or more computing devices is configured to initialize a sum, or an average value, or a sum and an average value, and one or more other components of skewness different from the sum and the average value, for a pre-adjustment computing window of a specified size n, n>4, of a data stream stored in a buffer of the computing system; A computing system based on a computing device receives a data element to be added to the pre-adjusted computing window; By a computing system based on the computing device, storing the received data element in a buffer; The pre-adjustment calculation window is adjusted by a computing system based on a computing device by: removing the earliest received data element from the pre-adjustment calculation window; as well as Adding the just received data element to the pre-adjustment calculation window; Iteratively calculating a sum, an average, or a sum and an average for the adjusted calculation window by a computing system based on the computing device; A computing system based on a computing device directly iteratively calculates one or more components of the skewness different from the sum and the average value for the adjusted computing window based on at least one or more other components of the skewness different from the sum and the average value of the computing window before adjustment, wherein the iterative calculation of the one or more components includes: Accessing the removed data elements and the added data elements avoids accessing all the data elements in the adjusted calculation window to reduce data access latency, save computing resources and reduce energy consumption; accessing the one or more components of the skewness of the pre-adjustment calculation window that are different from the sum and the mean; and Based on the removed data elements and the added data elements, mathematically removing any contribution of the removed data elements to the component and mathematically adding any contribution of the added data elements to the component from each component accessed to avoid using all data elements in the adjusted computation window to increase computation efficiency; and A skewness is generated for the adjusted calculation window by a computing system based on a computing device based on one or more components that iteratively calculate for the adjusted calculation window.

2. The method according to claim 1, characterized in that: The receiving of a data element to be added includes receiving multiple data elements to be added to the pre-adjustment calculation window. The method also further includes adjusting the pre-adjustment calculation window for each of the multiple data elements to be added, iteratively calculating the sum, or the average value, or the sum and the average value, and directly iteratively calculating one or more components.

3. The method according to claim 1, characterized in that: The receiving of a data element to be added includes receiving multiple data elements to be added to the pre-adjustment calculation window. The method also further includes adjusting the pre-adjustment calculation window for each of the multiple data elements to be added, iteratively calculating the sum, or the average value, or the sum and the average value, directly iteratively calculating one or more components, and generating a skewness for the post-adjustment calculation window.

4. The method according to claim 3, characterized in that: The generating of the skewness for the adjusted calculation window further includes indirectly iteratively calculating one or more components of the skewness for the adjusted calculation window by a computing system based on a computing device, and the indirect iterative calculation of the one or more components includes calculating the one or more components one by one based on one or more components other than the component to be calculated.

5. A computing system, characterized in that: one or more processors; One or more storage media, a buffer in which stream data elements are stored; and One or more computing modules, when executed by at least one of the one or more processors, determine a skewness for an adjusted computing window of a specified size of the buffer, wherein the determination of the skewness includes: a. Initialize a sum, or an average, or a sum and an average, and one or more other components of the skewness other than the sum and the average for a pre-adjustment calculation window of a specified size n for the buffer, where n>4; b. Receive a data element to be added to the pre-adjustment calculation window; c. Save the data element into a buffer; d. Adjust the calculation window before adjustment, including: Remove the earliest added data element from the pre-adjustment calculation window; and adding the received data element to the pre-adjustment calculation window; e. iteratively calculating a sum, an average, or a sum and an average for the adjusted calculation window; f. Directly iteratively calculate one or more components of the skewness other than the sum and the mean for the adjusted calculation window, including: Accessing the removed data elements and the added data elements while avoiding accessing all the data elements in the adjusted calculation window to reduce data access latency, save computing resources and reduce energy consumption; accessing the one or more components of the skewness of the pre-adjustment calculation window that are different from the sum and the mean; and Based on the removed data elements and the added data elements, mathematically removing any contribution of the removed data elements and mathematically adding any contribution of the added data elements from each component accessed to avoid using all data elements in the adjusted calculation window to improve calculation efficiency; and g. generating a skewness for the adjusted calculation window based on one or more components iteratively calculated for the adjusted calculation window.

6. The computing system according to claim 5, characterized in that: The one or more computing modules, when they are executed by at least one of the one or more processors, execute b, c, d, e, and f multiple times.

7. The computing system according to claim 5, characterized in that: The one or more computing modules, when they are executed by at least one of the one or more processors, execute b, c, d, e, f, and g multiple times.

8. The computing system according to claim 7, characterized in that: The g further includes one or more components of the skewness indirectly iteratively calculated by the computing system for the adjusted computing window, and the indirect iterative calculation of the one or more components includes calculating the one or more components one by one based on one or more components other than the component to be calculated.

9. A computing device readable medium comprising a plurality of computing device executable instructions, which, when executed by at least one computing device in a computing system comprising one or more computing devices and one or more storage media, causes the computing system to execute a method for calculating skewness, characterized in that: Initialize a sum, or a mean, or a sum and a mean, and one or more other components of skewness different from the sum and the mean, for a buffer with a pre-adjusted calculation window of size n, n>4; Receive a data element to be added to the pre-adjustment calculation window; Save the received data elements into a buffer; Adjust the pre-adjustment calculation window by: removing the earliest received data element from the pre-adjustment calculation window; as well as Adding the just received data element to the pre-adjustment calculation window; Iteratively calculate a sum, an average, or a sum and an average for the adjusted calculation window; At least based on the one or more other components of the skewness of the calculation window before adjustment that are different from the sum and the average value, directly iteratively calculate the one or more components of the skewness that are different from the sum and the average value for the calculation window after adjustment, where the iterative calculation of the one or more components includes: Accessing the removed data elements and the added data elements to avoid accessing all the data elements in the adjusted calculation window to reduce data access latency, save computing resources and reduce system energy consumption; accessing the one or more other components of the skewness of the pre-adjustment calculation window other than the sum and the mean; and Based on the removed data elements and the added data elements, mathematically removing any contribution of the removed data elements to the component and mathematically adding any contribution of the added data elements to the component from each component accessed to avoid using all the data elements in the adjusted calculation window to increase calculation efficiency; as well as A skewness is generated for the adjusted calculation window based on one or more components iteratively calculated for the adjusted calculation window.

10. A computing device readable medium, characterized in that: The computing device readable medium contains a plurality of computing device executable instructions. When the computing device executable instructions are executed by at least one computing device in a computing system containing one or more computing devices and one or more storage devices, the computing system implements the method as described in any one of claims 1 to 4.

Images