A method for judging the concentration degree of distribution density of big data or stream data in real time

Abstract

Kurtosis can be used to determine the concentration and dispersion of data density distribution. The present application discloses a method, system and computing system program product for calculating kurtosis of big data or stream data in real time by reducing the number of components of the kurtosis of the calculation subset of a given size, so as to determine the concentration of the distribution density of the big data or stream data in real time. The embodiments of the present application include reducing the number of components of the kurtosis of the adjusted calculation subset based on the two or more components of the kurtosis of the calculation subset before adjustment, and then generating the kurtosis of the adjusted calculation subset based on one or more components of the reduced calculation according to the need. The reduced calculation of kurtosis can update the calculation result in real time based on the latest data, avoid accessing all data elements in the adjusted calculation subset and performing repeated calculation, thereby improving the calculation efficiency, saving the computing resources and reducing the energy consumption of the computing system, making the real-time determination of the concentration of the distribution density of the big data or stream data efficient and low-cost, and in some scenarios, making the real-time determination of the concentration of the distribution density of the big data or stream data from impossible to possible.

Description

Technical Field

[0001] Big data or streaming data analysis. Background Technology

[0002] The internet, mobile communications, navigation, online games, sensing technologies, and large-scale computing infrastructure generate massive amounts of data every day. Big data refers to data whose sheer size, rapid changes, and growth rate exceed the processing capabilities of traditional database systems and the analytical capabilities of traditional analytical methods.

[0003] Streaming data is data that is continuously sent by a transmitter and received by a receiver. Streaming data can be real-time data collected by sensors and continuously transmitted to computing or electronic devices. It typically includes data elements with similar formats that are received at time intervals. Streaming data can also be data that is continuously read from memory, such as large datasets stored in the memories of multiple computing devices.

[0004] Streaming data processing algorithms can be extended to big data processing because large datasets accumulate over time and can be considered as data streams with irregular time intervals.

[0005] For processing large datasets or streaming data, it is challenging to efficiently and in real-time update analysis and judgment results, such as judging the concentration of data distribution density, after data updates, because many data elements may be accessed repeatedly.

[0006] Kurtosis reflects the degree of concentration of data density distribution relative to a normal distribution. Therefore, it is obvious to judge the degree of concentration of streaming data distribution density after calculating kurtosis. The difficulty and challenge lies in how to calculate kurtosis in real time on big data or streaming data.

[0007] For example, kurtosis might be calculated after a data element has been removed from a computed subset containing n data elements. When an existing data element is received, that data element is removed from the computed subset. Then, the kurtosis is recalculated by accessing the n-1 data elements in all computed subsets.

[0008] When recalculating kurtosis on all n-1 data elements, all n-1 data elements in the computed subset are accessed and used, and the time complexity of kurtosis calculation is typically O(n). Thus, whenever the computed subset changes, every data element in the computed subset is accessed for kurtosis calculation. Depending on the needs, the size of the computed subset n can be very large, so the data elements may be distributed across a cloud containing hundreds or thousands of computing devices. Recalculating kurtosis on large or streaming data after data changes is time-consuming and wasteful of resources. Summary of the Invention

[0009] Kurtosis can be used to determine the degree of concentration and dispersion of data density distribution. This invention extends to methods, systems, and computing system program products that calculate kurtosis in a decremental manner, thereby enabling real-time determination of the concentration of large or streaming data distribution density after adjusting the size of the computation subset. Decrementing kurtosis for an adjusted computation subset involves two or more components for decrementing the kurtosis of large or streaming data, and then calculating kurtosis based on one or more of the decremented components as needed. Decrementing kurtosis only requires receiving / accessing and using the removed data elements. This avoids accessing all data in the computation subset and also avoids retaining the entire computation subset as in traditional methods. Therefore, both time and space complexity are reduced, making real-time determination of the concentration of large or streaming data distribution density efficient and low-cost, and transforming the impossibility of real-time determination of the concentration of large or streaming data distribution density in some scenarios into a possibility.

[0010] The sum or average is a special component that must be calculated for reduction. Assume that the total number of components (including the sum or average) calculated in the same round of reduction is p (p>1). The number of components directly reduced is v (1≤v≤p), then the number of components indirectly reduced is w=pv (w≥0). The sum or average is a special component that must be calculated for reduction. The sum or average can be calculated for reduction directly or indirectly.

[0011] The computing system initializes a computational subset size counter n (n>1) for a data stream or a dataset stored on one or more storage devices on the computing system, a sum or an average or a sum and an average, and one or more other components of kurtosis besides the sum and the average. The computational subset counter specifies the number of data elements contained in the computational subset before adjustment.

[0012] The computing system can generate the kurtosis of the pre-adjustment computed subset based on the initialized components as needed.

[0013] The computing system accesses an existing big data element or receives a streaming data element to be removed.

[0014] The computing system removes received data elements from an optional data buffer as needed.

[0015] The computing system adjusts the unadjusted computation subset by removing existing data elements that have been accessed or received from the unadjusted computation subset and decrementing the computation subset size counter by 1.

[0016] The calculation system calculates a sum, an average, or a sum and an average of a subset after the reduction calculation adjustment.

[0017] The computational system directly subtracts v (1≤v≤p) components of the adjusted computational subset, excluding the sum and average. Directly subtracting these v components involves: accessing the removed data elements; accessing the v components of the original computational subset's kurtosis; and mathematically removing any contribution from the removed data elements from each accessed component.

[0018] The computing system indirectly reduces the kurtosis of the adjusted subset of computations by w = pv components as needed. Indirectly reducing the computation of w components involves indirectly reducing the computation of each of the w components one by one. Indirectly reducing the computation of a component involves accessing and using one or more other components to compute that component. These one or more components may be initialized, directly reduced, indirectly reduced, or computed in any other way.

[0019] The computing system is based on one or more components that perform subtraction calculations for the adjusted computation subset, and generates kurtosis for the adjusted computation subset. That is, kurtosis can be generated after receiving multiple data elements, not just one data element.

[0020] The computing system can continuously access or receive data elements to be removed, adjust the pre-adjustment computing subset, adjust the counter for the size of the computing subset, reduce the sum, average, or sum and average of the adjusted computing subset, directly reduce v components, indirectly reduce w = pv components and generate kurtosis as needed, and repeat the above process as needed. The adjusted computing subset in the previous round of calculation will become the pre-adjustment computing subset in the next round of calculation.

[0021] This brief description is intended to present some alternative concepts in a simplified manner, which will be described in further detail below. This brief description is not intended to identify key or essential features of the subject matter of the claims, nor is it intended to help determine the scope of the subject matter of the claims.

[0022] Other features and advantages of the invention will become apparent from the description which follows, or in part will be obvious from the description, or may be learned by practice of the invention. The features and advantages of the invention may be realized and obtained from the methods, apparatus, and combinations thereof particularly pointed out in the appended claims. These and other features of the invention will become more fully apparent from the description which follows and the appended claims, or from practice of the invention. Attached Figure Description

[0023] To illustrate how the above and other advantages and features of the present invention can be obtained, a more specific description of the invention briefly described above will be shown by referring to specific embodiments illustrated in the accompanying figures. The invention will be described and explained in more detail and specifically by means of the following figures; it should be understood that these figures only illustrate typical embodiments of the invention and therefore should not be construed as limiting the scope of the invention:

[0024] Figure 1 The diagram illustrates a high-level overview of an example computing system that supports decremental kurtosis calculation.

[0025] Figure 1-1 The diagram illustrates an example of a computing system architecture that calculates kurtosis for streaming data reduction and where all components perform direct reduction calculations.

[0026] Figure 1-2 The diagram illustrates an example of a computing system architecture that calculates kurtosis for stream data reduction, with some components performing reduction calculations directly and others indirectly.

[0027] Figure 1-3 The diagram illustrates an example of a computing system architecture that performs kurtosis calculations for large datasets with all components performing direct kurtosis calculations.

[0028] Figure 1-4 The diagram illustrates an example of a computing system architecture that reduces kurtosis for large datasets, with some components performing direct reduction calculations and others performing indirect reduction calculations.

[0029] Figure 2 The diagram illustrates an example flowchart of a method for calculating kurtosis for big data or streaming data reduction.

[0030] Figure 3-1 The diagram illustrates the data elements accessed in the kurtosis calculation window for decrementing streaming data.

[0031] Figure 3-2 The diagram illustrates the data elements accessed in the subset of kurtosis computation for reducing kurtosis in big data.

[0032] Figure 4-1 The diagram illustrates the definition of kurtosis and the conventional equation for calculating kurtosis on a subset of data.

[0033] Figure 4-2 The diagram illustrates the traditional equation for calculating kurtosis on the adjusted subset.

[0034] Figure 4-3 The first kurtosis reduction calculation algorithm (reduction algorithm 1) is shown.

[0035] Figure 4-4 The second kurtosis reduction calculation algorithm (reduction algorithm 2) is shown.

[0036] Figure 4-5 The third kurtosis reduction calculation algorithm (reduction algorithm 3) is shown.

[0037] Figure 5-1 The first subset of computations used for a computation instance is displayed.

[0038] Figure 5-2 The second subset of computations used for a computation instance is shown.

[0039] Figure 5-3 The third subset of computations used for a computation instance is shown.

[0040] Figure 6-1 The diagram illustrates a comparison of the computational workload between the traditional kurtosis algorithm and the decremented kurtosis algorithm when the subset size is 4.

[0041] Figure 6-2 The diagram illustrates a comparison of the computational workload between the traditional kurtosis algorithm and the decremented kurtosis algorithm when the subset size is 1,000,000. Specific implementation methods

[0042] Kurtosis can be used to determine the degree of concentration and dispersion of data density distribution. This invention extends to methods, systems, and computing system program products that calculate kurtosis in a decremental manner, thereby enabling real-time determination of the concentration of large or streaming data distribution density after adjusting the size of the computation subset. Decrementing kurtosis for an adjusted computation subset involves decrementing the kurtosis of large or streaming data using one or more components, and then calculating kurtosis based on one or more decremented components as needed. Decrementing kurtosis only requires receiving / accessing and using the removed data elements. This avoids accessing all data in the computation subset and also avoids retaining the entire computation subset as in traditional methods. Therefore, both time and space complexity are reduced, making real-time determination of the concentration of large or streaming data distribution density efficient and low-cost, and transforming the impossibility of real-time determination of the concentration of large or streaming data distribution density in some scenarios into a possibility.

[0043] A computation subset contains the data elements needed to calculate kurtosis. This data can originate from a large dataset stored on one or more storage devices or from one or more data streams. A computation subset is equivalent to a moving computation window when calculating kurtosis on streaming data or time series data. In this paper, the difference is that the data in a computation window is ordered, while the data in a computation subset can be unordered.

[0044] A component of kurtosis is a quantity or expression that appears in the defining equation of kurtosis or any transformation thereof. For example, calculating the mean of all data elements on a subset is a component of kurtosis, as is calculating the sum of squares of all data elements on a subset. The largest component of kurtosis is kurtosis itself. Below are some examples of components of kurtosis.

[0045] ·

[0046] ·

[0047] ·

[0048] ·

[0049] v

[0050] ·

[0051] ·

[0052] ·

[0053] ·

[0054] ·

[0055] ·

[0056] ·

[0057] v

[0058] Kurtosis can be calculated based on one or more components or a combination thereof, so multiple algorithms support decremented kurtosis calculation.

[0059] Kurtosis can be calculated as needed, i.e., only when it is accessed. When the kurtosis of a subset is not accessed for each data change, the computing system can calculate more than one component only for each data change. Kurtosis can be calculated and returned based on more than one component when accessed. As needed, the computing system can calculate the kurtosis of a subset using more than one component.

[0060] A component can be calculated by direct decrement or by indirect decrement. The difference is that when a component is calculated by direct decrement, it is calculated using the value of the component in the previous round, while when a component is calculated by indirect decrement, it is calculated using other components besides itself.

[0061] For a given component, it may be directly decremented in one algorithm and indirectly decremented in another algorithm.

[0062] For a given algorithm, assume that the total number of different components is p (p≥3), and the total number of components directly decremented is v (1≤v≤p). Then the total number of components indirectly decremented is w = p - v (0≤w<p). In a better algorithm, at least one component will be directly decremented. It is possible that all components are directly decremented (in which case v = p and w = 0). However, the components directly decremented must be calculated every round regardless of whether the kurtosis is accessed every round.

[0063] For a given algorithm, if a component is directly decremented, then that component needs to be calculated every round (i.e., whenever an existing data element is removed from the pre-adjustment calculation subset). However, if that component is indirectly decremented, then that component can be calculated as needed (i.e., only when the kurtosis needs to be calculated and accessed). Therefore, when the kurtosis is not accessed in a particular round and some components are indirectly decremented, only a small number of components can be directly decremented to save time. This will be useful when the kurtosis needs to be calculated after multiple data elements are removed from the calculation subset. The implementation of the present invention can be easily extended to handle the above situation by adding a loop to process the removed data elements one by one, directly decrementing more than one component inside the loop, and calculating the kurtosis after the loop or after indirectly decrementing more than one component or directly calculating the kurtosis.

[0064] The embodiments of the present invention include components that decrement calculate more than two kurtoses on the post-adjustment calculation subset based on more than two calculated components on the pre-adjustment calculation subset. Among them, the sum or average of the calculation subset is a special component that must be decremented calculated. The sum or average can be directly or indirectly decremented calculated. For a given kurtosis decrement algorithm, assume that the total number of all components decremented calculated (including the sum or average of the calculation subset) in the same round of decrement calculation is p (p≥3). The number of components directly decremented is v (1≤v≤p), then the number of components indirectly decremented is w = p - v (w≥0).

[0065] When the kurtosis decrement calculation starts from a non-empty pre-adjustment calculation subset, the scale counter of the pre-adjustment calculation subset is initially set to the number of data elements in the pre-adjustment calculation subset, and the initialization of more than one component includes calculating or setting to the results that have been calculated in various different ways based on the data elements in the pre-adjustment calculation subset through their definitions. Therefore, the calculation of the components in the previous round of decrement calculation can also be considered as the initialization of the components in the current round of calculation.

[0066] The computing system initializes a computation subset size counter n (n>1) for a pre-adjustment computation subset of at least one dataset or at least one data stream stored on one or more storage devices on the computing system, initializes a sum and or an average value or a sum and an average value for the pre-adjustment computation subset, and one or more other components of kurtosis other than the sum and the average value, the computation subset size counter specifying the number of data elements contained in the pre-adjustment computation subset.

[0067] The computing system accesses data elements of at least one dataset stored on one or more storage devices or receives a stream data element to be removed from at least one data stream.

[0068] For streaming data processing, the computing system removes received data elements from an optional data buffer as needed. Simple debulking kurtosis computation only needs to access the recently received data elements but not previously received ones; therefore, the computing system does not need to allocate storage space to retain all data elements of the computation subset, but will retain at least two components: the subset size counter and the subset's kurtosis. Adjusting the computation subset can be achieved by adjusting two or more components of the subset size counter and the subset's kurtosis. However, because iterative kurtosis computation requires access to previously received data elements, when combining debulking and iterative kurtosis computation, the received data elements need to be stored in a data buffer used by the iterative kurtosis computation. Therefore, for streaming data, the computing system may selectively require a data buffer to store the received data elements.

[0069] The computing system adjusts the unadjusted computation subset by removing accessed or received data elements from the unadjusted computation subset and decrementing the computation subset size counter by 1 accordingly.

[0070] The calculation system calculates a sum, an average, or a sum and an average of a subset after the reduction calculation adjustment.

[0071] The computational system reduces the kurtosis of the adjusted subset of computation by reusing v (1 ≤ v ≤ p) components from the original subset of computation. It then directly reduces the kurtosis of v components from the adjusted subset. Directly reducing the kurtosis of v components involves calculating each of the v components one by one. Directly reducing a component involves accessing data elements removed from the original subset of computation and the component calculated for the original subset of computation, mathematically removing any contribution of the accessed data elements to the component.

[0072] The computing system indirectly reduces the kurtosis of w = pv components of the adjusted computation subset as needed. Indirectly reducing the kurtosis of the w components involves sequentially reducing the kurtosis of each of the w components. Indirectly reducing the kurtosis of a component involves accessing and utilizing two or more components other than the component itself. These two or more components can be computed in various ways, such as being initialized, directly reduced, or indirectly reduced.

[0073] The computing system generates kurtosis based on one or more subtractive calculated components as needed.

[0074] The computing system can continuously access or receive data elements to be removed, adjust the pre-adjustment computation subset, reduce the sum, average, or sum and average of the adjusted computation subset, reduce v components, indirectly reduce w = pv components as needed, generate kurtosis based on one or more components that have been reduced as needed, and repeat the above process as needed. The adjusted computation subset in the previous round of calculation will become the pre-adjustment computation subset in the next round of calculation.

[0075] Embodiments of the present invention may include or utilize computing device hardware, such as one or more processors and storage devices as described in more detail below, whether dedicated or general-purpose computing devices. The scope of embodiments of the present invention also includes physical and other computing device-readable media used to carry or store computing device-executable instructions and / or data structures. These computing device-readable media may be any media accessible to general-purpose or dedicated computing devices. Computing device-readable media storing computing device-executable instructions is storage media (device). Computing device-readable media carrying computing device-executable instructions is transmission media. Therefore, by way of example only and not limitation, embodiments of the present invention may include at least two different types of computing device-readable media: storage media (devices) and transmission media.

[0076] Computing device storage media (devices) include random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), read-only optical disc storage (CD-ROM), solid-state drive (SSD), flash memory, phase-change memory (PCM), other types of memory, other optical disk storage, disk storage or other magnetic storage devices, or any other media that can be used to store program code in the form of computer device-executable instructions or data structures and that can be accessed by general-purpose or special-purpose computing devices.

[0077] A “network” is defined as one or more data links that enable computing devices and / or modules and other electronic devices to transmit electronic data. When information is transmitted or provided to a computing device via a network or other communication connection (wired, wireless, or a combination of wired and wireless), the computing device treats the connection as a transmission medium. The transmission medium may include a network and / or data link for carrying necessary program code in the form of instructions or data structures executable by the computing device, and which can be accessed by general-purpose or special-purpose computing devices. Combinations of the above should also be included within the scope of media readable by the computing device.

[0078] Furthermore, when using different computing device components, program code in the form of executable instructions or data structures can be automatically transferred from the transmission medium to the storage medium (device) (or vice versa). For example, executable instructions or data structures received from a network or data link can be temporarily stored in the random access memory (RAM) of a network interface module (e.g., NIC) and then eventually transferred to the RAM of the computing device and / or to a smaller, more volatile storage medium (device) of the computing device. Therefore, it should be understood that the storage medium (device) can be included in computing device components that also (or even primarily) utilize the transmission medium.

[0079] Computer device executable instructions include, for example, instructions and data, which, when executed by a processor, cause a general-purpose or special-purpose computing device to perform a specific function or set of functions. Computer device executable instructions can be, for example, binary, intermediate format instructions such as assembly code, or even source code. Although the described subject matter is described in a specific language of structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or actions described above. Rather, the described features or actions are disclosed only as examples of implementing the claims.

[0080] The embodiments of this invention can be implemented in a network computing environment configured with various types of computing devices, including personal computers, desktops, laptops, information processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network computers, minicomputers, mainframe computers, supercomputers, mobile phones, PDAs, tablets, pagers, routers, switches, and similar products. The embodiments of this invention can also be applied to a distributed system environment consisting of local or remote computing devices performing tasks via network interconnection (i.e., via wired data links, wireless data links, or a combination of wired and wireless data links). In a distributed system environment, program modules can be stored on local or remote storage devices.

[0081] Embodiments of the present invention can also be implemented in a cloud computing environment. In this description and the following claims, "cloud computing" is defined as a model that enables on-demand access to a shared pool of configurable computing resources over a network. For example, cloud computing can be marketed to provide widespread and convenient on-demand access to a shared pool of configurable computing resources. The shared pool of configurable computing resources can be quickly prepared via virtualization and provided with low management overhead or low service provider interaction, and then adjusted accordingly.

[0082] Cloud computing models can include various features such as on-demand self-service, broadband network access, resource collection, rapid deployment and deployment, metered services, and so on. Cloud computing models can also be embodied in various service models, such as Software as a Service (“SaaS”), Platform as a Service (“PaaS”), and Infrastructure as a Service (“IaaS”). Cloud computing models can also be deployed through different deployment models such as private cloud, community cloud, public cloud, hybrid cloud, and so on.

[0083] The embodiments of the present invention are also applicable to edge computing. Since the embodiments of the present invention do not require access to all data in the adjusted computation subset and have a small computational load, real-time computation can be performed at the data-collecting sensor using a low-end microprocessor without transmitting the data to the cloud for computation.

[0084] Figure 1 The diagram illustrates a high-level overview of a computing system 100 used for calculating kurtosis to reduce large or streaming data volume. (Reference) Figure 1 The computing system 100 includes multiple devices connected by different networks, such as a local area network 1021, a wireless network 1022, and the Internet 1023. These multiple devices include, for example, a data analysis engine 1007, a storage system 1011, a real-time data stream 1006, and multiple distributed computing devices, such as personal computers 1016, handheld devices 1017, and desktop computers 1018, capable of scheduling data analysis tasks and querying data analysis results.

[0085] The data analysis engine 1007 may include one or more processors, such as CPU 1009 and CPU 1010, one or more system memories, such as system memory 1008, and component computing module 131 and kurtosis computing module 191. Details of module 131 will be illustrated in more detail in other diagrams (e.g., Figure 1-1 and Figure 1-2 Storage system 1011 may include one or more storage media, such as storage media 1012 and storage media 1014, which can be used to store large datasets. For example, 1012 and / or 1014 may include dataset 123. The dataset in storage system 1011 can be accessed by data analysis engine 1007.

[0086] Typically, data stream 1006 can include streaming data from various data sources, such as stock prices, audio data, video data, geospatial data, internet data, mobile communication data, online game data, bank transaction data, sensor data, and / or closed caption data. Several examples are described here; real-time data 1000 can include data collected in real time from sensors 1001, stock data 1002, communication data 1003, and banking data 1004, etc. Data analysis engine 1007 can receive data elements from data stream 1006. Data from different data sources can be stored in storage system 1011 and accessed for big data analytics; for example, dataset 123 can come from different data sources and be accessed for big data analytics.

[0087] Please understand. Figure 1 The concepts are presented in a very simplified form. For example, distributed devices 1016 and 1017 may connect to the data analysis engine 1007 through a firewall, and the data accessed or received by the data analysis engine 1007 from the data stream 1006 and / or storage system 1011 may be filtered by a data filter, and so on.

[0088] Figure 1-1 The diagram illustrates a computational system architecture 100A for calculating kurtosis in streaming data reduction, where all components (v = p ≥ 1) are directly reduced in the calculation. Regarding computational system architecture 100A, this section will initially only introduce the functions and interrelationships of the main components. The process of how these components collaborate to complete the kurtosis reduction calculation will be discussed later. Figure 2 The process described in Figure 1 Let's begin the introduction. Figure 1-1 The diagram shows... Figure 1 Displayed numbers 1006 and 1007. (Reference) Figure 1-1The computing system architecture 100A includes a component computing module 131 and a kurtosis computing module 191. The component computing module 131 may be tightly coupled to one or more storage media via a high-speed data bus or loosely coupled to one or more storage media managed by the storage system via a network, such as a local area network, a wide area network, or even the Internet. Accordingly, the component computing module 131 and any other connected computing devices and their components can send and receive message-related data (e.g., Internet Protocol (“IP”) datagrams and other higher-level protocols using IP datagrams, such as User Datagram Protocol (“UDP”), Real-Time Streaming Protocol (“RTSP”), Real-Time Transport Protocol (“RTP”), Microsoft Media Server (“MMS”), Transmission Control Protocol (“TCP”), Hypertext Transfer Protocol (“HTTP”), Simple Mail Transfer Protocol (“SMTP”), etc.) on the network. The output of the component computing module 131 is used as input to the kurtosis computing module 191, which can generate kurtosis 192.

[0089] Typically, data stream 190 can be a sequence of digitally encoded signals (i.e., data packets or data packets) used to transmit or receive information during transmission. Data stream 190 can contain data from various sources, such as stock prices, audio data, video data, geospatial data, internet data, mobile communication data, online game data, bank transaction data, sensor data, closed caption data, and real-time text. Data stream 190 can be a real-time stream or streamed stored data.

[0090] Reduced kurtosis calculations begin with a non-empty subset of computations whose size and one or more components of kurtosis have been initialized or computed. If not, the kurtosis of one or more components can be computed one by one according to the definition by accessing all data in the subset of computations and utilizing that data. For example, in computational system architecture 100A, components Cd1141 to Cd... v 145 may have already been initialized. For example... Figure 1-1 As shown, component Cd1 141 includes contribution 151 (contribution of data element 101), contribution 152 (contribution of data element 102), and contribution 153 (contribution of other data elements). Similarly, component Cd... v 145 includes contribution 181 (the contribution of data element 101), contribution 182 (the contribution of data element 102), and contribution 183 (the contribution of other data elements).

[0091] The subset size counter 118 can be implemented in hardware or software. When implemented in hardware, counter 118 can be a small device that updates the subset size stored in counter 118. When implemented in software, counter 118 can be a variable stored in RAM, a hard disk, or any other computing device-readable storage medium, and the content of counter 118 is the subset size. The subset size counter 118 can be used to record the size of the subset. Whenever a data element is received, the computing device adjusts the subset by removing the received data element from the unadjusted subset and by decrementing the content or value of the subset size counter 118 by 1. The subset size counter 118 can be accessed or received by the component computing module 131 and can be set to a specific value by the reset module 129 when the decrementing kurtosis calculation begins operating from a non-empty subset.

[0092] For example, when data element 101 is received, the compute subset size counter 118 is decremented by 1. Both the compute subset size counter 118 and data element 101 are accessed by the component compute module 131. Data element 101 must be a data element in a compute subset.

[0093] Subsequently, data element 102 can be received. Upon receipt, the compute subset size counter 118 is decremented by 1. Both the adjusted compute subset size counter 118 and data element 102 can be accessed by the component compute module 131. Again, data element 102 must be a data element for which a compute subset exists.

[0094] refer to Figure 1-1 The computing system architecture 100A also includes an optional data buffer 125. The optional data buffer 125 is useful when combining debulking kurtosis calculation and iterative kurtosis calculation. As streaming data elements are received, they can be removed from the optional data buffer 125 as needed (i.e., when combining debulking kurtosis calculation and iterative kurtosis calculation).

[0095] Referring to the reference computing system architecture 100A, the reduction component computing module 131 contains v (v = p ≥ 1) component computing modules that calculate the kurtosis of a subset of data. The number v varies depending on the reduction algorithm used. For example... Figure 1-1 As shown, component Cd1 calculation module block 131 includes component calculation module 161 and component Cd. v Calculation module 162 has v-2 other component calculation modules between it. Calculation module 161 includes initialization module 132 for initializing component Cd1 and decrement algorithm 133 for directly decrementing component Cd1. Component Cd... v The calculation module 162 includes an initialization module 138 for initializing component Cd. v And reduction algorithm 139 for direct reduction calculation component Cdv Initialization module 132 is configured as computation subset initialization component Cd1, and initialization module 138 is configured as computation subset initialization component Cd1. v 141 is the initial value of component Cd1. Initialization module 132 can be used to initialize component Cd1 or when the kurtosis calculation is reset. Initialization module 132 initializes component 141 to a specific value composed of contributions 151, 152, and 153 passed in by reset module 129. Similarly, initialization module 138 can be used to initialize component Cd. v Or when the kurtosis calculation is reset. Initialization module 138 initializes component 145 with a specific value composed of contributions 181, 182 and 183 passed in by reset module 129.

[0096] The reduction algorithm is also configured to directly reduce the computation of v components from the computation subset. 133 receives the previous component Cd1 value and a data element to be removed from the computation subset as input. 133 uses the previous component Cd1 value and the removed data element to directly reduce the computation of a new component Cd1. The contribution removal module 133A can remove any contribution from the removed data element from the previous component Cd1. Removing any contribution from the removed data element can directly reduce the computation of component Cd1 from the adjusted computation subset. 139 operates similarly to 133. 139 receives the previous component Cd1 value... v The input consists of a value and a data element to be removed from the subset of computation. 139 uses the previous component Cd. v The value and the removed data elements are used to directly calculate a new component Cd. v The contribution removal module 139A can be derived from the previous component Cd. v Remove any contribution from the removed data elements. Removing any contribution from the removed data elements can directly reduce the computation component Cd in the adjusted computation subset. v .

[0097] refer to Figure 1-1 The computing system architecture 100A also includes a kurtosis calculation module 191. After p (p ≥ 1) kurtosis components are subtracted from the calculation by the component calculation module 131, 191 can generate kurtosis 192 based on one or more subtracted or initialized components. Kurtosis 192 can be sample kurtosis, overall kurtosis, sample excess kurtosis, and / or excess kurtosis.

[0098] Figure 1-2FIG. illustrates an example of a computing system architecture 100B for computing kurtosis for streaming data reduction, where some components (v (1≤v<p)) are directly reduced, and some components (w = p - v) are indirectly reduced. The numbers v and w depend on the algorithm of a specific application. Many components of computing system architectures 100B and 100A have the same reference numerals. Those components have the same architecture and work in the same way. The difference between computing system architecture 100B and computing system architecture 100A is that architecture 100B includes a component computing module 135. Except for 135 in 100B, the components with the same reference numerals as in 100A work in the same way. To avoid repeating what has been explained in the description of 100A before, only the different parts will be discussed here. The number v in 100B and the number v in 100A may be different because some components that are directly reduced in 100A are indirectly reduced in 100B. In 100A, v = p≥1, but in 100B, 1≤v<p. Refer to Figure 1-2 , computing system architecture 100B includes a component computing module 135. 135 generally includes w = p - v component computing modules for indirectly reducing w components. For example, 135 includes a component Ci1 computing module 163 for indirectly reducing component Ci1 and a component Ci w computing module 164 for indirectly reducing component Ci w , and there are w - 2 component computing modules between them. Indirectly reducing w components includes indirectly reducing each of the w components one by one. Indirectly reducing a component includes accessing and using more than one component other than itself. The more than one component may have been initialized, directly reduced, indirectly reduced, or computed in any other way.

[0099] Refer to computing system architecture 100B. After all p (p = v + w) components have been reduced, the kurtosis computing module 191 can generate kurtosis 192 using one or more reduced or initialized components as needed. Kurtosis 192 can be sample kurtosis, population kurtosis, sample excess kurtosis, and / or excess kurtosis.

[0100] Figure 1-3 FIG. illustrates an example of a computing system architecture 100C for reducing and computing kurtosis for a large data set, where all components are directly reduced. Regarding computing system architecture 100C, only the functions and interrelationships of the main components in this architecture will be introduced here first, and the process of how these components cooperate to jointly complete the reduction and kurtosis computing will be introduced later in combination with Figure 2 the process described in Figure 1 will be introduced. <​​​​The computing system architecture 100C includes a component computing module 131 and a kurtosis computing module 191. The component computing module 131 may be tightly coupled to one or more storage media via a high-speed data bus or loosely coupled to one or more storage media managed by the storage system via a network, such as a local area network, a wide area network, or even the Internet. Accordingly, the component computing module 131 and any other connected computing devices and their components can send and receive message-related data (e.g., Internet Protocol (“IP”) datagrams and other higher-level protocols using IP datagrams, such as User Datagram Protocol (“UDP”), Real-Time Streaming Protocol (“RTSP”), Real-Time Transport Protocol (“RTP”), Microsoft Media Server (“MMS”), Transmission Control Protocol (“TCP”), Hypertext Transfer Protocol (“HTTP”), Simple Mail Transfer Protocol (“SMTP”), etc.) on the network. The output of the component computing module 131 is used as input to the kurtosis computing module 191, which can generate kurtosis 192. Kurtosis 192 can be sample kurtosis, population kurtosis, sample overkurtosis, and / or overkurtosis.

[0101] Storage medium 121 includes dataset 123. As shown, dataset 123 includes multiple data elements located in multiple locations on storage medium 121. For example, data elements 101, 102, 103, 104, 105, and 106 are stored in locations 121A, 121B, 121C, 121D, 121E, and 121F, respectively, and multiple other data elements are located in other locations on storage medium 121.

[0102] Assume that the kurtosis calculation begins with computation subset 122. Computation subset 122, with a size of 6 (i.e., n = 6), includes data elements 101, 102, 103, 104, 105, and 106 from dataset 123. Data elements 101 through 106 in computation subset 122 can all be accessed for one or more components to initialize kurtosis.

[0103] Then, data element 101 can be accessed from position 121A. Data element 101 is removed from computation subset 122, and computation subset 122 becomes the adjusted computation subset 122A. The adjusted computation subset 122A includes five data elements: 102, 103, 104, 105, and 106. The computation system can calculate kurtosis by subtracting from the adjusted computation subset 122A.

[0104] Next, data element 102 can be accessed from position 121B. Data element 102 is removed from computation subset 122, and the adjusted computation subset 122A becomes the adjusted computation subset 122B. The adjusted computation subset 122B includes four data elements: 103, 104, 105, and 106. The computation system can calculate kurtosis by subtracting the selected data element 192 from the adjusted computation subset 122B.

[0105] Referring to the computing system architecture 100C, the reduction component computing module 131 contains v (v = p ≥ 1) component computing modules that calculate the kurtosis of a subset of data. The number v varies depending on the reduction algorithm used. For example... Figure 1-3 As shown, component Cd1 calculation module block 131 includes component calculation module 161 and component Cd. v Calculation module 162 has v-2 other component calculation modules between it. Calculation module 161 includes initialization module 132 for initializing component Cd1 and decrement algorithm 133 for directly decrementing component Cd1. Component Cd... v The calculation module 162 includes an initialization module 138 for initializing component Cd. v And reduction algorithm 139 for direct reduction calculation component Cd v Initialization module 132 is configured as computation subset initialization component Cd1, and initialization module 138 is configured as computation subset initialization component Cd1. v 141 is the initial value of component Cd1. Initialization module 132 can be used to initialize component Cd1 or when the kurtosis calculation is reset. Initialization module 132 initializes component 141 with a specific value consisting of contributions 151, 152, and 153 passed in by reset module 129. Similarly, initialization module 138 can be used to initialize component Cd. v Or when the kurtosis calculation is reset. Component 145 is component Cd. v The initial value. The initialization module 138 initializes the component 145 with the specific value consisting of contribution 181, contribution 182 and contribution 183 passed in by the reset module 129.

[0106] The reduction algorithm is also configured to directly reduce the computation of v components from the computation subset. 133 receives the previous component Cd1 value and a data element to be removed from the computation subset as input. 133 uses the previous component Cd1 value and the removed data element to directly reduce the computation of a new component Cd1. The contribution removal module 133A can remove any contribution from the removed data element from the previous component Cd1. Removing any contribution from the removed data element can compute component Cd1 for the adjusted computation subset. 139 operates similarly to 133. 139 receives the previous component Cd1 value... vThe input is a value and a data element to be removed from the computational subset. 139 uses the previous component Cd v The value and the removed data element are used to directly decrement and calculate a new component Cd v The contribution removal module 139A can remove any contribution of the removed data element from the previous component Cd v Removing any contribution of the removed data element can calculate the component Cd for the adjusted computational subset v .

[0107] Reference Figure 1-3 , the computing system architecture 100C also includes a kurtosis calculation module 191. After p (p≥1) kurtosis components are decremented and calculated by the component calculation module 131, 191 can generate the kurtosis 192 based on one or more decremented or initialized components. The kurtosis 192 can be sample kurtosis, population kurtosis, sample excess kurtosis, and / or excess kurtosis

[0108] Figure 1-4 Illustrates an example of a computing system architecture 100D for decrementing and calculating the kurtosis of big data, where part (v (1≤v<p)) of the components are directly decremented and calculated, and part (w = p - v) of the components are indirectly decremented and calculated. The numbers v and w depend on the algorithm of the specific application. Many components of the computing system architectures 100D and 100C have the same reference numbers. Those components have the same architecture and work in the same way. The difference between the computing system architecture 100D and the computing system architecture 100C is that the architecture 100D includes a component calculation module 135. Except for 135 in 100D, the components with the same reference numbers as in 100C work in the same way. To avoid repeating what has been explained in the description of 100C before, only the different parts will be discussed here. The number v in 100D and the number v in 100C may be different because some components that are directly decremented and calculated in 100C will be indirectly decremented and calculated in 100D. In 100C, v = p≥1, but in 100D, 1≤v<p. Reference Figure 1-4 , the computing system architecture 100D includes a component calculation module 135. 135 generally includes w = p - v component calculation modules for indirectly decrementing and calculating w components. For example, 135 includes a component Ci1 calculation module 163 for indirectly decrementing and calculating the component Ci1 and a component Ci w calculation module 164 for indirectly decrementing and calculating the component Ci w , and there are w - 2 component calculation modules between them. Indirectly decrementing and calculating w components includes indirectly decrementing and calculating each of the w components one by one. Indirectly decrementing and calculating a component includes accessing and using more than one component other than itself. The more than one component can be initialized, directly decremented, indirectly decremented, or calculated in any other way

[0109] Referring to the computing system architecture 100D, after all p (p = v + w) components have been subtracted from the calculation, the kurtosis calculation module 191 can generate kurtosis 192 using one or more subtracted or initialized components as needed. Kurtosis 192 can be sample kurtosis, overall kurtosis, sample overkurtosis, and / or overkurtosis.

[0110] Figure 2 The diagram illustrates a flowchart example of a method 200 for calculating kurtosis to reduce large or streaming data volume. Method 200 will be described separately along with the components and data for computing system architectures 100A, 100B, 100C, and 100D.

[0111] Method 200 includes initializing the value n (n>1) of the computation subset size counter 118 and the kurtosis v (1≤v≤p, p≥2) of a pre-adjusted computation subset of large or streaming data for components (201). For example, for computation system architectures 100A, 100B, 100C, and 100D, the reset module 129 can be component Cd1 accessible by the initialization module 132 and component Cd2 accessible by the initialization module 138. v All components are initialized, and corresponding initial values ​​are set for the subset size counter 118. 132 can access the initial values ​​of component Cd1 and initialize component Cd1141 with the values ​​that become contributions 151, 152, and 153. Similarly, 138 can access the initial values ​​of component Cd1141. v The initial values ​​are then used to initialize component Cd1145, which is then initialized with the values ​​181, 182, and 183.

[0112] Method 200 allows access to or receipt of a data element (202) to be removed from the unadjusted computational subset. For example, data element 101 can be accessed or received.

[0113] Method 200 includes removing the data element from an optional data buffer as needed (203). For example, when combining decremental kurtosis calculation and iterative kurtosis calculation, data element 101 can be removed from the optional data buffer 125 as needed.

[0114] Method 200 includes removing the data element from the unadjusted computed subset and adjusting the computed subset size counter (204). For example, data element 101 can be removed from the computed subset and the computed subset size counter 118 can be adjusted by decrementing the current value by 1 when data element 101 is received.

[0115] Method 200 includes directly subtracting v components of the kurtosis of the adjusted computed subset by reusing v components of the pre-adjustment computed subset and data elements removed from the pre-adjustment computed subset (205). For example, algorithm 133 can directly subtract Cd1143 using component Cd1141 and data element 101, and algorithm 139 can reuse component Cd1141 to calculate Cd1143. v 145 and data element 101 are directly subtracted to calculate Cd. v 147.

[0116] The v components of the direct reduction calculation of the kurtosis of the adjusted subset include access to data elements removed from the unadjusted subset (206). For example, for computing system architectures 100A, 100B, 100C, and 100D, reduction algorithm 133 can access data element 101, and reduction algorithm 139 can also access data element 101.

[0117] The v components that directly reduce the kurtosis of the adjusted subset include the v components that access the kurtosis of the unadjusted subset (207). For example, for computing system architectures 100A, 100B, 100C, and 100D, reduction algorithm 133 can access component Cd1141, and reduction algorithm 139 can access component Cd1141. v 145.

[0118] The direct debulking calculation of the next kurtosis of the v components includes mathematically removing any contribution from the data elements removed from the pre-adjusted computation subset from each accessed component (208). For example, for computation system architecture 100A, the debulking calculation component Cd1143 may include a contribution removal module 133A that mathematically removes contribution 151 from component Cd1141. v 147 may include a contribution removal module 139A mathematically from component Cd v Remove contribution 181 from 145. Both contributions 151 and 181 are contributions from data element 101.

[0119] like Figure 1-1 As shown, component Cd1143 includes contribution 152 (the contribution of data element 102) and contribution 153 (the contribution of other data elements). Similarly, component Cd... v 147 includes contribution 182 (the contribution of data element 102) and contribution 183 (the contribution of other data elements).

[0120] In addition to the direct decrement calculation components, some components can be indirectly decremented. When v < p (i.e., not all components are directly decremented), method 200 includes indirectly decrementing w = p - v components one by one as needed based on one or more other components in addition to the components themselves (209). Since the w indirectly decremented components are calculated using other components, they do not need to be initialized. In addition, those w indirectly decremented components do not need to be calculated in each round (i.e., when existing data elements are removed from the calculation subset). These w components only need to be calculated when the kurtosis is accessed. For example, referring to Figure 1-2 and Figure 1-4 , some components are directly decremented and some are indirectly decremented. The component Ci1 calculation module 163 can indirectly decrement Ci1 based on one or more components other than Ci1, and the component Ci w calculation module 164 can indirectly decrement Ci w based on one or more components other than Ci w . The one or more components can be initialized, directly decremented, or indirectly decremented. For a specific component, it can be directly decremented by one algorithm or indirectly decremented by another algorithm. For a specific algorithm, the directly decremented components must be calculated whenever a data element is removed from the data set, regardless of whether the kurtosis is accessed; however, the indirectly decremented components can be calculated as needed (i.e., only when the kurtosis is accessed).

[0121] Method 200 includes generating kurtosis as needed based on one or more initialized or decremented components (210). For example, the kurtosis calculation module 191 can generate kurtosis 192 based on one or more decremented components (e.g., component Cd1143 and component Cd v 147). Kurtosis 192 can be sample kurtosis, population kurtosis, sample excess kurtosis, and / or excess kurtosis. <0000�41>202 - 208 can be repeated as more data elements are accessed or received. 209 - 210 can be repeated as needed. For example, after calculating components Cd1143 to component Cd v 147, data element 102 can be accessed or received.

[0123] The decrement algorithm 133 can directly decrement component Cd1144 using component Cd1143. The decrement algorithm 133 can access data element 102 and component Cd1143. Directly decrementing component Cd1144 can include the contribution removal module 133A mathematically removing contribution 152 from component Cd1143. Contribution 15! is the contribution of data element 102. Similarly, the decrement algorithm 139 can use component Cd v147 Direct reduction calculation component Cd v 148. The decrement algorithm 139 can access data element 102 and component Cd. v 147. Direct reduction calculation component Cd v 148 may include a contribution removal module 139A from component Cd v 147 mathematically removes contribution 182. Contribution 182 is the contribution of data element 102. As shown in the figure, component Cd1144 includes contribution 153 (the contribution of other data elements), component Cd v 148 includes contribution 183 (contributions from other data elements). When the next data element is accessed or received, component Cd1144 can be used to directly decrement the calculation of the next component Cd1, and component Cd... v 148 can be used to directly reduce the Cd of the next component. v .

[0124] like Figure 2 As shown, Reset 211 can be used to reset the decremental kurtosis calculation. When Reset 211 is called after 205 or 210, the v components that compute the subset size counter and kurtosis are reset or initialized. For example, if the computed subset is not empty and component Cd1141 has already been computed, component Cd1141 can be initialized to a specific value. This can occur when decremental kurtosis calculation is combined with iterative or incremental kurtosis calculations. Component Cd v 145 can be initialized in the same way.

[0125] Figure 3-1 The diagram illustrates data elements accessed within a calculation window 300A for calculating kurtosis on streaming data. The difference between a calculation window and a calculation subset is that the data elements within a calculation window are ordered (e.g., data elements are removed from the left or right side of the calculation window each time). For example, see reference... Figure 3-1The oldest data element is removed from the left side of the computation window 300A each time. Over time, the oldest data elements, for example, data elements x1, then x2, then x3, ..., are removed from the computation window and accessed to directly subtract the next v components of kurtosis and indirectly subtract w = pv components. One or more of these subtracted components are used to calculate kurtosis. These v components can be directly subtracted from the data elements removed from the computation window and the v components of the previous data window without accessing other data elements in the computation window 300A. For a given subtraction algorithm, v is a constant, so the operands for directly subtracting v components are also constant, and the operands for indirectly subtracting w = pv components are also constant. Therefore, the time complexity of calculating all p components of a given computation window of size n is reduced from O(n) to O(1). Kurtosis can be calculated using one or more subtracted components, so the overall time complexity is reduced from O(n) to O(1). The larger n is, the more significant the reduction in computational complexity.

[0126] Figure 3-2 The diagram illustrates data elements accessed within a computed subset 300B for reducing kurtosis in large datasets. The difference between a computed subset and a computed window is that data elements on a computed subset are not ordered (e.g., data elements can be removed from any position within the computed subset). For example, see reference... Figure 3-2 An existing data element can be removed from any position in the computed subset 300B. Over time, data elements, for example, data element x... r The components are removed from the gray areas of the computation subset and accessed for directly subtracting the new values ​​of the v components for calculating kurtosis and indirectly subtracting w = pv components. One or more of these subtracted components are used to generate the kurtosis. These v components can be directly subtracted from the removed data elements and the v components of the original computation subset without accessing other data elements in the computation subset 300B. For a given subtraction algorithm, v is a constant, so the number of operands for directly subtracting v components is also a constant, and the number of operands for indirectly subtracting w = pv components is also a constant. Therefore, the time complexity of computing all p components of a given computation subset of size n is reduced from O(n) to O(1). Kurtosis can be calculated using one or more subtracted components, so the overall time complexity is reduced from O(n) to O(1). The larger n is, the more significant the reduction in computational cost.

[0127] The following section provides examples of kurtosis components and decremental kurtosis calculation algorithms.

[0128] Figure 4-1 The definition of kurtosis is illustrated. Assume the subset X = {x} is being calculated.i The group |i=1,…,n} contains the data elements x1,x2,x3,x4,……,x that participate in the kurtosis calculation. n .

[0129] Equations 401 and 402 are the sums S of all data elements of X calculated in the k-th round, respectively. k and average The traditional equation. Equation 403 is for calculating the kurtosis γ of the k-th round for subset X. k (2p) The traditional equations are: Equation 404 is the traditional equation for calculating the excess kurtosis of X in the k-th round; Equation 405 is the traditional equation for calculating the sample kurtosis of X in the k-th round; Equation 406 is the traditional equation for calculating the sample excess kurtosis of X in the k-th round. Assume that after a certain period of time, data element x... r (1≤r≤n) is removed from the computation subset X. Whenever a data element is removed from the computation subset, the kurtosis needs to be recalculated. The computation subset after removing the data element is considered the adjusted computation subset and is defined as X′. A new round of reduction calculation begins.

[0130] Figure 4-2 The diagram illustrates the traditional equations for calculating the kurtosis of the adjusted subset X′. Equations 407 and 408 are the sums S of all data elements of the adjusted X′ calculated in the (k+1)th round, respectively. k+1 and average The traditional equations are as follows: Equation 409 is the traditional equation for calculating the adjusted overall kurtosis of X′ in the (k+1)th round. Equation 410 is the traditional equation for calculating the adjusted excess kurtosis of X′ in the (k+1)th round. Equation 411 is the traditional equation for calculating the adjusted sample kurtosis of X′ in the (k+1)th round. Equation 412 is the traditional equation for calculating the adjusted sample excess kurtosis of X′ in the (k+1)th round.

[0131] To demonstrate how to calculate kurtosis using component deflator, three different deflator kurtosis algorithms are provided as examples.

[0132] Figure 4-3 This section explains the first example of the decremented kurtosis calculation algorithm (decremented algorithm 1). Equation 401 can be used to initialize the sum S of all data elements in X. k Equation 402 can be used to initialize the average value of all data elements in X. Equations 413, 414, 415, and 416 can be used to initialize SS for the data elements of the first computational subset, respectively. k SX k CS k QX k Equation 417 can be used to generate kurtosis γ based on the initialized components, as needed. k (2p)Equation 418 can be used to calculate the excess kurtosis γ based on the initialized components, as needed. k (2e) Equation 419 can be used to calculate the sample kurtosis γ based on the initialized components, as needed. k (2s) Equation 420 can be used to calculate the sample excess kurtosis γ based on the initialized components, as needed. k (2se) Assume that after a period of time, data element x... r To be removed from the computation subset X. Based on component S. k and or Equations 421 and 422 can be used to reduce the amount of S calculated after adjustment for subset X′. k+1 and or Based on component SS k Equation 423 can be used to reduce the adjusted SS of subset X′. k+1 Based on component SX k Equation 424 can be used to reduce the adjusted subset X′ of SX. k+1 Based on component XS k Equation 425 can be used to reduce the CS of the adjusted subset X′. k+1 Based on component QX k Equation 426 can be used to reduce the amount of QX calculated for the adjusted subset X′. k+1 Based on the component of the reduction calculation, equation 427 can calculate the kurtosis γ of the subset X′ after the reduction calculation adjustment. k+1 (2p) Equation 428 can reduce the excess kurtosis γ of the adjusted calculation window X′. k+1 (2e) Equation 429 can reduce the amount of data used to calculate the sample kurtosis γ of the adjusted calculation window X′. k+1 (2s) Equation 430 can reduce the calculated sample excess kurtosis γ of the adjusted calculation window X′. k+1 (2se) 421, 422, 424, and 426 each contain multiple equations, but only one of them is needed, depending on whether sum or average or both are available.

[0133] Figure 4-4 This illustrates the second example of the decremented kurtosis calculation algorithm (decremented algorithm 2). Equation 401 can be used to initialize the sum S of all data elements in X. k Equation 402 can be used to initialize the average value of all data elements in X. Equations 431, 432, 433, 434, and 435 can be used to initialize SS for the data elements of the first computational subset, respectively. k SXk CS k QS k QX k Equation 436 can be used to generate kurtosis γ based on the initialized components, as needed. k (2p) Equation 437 can be used to calculate the excess kurtosis γ based on the initialized components, as needed. k (2e) Equation 438 can be used to calculate the sample kurtosis γ based on the initialized components, as needed. k (2s) Equation 439 can be used to calculate the sample excess kurtosis γ based on the initialized components, as needed. k (2se) Assume that after a period of time, data element x... r To be removed from the computation subset X. Based on component S. k and or Equations 440 and 441 can be used to reduce the amount of S calculated after adjustment for subset X′. k+1 and or Based on component SS k Equation 442 can be used to reduce the adjusted SS of subset X′. k+1 Based on component SX k Equation 443 can be used to reduce the adjusted subset X′ of SX. k+1 Component-based CS k Equation 444 can be used to reduce the CS of the adjusted subset X′. k+1 Component-based QS k Equation 445 can be used to reduce the QS of the adjusted subset X′. k+1 Based on the component reduction calculation, equation 446 can be used to calculate the QX of the adjusted subset X′ after reduction calculation. k+1 Based on the component of the reduction calculation, equation 447 can reduce the adjusted kurtosis γ of the subset X′. k+1 (2p) Equation 448 can reduce the excess kurtosis γ of the adjusted calculation window X′. k+1 (2e) Equation 449 can reduce the amount of data used to calculate the sample kurtosis γ of the adjusted calculation window X′. k+1 (2s) Equation 450 can reduce the calculated sample excess kurtosis γ of the adjusted calculation window X′. k+1 (2se) 440, 441, 443, and 446 each contain multiple equations, but only one of them is needed, depending on whether sum or average or both are available.

[0134] Figure 4-5This illustrates the third example of the decremented kurtosis calculation algorithm (decremented algorithm 3). Equation 401 can be used to initialize the sum S of all data elements in X. k Equation 402 can be used to initialize the average value of all data elements in X. Equations 451, 452, 453, 454, and 455 can be used to initialize SS for the data elements of the first computational subset, respectively. k SX k CS k QS k QX k Equation 456 can be used to generate kurtosis γ based on the initialized components, as needed. k (2p) Equation 457 can be used to calculate the excess kurtosis γ based on the initialized components, as needed. k (2e) Equation 458 can be used to calculate the sample kurtosis γ based on the initialized components, as needed. k (2s) Equation 459 can be used to calculate the sample excess kurtosis γ based on the initialized components, as needed. k (2se) Assume that after a period of time, data element x... r To be removed from the computation subset X. Based on component S. k and or Equations 460 and 461 can be used to reduce the amount of S calculated after adjustment for subset X′. k+1 and or Based on component SS k Equation 462 can be used to reduce the adjusted SS of subset X′. k+1 Based on component SX k Equation 463 can be used to reduce the adjusted subset X′ of SX. k+1 Component-based CS k Equation 464 can be used to reduce the CS of the adjusted subset X′. k+1 Component-based QS k Equation 465 can be used to reduce the QS of the adjusted subset X′. k+1 Based on component reduction calculations, equation 466 can be used to calculate the QX of the adjusted subset X′ after reduction calculations. k+1 Based on the component of the reduction calculation, equation 467 can reduce the adjusted kurtosis γ of the subset X′. k+1 (2p) Equation 468 can reduce the excess kurtosis γ of the adjusted calculation window X′. k+1 (2e) Equation 469 can reduce the amount of data used to calculate the sample kurtosis γ of the adjusted calculation window X′. k+1(2s) Equation 470 can reduce the sample excess kurtosis γ of the adjusted calculation window X′. k+1 (2se) 460, 461, 463, and 466 each contain multiple equations, but only one of them is needed, depending on whether sum or average or both are available.

[0135] To demonstrate debulking kurtosis algorithms and their comparison with traditional algorithms, three examples are given below. Data from three computational subsets are used. For traditional algorithms, the computation process for all three subsets is identical. For debulking algorithms, the first subset initializes one or more components, while the second and third subsets perform debulking computations.

[0136] Figure 5-1 , Figure 5-2 , Figure 5-3 The first, second, and third computation subsets used for a single computation instance are shown. Computation subset 502 consists of 6 data elements from the large dataset 501: 8, 3, 6, 1, 4, 2. The size of computation subset 503(n) is 6. Computation subset 504 consists of 5 data elements from the large dataset 501: 3, 6, 1, 4, 2. The size of computation subset 505(n) is 5. Computation subset 506 consists of 4 data elements from the large dataset 501: 6, 1, 4, 2. The size of computation subset 507(n) is 4.

[0137] First, the sample kurtosis of subsets 502, 504, and 506 is calculated using a traditional algorithm.

[0138] To calculate the sample kurtosis for subset 502:

[0139]

[0140] Without any optimization, calculating the sample kurtosis for a computation subset of size 6 involves 2 divisions, 24 multiplications, 16 additions, and 15 subtractions.

[0141] The same equations and processes can be used to respectively Figure 5-2 The calculated subset 504 shows the sum of sample kurtosis. Figure 5-3 The displayed calculation subset 506 calculates the sample kurtosis. The calculation subset 504, with a size of 5, calculates the kurtosis. This calculation, without optimization, involves 2 divisions, 21 multiplications, 13 additions, and 13 subtractions. The kurtosis of a 506-sample subset of the calculation of size 4 is also considered. This calculation, without optimization, involves 2 divisions, 18 multiplications, 10 additions, and 11 subtractions. Traditional algorithms, without optimization, typically require 2 divisions, 3n+6 multiplications, 3n-2 additions, and 2n+3 subtractions when calculating the sample kurtosis of a computational subset of size n.

[0142] The sample kurtosis of subsets 502, 504, and 506 is calculated using subtraction algorithm 1.

[0143] Calculate the sample kurtosis for a computational subset of size 6, 502:

[0144] The first round is initialized using equations 402, 413, 414, 415, and 416 respectively. SS1, SX1, CS1, and QX1:

[0145]

[0146] Calculate the kurtosis of the first round of samples using Equation 419:

[0147]

[0148] There are 2 divisions, 30 multiplications, 26 additions, and 9 subtractions when calculating the sample kurtosis for subset 502.

[0149] To calculate the sample kurtosis for a computational subset of size 504, the following steps are performed:

[0150] The components for the second round are calculated by subtracting amounts using equations 422, 423, 424, 425, and 426, respectively. SS2, SX2, CS2, and QX2:

[0151]

[0152] SS2 = SS1 - x r 2 =130-8 2 =66

[0153]

[0154] CS2 = CS1 - x r 3 =828-8 3 =316

[0155]

[0156] Calculate the kurtosis of the second round of samples using equation 429:

[0157]

[0158] When calculating the sample kurtosis for the subset 504 reduction, there are 2 divisions, 21 multiplications, 4 additions, and 15 subtractions.

[0159] To calculate the sample kurtosis for a computational subset of size 4, 506:

[0160] The components for the third round are calculated by subtracting amounts using equations 422, 423, 424, 425, and 426, respectively. SS3, SX3, CS3, and QX3:

[0161] (Keep the result of 5-1)

[0162] SS3 = SS2 - x r 2 =66-3 2 =57 (retain x) r 2 (Result)

[0163] (Using the results of 5-1)

[0164] CS3 = CS2 - x r 3 =316-3 3 =289 (using x) r 2 The result and retain x r 3 (Result)

[0165] (Use 5-1, x) r 2 x r 3 The result) uses Equation 429 to calculate the kurtosis of the third round of samples:

[0166]

[0167] When calculating the sample kurtosis for the subset 506 reduction, there are 2 divisions, 21 multiplications, 4 additions, and 15 subtractions.

[0168] The sample kurtosis of subsets 502, 504, and 506 is calculated using subtraction algorithm 2.

[0169] Calculate the sample kurtosis for a computational subset of size 6, 502:

[0170] Initialize the first round using equations 402, 431, 432, 433, 434, and 435. SS1, SX1, CS1, QS1 and QX1:

[0171]

[0172] Calculate the kurtosis of the first round of samples using equation 438:

[0173]

[0174] There are 2 divisions, 36 multiplications, 31 additions, and 9 subtractions when calculating the sample kurtosis for subset 502.

[0175] To calculate the sample kurtosis for a computational subset of size 504, the following steps are performed:

[0176] The components for the second round are calculated by subtracting amounts using equations 441, 442, 443, 444, 445, and 446, respectively. SS2, SX2, CS2, QS2, and QX2:

[0177] (Keep the result of 6-1)

[0178] SS2 = SS1 - x r 2 =130-8 2 =66 (retain x) r 2 (Result)

[0179]

[0180] CS2 = CS1 - x r 3 =828-8 3 =316 (using x) r 2 The result and retain x r 3 (Result)

[0181] QS2 = QS1 - x r 4 =5746-8 4 =1650 (using x) r 3 (Result)

[0182] (Using the results of 6-1) Calculate the kurtosis of the second round of samples using Equation 449:

[0183]

[0184] When calculating the sample kurtosis for the subset 504 reduction, there are 2 divisions, 18 multiplications, 1 addition, and 14 subtractions.

[0185] To calculate the sample kurtosis for a computational subset of size 4, 506:

[0186] The components for the third round are calculated by subtracting amounts using equations 441, 442, 443, 444, 445, and 446, respectively. SS3, SX3, CS3, QS3, and QX3:

[0187]

[0188] SS3 = SS2 - x r 2 =66-3 2 =57 (retain x) r 2 (Result)

[0189]

[0190] CS3 = CS2 - x r 3 =316-3 3 =289 (using x) r 2 The result and retain x r 3 (Result)

[0191] QS3 = QS2 - x r 4 =1650-3 4 =1569 (using x) r 3 (Result)

[0192] (Using the results of 5-1)

[0193] Calculate the kurtosis of the third round of samples using equation 449:

[0194]

[0195] When calculating the sample kurtosis for the subset 506 reduction, there are 2 divisions, 18 multiplications, 1 addition, and 14 subtractions.

[0196] The sample kurtosis of subsets 502, 504, and 506 is calculated using subtraction algorithm 3.

[0197] Calculate the sample kurtosis for a computational subset of size 6, 502:

[0198] Initialize the first round using equations 402, 451, 452, 453, 454, and 455. SS1, SX1, CS1, QS1 and QX1:

[0199]

[0200] Calculate the kurtosis of the first round of samples using equation 458:

[0201]

[0202] There are 2 divisions, 36 multiplications, 31 additions, and 9 subtractions when calculating the kurtosis of subset 502.

[0203] To calculate the sample kurtosis for a computational subset of size 504, the following steps are performed:

[0204] The components for the second round are calculated by subtracting amounts using equations 461, 462, 463, 464, 465, and 466, respectively. SS2, SX2, CS2, QS2, and QX2:

[0205]

[0206] SS2 = SS1 - x r 2 =130-8 2 =66

[0207]

[0208] CS2 = CS1 - x r 3 =828-8 3 =316

[0209] QS2 = QS1 - x r 4 =5746-8 4 =1650

[0210]

[0211] Calculate the kurtosis of the second round of samples using equation 469:

[0212]

[0213] When calculating the sample kurtosis for the subset 504 reduction, there are 2 divisions, 19 multiplications, 1 addition, and 11 subtractions.

[0214] To calculate the sample kurtosis for a computational subset of size 4, 506:

[0215] The components for the third round are calculated by subtracting amounts using equations 461, 462, 463, 464, 465, and 466, respectively. SS3, SX3, CS3, QS3, and QX3:

[0216]

[0217] SS3 = SS2 - xr2 = 66 - 3 2 =57

[0218]

[0219] CS3 = CS2 - x r 3 =316-3 3 =289

[0220] QS3 = QS2 - x r 4 =1650-3 4 =1569

[0221]

[0222] Calculate the kurtosis of the third round of samples using equation 469:

[0223]

[0224] When calculating the sample kurtosis for the subset 506 reduction, there are 2 divisions, 19 multiplications, 1 addition, and 11 subtractions.

[0225] In the three examples above, the average value was used to calculate kurtosis reduction. Both and can also be used for kurtosis reduction calculations, only with different operands.

[0226] Figure 6-1 The diagram illustrates a comparison of the computational complexity of the traditional kurtosis calculation algorithm and the decremented kurtosis calculation algorithm when n=4. As shown in the figure, the number of operands for either decremented algorithm is similar to that of the traditional algorithm.

[0227] Figure 6-2 The diagram illustrates a comparison of the computational complexity of traditional kurtosis calculation algorithms and decremented kurtosis calculation algorithms when n = 1,000,000. As shown, both decremented and decremented kurtosis algorithms significantly reduce the number of multiplication, addition, and subtraction operations compared to traditional algorithms. Decremented kurtosis algorithms can process data that might otherwise require thousands of computers on a single machine, thereby greatly improving computational efficiency, reducing computational resource requirements, and lowering the energy consumption of the computing system. This makes it possible, in some scenarios, to determine the density and concentration of large or streaming data distributions in real time, which was previously impossible.

[0228] This invention can be implemented in other specific ways without departing from its concept or essential characteristics. The implementations described in this application are illustrative rather than restrictive in all respects. Therefore, the specific scope of protection of this invention should be determined by the relevant parts of this application, rather than by the description of the above embodiments. All changes or improvements equivalent to the described technical solutions should be considered within the scope of this invention without departing from the core idea of ​​this invention.

Claims

1. A method for real-time determination of the degree of distribution concentration of time series or streaming big data, characterized in that: A computing system based on one or more computing devices, for a pre-adjusted computed subset of a data stream or a dataset stored on one or more storage devices on the computing system, initializes a computed subset size counter n, n>5, a sum, or an average, or a sum and an average, and one or more other components of kurtosis besides the sum and the average, the computed subset size counter indicating the number of data elements contained in the pre-adjusted computed subset; The computing system based on the computing device receives or accesses a data element to be removed from the pre-adjustment computation subset; The computing system based on the computing device adjusts the original computational subset by: Remove the data elements to be removed from the original computation subset; as well as Adjust the corresponding subset size counter; The computing system based on the computing device calculates a sum, or an average, or a sum and an average for the adjusted subset of calculations. The computing system based on the computing device, at least based on one or more components other than the sum and average of the kurtosis of the pre-adjustment computing subset, directly reduces one or more components of the kurtosis calculation for the adjusted computing subset, wherein the reduction of the one or more components of the calculation includes: Accessing the removed data elements thereby avoiding access to all data elements in the adjusted computation subset reduces data access latency, saves computing resources, and reduces energy consumption; Access one or more components of the kurtosis of the subset before adjustment, excluding the sum and average; and Based on the removed data elements, any contribution of the removed data elements to the accessed component is mathematically removed, thus avoiding the use of all data elements in the adjusted computation subset to directly reduce the kurtosis of one or more components, thereby improving computational efficiency; and The computing system based on the computing device generates kurtosis for the adjusted computation subset based on one or more components that perform subtraction calculations for the adjusted computation subset.

2. The method according to claim 1, characterized in that: Receiving or accessing a data element to be removed includes receiving or accessing multiple data elements to be removed from the pre-adjustment computed subset. The method further includes adjusting the pre-adjustment computed subset for each of the multiple data elements to be removed, reducing a sum, or an average, or a sum and an average, directly reducing one or more components of the kurtosis other than the sum and the average, and generating kurtosis for the adjusted computed subset.

3. The method according to claim 2, characterized in that: The kurtosis is generated for the adjusted subset if and only if the kurtosis is accessed.

4. The method according to claim 3, characterized in that: The generation of kurtosis for the adjusted computed subset further includes one or more components of the computing system based on the computing device indirectly reducing the kurtosis for the adjusted computed subset. The indirect reduction calculation of the one or more components includes calculating the one or more components one by one based on one or more components other than the components to be calculated.

5. A computing system, characterized in that: One or more computing devices; Each computing device contains one or more processors; One or more storage media; as well as One or more computing modules, when executed by at least one of one or more computing devices, determine in real time the degree of concentration of streaming data or big data distribution. These one or more computing modules are configured to: a. For a pre-adjusted computed subset of a data stream or a dataset stored on one or more storage devices on the computing system, initialize a computed subset size counter n, n>5, a sum, or an average, or a sum and an average, and one or more other components of kurtosis besides the sum and the average, the computed subset size counter indicating the number of data elements contained in the pre-adjusted computed subset. b. Receive or access a data element to be removed from the subset calculated before the adjustment; c. Adjust the subset of data that was not previously computed, including: Remove the data elements to be removed from the pre-adjustment subset; and Adjust the corresponding subset size counter; d. Calculate a sum, or an average, or a sum and an average for the adjusted subset reduction; e. Based at least on the kurtosis of the pre-adjustment subset, one or more components other than the sum and average of the kurtosis are directly decomputed for the adjusted subset, and during the decomputation of the one or more components, access to and use of all data elements in the adjusted subset are avoided to reduce data access latency, improve computational efficiency, save computational resources, and reduce the energy consumption of the computing system; and f. Generate kurtosis for the adjusted computed subset based on one or more components that perform reduction calculations for the adjusted computed subset.

6. The computing system according to claim 5, characterized in that: The one or more computing modules, when executed by at least one of the one or more computing devices, execute b, c, d, e, and f multiple times.

7. The computing system according to claim 6, characterized in that: The function f is executed if and only if the kurtosis of the adjusted computed subset is accessed.

8. The computing system according to claim 7, characterized in that: The f further includes one or more components for indirectly reducing the kurtosis of the adjusted computation subset by the computing system, wherein the indirect reduction of the one or more components includes calculating the one or more components one by one based on one or more components other than the components to be calculated.

9. A computing device readable medium comprising a plurality of computing device executable instructions, wherein when these computing device executable instructions are executed by at least one computing device in a computing system comprising one or more computing devices and one or more storage media, the computing system executes a method for real-time determination of the density concentration of streaming data or big data distribution, characterized in that: The computing system initializes a computational subset size counter n, n>5, a sum, or an average, or a sum and an average, and one or more other components of kurtosis other than the sum and the average for a pre-adjustment computational subset of a data stream or a dataset stored on at least one storage medium of the computing system. The computational subset size counter indicates the number of data elements contained in the pre-adjustment computational subset. The computing system receives or accesses a data element to be removed from the pre-adjustment subset of computation; The computational system adjusts the original computational subset by: Remove the data elements to be removed from the original computation subset; as well as Adjust the corresponding subset size counter; The calculation system calculates a sum, or an average, or a sum and an average for the adjusted subset of calculations. The computing system directly reduces the computation of one or more components of the kurtosis of the pre-adjustment computing subset, excluding the sum and average, for the adjusted computing subset. In the process of reducing the computation of the one or more components, access to and use of all data elements in the adjusted computing subset are avoided to reduce data access latency, improve computing efficiency, save computing resources and reduce the energy consumption of the computing system. as well as The computing system generates kurtosis for the adjusted computation subset based on one or more components that perform reduction calculations for the adjusted computation subset.

10. A computing device readable medium comprising a plurality of computing device executable instructions, which, when executed by at least one computing device in a computing system comprising one or more computing devices, cause the computing system to perform the method as described in any one of claims 1-4.

Images