Data stream feature determination method and device, equipment and storage medium

By storing and analyzing the frequency of elements in the data stream and determining the smooth elements, the problem of data stream trend prediction is solved, and fast and accurate network performance management is achieved.

CN119030902APending Publication Date: 2024-11-26HUAWEI TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310593825.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The prior art is difficult to predict the trend of data flows quickly and accurately, especially in the case of rapid changes in network bandwidth and device transmission speed, resulting in network congestion and abnormal traffic being difficult to effectively manage.

Method used

Through the smoothing data structure, the frequency of elements appearing in different windows in the data stream is stored, and whether the element is a smooth element is determined based on the frequency relationship, thereby reflecting the trend of the data stream and real-time and accurate data stream characteristics determination.

Benefits of technology

Improve the accuracy and speed of data flow trend prediction, reduce storage space usage, and enhance network performance management capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119030902A_ABST
    Figure CN119030902A_ABST
Patent Text Reader

Abstract

The invention provides a data stream feature determination method and device, equipment and a storage medium. In an embodiment, a data stream is received, the data stream comprising elements, the data stream being divided into a plurality of sub-data streams, each sub-data stream of the plurality of sub-data streams corresponding to a window; mapping the element into a bucket of a second unit of the smooth data structure, and storing a first frequency number of the element in the current window in the bucket; determining a smooth element from the plurality of elements according to the first frequency numbers of the plurality of elements in the p continuous windows under the condition that the first frequency numbers of the plurality of elements in the p continuous windows are stored in the bucket of the second unit; wherein the first frequencies of the elements in the p windows are in the same row of the bucket. According to the embodiment, the smooth elements in the data stream can be accurately determined in real time through the smooth data structure, and the characteristics of the data stream are reflected.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a method, device, equipment and storage medium for determining characteristics of a data stream. Background Art

[0002] With the rapid development of data networks, network bandwidth and device transmission speed have been greatly improved, which has led to a sharp increase in network traffic and data transmission speed. Therefore, network performance is also changing rapidly.

[0003] In order to deal with problems caused by rapid changes in network performance (such as network congestion, abnormal traffic, etc.), it is necessary to predict the data flow trend in the next period of time. The trend of data flow can be reflected through the characteristics of data flow.

[0004] Therefore, there is an urgent need for a technical solution that can accurately and real-time discover the characteristics of data streams. Summary of the invention

[0005] The embodiments of the present application provide a method, apparatus, device and storage medium for determining characteristics of a data stream, which can determine smooth elements in the data stream in real time and accurately, and reflect the characteristics of the data stream.

[0006] In a first aspect, an embodiment of the present application provides a method for determining characteristics of a data stream, including:

[0007] receiving a data stream, the data stream comprising elements, the data stream being divided into a plurality of sub-data streams, each of the plurality of sub-data streams corresponding to a window;

[0008] Map the element to the bucket of the second unit of the smooth data structure, and store the first frequency of the element in the current window in the bucket;

[0009] In a case where the first frequencies of a plurality of elements in p consecutive windows are stored in the bucket of the second unit, a smoothing element is determined from the plurality of elements according to the first frequencies of the plurality of elements in the p windows;

[0010] Among them, the first frequency of the element in the p windows is in the same row of the bucket.

[0011] According to this solution, the frequency of an element in p consecutive windows can be stored through a smooth data structure, so that based on the first frequency of the element in p consecutive windows, it is possible to determine in real time and accurately whether the element is a smooth element, and further based on the smooth element, the trend of the data flow can be reflected.

[0012] In one possible implementation, mapping the element to the bucket of the second unit of the smooth data structure includes:

[0013] mapping the element to a first storage location of a first location in the smoothed data structure;

[0014] Record the first frequency of the element in the consecutive s windows through the first storage unit;

[0015] According to the first frequency of the element in s windows, determine whether the element has the potential to become a smooth element;

[0016] In case the element has the potential to be a smooth element, the element is mapped into the bucket of the second unit of the smooth data structure.

[0017] In this way, the elements in the data stream that cannot be called smooth elements can be filtered out, thereby reducing the storage space occupied.

[0018] In a possible implementation, storing the first frequency of the element in the current window includes:

[0019] The first frequency of the element in the first storage unit in the s windows is stored in the bucket where the element is located.

[0020] In a possible implementation, determining whether an element has the potential to become a smoothing element according to the first frequency of the element in s windows includes:

[0021] Fit the first frequency of the element in s windows to obtain the first function after fitting;

[0022] According to the first function, calculate the second frequency of the element in each of the s windows;

[0023] Whether the element has the potential to be a smooth element is determined based on the first frequency of the element in each of the s windows and the second frequency of the element in each of the s windows.

[0024] In this way, the elements in the data stream that cannot be called smooth elements can be filtered out, thereby reducing the storage space occupied.

[0025] In a possible implementation, determining whether an element has the potential to become a smoothing element according to the first frequency of the element in each of the s windows and the second frequency of each of the s windows includes:

[0026] If the relationship between the first frequency of an element in each of the s windows and the second frequency of an element in each of the s windows satisfies the first formula, then the element has the potential to become a smooth element;

[0027] The first formula includes f(i)>0 and Wherein, f′(i) represents the first frequency of the element in the i-th window, f(i) represents the second frequency of the element in the i-th window, ε represents the root mean square of the error between the first frequency and the second frequency of the element in s consecutive windows, T represents the preset first threshold, i=0, 1, 2, ..., s.

[0028] In this way, the elements in the data stream that cannot be called smooth elements can be filtered out, thereby reducing the storage space occupied.

[0029] In a possible implementation, the first formula also includes μ≥G, Wherein, μ represents the potential of an element to become a smoothing element, G represents a preset second threshold, and Δ represents a value infinitely close to 0 but not equal to 0.

[0030] In this way, the elements in the data stream that cannot be called smooth elements can be filtered out, thereby reducing the storage space occupied.

[0031] In a possible implementation, the bucket of the second unit also stores the number of the target window where the target element starts to be smoothed;

[0032] When all buckets in the second unit are occupied, mapping the element to the bucket of the second unit of the smooth data structure includes:

[0033] When all buckets in the second unit are occupied, the duration of smoothing of each target element is determined according to the number of the target window of the target element stored in the second unit and the number of the current window;

[0034] If the duration of the target element starting smoothing in the second unit meets the preset condition, the element is inserted into the bucket where the target element is located, and the information of the target element in the bucket is deleted.

[0035] In this way, the element that is most likely to become a smoothing element can be stored in the second unit, thereby reducing the occupation of storage space.

[0036] In a possible implementation, determining a smoothing element from a plurality of elements according to first frequencies of each of the plurality of elements in p windows includes:

[0037] Determine the second frequency of the element in the p windows according to the first frequency of the element in the p windows;

[0038] Determine whether the element is a smooth element based on the first frequency and the second frequency of the element in p windows.

[0039] In a possible implementation, determining the second frequency of the element in the p windows according to the first frequency of the element in the p windows includes:

[0040] Fit the first frequency of the element in p windows to obtain the second function;

[0041] According to the second function, the second frequencies of the elements in the p windows are calculated.

[0042] In a possible implementation, determining whether an element is a smooth element according to the first frequency and the second frequency of the element in p windows includes:

[0043] When the element satisfies the second formula, the element is determined to be a smooth element;

[0044] The second formula is f(n)>0 and Wherein, f′(n) represents the first frequency of the element in the nth window, f(n) represents the second frequency of the element in the nth window, ε′ represents the root mean square of the error between the first frequency and the second frequency of the element in p consecutive windows, T represents a preset first threshold, and n=0, 1, 2, ..., p.

[0045] In a second aspect, an embodiment of the present application provides a device for determining characteristics of a data stream, characterized in that it includes:

[0046] A receiving module, used for receiving a data stream, the data stream comprising elements;

[0047] A processing module, used for mapping the element to a bucket of a second unit of the smooth data structure, and storing a first frequency of the element in a current window in the bucket, wherein the window is used for dividing the data stream;

[0048] A determination module, configured to determine a smoothing element from a plurality of elements according to the first frequencies of the plurality of elements in the p windows when the first frequencies of the plurality of elements in the p windows are stored in the bucket of the second unit;

[0049] Among them, the first frequency of the element in p windows is in the same row of the bucket, n = 0, 1, 2...p.

[0050] According to this solution, the frequency of an element in p consecutive windows can be stored through a smooth data structure, so that based on the first frequency of the element in p consecutive windows, it is possible to determine in real time and accurately whether the element is a smooth element, and further based on the smooth element, the trend of the data flow can be reflected.

[0051] In a possible implementation, the processing module is used to:

[0052] mapping the element to a first storage location of a first cell in the smoothed data structure;

[0053] Record the first frequency of the element in the consecutive s windows through the first storage unit;

[0054] According to the first frequency of the element in s windows, determine whether the element has the potential to become a smooth element;

[0055] In case the element has the potential to be a smooth element, the element is mapped into the bucket of the second unit of the smooth data structure.

[0056] In this way, the elements in the data stream that cannot be called smooth elements can be filtered out, thereby reducing the storage space occupied.

[0057] In a possible implementation, the processing module is used to:

[0058] The first frequency of the element in the first storage unit in the s windows is stored in the bucket where the element is located.

[0059] In a possible implementation, determining whether an element has the potential to become a smoothing element according to the first frequency of the element in s windows includes:

[0060] Fit the first frequency of the element in s windows to obtain the first function after fitting;

[0061] According to the first function, calculate the second frequency of the element in each of the s windows;

[0062] Whether the element has the potential to be a smooth element is determined based on the first frequency of the element in each of the s windows and the second frequency of the element in each of the s windows.

[0063] In this way, the elements in the data stream that cannot be called smooth elements can be filtered out, thereby reducing the storage space occupied.

[0064] In a possible implementation, the processing module is used to:

[0065] If the relationship between the first frequency of an element in each of the s windows and the second frequency of an element in each of the s windows satisfies the first formula, then the element has the potential to become a smooth element;

[0066] The first formula includes f(i)>0 and Wherein, f′(i) represents the first frequency of the element in the i-th window, f(i) represents the second frequency of the element in the i-th window, ε represents the root mean square of the error between the first frequency and the second frequency of the element in s consecutive windows, T represents the preset first threshold, i=0, 1, 2, ..., s.

[0067] In this way, the elements in the data stream that cannot be called smooth elements can be filtered out, thereby reducing the storage space occupied.

[0068] In a possible implementation, the first formula also includes μ≥G, Wherein, μ represents the potential of an element to become a smoothing element, G represents a preset second threshold, and Δ represents a value infinitely close to 0 but not equal to 0.

[0069] In this way, the elements in the data stream that cannot be called smooth elements can be filtered out, thereby reducing the storage space occupied.

[0070] In a possible implementation, the bucket of the second unit also stores the number of the target window where the target element starts to be smoothed;

[0071] When all the buckets in the second unit are occupied, the processing module is used to:

[0072] When all buckets in the second unit are occupied, the duration of smoothing of each target element is determined according to the number of the target window of the target element stored in the second unit and the number of the current window;

[0073] If the duration of the target element starting smoothing in the second unit meets the preset condition, the element is inserted into the bucket where the target element is located, and the information of the target element in the bucket is deleted.

[0074] In this way, the element that is most likely to become a smoothing element can be stored in the second unit, thereby reducing the occupation of storage space.

[0075] In a possible implementation, the determination module is used to:

[0076] Determine the second frequency of the element in the p windows according to the first frequency of the element in the p windows;

[0077] Determine whether the element is a smooth element based on the first frequency and the second frequency of the element in p windows.

[0078] In a possible implementation, the determination module is used to:

[0079] Fit the first frequency of the element in p windows to obtain the second function;

[0080] According to the second function, the second frequencies of the elements in the p windows are calculated.

[0081] In a possible implementation, the determination module is used to:

[0082] When the element satisfies the second formula, the element is determined to be a smooth element;

[0083] The second formula is f(n)>0 and Wherein, f′(n) represents the first frequency of the element in the nth window, f(n) represents the second frequency of the element in the nth window, ε′ represents the root mean square of the error between the first frequency and the second frequency of the element in p consecutive windows, T represents a preset first threshold, and n=0, 1, 2, ..., p.

[0084] In a third aspect, an embodiment of the present application provides a computing device, comprising: at least one memory for storing programs; and at least one processor for executing the programs stored in the memory, wherein when the programs stored in the memory are executed, the processor is used to execute the method provided in the first aspect.

[0085] In a fourth aspect, an embodiment of the present application provides a device for determining characteristics of a data stream, characterized in that the device runs computer program instructions to execute the method provided in the first aspect.

[0086] In one example, the device may include a processor, which may be coupled to a memory, read instructions in the memory and execute the method provided in the first aspect according to the instructions. The memory may be integrated in a chip or a processor, or may be independent of the chip or the processor.

[0087] In a fifth aspect, an embodiment of the present application provides a computer storage medium, in which instructions are stored. When the instructions are executed on a computer, the computer executes the method provided in the first aspect.

[0088] In a sixth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method provided in the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0089] Figure 1 It is a structural schematic diagram of a tower-type data structure provided in an embodiment of the present application;

[0090] Figure 2 It is a schematic diagram of a tower-shaped data structure for storing data provided by an embodiment of the present application;

[0091] Figure 3 is a schematic diagram of a window division provided in an embodiment of the present application;

[0092] Figure 4 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0093] Figure 5 is a structural diagram of a smoothed data structure provided in an embodiment of the present application;

[0094] Figure 6It is a flowchart of a method for determining characteristics of a data stream provided in an embodiment of the present application;

[0095] Figure 7 It is a flowchart of another method for determining characteristics of a data stream provided in an embodiment of the present application;

[0096] Figure 8 It is a schematic diagram of inserting an element into a bucket in a smooth data structure provided by an embodiment of the present application;

[0097] Fig. 9 It is a structural diagram of a data stream feature determination device provided in an embodiment of the present application;

[0098] Fig.10 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0099] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0100] In the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete way.

[0101] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "multiple" means two or more. For example, multiple systems refers to two or more systems, and multiple terminals refers to two or more terminals.

[0102] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0103] Hereinafter, some terms used in this embodiment are explained. It should be noted that these explanations are for the convenience of understanding by those skilled in the art and do not constitute a limitation on the scope of protection claimed by the present invention.

[0104] Data structure (Sketch): Sketch refers to a two-dimensional table data structure.

[0105] For example, Figure 1 The tower data structure (TowerSketch) shown in FIG. 1 includes d layers, wherein the i-th layer is represented by A i Each layer includes an array, a total of d arrays. Among them, each layer includes q i A counter, A i The counters of the layer, 1≤i≤d,. Each counter contains δ i Position, such as Figure 1 The counter in the middle layer A1 is the bit δ i 4 bits. Each array is associated with a hash function hi(*), 1≤hi(*)≤d. h1, h2, ..., hd are independent of each other.

[0106] The tower data structure can achieve the matching of the counter bit number and flow rate by automatic balancing. The automatic balancing method is as follows: each layer in the tower data structure contains the same space size, such as Figure 1 As shown, the tower data structure increases from top to bottom, the bits of the counter increase, and the number of counters q i reduce.

[0107] In this example, if Figure 2 As shown, during the insertion operation on the data structure, given an element of a stream with an identity document (ID) of e, TowerSketch first calculates d hash functions h1(e), h2(e), ..., hd(e) to locate d mapped counters L1[h1(e)], L2[h2(e)], ..., Ld[hd(e)]. Then, TowerSketch increases each of the d mapped counters by 1. Note that if there is a counter overflow, TowerSketch sets the counter to its maximum value, indicating that it has overflowed, and the overflowed counter will no longer be modified.

[0108] Element: Also known as a flow, in the data flow of the network, an element is usually defined as part or all of the information of a five-tuple, which includes the source Internet Protocol (IP) address, destination IP address, source port, destination port, and communication protocol.

[0109] Frequency of an element: the number of times an element appears in a data stream.

[0110] Data flow pattern: refers to a series of characteristics exhibited by the elements that make up the data flow.

[0111] Window: Divide the data stream into several small and continuous sub-data streams, and the sub-data streams are windows. Figure 3 , the data stream Data is evenly divided into n continuous sub-data streams, namely sub-data stream data1, sub-data stream data2, ..., sub-data stream data n. Among them, the sub-data stream data1, sub-data stream data2, ..., sub-data stream datan have the same size. Sub-data stream data1 can be regarded as window 1, sub-data stream data2 can be regarded as window 2, ..., sub-data stream data n can be regarded as window n.

[0112] Smooth element: used to characterize a continuous and smooth pattern in a data stream. When the data stream is divided into several identical and continuous windows, if the frequency of an element e in p continuous windows can be fitted by a polynomial of no more than k degrees, then the element e is a smooth element.

[0113] With the rapid development of data networks, network bandwidth and device transmission speed have been greatly improved, which has led to a sharp increase in network traffic and data transmission speed. Therefore, in the process of network data transmission, various important traffic patterns may appear, which often have a great impact on network performance. For example, burst elements that burst and disappear in the network may cause network congestion and packet loss.

[0114] At present, in practical applications, abnormal traffic detection (such as advanced persistent threats, Trojans, etc.), network congestion control and network traffic shaping all require the prediction of data traffic trends in advance. However, during the high-speed transmission of data streams, the prediction of data traffic trends needs to be fast enough to catch up with the high-speed data streams.

[0115] In the related art, the trend of data traffic is predicted by machine learning, such as by linear regression model or time series model. However, the method of predicting data traffic by machine learning cannot meet the requirements of fast prediction.

[0116] Based on this, an embodiment of the present application provides a method, device, equipment and storage medium for determining a smooth element in a data stream. The method can store the frequency of occurrence of elements in the data stream in different windows of the data stream through a smooth data structure, so that according to the frequency of occurrence of the element in different windows, it is possible to determine in real time and accurately whether the element is a smooth element, thereby reflecting the trend of the data stream. This is only a brief description of the method, and the details of the method are described below.

[0117] Next, possible application scenarios of the method for determining smoothing elements in a data stream provided by an embodiment of the present invention are introduced. Figure 4 Schematic diagram of an application scenario of network congestion control provided by an embodiment of the present application. Figure 4 As shown, in the application scenario of network congestion control, the switch is divided into two areas, namely the control plane 41 and the data plane 42. The functions of the control plane include routing selection and management, such as operating system management, administrator user interface, routing protocol processing, etc. The functions of the data plane include data forwarding, etc.

[0118] Exemplarily, in an embodiment of the present application, the data plane of the switch can be divided into a smoothing element determination module 411 and a routing list storage module 412. The control plane of the switch can be divided into a traffic analysis prediction module 421 and a routing rule generation module 422. Among them, the smoothing element determination module is used to detect the smoothing elements in the data flow and report the frequency of the smoothing elements in the data flow to the traffic analysis prediction module. The traffic analysis prediction module performs analysis based on the frequency of the smoothing elements to determine the smoothing trend of the smoothing elements, and then predicts the number of data packets in the next time period. The routing rule generation module performs routing analysis based on the number of data packets in the next time period, and then determines the routing path of the data flow. The routing rule generation module sends the routing path of the data flow to the routing list storage module. According to the routing path, the next hop switch of the data flow is determined.

[0119] Here, the embodiments of the present application do not specifically limit the scenarios in which the method for determining a smoothing element in a data stream can be applied. The method for determining a smoothing element in a data stream provided by the embodiments of the present application can also be applied to data forwarding devices in the scenarios of abnormal traffic detection and network traffic shaping. For details, see Figure 4 The detailed description of the switch is not repeated here.

[0120] Before describing the method for determining smoothing elements in a data stream provided by an embodiment of the present application, first, a new type of data structure provided by an embodiment of the present application is described in detail.

[0121] Figure 5Schematic diagram of the structure of the smooth data structure (Smooth Sketch) provided in the embodiment of the present application. Figure 5 As shown in Figure 1, Smooth Sketch consists of two parts, namely the improved tower data structure (TowerSketch) and the hash table. The improved tower data structure is the first stage, which is used to record the frequency of the element with stream ID e in the most recent s consecutive windows, where s can be a pre-configured value and s is less than p. The tower data structure can be found in Figure 1 The data structure described will not be described in detail here. Specifically, Figure 5 As shown, each counter Count in the improved tower data structure includes s sub-counters Count'. The i-th sub-counter among the s sub-counters is used to record the frequency f(·) of the element with ID e in the i-th window.

[0122] The hash table is the second stage, which is used to record the ID of the element, the frequency of the element in p consecutive windows in the recent period, and the number of the window where the element starts smoothing (hereinafter referred to as the starting window). The hash table includes m buckets, such as Figure 5 As shown in the figure, the m buckets are B1, B2, ..., Bm. Each bucket has u cells, each of which is used to store information of three fields, namely the flow ID field, the value Count of the counter (that is, the frequency f′(·) of the element in the window), and the number w of the starting window. str The count field of the counter includes p sub-counters, which are used to record the frequency of the element with ID e in p consecutive windows. The hash function can map an element to a bucket in the hash table.

[0123] Next, taking a switch as an execution subject, the method for determining a smoothing element in a data stream provided in an embodiment of the present application is exemplarily described.

[0124] Figure 6 FIG. 1 is a flow chart of a method for determining a smoothing element in a data stream provided by an embodiment of the present application. Figure 6 As shown, the method for determining smoothing elements in a data stream provided by an embodiment of the present application includes the following steps S601 to S620.

[0125] S601, receiving a data stream.

[0126] The data stream may include multiple ordered data packets. Each data packet includes information such as source IP address, destination IP address, source port, destination port, and communication protocol. Among them, part or all of the information in the source IP address, destination IP address, source port, destination port, and communication protocol can be regarded as one element.

[0127] S602, map the element to the hash table, determine whether the element has been inserted into the bucket in the hash table, if so, execute S603, if not, execute S606.

[0128] If the element has been inserted into the bucket in the hash table, the element is the target element. Wherein, the target element refers to an element with the potential to become a smooth element. In an embodiment of the present application, the element can be mapped to the bucket in the hash table through a hash function, thereby determining whether the element has been inserted into the bucket in the hash table.

[0129] S603, update the element information in the hash table.

[0130] Specifically, the counter corresponding to the current window is increased by 1. The switch can determine the counter representing the current window by calculating wmodp.

[0131] S604, when all windows of the element have been recorded, fitting the frequency of the element in p consecutive windows, and then calculating the fitting frequency of the element in each of the p consecutive windows.

[0132] By fitting the frequency of elements in p consecutive windows, the polynomial f(n) can be calculated.

[0133] The fitting frequency of the element in the nth window satisfies the following formula (1):

[0134]

[0135] Wherein, n=0, 1, ..., p, p, k are preset positive integers.

[0136] S605 , determining a smoothing element from all elements recorded in the hash table according to the actual frequencies and fitting frequencies corresponding to the elements in the p consecutive windows.

[0137] The fitting frequency of the element corresponding to each of the p consecutive windows is greater than 0, and the root mean square of the error between the actual frequency and the fitting frequency of the target element corresponding to each of the p consecutive windows is less than or equal to the preset root mean square threshold T.

[0138] If the fitted frequency f(n) of an element, the root mean square ε′ of the error between the fitted frequency f(n) of the element in p consecutive windows and the actual frequency f′(n), the element is a smooth element.

[0139]

[0140] It should be noted that the elements mentioned in S603 to S605 are elements that can be inserted into the hash table, that is, the elements mentioned in S603 to S605 are actually target elements. The definition of the target element can refer to the target element defined in S602, which will not be repeated here.

[0141] S606, recording the actual frequency of the element in the window through a tower data structure.

[0142] The actual frequency of an element in a window is the frequency counted by the switch. For example, if the source IP address 192.168.182.48 has a frequency of 50 in the 8th window, then the actual frequency of the element in the 8th window is 50.

[0143] In an embodiment of the present application, when an element cannot be inserted into a hash table, it is necessary to map the element to a tower data structure. Specifically, assuming that there are d arrays in the tower data structure, and each array corresponds to a hash function, then d hash functions are used to perform hash calculations on element e to obtain hash values ​​h1(e), h2(e), ..., hd(e). Next, element e is mapped to counters A1[h1(e)], A2[h2(e)], ..., Ad[hd(e)]. Since there are multiple sub-counters in each array, each sub-counter corresponds to a window, which is used to record the frequency of element e in the window. Since different sub-counters correspond to a window, the switch can determine the sub-counter corresponding to the i-th window by calculating w mod s. According to the sub-counter corresponding to the i-th window, the frequency of element e is recorded. Here, the window corresponds to a sub-counter in different arrays. Therefore, there are d counters corresponding to the i-th window. The frequency of element e is recorded by d counters.

[0144] As a possible implementation, the frequency of element e in the i-th window may be recorded through the CM version of SmoothSketch (expressed as SmoothSketch-CM). Exemplarily, the d sub-counters corresponding to the i-th window may all be increased by 1.

[0145] As another possible implementation, the frequency of the element in the i-th window may be recorded by the CU version of SmoothSketch. Exemplarily, the sub-counter with the minimum value among the d sub-counters corresponding to the i-th window may be increased by 1.

[0146] S607, fitting the frequencies of the element in s consecutive windows to calculate the fitting frequency of the element in each window.

[0147] By fitting the frequencies of elements in s consecutive windows, we can get the polynomial f(i).

[0148] The fitting frequency of the element in the i-th window satisfies the following formula (3):

[0149]

[0150] Among them, a0, a1, ..., a k are polynomial coefficients, t is a positive integer greater than or equal to 0, and i is a positive integer greater than or equal to 0 and less than or equal to s.

[0151] Here, s is less than p and greater than or equal to k+1.

[0152] It should be noted that there may be an element whose frequency in the window is 0. In this case, there is no need to perform polynomial fitting on the frequency of the element in the window. You can then perform polynomial fitting on the frequency of the element in the next window, or then perform polynomial fitting on the frequency of the next element in the window.

[0153] S608, judging whether the element meets the preliminary conditions according to the actual frequencies and fitting frequencies corresponding to the element in the s consecutive windows, if so, executing S609, if not, jumping to S601, judging whether the next element can become a smoothing element.

[0154] The preliminary condition is that the fitting frequency corresponding to each element in s consecutive windows is greater than 0, and the root mean square of the error between the actual frequency and the fitting frequency corresponding to each element in s consecutive windows is less than or equal to a preset error threshold.

[0155] Specifically, in order to avoid storing a large number of frequencies and occupying a large amount of memory space, the switch filters out elements that cannot be called smooth elements through S604 to save memory space in the switch. The preliminary condition can satisfy the following formula (4):

[0156]

[0157] f′(i) represents the actual frequency of the element in the i-th window, T represents the preset error threshold, and ε represents the root mean square of the error between the actual frequency and the fitted frequency of the element in s consecutive windows.

[0158] S609, determining a target element from the elements satisfying the preliminary condition.

[0159] The target element refers to an element that has the potential to become a smoothing element. The target element can be inserted into the second stage to determine the final smoothing element.

[0160] In a possible implementation, in S609, the highest order polynomial coefficient a of the element in the polynomial fitting process may be used. kThe sum of the root mean square ε determines the potential of an element to become a smoothing element. The potential μ of an element to become a smoothing element satisfies the following formula (5).

[0161]

[0162] Δ represents a very small number, which is used to prevent the denominator in formula (3) from being equal to 0. For example, Δ can be a number that is infinitely close to 0 but not equal to 0.

[0163] If the potential of an element that meets the preliminary conditions is greater than or equal to a preset potential threshold, the element is determined as a target element, that is, the element has the potential to become a smoothing element.

[0164] S610, traverse the hash table to determine whether there is an empty bucket in the hash table. If so, execute S611, if not, execute S612.

[0165] S611, insert the frequency of the target element in the s windows into the empty bucket.

[0166] In the case where the hash table includes an empty bucket, the frequency of the target element in the window can be inserted into the empty bucket. Here, if the hash table has multiple empty buckets, the frequency of the target element in the window can be randomly inserted into the empty bucket.

[0167] S612, calculating the weights of the elements in the hash table.

[0168] The weight of an element identifies the duration of element smoothing, that is, the number of windows in which the element is continuously smoothed. Here, the weight of the target element in the hash table is the difference between the number of the element in the window and the number of the starting window.

[0169] S613, determine the element with the smallest weight value in the hash table.

[0170] S614, according to the minimum weight value, determine whether to insert the target element into the hash table. If yes, execute S605. If not, the target element cannot become a smooth element, and continue to determine whether the next element can become a smooth element. The determination method can be found in S601 to S614, which will not be repeated here.

[0171] According to the minimum weight value, the probability value P corresponding to the element can be calculated. Among them, if the probability value meets the replacement condition, the target element replaces the element with the smallest weight value. Specifically, the identity of the element with the smallest weight value, the actual frequency in the window, and the window number are deleted, and the identity of the target element, the actual frequency of the target element in the window, and the window number are inserted into the bucket of the hash table.

[0172] Here, the probability value P of the element satisfies the following formula (6):

[0173]

[0174] The probability value of an element indicates the probability of the element being replaced by the target element. The lower the probability, the smaller the probability of being replaced by the target element. The larger the probability, the greater the probability of being replaced by the target element. Here, the replacement condition can be a preset probability threshold. The user can set a specific probability threshold according to their own needs.

[0175] Based on the data stream feature determination method in the above-mentioned embodiment, the data stream feature determination method provided in the embodiment of the present application is described in detail.

[0176] Figure 7 The method for determining the characteristics of a data stream provided in the embodiment of the present application can be applied to Figure 4 A switch or other computing device that receives a data stream as shown in FIG. Figure 7 As shown, the method for determining characteristics of a data stream provided in an embodiment of the present application includes the following steps S701 to S703.

[0177] S701, receiving a data stream, the data stream including elements, the data stream being divided into a plurality of sub-data streams, each of the plurality of sub-data streams corresponding to a window.

[0178] The data stream can be divided into multiple sub-data streams, each of which corresponds to a window. For details, see Figure 3 The detailed description of the window in the corresponding embodiment will not be repeated here. Here, the sizes of the multiple sub-data streams are the same, that is, the size of each window is also the same.

[0179] The data stream includes multiple elements, and the frequency of occurrence of elements in different sub-data streams can reflect the changing trend of the elements.

[0180] S702, mapping the element to the bucket of the second unit of the smooth data structure, and storing the first frequency of the element in the current window in the bucket.

[0181] In order to predict the trend of the data stream, it is necessary to predict the smooth trend of the elements. Based on this, the embodiment of the present application proposes a smooth data structure. The smooth data structure provided by the embodiment of the present application includes a second unit, and the second unit includes a plurality of buckets. Here, the second unit can be regarded as a unit with a storage function, and each bucket can be regarded as a storage unit, for example, Figure 5 The hash table shown.

[0182] The first frequency is the number of times the element actually appears in the sub-data stream. Figure 6In the corresponding embodiment, the actual frequency is the first frequency. The bucket includes multiple counters that can store the first frequencies of elements in different windows. When the element arrives at the device, the first frequency of the element in the current window is stored in the bucket in the second unit.

[0183] In the embodiment of the present application, before inserting the first frequency of the element in the current window into the bucket, it is necessary to traverse all the buckets in the second unit to determine whether the first frequency of the element in the window before the current window is stored in the second unit.

[0184] As a possible implementation manner, when the first frequency of the element in the window before the current window is already stored in the second unit, the first frequency of the element in the current window is inserted into the bucket of the second unit.

[0185] For example, Figure 8 As shown, the second unit stores the first frequencies of elements e2, e3, e6, and e7 in multiple windows. The current window is numbered 28, the frequency of e3 in window 28 is 2, and the bucket B3 in the second unit stores the first frequencies of element e3 in windows 21 to 27. Then, the first frequency of element e3 in window 28 is inserted into the row where element e3 is located.

[0186] As another possible implementation, when the first frequency of the element in any window is not stored in the second unit, it is necessary to evaluate the potential of the element to become a smoothing element. When the potential of the element meets the conditions, the element is inserted into the bucket of the second unit. First, the computing device maps the element to the first storage unit of the first unit in the smoothing data structure. Next, the first storage unit in the computing device can record the first frequency of the element in s consecutive windows. Then, based on the first frequency of the element in the s windows, it is determined whether the element has the potential to become a smoothing element; finally, when the element has the potential to become a smoothing element, the element is mapped to the bucket of the second unit of the smoothing data structure.

[0187] Specifically, the smoothing data structure also includes a first unit, such as Figure 5 The first stage shown in FIG. The first unit may be a unit with a storage function, such as a processor with a memory in a computing device (switch), and the first storage unit may store the first frequency of the element in different windows, such as Figure 1 and Figure 2 In the array shown in , the first unit can be divided into a plurality of first storage units, such as Figure 1 and Figure 2Each first storage unit includes a plurality of counters, each counter includes a plurality of sub-counters, so that the counter in the first storage unit can record the first frequency of the element in the continuous s windows. The specific mapping process of the first frequency of the element can be referred to Figure 6 The detailed description of S606 in the corresponding embodiment will not be repeated here.

[0188] In the case where the first unit stores the first frequency of the element in the continuous s windows, the computing device fits the first frequency of the element in the continuous s windows to obtain a fitted first function. The first function can be found in Figure 6 The corresponding formula (3) in the embodiment will not be described in detail here. According to the first function and the number i of the window, the second frequency of the element in the i-th window can be calculated. Here, the second frequency is equivalent to Figure 6 The calculation process of the second frequency can be specifically referred to in Figure 6 The detailed description of S607 in the corresponding embodiment will not be repeated here.

[0189] According to the first frequency and the second frequency of the element in the continuous s windows, it is determined whether the element has the potential to become a smooth element. Specifically, when the element satisfies the first formula between the first frequency and the second frequency of the continuous s windows, the element has the potential to become a smooth element. Here, the first formula may include Figure 6 Corresponding to formula (4) in the embodiment, when an element satisfies the first formula, the element has the potential to become a smooth element.

[0190] As a possible implementation, the first formula may also include Figure 6 The corresponding formula (5) in the embodiment. According to formula (5), the potential of an element to become a smoothing element can be further evaluated. In the embodiment of the present application, the evaluation of the potential of a smoothing element can be specifically referred to Figure 6 The detailed description of S608 and S609 in the embodiment will not be described in detail here.

[0191] In the case where the element has the potential to become a smoothing element, the element is inserted into the second unit, specifically, the first frequency of the element stored in the first unit in the continuous s windows is inserted into the second unit. Here, the information stored in the second unit includes the ID of the element, the first frequency of the element in each window, and the number of the window where the element starts smoothing. In an embodiment of the present application, inserting the element into the second unit includes storing the ID of the element and the first frequency of the element in the continuous s windows in the second unit.

[0192] In some embodiments, if the storage space of the second unit is full, it is necessary to compare the elements that have been inserted into the second unit with the elements that are about to be inserted into the second unit. For example, the storage space of the second unit has 8 buckets, which respectively store the relevant information of element e1, element e2, element e3, ..., element e8. Element e9 has the potential to become a smoothing element, so it is necessary to compare element e9 with element e1, element e2, element e3, ..., element e8 to determine whether it is necessary to replace the element already stored in the second unit with element e9.

[0193] Specifically, when all the buckets in the second unit are occupied, the duration of smoothing of each target element is determined according to the number of the target window of the target element stored in the second unit and the number of the current window; when the duration of smoothing of the target element in the second unit meets the preset conditions, the element is inserted into the bucket where the target element is located, and the information of the target element in the bucket is deleted.

[0194] The target element is the element that has been inserted into the bucket of the second unit. The target window refers to the window where the element starts to be smoothed. As a possible implementation method, the weight value of each target element is calculated, and the probability of the target element with the smallest weight value being replaced is calculated. If the probability is greater than the preset probability threshold, the target element with the smallest weight value is deleted from the second unit, and the element is inserted into the bucket of the second unit. For details, see Figure 6 The description of S612 to S614 in the corresponding embodiment will not be repeated here.

[0195] S703, when the first frequencies of multiple elements in p consecutive windows are stored in the bucket of the second unit, a smoothing element is determined from the multiple elements according to the first frequencies of the multiple elements in the p windows, and the p windows have the same size; wherein the first frequencies of the elements in the p windows are in the same row of the bucket.

[0196] In the case where the first frequencies of the element in the continuous p windows are stored in the second unit, whether the element is a smooth element can be evaluated. Specifically, the second frequency of the element in the p windows is determined according to the first frequency of the element in the p windows; and whether the element is a smooth element is determined according to the first frequency and the second frequency of the element in the p windows.

[0197] Among them, the first frequency of the element in p windows is fitted to obtain the second function, which is Figure 6 For details, please refer to the formula (1) in the corresponding embodiment. Figure 6 The detailed description of formula (1) in the corresponding embodiment will not be repeated here.

[0198] According to the second function, the second frequency of the element in each of the p windows is calculated. Substituting the numbers of the p windows into the second function, the second frequency of the element in each of the p consecutive windows can be calculated. If the first frequency and the second frequency of the element in the p consecutive windows satisfy the second formula, the element is a smooth element. The second formula is Figure 6 For details, please refer to the formula (2) in the corresponding embodiment. Figure 6 The detailed description of formula (2) in the corresponding embodiment will not be repeated here.

[0199] In order to objectively evaluate the prediction effect of the smooth data structure on the frequency of the next time window, it is compared with two machine learning models: linear regression model and time series model (differential integrated moving average autoregressive model). Specifically, the smooth data structure is used to screen out the most likely smooth elements and predict their frequencies through polynomial fitting. For other non-smooth elements, their frequencies in the upcoming window are directly predicted by two machine learning models. 300KB of memory is allocated for the smooth data structure, while as much memory as possible is allocated for the two machine learning models.

[0200] The experiment uses accuracy and running time as evaluation indicators. 1) Accuracy: the ratio of the number of accurate predictions to the number of predictions. For each element, if the deviation between the prediction result and the actual result is small, the prediction is classified as accurate; 2) Running time: the total running time to complete all predictions, which is used to measure the execution speed of the three schemes.

[0201] The experiment uses two open source datasets: a real dataset (IP Trace dataset) and a synthetic dataset (transaction dataset). For each of the above datasets, the first 300,000 elements are used and divided into 30 windows, each containing 10,000 elements. The results of the experimental evaluation are shown in Tables 1 and 2, indicating that smoothing the data structure can improve the prediction runtime by at least 100 times while still maintaining high accuracy.

[0202] Experimental results on the IP Trace dataset. As shown in Table 1, on this dataset, for the polynomial degree k = 0, 1, 2, the running time of the smooth data structure prediction is 530 times, 660 times, 670 times faster than that of the linear regression and time series respectively, and 23160 times, 1345 times, 10825 times faster. Among them, the accuracy of the smooth data structure exceeds 90% within 0.05 seconds, while the linear regression model takes nearly 30 seconds to achieve a similar accuracy. In addition, although the time series model predicts these elements more accurately on this dataset, it usually takes several minutes to get satisfactory results.

[0203] Table 1

[0204]

[0205] The experimental results of the transaction data set are shown in Table 2. On the transaction data set, for polynomial degree k = 0, 1, 2, the running time of the smooth data structure prediction is 152.9 times, 162.3 times, 142 times faster than that of linear regression and time series respectively, and 5071.4 times, 2892.3 times, 4786.7 times faster. Among them, the smooth data structure only takes 0.02 seconds to achieve 85% accuracy, while the linear regression model needs more than 2 seconds to achieve the same performance.

[0206] Table 2

[0207]

[0208] According to an embodiment of the present application, the frequency of an element in p consecutive windows can be stored through a smooth data structure, so that based on the first frequency of the element in p consecutive windows, it is possible to determine in real time and accurately whether the element is a smooth element, and further based on the smooth element, the trend of the data flow can be reflected.

[0209] Based on the same concept as the embodiment of the method of the present invention, the embodiment of the present invention also provides a device for determining the characteristics of a data stream. The device for determining the characteristics of a data stream includes several modules, each of which is used to execute each step in the method for identifying personal information provided in the embodiment of the present invention, and the division of the modules is not limited here. A person skilled in the art can clearly understand that in practical applications, the various steps in the method for determining the characteristics of a data stream provided in the embodiment of the present invention can be assigned to different modules as needed, that is, the internal structure of the device is divided into different modules to complete all or part of the functions described above. Each module in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more modules can be integrated into one unit, and the above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each module are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the module in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0210] For example, the data stream feature determination device is used to execute the data stream feature determination method provided by the embodiment of the present invention, Fig. 9 Schematic diagram of the structure of the data stream feature determination device provided in the embodiment of the present application. Fig. 9 As shown, the data stream feature determination device provided in the embodiment of the present application includes:

[0211] The receiving module 901 is used to receive a data stream, where the data stream includes elements;

[0212] A processing module 902 is used to map the element to a bucket of a second unit of the smooth data structure, and store a first frequency of the element in a current window in the bucket, where the window is used to divide the data stream;

[0213] A determination module 903 is used to determine a smoothing element from a plurality of elements according to the first frequencies of the plurality of elements in the p windows when the first frequencies of the plurality of elements in the p windows are stored in the bucket of the second unit;

[0214] Among them, the first frequency of the element in p windows is in the same row of the bucket, n = 0, 1, 2...p.

[0215] According to this solution, the frequency of an element in p consecutive windows can be stored through a smooth data structure, so that based on the first frequency of the element in p consecutive windows, it is possible to determine in real time and accurately whether the element is a smooth element, and further based on the smooth element, the trend of the data flow can be reflected.

[0216] In a possible implementation, the processing module is used to:

[0217] mapping the element to a first storage location of a first cell in the smoothed data structure;

[0218] Record the first frequency of the element in the consecutive s windows through the first storage unit;

[0219] According to the first frequency of the element in s windows, determine whether the element has the potential to become a smooth element;

[0220] In case the element has the potential to be a smooth element, the element is mapped into the bucket of the second unit of the smooth data structure.

[0221] In this way, the elements in the data stream that cannot be called smooth elements can be filtered out, thereby reducing the storage space occupied.

[0222] In a possible implementation, the processing module is used to:

[0223] The first frequency of the element in the first storage unit in the s windows is stored in the bucket where the element is located.

[0224] In a possible implementation, determining whether an element has the potential to become a smoothing element according to the first frequency of the element in s windows includes:

[0225] Fit the first frequency of the element in s windows to obtain the first function after fitting;

[0226] According to the first function, calculate the second frequency of the element in each of the s windows;

[0227] Whether the element has the potential to be a smooth element is determined based on the first frequency of the element in each of the s windows and the second frequency of the element in each of the s windows.

[0228] In this way, the elements in the data stream that cannot be called smooth elements can be filtered out, thereby reducing the storage space occupied.

[0229] In a possible implementation, the processing module is used to:

[0230] If the relationship between the first frequency of an element in each of the s windows and the second frequency of an element in each of the s windows satisfies the first formula, then the element has the potential to become a smooth element;

[0231] The first formula includes f(i)>0 and Wherein, f′(i) represents the first frequency of the element in the i-th window, f(i) represents the second frequency of the element in the i-th window, ε represents the root mean square of the error between the first frequency and the second frequency of the element in s consecutive windows, T represents the preset first threshold, i=0, 1, 2, ..., s.

[0232] In this way, the elements in the data stream that cannot be called smooth elements can be filtered out, thereby reducing the storage space occupied.

[0233] In a possible implementation, the first formula also includes μ≥G, Wherein, μ represents the potential of an element to become a smoothing element, G represents a preset second threshold, and Δ represents a value infinitely close to 0 but not equal to 0.

[0234] In this way, the elements in the data stream that cannot be called smooth elements can be filtered out, thereby reducing the storage space occupied.

[0235] In a possible implementation, the bucket of the second unit also stores the number of the target window where the target element starts to be smoothed;

[0236] When all the buckets in the second unit are occupied, the processing module is used to:

[0237] When all buckets in the second unit are occupied, the duration of smoothing of each target element is determined according to the number of the target window of the target element stored in the second unit and the number of the current window;

[0238] If the duration of the target element starting smoothing in the second unit meets the preset condition, the element is inserted into the bucket where the target element is located, and the information of the target element in the bucket is deleted.

[0239] In this way, the element that is most likely to become a smoothing element can be stored in the second unit, thereby reducing the occupation of storage space.

[0240] In a possible implementation, the determination module is used to:

[0241] Determine the second frequency of the element in the p windows according to the first frequency of the element in the p windows;

[0242] Determine whether the element is a smooth element based on the first frequency and the second frequency of the element in p windows.

[0243] In a possible implementation, the determination module is used to:

[0244] Fit the first frequency of the element in p windows to obtain the second function;

[0245] According to the second function, the second frequencies of the elements in the p windows are calculated.

[0246] In a possible implementation, the determination module is used to:

[0247] When the element satisfies the second formula, the element is determined to be a smooth element;

[0248] The second formula is f(n)>0 and Wherein, f′(n) represents the first frequency of the element in the nth window, f(n) represents the second frequency of the element in the nth window, ε′ represents the root mean square of the error between the first frequency and the second frequency of the element in p consecutive windows, T represents a preset first threshold, and n=0, 1, 2, ..., p.

[0249] Based on the same concept as the embodiment of the method of the present invention, the embodiment of the present invention further provides a computing device. The computing device is also a personal information identification device. The computing device can be a server or a terminal device. Among them, the terminal device can be a mobile phone, a tablet computer, a wearable device, a smart TV, a Huawei smart screen, a smart speaker, a car machine, etc.

[0250] Fig.10 It is a structural diagram of a computing device provided in an embodiment of the present application.

[0251] like Fig.10 As shown, computing device 1000 includes a processor 1001 , a memory 1002 , and a network interface 1003 .

[0252] The processor 1001 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0253] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0254] For example, the memory 1002 may store a computer program, and when the processor 1001 executes the computer program, the steps in the above-mentioned data stream feature determination method embodiment are implemented, for example Figure 7Steps 701 to 703 shown. Alternatively, the processor 1001 implements the functions of each module in the above-mentioned device embodiment when executing the computer program. Exemplarily, the computer program can be divided into one or more modules / units, and the one or more modules / units can be a series of computer program instruction segments that can perform specific functions. The one or more modules / units are stored in the memory 1002 and executed by the processor 1001 to complete the present invention. For example, the computer program can be divided into a receiving module, a processing module, and a determining module. The specific functions of each module are described above.

[0255] The network interface 1003 is used to send and receive data, for example, to send data processed by the processor 1001 to other computing devices, or to receive data sent by other computing devices.

[0256] Of course, to simplify, Fig.10 Only some of the components of the computing device 1000 related to the present application are shown, and components such as buses, input / output interfaces, etc. are omitted. In addition, according to specific application situations, the computing device 1000 may also include any other appropriate components. In addition, the computing device may be a desktop computer, a notebook, a handheld computer, a cloud server, and other computing devices. Those skilled in the art will understand that Fig.10 This is only an example of the computing device 1000 and does not constitute a limitation of the computing device. The computing device may include more or fewer components than shown in the figure, or may combine certain components, or may have different components. For example, the computing device may also include an input device, an output device, a network access device, a bus, etc. By way of example, the input device may be a microphone array, and may also include, for example, a keyboard, a mouse, etc. By way of example, the output device may output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, etc.

[0257] In addition to the above methods, apparatuses and computing devices, embodiments of the present application may also provide a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform the steps in the method for determining the characteristics of the data stream of various embodiments of the present application described in the "Method" section of this specification. Wherein, the computer program product can be written in any combination of one or more programming languages ​​to perform computer program codes for the operation of the embodiments of the present application, and the programming languages ​​include object-oriented programming languages, such as Java, C++, etc., and also include conventional procedural programming languages, such as "C" language or similar programming languages. Wherein, the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer program code can be executed completely on the user computing device, partially on the user device, as an independent software package, partially on the user computing device, partially on the remote computing device, or completely on the remote computing device or server.

[0258] In addition, the embodiment of the present application may also provide a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions, when executed by the processor, cause the processor to perform the steps in the display control method according to various embodiments of the present disclosure described in the above "method" section of this specification. The computer-readable storage medium may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of readable storage media (non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0259] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0260] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0261] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0262] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0263] The basic principles of the present application are described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, effects, etc. mentioned in the present application are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. are required by each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to the necessity of adopting the above specific details to be implemented.

[0264] The block diagrams of the devices, apparatuses, equipment, and systems involved in this disclosure are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, apparatuses, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including," "comprising," "having," and the like are open words, referring to "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or," and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0265] It should also be noted that in the apparatus, device and method of the present disclosure, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present disclosure.

[0266] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present disclosure to the forms disclosed herein. Although multiple example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

[0267] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

Claims

1. A method for determining characteristics of a data stream, It is characterized in that include: Receive a data stream, the data stream comprising elements, the data stream being divided into a plurality of sub-data streams, each of the plurality of sub-data streams corresponding to a window; Mapping the element to a bucket of a second unit of a smooth data structure, and storing a first frequency of the element in a current window in the bucket; In a case where the first frequencies of a plurality of the elements in p consecutive windows are stored in the bucket of the second unit, a smoothing element is determined from the plurality of the elements according to the first frequencies of the plurality of the elements in the p windows, and the p windows have the same size; The first frequencies of the elements in the p windows are in the same row of the bucket.

2. The method according to claim 1, It is characterized in that Mapping the element to a bucket of a second unit of a smooth data structure comprises: mapping the element to a first storage unit of a first unit in the smoothed data structure; Recording the first frequency of the element in s consecutive windows by the first storage unit; Determine whether the element has the potential to become a smoothing element according to the first frequency of the element in the s windows; In case the element has the potential to be a smooth element, the element is mapped into a bucket of a second unit of the smooth data structure.

3. The method according to claim 2, It is characterized in that The storing the first frequency of the element in the current window comprises: The first frequency of the element in the first storage unit in the s windows is stored in the bucket where the element is located.

4. The method according to claim 2, It is characterized in that The determining, according to the first frequency of the element in the s windows, whether the element has the potential to become a smoothing element comprises: Fitting the first frequency of the element in the s windows to obtain a first fitted function; Calculate, according to the first function, a second frequency of the element in each of the s windows; Whether the element has the potential to become a smoothing element is determined according to a first frequency of the element in each of the s windows and a second frequency of the element in each of the s windows.

5. The method according to claim 4, It is characterized in that The determining, according to the first frequency of the element in each of the s windows and the second frequency of each of the s windows, whether the element has the potential to become a smoothing element comprises: If the relationship between the first frequency of the element in each of the s windows and the second frequency of each of the s windows satisfies the first formula, then the element has the potential to become a smoothing element; The first formula includes f(i)>0 and Wherein, f′(i) represents the first frequency of the element in the i-th window, f(i) represents the second frequency of the element in the i-th window, ε represents the root mean square of the error between the first frequency and the second frequency of the element in s consecutive windows, T represents a preset first threshold, i=0, 1, 2, ..., s.

6. The method according to claim 5, It is characterized in that The first formula also includes μ≥G, Wherein, μ represents the potential of the element to become a smoothing element, G represents a preset second threshold, and Δ represents a value infinitely close to 0 but not equal to 0.

7. The method according to any one of claims 2 to 6, It is characterized in that The bucket of the second unit also stores the number of the target window where the target element starts smoothing; In a case where all buckets in the second unit are occupied, mapping the element to the bucket of the second unit of the smooth data structure comprises: When all buckets in the second unit are occupied, determining the duration of smoothing of each target element according to the number of the target window of the target element stored in the second unit and the number of the current window; If the duration of the target element starting smoothing in the second unit satisfies a preset condition, the element is inserted into the bucket where the target element is located, and the information of the target element in the bucket is deleted.

8. The method according to any one of claims 1 to 6, It is characterized in that Determining a smoothing element from the plurality of elements according to the first frequencies of the plurality of elements in the p windows, comprising: Determine a second frequency of the element in the p windows according to a first frequency of the element in the p windows; It is determined whether the element is the smoothing element according to the first frequency and the second frequency of the element in the p windows.

9. The method according to claim 8, It is characterized in that Determining the second frequency of the element in the p windows according to the first frequency of the element in the p windows includes: Fitting the first frequency of the element in the p windows to obtain a second function; According to the second function, second frequencies corresponding to the element in each of the p windows are calculated.

10. The method according to claim 8, It is characterized in that The determining, according to the first frequency and the second frequency of the element in the p windows, whether the element is the smooth element comprises: In the case where the element satisfies the second formula, determining that the element is the smooth element; Wherein, the second formula is f(n)>0 and Wherein, f′(n) represents the first frequency of the element in the nth window, f(n) represents the second frequency of the element in the nth window, ε′ represents the root mean square of the error between the first frequency and the second frequency of the element in p consecutive windows, T represents a preset first threshold, and n=0, 1, 2, ..., p.

11. A device for determining characteristics of a data stream, It is characterized in that include: A receiving module, configured to receive a data stream, wherein the data stream includes elements, the data stream is divided into a plurality of sub-data streams, and each of the plurality of sub-data streams corresponds to a window; a processing module, configured to map the element to a bucket of a second unit of a smooth data structure, and store a first frequency of the element in a current window in the bucket; a determination module, configured to determine a smoothing element from a plurality of the elements according to the first frequencies of the plurality of the elements in the p windows, when the first frequencies of the plurality of the elements in the p windows are stored in the bucket of the second unit; The first frequencies of the elements in the p windows are in the same row of the bucket.

12. A computing device, It is characterized in that include: at least one memory for storing a program; At least one processor is used to execute the program stored in the memory. When the program stored in the memory is executed, the processor is used to execute the method according to any one of claims 1 to 10.

13. A device for determining characteristics of a data stream, It is characterized in that The device runs computer program instructions to perform the method according to any one of claims 1-10.

14. A computer storage medium, wherein instructions are stored in the computer storage medium, and when the instructions are executed on a computer, the computer executes the method according to any one of claims 1 to 10.

15. A computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 10.