Traffic feature detection method, device and system based on time series

Abstract

The present invention relates to the field of computer technology, and specifically provides a method, device and system for detecting traffic characteristics based on time series. The method includes: at the start time of the current time window, inserting a sample element into a data structure, wherein the data structure includes a state tracker, a state converter and a filter; at the end time of the current time window, checking the basic mode of the sample element in the state tracker, and checking the current state of the sample element in the state converter; obtaining the output state through state conversion, and reporting according to the output state, and the report result is used to characterize the change of traffic characteristics. The invention creates a combination conversion of the target mode change of the element abstracted into the current state and the basic mode input, and has strong versatility. The invention only focuses on the current basic mode and the current state that may trigger the state conversion, which saves more memory than the existing detection methods.

Description

Technical Field

[0001] The invention relates to the field of computer technology, and in particular to a flow characteristic detection method, device and system based on time series. Background Art

[0002] In a high-speed network environment, traffic detection is the basis for understanding network operation status and monitoring abnormal network behavior. It has important applications in key network functions such as load balancing, congestion control and routing selection.

[0003] Traffic characteristics are various statistical information of network flows, which are used to reflect the current state of network communication traffic in high-speed networks. Traffic characteristics based on time series are one type of traffic characteristics, which involve the continuity of traffic and focus on the changes of traffic in the time dimension. Traffic characteristics based on time series help to understand the dynamic changes of network traffic and provide important references for network optimization and management.

[0004] Existing research on the detection of time-series-based traffic characteristics requires recording complete element arrival rate data and judging pattern changes based on arrival rate changes. The counter occupies a lot of storage space on the switch chip, the overall throughput of the data structure is insufficient, and the measurement accuracy is low. Summary of the invention

[0005] The embodiments of the present invention provide a time series-based traffic feature detection method, device and system, which focus on the current state that can trigger state transition, achieve efficient detection and reduce memory usage.

[0006] In a first aspect, the present invention provides a method for detecting flow characteristics based on time series, which includes:

[0007] At the start time of the current time window, the sample element is inserted into a data structure, wherein the data structure includes a state tracker and a state converter, and the state tracker is further provided with a filter, and the filter is used to forward the sample element that needs attention to the state tracker; the insertion position of the sample element is the filter or the state tracker, and the insertion position is determined according to the storage content of the filter and the state tracker at the end time of the previous time window;

[0008] At the end of the current time window, check the sample elements in the state tracker one by one, and match the sample elements with the basic pattern with the storage content of the state converter;

[0009] If the sample element is stored in the state converter, the current state of the sample element in the state converter is obtained, and the state conversion is performed to obtain the output state, and a report is made based on the result of the output state, wherein the reported result is used to characterize the flow characteristic change of the sample element.

[0010] The time series-based traffic feature detection method provided by the embodiment of the present invention, before the start time of the current time window, the method further includes:

[0011] Creating a filter, wherein the filter comprises a plurality of filter buckets and a plurality of first hash functions; a filter unit is set in each filter bucket of the filter, and each filter unit comprises a first element field and a storage frequency associated with the first element field;

[0012] Creating a state tracker, wherein the state tracker is a tracking bucket array with a second hash function; each tracking bucket in the tracking bucket array is provided with a plurality of tracking units, each of the tracking units includes a second element field, and a previous storage frequency counter and a current storage frequency counter associated with the second element field;

[0013] A state converter is created, wherein the state converter is a conversion bucket array with a third hash function; each conversion bucket in the conversion bucket array is provided with a plurality of conversion units, each of the conversion units includes a third element field, and a current state, a state update flag and a state frequency counter associated with the third element field, and the state update flag is used to mark whether the state of the third element field in the current time window is updated.

[0014] The time series-based traffic feature detection method provided by the embodiment of the present invention inserts the sample element into the data structure, including:

[0015] Inputting the sample element into a second hash function to obtain a second hash value;

[0016] If the sample element is not stored in the tracking bucket to which the second hash value is mapped, increasing the count result of the current storage frequency counter associated with the sample element in the state tracker by 1;

[0017] If the sample element has been stored in the tracking bucket to which the second hash value is mapped, the sample element is inserted into the filter.

[0018] The method for detecting traffic characteristics based on time series provided by an embodiment of the present invention inserts the sample element into the filter, comprising:

[0019] The sample elements are respectively input into a plurality of first hash functions, and mapped to obtain a plurality of mapping buckets in the filter;

[0020] Matching the sample elements with multiple mapping buckets one by one;

[0021] If the sample element is stored in the mapping bucket, the storage frequency associated with the sample element in the mapping bucket is increased by 1;

[0022] If the sample element is not stored in the mapping bucket and an irrelevant element is stored, the storage frequency associated with the irrelevant element is reduced by 1; when the storage frequency of the irrelevant element is reduced to 0, the mapping bucket is cleared;

[0023] If the mapping bucket is an empty bucket, or the irrelevant element is cleared, the sample element is inserted and the storage frequency associated with the sample element is set to 1.

[0024] The present invention provides a method for detecting flow characteristics based on time series, the method further comprising:

[0025] In the filter, if the storage frequency of the sample element is greater than or equal to a potential mode threshold, the sample element is forwarded by the filter and inserted into the state tracker; wherein the potential mode threshold is preset by the data structure.

[0026] The time series-based traffic feature detection method provided by the embodiment of the present invention matches the sample elements having the basic pattern with the storage content of the state converter, including:

[0027] Inputting the sample element into a third hash function to obtain a third hash value;

[0028] If the sample element is not stored in the conversion bucket to which the third hash value is mapped, insert the sample element into an empty cell of the state converter, evict an element according to the state priority setting when there is no empty cell, and then insert the sample element into the state converter; after the insertion, assign the basic mode to the sample element as the current state of the sample element;

[0029] If the sample element is already stored in the conversion bucket to which the third hash value is mapped, the state conversion is completed in the state converter to obtain the output state, and the state update flag of the sample element is marked as updated.

[0030] The flow characteristic detection method based on time series provided by the embodiment of the present invention performs state conversion to obtain an output state, and reports according to the result of the output state include:

[0031] Input the basic mode of the sample element and the current state of the sample element in the state converter, perform state conversion to obtain an output state;

[0032] Matching the output state with the current state of the sample element;

[0033] If the match is consistent, the count result of the state frequency counter associated with the sample element in the state converter is increased by 1;

[0034] If the match fails and the output state is an accepting state, target pattern recognition is performed and reported.

[0035] The time series-based traffic feature detection method provided by the embodiment of the present invention, after updating the storage content in the state converter according to the matching result, before the next time window starts, the method further includes:

[0036] Clearing the first element field and the storage frequency associated with the first element field;

[0037] expelling the second element field in the state tracker, for which the counting results of the previous storage frequency counter and the current storage frequency counter are both smaller than the potential mode threshold;

[0038] Evicting the second element field whose count result of the current storage frequency counter in the state tracker is 0;

[0039] expelling the third element field whose counting result of the state frequency counter exceeds a duration threshold, wherein the duration threshold is preset in the data structure;

[0040] expelling the third element field marked as not updated by the status update identifier at the end of the current time window;

[0041] The status update flag associated with the third element field in the conversion bucket is reset from updated to not updated.

[0042] In a second aspect, the present invention provides an electronic device, comprising: a processor, and a memory for storing a program, wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes the time series-based traffic feature detection method described in any of the above embodiments.

[0043] In a third aspect, the present invention provides an embodiment of a flow feature detection system based on a time series, and the flow feature detection system based on a time series includes the flow feature detection method based on a time series described in any of the above embodiments.

[0044] The time series-based traffic feature detection method, device and system provided by the embodiments of the present invention divide the arrival process of data elements into multiple time windows, and abstract the target mode change of the element into a combination of the current state and the basic mode input that meets the conversion conditions, so that the detection process can adapt to different modes, and the detection framework is highly versatile. There is no need to fully record the changes in the arrival rate of the elements, and only focus on the current state that may cause the state transition, which significantly saves memory compared to the existing traffic detection methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other embodiments can be obtained based on these drawings without creative work.

[0046] Figure 1 It is a flow chart of the initialization module of the embodiment created by the present invention.

[0047] Figure 2 It is a flow chart of the insertion module of an embodiment created by the present invention.

[0048] Figure 3 It is a flow chart of the detection module of an embodiment created by the present invention.

[0049] Figure 4 It is a flow chart of the update module of the embodiment created by the present invention. DETAILED DESCRIPTION

[0050] The embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided to provide a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0051] The object of the traffic detection task is mainly to obtain traffic characteristics that are of certain value to network data. Through these valuable traffic characteristics, the operating status of the network infrastructure can be grasped, and then the necessary decision-making basis can be provided for diversified network management needs.

[0052] In traffic detection tasks, a network flow is usually modeled as a set of data packets, where each flow is uniquely identified by a flow label, and each data packet in each flow is modeled as a tuple consisting of a flow label and an element label. For example, a flow of data packets sent to the same host is a destination address flow.

[0053] Traffic characteristics include flow size, flow cardinality, and quantiles. Flow size refers to the number of packets contained in a network flow or the sum of the number of bytes carried by packets. Flow size has a very good effect in load balancing, elephant flow detection, and routing redirection. Flow cardinality refers to the number of different elements contained in a network flow, which is of great significance in the detection of abnormal traffic behavior.

[0054] Time series-based traffic feature detection focuses on the changes in traffic in the time dimension, including traffic peaks, fluctuation patterns, and periodicity; these features can reveal the laws of network usage and potential anomalies, help formulate more effective traffic management strategies, and have a wide range of application scenarios. Time series features are essentially a combination of flow size / cardinality features and flow continuity features. In other words, time series features are changes in size / cardinality features of network flows based on continuity features.

[0055] On the one hand, the existing traffic feature detection method needs to set up counters or registers in the switch chip space, complete the storage and recording of traffic information in each cycle, and determine whether it meets the timing feature pattern and report it at the end of each cycle based on the recorded information. The memory usage of the data structure is too large. The allocated space of the switch chip storage space for the traffic measurement module is less than 1MB, which limits the operation of the traffic feature detection method.

[0056] On the other hand, real-time measurement of traffic characteristics requires that the packet processing speed of the deployed measurement structure matches the packet forwarding speed of the link. The throughput capacity of the data structure in the existing traffic characteristic detection method cannot bear the throughput requirements of the high-speed on-chip packet forwarding speed.

[0057] In addition, due to the associated time series, the detection of traffic features has high accuracy requirements. Since the purpose of detecting time series features is to detect abnormal traffic changes in the network, and the anomaly needs to be defined based on the actual network environment, most existing solutions are to discover new time series features based on the actual situation, propose a new detection task, and design a corresponding detection solution. The above limitations make the existing data structure lack of universality.

[0058] Due to the huge amount of data, research on finding time series patterns in data streams usually designs simplified data structures (sketch) based on the specific characteristics of the patterns so that these patterns can be recorded and queried in real time. Among them, bursts and across-period bursts are common patterns in data streams. For these patterns, the sudden increase in the arrival rate of data elements between consecutive time periods is considered as the definition.

[0059] The invention proposes a general model for defining patterns, namely, Pattern Automaton. Pattern Automaton can cover more abnormal changes in data streams. The generation of data streams is manifested as a sequence of data items that arrive dynamically and continuously in order (usually with timestamps).

[0060] The data stream is divided into fixed-width time windows. The arrival rate standard is set for each time window to determine whether the data element presents a time series pattern. The arrival rate standard is set based on actual abnormal events. By observing each pattern, it can be found that the key to determining whether the pattern is established lies in the increase or decrease of the arrival rate in each two adjacent time windows.

[0061] It should be noted that the time series is divided into multiple time windows, and each time window obtained after the division has its own start time and end time. The time series before the current time window W_i is called the previous time window or the past time window; wherein, the previous time window W_{i-1} is relative to the current time window W_i, and a time window occurs before it. The end time of the previous time window W_{i-1} is the start time of the current time window W_i. The next time window or the next time window W_{i+1} is relative to the current time window W_i, and a time window occurs after it. The start time of the next time window W_{i+1} is the end time of the current time window W_i. In an embodiment of the present invention, the traffic characteristics in the current time window W_i are taken as an example, and the arrival rate changes of the current time window W_i and the previous time window W_{i-1} are considered for detection.

[0062] The core idea of ​​Pattern Automaton is based on Finite State Automaton. The changes of data elements in each time window are abstracted into basic patterns as input. According to the pattern, it switches between different states and can cover multiple types of patterns.

[0063] The invention also provides an efficient, fast and universal framework for simultaneously discovering multiple time series patterns. In this framework, there are two important parts: basic mode collection (Mode Collection) and state transition (State Transition). Based on the mode automaton, each mode can be mapped to a sequence of modes captured by each time window. Basic mode collection is used to track the changes of data elements in the current time window. By comparing the conditions of the basic mode, the current state that meets the transition conditions is reported to the state transition. The new mode can be expanded and customized according to user needs, making the framework more flexible and easy to deploy. State transition completes state switching after each time window and detects various time series patterns in the data stream.

[0064] Based on the pattern automaton and the universal framework, the embodiment of the present invention provides a flow feature detection method based on time series, which can detect a variety of flow features based on time series, determine whether the element generates a new pattern and report it, and use the report result to characterize the change of flow features in the time series. The universal framework can reduce the memory usage, only focus on the current state that meets the conversion conditions, and overcome the limitation of the switch chip space. At the same time, the present application performs high-precision detection of flow features based on the consideration of time series.

[0065] The time series-based traffic feature detection method provided in the present application is executed through four modules: initialization module, insertion module, detection module and update module. Among them, the initialization module is used to create a data structure. At the start time of the time window, the insertion module selects and inserts the position of the incoming elements. After completing the element insertion, at the end time of the time window, the update module updates and resets the state of each element in the data structure. When receiving the detection demand, the detection module obtains the state update of the element through traversal inspection at the end time of the time window, detects whether the element can produce a certain pattern, and further outputs the report result to characterize the traffic change.

[0066] Reference Figures 1 to 4 As shown, the flow feature detection method based on time series includes the following steps:

[0067] Before the start time of the current time window W_i, a data structure is created in the initialization module.

[0068] The data structure consists of three parts, including a state tracker, a state converter, and a filter. For elements already in the state tracker, they can be directly inserted into the state tracker for state tracking when the data structure is passed in; since some elements are not stored in the state tracker, the filter acts as a cache to provide an insertion location for these elements and forward the elements that need attention to the state tracker. At the end of the time window, pay attention to whether the elements in the state tracker meet the state transition conditions, and match and compare with the state converter to obtain the traffic feature detection results.

[0069] Reference Figure 1 As shown, creating a data structure specifically includes:

[0070] Create a filter, wherein the filter includes multiple filter buckets, each filter bucket of the filter is set with a filter unit, each filter unit includes a first element field, that is, an "item", which is identified by an "ID" or "key"; and also includes a storage frequency associated with the first element field. In addition, the filter is also associated with d first hash functions , , … .

[0071] Create a state tracker, where the state tracker is a state tracker with a second hash function The tracking bucket array PB[ ]; Tracking bucket array PB[ Each tracking bucket in ] is provided with a plurality of tracking units, each tracking unit includes a second element field, and the second element field has the same property as the first element field; each tracking unit also includes a pre-order storage frequency counter C associated with the second element field pre and the current storage frequency counter C cur . Preamble storage frequency counter C pre The counting result in is the storage frequency of the second element field in the previous time window W_{i-1}. The current storage frequency counter C cur The counting result within is the storage frequency of the second element field in the current time window W_i.

[0072] Create a state converter, where the state converter is a state converter with a third hash function The conversion bucket array PA[ ]; Convert bucket array PA[ Each conversion bucket in the ] is provided with a plurality of conversion units, each conversion unit includes a third element field, and the third element field has the same properties as the first element field and the second element field; each conversion unit also includes a current state S, a state update identifier N and a state frequency counter S associated with the third element field num The state update flag N is used to mark whether the state of the third element field in the current time window W_i is updated. In the embodiment of the present application, when the state update flag N is true, it indicates that the state of the third element field in the conversion unit in the current time window W_i is updated. When the state update flag N is false, it indicates that the state is not updated. State frequency counter S num It is used to record the number of occurrences of the current state. In addition, a state transition table is pre-established in the state converter.

[0073] At the start time of the current time window W_i, the insertion of elements is performed in the insertion module, and the elements are inserted into the data structure created during initialization.

[0074] In the embodiment of the present application, the element inserted in the current time window W_i is set as the sample element e. The insertion position of the sample element e is the filter or the state tracker, and the insertion position is determined according to the storage content of the filter and the state tracker at the end time of the previous time window W_{i-1}.

[0075] Specifically, refer to Figure 2 As shown, inserting the sample element e into the data structure includes:

[0076] Try to insert the sample element e into the state tracker: Input the sample element e into the second hash function Get the second hash value .

[0077] If the second hash value Mapped to the tracking bucket PB[ ] has stored the sample element e, and only needs to store the current storage frequency counter eC associated with the sample element e in the state tracker cur The count result increases by 1.

[0078] If the second hash value Mapped to the tracking bucket PB[ ], the attempt to insert the sample element e into the state tracker fails. In this case, the sample element e is inserted into the filter.

[0079] Next, inserting the sample element e into the filter includes: inputting the sample element e into the d first hash functions of the filter respectively , , … , locate the d mapping buckets of the sample element e in the filter, match the sample element e with the d mapping buckets one by one, and perform the following matching judgment for each mapping bucket.

[0080] If the mapping bucket stores the sample element e, it is considered that the first element field has stored the sample element e. In this case, it is only necessary to increase the storage frequency associated with the sample element e in the mapping bucket by 1.

[0081] If the mapping bucket does not store the sample element e and stores an irrelevant element, that is, the first element field of the filtering unit is an element other than the sample element e, the storage frequency associated with the irrelevant element is reduced by 1.

[0082] It should be noted that in this application, only one unit is set for each bucket of the filter and a first element field is stored. Therefore, when the storage frequency of the irrelevant element is reduced to 0, the mapping bucket is cleared. At this time, the sample element e is inserted into the cleared mapping bucket as the first element field, and its associated storage frequency is set to 1.

[0083] If the sample element e is not stored in the mapping bucket, and the mapping bucket is an empty bucket and does not store irrelevant elements, the sample element e is inserted into the emptied mapping bucket as the first element field, and its associated storage frequency is set to 1.

[0084] After completing the insertion of the above sample element e, the elements that need to be paid attention to in the filter must be forwarded to the state tracker, including: the storage frequency of the sample element e in the filter and the potential mode threshold Comparison, where the potential mode threshold It is preset when creating the data structure.

[0085] If the storage frequency of sample element e is less than the potential mode threshold , it is considered that the sample element e stored in the filter does not have the conditions to trigger the potential mode, and the sample element e is still stored in the filter unchanged.

[0086] If the storage frequency of sample element e is greater than or equal to the potential mode threshold , it is considered that the sample element e stored in the filter needs attention and state tracking. At this time, the sample element e is forwarded by the filter and inserted into the state tracker.

[0087] Next, forwarding the sample element e from the filter and inserting it into the state tracker includes:

[0088] If the tracking bucket PB[ ] can find an empty cell, then insert the sample element e and the storage frequency associated with the sample element e into the empty cell. The sample element e is now stored as the second element field, and the storage frequency originally associated with the sample element e in the filter is now used as the current storage frequency counter eC cur The count result is inserted into the state tracker.

[0089] If the tracking bucket PB[ ] fails to find an empty cell, then evict the tracking bucket PB[ The frequency counter C currently stored in each tracking unit of cur The second element field with the smallest count result is expelled and the sample element e and the storage frequency associated with the sample element e are inserted. The storage frequency originally associated with the sample element e in the filter is now used as the current storage frequency counter eC cur The count result is inserted into the state tracker.

[0090] After the element is inserted, at the end of the current time window W_i, refer to Figure 3 As shown, pattern detection is performed in the detection module.

[0091] For the sample element e stored in the state tracker, a traversal check is performed to detect whether the sample element e satisfies a certain basic pattern.

[0092] Specifically, for a sample element e, the frequency counter eC is currently stored in the state converter cur The counting result is f_i, and the pre-order storage frequency counter C pre The counting result is f_{i-1}. If one of the following two conditions is met, a basic mode mode of the sample element e can be identified in the current time window W_i.

[0093] f_{i-1}≥T and f_i≥T, and f_i≥α·f_{i-1};

[0094] f_{i-1}≥T and f_i≥T, and f_{i-1}≥β·f_i.

[0095] Among them, α and β are user-defined parameters. T is the basic mode threshold.

[0096] Furthermore, the combined pattern is a sequence of specific basic patterns captured in each time window. For a sample element e, if there are t consecutive time windows W_i, W_{i+1}, ... W_{i+t-1}, and each time window can observe a specific basic pattern mode_1(e), mode_2(e), ... mode_t(e), combining them can obtain a target pattern pattern(e) of the sample element e.

[0097] pattern(e)=<mode_1(e),mode_2(e),...,mode_t(e)> .

[0098] The present invention incorporates the concept of Finite-State Automaton (FSA) into the detection of target patterns. The entire detection process is divided into a series of key states, and the transitions of these states are triggered in sequence by specific changes in the arrival rate, thereby realizing the step-by-step detection of the target pattern. Although the arrival process of the sample element may experience multiple transitional changes, the overall trend of its arrival rate is mainly affected by a few key changes.

[0099] For example, the detection of burst patterns is mainly triggered by a “significant increase” and confirmed by a subsequent “significant decrease”. Therefore, the essence of target pattern detection is a process of gradually identifying these specific changes in the arrival rate of elements.

[0100] The present invention abstracts the change of arrival rate in two adjacent time windows into a predefined basic pattern, and then converts the numerical change of the element in the entire arrival process into the change of the basic pattern. Since the arrival pattern is essentially composed of several key basic patterns, and these basic patterns reflect the increase or decrease of the arrival rate in a specific order, the recognition of these key basic patterns can be abstracted into states, and these basic patterns are used as conditions for state transition to drive the state change of pattern detection. Whenever a new basic pattern is updated, according to the matching result with the transition condition, the process of target pattern detection will advance in a possible direction, or remain unchanged. Through the transfer of states, the recognition of the target pattern of the sample element e can be gradually completed.

[0101] The present invention converts the arrival rate of the sample element e into a change sequence of the basic pattern according to the continuous change. The pattern automaton starts from the initial state and switches its current state according to the matching result of the new basic pattern and the state transition condition. When the input basic pattern meets the transition condition, the automaton switches to the new state and gradually completes the recognition of the target pattern.

[0102] The basic input patterns can reflect frequency changes and specific trends, describe the behavior of data elements, and more effectively classify and analyze frequency changes in data streams. In the embodiment of the present invention, the basic input patterns of the finite state automaton include the following five patterns.

[0103] Sharp Increase (SI). For sample element e, if \(f_{i - 1}\geq T\), \(f_i\geq T\) and \(f_i\geq s\cdot f_{i - 1}(s > 1)\), where s is a user-defined parameter, it indicates a sharp increase. The basic pattern of a sharp increase shows that the frequency of sample element e rises sharply within a short period, usually a sign of a sudden event or abnormal behavior.

[0104] Sharp Decrease (SD). For sample element e, if \(f_{i - 1}\geq T\), \(f_i\geq T\) and \(f_i\leq q\cdot f_{i - 1}(q < 1)\), where q is a user-defined parameter, it indicates a sharp decrease. The pattern of a sharp decrease shows that the frequency of sample element e drops suddenly, such as a sharp reduction in the demand for a certain product in the market.

[0105] Gradual Increase (GI). For sample element e, if \(f_{i - 1}\geq T\), \(f_i\geq T\) and \(f_i\geq g\cdot f_{i - 1}(1 < g < s)\), where g is a user-defined parameter, it indicates a gradual increase. Different from a sharp increase, a gradual increase reflects a slower growth trend, which may indicate the steady development of a business or the continuous popularity of a certain product.

[0106] Gradual Decrease (GD). For sample element e, if \(f_{i - 1}\geq T\), \(f_i\geq T\) and \(f_i\leq p\cdot f_{i - 1}(q < p < 1)\), where p is a user-defined parameter, it indicates a gradual decrease. The pattern of a gradual decrease represents the gradual decline of a certain item or activity, such as the natural decrease in the demand for seasonal goods.

[0107] Stable (ST). For sample element e, if none of the above basic patterns are met, it indicates stability. This means that the frequency of sample element e remains basically unchanged within the observation time window.

[0108] In addition, the basic patterns input into the finite state automaton can also be other basic patterns that can reflect data changes, not limited to this.

[0109] At the end of the current time window \(W_i\), check each sample element e in the state tracker one by one. If the frequency change of sample element e in the state tracker does not meet the conditions of the basic pattern mode, then sample element e is temporarily retained in the state tracker.

[0110] If the frequency change of sample element e can be classified into one of the above five basic patterns, then insert the sample element e with any basic pattern mode into the state converter. The insertion process includes:

[0111] Input sample element e into the third hash function Get the third hash value .

[0112] If the third hash value Mapped to the conversion bucket PA[ ] does not store the sample element e, then insert the sample element e into the empty cell of the state converter, and when there is no empty cell, evict an element according to the state priority setting, and then insert the sample element e into the state converter; after the insertion, assign the basic mode mode of the sample element e to the sample element e as the current state eS of e. The state update flag eN of the sample element e is marked as updated, that is, eN is set to true. The state frequency counter eS associated with the sample element e num The count result is set to 1.

[0113] If the third hash value Mapped to the conversion bucket PA[ ] If the sample element e has been stored, the state update flag eN of the sample element e is marked as updated in the state converter, that is, eN is set to true, indicating that the mode state of the sample element e has been updated in the current time window W_i. Then the state conversion of the sample element e is performed, specifically including:

[0114] The basic mode mode of the sample element e and the current state eS are input into the state transition table P, and the state transition is performed to obtain the output state P[eS, mode].

[0115] Among them, the input basic mode mode is the frequency change classification of the sample element e at the end of the current time window W_i, which is specifically one of a sudden increase, a sudden decrease, a gradual increase, a gradual decrease and a stable state in this embodiment. The current state eS that meets the state transition condition is specifically one of a sudden increase, a sudden decrease, a gradual increase, a gradual decrease, a stable, a pre-burst, an Eruption, a Burst, a disturbance, and a plunge. As shown in Table 1, the state transition table is implemented by a two-dimensional array. In the table, Burst is a defined target mode, Eruption and Plunge are newly defined target modes, all of which are accepting states, and the results of other output states are intermediate states, and blank spaces are illegal modes.

[0116] Table 1

[0117]

[0118] Further, reporting based on the result of the output status includes:

[0119] Match the output state P[eS, mode] with the current state eS of the sample element e;

[0120] If eS is equal to P[eS, mode], the match is consistent, and the state frequency counter eS associated with the sample element e in the state converter is set to num The count result increases by 1.

[0121] For example, if the basic mode mode of the sample element is stable ST, the current state eS obtained by traversal check is stable Stable, and the output state P[eS, mode] obtained by conversion is still stable. At this time, the state frequency counter eS associated with the sample element e is num The count result increases by 1.

[0122] If the current state eS fails to match the output state P[eS, mode], and the output state P[eS, mode] is an accepting state, the target mode is recognized and reported, and the reported result is used to characterize the flow characteristic changes of the sample element e.

[0123] For example, the basic mode mode of the sample element is a sudden decrease SD, the current state eS obtained by traversal inspection is a sudden decrease Sudden decrease, the output state P[eS, mode] obtained by conversion is a plunge, Plunge is not equal to a sudden decrease Sudden decrease, and Plunge is an accepting state, then it is reported.

[0124] If the current state eS does not match the output state P[eS, mode] and the output state P[eS, mode] is an intermediate state, then update the current state eS to the output state P[eS, mode] and set the associated state frequency counter eS num The count result is set to 1.

[0125] For example, if the basic mode mode of the sample element is sudden increase SI, the current state eS obtained by traversal inspection is gradual increase Gradual increase, and the output state P[eS, mode] obtained by conversion is sudden increase Sudden increase. Sudden increase is not equal to gradual increase, and Sudden increase is an intermediate state, then the current state eS is updated to sudden increase Sudden increase, and the associated state frequency counter eS numThe count is set to 1.

[0126] It should be noted that the accepting state represents the final state reached by the element after multiple time windows. The intermediate state represents the transition stage between the initial state and the accepting state, capturing the dynamic changes in the element frequency. For example, depending on the input, the element will transform into the corresponding five intermediate states: sharp increase, sharp decrease, gradual increase, gradual decrease, and stable. When an element is currently in a sharp increase state, but the input is not sharply reduced, it cannot directly transform into the accepting state (burst state), but enters the pre-burst intermediate state.

[0127] If the unrecorded mode P[eS, mode] is an acceptance state, it means that the mode has been identified and defined. At this time, the acceptance state is reported to characterize the traffic change.

[0128] After updating the storage content in the state converter and / or completing the report, before the next time window W_{i+1} starts, the detection method provided by the present application also includes: referring to Figure 4 As shown, the data structure is updated and reset in the update module to initialize the traffic feature detection in the next time window W_{i+1} or subsequent time series.

[0129] At the end of each time window, the data structure is updated and reset including:

[0130] Update the filter; at the end of each time window, the filter bucket is reset, the stored first element field and the stored frequency associated with the first element field are cleared, and preparation is made for the insertion of elements in the next time window W_{i+1}.

[0131] Update the state tracker; first, in the state tracker, the pre-order storage frequency counter C pre and the current storage frequency counter C cur The count results are all less than the potential pattern threshold The second element field of is expelled and its state is no longer tracked. Second, for the frequency counter C currently stored in the state tracker cur The second element field whose count result is 0 is evicted.

[0132] Update the state converter; Update the state converter including data sorting and state reset, wherein data sorting is performed for each third element field stored, specifically according to its state frequency counter S numThe counting result and state update flag N are updated. First, the third element field whose counting result of the state frequency counter exceeds the duration threshold is expelled, where the duration threshold is preset in the data structure. Second, the third element field marked as not updated by the state update flag N at the end of the current time window W_i is also expelled. The fact that the state update flag N is not updated indicates that the element is not inserted in the current time window W_i, but in the past time window, and does not appear in the current time window W_i, so it is expelled.

[0133] Next, the state reset is to reset the state update flag N from true to false for the third element field marked as updated by the state update flag N. These elements are inserted for the first time in the current time window W_i, or have been stored in the state converter in the past time window and have a state update in the current time window W_i, so they continue to be retained in the state converter and the state update flag N is reset.

[0134] An embodiment of the present invention also provides a time series-based traffic feature detection system, which includes a computer program, wherein the computer program, when executed by a computer processor, is used to enable the computer to execute the time series-based traffic feature detection method of an embodiment of the present invention.

[0135] An embodiment of the present invention also provides a non-transitory machine-readable medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to enable the computer to execute the time series-based traffic feature detection method of an embodiment of the present invention.

[0136] The embodiment of the invention also provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor. The memory stores a computer program that can be executed by the at least one processor, and when the at least one processor executes the computer program, the electronic device executes the flow feature detection method based on time series of the embodiment of the invention.

[0137] Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the inventions described and / or claimed herein.

[0138] The electronic device includes a computing unit, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) or a computer program loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for the operation of the electronic device can also be stored. The computing unit, ROM, and RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0139] Multiple components in the electronic device are connected to the I / O interface, including: an input unit, an output unit, a storage unit, and a communication unit. The input unit can be any type of device that can input information to the electronic device, and the input unit can receive input digital or character information, and generate key signal input related to user settings and / or function control of the electronic device. The output unit can be any type of device that can present information, and can include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit can include but is not limited to a disk, an optical disk. The communication unit allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and can include but is not limited to a modem, a network card, an infrared communication device, and / or a wireless communication transceiver, such as a Bluetooth device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0140] The computing unit may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of computing units include, but are not limited to, CPUs, graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing units, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit performs the various methods and processes described above. For example, in some embodiments, the method embodiments created by the present invention may be implemented as a computer program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on an electronic device via a ROM and / or a communication unit. In some embodiments, the computing unit may be configured to perform the above method in any other appropriate manner (e.g., by means of firmware).

[0141] The computer program for implementing the method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the computer program, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The computer program can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or apparatus. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable signal medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, or infrared systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] It should be noted that the term "including" and its variations used in the embodiments of the present invention are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". The modifications of "one" and "multiple" mentioned in the embodiments of the present invention are illustrative and not restrictive. Those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0144] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0145] The various steps described in the method implementation methods provided by the embodiments of the present invention can be performed in different orders and / or in parallel. In addition, the method implementation methods may include additional steps and / or omit the steps shown. The scope of protection of the present invention is not limited in this respect.

[0146] The term "embodiment" in this specification refers to specific features, structures or characteristics described in conjunction with the embodiment that can be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. The various embodiments in this specification are described in a related manner, and the same and similar parts between the various embodiments refer to each other. In particular, for the device, equipment, and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment.

[0147] The above-described embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of protection. It should be pointed out that, for a person of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the attached claims.

Claims

1. A flow feature detection method based on time series, characterized in that: include: At the start time of the current time window, the sample element is inserted into a data structure, wherein the data structure includes a state tracker and a state converter, and the state tracker is further provided with a filter, and the filter is used to forward the sample element that needs attention to the state tracker; the insertion position of the sample element is the filter or the state tracker, and the insertion position is determined according to the storage content of the filter and the state tracker at the end time of the previous time window; At the end of the current time window, check the sample elements in the state tracker one by one, and match the sample elements with the basic pattern with the storage content of the state converter; If the sample element is stored in the state converter, the current state of the sample element in the state converter is obtained, and the state conversion is performed to obtain the output state, and a report is made based on the result of the output state, wherein the reported result is used to characterize the flow characteristic change of the sample element.

2. The flow characteristic detection method based on time series according to claim 1 is characterized in that: Before the start time of the current time window, the method further includes: Creating a filter, wherein the filter comprises a plurality of filter buckets and a plurality of first hash functions; a filter unit is set in each filter bucket of the filter, and each filter unit comprises a first element field and a storage frequency associated with the first element field; Creating a state tracker, wherein the state tracker is a tracking bucket array with a second hash function; each tracking bucket in the tracking bucket array is provided with a plurality of tracking units, each of the tracking units includes a second element field, and a previous storage frequency counter and a current storage frequency counter associated with the second element field; A state converter is created, wherein the state converter is a conversion bucket array with a third hash function; each conversion bucket in the conversion bucket array is provided with a plurality of conversion units, each of the conversion units includes a third element field, and a current state, a state update flag and a state frequency counter associated with the third element field, and the state update flag is used to mark whether the state of the third element field in the current time window is updated.

3. The flow characteristic detection method based on time series according to claim 1 is characterized in that: Inserting a sample element into a data structure involves: Inputting the sample element into a second hash function to obtain a second hash value; If the sample element is not stored in the tracking bucket to which the second hash value is mapped, increasing the count result of the current storage frequency counter associated with the sample element in the state tracker by 1; If the sample element has been stored in the tracking bucket to which the second hash value is mapped, the sample element is inserted into the filter.

4. The flow characteristic detection method based on time series according to claim 3 is characterized in that: Inserting a sample element into the filter involves: The sample elements are respectively input into a plurality of first hash functions, and mapped to obtain a plurality of mapping buckets in the filter; Matching the sample elements with multiple mapping buckets one by one; If the sample element is stored in the mapping bucket, the storage frequency associated with the sample element in the mapping bucket is increased by 1; If the sample element is not stored in the mapping bucket and an irrelevant element is stored, the storage frequency associated with the irrelevant element is reduced by 1; when the storage frequency of the irrelevant element is reduced to 0, the mapping bucket is cleared; If the mapping bucket is an empty bucket, or the irrelevant element is cleared, the sample element is inserted and the storage frequency associated with the sample element is set to 1.

5. The flow characteristic detection method based on time series according to claim 4 is characterized in that: The method further comprises: In the filter, if the storage frequency of the sample element is greater than or equal to a potential mode threshold, the sample element is forwarded by the filter and inserted into the state tracker; wherein the potential mode threshold is preset by the data structure.

6. The flow characteristic detection method based on time series according to claim 1 is characterized in that: Matching the sample elements having the basic pattern with the storage contents of the state converter comprises: Inputting the sample element into a third hash function to obtain a third hash value; If the sample element is not stored in the conversion bucket to which the third hash value is mapped, insert the sample element into an empty cell of the state converter, evict an element according to the state priority setting when there is no empty cell, and then insert the sample element into the state converter; after the insertion, assign the basic mode to the sample element as the current state of the sample element; If the sample element is already stored in the conversion bucket to which the third hash value is mapped, the state conversion is completed in the state converter to obtain the output state, and the state update flag of the sample element is marked as updated.

7. The flow characteristic detection method based on time series according to claim 6 is characterized in that: Performing state conversion to obtain an output state, and reporting according to a result of the output state includes: Input the basic mode of the sample element and the current state of the sample element in the state converter, perform state conversion to obtain an output state; Matching the output state with the current state of the sample element; If the match is consistent, the count result of the state frequency counter associated with the sample element in the state converter is increased by 1; If the match fails and the output state is an accepting state, target pattern recognition is performed and reported.

8. The flow characteristic detection method based on time series according to claim 2 is characterized in that: After updating the storage content in the state converter according to the matching result, before the next time window starts, the method further includes: Clearing the first element field and the storage frequency associated with the first element field; expelling the second element field in the state tracker, for which the counting results of the previous storage frequency counter and the current storage frequency counter are both smaller than the potential mode threshold; Evicting the second element field whose count result of the current storage frequency counter in the state tracker is 0; expelling the third element field whose counting result of the state frequency counter exceeds a duration threshold, wherein the duration threshold is preset in the data structure; expelling the third element field marked as not updated by the status update identifier at the end of the current time window; The status update flag associated with the third element field in the conversion bucket is reset from updated to not updated.

9. An electronic device, comprising: A processor, and a memory storing a program, characterized in that the program includes instructions, which, when executed by the processor, cause the processor to execute the time series-based traffic feature detection method according to any one of claims 1 to 8.

10. A flow characteristic detection system based on time series, characterized in that: The time series-based traffic feature detection system includes the electronic device as described in claim 9.

Images