Big data stream processing method and device and storage medium

Abstract

The invention relates to the technical field of big data, and provides a big data stream processing method, which comprises the following steps: judging whether a target data stream is an active data stream in big data streams or not; if the target data stream is an inactive data stream in the big data stream, marking the inactive data stream; if the target data stream is the active data stream in the big data stream, judging whether the active data stream is a persistent active data stream: if the active data stream is the persistent active data stream, storing and marking the persistent active data stream; and if the active data stream is a non-persistent active data stream, temporarily storing and marking the non-persistent active data stream as a temporary active data stream. According to the technical scheme, invalid consumption of computing resources can be reduced, and the preliminary screening efficiency of data processing and the scientificity of business decision making are improved.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a big data stream processing method, device and storage medium. Background Art

[0002] In the field of data stream processing, especially in big data and real-time data analysis, a variety of techniques have been developed to monitor and analyze the continuous inflow of data. These techniques aim to quickly capture and process data to support immediate and long-term decision-making. For example, one solution helps detect persistent behaviors in network security by estimating the persistence of data items, and another solution focuses on discovering those persistent but infrequent data flows, which is particularly important for identifying security threats such as advanced persistent threats (APTs).

[0003] Although the above-mentioned big data stream processing methods have advantages in processing speed and efficiency, they still have shortcomings such as indiscriminate processing of all data, resulting in waste of computing resources and reduced processing efficiency. There is also the defect that it is difficult to screen out those data elements that have a long-term impact on business operations, thus affecting the quality of long-term decision-making and analysis. Summary of the invention

[0004] The present application provides a large data stream processing method, device and storage medium, which can reduce the ineffective consumption of computing resources, improve the initial screening efficiency of data processing and the scientific nature of business decision-making.

[0005] On the one hand, the present application provides a method for processing a large data stream, the method comprising:

[0006] Determine whether the target data stream is an active data stream in the large data stream;

[0007] If the target data flow is an inactive data flow in the large data flow, marking the inactive data flow;

[0008] If the target data stream is an active data stream in the large data stream, determining whether the active data stream is a persistent active data stream;

[0009] If the active data stream is a persistent active data stream, storing and marking the persistent active data stream;

[0010] If the active data flow is a non-persistent active data flow, the non-persistent active data flow is temporarily stored and marked as a temporary active data flow.

[0011] On the other hand, the present application provides a large data stream processing device, the device comprising:

[0012] A first judgment module is used to judge whether the target data flow is an active data flow in the large data flow;

[0013] A first marking module, configured to mark the inactive data flow if the target data flow is an inactive data flow in the large data flow;

[0014] A second judgment module, configured to judge whether the active data flow is a persistent active data flow if the target data flow is an active data flow in the large data flow;

[0015] a second marking module, configured to store and mark the persistent active data flow if the active data flow is a persistent active data flow;

[0016] The third marking module is configured to temporarily store and mark the non-persistent active data flow as a temporary active data flow if the active data flow is a non-persistent active data flow.

[0017] In a third aspect, the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the technical solution of the above-mentioned large data stream processing method when executing the computer program.

[0018] In a fourth aspect, the present application provides a storage medium storing a computer program, which, when executed by a processor, implements the steps of the technical solution of the above-mentioned large data stream processing method.

[0019] From the technical solution provided by the present application, it can be seen that, on the one hand, after judging that the target data stream is an inactive data stream in the large data stream, by marking these inactive data streams, the inactive elements in the small stream data can be effectively identified and filtered out, significantly reducing the ineffective consumption of computing resources and improving the initial screening efficiency of data processing; on the other hand, after further judging that the active data stream is a persistent active data stream, by storing and marking these persistent active data streams, the depth and accuracy of the large data stream processing are ensured, and the system can also identify and focus on key data items that have a long-term impact on business decisions, greatly improving the scientific nature of business decisions. In summary, the technical solution of the present application can reduce the ineffective consumption of computing resources, improve the initial screening efficiency of data processing and the scientific nature of business decisions. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 is a flowchart of a large data stream processing method provided by an embodiment of the present application;

[0022] Figure 2 It is a structural diagram of a large data stream processing device provided in an embodiment of the present application;

[0023] Figure 3 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] In this specification, adjectives such as first and second may be used only to distinguish one element or action from another element or action, without necessarily requiring or implying any actual such relationship or order. Where circumstances permit, reference to an element or component or step (etc.) should not be interpreted as being limited to only one of the elements, components, or steps, but may be one or more of the elements, components, or steps, etc.

[0026] In this specification, for the convenience of description, the sizes of various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0027] In the field of data stream processing, especially in big data and real-time data analysis, a variety of technologies have been developed to monitor and analyze the continuously flowing data. These technologies are designed to quickly capture and process data to support immediate and long-term decision-making. For example, one solution is to help detect persistent behaviors in network security by estimating the persistence of data items, and another solution is to focus on discovering those persistent but infrequent data flows, which is particularly important for identifying security threats such as advanced persistent threats (APTs). Although the above big data stream processing methods have advantages in processing speed and efficiency, there are still shortcomings such as indiscriminate processing of all data, resulting in a waste of computing resources and reduced processing efficiency. There is also a defect that it is difficult to screen out those data elements that have a long-term impact on business operations, thereby affecting the quality of long-term decision-making and analysis.

[0028] In view of the above problems in the prior art, this application proposes a method for processing large data streams, the flow chart of which is shown in the attached figure. Figure 1 As shown, it mainly includes steps S101 to S105, which are described in detail as follows:

[0029] Step S101: Determine whether the target data flow is an active data flow in the large data flow.

[0030] In the field of large data stream processing, not every or every data stream in a large data stream plays a role in decision-making. In fact, those small stream data or data streams with low activity in a large data stream are not only unhelpful for business decision-making, but may also cause ineffective consumption of computing resources. Therefore, in order to reduce the ineffective consumption of computing resources and improve the initial screening efficiency of data processing, the technical solution of the present application can first determine whether the target data stream is an active data stream in a large data stream. It should be noted that the target data stream refers to the data stream that needs to be monitored in the embodiment of the present application in the large data stream.

[0031] As an embodiment of the present application, determining whether the target data stream is an active data stream in the large data stream may be implemented through steps S1011 to S1013, which are described in detail as follows:

[0032] Step S1011: Determine whether the evaluation target data stream is a dimension of an active data stream in a large data stream.

[0033] In an embodiment of the present application, dimensions for evaluating whether a target data stream is an active data stream in a large data stream include an update frequency of the target data stream per unit time, an update frequency of the large data stream per unit time, and a diversity or uncertainty of the target data stream, etc., wherein the update frequency of the target data stream per unit time may be the number of events of the target data stream per unit time, and the update frequency of the large data stream per unit time is the number of events of all large data streams.

[0034] Step S1012: Calculate the target data flow in the time window w according to the dimension of evaluating whether the target data flow is an active data flow in the large data flow. j Activity index within.

[0035] If the update frequency of the target data stream per unit time is f target , let the update frequency of the large data flow in unit time be total_flow_rate, and let the diversity or uncertainty of the target data flow be H target , then according to the dimension of evaluating whether the target data stream is an active data stream in the big data stream, calculate the target data stream in the time window w j The activity index can be expressed by the formula Calculate the target data flow in time window w j Activity index A target In the above embodiment, the diversity or uncertainty of the target data stream is H target It can be determined by: obtaining a set S of events contained in the target data stream; calculating jAny event s in the inner set S i The probability of presentation in the target data stream is p(s i ); According to the formula Calculate the diversity or uncertainty H of the target data stream target , where k is the number of types of events contained in the set S. In the above embodiment, if the time window w j Any event s in the inner set S i The number of presentations in the target data stream is count(s i ), the total number of events in the big data stream is N, then in the time window w j Any event s in the inner set S i The probability of presentation in the target data stream is p(s i ) can be calculated according to the formula to calculate.

[0036] Step S1013: If the target data stream is within the time window w j If the activity index within exceeds a first preset threshold, the target data stream is determined to be an active data stream in the large data stream.

[0037] If the target data stream is in the time window w j If the activity index in the target data stream exceeds the first preset threshold, it means that the target data stream is highly active, and the target data stream can be determined to be an active data stream in the big data stream. The first preset threshold is an important parameter that determines when the target data stream is considered "active". Therefore, the method of the above embodiment may also include: extracting the activity index from the historical big data stream and calculating the activity of all big data streams; obtaining the statistical characteristics of the activity of all big data streams, calculating their mean μ and standard deviation σ; according to the formula θ active =μ+α 1 *σCalculate θ active , and θ active Determined as the first preset threshold, α 1 is the first adjustment coefficient. Usually, the first adjustment coefficient α 1 The value of can be 1 or 2. Of course, the first adjustment coefficient α can be adjusted dynamically 1 As an embodiment of the present application, the first adjustment coefficient α is dynamically adjusted. 1 The value of can be: through the standard deviation σ of the big data stream, analyze the volatility of the big data stream, that is, the larger the standard deviation σ, the greater the volatility of the big data stream, and vice versa; according to the volatility of the data, choose a reasonable α 1 Range, that is, if the standard deviation σ of the large data stream is large, choose α 1 In the range [1, 2], the first preset threshold θ activeRelatively stable, not easily affected by small fluctuations. If the standard deviation σ of the large data stream is small, α can be selected. 1 In the range [2, 5], the first preset threshold θ active More sensitive to changes; or, dynamically adjust α based on the periodicity, trend, and volatility of large data streams 1 That is, if the large data flow is in a period of high volatility, increase the first adjustment coefficient α 1 The value of the first preset threshold θ active More flexible, if the large data flow is in a low volatility period, reduce the first adjustment coefficient α 1 The value of the first preset threshold θ active More stable.

[0038] As another embodiment of the present application, the first preset threshold θ aetive It can also be set by weighted average of historical activity data. The specific scheme is as follows: Step S1 to Step S3:

[0039] Step S1: Collect activity data of the target data stream in the past n time windows (for example, every hour or every day) {A 1 , A 2 , ..., A i , ..., A n}, where A i Indicates the activity value of the target data stream in the i-th time window.

[0040] Step S2: Calculate the weighted average activity A of the target data stream according to the following formula: weighted :

[0041]

[0042] Among them, ω i It is the weight associated with the time window, which is usually set according to the distance between the time window and the current time; a linear attenuation method can be selected, that is:

[0043]

[0044] In this way, the closer the target data stream is to the current time window, the higher its weight ω i The bigger.

[0045] Step S3: The weighted average activity A of the target data stream weighted Multiply by a threshold ratio γ, for example γ = 90%, and the product A weighted *γ is used as the first preset threshold value θ active , that is, θ active =A weighted *γ.

[0046] Step S102: If the target data flow is an inactive data flow in the large data flow, the inactive data flow is marked.

[0047] When the target data stream is an inactive data stream in the large data stream, marking it as an inactive data stream means that less attention can be paid to the target data stream and it will not be considered as an influencing factor for business decisions.

[0048] Step S103: If the target data stream is an active data stream in the large data stream, it is determined whether the active data stream is a persistent active data stream.

[0049] In a large data stream, the activity of the data stream changes dynamically. Some data streams may be active only for a short period of time, while other data streams may be continuously active. In this case, simply treating all "active" data streams equally may lead to resource waste or inefficient processing. In other words, for each data stream, especially a large data stream, certain computing resources are required for processing, storage, indexing and retrieval. If a data stream is judged to be active for a short period of time (i.e., it generates certain events or data), but it quickly stops being active afterwards, it will lead to waste of computing resources to continue to reserve it. Persistently active data streams usually contain valuable or high-priority data that need to be continuously tracked and analyzed. For example, financial transaction streams, social media active user streams, or system log streams may reflect the long-term trend of system status or user behavior. Based on the above facts, when judging that the target data stream is an active data stream in a large data stream, it is necessary to further judge whether the active data stream is a persistent active data stream.

[0050] As an embodiment of the present application, determining whether the active data stream is a persistent active data stream may be implemented through steps S1031 to S1033, which are described in detail as follows:

[0051] Step S1031: Determine whether the evaluation target data stream is a dimension of an active data stream in a large data stream.

[0052] In an embodiment of the present application, the dimensions for evaluating whether the target data stream is an active data stream in a large data stream may be the activity change rate (Active Change Rate) and the activity persistence (Persistence of Activity). The former indicates the change in the activity of the data stream in different time windows, and the latter can be measured by the average activity of the target data stream in multiple time windows.

[0053] Step S1032: Calculate the activity index of the target data stream in multiple time windows according to the dimension of evaluating whether the target data stream is an active data stream in the large data stream.

[0054] If the target data stream is in time window t i The activity index is A target (t i ), you can use the formula Calculate the activity index P of the target data stream in n time windows target It should be noted that the target data stream is in the time window t i The activity index is A target (t i ) is the same as the calculation scheme in the above embodiment, which is to calculate the target data stream in the time window w according to the dimension of evaluating whether the target data stream is an active data stream in the large data stream. j The technical solution of the activity index within is the same as that of the embodiment, and reference may be made to the relevant description of the aforementioned embodiment, which will not be repeated here.

[0055] Step S1033: If the activity index of the target data flow in multiple time windows exceeds a second preset threshold, the target data flow is determined to be a persistent active data flow.

[0056] If the second preset threshold is θ persistent , then when the activity index of the target data stream in multiple time windows exceeds the second preset threshold, that is, P target >θ persistent When , the target data stream can be determined to be a persistent active data stream. It should be noted that, as in the above embodiment, the first preset threshold θ active Similarly, in the embodiment of the present application, the second preset threshold is θ persistent It is also an important parameter for determining whether the active data flow is a persistent active data flow. As an embodiment of the present application, the second preset threshold is determined to be θ persistent The method may be: obtaining the activity of the target data stream in T historical time windows; calculating the weighted average activity A of the activity of the target data stream in T historical time windows weighted According to the formula θ persistent =A weighted +α 2 *β calculate θ persistent , and θ persistent is determined as the second preset threshold, where α 2 is the second adjustment coefficient, and β is the rate of change of activity over time obtained by linear regression, that is, the trend slope. If the activity of the target data stream in T historical time windows is recorded as {A 1 , A 2 , ..., A t , ..., A T}, where 4 represents the activity of the target data stream in the tth historical time window, then it can be calculated according to the formula Calculate the weighted average activity A of the target data stream in T historical time windows weighted , where ω t Indicates the activity of the data flow A t In the time window, the data stream closer to the current moment contributes more to the activity evaluation, and the weight ω t It can be set by a linear function, that is, the closer the data flow is to the current time, the greater the weight, and the farther the data flow is from the current time, the smaller the weight. Specifically, if the current time is t now The length of the time window is T, and the data flow in the time window is in chronological order from t 0 to T (where t 0 is the earliest moment, t T is the latest moment), the weight ω can be set as follows t :

[0057]

[0058] Among them, t is the timestamp of a data stream in the time window, t now is the current moment, and T is the total length of the time window. This means that when, the closer to the current moment t now , that is, t now -, the smaller the weight ω t The larger it is; the farther away from the current moment t now , that is, t now -t is larger, the weight ω t The smaller the weight ω t The value range is between 0 and 1.

[0059] In addition to setting the weight ω in the above way t , and the weight ω can also be set by exponential weighting t ,Right now:

[0060]

[0061] Where α is a decay factor (usually 0<α<1), which determines the rate of time decay, t is the timestamp of a data point in the time window, and t now is the current moment. The weight ω is set in an exponentially weighted manner t The advantage of is that it can respond more sensitively to short-term changes, and the decay rate can be controlled by adjusting α. In general, α is small (such as 0.9 or 0.95), so that the newer data dominates.

[0062] In the above embodiment, the second preset threshold θ persistent The calculation formula of θpersistent =A weighted +α 2 *β, and α 2 is a hyperparameter, where β can be obtained by linear regression, for example, by fitting a linear regression model through the least squares method, and α 2 is a control trend for the second preset threshold value θ persistent The selection of α affects the sensitivity of the threshold. 2 The method is to define a loss function L(α 2 ), which is used to evaluate the performance of the model in predicting long-term activity, that is, for a set of real long-term activity labels θ real , the loss function can be set as:

[0063]

[0064] Among them, A weighted，i is the weighted average activity of the ith data stream, β i is the corresponding trend slope, θ real，i is the actual persistent activity threshold; then, an optimization algorithm such as gradient descent is used to minimize the loss function L(α 2 ), and obtain the optimal second adjustment coefficient α 0 .

[0065] Step S104: If the active data flow is a persistent active data flow, the persistent active data flow is stored and marked.

[0066] Persistent active data streams usually contain valuable or high-priority data, which may be related to real-time monitoring, early warning mechanisms or business decisions. They not only require more computing resources, but also need to be processed continuously. Therefore, if the active data stream is a persistent active data stream, it is necessary to store and mark these persistent active data streams, that is, use persistent storage to ensure that they can be fully processed throughout the life cycle. For example, dynamically adjust the scheduling strategy of computing and storage to achieve more efficient load balancing and resource allocation.

[0067] Step S105: If the active data flow is a non-persistent active data flow, temporarily store and mark the non-persistent active data flow as a temporary active data flow.

[0068] The timeliness and importance of non-persistent active data streams are relatively low. They may only fluctuate at a certain moment and will not have a significant impact on the overall operation or decision-making of the system. For example, a small-scale emergency may cause the data stream to increase in a short period of time, but it will return to normal after a period of time. For non-persistent active data streams, you can temporarily store and mark the non-persistent active data streams as temporary active data streams so that they can be processed on demand. For example, data is processed during the active period of the data stream, and the cache is cleared or the processing frequency is reduced afterwards. The benefit of this strategy is that it can avoid unnecessary calculations and storage when the activity of the stream is low, thereby improving system efficiency and reducing resource waste.

[0069] From the above attached Figure 1 It can be seen from the example of the large data stream processing method that, on the one hand, after judging that the target data stream is an inactive data stream in the large data stream, by marking these inactive data streams, the inactive elements in the small stream data can be effectively identified and filtered out, significantly reducing the ineffective consumption of computing resources and improving the initial screening efficiency of data processing; on the other hand, after further judging that the active data stream is a persistent active data stream, by storing and marking these persistent active data streams, the depth and accuracy of the large data stream processing are ensured, and the system can also identify and focus on processing key data items that have a long-term impact on business decisions, greatly improving the scientific nature of business decisions. In summary, the technical solution of the present application can reduce the ineffective consumption of computing resources, improve the initial screening efficiency of data processing and the scientific nature of business decisions.

[0070] Please see attached Figure 2 , is a large data stream processing device provided in an embodiment of the present application, the device may include a first judgment module 201, a first marking module 202, a second judgment module 203, a second marking module 204 and a third marking module 205, which are described in detail as follows:

[0071] The first determination module 201 is used to determine whether the target data flow is an active data flow in the large data flow;

[0072] A first marking module 202, configured to mark an inactive data flow if the target data flow is an inactive data flow in the large data flow;

[0073] A second determination module 203 is configured to determine whether the active data flow is a persistent active data flow if the target data flow is an active data flow in the large data flow;

[0074] A second marking module 204, configured to store and mark the persistent active data flow if the active data flow is a persistent active data flow;

[0075] The third marking module 205 is configured to temporarily store and mark the non-persistent active data flow as a temporary active data flow if the active data flow is a non-persistent active data flow.

[0076] From the above attached Figure 2 It can be seen from the example of a large data stream processing device that, on the one hand, after determining that the target data stream is an inactive data stream in the large data stream, by marking these inactive data streams, the inactive elements in the small stream data can be effectively identified and filtered out, significantly reducing the ineffective consumption of computing resources and improving the initial screening efficiency of data processing; on the other hand, after further determining that the active data stream is a persistent active data stream, by storing and marking these persistent active data streams, the depth and accuracy of the large data stream processing are ensured, and the system can also identify and focus on processing key data items that have a long-term impact on business decisions, greatly improving the scientific nature of business decisions. In summary, the technical solution of the present application can reduce the ineffective consumption of computing resources, improve the initial screening efficiency of data processing and the scientific nature of business decisions.

[0077] Figure 3 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 3 As shown, the electronic device 3 of this embodiment mainly includes: a processor 30, a memory 31, and a computer program 32 stored in the memory 31 and executable on the processor 30, such as a program of a large data stream processing method. When the processor 30 executes the computer program 32, the steps in the above-mentioned large data stream processing method embodiment are implemented, such as Figure 1 Alternatively, when the processor 30 executes the computer program 32, the functions of each module / unit in the above-mentioned device embodiments are implemented, for example Figure 2 The functions of the first judging module 201 , the first marking module 202 , the second judging module 203 , the second marking module 204 and the third marking module 205 are shown.

[0078] Exemplarily, the computer program 32 of the large data stream processing method mainly includes: determining whether the target data stream is an active data stream in the large data stream; if the target data stream is an inactive data stream in the large data stream, marking the inactive data stream; if the target data stream is an active data stream in the large data stream, determining whether the active data stream is a persistent active data stream; if the active data stream is a persistent active data stream, storing and marking the persistent active data stream; if the active data stream is a non-persistent active data stream, temporarily storing and marking the non-persistent active data stream as a temporary active data stream. The computer program 32 can be divided into one or more modules / units, one or more modules / units are stored in the memory 31, and executed by the processor 30 to complete the present application. One or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program 32 in the electronic device 3. For example, the computer program 32 can be divided into the functions of a first judgment module 201, a first marking module 202, a second judgment module 203, a second marking module 204 and a third marking module 205 (modules in the virtual device), and the specific functions of each module are as follows: the first judgment module 201 is used to judge whether the target data stream is an active data stream in the large data stream; the first marking module 202 is used to mark the inactive data stream if the target data stream is an inactive data stream in the large data stream; the second judgment module 203 is used to judge whether the active data stream is a persistent active data stream if the target data stream is an active data stream in the large data stream; the second marking module 204 is used to store and mark the persistent active data stream if the active data stream is a persistent active data stream; the third marking module 205 is used to temporarily store and mark the non-persistent active data stream as a temporary active data stream if the active data stream is a non-persistent active data stream.

[0079] The electronic device 3 may include but is not limited to a processor 30 and a memory 31. Those skilled in the art will appreciate that Figure 3 It is only an example of the electronic device 3 and does not constitute a limitation of the electronic device 3. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0080] The processor 30 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0081] The memory 31 may be an internal storage unit of the electronic device 3, such as a hard disk or memory of the electronic device 3. The memory 31 may also be an external storage device of the electronic device 3, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 3. Further, the memory 31 may also include both an internal storage unit of the electronic device 3 and an external storage device. The memory 31 is used to store computer programs and other programs and data required by the electronic device. The memory 31 may also be used to temporarily store data that has been output or is to be output.

[0082] Those skilled in the art can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0083] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0084] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0085] In the embodiments provided in the present application, it should be understood that the disclosed devices / equipment and methods can be implemented in other ways. For example, the device / equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0086] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0087] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0088] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program of the large data stream processing method can be stored in a storage medium. When the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments, that is, determine whether the target data stream is an active data stream in the large data stream; if the target data stream is an inactive data stream in the large data stream, mark the inactive data stream; if the target data stream is an active data stream in the large data stream, determine whether the active data stream is a persistent active data stream; if the active data stream is a persistent active data stream, store and mark the persistent active data stream; if the active data stream is a non-persistent active data stream, temporarily store and mark the non-persistent active data stream as a temporary active data stream. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. Storage media may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content contained in the storage medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, storage media do not include electric carrier signals and telecommunication signals.

[0089] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application is described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. The specific implementation methods described above further describe the purpose, technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only the specific implementation method of the present application, and is not used to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present invention.

Claims

1. A method for processing a large data stream, characterized in that: The method comprises: Determine whether the target data stream is an active data stream in the large data stream; If the target data flow is an inactive data flow in the large data flow, marking the inactive data flow; If the target data stream is an active data stream in the large data stream, determining whether the active data stream is a persistent active data stream; If the active data stream is a persistent active data stream, storing and marking the persistent active data stream; If the active data flow is a non-persistent active data flow, the non-persistent active data flow is temporarily stored and marked as a temporary active data flow.

2. The method for processing a large data stream according to claim 1, characterized in that: The determining whether the target data stream is an active data stream in the large data stream includes: Determine a dimension for evaluating whether the target data stream is an active data stream in the large data stream; According to the dimension of evaluating whether the target data stream is an active data stream in the large data stream, the target data stream in the time window w is calculated. j Activity index within If the target data stream is in the time window w j If the activity index within exceeds a first preset threshold, the target data stream is determined to be an active data stream in the large data stream.

3. The method for processing a large data stream as claimed in claim 2, characterized in that: The dimensions for evaluating whether the target data stream is an active data stream in the large data stream include the update frequency of the target data stream within a unit time, the update frequency of the large data stream within the unit time, and the diversity or uncertainty of the target data stream.

4. The method for processing a large data stream as claimed in claim 3, characterized in that: The diversity or uncertainty of the target data stream is determined by: Obtain a set S consisting of events contained in the target data stream; Calculate in the time window w j Any event s in the set S i The probability of presentation p(s) in the target data stream i ); According to the formula Calculate the diversity or uncertainty H of the target data stream target , k is the number of types of events contained in the set S.

5. The method for processing a large data stream as claimed in claim 2, characterized in that: The method further comprises: Extract activity index from historical big data streams and calculate the activity of all big data streams; Obtain the statistical characteristics of the activity of all the large data streams, and calculate the mean μ and standard deviation σ thereof; According to the formula θ active =μ+α1*σCalculate θ active , the θ active The first preset threshold is determined, and the α1 is the first adjustment coefficient.

6. The method for processing a large data stream according to claim 1, characterized in that: The determining whether the active data flow is a persistent active data flow includes: Determine a dimension for evaluating whether the target data stream is an active data stream in the large data stream; Calculating an activity index of the target data stream in multiple time windows according to a dimension for evaluating whether the target data stream is an active data stream in a large data stream; If the activity index of the target data stream in multiple time windows exceeds a second preset threshold, the target data stream is determined to be the persistent active data stream.

7. The method for processing a large data stream according to claim 6, characterized in that: The method further comprises: Obtaining the activity of the target data stream in T historical time windows; Calculate the weighted average activity A of the target data stream in T historical time windows weighted ; According to the formula θ persistent =A weighted +α2*β to calculate θ persistent , the θ persistent is determined as the second preset threshold, α2 is the second adjustment coefficient, and β is the rate of change of activity over time obtained by linear regression.

8. A large data stream processing device, characterized in that: The device comprises: A first judgment module is used to judge whether the target data flow is an active data flow in the large data flow; A first marking module, configured to mark the inactive data flow if the target data flow is an inactive data flow in the large data flow; A second judgment module, configured to judge whether the active data flow is a persistent active data flow if the target data flow is an active data flow in the large data flow; a second marking module, configured to store and mark the persistent active data flow if the active data flow is a persistent active data flow; The third marking module is configured to temporarily store and mark the non-persistent active data flow as a temporary active data flow if the active data flow is a non-persistent active data flow.

9. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images