A method for data flow calculation based on dynamic time slice cache

Abstract

The present application provides a method for calculating data flow based on dynamic time slice cache, including collecting data flow information of each business system according to a preset time threshold; acquiring an intermediate variable sequence and a timestamp of the data flow information in real-time streaming; reading a first cache data flow information of a computer memory according to the intermediate variable sequence; combining the first cache data flow information, current time information and the intermediate variable sequence, generating a second cache data flow information by transforming the intermediate result; combining the second cache data flow information with data flow business information, generating a third cache data flow information; dynamically dividing the granularity of the timestamp according to business requirements to form a dynamic time slice; based on the dynamic time slice, merging the second cache data flow information specified by the timestamp according to the performance requirements to reduce the query time of the third cache data flow information.

Description

Technical Field

[0001] The present application relates to the field of data flow computing, and in particular to a method for data flow computing based on dynamic time slice caching. Background Art

[0002] In recent years, Internet technology has made great progress. The behavior data of network users contains a lot of valuable and meaningful information. Analyzing massive data contains huge value. In the process of analyzing massive data, the accuracy and efficiency of the analysis are related to the reasonable use of data. The industry urgently needs to introduce new technologies and tools to provide effective support for the calculation of business system data volume.

[0003] One implementation method is to slice the time data in the data stream information into fixed sizes, but this approach has certain defects. For example, it cannot be adjusted according to actual needs, resulting in time data slices that are too large or too small, making it difficult to ensure the reasonable use of data stream information; the data stream information cache lacks a merging mechanism, resulting in excessive server memory, low cache processing efficiency, normal requests being forced to enter a waiting state, and complex real-time data access, resulting in slow data stream calculation. Summary of the invention

[0004] The present application provides a method for data stream calculation based on dynamic time slice caching to solve the problems of unreasonable time data slicing and slow data stream information query in the process of analyzing massive data.

[0005] The method for data flow calculation based on dynamic time slice caching includes:

[0006] Collect data flow information of each business system according to preset time thresholds;

[0007] Real-time streaming acquisition of intermediate variable sequences and timestamps of the data stream information;

[0008] Reading first cache data flow information of a computer memory according to the intermediate variable sequence;

[0009] Combining the first cache data flow information, the current time information and the intermediate variable sequence, and transforming the intermediate result to generate the second cache data flow information;

[0010] Combining the second cache data flow information with the data flow service information, generating third cache data flow information;

[0011] Dynamically divide the timestamp into granularities according to business requirements to form dynamic time slices;

[0012] Based on the dynamic time slice, the second cache data flow information specified by the timestamp according to the performance requirement is merged to reduce the query time of the third cache data flow information.

[0013] Optionally, after the step of combining the second cached data flow information with the data flow service information to generate the third cached data flow information, the method further includes:

[0014] The third cache data flow information is written into the cache of the computer memory.

[0015] Optionally, before the step of dynamically dividing the timestamp into granularities according to business requirements, the method further includes:

[0016] The upper and lower bounds of the timestamp are specified according to business requirements, and the timestamp is sliced ​​according to the upper and lower bounds.

[0017] Optionally, the data stream information includes indicator code, service field, time interval, and time unit.

[0018] Optionally, the intermediate variable sequence information includes a hash indicator code and a hash business field.

[0019] Optionally, before the step of acquiring the intermediate variable sequence and timestamp of the data stream information in real-time streaming, the method further includes:

[0020] The streaming acquisition process and streaming acquisition content are formulated through the preset program interface.

[0021] Optionally, the streaming content acquisition includes real-time acquisition of a series of dynamic data aggregates.

[0022] Optionally, the streaming acquisition has no restrictions on time distribution and quantity.

[0023] Optionally, the cache information of the computer memory includes array index, disk and main memory transmission unit, and information fixed block.

[0024] Optionally, the third cache data flow information is service information.

[0025] It can be seen from the above technical scheme that the present application provides a method for calculating data flow based on dynamic time slice cache, including collecting data flow information of each business system according to a preset time threshold; acquiring the intermediate variable sequence and timestamp of the data flow information in real-time streaming; reading the first cache data flow information of the computer memory according to the intermediate variable sequence; combining the first cache data flow information, current time information and the intermediate variable sequence, and generating the second cache data flow information through intermediate result transformation; combining the second cache data flow information with the data flow business information, generating the third cache data flow information; dynamically dividing the timestamp into granularity according to business needs to form a dynamic time slice; based on the dynamic time slice, merging the second cache data flow information specified by the timestamp according to the performance requirements to reduce the query time of the third cache data flow information. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0027] Figure 1 The figure is a flowchart of a method for caching data stream calculation based on dynamic time slices. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below in conjunction with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present application. The technical solutions provided by the embodiments of the present application are described in detail below in conjunction with the drawings.

[0029] In this application, a time slice, which may also be referred to as a quantum or processor slice, is a microscopic section of CPU (central processing unit, CPU for short) time allocated to each running process by a time-sharing operating system. Usually, an operating system allows multiple processes to run simultaneously, for example, you can use a browser to browse the web and download files while opening a music player to listen to music, but the same CPU can never really run multiple tasks simultaneously. When only one CPU is considered, these processes appear to be running simultaneously, but in fact they are running in turns, and the processing time allocated to each process by the scheduler of the operating system kernel can be understood as a time slice.

[0030] This application is based on the processing technology of dynamic time slices. Dynamic time slices use the fragmented cache of intermediate variable sequence time to dynamically divide the granularity according to data usage, thereby improving the efficiency of subsequent data usage. At the same time, it assists in data flow calculation, accelerates data flow calculation, effectively integrates data resources, and quickly judges and processes the input business and data information in a timely manner.

[0031] See also Figure 1 , Figure 1 The flowchart of the method based on dynamic time slice caching data flow calculation is as follows: Figure 1 It can be seen that the method specifically includes:

[0032] S1: Collect data flow information of each business system according to the preset time threshold.

[0033] Users can set the time threshold according to their needs. For example, they can set the collection of business system data, that is, data flow information, every 3 minutes. Assume that there are n business systems, where n is a positive integer and n is greater than or equal to 1. Figure 1 In step S1, data is collected from business system 1, business system 2, ... business system n every preset time threshold. Data flow information can be preset according to user needs. For example, data flow information can include but is not limited to indicator code, business field, time interval, time unit, etc.

[0034] S2: Real-time streaming acquisition of the intermediate variable sequence and timestamp of the data stream information.

[0035] For the data stream information collected in step S1, an acquisition behavior is performed on the intermediate variable sequence and timestamp of the data stream information. The acquisition behavior realizes real-time streaming acquisition by providing a program interface. It should be noted that the streaming acquisition process and streaming acquisition content can be formulated through a preset program interface. The streaming acquisition in the embodiment of the present application is the real-time acquisition of a series of dynamic data aggregates that are infinite in time distribution and quantity, that is, the streaming acquisition content includes the real-time acquisition of a series of dynamic data aggregates, and the streaming acquisition does not limit the time distribution and quantity.

[0036] Providing a program interface to achieve real-time streaming acquisition ensures real-time acquisition of a series of dynamic data collections that are infinite in time distribution and quantity, reduces the complexity of real-time access, and thus improves the accuracy of real-time acquisition and the simplicity of access.

[0037] The intermediate variable sequence information can also be formulated according to user needs. For example, the intermediate variable sequence information can include, but is not limited to, hash indicator codes, hash business fields, etc. Timestamps are data generated using digital signature technology. The object of the signature includes information such as the original file information, signature parameters, and signature time. The timestamp system is used to generate and manage timestamps, and digitally sign the signature object to generate a timestamp to prove that the original file existed before the signature time.

[0038] S3: Reading first cache data flow information of a computer memory according to the intermediate variable sequence.

[0039] When reading computer memory, in order to reduce the operation of computer input and output, the embodiment of the present application adopts the reading method of cache stream. In the past, when reading by byte stream or character stream, hard disk data would be accessed and read again every time reading and writing. If the reading and writing frequency is relatively high, the burden of computer input and output operation would be increased, resulting in poor computer performance.

[0040] When the cache stream reads data, it will read more data into the cache at one time. Each subsequent read will be accessed in the cache until the data in the cache is read and then read from the hard disk. When the cache stream writes data, it will first write the data to the cache area, and will not write the data to the hard disk until the cache area reaches a certain amount. In step S3, the first cache data stream information of the computer memory is read according to the intermediate variable sequence generated in step S2. Among them, the first cache data stream information is the cache information of the computer memory, and the cache information may include array index, disk and main memory transmission unit, information fixed block, etc.

[0041] S4: combining the first cache data flow information, the current time information and the intermediate variable sequence, and generating second cache data flow information by transforming the intermediate result;

[0042] The specific content of the intermediate result transformation can be formulated according to actual needs. The second cache data flow information is the first cache data flow information and the time information combined with the intermediate variable sequence, and the intermediate transformation is performed on it.

[0043] S5: Combining the second cache data flow information with the data flow service information, generating third cache data flow information.

[0044] The third cache data flow information is obtained by combining the second cache data flow information with the data flow service information, and the third cache data flow information is written into the cache of the computer memory, and the third cache data flow information is service information.

[0045] Combined with steps S3-S5, assuming that the first cache data flow information is cache information k1, the second cache data flow information is k2, and the third cache data flow information is k3, the data flow relationship between the three is: according to the intermediate variable sequence of the data flow information, read the cache information k1 of the computer memory, use the cache result k1 and the time information, combined with the intermediate variable sequence, to construct the intermediate result and obtain k2; obtain k3 by combining k2 with the data flow business information, and write k3 into the computer memory cache.

[0046] S6: Dynamically divide the timestamp into granularities according to business requirements to form dynamic time slices.

[0047] Before the step of dynamically dividing the granularity of the timestamp according to business needs, it also includes specifying the upper and lower bounds of the timestamp according to business needs, and slicing the timestamp according to the upper and lower bounds. In order to facilitate the understanding of the upper and lower bounds, an example is given here to illustrate. For example, for the collected data stream information, it can be defined as intermediate variable time fragmentation from a certain perspective. The so-called intermediate variable time fragmentation can include intermediate variable sequences and time fragmented partition segments. The partition segment information can include the upper bound of the timestamp interval, the lower bound of the timestamp interval, and the time unit. The partition segment specifies the upper and lower bounds according to business needs. This is the source of the concept of upper and lower bounds.

[0048] Normally, the computer kernel assigns an equal initial time slice to each process, and then each process takes turns to execute the corresponding time. When all processes are in a state where the time slice is exhausted, the kernel will recalculate and assign time slices to each process, and so on. The length of time slices assigned to all processes in a system is not completely equal, although the initial time slices are basically equal. The system calculates the interactivity of each process by measuring the length of time the process is in the "sleeping" and "running" states. The superposition of interactivity and the preset static priority of each process is the dynamic priority. The dynamic priority is proportionally scaled to the length of the time slice to be assigned to that process. Generally, in order to obtain a faster response speed, processes with strong interactivity are assigned longer time slices than processes with weak interactivity. The use of dynamic time slices can match the process intensity to the maximum extent, and avoid resource waste while meeting the process operation intensity.

[0049] The embodiment of the present application slices the timestamp according to the upper and lower bounds specified by the business requirements based on the timestamp of the data stream information, and the slices are dynamically divided into granularity according to the data usage to ensure the reasonable use of the data stream information. The data granularity can be understood as the smallest unit used to represent the composition of a data set. The timestamp is sliced ​​according to the upper and lower bounds specified by the business requirements, and the slices are dynamically divided into granularity according to the data usage. For example, the indicator of the number of times a certain IP appears in the past 30 minutes is counted. When the event enters the stream computing system, the minute of the current time plus 29 minutes forward is cut out according to the event time, a total of 30 time slices are constructed, and 30 cache keys are constructed. The key in the embodiment of the present application can be understood as the elements of the query, the query items, etc., and then the cache is queried for data, the cache results are accumulated and calculated to obtain the value of the indicator, the value of the current time slice is increased by 1 and the cache is updated.

[0050] Through the above steps, the memory storage occupied by time slices is reduced, and the unreasonable time slices of business information are reduced. The granularity is dynamically divided based on data usage, which improves the rationality of time slices for data flow calculation and effectively improves the utilization rate of memory resources.

[0051] S7: Based on the dynamic time slice, merge the second cache data flow information specified by the timestamp according to the performance requirement to reduce the query time of the third cache data flow information.

[0052] According to the timestamp of the data flow information, the timestamp is merged according to the K2 specified by the performance requirements, that is, the second cache data flow information, and the cache sequence of the computer memory K2 is merged, thereby reducing the complexity and number of cache information key values, thereby improving the query rate and query number of the cache information, and effectively reducing k3, that is, the business information query time.

[0053] For example, to count the number of IP occurrences in the past 72 hours, according to the conventional algorithm, 72 keys need to be queried each time for calculation. In this application, by merging historical data, the number of cache keys required for querying each indicator calculation is reduced. For example, historical data can be merged in steps of 5 or 10. After 72 keys are merged in steps of 10, only 9 keys need to be queried in the best case, which greatly reduces the number of keys to be queried for each indicator calculation, thereby improving the query speed.

[0054] It can be seen from the above technical scheme that the present application provides a method for calculating data flow based on dynamic time slice cache, including collecting data flow information of each business system according to a preset time threshold; acquiring the intermediate variable sequence and timestamp of the data flow information in real-time streaming; reading the first cache data flow information of the computer memory according to the intermediate variable sequence; combining the first cache data flow information, current time information and the intermediate variable sequence, and generating the second cache data flow information through intermediate result transformation; combining the second cache data flow information with the data flow business information, generating the third cache data flow information; dynamically dividing the timestamp into granularity according to business needs to form a dynamic time slice; based on the dynamic time slice, merging the second cache data flow information specified by the timestamp according to the performance requirements to reduce the query time of the third cache data flow information. The present application realizes real-time streaming acquisition by providing a program interface, ensures real-time acquisition of a series of dynamic data collections that are infinite in time distribution and quantity, reduces the complexity of access to real-time acquisition, and thus improves the accuracy of real-time acquisition and the simplicity of access; by slicing and dynamically dividing the granularity of the timestamp according to the upper and lower bounds specified by business needs, the memory storage occupied by the time slice is reduced, and the irrationality of the time slice of business information is reduced; the granularity is dynamically divided according to the data usage, which improves the rationality of the time slice of data flow calculation and effectively improves the utilization rate of memory resources. By merging the cache sequence of the second cache data stream information of the computer memory, the complexity and quantity of the cache information key value are reduced, thereby improving the query rate and query quantity of the cache information, and effectively reducing the query time of business information.

[0055] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

Claims

1. A method for calculating data flow based on dynamic time slice caching, characterized in that: include: Collect data flow information of each business system according to preset time thresholds; Real-time streaming acquisition of intermediate variable sequences and timestamps of the data stream information; Reading first cache data flow information of a computer memory according to the intermediate variable sequence; Combining the first cache data flow information, the current time information and the intermediate variable sequence, and transforming the intermediate result to generate the second cache data flow information; Combining the second cache data flow information with the data flow service information, generating third cache data flow information; Dynamically divide the timestamp into granularities according to business requirements to form dynamic time slices; Based on the dynamic time slice, the second cache data flow information specified by the timestamp according to the performance requirement is merged to reduce the query time of the third cache data flow information.

2. The method for data flow calculation based on dynamic time slice caching according to claim 1 is characterized in that: After the step of combining the second cached data flow information with the data flow service information to generate the third cached data flow information, the method further includes: The third cache data flow information is written into the cache of the computer memory.

3. The method for data flow calculation based on dynamic time slice caching according to claim 1, characterized in that: Before the step of dynamically dividing the timestamp into granularities according to business requirements, the method further includes: The upper and lower bounds of the timestamp are specified according to business requirements, and the timestamp is sliced ​​according to the upper and lower bounds.

4. The method for data flow calculation based on dynamic time slice caching according to claim 1, characterized in that: The data stream information includes indicator code, service field, time interval, and time unit.

5. The method for data flow calculation based on dynamic time slice caching according to claim 1, characterized in that: The intermediate variable sequence information includes a hash indicator code and a hash business field.

6. The method for data flow calculation based on dynamic time slice caching according to claim 1, characterized in that: Before the step of acquiring the intermediate variable sequence and timestamp of the data stream information in real-time streaming, the method further includes: The streaming acquisition process and streaming acquisition content are formulated through the preset program interface.

7. The method for data flow calculation based on dynamic time slice caching according to claim 6 is characterized in that: The streaming content acquisition includes real-time acquisition of a series of dynamic data aggregates.

8. The method for data flow calculation based on dynamic time slice caching according to claim 6 is characterized in that: The streaming acquisition has no restrictions on time distribution and quantity.

9. The method for data flow calculation based on dynamic time slice caching according to claim 1, characterized in that: The cache information of the computer memory includes array index, disk and main memory transmission unit, and information fixed block.

10. The method for data flow calculation based on dynamic time slice caching according to claim 1, characterized in that: The third cache data flow information is business information.

Images