Data processing method and device based on time window, equipment and medium

Abstract

The invention discloses a data processing method and device based on time windows, equipment and a medium, and relates to the technical field of computer data processing, and the method comprises the steps: receiving each piece of to-be-processed data, determining a time window corresponding to each piece of to-be-processed data according to a timestamp corresponding to each piece of to-be-processed data, obtaining a corresponding relation between each piece of to-be-processed data and each time window; storing the to-be-processed data to the corresponding window container based on the corresponding relationship; and submitting each target to-be-processed data in the expired time window to a preset blocking queue, and acquiring and processing each target to-be-processed data in batches from the preset blocking queue by using a preset working thread group. By submitting the expired window data to the blocking queue based on the preset time interval, the system throughput and the data processing performance are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer data processing, and particularly relates to a data processing method and device based on a time window, equipment and a medium. BACKGROUND

[0002] At present, in the process of log receiving, data single-thread processing outputs a single message from the accept of a socket, and only through queue-based asynchronous processing, this mode has the problem that high-frequency calls will cause queue blocking, which will greatly reduce the performance of data acceptance.

[0003] Therefore, how to improve the data throughput of the system has become a technical problem to be solved at present. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a data processing method and device based on a time window, equipment and a medium, which can improve the system throughput and data processing performance by submitting expired window data to a blocking queue based on a preset time interval. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a data processing method based on a time window, comprising:

[0006] Using a preset data receiving thread to receive each to-be-processed data transmitted by a target network socket, and determining the time window corresponding to each to-be-processed data according to the time stamp corresponding to each to-be-processed data, to obtain the correspondence between each to-be-processed data and each time window;

[0007] Based on the correspondence, each to-be-processed data is stored in a corresponding window container; wherein the window container is a container for storing to-be-processed data corresponding to a corresponding time window;

[0008] Based on a preset time interval, each target to-be-processed data in the expired time window is submitted to a preset blocking queue, and a preset work thread group is used to batch obtain each target to-be-processed data from the preset blocking queue, and each target to-be-processed data is processed in parallel.

[0009] Optionally, the determination of the time window corresponding to each to-be-processed data according to the time stamp corresponding to each to-be-processed data comprises:

[0010] Respectively extracting the corresponding time stamp from each to-be-processed data, and respectively calculating the target remainder of each time stamp and a preset time window duration;

[0011] determining a target difference between each of the timestamps and a corresponding target remainder, and determining the time window corresponding to each of the to-be-processed data according to each of the target differences.

[0012] Optionally, before storing each of the to-be-processed data into a corresponding window container based on the correspondence, the method further comprises:

[0013] organizing each of the window containers corresponding to each of the time windows by using a concurrent hash table, wherein a key of each of the window containers is a corresponding time window identifier, and a value of each of the window containers is a corresponding list of to-be-processed data.

[0014] Optionally, submitting each of the target to-be-processed data in an expired time window to a preset blocking queue based on a preset time interval comprises:

[0015] In a current scheduling period, creating a window container corresponding to a future time window by using a preset scheduling thread, and submitting to-be-processed data corresponding to a previous time window of the current scheduling period to the preset blocking queue.

[0016] Optionally, after processing each of the target to-be-processed data in parallel, the method further comprises:

[0017] stopping receiving new to-be-processed data, and submitting all unprocessed to-be-processed data in each of the window containers to the preset blocking queue by using the preset scheduling thread.

[0018] Optionally, processing each of the target to-be-processed data in parallel comprises:

[0019] concurrently sending each of the target to-be-processed data to a target message queue by using the preset group of worker threads, so as to perform distributed computing on each of the target to-be-processed data.

[0020] In a second aspect, the present application provides a data processing apparatus based on time windows, comprising:

[0021] a time window determination module, configured to receive each of to-be-processed data transmitted by a target network socket by using a preset data receiving thread, and determine a time window corresponding to each of the to-be-processed data according to a timestamp corresponding to each of the to-be-processed data, so as to obtain a correspondence between each of the to-be-processed data and each of the time windows;

[0022] a data storage module, configured to store each of the to-be-processed data into a corresponding window container based on the correspondence; wherein the window container is a container for storing to-be-processed data corresponding to a corresponding time window.

[0023] The data processing module is configured to submit each target to-be-processed data in an expired time window to a preset blocking queue based on a preset time interval, to use a preset group of worker threads to batch acquire each target to-be-processed data from the preset blocking queue, and to concurrently process each target to-be-processed data.

[0024] Optionally, the time window determination module comprises:

[0025] The remainder calculation unit is configured to extract a corresponding time stamp from each to-be-processed data and to calculate a target remainder of each time stamp and a preset time window length.

[0026] The time window determination unit is configured to determine a target difference between each time stamp and a corresponding target remainder, and to determine a time window corresponding to each to-be-processed data according to each target difference.

[0027] In a third aspect, the present application provides an electronic device, comprising:

[0028] The memory is configured to save a computer program.

[0029] The processor is configured to execute the computer program to implement the aforementioned time window-based data processing method.

[0030] In a fourth aspect, the present application provides a computer readable storage medium configured to save a computer program, wherein the computer program is executed by a processor to implement the aforementioned time window-based data processing method.

[0031] The application first receives each to-be-processed data transmitted by a target network socket by using a preset data receiving thread, determines a time window corresponding to each to-be-processed data according to a time stamp corresponding to each to-be-processed data, to obtain a corresponding relationship between each to-be-processed data and each time window, and then stores each to-be-processed data into a corresponding window container based on the corresponding relationship; the window container is a container for storing to-be-processed data corresponding to a corresponding time window; finally, each target to-be-processed data in an expired time window is submitted to a preset blocking queue based on a preset time interval, each target to-be-processed data is batched from the preset blocking queue by using a preset work thread group, and each target to-be-processed data is processed concurrently. As can be seen, the application realizes the collection and buffering of data in the same time period by storing data into a corresponding window container, avoids high-frequency single data processing; the application realizes the asynchronous decoupling and flow control of data from the receiving link to the processing link by submitting expired window data to the blocking queue based on the preset time interval, effectively alleviates the back pressure phenomenon; the application realizes the horizontal expansion of data processing capacity by batch processing and concurrent processing of data from the blocking queue by the work thread group, converts single-thread serial processing into multi-thread parallel processing, and thus significantly improves the system throughput and data processing performance. BRIEF DESCRIPTION OF DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on the provided drawings.

[0033] Figure 1 A flow chart of a data processing method based on a time window disclosed by the present application;

[0034] Figure 2 A flow chart of a data processing method based on a time window disclosed by the present application;

[0035] Figure 3 A structure diagram of a data processing device based on a time window disclosed by the present application;

[0036] Figure 4 A structure diagram of an electronic device disclosed by the present application. DETAILED DESCRIPTION

[0037] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the protection scope of the present application.

[0038] The current log receiving mode has the problem that high-frequency calling will cause queue blocking, which will greatly reduce the data receiving performance. Therefore, the application provides a data processing method based on a time window, which submits the expired window data to the blocking queue based on a preset time interval, thereby improving the system throughput and data processing performance.

[0039] Referring to Figure 1 The embodiments of the present application disclose a data processing method based on a time window, comprising:

[0040] In step S11, a preset data receiving thread is used to receive each to-be-processed data transmitted by a target network socket, and a time window corresponding to each to-be-processed data is determined according to a time stamp corresponding to each to-be-processed data, so as to obtain a corresponding relationship between each to-be-processed data and each time window.

[0041] The present embodiment is realized based on keepalived (a service software for realizing load balancing) and udpcopy (a data replication tool), and has certain guarantee for system high availability or traffic replication. The keepalived mainly constructs a virtual ip, and two nodes equipped with the keepalived need to be mounted in the virtual ip; the udpcopy refers to a forwarding operation of copying a data packet of a current host to another node. The present embodiment is realized based on the java language, and all the technologies used are constructed based on the java.

[0042] The main purpose of the present embodiment is to distribute to different arrays through windowed calculation when accepting data, and to improve the throughput through asynchronous data processing. The specific process is as shown in Figure 2 The specific process is as shown in

[0043] When starting the service, there is an acceptance thread (i.e. a preset data receiving thread), a scheduling thread and a working thread group in the system. The blocking queue submits tasks from the acceptance thread and executes tasks from the working thread.

[0044] In this embodiment, the process of determining the time window corresponding to each to-be-processed data according to the time stamp corresponding to each to-be-processed data can specifically include: extracting the corresponding time stamp from each to-be-processed data, respectively, and calculating the target remainder of each time stamp and the target time window length, respectively; determining the target difference value between each time stamp and the corresponding target remainder, and determining the time window corresponding to each to-be-processed data according to each target difference value. Specifically,

[0045] After the data comes out of the kernel, the time stamp is calculated. The calculation formula is:

[0046] long div = now % (time * 1000);

[0047] long win = now - div;

[0048] return win;

[0049] Where now is the current time stamp, that is, the time stamp corresponding to the current to-be-processed data, and time is the time window calculated in seconds. By calculating, the scheduled thread of the timing task creates an array in ConcurrentHashMap<long, List> according to the value of win, which divides the data time stamp by the window. The above process is equivalent to finding the nearest time point that can be divided by (time * 1000) from the current time.

[0050] Through the above time window calculation, it can be ensured that the data of the same period is divided into the same window, all window boundaries are neat time points, and the scheduling thread can accurately know when to process which window.

[0051] Step S12, store each to-be-processed data to the corresponding window container based on the correspondence; wherein the window container is a container for storing to-be-processed data corresponding to the corresponding time window.

[0052] In this embodiment, before storing each to-be-processed data to the corresponding window container based on the correspondence, it further includes: organizing each window container corresponding to each time window by using a concurrent hash table; wherein the key corresponding to each window container is the identifier of the corresponding time window, and the value is the list of to-be-processed data.

[0053] That is, the embodiment uses the get operation of the hash table to construct the window container corresponding to each time window. It should be noted that the window container in this embodiment can be in the form of an array, or other structures that can be used for data storage.

[0054] Through window calculation, the data is divided by the accepted time window, the window data is processed in batches, a certain delay is exchanged for multi-thread concurrent processing, and the log processing capacity is improved.

[0055] In step S13, each target to-be-processed data in the expired time window is submitted to the preset blocking queue based on a preset time interval, each target to-be-processed data is batched from the preset blocking queue by using a preset work thread group, and concurrent processing is performed on each target to-be-processed data.

[0056] In this embodiment, submitting each target to-be-processed data in the expired time window to the preset blocking queue based on a preset time interval includes: in the current scheduling period, a window container corresponding to a future time window is created by using a preset scheduling thread at a regular time, and to-be-processed data corresponding to a time window in a previous scheduling period is submitted to the preset blocking queue.

[0057] That is, the scheduling thread needs to newly create window arrays of the next two time units at a regular time, and submit data of the previous two time units to the blocking queue for processing. It should be noted that the number of newly created and submitted arrays in this embodiment can be dynamically set according to actual needs, and is not specifically limited here.

[0058] It should be noted that the "expired time window" in this embodiment refers to a time window whose time range has ended, no longer receives new data, and its data can be submitted to a subsequent link for processing. Assuming that the time window size in this embodiment is 1 second, and the system stipulates that it is submitted 1 second after the end of a certain window (that is, the processing delay is 1 second):

[0059] Current time: 2023-10-27 10:00:05.500 (millisecond-level timestamp 1698393605500);

[0060] Current active window: 2023-10-27 10:00:05.000 to 2023-10-27 10:00:06.000 (timestamp identifier 1698393605000); this window is receiving data generated at the current time (10:00:05.500).

[0061] For window W1: 10:00:04.000 - 10:00:05.000 (identified as 1698393604000); the end time (10:00:05.000) of the window is earlier than the current time (10:00:05.500). The specified 1-second processing delay has been met (0.5 seconds have passed from the end of the window, 10:00:05.000, to the current time, 10:00:05.500, although it is not yet 1 second, the scheduling thread can be preparing to commit W1. Under the strict 1-second delay, the window of 10:00:03.000 - 10:00:04.000 is the only expired and ready window). That is, W1 is an "expired time window" at this time. The scheduling thread takes out the data in the window container corresponding thereto and submits the data in batches to the blocking queue.

[0062] By submitting each target to-be-processed data corresponding to an expired time window to the preset blocking queue, it is ensured that a window is processed after the window is closed and no new data is added. This avoids new data from arriving during processing, resulting in incomplete data or complex processing logic. "Delay for throughput" is achieved: instead of processing one piece of data at a time, the system waits for all data in a window to arrive and processes the data as a batch, greatly improving efficiency; at the same time, a clear time point is provided for scheduling and processing of the system. The scheduling thread only needs to process the "expired" windows after a fixed time interval.

[0063] In this embodiment, the data in the blocking queue is processed through a business process, and the data is taken out in batches, so that the CPU cache hit rate can be improved through batch processing to improve performance.

[0064] In addition, in this embodiment, after the concurrent processing of the target to-be-processed data, the method further includes: stopping receiving new to-be-processed data, and submitting the unprocessed to-be-processed data in each window container to the preset blocking queue by using a preset scheduling thread.

[0065] That is, when the service is closed, the receiving thread is stopped. Then, the remaining data in the window is submitted. The threads are gradually closed. The service is completed.

[0066] In addition, in this embodiment, the concurrent processing of the target to-be-processed data includes: concurrently sending each target to-be-processed data to a target message queue by using a preset group of work threads, so as to perform distributed computing on each target to-be-processed data.

[0067] It can be seen that, by means of storing data to corresponding window containers, the application realizes the collection and buffering of data in the same time period, avoiding high-frequency single data processing; by means of submitting expired window data to a blocking queue based on a preset time interval, the application realizes the asynchronous decoupling and flow control of data from the receiving link to the processing link, effectively relieving the back pressure phenomenon; by means of batch acquisition and concurrent processing of the target data by the working thread group from the blocking queue, the application realizes the horizontal expansion of data processing capacity, converts single-thread serial processing into multi-thread parallel processing, thereby significantly improving the system throughput and data processing performance.

[0068] Referring to Figure 3 The embodiment of the application discloses a time window-based data processing device, which comprises:

[0069] A time window determination module 11 is configured to receive each to-be-processed data transmitted by a target network socket using a preset data receiving thread, and determine a time window corresponding to each to-be-processed data according to a time stamp corresponding to each to-be-processed data, so as to obtain a corresponding relationship between each to-be-processed data and each time window.

[0070] A data storage module 12 is configured to store each to-be-processed data in a corresponding window container based on the corresponding relationship; wherein the window container is a container for storing to-be-processed data corresponding to a corresponding time window.

[0071] A data processing module 13 is configured to submit each target to-be-processed data in an expired time window to a preset blocking queue based on a preset time interval, batch acquire each target to-be-processed data from the preset blocking queue using a preset working thread group, and perform concurrent processing on each target to-be-processed data.

[0072] In some specific embodiments, the time window determination module can specifically comprise:

[0073] A remainder calculation unit is configured to extract a corresponding time stamp from each to-be-processed data, and calculate a target remainder of each time stamp and a preset time window duration.

[0074] A time window determination unit is configured to determine a target difference between each time stamp and a corresponding target remainder, and determine a time window corresponding to each to-be-processed data according to each target difference.

[0075] In some specific embodiments, the data storage module 12 further comprises:

[0076] The window container organization unit is configured to organize each window container corresponding to each time window by using a concurrent hash table, wherein a key of each window container is a corresponding time window identifier, and a value is a corresponding to-be-processed data list.

[0077] In some embodiments, the data processing module 13 can specifically include:

[0078] The first data submission unit is configured to create a window container corresponding to a future time window by using a preset scheduling thread at a preset time interval in a current scheduling period, and submit to-be-processed data corresponding to a previous time window of the current scheduling period to the preset blocking queue.

[0079] In some embodiments, the data processing module 13 further includes:

[0080] The second data submission unit is configured to stop receiving new to-be-processed data, and submit all unprocessed to-be-processed data in each window container to the preset blocking queue by using the preset scheduling thread.

[0081] In some embodiments, the data processing module 13 can specifically include:

[0082] The data sending unit is configured to send each target to-be-processed data to a target message queue by using the preset group of working threads, so as to perform distributed computing on each target to-be-processed data.

[0083] Further, the embodiment of the present application further discloses an electronic device, Figure 4 The electronic device 20 is shown in accordance with an exemplary embodiment, and the content in the figure cannot be considered as any limitation on the use range of the present application.

[0084] Figure 4 A structural schematic diagram of an electronic device 20 is provided in the embodiment of the present application. The electronic device 20 can specifically include at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is configured to store a computer program, and the processor 21 is configured to load and execute the computer program to implement the related steps in the time window-based data processing method disclosed in any of the preceding embodiments. In addition, the electronic device 20 in the embodiment can be specifically an electronic computer.

[0085] In this embodiment, the power supply 23 is configured to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 is configured to create a data transmission channel between the electronic device 20 and external devices, and the communication protocol followed by the communication interface 24 can be any communication protocol applicable to the technical solution of the present application, which will not be specifically limited herein; the input / output interface 25 is configured to obtain external input data or output data to the outside, and the specific interface type can be selected according to the specific application needs, which will not be specifically limited herein.

[0086] In addition, the memory 22 as a carrier for storing resources can be a read-only memory, a random access memory, a magnetic disk or an optical disk, etc., and the resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage mode can be temporary storage or permanent storage.

[0087] The operating system 221 is configured to manage and control each hardware device on the electronic device 20 and the computer program 222, and can be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program capable of completing the time window-based data processing method executed by the electronic device 20 disclosed in any of the preceding embodiments, the computer program 222 can further include a computer program capable of completing other specific work.

[0088] Further, the present application also discloses a computer readable storage medium for storing a computer program; wherein the computer program is executed by a processor to implement the time window-based data processing method disclosed above. For the specific steps of the method, please refer to the corresponding content disclosed in the preceding embodiments, which will not be described here.

[0089] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. For the same or similar parts between the embodiments, please refer to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and please refer to the method part for the relevant part.

[0090] The skilled person can further realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of the two. In order to clearly show the interchangeability of hardware and software, the components and steps of each example have been described in the above description. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0091] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM, flash memory, ROM, electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. The

[0092] Finally, it should be noted that the terms "first", "second", and the like, herein do not denote any order, quantity, combination, or importance, but rather are used to distinguish one element from another, and do not imply or require any such actual relationship or order. Moreover, the terms "include", "contain", or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements does not include only those elements recited, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by the phrase "comprising a" does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0093] The above provides a detailed description of the technical solutions of the present application. The principles and implementation modes of the present application are described by applying specific examples. The above description of the embodiments is only to help understand the method and its core idea of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range can be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A data processing method based on a time window, characterized in that, include: The system uses a preset data receiving thread to receive each piece of data to be processed transmitted from the target network socket, and determines the time window corresponding to each piece of data to be processed based on the timestamp corresponding to each piece of data to be processed, so as to obtain the correspondence between each piece of data to be processed and each time window. Based on the correspondence, each of the data to be processed is stored in a corresponding window container; wherein, the window container is a container used to store the data to be processed corresponding to the corresponding time window; Based on a preset time interval, each target data to be processed in the expired time window is submitted to a preset blocking queue. A preset group of worker threads is used to retrieve each target data to be processed in batches from the preset blocking queue and to process each target data to be processed concurrently.

2. The data processing method based on a time window according to claim 1, characterized in that, The step of determining the time window corresponding to each of the data to be processed based on the timestamps corresponding to each of the data to be processed includes: Extract the corresponding timestamps from each of the data to be processed, and calculate the target remainder between each timestamp and the preset time window duration. The target difference between each timestamp and the corresponding target remainder is determined, and the time window corresponding to each piece of data to be processed is determined based on each target difference.

3. The data processing method based on a time window according to claim 1, characterized in that, Before storing each of the data to be processed into the corresponding window container based on the correspondence, the method further includes: A concurrent hash table is used to organize the window containers corresponding to each time window; wherein the key of each window container is the corresponding time window identifier, and the value is the corresponding list of data to be processed.

4. The data processing method based on a time window according to claim 1, characterized in that, The step of submitting each target data to be processed within the expired time window to a preset blocking queue based on a preset time interval includes: Within the current scheduling cycle, a window container corresponding to the future time window is created periodically using a preset scheduling thread, and the data to be processed corresponding to the previous time window of the current scheduling cycle is submitted to the preset blocking queue.

5. The data processing method based on a time window according to claim 4, characterized in that, After performing concurrent processing on each of the target data to be processed, the method further includes: Stop receiving new pending data, and use the preset scheduling thread to submit all unprocessed pending data in each window container to the preset blocking queue.

6. The data processing method based on a time window according to any one of claims 1 to 5, characterized in that, The concurrent processing of each of the target data to be processed includes: The preset working thread group is used to send each of the target data to be processed concurrently to the target message queue so as to perform distributed computing on each of the target data to be processed.

7. A data processing device based on a time window, characterized in that, include: The time window determination module is used to receive each piece of data to be processed transmitted by the target network socket using a preset data receiving thread, and determine the time window corresponding to each piece of data to be processed according to the timestamp corresponding to each piece of data to be processed, so as to obtain the correspondence between each piece of data to be processed and each time window. A data storage module is used to store each of the data to be processed into a corresponding window container based on the correspondence; wherein, the window container is a container used to store the data to be processed corresponding to a corresponding time window; The data processing module is used to submit each target data to be processed in the expired time window to a preset blocking queue based on a preset time interval, use a preset working thread group to retrieve each target data to be processed in batches from the preset blocking queue, and perform concurrent processing on each target data to be processed.

8. The data processing device based on a time window according to claim 7, characterized in that, The time window determination module includes: The remainder calculation unit is used to extract the corresponding timestamps from each of the data to be processed, and to calculate the target remainder between each timestamp and the preset time window duration. The time window determination unit is used to determine the target difference between each timestamp and the corresponding target remainder, and to determine the time window corresponding to each piece of data to be processed based on each target difference.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the time window-based data processing method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the time-window-based data processing method as described in any one of claims 1 to 6.

Images