Data batch processing method and device, equipment and storage medium
By combining the sliding window algorithm in batch processing, the data is divided into events and processing time windows, the problem of low efficiency of disordered data processing in the existing technology is solved, and efficient data pattern mining and high-dimensional analysis are realized.
Patent Information
- Application Number
- CN202411417740.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-11
- Publication Date
- 2025-01-10
AI Technical Summary
The existing data batch processing methods are less efficient when processing and analyzing disordered device behavior data or user behavior data, and cannot dig deeper patterns or knowledge, and cannot achieve higher-dimensional data processing and analysis.
By receiving batch requests, the target batch data is divided into multiple events, and multiple time windows are found according to the timestamp of the event. The sliding window algorithm is used to process the data in each time window, combining the acquisition of intermediate state, accumulation calculation or Aggregation computing enables efficient data processing and analysis.
Improve data processing efficiency, and can mine deep-level patterns or knowledge in batch processing to achieve higher-dimensional data processing and analysis.
Smart Images
Figure CN119293449A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a data batch processing method, device, equipment and storage medium. Background Art
[0002] Batch processing is a computing mode that refers to processing a large amount of data or executing a series of instructions at one time, rather than processing a single data or instruction in real time. Batch processing is often used in scenarios where you need to process a large amount of data, run complex computing tasks, or perform repetitive tasks.
[0003] Unordered data refers to a set of data that has no particular sort or order. That is, the elements in the data are not arranged in any particular order, such as time, size, alphabetical order, etc.
[0004] At present, data processing systems often use batch processing to process and analyze data. However, the existing data batch processing methods have low data processing efficiency when processing and analyzing disordered (the disorder refers to time disorder) device behavior data or user behavior data, and usually cannot mine deep patterns or knowledge, and cannot achieve higher-dimensional data processing and analysis. Summary of the invention
[0005] The purpose of this application is to provide a data batch processing method, device, electronic device and storage medium.
[0006] In a first aspect, an embodiment of the present application provides a data batch processing method, comprising:
[0007] receiving a batch processing request for target batch data, wherein the batch processing request carries a processing method;
[0008] According to the minimum unit of data processing by the system, the target batch data is divided into multiple events, and the start time of each event is used as the timestamp corresponding to the event;
[0009] For each of the events, find multiple time windows to which the event belongs according to the timestamp corresponding to the event;
[0010] The data in each time window is processed according to the processing method to obtain a processing result.
[0011] In a possible implementation, dividing the target batch data into multiple events according to the minimum unit of data processing by the system includes:
[0012] The target batch data is divided into a plurality of events according to a minimum unit of a preset multiple of system processing data, where the preset multiple is less than or equal to 1.
[0013] In a possible implementation, the processing method includes:
[0014] Get intermediate status, cumulative calculations, or aggregate calculations.
[0015] In a possible implementation, the method further includes:
[0016] receiving an update request for the time window, the update request including a size of the time window and a sliding step size;
[0017] The time window is updated according to the update request.
[0018] In a second aspect, an embodiment of the present application provides a data batch processing device, including:
[0019] A receiving module, configured to receive a batch processing request for target batch data, wherein the batch processing request carries a processing method;
[0020] A partitioning module, used to partition the target batch data into multiple events according to the minimum unit of data processing by the system, and use the start time of each event as the timestamp corresponding to the event;
[0021] A matching module, used for finding, for each of the events, multiple time windows to which the event belongs according to a timestamp corresponding to the event;
[0022] The processing module is used to process the data in each time window according to the processing method to obtain a processing result.
[0023] In a possible implementation, the partitioning module is specifically used to:
[0024] The target batch data is divided into a plurality of events according to a minimum unit of a preset multiple of system processing data, where the preset multiple is less than or equal to 1.
[0025] In a possible implementation, the processing method includes:
[0026] Get intermediate status, cumulative calculations, or aggregate calculations.
[0027] In a possible implementation manner, the device further includes:
[0028] An update module is used to receive an update request for the time window, wherein the update request includes the size of the time window and the sliding step; and update the time window according to the update request.
[0029] In a third aspect, the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.
[0030] In a fourth aspect, the present application provides a computer-readable storage medium having computer-readable instructions stored thereon, wherein the computer-readable instructions can be executed by a processor to implement the method as described in the first aspect.
[0031] Compared with the prior art, the data batch processing method provided by the present application receives a batch processing request for target batch data, wherein the batch processing request carries a processing method; the target batch data is divided into multiple events according to the smallest unit of data processing by the system, and the start time of each event is used as the timestamp corresponding to the event; for each of the events, the multiple time windows to which the event belongs are found according to the timestamp corresponding to the event; the data in each of the time windows is processed according to the processing method to obtain a processing result. Compared with the prior art, the present application combines a sliding window algorithm in batch processing, so that a sliding algorithm can be used in batch processing to process and analyze disordered device behavior data or user behavior data, thereby mining deep-level patterns or knowledge, with high data processing efficiency, and can achieve higher-dimensional data processing and analysis. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:
[0033] Figure 1 A flow chart of a data batch processing method provided in an embodiment of the present application is shown;
[0034] Figure 2 A schematic diagram of a sliding window algorithm provided in an embodiment of the present application is shown;
[0035] Figure 3 A schematic diagram showing a time window to which a matching event provided in an embodiment of the present application belongs;
[0036] Figure 4 A schematic diagram of a data batch processing device provided in an embodiment of the present application is shown;
[0037] Figure 5 A schematic diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION
[0038] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.
[0039] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by technicians in the field to which this application belongs.
[0040] In addition, the terms "first" and "second" etc. are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or devices.
[0041] Please refer to Figure 1 , Figure 1 1 is a flow chart of a data batch processing method provided in an embodiment of the present application, comprising the following steps S101 to S104:
[0042] S101, receiving a batch processing request for target batch data, wherein the batch processing request carries a processing method;
[0043] The execution subject of the embodiment of the present application may be a data processing system, and the data processing system may receive batch processing requests for target batch data such as batch data 1, batch data 2, batch data 3, etc.
[0044] The processing method carried in the batch processing request can be obtaining intermediate status, cumulative calculation or aggregate calculation. Obtaining intermediate status refers to obtaining the intermediate status of batch data, which usually refers to the temporary storage state of data before the final processing is completed during the batch processing process. Aggregate calculation refers to summing, calculating average, counting, etc.
[0045] S102, dividing the target batch data into multiple events according to the minimum unit of system data processing, and using the start time of each event as the timestamp corresponding to the event;
[0046] The smallest unit of data processed by the system refers to the smallest unit that the data processing system can process. In the data processing system, this unit is generally a row.
[0047] Specifically, the step of dividing the target batch data into multiple events according to the minimum unit of system processing data in step S102 can be implemented as follows: dividing the target batch data into multiple events according to the minimum unit of system processing data of a preset multiple, where the preset multiple is less than or equal to 1, for example, the preset multiple can be 1 / 2, 1, etc.
[0048] For example, the target batch data may be divided into multiple events according to the smallest unit of data processed by the system, where an event is the smallest unit of data after division. The batch data to be processed is split, and each batch data is split into the smallest unit that the system can process.
[0049] S103: for each of the events, find multiple time windows to which the event belongs according to the timestamp corresponding to the event;
[0050] The sliding window algorithm is a technique for processing continuous, fast-arriving data streams. It allows performing aggregation operations (such as summing, calculating averages, counting, etc.) on a continuously updating data stream and calculating the results within a specific time window.
[0051] Stream processing is a computing model that aims to process continuously arriving data streams in real time. This is different from batch processing, which processes pre-existing, static data sets. Stream processing can perform instant analysis and calculations on real-time data.
[0052] The existing sliding window algorithm has been maturely applied in stream processing, but in batch processing, there is no mature framework method for applying the sliding window algorithm in batch processing. The present application combines the sliding window algorithm in stream processing with batch processing, so that the sliding algorithm can be used in batch processing to process and analyze disordered device behavior data and user behavior data, so as to mine deep patterns or knowledge.
[0053] Event: represents an atomic action or state change that occurs in the system. Each event contains various information related to the action or change, such as timestamp, event type, and other contextual data.
[0054] In this application, the time window is obtained according to the preset time window parameters, which include the size of the time window and the sliding step size, for example, the size is 30 minutes and the sliding step size is 1 minute. Figure 2 Shown is a schematic diagram of the sliding window algorithm in this application.
[0055] Specifically, for each event, the event itself has an event timestamp, which represents the real time in the physical world when the event occurs. According to the event timestamp, find the N time windows to which the event belongs. For example, the data of 12:01 can belong to [11:31,12:01]....
[0056] [12:00,12:30)[12:01,12:31] and other time windows. For ease of understanding, this application provides the following Figure 3 The schematic diagram of the time window to which the matching event belongs is shown in Figure 3 As shown, after the batch data 1, 2, and 3 are divided into events, they are matched to the time windows corresponding to the events. Then, aggregation analysis can be performed on a single time window or multiple time windows to obtain analysis results.
[0057] S104: Process the data in each of the time windows according to the processing method to obtain a processing result.
[0058] Specifically, after finding the time window to which each event belongs, the data in each time window is processed according to the above processing method according to the needs of data processing or analysis, and the intermediate state or cumulative and aggregated calculation results are saved in the cache of each time window. The data in the time window refers to the data belonging to a certain time window in the target batch data.
[0059] The cache in this application refers to a memory area stored in the computer memory, which can temporarily store some data such as text, characters, values, time, etc.
[0060] When all batch data have been calculated, the status of all time windows is obtained. These time windows can be further calculated according to the needs of data processing or analysis, and subsequent data processing and analysis processes can be performed. The embodiment of the present application can use the sliding window algorithm for data processing and analysis in batch processing, thereby more efficiently realizing the sliding window calculation in batch processing, which is suitable for various time series analysis and log data aggregation scenarios.
[0061] In some embodiments of the present application, the above data batch processing method may further include the following steps:
[0062] An update request for the time window is received, wherein the update request includes a size of the time window and a sliding step; and the time window is updated according to the update request.
[0063] For example, the current time window size and sliding step size are 30 minutes and 1 minute respectively, which can be updated to 20 minutes and 1 minute as needed.
[0064] The data batch processing method provided in the embodiment of the present application has the following beneficial effects:
[0065] In the present application, a batch processing request for target batch data is received, and the batch processing request carries a processing method; the target batch data is divided into multiple events according to the smallest unit of data processing by the system, and the start time of each event is used as the timestamp corresponding to the event; for each of the events, the multiple time windows to which the event belongs are found according to the timestamp corresponding to the event; the data in each of the time windows is processed according to the processing method to obtain a processing result. Compared with the prior art, the present application combines a sliding window algorithm with batch processing, so that a sliding algorithm can be used in batch processing to process and analyze disordered device behavior data or user behavior data, so as to mine deep patterns or knowledge, with high data processing efficiency, and can achieve higher-dimensional data processing and analysis.
[0066] In the above embodiment, a data batch processing method is provided. Correspondingly, the present application also provides a data batch processing device. The data batch processing device provided in the embodiment of the present application can implement the above data batch processing method. The data batch processing device can be implemented by software, hardware, or a combination of software and hardware. For example, the data batch processing device may include integrated or separate functional modules or units to perform the corresponding steps in the above methods. Please refer to Figure 4 As shown, the data batch processing device 10 of the present application includes:
[0067] A receiving module 101 is used to receive a batch processing request for target batch data, wherein the batch processing request carries a processing method;
[0068] A division module 102 is used to divide the target batch data into multiple events according to the minimum unit of system data processing, and use the start time of each event as the timestamp corresponding to the event;
[0069] A matching module 103 is used to find, for each event, multiple time windows to which the event belongs according to a timestamp corresponding to the event;
[0070] The processing module 104 is used to process the data in each of the time windows according to the processing method to obtain a processing result.
[0071] In a possible implementation, the division module 102 is specifically configured to:
[0072] The target batch data is divided into a plurality of events according to a minimum unit of a preset multiple of system processing data, where the preset multiple is less than or equal to 1.
[0073] In a possible implementation, the processing method includes:
[0074] Get intermediate status, cumulative calculations, or aggregate calculations.
[0075] In a possible implementation manner, the device further includes:
[0076] An update module is used to receive an update request for the time window, wherein the update request includes the size of the time window and the sliding step; and update the time window according to the update request.
[0077] The data batch processing device provided in the embodiment of the present application and the data batch processing method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented therein.
[0078] An embodiment of the present application also provides an electronic device corresponding to the method provided in the aforementioned embodiment, and the electronic device may be an electronic device for a client, such as a mobile phone, a laptop computer, a tablet computer, a desktop computer, etc., to execute the above-mentioned data batch processing method.
[0079] Please refer to Figure 5 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 5 As shown, the electronic device 20 includes: a processor 200, a memory 201, a bus 202 and a communication interface 203, and the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and when the processor 200 runs the computer program, it executes the data batch processing method provided in any of the aforementioned embodiments of the present application.
[0080] The memory 201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 203 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.
[0081] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs, and the processor 200 executes the programs after receiving execution instructions. The data batch processing method disclosed in any implementation of the embodiment of the present application may be applied to the processor 200, or implemented by the processor 200.
[0082] The processor 200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 200. The above processor 200 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the above method in combination with its hardware.
[0083] The electronic device provided in the embodiment of the present application and the data batch processing method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented therein.
[0084] The present application also provides a computer-readable storage medium corresponding to the data batch processing method provided in the aforementioned embodiment, on which a computer program (i.e., a program product) is stored. When the computer program is executed by a processor, it will execute the data batch processing method provided in any of the aforementioned embodiments.
[0085] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.
[0086] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the data batch processing method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.
[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application.
Claims
1. A data batch processing method, characterized in that: include: receiving a batch processing request for target batch data, wherein the batch processing request carries a processing method; According to the minimum unit of data processing by the system, the target batch data is divided into multiple events, and the start time of each event is used as the timestamp corresponding to the event; For each of the events, find multiple time windows to which the event belongs according to the timestamp corresponding to the event; The data in each time window is processed according to the processing method to obtain a processing result.
2. The method according to claim 1, characterized in that The target batch data is divided into multiple events according to the minimum unit of data processing by the system, including: The target batch data is divided into a plurality of events according to a minimum unit of a preset multiple of system processing data, where the preset multiple is less than or equal to 1.
3. The method according to claim 1, characterized in that: The processing methods include: Get intermediate status, cumulative calculations, or aggregate calculations.
4. The method according to claim 1, characterized in that The method further comprises: receiving an update request for the time window, the update request including a size of the time window and a sliding step size; The time window is updated according to the update request.
5. A data batch processing device, characterized in that: include: A receiving module, configured to receive a batch processing request for target batch data, wherein the batch processing request carries a processing method; A partitioning module, used to partition the target batch data into multiple events according to the minimum unit of data processing by the system, and use the start time of each event as the timestamp corresponding to the event; A matching module, configured to find, for each event, multiple time windows to which the event belongs according to a timestamp corresponding to the event; The processing module is used to process the data in each time window according to the processing method to obtain a processing result.
6. The device according to claim 5, characterized in that The division module is specifically used for: The target batch data is divided into a plurality of events according to a minimum unit of a preset multiple of system processing data, where the preset multiple is less than or equal to 1.
7. The device according to claim 5, characterized in that The processing methods include: Get intermediate status, cumulative calculations, or aggregate calculations.
8. The device according to claim 5, characterized in that The device also includes: An update module is used to receive an update request for the time window, wherein the update request includes the size of the time window and the sliding step; and update the time window according to the update request.
9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 4 when executing the computer program.
10. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and the computer-readable instructions can be executed by a processor to implement the method according to any one of claims 1 to 4.