SQL statement processing method, device, medium and electronic device

Abstract

The present invention provides a method, device, medium and electronic device for processing SQL statements. In the scheme, in the process of big data processing, by adding a window offset and a periodic output interval in the SQL statement in the task, in the process of processing the SQL statement, the start time of the data processing window is determined according to the window offset, and the trigger of the periodic output result is determined according to the periodic output interval. After the data enters the window operation, the intermediate result of the data processing is output according to the start time of the window and the trigger. The scheme supports adjusting the start time of the window and can grasp the change of the data in real time by periodically outputting the intermediate results.

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a method, device, medium and electronic device for processing structured query language (SQL) statements. Background Art

[0002] In recent years, with the development of big data technology, excellent big data computing engine frameworks such as Storm, Spark, and Flink have emerged. Among them, Flink's position in big data has become increasingly important. In the field of big data, SQL is also the most user-friendly. It is a user-oriented application programming interface (API) layer. In the traditional streaming computing field, Storm and Spark Streaming provide some Function or Datastream APIs. Users write business logic in Java or Scala. This method has some problems, such as having a certain threshold and difficulty in tuning. With the continuous update of versions, many incompatibilities have appeared in the API. Flink SQL is a set of development languages ​​that conform to standard SQL semantics designed by Flink real-time computing to simplify the computing model and lower the threshold for users to use real-time computing. At present, Spark and Flink are actively turning to the implementation of SQL. In the SQL streaming computing engine, there is an important concept called window. The above two big data computing engines support the use of windows, such as sliding windows, rolling windows, and session windows.

[0003] However, the current technical solution cannot support adjusting the start time of the window, and cannot grasp the status of intermediate results in real time, and cannot grasp the changes in data in real time. Summary of the invention

[0004] The embodiments of the present invention provide a method, device, medium and electronic device for processing SQL statements to solve the problems in the prior art that the start time of the window cannot be adjusted, the intermediate result status cannot be grasped in real time, and the change of data cannot be grasped in real time.

[0005] In a first aspect, an embodiment of the present invention provides a method for processing an SQL statement, comprising:

[0006] Obtaining a structured query language SQL statement to be processed, wherein the SQL statement includes a window offset and a periodic output interval;

[0007] Determine the start time of the data processing window according to the window offset;

[0008] Determine a trigger for a periodic output result according to the periodic output interval;

[0009] After the data enters the window operation, the intermediate result of the data processing is output according to the start time of the window and the trigger.

[0010] In a specific embodiment, the method further comprises:

[0011] The time field is output each time the intermediate result of data processing is output.

[0012] In a specific embodiment, the method further comprises:

[0013] A time attribute in the data processing task to which the SQL statement belongs is obtained, where the time attribute is used to indicate that the window time field type is processing time or event time.

[0014] In a specific implementation, the time attribute is used to indicate that the window time field type is processing time, then after the data enters the window operation, according to the start time of the time window and the trigger, the intermediate result of the data processing is output, including:

[0015] After the data enters the window operation, registering a timer according to the trigger and the system time, wherein the system time is the system time when the data enters the window operation;

[0016] triggering the timer at the first integer multiple of the periodic output interval after the system time;

[0017] After the timer time is reached, the callback function is triggered to output the intermediate result of data processing, and the timer is triggered again after the periodic output interval, and this step is repeated until the triggering time of the timer exceeds the end time of the window.

[0018] In a specific implementation, the time attribute is used to indicate that the window time field type is event time, then after the data enters the window operation, according to the start time of the time window and the periodic output interval, the intermediate result of the data processing is output, including:

[0019] After the data enters the window operation, registering a timer according to the trigger and the data time, the data time being the time included in the data;

[0020] triggering the timer after the periodic output interval after the window;

[0021] After the timer time is reached, the callback function is triggered to output the intermediate result of data processing, and the timer is triggered again after the periodic output interval, and this step is repeated until the triggering time of the timer exceeds the end time of the window.

[0022] In a specific implementation, the SQL statement also includes a time field and a window size;

[0023] Accordingly, determining the start time of the data processing window according to the window offset includes:

[0024] The start time of the window for data processing is determined based on the time field and the window offset.

[0025] In a specific embodiment, the method further comprises:

[0026] An end time of the window for data processing is determined according to the start time and the window size.

[0027] In a specific embodiment, the method further comprises:

[0028] A data processing task sent by a WEB client is received, wherein the data processing task includes the SQL statement and the time attribute.

[0029] In a second aspect, an embodiment of the present invention provides a device for processing SQL statements, including:

[0030] An acquisition module, used for acquiring a SQL statement to be processed, wherein the SQL statement includes a window offset and a periodic output interval;

[0031] A processing module, used to determine the start time of the data processing window according to the window offset;

[0032] The processing module is also used to determine a trigger of a periodic output result according to the periodic output interval;

[0033] The output module is used to output the intermediate result of data processing according to the start time of the window and the trigger after the data enters the window operation.

[0034] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0035] processor, interactive interface; and,

[0036] A memory, configured to store executable instructions of the processor;

[0037] The processor is configured to execute the SQL statement processing method described in any one of the first aspects by executing the executable instructions.

[0038] In a fourth aspect, an embodiment of the present invention provides a storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for processing an SQL statement as described in any one of the first aspects.

[0039] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the SQL statement processing method described in any one of the first aspects.

[0040] The SQL statement processing method, device, medium and electronic device provided by the embodiments of the present invention, in the process of big data processing, by adding a window offset and a periodic output interval in the SQL statement in the task, in the process of processing the SQL statement, the start time of the data processing window is determined according to the window offset, and the trigger of the periodic output result is determined according to the periodic output interval. After the data enters the window operation, the intermediate result of the data processing is output according to the window start time and the trigger. This solution supports adjusting the start time of the window and can grasp the change of data in real time by periodically outputting the intermediate results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 A flowchart of a first embodiment of a method for processing SQL statements provided by the present invention;

[0043] Figure 2 A flowchart of a second embodiment of a method for processing SQL statements provided by the present invention;

[0044] Figure 3 A flowchart of a third embodiment of a method for processing SQL statements provided by the present invention;

[0045] Figure 4 A structural diagram of a first embodiment of a device for processing SQL statements provided by the present invention;

[0046] Figure 5 It is a structural schematic diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all the embodiments. Based on the embodiment of the present invention, all other embodiments made by ordinary technicians in this field under the enlightenment of this embodiment belong to the scope of protection of the present invention.

[0048] In the specification and other parts of the present invention, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0049] Spark and Flink provided in the existing technology are actively turning to SQL implementation, but there are still the following problems:

[0050] (1) Flink SQL supports Tumble Window, Hop Window, and Session Window, but cannot support customizing the start time of the window.

[0051] (2. The windows supported by Flink SQL cannot customize the interval parameters, and then periodically output the intermediate aggregation results of the unfinished windows according to the interval. Therefore, it is impossible to grasp the status of the intermediate results in real time, and it is impossible to display reports in real time.

[0052] (3) In Flink SQL, if the window size is at the daily level (a window size that is a multiple of 24 hours), there will be errors in aggregation statistics. For example, if the window size is 24 hours, there are two data points 2020-01-01 06:30:30 and 2020-01-01 16:30:30. Assuming that there are only two data points on 2020-01-01, the correct result of the aggregation statistics of the data volume in the window should be 2. However, due to the East 8 time zone issue, 2020-01-01 06:30:30 and 020-01-01 16:30:30 cannot be correctly assigned to the windows (2020-01-01 00:00:00, 2020-01-02 00:00:00), so the aggregation result is wrong.

[0053] In the above content, it should be understood that Flink is an open source computing engine under Apache that can handle stream tasks, batch tasks, and support SQL consumption data processing and persistence to external storage systems.

[0054] Time zone: The Flink SQL platform runs in Kubernetes, and the default time zone is East 8. When calculating a window that spans multiple days, the start time of the window is from 8:00 a.m. each day to 8:00 a.m. the next day, which results in errors in aggregate statistics. The start and end times of windows that span multiple days are normal.

[0055] In view of the above problems, the present invention provides a method for processing SQL statements, which supports adjusting the start time of a rolling window and a sliding window in an Event Time\Processint Time scenario, supports periodic output of the interval between the two windows, and can solve the problem of aggregate statistical errors in windows of daily size.

[0056] The following is a method for processing SQL statements provided by the present application through several specific embodiments.

[0057] Figure 1 A flowchart of a method for processing SQL statements provided by the present invention is shown in FIG. Figure 1 As shown, the SQL statement processing method can be applied to electronic devices with data processing functions, such as servers, cloud servers, computers, etc. Specifically, the SQL statement processing method includes the following steps:

[0058] S101: Obtain an SQL statement to be processed, where the SQL statement includes a window offset and a periodic output interval.

[0059] Generally speaking, when users use SQL related functions, the rolling window only supports two parameters: time field and window size, and the sliding window only supports three parameters: time field, sliding interval, and window size.

[0060] In this step, for the definition and processing of SQL statements, two additional parameters are introduced while being compatible with native ones: window offset and periodic output interval. When users enter SQL statements, they include window offset and periodic output interval. These two parameters are used to adjust the start time of the window and can be used to support the intermediate output of data processing.

[0061] In a specific implementation of this solution, the user can input the SQL statement through a client or a web, and the electronic device receives the SQL statement sent by the client or the web.

[0062] For the electronic device, a data processing task sent by a WEB client is received, wherein the data processing task includes the SQL statement and the time attribute. The time attribute is used to indicate that the window time field type is processing time or event time.

[0063] S102: Determine the start time of the data processing window according to the window offset.

[0064] In this step, after obtaining the SQL statement, the electronic device can determine the start time of the data processing window according to the window offset therein. The increased window offset can change the start time and end time of the window.

[0065] In a specific implementation, the electronic device may determine the start time of the window for data processing based on the time field and the window offset.

[0066] Window offset refers to the operation of moving the window forward or backward in the time dimension, thereby changing the start and end time of the window. For example, if the window size is 1 hour and the data time is 2020-01-0112:30:00, then the window is (2020-01-01 12:00:00, 2020-01-01 13:00:00). By introducing a window offset of 5 minutes, the window time can be made (2020-01-01 12:05:00, 2020-01-0113:05:00). Alternatively, if the time field is 0:00 to 24:00, after obtaining the window offset of 8h, the time window can be determined to be 8:00 to 8:00 of the next day, and the new window start time is obtained.

[0067] S103: Determine a trigger for a periodic output result according to the periodic output interval.

[0068] In this step, the electronic device can determine a trigger based on the periodic output interval, and the trigger is used to trigger the callback process, and send the intermediate results of data processing periodically with the periodic output interval as a period. The trigger is used to periodically trigger the output of the intermediate results and is used to register the timing scheduler. According to the registered time, the callback function is automatically triggered at the time. This function is mainly used to output the intermediate results of the window.

[0069] S104: After the data enters the window operation, the intermediate result of the data processing is output according to the start time and trigger of the window.

[0070] In this step, after the start time and trigger of the window are determined, the result of data processing is periodically output based on the start time and trigger after the data enters the window operation. In this solution, due to different types of time, the specific callback processing based on the trigger is different.

[0071] In this solution, the meaning of periodic output of window intermediate results is: for example, if the window size is 24 hours, the window will only output results at the end of the window. In this solution, in order to display the intermediate results of the window in real time, you can customize a periodic time interval parameter (incrementInterval). For example, if the size is 1 hour, the current result of the window will be output once every 1 hour.

[0072] In a specific implementation, the electronic device outputs a time field each time it outputs an intermediate result of data processing, where the time field is used to indicate the time when the corresponding intermediate result is obtained.

[0073] In a specific implementation of this step, if the time attribute in the data processing task to which the SQL statement belongs indicates that the window time field type is processing time, then after the data enters the window operation, a timer is registered according to the trigger and the system time, and the system time is the system time when the data enters the window operation.

[0074] The timer is triggered at the first integer multiple of the periodic output interval after the system time, the callback function is triggered after the timer time is reached, the intermediate result of data processing is output, and the timer is triggered again after the periodic output interval, and this step is repeated until the triggering time of the timer exceeds the end time of the window.

[0075] In another specific implementation of this step, if the time attribute in the data processing task to which the SQL statement belongs indicates that the window time field type is event time, then after the data enters the window operation, a timer is registered according to the trigger and the data time, where the data time is the time included in the data;

[0076] The timer is triggered after the periodic output interval after the window; the callback function is triggered after the timer time is reached, the intermediate result of data processing is output, and the timer is triggered again after the periodic output interval, and this step is repeated until the triggering time of the timer exceeds the end time of the window.

[0077] The SQL statement processing method provided in this embodiment, in the process of big data processing, by adding a window offset and a periodic output interval in the SQL statement in the task, in the process of processing the SQL statement, the start time of the data processing window is determined according to the window offset, and the trigger of the periodic output result is determined according to the periodic output interval. After the data enters the window operation, the intermediate result of the data processing is output according to the window start time and the trigger. This solution supports adjusting the start time of the window and can grasp the changes of the data in real time by periodically outputting the intermediate results.

[0078] Figure 2 A flowchart of a second embodiment of a method for processing SQL statements provided by the present invention is shown in FIG. Figure 2 As shown, based on the above embodiment, this embodiment provides an implementation scheme of a specific SQL statement processing method, including the following steps:

[0079] S201: Obtain an input SQL statement, which includes a time field, a window size, a window offset, and a periodic output interval.

[0080] S202: Parse the SQL statement, which includes a time field, a window size, a window offset, and a periodic output interval.

[0081] In this step, the SQL statement is parsed so that the two parameters (window offset and periodic output interval) newly added in the window function TUMBLE in the SQL can be correctly identified.

[0082] In a specific implementation, the relevant functions using SQL at the user level are as follows:

[0083] Introduction to rolling window function:

[0084] TUMBLE(time field, window size)

[0085] TUMBLE(time field, window size, window offset)

[0086] TUMBLE(time field, window size, window offset, periodic output interval)

[0087] The rolling window only supports two parameters (time field, window size). This solution introduces two more parameters: window offset and periodic output interval while being compatible with the native solution. The above three methods are supported to use the rolling window.

[0088] The window offset can change the start time and end time of the window. If there is a periodic output interval, the intermediate result output of the window can be sent to the downstream regularly according to the interval size.

[0089] Introduction to sliding window function:

[0090] HOP (time field, sliding interval, window size)

[0091] TUMBLE (time field, sliding interval, window size, window offset)

[0092] TUMBLE (time field, sliding interval, window size, window offset, periodic output interval)

[0093] The implementation principle of sliding window is the same as that of rolling window.

[0094] S203: Adjust the start time of the window according to the window offset.

[0095] S204: Determine a trigger for a periodic output result according to the periodic output interval.

[0096] In the above two steps, after the window offset and the periodic output interval are obtained, the start time and end time of the adjusted window can be obtained according to the time field, the window size and the window offset.

[0097] Determine the implementation of the trigger based on the periodic output interval:

[0098] Processing Time: Under the semantics of processing time, the periodic output of the intermediate results of the window is realized by registering a timer. The timer triggers the callback function and then outputs the result. At the same time, a new timer is registered. The interval of the timer is fixed, that is, the size of the periodic output interval set in the window function.

[0099] Event Time: Under the semantics of event time, it is different from processing time. It is determined by event time. Events are processed data, and the data specifies a field of a certain time type as the event time of the data. Timers are also registered. The output of the intermediate results of the window is triggered only when the time contained in the data is greater than or equal to the timer time. It is completely driven by events and has nothing to do with system time.

[0100] S205: Binding the time field of the trigger window output result in the window optimization rule.

[0101] S206: Output the intermediate results of the window.

[0102] In the above steps, the window optimization rule is pre-configured, and it is set to output the time field each time the intermediate result is output, so that the intermediate result and the time corresponding to the generation of the intermediate result can be obtained at the same time.

[0103] This solution provides a method in which users can customize the start time of the time window by setting the window offset. At the same time, they can customize the output interval by setting the periodic output interval. During the processing, the intermediate aggregation results of the window are output, so as to grasp the status of the intermediate results in real time.

[0104] Figure 3 A flowchart of a third embodiment of a method for processing SQL statements provided by the present invention is shown in FIG. Figure 3 As shown, based on the above embodiment, this embodiment provides a more specific implementation steps of a method for processing SQL statements:

[0105] S301: Window function (time field, window size, window offset, periodic output interval).

[0106] S302: Parse the parameters of the window function in the SQL statement, and obtain the parameter values ​​if there are four parameters.

[0107] In this solution, it should be understood that the implementation of the window offset is mainly introduced through the function parameters at the SQL level, and then the corresponding offset parameters are configured through SQL parsing to the specific window. Finally, the corresponding window allocator will be used in the window operator to allocate windows for the data. The allocated window is implemented through the following public statement. By introducing the offset, the start time of the window can be changed. Because the window size is fixed, you only need to know the start time to get the window end time.

[0108] For example: public static long getwindowStartWithOffset(long timestamp, longoffset, longwindowSize){

[0109] Return timestamp-(timestamp-Offset+windowSize)%windowSize;

[0110] }

[0111] Among them, timestamp: the time in the data, the specific value of the time field set in the window function; offset: the window offset size set in the window function; windowSize: the window size set in the window function.

[0112] S303: Determine whether it is processing time or event time according to the window time attribute.

[0113] S304: Event time.

[0114] In the context of the invention, event time is the time at which each event occurs on its production device. This time is typically embedded in the record before entering Flink, and the event timestamp can be extracted from each record. In event time, the progression of time depends on the data, not any clock. Event time programs must specify how to generate event time watermarks, which are a mechanism for representing the progression of event time.

[0115] S305: Processing time.

[0116] In the scheme of the present invention, processing time refers to the system time of the machine executing the corresponding operation. When the stream processing program runs based on processing time, all time-based operations (such as time windows) will use the system clock of the machine running the corresponding operator. The hourly processing time window will include all records that arrived at a specific operator between the whole hour indicated by the system clock. For example, if the application starts running at 9:15 am, the first hourly processing time window will include events processed between 9:15 and 10:00 am, the next window will include events processed between 10:00 and 11:00 am, and so on.

[0117] S306: Create a trigger for periodically outputting results.

[0118] S307: Data enters the window operation.

[0119] S308: After allocating a window for the data, a corresponding timer is registered using a trigger.

[0120] S309: Event time: register a timer according to the data time.

[0121] S310: Processing time: registering a timer according to the system time.

[0122] S311: The timer automatically calls back when it expires and outputs the intermediate results of the window.

[0123] like Figure 3 As shown in the figure, during the processing of the SQL statement and the specific data processing, the algorithm of the trigger registration timer (Timer) is:

[0124] (1) Processing Time:

[0125] Assuming that the window where a piece of data is located starts at start and ends at end, the system time corresponding to the data is timestamp, and the periodic output interval is incrementInterval, then two timers will be registered at the beginning.

[0126] The trigger time of the first Timer is: start+incrementInterval*(1+(timestamp-start) / incrementInterval); the trigger time of the second Timer is: end.

[0127] The second Timer trigger time above is when the window ends, which is the last time the window result is output. The first Timer trigger time is the first integer multiple of incrementInterval after the current data enters the system time of the window operation. When the first Timer time arrives, the callback function is triggered, thereby triggering the output of the intermediate results. At the same time, a new Timer is registered to achieve periodic output of the intermediate results. The new Timer trigger time = the time of the Timer that triggers the callback + the periodic output interval size incrementInterval. If the new Timer trigger time exceeds the end time of the window, it will not be registered.

[0128] (2) Event Time:

[0129] Data under event time will also register two timers at the beginning. The time of the first timer is: start+incrementInterval, and the time of the second timer is end. The callback function is automatically triggered when the timer expires. The specific function is the same as processint time, but the time algorithm for registering a new timer is different. The new timer trigger time = start+incrementInterval*(1+(ctx.getCurrentWatermark()-start) / incrementInterval), and the timer will be registered only if it does not exceed the end time. In Flink, the window result output under event time is driven by watermark, that is, when the watermark time exceeds the timer time, the timer will be triggered to execute the callback function and output the corresponding intermediate results.

[0130] In order to avoid registering unnecessary timers, it is ensured that the registered timer will not be lower than the current watermark time.

[0131] In the specific implementation of this solution, in this way, if it is based on Processing time, then the callback can be directly triggered when the timer time arrives. If it is Event Time, then the timer trigger depends on the watermark. Only when the time represented by the watermark exceeds the timer time, the timer will be called back. Watermark is a concept of the flink computing engine, which is mainly used in the scenario of windows under event time. After each data arrives at the flink engine, the time will be extracted from the data and then saved with the watermark. Then the watermark is sent downstream regularly. When the watermark passes through the window, if the time represented by the watermark exceeds the end time of the window, or the timer time registered for the incremental window, the output of the window result will be triggered.

[0132] In the specific implementation of this solution, when using the window function TUMBLE (time field, window size) instead of TUMBLE (time field, window size, window offset), for example, the window size is 24 hours and the time field is rowtime, the previous usage method is: TUMBLE (rowtime, INTERVAL'24'HOUR) is changed to TUMBLE (rowtime, INTERVAL'24'HOUR, INTERVAL'-8'HOUR) to solve the problem of window statistics errors. For example: for a day-level window, that is, a window size of 24 hours or 48 hours, if the data time is 2020-01-21 06:00:00, then the window to which this data belongs is (2020-01-20 08:00:00, 2020-01-21 08:00:00), but this data actually belongs to the window (2020-01-21 00:00:00, 2020-01-22 By introducing the offset, the window can be corrected to obtain the correct window for this data (2020-01-21 00:00:00, 2020-01-22 00:00:00).

[0133] On the whole, the core points of the technical solution provided by this solution are: introducing window offset and periodic output interval, realizing periodic output of window intermediate results for two different time semantics of Event Time and Processing Time, and realizing corresponding triggers (Trigger), as well as how the SQL-level API is bound to the SQL optimization bottom layer, correctly parsing and identifying the two parameters of the introduced periodic output interval (IncrementInterval (time interval for periodic output of window results)) and offset (window offset), thereby solving the problem in the prior art that the start time of the window cannot be customized and the intermediate results cannot be mastered in real time, and at the same time, it can also avoid the problem of errors in the aggregation results of the data in the window.

[0134] Figure 4 A structural diagram of a first embodiment of a SQL statement processing device provided by the present invention, such as Figure 4 As shown, the SQL statement processing device 10 also includes:

[0135] An acquisition module 11 is used to acquire a SQL statement to be processed, wherein the SQL statement includes a window offset and a periodic output interval;

[0136] A processing module 12, configured to determine a start time of a data processing window according to the window offset;

[0137] The processing module 12 is also used to determine a trigger for a periodic output result according to the periodic output interval;

[0138] The output module 13 is used to output the intermediate result of data processing according to the start time of the window and the trigger after the data enters the window operation.

[0139] Based on the above embodiment, the output module 13 is further used for:

[0140] The time field is output each time the intermediate result of data processing is output.

[0141] In the specific implementation of the above embodiment, the acquisition module 11 is also used for:

[0142] A time attribute in the data processing task to which the SQL statement belongs is obtained, where the time attribute is used to indicate that the window time field type is processing time or event time.

[0143] Optionally, the time attribute is used to indicate that the window time field type is processing time, and the output module 13 is specifically used to:

[0144] After the data enters the window operation, registering a timer according to the trigger and the system time, wherein the system time is the system time when the data enters the window operation;

[0145] triggering the timer at the first integer multiple of the periodic output interval after the system time;

[0146] After the timer time is reached, the callback function is triggered to output the intermediate result of data processing, and the timer is triggered again after the periodic output interval, and this step is repeated until the triggering time of the timer exceeds the end time of the window.

[0147] Optionally, the time attribute is used to indicate that the window time field type is event time, and the output module 13 is specifically used to:

[0148] After the data enters the window operation, registering a timer according to the trigger and the data time, the data time being the time included in the data;

[0149] triggering the timer after the periodic output interval after the window;

[0150] After the timer time is reached, the callback function is triggered to output the intermediate result of data processing, and the timer is triggered again after the periodic output interval, and this step is repeated until the triggering time of the timer exceeds the end time of the window.

[0151] Optionally, the SQL statement also includes a time field and a window size;

[0152] Accordingly, the processing module 12 is specifically used for:

[0153] The start time of the window for data processing is determined based on the time field and the window offset.

[0154] Optionally, the processing module 12 is further used for:

[0155] An end time of the window for data processing is determined according to the start time and the window size.

[0156] Optionally, the acquisition module 11 is further used for:

[0157] A data processing task sent by a WEB client is received, wherein the data processing task includes the SQL statement and the time attribute.

[0158] The SQL statement processing device provided in any of the above embodiments is used to execute the technical solution of any of the above method embodiments, and its implementation principle and technical effect are similar. By adding a window offset and a periodic output interval in the SQL statement, during the processing of the SQL statement, the start time of the data processing window is determined according to the window offset, and the trigger of the periodic output result is determined according to the periodic output interval. After the data enters the window operation, the intermediate result of the data processing is output according to the window start time and the trigger. It supports adjusting the start time of the window and can grasp the changes of the data in real time by periodically outputting the intermediate results.

[0159] Figure 5 Schematic diagram of the structure of an electronic device provided by the present invention. Figure 5 As shown, an electronic device 20 provided in this embodiment includes:

[0160] Processor 21, interactive interface 23; and,

[0161] A memory 22, used to store executable instructions of the processor;

[0162] The processor 21 is configured to execute the SQL statement processing method provided by any of the aforementioned method embodiments by executing the executable instructions.

[0163] Optionally, the memory 22 may be independent or integrated with the processor 21 .

[0164] When the memory 22 is a device independent of the processor 21, the electronic device 20 may further include:

[0165] The bus 24 is used to connect the processor 21 and the memory 22 .

[0166] An embodiment of the present invention further provides a storage medium, in which a computer program is stored. When the computer program is executed by a processor, the method for processing SQL statements provided by any of the above method embodiments is implemented.

[0167] The embodiment of the present invention further provides a computer program product, which includes a computer program stored in a storage medium. At least one processor can read the computer program from the readable storage medium, and at least one processor executes the computer program to implement the SQL statement processing method provided by any of the above method embodiments.

[0168] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing SQL statements, characterized in that: include: Obtaining a structured query language SQL statement to be processed, wherein the SQL statement includes a window offset and a periodic output interval; Determine the start time of the data processing window according to the window offset; Determine a trigger for a periodic output result according to the periodic output interval; After the data enters the window operation, outputting the intermediate result of the data processing according to the start time of the window and the trigger; The method further comprises: Obtaining a time attribute in the data processing task to which the SQL statement belongs, wherein the time attribute is used to indicate that the window time field type is processing time or event time; Wherein, after the data enters the window operation, outputting the intermediate result of the data processing according to the start time of the window and the trigger includes: The timer is triggered at the preset time after the data enters the window operation. Different window time field types correspond to different preset times. After the timer time is reached, the callback function is triggered to output the intermediate result of the data processing.

2. The method according to claim 1, characterized in that The method further comprises: The time field is output each time the intermediate result of data processing is output.

3. The method according to claim 1, characterized in that The time attribute is used to indicate that the window time field type is processing time, and before the preset time after the data enters the window operation triggers the timer, it also includes: Register a timer according to the trigger and the system time, wherein the system time is the system time when the data enters the window operation; The preset time is the periodic output interval that is the first integer multiple of the system time; After outputting the intermediate result of the data processing, the method further includes: triggering the timer again after the periodic output interval, and repeating this step until the triggering time of the timer exceeds the end time of the window.

4. The method according to claim 1, characterized in that: The time attribute is used to indicate that the window time field type is an event time, and before the preset time after the data enters the window operation triggers the timer, it also includes: registering a timer according to the trigger and the data time, the data time being the time included in the data; The preset time is the periodic output interval after the window; After outputting the intermediate result of the data processing, the method further includes: triggering the timer again after the periodic output interval, and repeating this step until the triggering time of the timer exceeds the end time of the window.

5. The method according to claim 1 or 2, characterized in that: The SQL statement also includes a time field and a window size; Accordingly, determining the start time of the data processing window according to the window offset includes: The start time of the window for data processing is determined based on the time field and the window offset.

6. The method according to claim 5, characterized in that The method further comprises: An end time of the window for data processing is determined according to the start time and the window size.

7. The method according to claim 1, characterized in that The method further comprises: A data processing task sent by a WEB client is received, wherein the data processing task includes the SQL statement and the time attribute.

8. A SQL statement processing device, characterized in that: include: An acquisition module, used for acquiring a SQL statement to be processed, wherein the SQL statement includes a window offset and a periodic output interval; A processing module, used to determine the start time of the data processing window according to the window offset; The processing module is also used to determine a trigger of a periodic output result according to the periodic output interval; An output module, used for outputting the intermediate result of data processing according to the start time of the window and the trigger after the data enters the window operation; The acquisition module is further used for: Obtaining a time attribute in the data processing task to which the SQL statement belongs, wherein the time attribute is used to indicate that the window time field type is processing time or event time; Wherein, the output module is specifically used for: The timer is triggered at the preset time after the data enters the window operation. Different window time field types correspond to different preset times. After the timer time is reached, the callback function is triggered to output the intermediate result of the data processing.

9. An electronic device, characterized in that: include: Processor, interactive interface; as well as, A memory, configured to store executable instructions of the processor; The processor is configured to execute the SQL statement processing method according to any one of claims 1 to 7 by executing the executable instructions.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for processing the SQL statement according to any one of claims 1 to 7 is implemented.

11. A computer program product, characterized in that It comprises a computer program, which is used to implement the SQL statement processing method according to any one of claims 1 to 7 when being executed by a processor.

Images