Service data processing method and related device

Abstract

The invention discloses a business data processing method and a related device, and the method comprises the steps: obtaining a to-be-processed structured query statement, analyzing the structured query statement, and determining a target table value function corresponding to the structured query statement; wherein the target table value function predefines a storage list and a batch processing quantity threshold matched with the storage list; obtaining to-be-processed service data, and determining a snapshot signal corresponding to the service data; and storing the business data to the storage list by using the target table value function, and obtaining a processing result matched with the business data based on the snapshot signal, the business data in the storage list and the corresponding batch processing quantity threshold. In this way, the efficiency and flexibility of business data processing can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a business data processing method and related devices. Background Art

[0002] With the advent of the big data era, the amount of data has exploded, and traditional data processing frameworks are gradually unable to meet the needs of real-time and large-scale data processing. In order to solve these problems, Apache Flink came into being. Flink is an open source stream processing framework designed for distributed data stream processing and batch processing. It provides low latency, high throughput, and exactly-once state consistency guarantees, becoming an important component in modern data architecture. In the data processing process, Flink uploads each business data to the processor separately according to the custom table-valued function for calculation. Since only a single data is processed each time, the parallel processing capability of the processor cannot be brought into play, resulting in low overall throughput; even by adding batch logic in the custom table-valued function, the business data is uploaded to the processor in batches to improve the computing efficiency, but when the volume of a single business data is large, it will bring about the problem of too high resource occupation by the task snapshot.

[0003] In view of this, how to improve the flexibility of business data processing has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a business data processing method and related devices, which can improve the efficiency and flexibility of business data processing.

[0005] In order to solve the above technical problems, a technical solution adopted in the present application is: to provide a business data processing method, which is applied to a target processing engine, and the method includes: obtaining a structured query statement to be processed, parsing the structured query statement, and determining a target table-valued function corresponding to the structured query statement; wherein the target table-valued function predefines a storage list and a batch processing quantity threshold for matching the storage list; obtaining business data to be processed, and determining a snapshot signal corresponding to the business data; using the target table-valued function to store the business data in the storage list, and obtaining a processing result of the business data matching based on the snapshot signal, the business data in the storage list and its corresponding batch processing quantity threshold.

[0006] To solve the above technical problems, another technical solution adopted in the present application is: to provide a business data processing device, comprising: a memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the method mentioned in the above technical solution.

[0007] The beneficial effects of the present application are as follows: Different from the prior art, the business data processing method proposed in the present application determines the target table-valued function by parsing the structured query statement, and uses the target table-valued function to store the acquired business data in the corresponding storage list. A snapshot signal matching the business data is generated according to the target interval duration, and whether the preset execution conditions are met is determined according to the snapshot signal or the number of business data in the storage list. When the preset execution conditions are met, the business data in the storage list is processed to obtain the corresponding processing results. This method combines the snapshot signal and the number of business data in the storage list to batch process the business data while avoiding excessive amounts of business data in the storage list, thereby improving the flexibility of business data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them: Figure 1 It is a flowchart of an implementation method of the business data processing method of the present application; Figure 2 yes Figure 1 Step S101 corresponds to a flow chart of another implementation method; Figure 3 yes Figure 1 Step S102 corresponds to a flow chart of another implementation method; Figure 4 yes Figure 1 Step S103 corresponds to a flow chart of another implementation method; Figure 5 yes Figure 4 Step S403 corresponds to a flowchart of another implementation method; Figure 6 It is a structural diagram of an implementation method of a business data processing device of the present application. DETAILED DESCRIPTION

[0009] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments, and different embodiments can be adaptively combined. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0010] See also Figure 1 , Figure 1 This is a flow chart of an implementation method of a business data processing method of the present application. The method is applied to a target processing engine, and the specific implementation process includes: S101: Obtain a structured query statement to be processed, parse the structured query statement, and determine a target table-valued function corresponding to the structured query statement; wherein the target table-valued function predefines a storage list and a batch processing quantity threshold for storage list matching.

[0011] In one implementation, a structured query statement to be processed is obtained, and the query statement is parsed to determine its corresponding target table-valued function.

[0012] In one implementation scenario, the target processing engine is a Flink engine, and the structured query statement obtained is a SQL (Structured Query Language) statement. And the determined target table-valued function is a UDTF (User-DefinedTable-Generating Function). The target table-valued function is defined by a user or a relevant technical staff, that is, the target table-valued function is pre-defined with a storage list and a batch processing quantity threshold matching the storage list, and the target table-valued function is used to implement batch processing of subsequent business data.

[0013] S102: Acquire the business data to be processed, and determine the snapshot signal corresponding to the business data.

[0014] In one embodiment, the business data to be processed is acquired in real time, and a snapshot signal matching the current business data is generated at a target interval, and the generated snapshot information is added to the current business data. The snapshot signal is used to generate a corresponding snapshot in the subsequent processing of the business data, and the target interval can be set according to the actual scenario.

[0015] In one implementation scenario, the business data may be a bounded data stream or an unbounded data stream. In addition, the business data may also be at least one of data from a database, real-time data in the Internet of Things, and log data.

[0016] S103: Using the target table-valued function to store the business data in a storage list, and obtaining a processing result of business data matching based on the snapshot signal, the business data in the storage list and its corresponding batch processing quantity threshold.

[0017] In one embodiment, during the process of processing the business data, the target table value function is used to store the business data one by one into a storage list corresponding to the target table value function. According to the snapshot signal, the business data in the storage list and its corresponding batch processing quantity threshold, a processing result of the business data matching is obtained.

[0018] Specifically, when the business data matches the corresponding snapshot signal, a corresponding snapshot is generated when the business data is processed, so that the data can be restored according to the generated snapshot when a failure occurs in the subsequent processing process. Further, the business data is stored in the storage list using the target table value function. For the business data in the current storage list, it is detected whether there is a matching snapshot or whether the number reaches the batch processing number threshold. If so, the business data currently stored in the storage list is processed to obtain the corresponding processing result.

[0019] The business data processing method proposed in the present application parses the structured query statement to determine the target table-valued function, and uses the target table-valued function to store the acquired business data in the corresponding storage list. A snapshot signal matching the business data is generated according to the target interval duration, and whether the preset execution conditions are met is determined according to the snapshot signal or the number of business data in the storage list. When the preset execution conditions are met, the business data in the storage list is processed to obtain the corresponding processing results. This method combines the snapshot signal and the number of business data in the storage list to batch process the business data while avoiding excessive amounts of business data in the storage list, thereby improving the flexibility of business data processing.

[0020] See also Figure 2 , Figure 2 yes Figure 1 Step S101 in the flowchart corresponds to another embodiment. Specifically, the implementation process of step S101 includes: S201: Obtain a structured query statement, parse the structured query statement, and obtain a corresponding parsed text.

[0021] In one implementation, a structured query statement is obtained, and the structured query statement is parsed using a target processing engine to obtain a corresponding parsed text.

[0022] Specifically, a configuration instruction triggered by a user or a relevant technical staff is obtained, and the configuration instruction includes a generation logic related to the target table-valued function. According to the configuration instruction, a structured query statement containing the target table-valued function is written. The generation logic is to store the business data input into the target processing engine one by one into the corresponding storage list, and when the number of business data in the storage list reaches the batch processing number threshold, all business data in the storage list are processed.

[0023] S202: In response to the parsed text including the target table-valued function, multiple executable operators corresponding to the structured query statement are obtained based on the parsed text; wherein the multiple executable operators include an extended operator corresponding to the target table-valued function.

[0024] In one embodiment, the target processing engine includes a code text writing module. In response to the target table value function defined by the user or relevant technical personnel in the parsed text, the code text writing module is called to generate an extended code text that meets the generation logic based on the target table value function. Based on the parsed text and the code text, multiple executable operators are converted.

[0025] Specifically, the code text writing module is set in the target processing engine in the form of a plug-in. In response to the parsed text including the target table-valued function, the code text writing module is called according to the generation logic corresponding to the target table-valued function to generate the corresponding extended code text, and the generated code text is added to the parsed text. The parsed text including the code text is converted to obtain multiple executable operators corresponding to the target processing engine. Among them, the multiple executable operators are respectively used to process the business data subsequently acquired, and the multiple executable operators include an extended operator corresponding to the target table-valued function.

[0026] The above scheme generates the code text corresponding to the target table-valued function by utilizing the code text writing module, thereby converting the target table-valued function into the corresponding executable operator, so that subsequent business data is stored in the storage list after passing through the executable operator corresponding to the target table-valued function, thereby realizing batch processing of at least one business data and improving the efficiency of business data processing.

[0027] In another embodiment, step S101 may not be combined with the configuration instruction in the process of obtaining the structured query statement, that is, the obtained structured query statement is not combined with the target table-valued function. Based on this, after parsing the structured query statement and obtaining the corresponding parsed text, the code text writing module is directly called to generate the code text corresponding to the target table-valued function according to the preset generation logic. The generated code text is added to the parsed text, and the parsed text including the code text is converted to obtain multiple executable operators corresponding to the target processing engine.

[0028] See also Figure 3 , Figure 3 yes Figure 1 Step S102 in the flowchart corresponds to another embodiment. Specifically, in response to the target processing engine including a snapshot signal sending module, the implementation process of step S102 includes: S301: Acquire initial data to be processed.

[0029] In one embodiment, the initial data to be processed is acquired in real time, wherein the initial data is data that needs to be processed or analyzed in real time.

[0030] S302: Generate a snapshot signal matching the current initial data at a target interval using a snapshot signal sending module, and use the initial data including the snapshot signal as service data.

[0031] In one embodiment, the target processing engine includes a snapshot signal sending module of a pre-device, and the snapshot signal is generated by the snapshot signal sending module, and the generated snapshot signal is added to the initial data to obtain the business data.

[0032] Specifically, the snapshot signal sending module is set as a plug-in in the target processing engine. The target interval duration is predetermined, and the snapshot signal is generated by the snapshot signal sending module at each target interval duration, and the generated snapshot signal is added to the business data. The target interval duration can be set according to the actual scenario, and the corresponding snapshot signal is generated by each target interval duration to avoid the excessive amount of business data in the subsequent storage list affecting the performance of data processing.

[0033] See also Figure 4 , Figure 4 yes Figure 1 Step S103 in the flowchart corresponds to another implementation. Specifically, the target processing engine includes a batch processing module that matches the storage list. Based on this, the implementation process of step S103 includes: S401: Process the business data in sequence using executable operators.

[0034] In one implementation, the business data is processed in sequence using the acquired executable operators.

[0035] Specifically, after obtaining the business data to be processed, the business data is processed in sequence using multiple executable operators corresponding to the target processing engine. In addition, in response to any executable operator receiving a business data matching snapshot signal, a corresponding snapshot is generated. The generated snapshot is used to batch process the business data in the storage list, and is also used for data recovery when a failure occurs.

[0036] S402: In response to the extended operator corresponding to the target table-valued function receiving the processed business data, the processed business data is stored in a storage list.

[0037] In one implementation, in response to an expansion operator corresponding to the target table-valued function receiving processed business data, the expansion operator is used to store the received business data into a corresponding storage list.

[0038] In a specific application scenario, the executable operators corresponding to the target processing engine obtained based on the method mentioned in the above corresponding implementation manner include operator A, operator B, operator C and operator D, and operator C is an extended operator corresponding to the target table-valued function. After obtaining the business data, the business data is processed by operator A and operator B in turn, and when operator C obtains the business data processed by operator B, the processed business data is stored in the storage list.

[0039] In another embodiment, in response to the extended operator corresponding to the target table-valued function receiving processed business data, the extended operator is used to assign a business data storage identifier, and after all executable operators corresponding to the target processing engine process the business data, the business data is stored in the corresponding storage list according to the storage identifier of the business data.

[0040] In a specific application scenario, the executable operators corresponding to the target processing engine obtained based on the method mentioned in the corresponding implementation method include operator A, operator B, operator C and operator D, and operator C is the extended operator corresponding to the target table-valued function. After obtaining the business data, the business data is processed by operator A and operator B in turn, and when operator C obtains the business data processed by operator B, it assigns a storage identifier to the processed business data. In addition, the business data processed by operator C is input to operator D. When operator D completes the processing of the business data, the target processing engine stores the business data with the storage identifier in the storage list.

[0041] S403: In response to the business data in the storage list satisfying a preset processing condition, the business data in the storage list is input into a batch processing module in the target processing engine to obtain a processing result.

[0042] In one embodiment, in response to satisfying a preset processing condition, all the business data stored in the storage list are input into a batch processing module in a target processing engine, so that the batch processing module processes the input business data to obtain a corresponding processing result. The preset processing condition is related to the snapshot matched by the business data in the storage list or the number of business data in the storage list.

[0043] In a specific application scenario, when the business data is view business data, the batch processing module is a graphics processing unit (GPU) capable of processing the business data.

[0044] The above scheme triggers preset processing conditions in different ways, so as to avoid excessive amount of business data in the storage list while batch processing the business data, thereby improving the efficiency and flexibility of business data processing.

[0045] See also Figure 5 , Figure 5 yes Figure 4 Step S403 in the flowchart corresponds to another implementation method. Specifically, the implementation process of step S403 includes: S501: Detect whether the business data in the storage list meets the batch execution condition or the snapshot execution condition; wherein the batch execution condition is determined based on the batch processing quantity threshold, and the snapshot execution condition is determined based on the snapshot signal matched by the business data.

[0046] In one implementation, when storing the service data in the storage list, it is detected whether the service data in the storage list meets the batch execution condition or meets the snapshot execution condition.

[0047] Specifically, to improve the processing efficiency of business data, at least part of the business data processed by the executable terminal is stored in a storage list, and when the amount of business data in the storage list is greater than or equal to the batch processing amount threshold, it is determined that the batch execution condition is met.

[0048] Alternatively, when storing business data in the storage list, the currently stored business data is checked to determine whether the business data matches a corresponding snapshot. If the business data matches a snapshot, the snapshot execution condition is triggered. If the business data does not match a snapshot, the subsequent business data is processed using the executable operator and stored in the storage list until the latest business data stored in the storage list matches a snapshot or the number of business data in the storage list reaches the batch processing quantity threshold.

[0049] It should be noted that, in actual application, when any one of the snapshot execution condition and the batch execution condition is met, it is determined that the preset processing condition mentioned in step S403 is met.

[0050] S502: In response to the business data in the storage list satisfying the batch execution condition or the snapshot execution condition in advance, all the business data in the storage list are input into the batch processing module to obtain a processing result.

[0051] Specifically, when any one of the batch execution condition or the snapshot execution condition is met first, all the business data in the storage list are input into the batch processing module to obtain the corresponding processing result.

[0052] In one embodiment, in response to the number of business data in the storage list reaching a batch processing number threshold, a batch execution condition is triggered, and all business data in the storage list is input into a batch processing module to obtain a processing result.

[0053] In another embodiment, in response to the business data in the storage list matching a snapshot corresponding to any processing operator, the snapshot execution condition is triggered, and all the business data in the storage list are input into the batch processing module to obtain the processing result.

[0054] In one implementation scenario, a synchronization interface is provided in the target processing engine, and the synchronization interface is provided in the code text writing module or coupled with the code text encoding module. When the parsed text includes a target table value function, the synchronization interface is triggered. In response to the synchronization interface being in a triggered state, in the process of processing the business data, the synchronization interface is used to detect whether the business data in the storage list matches a corresponding snapshot. When the business data in the storage list matches a snapshot corresponding to any processing operator, the snapshot execution condition is triggered through the synchronization interface to input all the business data in the storage list into the batch processing module to obtain a processing result. In addition, when the business data in the storage list matches a snapshot, the above-mentioned synchronization interface is also used to save the current state of the corresponding business data, thereby improving the security and reliability of the data.

[0055] Furthermore, after all the business data in the storage list are input into the batch processing module and the corresponding processing results are obtained, the business data in the storage list is cleared and the process returns to step S103, that is, the business data subsequently acquired is processed by the executable operator through the method mentioned in the above corresponding implementation mode and then stored in the storage list.

[0056] Optionally, in another embodiment, the synchronization interface may also be used only to save the current state of the corresponding business data when it is detected that the business data in the storage list matches the corresponding snapshot, that is, relevant technical personnel can configure the function of the synchronization interface to improve the flexibility of processing business data.

[0057] The present application also proposes a target processing engine, which is used to implement the business data processing method mentioned in any of the above-mentioned embodiments. A synchronization interface is provided in the target processing engine, and the synchronization interface is used to store the state of the business data or batch process the business data according to the snapshot signal corresponding to the business data.

[0058] See also Figure 6 , Figure 61 is a schematic diagram of the structure of an embodiment of a business data processing device of the present application. The business data processing device includes: a memory 10 and a processor 20 coupled to each other. Among them, the memory 10 stores program data, and the processor 20 is used to call the program data to execute the business data processing method mentioned in any of the above embodiments. Specifically, the electronic device includes but is not limited to: a desktop computer, a laptop computer, a tablet computer, a server, etc., which are not limited here. In addition, the processor 20 can also be called a CPU (Center Processing Unit). The processor 20 may be an integrated circuit chip with signal processing capabilities. The processor 20 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 20 can be implemented by an integrated circuit chip.

[0059] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0060] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0061] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0062] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0063] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A business data processing method, characterized in that: The business data processing method is applied to a target processing engine, and the method comprises: Obtaining a structured query statement to be processed, parsing the structured query statement, and determining a target table-valued function corresponding to the structured query statement; wherein the target table-valued function predefines a storage list and a batch processing quantity threshold for matching the storage list; Acquire the business data to be processed, and determine the snapshot signal corresponding to the business data; The target table-valued function is used to store the business data in the storage list, and based on the snapshot signal, the business data in the storage list and the corresponding batch processing quantity threshold, a processing result of the business data matching is obtained.

2. The business data processing method according to claim 1, characterized in that: The step of obtaining a structured query statement to be processed, parsing the structured query statement, and determining a target table-valued function corresponding to the structured query statement includes: Obtaining the structured query statement, parsing the structured query statement, and obtaining a corresponding parsed text; In response to the parsed text including the target table-valued function, a plurality of executable operators corresponding to the structured query statement are obtained based on the parsed text; wherein the plurality of executable operators include an extended operator corresponding to the target table-valued function.

3. The business data processing method according to claim 2, characterized in that: The target processing engine includes a code text writing module, and the code text writing module is provided with generation logic, and the response to the parsed text including the target table-valued function, based on the parsed text, a plurality of executable operators corresponding to the structured query statement are obtained, including: Based on the target table-valued function, calling the code text writing module to generate an extended code text that satisfies the generation logic; Based on the parsed text and the code text, a plurality of executable operators are obtained.

4. The business data processing method according to claim 1, characterized in that: The target processing engine includes a snapshot signal sending module, and the acquiring of the business data to be processed and determining the snapshot signal corresponding to the business data include: Get the initial data to be processed; The snapshot signal sending module is used to generate a snapshot signal matching the current initial data at target intervals, and the initial data including the snapshot signal is used as the service data.

5. The business data processing method according to claim 2, characterized in that: The target processing engine includes a batch processing module that matches the storage list, and the target table value function is used to store the business data in the storage list, and based on the snapshot signal, the business data in the storage list and the corresponding batch processing quantity threshold, a processing result of the business data matching is obtained, including: Processing the business data in sequence using the executable operators; In response to the extended operator corresponding to the target table-valued function receiving the processed business data, storing the processed business data in the storage list; In response to the business data in the storage list satisfying a preset processing condition, the business data in the storage list is input into the batch processing module to obtain the processing result.

6. The business data processing method according to claim 5, characterized in that: In response to the business data in the storage list satisfying a preset processing condition, inputting the business data in the storage list into the batch processing module to obtain the processing result, including: Detecting whether the business data in the storage list meets a batch execution condition or a snapshot execution condition; wherein the batch execution condition is determined based on the batch processing quantity threshold, and the snapshot execution condition is determined based on the snapshot signal matched by the business data; In response to the business data in the storage list satisfying the batch execution condition or the snapshot execution condition in advance, all the business data in the storage list are input into the batch processing module to obtain the processing result.

7. The business data processing method according to claim 6, characterized in that: In response to the business data in the storage list satisfying the batch execution condition in advance, inputting all the business data in the storage list into the batch processing module to obtain the processing result, including: In response to the quantity of the business data in the storage list reaching a batch processing quantity threshold, the batch execution condition is triggered, and all the business data in the storage list are input into the batch processing module to obtain the processing result.

8. The business data processing method according to claim 6, characterized in that: The using the executable operators to process the business data in sequence includes: When any of the executable operators receives the snapshot signal matching the business data, a corresponding snapshot is generated; In response to the business data in the storage list satisfying the snapshot execution condition in advance, inputting all the business data in the storage list into the batch processing module to obtain the processing result, including: In response to detecting that the business data currently stored in the storage list matches the snapshot, the snapshot execution condition is triggered, and all the business data in the storage list are input into the batch processing module to obtain the processing result.

9. A target processing engine, characterized in that: The target processing engine is used to implement the business data processing method described in any one of claims 1-8. A synchronization interface is provided in the target processing engine, and the synchronization interface is used to store the state of the business data or batch process the business data according to a snapshot signal corresponding to the business data.

10. A business data processing device, characterized in that: include: A memory and a processor coupled to each other, wherein the memory stores program data, and the processor calls the program data to execute the business data processing method according to any one of claims 1-8.

Images