A data processing method and device based on stream computing

Abstract

The application provides a data processing method and device based on stream computing. When the utilization rate of the storage space in the local memory of the stream computing node is less than a first preset threshold, the stream computing node obtains stream data to be calculated from a data source, calculates the stream data to be calculated according to a calculation mode specified in a stream computing task, and stores the calculation result in the local memory of the stream computing node. Since the utilization rate of the storage space in the local memory of the stream computing node is less than the first preset threshold, that is, the local memory of the stream computing node has sufficient free storage space, the calculation result can be successfully stored in the local memory of the stream computing node, and the stream computing node will not be constantly attempting to store the calculation result in the local memory of the stream computing node due to storage failure, and thus the stream computing node will not be down or dead due to multiple unsuccessful storage.

Description

Technical Field

[0001] This application relates to the field of streaming computing, and in particular to a data processing method and apparatus based on streaming computing. Background Technology

[0002] With the rapid development of computer technology, the data production rate on the network is getting faster and faster. Correspondingly, the requirements for the timeliness of data processing are also getting higher and higher. For example, in some high-concurrency scenarios, e-commerce needs to count quantities in real time, such as real-time statistics on the transaction volume or transaction amount of goods. Summary of the Invention

[0003] This application discloses a data processing method and apparatus based on streaming computing.

[0004] Firstly, a data processing method based on streaming computing is shown, applied to one of multiple streaming computing nodes in a streaming computing cluster, comprising: in a scenario where the streaming computing node processes streaming data, detecting whether the utilization rate of the storage space of the local memory of the streaming computing node is greater than or equal to a first preset threshold; if the utilization rate of the storage space of the local memory of the streaming computing node is greater than or equal to the first preset threshold, determining whether the tolerance of the scenario for processing timeliness is higher than or equal to a preset tolerance; if the tolerance of the scenario for processing timeliness is higher than or equal to the preset tolerance, interrupting the acquisition of streaming data to be computed from the data source, wherein the data source is used to store the streaming data to be computed, and the streaming data in the data source is used to be computed by the streaming computing node through a computation method specified in the streaming computing task; If the utilization rate of the local storage space of the streaming computing node is less than a first preset threshold, the streaming data to be computed is then obtained from the data source. The streaming data to be computed is computed according to the computing method specified in the streaming computing task, and the computing result is stored in the local storage of the streaming computing node. Alternatively, if the tolerance for processing timeliness in the scenario is lower than a preset tolerance, an allocation request is submitted to the control node in the streaming computing cluster. The allocation request is used to request more storage space to be allocated to the local storage of the streaming computing node. If the control node allocates more storage space to the local storage of the streaming computing node according to the allocation request, the streaming data to be computed is obtained from the data source. The streaming data to be computed is computed according to the computing method specified in the streaming computing task, and the computing result is stored in the allocated storage space.

[0005] Secondly, a data processing method based on streaming computing is shown, applied to a control node outside of multiple streaming computing nodes in a streaming computing cluster. The method includes: receiving allocation requests submitted by streaming computing nodes, the allocation requests requesting more storage space for the local memory of the streaming computing nodes; the allocation request is sent to the control node when, in a scenario where the streaming computing nodes are processing streaming data, the utilization rate of the storage space in the local memory of the streaming computing nodes is detected to be greater than or equal to a first preset threshold, and it is determined that the tolerance for processing timeliness in the scenario is lower than a preset tolerance; or, the allocation request is sent to the control node when it is detected that the utilization rate of the storage space in the local memory of the streaming computing nodes is greater than or equal to a first preset threshold, and it is determined that the tolerance for processing timeliness in the scenario is lower than a preset tolerance. If the utilization rate of the local storage space of the streaming computing node is greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario for processing timeliness is higher than a preset tolerance, after the interruption obtains the streaming data to be calculated from the data source, and it is determined that the duration of the interruption in obtaining the streaming data to be calculated from the data source is greater than the preset duration corresponding to the scenario, the interruption is sent to the control node; according to the allocation request, more storage space is allocated to the local storage of the streaming computing node so that the streaming computing node obtains the streaming data to be calculated from the data source, calculates the streaming data to be calculated according to the calculation method specified in the streaming computing task, and stores the calculation results in the allocated storage space.

[0006] Thirdly, a data processing method based on streaming computing is shown, applied to a development node, comprising: obtaining computer program code developed for a streaming computing task; the computer program code including at least: a first table creation statement and a write statement; the first table creation statement including at least a table creation instruction, a first table name, and a first data storage type; the first table creation statement instructing the creation of a first database table with the first table name according to the table creation instruction and instructing the data storage type applicable to the first database table to be the first data storage type; the first data storage type includes an external storage type; the write statement including at least a write instruction and the first table name, instructing the writing of calculation results to the first database table according to the write instruction and the first table name; searching for the first data storage type in the first table creation statement; replacing the first data storage type in the first table creation statement with a pre-set third data storage type to obtain new computer program code, the third data storage type including a local storage type; generating a streaming computing task based on the new computer program code, the streaming computing task being suitable for both offline verification environments and online environments.

[0007] Fourthly, a data processing method based on streaming computing is shown, applied to one of multiple streaming computing nodes in a streaming computing cluster. The method includes: receiving a request sent by a development node, the request being used to retrieve computation results stored in the local memory of the streaming computing node; the computation results are obtained when, in a scenario where the streaming computing node processes streaming data, it is detected that the utilization rate of the storage space in the local memory of the streaming computing node is greater than or equal to a first preset threshold, and it is determined that the tolerance for processing timeliness in the scenario is greater than or equal to a preset tolerance. In this case, the retrieval of the streaming data to be computed from the data source is interrupted, and the retrieval is resumed when the utilization rate of the storage space in the local memory of the streaming computing node is less than the first preset threshold. The streaming data to be computed is then computed according to the computation method specified in the streaming computing task and stored in the free space of the local memory of the streaming computing node; the data source is used to store... The system stores streaming data to be computed. The streaming data in the data source is used by streaming computing nodes to compute the data using the computation method specified in the streaming computing task. Alternatively, the computation result is obtained when, in a scenario where the streaming computing node processes streaming data, it detects that the utilization rate of the storage space in the local memory of the streaming computing node is greater than or equal to a first preset threshold, and it is determined that the tolerance for processing timeliness in the scenario is lower than a preset tolerance. In this case, an allocation request is submitted to the control node in the streaming computing cluster. The allocation request requests more storage space for the local memory of the streaming computing node. If the control node allocates more storage space to the local memory of the streaming computing node according to the allocation request, the system retrieves the streaming data to be computed from the data source, computes the data according to the computation method specified in the streaming computing task, and stores it in the allocated storage space. The system then reads the computation result stored in the local memory of the streaming computing node and sends the read computation result to the development node.

[0008] Fifthly, a data processing method based on streaming computing is shown, applied to a development node, comprising: sending an acquisition request to one of multiple streaming computing nodes in a streaming computing cluster, the acquisition request being used to acquire computation results stored in the local memory of the streaming computing node; wherein the computation results are obtained when, in a scenario where the streaming computing node processes streaming data, it is detected that the utilization rate of the storage space in the local memory of the streaming computing node is greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario for processing timeliness is greater than or equal to a preset tolerance, the acquisition of streaming data to be computed from the data source is interrupted, and the acquisition of streaming data to be computed is resumed when the utilization rate of the storage space in the local memory of the streaming computing node is less than the first preset threshold, the streaming data to be computed is then acquired from the data source, and the streaming data to be computed is computed according to the computation method specified in the streaming computing task and stored in the free space of the local memory of the streaming computing node. The data source is used to store the streaming data to be computed. The streaming data in the data source is used by the streaming computing nodes to compute the data using the computation method specified in the streaming computing task. Alternatively, the computation result is that in the scenario where the streaming computing node processes streaming data, if the utilization rate of the storage space in the local memory of the streaming computing node is detected to be greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario for processing timeliness is lower than a preset tolerance, an allocation request is submitted to the control node in the streaming computing cluster. The allocation request is used to request the allocation of more storage space for the local memory of the streaming computing node. And if the control node allocates more storage space for the local memory of the streaming computing node according to the allocation request, the streaming data to be computed is obtained from the data source, computed according to the computation method specified in the streaming computing task, and stored in the allocated storage space. The computation result returned by the streaming computing node according to the acquisition request is received.

[0009] Sixthly, a data processing apparatus based on streaming computing is shown, applied to one of multiple streaming computing nodes in a streaming computing cluster, comprising: a detection module, used to detect whether the utilization rate of the local storage space of the streaming computing node is greater than or equal to a first preset threshold in a scenario where the streaming computing node processes streaming data; a determination module, used to determine whether the tolerance of the scenario for processing timeliness is higher than or equal to a preset tolerance when the utilization rate of the local storage space of the streaming computing node is greater than or equal to the first preset threshold; and an interruption module, used to interrupt the acquisition of streaming data to be computed from a data source when the tolerance of the scenario for processing timeliness is higher than or equal to the preset tolerance, wherein the data source is used to store the streaming data to be computed, and the streaming data in the data source is used by the streaming computing node to process the data through a streaming computing task specified in the streaming computing task. The calculation method is as follows: If the utilization rate of the local storage space of the streaming computing node is less than a first preset threshold, the streaming data to be calculated is then obtained from the data source. The streaming data to be calculated is then calculated according to the calculation method specified in the streaming computing task, and the calculation result is stored in the local storage of the streaming computing node. The first submission module is used to submit an allocation request to the control node in the streaming computing cluster when the tolerance for processing timeliness in the scenario is lower than a preset tolerance. The allocation request is used to request more storage space to be allocated to the local storage of the streaming computing node. If the control node allocates more storage space to the local storage of the streaming computing node according to the allocation request, the streaming data to be calculated is obtained from the data source. The streaming data to be calculated is then calculated according to the calculation method specified in the streaming computing task, and the calculation result is stored in the allocated storage space.

[0010] A seventh aspect discloses a data processing apparatus based on streaming computing, applied to a control node outside of multiple streaming computing nodes in a streaming computing cluster, comprising: a first receiving module, configured to receive allocation requests submitted by streaming computing nodes, the allocation requests requesting the allocation of more storage space for the local memory of the streaming computing nodes, the allocation request being sent to the control node when, in a scenario where the streaming computing nodes are processing streaming data, the utilization rate of the storage space in the local memory of the streaming computing nodes is detected to be greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario for processing timeliness is lower than a preset tolerance; or, the allocation request is sent to the control node when it is detected that the utilization rate of the storage space in the local memory of the streaming computing nodes is greater than or equal to a first preset threshold. The allocation module is used to allocate more storage space to the local memory of the streaming computing node according to the allocation request, so that the streaming computing node can obtain the streaming data to be calculated from the data source, calculate the streaming data to be calculated according to the calculation method specified in the streaming computing task, and store the calculation results in the allocated storage space. This is done when the utilization rate of the local memory is greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario for processing timeliness is higher than a preset tolerance.

[0011] Eighthly, a data processing apparatus based on streaming computing is shown, applied to a development node, comprising: an acquisition module for acquiring computer program code developed for a streaming computing task; the computer program code includes at least: a first table creation statement and a write statement; the first table creation statement includes at least a table creation instruction, a first table name, and a first data storage type; the first table creation statement instructs the creation of a first database table with the first table name according to the table creation instruction and instructs the data storage type applicable to the first database table to be the first data storage type; the first data storage type includes an external storage type; the write statement includes at least a write instruction and the first table name, instructing the writing of calculation results to the first database table according to the write instruction and the first table name; a search module for searching for the first data storage type in the first table creation statement; a replacement module for replacing the first data storage type in the first table creation statement with a pre-set third data storage type to obtain new computer program code, the third data storage type including a local storage type; and a generation module for generating a streaming computing task based on the new computer program code, the streaming computing task being suitable for both offline verification environments and online environments.

[0012] Ninthly, a data processing apparatus based on streaming computing is shown, applied to one of multiple streaming computing nodes in a streaming computing cluster, comprising: a third receiving module, configured to receive an acquisition request sent by a development node, the acquisition request being used to acquire computation results stored in the local memory of the streaming computing node; the computation results are obtained when, in a scenario where the streaming computing node processes streaming data, it is detected that the utilization rate of the storage space in the local memory of the streaming computing node is greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario for processing timeliness is greater than or equal to a preset tolerance, the acquisition of streaming data to be computed from the data source is interrupted, and the acquisition of streaming data to be computed is resumed when the utilization rate of the storage space in the local memory of the streaming computing node is less than the first preset threshold, the streaming data to be computed is then acquired from the data source, and the streaming data to be computed is computed according to the computation method specified in the streaming computing task and stored in the free space of the local memory of the streaming computing node; the data source is used to store the data to be computed. The streaming data in the data source is used by the streaming computing node to perform calculations using the calculation method specified in the streaming computing task; or, the calculation result is submitted to the control node in the streaming computing cluster when the utilization rate of the storage space in the local memory of the streaming computing node is detected to be greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario for processing timeliness is lower than a preset tolerance. The allocation request is used to request more storage space to be allocated to the local memory of the streaming computing node; and when the control node allocates more storage space to the local memory of the streaming computing node according to the allocation request, the streaming data to be calculated is obtained from the data source, and the streaming data to be calculated is calculated according to the calculation method specified in the streaming computing task and stored in the allocated storage space; a reading module is used to read the calculation result stored in the local memory of the streaming computing node; a first sending module is used to send the read calculation result to the development node.

[0013] Tenthly, a data processing apparatus based on streaming computing is shown, applied to a development node, comprising: a second sending module, configured to send an acquisition request to one of a plurality of streaming computing nodes in a streaming computing cluster, the acquisition request being used to acquire computation results stored in the local memory of the streaming computing node; wherein the computation results are obtained when, in a scenario where the streaming computing node processes streaming data, it is detected that the utilization rate of the storage space in the local memory of the streaming computing node is greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario for processing timeliness is greater than or equal to a preset tolerance, the acquisition of streaming data to be computed from the data source is interrupted, and the acquisition of streaming data to be computed is resumed when the utilization rate of the storage space in the local memory of the streaming computing node is less than the first preset threshold, the streaming data to be computed is then acquired from the data source, and the streaming data to be computed is computed according to the computation method specified in the streaming computing task and stored in the free space of the local memory of the streaming computing node. The data source is used to store the streaming data to be computed. The streaming data in the data source is used to be computed by the streaming computing node using the computing method specified in the streaming computing task. Alternatively, the computation result is that in the scenario where the streaming computing node processes streaming data, if the utilization rate of the storage space in the local memory of the streaming computing node is detected to be greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario for processing timeliness is lower than a preset tolerance, an allocation request is submitted to the control node in the streaming computing cluster. The allocation request is used to request the allocation of more storage space for the local memory of the streaming computing node. And if the control node allocates more storage space for the local memory of the streaming computing node according to the allocation request, the streaming data to be computed is obtained from the data source, computed according to the computing method specified in the streaming computing task, and stored in the allocated storage space. The fourth receiving module is used to receive the computation result returned by the streaming computing node according to the acquisition request.

[0014] Eleventhly, an electronic device is shown, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to perform the method as shown in any of the preceding aspects.

[0015] In a twelfth aspect, a non-transitory computer-readable storage medium is disclosed, which, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the methods shown in any of the foregoing aspects.

[0016] In a thirteenth aspect, a computer program product is shown that, when the instructions in the computer program product are executed by a processor of an electronic device, enables the electronic device to perform the method as shown in any of the foregoing aspects.

[0017] Compared with the prior art, this application has the following advantages:

[0018] In this application, in the scenario of streaming computing nodes processing streaming data, if the utilization rate of the local storage space of the streaming computing node is greater than or equal to a first preset threshold, it indicates that the local storage space of the streaming computing node cannot currently store more computation results. Therefore, in order to avoid the streaming computing node crashing or freezing during the execution of streaming computing tasks as much as possible, it can be determined whether the tolerance for processing timeliness in this scenario is higher than or equal to a preset tolerance. If the tolerance for processing timeliness in this scenario is higher than or equal to the preset tolerance, the streaming computing node may not acquire streaming data. That is, the streaming computing node interrupts the acquisition of streaming data to be computed from the data source. In this way, the streaming computing node will not compute the streaming data according to the computing method specified in the streaming computing task, and will not obtain computing results. Therefore, it will not attempt to store computing results in the local storage of the streaming computing node, and will not fail to store computing results in the local storage of the streaming computing node. It will also not continuously attempt to store computing results in the local storage of the streaming computing node, and thus will not cause the streaming computing node to crash or freeze due to failure to successfully store the results.

[0019] Furthermore, if at least one computation result is subsequently extracted from the local memory of the streaming computing node, such that the utilization rate of the local memory of the streaming computing node is less than a first preset threshold, the streaming computing node then obtains the streaming data to be computed from the data source, computes the streaming data to be computed according to the computation method specified in the streaming computing task, and stores the computation result in the local memory of the streaming computing node. Since the utilization rate of the local memory of the streaming computing node is less than the first preset threshold, that is, there is enough free storage space in the local memory of the streaming computing node, the computation result can be successfully stored in the local memory of the streaming computing node. It will not continuously attempt to store the computation result in the local memory of the streaming computing node due to storage failure, and thus will not cause the streaming computing node to crash or freeze due to multiple unsuccessful storage attempts.

[0020] Furthermore, in scenarios where the tolerance for processing timeliness is lower than a preset tolerance, an allocation request is submitted to the control node in the streaming computing cluster. This allocation request requests more storage space for the local storage of the streaming computing node. If the control node allocates more storage space to the local storage of the streaming computing node according to the allocation request, the streaming data to be computed is obtained from the data source. The streaming data to be computed is computed according to the computation method specified in the streaming computing task. The computation results are stored in the allocated storage space. Since more storage space is allocated to the local storage of the streaming computing node, the computation results can be successfully stored in the local storage of the streaming computing node. This avoids the continuous attempts to store the computation results in the local storage of the streaming computing node due to storage failures, and thus prevents the streaming computing node from crashing or freezing due to multiple unsuccessful storage attempts.

[0021] Furthermore, this application can decide whether to "interrupt the acquisition of streaming data to be computed from the data source and, if the utilization rate of the local storage space of the streaming computing node is less than a first preset threshold, then acquire the streaming data to be computed from the data source" or "request the control node to allocate more storage space for the local storage of the streaming computing node" based on the tolerance for processing timeliness in the scenario, so as to avoid delaying the timely processing of streaming data involved in scenarios with low tolerance for processing timeliness.

[0022] Secondly, it can support a scheme that allows the execution of streaming computing tasks to verify the results without involving external storage devices, which can improve the efficiency of verifying the results, reduce labor costs, reduce hardware costs, and improve the efficiency of officially launching streaming computing tasks. Attached Figure Description

[0023] Figure 1 This is a structural block diagram of a streaming computing cluster according to this application.

[0024] Figure 2 This is a flowchart of the steps of a data processing method based on streaming computing according to this application.

[0025] Figure 3 This is a flowchart of the steps of a data processing method based on streaming computing according to this application.

[0026] Figure 4 This is a flowchart of the steps of a data processing method based on streaming computing according to this application.

[0027] Figure 5 This is a structural block diagram of a data processing device based on streaming computing according to this application.

[0028] Figure 6This is a structural block diagram of a data processing device based on streaming computing according to this application.

[0029] Figure 7 This is a structural block diagram of a data processing device based on streaming computing according to this application.

[0030] Figure 8 This is a structural block diagram of a data processing device based on streaming computing according to this application.

[0031] Figure 9 This is a structural block diagram of a data processing device based on streaming computing according to this application.

[0032] Figure 10 This is a structural block diagram of a device according to this application. Detailed Implementation

[0033] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0034] Currently, stream computing engines are widely used in real-time computing scenarios. Examples include Flink, also known as Apache Flink, an open-source stream processing framework and distributed processing engine. Its core is a distributed data stream engine written in Java and Scala. Flink executes arbitrary streaming data programs in a data-parallel and pipelined manner. Flink's pipelined runtime system can execute batch and stream processing programs, performing stateful computations on unbounded and bounded data streams. The Flink runtime itself also supports the execution of iterative algorithms. Flink provides a unified abstraction for both streaming and batch processing, and based on its structural characteristics, it boasts high throughput and low latency.

[0035] Among them, SQL (Structured Query Language) based on stream computing, also known as stream computing SQL, is one of the development methods for stream computing tasks. Due to its structured query method, it is widely used by many vendors. Stream computing SQL is used for storing and retrieving data and performing calculations on the data.

[0036] The process of developing and deploying streaming computing tasks involves stages such as writing and debugging streaming SQL code. Debugging streaming SQL code includes verifying the syntax and logic of the streaming SQL code.

[0037] Regarding "validating the syntax of streaming SQL code", vendors need to verify whether the syntax of the streaming SQL code they write conforms to the syntax requirements.

[0038] Regarding "validating the logic of streaming SQL code", vendors often need to verify whether the table creation parameters in their streaming SQL code are correct and reasonable (incorrect or unreasonable table creation parameters will lead to subsequent data read and write failures), whether the custom logic (such as instruction names or function names) in their streaming SQL code is correct and reasonable (incorrect or unreasonable custom logic will lead to incorrect calculation results in subsequent data calculations), and whether the calculation results obtained from executing the streaming calculation task corresponding to their streaming SQL code are correct and reasonable (incorrect or unreasonable calculation results do not meet the vendor's actual needs), etc.

[0039] In short, vendors will only launch streaming computing tasks online if they can ensure that the calculation results are correct and reasonable.

[0040] In cases where it's necessary to "verify the correctness and reasonableness of the computation results obtained from the streaming computation task corresponding to the streamed SQL code," one approach involves the vendor generating a streaming computation task based on the written streamed SQL code after completion. (For example, they can write Java executable code including the streamed SQL code, compile it into an executable file, and thus obtain a streaming computation task.) The streaming computation task can then be submitted to a streaming computation node. The streaming computation node can then execute the task. For instance, during the execution of the task, the node can obtain streaming data from a data source, perform computations on the data according to the specified computation method in the task, obtain the results, and write them to an external storage device.

[0041] External storage devices are purchased or leased separately by the vendor. The vendor needs to deploy these external storage devices beforehand, ensuring a communication connection between them and the streaming computing nodes. This allows the streaming computing nodes to write computation results to the external storage devices. The external storage devices and streaming computing nodes can operate independently. Afterward, the vendor can retrieve the computation results from the external storage devices and verify their accuracy and reasonableness.

[0042] However, after the above method was widely provided to a large number of manufacturers, the inventors received a lot of feedback from manufacturers: the above method is inefficient and costly in verifying calculation results.

[0043] Thus, the need arose to "improve the efficiency of verifying calculation results and reduce the cost of verification." To achieve this goal, the inventors conducted a statistical analysis of the aforementioned method and discovered that the reasons for the "low efficiency and high cost of verifying calculation results" include:

[0044] On the one hand, in the scenario of "verifying whether the calculation results are correct and reasonable", the above method involves multiple systems, such as the vendor's development node, streaming computing node, and external storage devices. The calculation results need to flow between these devices. For example, the calculation results need to flow from the streaming computing node to the external storage device, and from the external storage device to the vendor's development node. The process of the calculation results flowing from the streaming computing node to the external storage device to the vendor's development node will consume time. In addition, the process of the vendor deploying external storage devices (for example, adapting the external storage device and the streaming computing node by setting the address and other parameters, such as setting the address of the external storage device in the streaming computing node) will also consume time. This will lead to low efficiency in verifying the calculation results and increase the workload of manual labor, resulting in high labor costs.

[0045] On the other hand, the above method requires the introduction of external storage devices, which incur certain costs, such as hardware costs, resulting in high costs for verifying the calculation results.

[0046] On the other hand, once the accuracy and reasonableness of the verification calculation results are confirmed, the streaming computing task can usually be deployed online.

[0047] However, the inventors discovered that some manufacturers have both online and offline verification environments. Both online and offline verification environments utilize external storage devices. However, the external storage devices used in the online and offline verification environments differ. The external storage devices used in the offline verification environment store the computation results of streaming tasks within the offline verification environment, while those used in the online environment store the computation results of streaming tasks within the online environment. Furthermore, the address and other parameters of the external storage devices typically differ between the online and offline verification environments.

[0048] The scenario of "verifying whether the calculation results are correct and reasonable" involves the offline verification environment. In these scenarios, in order for the streaming computing node to communicate with the external storage device of the offline verification environment, the vendor needs to configure the parameters between the external storage device of the offline verification environment and the streaming computing node.

[0049] Setting up parameter compatibility between external storage devices and streaming computing nodes for offline verification environments is time-consuming, leading to low efficiency in verifying calculation results and increased manual workload, resulting in high labor costs.

[0050] Furthermore, once the calculation results are confirmed to be accurate and reasonable, the streaming computing task can usually be officially launched online. Before the streaming computing task is officially launched online, in order to enable the streaming computing node to communicate with the external storage device of the online environment application, the manufacturer also needs to configure the parameters between the external storage device of the online environment application and the streaming computing node.

[0051] Setting up parameter compatibility between external storage devices and streaming computing nodes for online applications is time-consuming, leading to low efficiency in the formal online execution of streaming computing tasks and increased manual workload, resulting in high labor costs.

[0052] Therefore, in order to achieve the goal of "improving the efficiency of verifying calculation results and reducing the cost of verifying calculation results", the inventors summarized and analyzed the above reasons and found that one of the main reasons is that the scenario of "verifying whether the calculation results are correct and reasonable" requires the use of external storage devices, resulting in low efficiency and high cost of verifying calculation results.

[0053] Therefore, the inventors came up with the idea of ​​not using external storage devices in scenarios where "the calculation results are verified to be correct and reasonable".

[0054] For example, the inventors conceived of another approach: after a streaming computing node executes a streaming computing task and obtains the computing results, the computing results can be written into the local memory of the streaming computing node. Then, the manufacturer can retrieve the computing results from the local memory of the streaming computing node and verify whether the retrieved computing results are correct and reasonable.

[0055] It is evident that, on the one hand, in the aforementioned alternative approach, in the scenario of "verifying whether the calculation results are correct and reasonable," no external storage devices are involved. The calculation results do not need to be transferred from the streaming computing node to the external storage device, nor do they need to be transferred from the external storage device to the vendor's development node. This saves the process of transferring the calculation results through the external storage device. Furthermore, since no external storage devices are involved, the vendor does not need to deploy external storage devices, saving the vendor the process of deploying external storage devices and saving time. This can improve the efficiency of verifying the calculation results and reduce the amount of manual work, thereby reducing labor costs.

[0056] On the other hand, in the above alternative approach, external storage devices can be eliminated, thereby reducing hardware costs and consequently reducing the cost of verifying the computation results.

[0057] On the other hand, in the aforementioned alternative approach, the offline verification environment does not require external storage devices. Manufacturers do not need to consider the communication connection between the streaming computing nodes and the external storage devices used in the offline verification environment. This eliminates the need for parameter adaptation settings between the external storage devices and streaming computing nodes used in the offline verification environment, saving time and improving the efficiency of verifying computation results. Furthermore, it reduces manual workload and consequently lowers labor costs.

[0058] Furthermore, the online environment can operate without external storage devices. This eliminates the possibility of discrepancies between the addresses and other parameters of the external storage devices used in the online environment and those used in the offline verification environment. Thus, assuming the verification calculation results are accurate and reasonable, before officially deploying the streaming task, vendors do not need to consider the communication connections between the streaming nodes and the external storage devices used in the online environment. This eliminates the need for parameter compatibility settings between the external storage devices and streaming nodes, saving time and improving the efficiency of deploying streaming tasks. It also reduces manual workload and consequently, lowers labor costs.

[0059] However, after widely disseminating this alternative method to a large number of manufacturers, the inventors received feedback from numerous manufacturers, including:

[0060] During the execution of streaming computing tasks, streaming computing nodes may occasionally crash or freeze. Once a streaming computing node crashes or freezes, it takes a long time to recover, ranging from several hours to several days. During the downtime or freeze, streaming computing tasks cannot be executed, which seriously affects the vendor's need for real-time data computation. As a result, many vendors are unwilling to use the other method mentioned above to verify the correctness and reasonableness of the calculation results, and instead return to the first method to verify the correctness and reasonableness of the calculation results, which is inefficient and costly.

[0061] Therefore, in order to encourage a large number of manufacturers to use the above-mentioned alternative method to verify whether the calculation results are correct and reasonable, so as to achieve the goal of "improving the efficiency of verifying calculation results and reducing the cost of verifying calculation results", the requirement of "on the basis of the above-mentioned alternative method, streaming computing tasks can be executed to meet the manufacturers' needs for real-time data computing" has been proposed.

[0062] In order to achieve the goal of "being able to execute streaming computing tasks to meet the vendor's real-time computing needs for data based on another method", the requirement of "avoiding downtime or crashing as much as possible during the execution of streaming computing tasks based on the above-mentioned alternative method" is proposed.

[0063] To achieve the goal of "minimizing downtime or crashes of streaming computing nodes during the execution of streaming computing tasks using the aforementioned alternative method," the inventors conducted a statistical analysis of numerous cases involving streaming computing nodes from various vendors executing streaming computing tasks using this alternative method. They discovered the reason why "streaming computing nodes occasionally experience downtime or crashes during the execution of streaming computing tasks":

[0064] In this process, during the execution of streaming computing tasks, streaming computing nodes can obtain streaming data from a data source, perform calculations on the streaming data according to the calculation method specified in the streaming computing task, obtain the calculation results, and write the calculation results into the local storage of the streaming computing node. This allows vendors to retrieve the calculation results from the local storage of the streaming computing node. After the vendor retrieves the calculation results from the local storage of the streaming computing node, the storage space occupied by the retrieved calculation results will be freed up, and the freed storage space can be used to store subsequent calculation results.

[0065] However, one scenario for using streaming computing nodes to perform streaming computing tasks is a high-concurrency scenario. In a high-concurrency scenario, the data source continuously generates streaming data to be computed, and the streaming computing node continuously consumes the streaming data from the data source. For example, the streaming computing node continuously extracts streaming data from the data source and continuously performs calculations on the extracted streaming data according to the calculation method specified in the streaming computing task, obtains the calculation results, and attempts to store the obtained calculation results in the streaming computing node's storage.

[0066] However, if the local memory of the streaming computing node has limited storage space and the vendor retrieves computing results from the local memory of the streaming computing node slowly, the local memory of the streaming computing node may become full.

[0067] If the local storage space of a streaming computing node is full, and the node continues to obtain new streaming data from the data source and performs calculations on the new data according to the calculation method specified in the streaming computing task, obtaining new calculation results, and continues to attempt to store the new calculation results in the local storage space of the streaming computing node, it will be unable to successfully store the new calculation results in the local storage space of the streaming computing node because the local storage space of the streaming computing node is full. That is, storage will fail after attempting to store. However, in order to successfully store the new calculation results, the streaming computing node will continue to attempt to store the new calculation results in the local storage space of the streaming computing node. However, in practice, multiple failures to store often lead to the streaming computing node crashing or freezing.

[0068] In view of this, in order to achieve the goal of "avoiding down or crashing as much as possible during the execution of streaming computing tasks by streaming computing nodes in the above-mentioned other way", the inventors thought of avoiding constantly trying to store new computing results in the local memory of streaming computing nodes when the storage space of the local memory of the streaming computing node is full.

[0069] However, the inventors discovered that in scenarios where streaming computing nodes perform streaming computing tasks, if a streaming computing node performs calculations on streaming data according to the calculation method specified in the streaming computing task and obtains the calculation results, the streaming computing node will inevitably attempt to store the calculation results in its local memory.

[0070] Therefore, in order to achieve the goal of "avoiding the continuous attempt to store new computation results in the local memory of the streaming computing node when the local memory of the streaming computing node is full", the inventors thought of a way to prevent the streaming computing node from performing computations on the streaming data according to the computation method specified in the streaming computing task when the local memory of the streaming computing node is full.

[0071] However, the inventors discovered that in scenarios where streaming computing nodes execute streaming computing tasks, if a streaming computing node acquires streaming data, it will inevitably perform calculations on the streaming data according to the calculation method specified in the streaming computing task.

[0072] Therefore, in order to achieve the goal of "the streaming computing node does not perform calculations on streaming data according to the calculation method specified in the streaming computing task when the local storage space of the streaming computing node is full", the inventors conceived of the idea that the streaming computing node does not acquire streaming data when the local storage space of the streaming computing node is full. For example, the streaming computing node does not acquire streaming data from the data source.

[0073] Thus, if the local storage space of a streaming computing node is full, and the streaming computing node does not acquire streaming data, it will not perform calculations on the streaming data according to the calculation method specified in the streaming computing task. Consequently, it will not obtain calculation results, will not attempt to store the calculation results in its local storage, will not fail to store the calculation results in its local storage, will not continuously attempt to store the calculation results in its local storage, and will not crash or freeze due to failure to successfully store the results.

[0074] Furthermore, if at least one computation result is extracted from the local memory of the streaming computing node so that the local memory of the streaming computing node is not full (i.e., there is free storage space), the streaming computing node then obtains new streaming data from the data source, performs computation on the new streaming data according to the computation method specified in the streaming computing task, obtains new computation results, and then attempts to store the new computation results in the local memory of the streaming computing node. Since the local memory of the streaming computing node is not full, that is, there is free storage space in the local memory of the streaming computing node, the new computation results can be successfully stored in the local memory of the streaming computing node. It will not continuously attempt to store computation results in the local memory of the streaming computing node due to storage failures, and thus will not cause the streaming computing node to crash or freeze due to multiple unsuccessful storage attempts.

[0075] Specifically, see Figure 1The diagram illustrates a data processing system, which includes a development node 01 and a streaming computing cluster 02, which includes a streaming computing node 021 and a control node 022.

[0076] The development node and the streaming computing node can communicate with each other. The streaming computing node and the control node can also communicate with each other.

[0077] Development nodes can include terminals used by the vendor's developers to develop streaming computing tasks, such as tablets or laptops.

[0078] Streaming computing nodes can include physical machines, such as servers. Each streaming computing node has local storage, which can include memory or persistent storage, such as hard drives. This local storage is present in the streaming computing node before it leaves the factory.

[0079] Memory is a crucial component of a computer, also known as internal memory or main memory. It temporarily stores data processed by the CPU, as well as data exchanged with external storage devices such as hard drives. It acts as a bridge between external storage and the CPU. All programs in a computer run within memory, and the performance of the memory directly impacts the overall performance of the computer.

[0080] In one embodiment, the streaming computing node and the development node can both be assets belonging to the same vendor.

[0081] Alternatively, in another embodiment, the streaming computing node and the development node can be assets belonging to different vendors. For example, the development node is an asset of vendor A, while the streaming computing node is an asset of vendor B. Vendor A can lease streaming computing nodes from vendor B to perform streaming computing tasks.

[0082] In another embodiment, the streaming computing node may also include virtual machines deployed on a physical machine. The physical machine can allocate system resources to the streaming computing node acting as a virtual machine, including computing resources and storage resources, that is, the streaming computing node acting as a virtual machine can receive at least a portion of the computing resources and at least a portion of the storage resources allocated by the physical machine. The local storage of the streaming computing node acting as a virtual machine may include the storage space allocated by the physical machine to the virtual machine (at least a portion of the storage space in the local storage of the physical machine), etc.

[0083] In one embodiment, see Figure 2 This paper demonstrates a data processing method based on streaming computing, which is applied to... Figure 1The development node shown includes the following method:

[0084] In step S101, the computer program code developed for the streaming computing task is obtained.

[0085] Computer program code is developed by the vendor's developers using development nodes. For example, the vendor's developers can start development tools in the development node, create engineering projects in the development tools, and write computer program code in the engineering projects.

[0086] When it is necessary to verify the computation results obtained from the subsequent execution of streaming computing tasks, the debug button in the development tool can be manipulated. When the development node receives the operation of the debug button in the development tool, it is determined that the developer needs to verify the computation results obtained from the subsequent execution of streaming computing tasks, thus obtaining the computer program code developed for streaming computing tasks.

[0087] In one approach, for scenarios where vendors need to verify the computation results obtained from subsequent streaming computation tasks, most vendors, in addition to using streaming computation nodes (such as Flink streaming computation nodes), also need to use external storage devices, as mentioned earlier. The streaming computation nodes and external storage devices communicate with each other. Based on the development node, the vendor can retrieve the computation results from the external storage device of the streaming computation node and verify whether the computation results are correct and reasonable. For example, the computation results can be displayed so that developers can view the computation results and evaluate whether the computation results are correct and reasonable.

[0088] In order for streaming computing nodes to store the computation results on external storage devices after executing streaming computing tasks, the computer program code written by the developers needs to specify that the computation results should be stored on external storage devices.

[0089] For example, in one instance, the computer program code includes at least: a first table creation statement and a write statement.

[0090] The first table creation statement includes at least a table creation instruction, a first table name, and a first data storage type; the first table creation statement is used to instruct the creation of a first database table with the first table name according to the table creation instruction and to instruct the data storage type applicable to the first database table to be the first data storage type; the first data storage type includes external storage type.

[0091] The write statement includes at least a write instruction and a first table name, which instructs the calculation results to be written to the first database table according to the write instruction and the first table name.

[0092] The first database table corresponding to the first table name is used to store calculation results, and the first database table is applicable to the first data storage type. The first data storage type includes external storage, which can achieve the following indication: the first database table is located in external storage, the calculation results stored in the first database table are located in external storage, and storing calculation results in the first database table can actually be understood as storing calculation results in the first database table in external storage.

[0093] Moreover, one of the objectives of this application is to enable the computation results to be stored in local memory rather than in external memory.

[0094] The distinction between "local" and "external" in local and external storage can be based on the device itself. A device may have local storage, which is located in the device before it leaves the factory. For example, a device's local storage may include the device's RAM or its persistent storage space (such as a hard drive).

[0095] External memory and devices are independent of each other, and external memory and devices can be connected in a communicative manner.

[0096] In one scenario, the local memory of this application may include the local memory in a streaming computing node that performs streaming computing tasks.

[0097] In order to store the calculation results in local memory instead of external memory, it is necessary to modify the computer program code already written by the developers to obtain new computer program code that instructs the calculation results to be stored in local storage devices.

[0098] However, if the manufacturer's developers modify the computer program code already written by the developers, it will consume the developers' time and increase their workload. Therefore, in this application, the computer program code already written by the developers can be automatically modified to obtain new computer program code. The new computer program code instructs the calculation results to be stored in the local storage device, without requiring manual modification by the developers, thus saving the developers' time and reducing their workload.

[0099] The specific process of "automatically modifying the computer program code already written by the developer to obtain new computer program code, and the new computer program code instructing the calculation results to be stored in the local storage device" can be found in steps S102 and S103, and will not be described in detail here.

[0100] Furthermore, in another embodiment of this application, the computer program code may also include: a second table creation statement and a read statement.

[0101] The second table creation statement includes at least a table creation instruction, a second table name, and a second data storage type; the second table creation statement is used to instruct the creation of a second database table with the second table name according to the table creation instruction and to instruct the data storage type that the second database table is connected to as the second data storage type.

[0102] A read statement must include at least a read instruction and a second table name, indicating that streaming data should be read from a second database table based on the read instruction and the second table name. The second database table is the data source.

[0103] The example provided is intended to illustrate the solution of this application, but is not intended to limit the solution of this application.

[0104] For example, suppose the computer program code written by the developers for a streaming computing task includes the following content in quotation marks:

[0105] “

[0106] create TEMPORARY table`datagen_source`(id varchar,name varchar)with('connector'='datagen');

[0107] create TEMPORARY table`blackhole`(id varchar,name varchar)with('connector'='print');

[0108] insert into blackhole;

[0109] select id,MyScalarFunc(name)from datagen_source;

[0110] "

[0111] In the above computer program code, "create TEMPORARY table `datagen_source`(id varchar, name varchar)with('connector' = 'datagen')" is the second table creation statement.

[0112] The first statement to create a table is "create TEMPORARY table `blackhole`(id varchar,name varchar)with('connector'='print')".

[0113] "insert into blackhole" is the write statement.

[0114] The statement “select id,MyScalarFunc(name)from datagen_source” is a read statement.

[0115] In the first table creation statement, "create TEMPORARY table" is the table creation command, "blackhole" is the first table name, and "print" is the first data storage type.

[0116] In the second table creation statement, "create TEMPORARY table" is the table creation command, "datagen_source" is the second table name, and "datagen" is the second data storage type.

[0117] In the write statement, "insert into" is the write command, and "blackhole" is the first table name.

[0118] In the read statement, "select from" is the read command, and "datagen_source" is the name of the second table.

[0119] In step S102, the first data storage type is searched in the first table creation statement.

[0120] For example, in the example above, the first data storage type "print" is searched in the first table creation statement "create TEMPORARY table `blackhole`(idvarchar,name varchar)with('connector'='print')".

[0121] Specifically, "connector" is the command name of the connection instruction. Thus, the command name "connector" can be indexed in the first table creation statement, and then the string "print" connected to the command name "connector" by the specific symbol "=" can be indexed in the first table creation statement. Then, the string "print" connected to the command name "connector" by the specific symbol "=" can be used as the first data storage type.

[0122] The first data storage type "print" indicates that the first database table "blackhole" is located in external storage, for example, the first database table "blackhole" is located in an external storage device.

[0123] In step S103, the first data storage type in the first table creation statement is replaced with a pre-set third data storage type to obtain new computer program code. The third data storage type includes the local memory type.

[0124] The third data storage type indicates that the data in the database table is intended to be stored in local storage.

[0125] According to the new computer program code, the first database table corresponding to the first table name is used to store calculation results, and the first database table is applicable to the third data storage type, which includes local memory. This can achieve the following instruction: the first database table is located in local memory, the calculation results stored in the first database table are located in local memory, and storing calculation results in the first database table can actually be understood as storing calculation results in the first database table in local memory.

[0126] In one example, the third data storage type could include "preview". The third data storage type "preview" indicates that the data in the first database table "blackhole" is intended to be stored in local storage.

[0127] For example, in one instance, the first data storage type "print" in the first table creation statement "create TEMPORARY table `blackhole`(id varchar,name varchar)with('connector'='print')" can be replaced with the third data storage type "preview" to obtain the new table creation statement "create TEMPORARYtable `blackhole`(id varchar,name varchar)with('connector'='preview')", resulting in new computer program code.

[0128] For example, in one instance, the new computer program code includes the following content within “”:

[0129] “

[0130] create TEMPORARY table`datagen_source`(id varchar,name varchar)with('connector'='datagen');

[0131] create TEMPORARY table`blackhole`(id varchar,name varchar)with('connector'='preview');

[0132] insert into blackhole;

[0133] select id,MyScalarFunc(name)from datagen_source;

[0134] "

[0135] Local storage includes the memory in a device that executes new computer program code. This memory is present in the device before it leaves the factory; for example, it includes the device's RAM or its persistent storage space (e.g., a hard drive).

[0136] In one scenario, the local memory of this application may include the local memory in a streaming computing node that performs streaming computing tasks.

[0137] In step S104, a streaming computing task is generated based on the new computer program code. The streaming computing task is suitable for both offline verification environments and online environments.

[0138] For example, based on new computer program code, Java executable code can be written, which then compiles the Java executable code into an executable file, thus obtaining a streaming computing task. This streaming computing task can then be passed to a streaming computing node for execution. For instance, during the execution of the streaming computing task, the node can obtain streaming data from a data source, perform calculations on the streaming data according to the calculation method specified in the task, obtain the results, and write the results to its local memory. This allows the vendor to later retrieve the results from the local memory of the streaming computing node using the development node, and verify the correctness and reasonableness of the results.

[0139] The specific process of "the streaming computing node obtaining streaming data from the data source, performing calculations on the streaming data according to the calculation method specified in the streaming computing task, obtaining the calculation results, and writing the calculation results into the local storage of the streaming computing node" can be found later. Figure 3 The embodiments shown are not described in detail here.

[0140] This application enables the modification of computer program code developed by various manufacturers for streaming computing tasks without human intervention, thus achieving automated modification, improving efficiency, and reducing labor costs.

[0141] Furthermore, the solution in this application supports the following: "In scenarios where streaming computing nodes process streaming data, if the utilization rate of the local storage space of the streaming computing node is greater than or equal to a first preset threshold, it indicates that the local storage space of the streaming computing node is currently unable to store more computation results. Therefore, to minimize the risk of the streaming computing node crashing or freezing during the execution of streaming computing tasks, it can be determined whether the tolerance for processing timeliness in this scenario is higher than or equal to a preset tolerance. If the tolerance for processing timeliness in this scenario is higher than or equal to the preset tolerance, the streaming computing node..." A streaming node can choose not to acquire streaming data. That is, the streaming computing node can interrupt the acquisition of streaming data to be computed from the data source. In this way, the streaming computing node will not compute the streaming data according to the computing method specified in the streaming computing task, and will not obtain the computing result. Therefore, it will not attempt to store the computing result in the local memory of the streaming computing node, and will not fail to store the computing result in the local memory of the streaming computing node. It will also not continuously attempt to store the computing result in the local memory of the streaming computing node, and will not cause the streaming computing node to crash or freeze due to failure to store the result.

[0142] Furthermore, it supports the following: "If at least one computation result is extracted from the local memory of the streaming computing node, causing the utilization rate of the local memory of the streaming computing node to be less than a first preset threshold, the streaming computing node then obtains the streaming data to be computed from the data source, computes the streaming data to be computed according to the computation method specified in the streaming computing task, and stores the computation result in the local memory of the streaming computing node. Since the utilization rate of the local memory of the streaming computing node is less than the first preset threshold, that is, there is enough free storage space in the local memory of the streaming computing node, the computation result can be successfully stored in the local memory of the streaming computing node. It will not continuously attempt to store the computation result in the local memory of the streaming computing node due to storage failure, and thus will not cause the streaming computing node to crash or freeze due to multiple unsuccessful storage attempts."

[0143] Furthermore, it supports the following: "When the tolerance for processing timeliness in this scenario is lower than the preset tolerance, an allocation request is submitted to the control node in the streaming computing cluster. The allocation request is used to request more storage space for the local storage of the streaming computing node. When the control node allocates more storage space for the local storage of the streaming computing node according to the allocation request, the streaming data to be computed is obtained from the data source, and the streaming data to be computed is computed according to the computing method specified in the streaming computing task. The computing results are stored in the allocated storage space. Since more storage space is allocated to the local storage of the streaming computing node, the computing results can be successfully stored in the local storage of the streaming computing node. It will not continuously try to store the computing results in the local storage of the streaming computing node due to storage failure, and thus will not cause the streaming computing node to crash or freeze due to multiple failures to store."

[0144] Furthermore, this application can decide whether to "interrupt the acquisition of streaming data to be computed from the data source and, if the utilization rate of the local storage space of the streaming computing node is less than a first preset threshold, then acquire the streaming data to be computed from the data source" or "request the control node to allocate more storage space for the local storage of the streaming computing node" based on the tolerance for processing timeliness in the scenario, so as to avoid delaying the timely processing of streaming data involved in scenarios with low tolerance for processing timeliness.

[0145] Secondly, see Figure 1 As described in the preceding text of the illustrated embodiments, this application can support a scheme that "executes streaming computing tasks to verify the obtained computing results without involving external storage devices, which can improve the efficiency of verifying computing results, reduce labor costs, reduce hardware costs, and improve the efficiency of officially launching streaming computing tasks."

[0146] In one embodiment, see Figure 3 This paper demonstrates a data processing method based on streaming computing, which is applied to... Figure 1 The streaming computing node shown in the diagram includes the following method:

[0147] In step S201, in the scenario of streaming computing node processing streaming data, it is detected whether the utilization rate of the local memory of the streaming computing node is greater than or equal to a first preset threshold.

[0148] In this application, during the execution of a streaming computing task by a streaming computing node, the streaming computing node can obtain streaming data from a data source, perform calculations on the streaming data according to the calculation method specified in the streaming computing task, obtain the calculation results, and write the calculation results into the local storage of the streaming computing node.

[0149] However, the storage space of the local memory of a streaming computing node is often limited. If the manufacturer retrieves the computing results from the local memory of the streaming computing node slowly, the utilization rate of the local memory of the streaming computing node may be greater than or equal to a first preset threshold. If the utilization rate of the local storage space of the streaming computing node is greater than or equal to a first preset threshold, and the streaming computing node continues to obtain new streaming data from the data source and continues to perform calculations on the new streaming data according to the calculation method specified in the streaming computing task, and attempts to store the new calculation results in the local storage of the streaming computing node, then since the utilization rate of the local storage space of the streaming computing node is greater than or equal to the first preset threshold, it means that the storage space of the local storage of the streaming computing node cannot currently store more calculation results. Therefore, it is impossible to successfully store new calculation results in the local storage of the streaming computing node. That is, the storage attempt will fail. However, in order to successfully store the results, the streaming computing node will continue to try to store new calculation results in the local storage of the streaming computing node. However, in practice, multiple failures to store the results often lead to the streaming computing node crashing or freezing.

[0150] Therefore, in order to avoid the streaming computing node from crashing or freezing during the execution of streaming computing tasks as much as possible, it is possible to detect whether the utilization rate of the local memory of the streaming computing node is greater than or equal to a first preset threshold.

[0151] If the utilization rate of the local memory of the streaming computing node is greater than or equal to a first preset threshold, step S202 is executed: determine whether the tolerance of the scenario for processing timeliness is higher than or equal to a preset tolerance.

[0152] In the process of executing streaming computing tasks, the utilization rate of the local memory of the streaming computing node can be detected in real time or periodically to see if it is greater than or equal to a first preset threshold.

[0153] The first preset threshold includes 100%, 99%, 98%, 97%, or 96%, etc., and the specific threshold can be determined according to the actual situation. This application does not limit it.

[0154] If the utilization rate of the local memory of the streaming computing node is greater than or equal to a first preset threshold, in step S202, it is determined whether the tolerance of the scenario for processing timeliness is higher than or equal to a preset tolerance.

[0155] Streaming computing nodes may handle streaming data processing in different scenarios, such as live streaming, gaming, shopping, or statistics. Different scenarios have different tolerances for timeliness. Scenarios with higher tolerance for timeliness can tolerate longer delays in the streaming data processing, while scenarios with lower tolerance for timeliness can tolerate shorter delays in the streaming data processing.

[0156] In the scenario where the tolerance for processing timeliness is higher than or equal to the preset tolerance, in step S203, the acquisition of streaming data to be calculated from the data source is interrupted. The data source is used to store the streaming data to be calculated, and the streaming data in the data source is used by the streaming computing node to calculate the data using the calculation method specified in the streaming computing task.

[0157] In another embodiment of this application, if the utilization rate of the local memory of the streaming computing node is greater than or equal to a first preset threshold, it indicates that the local memory of the streaming computing node cannot currently store more computation results. Thus, in this scenario where the tolerance for processing timeliness is higher than or equal to the preset tolerance, in order to avoid the streaming computing node crashing or freezing during the execution of the streaming computing task as much as possible, the streaming computing node may not acquire streaming data. That is, the streaming computing node interrupts the acquisition of streaming data to be computed from the data source. In this way, the streaming computing node will not compute the streaming data according to the computation method specified in the streaming computing task, and will not obtain computation results. Therefore, it will not attempt to store the computation results in the local memory of the streaming computing node, and will not fail to store the computation results in the local memory of the streaming computing node. It will also not continuously attempt to store the computation results in the local memory of the streaming computing node, and thus will not cause the streaming computing node to crash or freeze due to failure to successfully store the results.

[0158] Alternatively, in another embodiment of this application, if the utilization rate of the local memory of the streaming computing node is less than a first preset threshold, it indicates that the local memory of the streaming computing node can still store the calculation results. In this case, the streaming computing node can obtain the streaming data to be calculated from the data source, calculate the streaming data to be calculated according to the calculation method specified in the streaming computing task, and store the calculation results in the local memory of the streaming computing node.

[0159] In step S204, if the utilization rate of the storage space in the local memory of the streaming computing node is less than the first preset threshold, the streaming data to be calculated is obtained from the data source, the streaming data to be calculated is calculated according to the calculation method specified in the streaming computing task, and the calculation result is stored in the local memory of the streaming computing node.

[0160] If the tolerance for processing timeliness in this scenario is lower than the preset tolerance, in step S205, an allocation request is submitted to the control node in the streaming computing cluster. The allocation request is used to request more storage space to be allocated to the local memory of the streaming computing node.

[0161] The preset tolerance can be set in advance according to the actual situation, and this application does not limit it.

[0162] Accordingly, the control node can receive allocation requests submitted by the streaming computing node and allocate more storage space to the local memory of the streaming computing node according to the allocation request, so that the local memory of the streaming computing node has free storage space to store the computing results. In this way, the streaming computing node can obtain the streaming data to be computed from the data source, compute the streaming data to be computed according to the computing method specified in the streaming computing task, and store the computing results in the allocated storage space.

[0163] In step S206, when the control node allocates more storage space to the local memory of the streaming computing node according to the allocation request, the streaming data to be computed is obtained from the data source, the streaming data to be computed is computed according to the computing method specified in the streaming computing task, and the computing results are stored in the allocated storage space.

[0164] In another embodiment, if the interruption duration of obtaining streaming data to be computed from the data source is longer than the preset duration corresponding to the scenario, step S205 is executed: submit an allocation request to the control node in the streaming computing cluster to avoid not processing the streaming data involved in the scenario for a long time, thereby avoiding service interruption and the like.

[0165] Since different scenarios have varying tolerances for processing timeliness, the preset durations are often also different. Therefore, the preset durations for each scenario can be statistically analyzed and set in advance. Thus, in this embodiment, the preset duration corresponding to the scenario, which has been statistically analyzed and set in advance, can be obtained and compared with the interruption duration of the interruption in retrieving streaming data from the data source.

[0166] Furthermore, when the utilization rate of the local memory of the streaming computing node falls below a second preset threshold, a reclamation request is submitted to the control node. This reclamation request reclaims a portion of the storage space allocated to the streaming computing node's local memory. The second preset threshold is lower than the first preset threshold, allowing the control node to reclaim a portion of the storage space in the streaming computing node's local memory according to the reclamation request. This reclaims excess, idle storage space, preventing waste. The control node can then schedule the reclaimed storage space according to actual needs to improve storage utilization.

[0167] In this application, the following actions can be implemented using a backpressure mechanism: "When the utilization rate of the local storage space of the streaming computing node is greater than or equal to a first preset threshold and the tolerance for processing timeliness in this scenario is higher than or equal to a preset tolerance, the acquisition of streaming data to be computed from the data source is interrupted" and "The acquisition of streaming data to be computed from the data source is resumed when the utilization rate of the local storage space of the streaming computing node is less than the first preset threshold." For example, in one instance, the streaming computing node has a streaming computing engine, and a backpressure mechanism can be deployed in the streaming computing engine. This allows the streaming computing node to implement the following actions based on the backpressure mechanism: "When the utilization rate of the local storage space of the streaming computing node is greater than or equal to a first preset threshold and the tolerance for processing timeliness in this scenario is higher than or equal to a preset tolerance, the acquisition of streaming data to be computed from the data source is interrupted" and "The acquisition of streaming data to be computed from the data source is resumed when the utilization rate of the local storage space of the streaming computing node is less than the first preset threshold."

[0168] In this application, in the scenario of streaming computing nodes processing streaming data, if the utilization rate of the local storage space of the streaming computing node is greater than or equal to a first preset threshold, it indicates that the local storage space of the streaming computing node cannot currently store more computation results. Therefore, in order to avoid the streaming computing node crashing or freezing during the execution of streaming computing tasks as much as possible, it can be determined whether the tolerance for processing timeliness in this scenario is higher than or equal to a preset tolerance. If the tolerance for processing timeliness in this scenario is higher than or equal to the preset tolerance, the streaming computing node may not acquire streaming data. That is, the streaming computing node interrupts the acquisition of streaming data to be computed from the data source. In this way, the streaming computing node will not compute the streaming data according to the computing method specified in the streaming computing task, and will not obtain computing results. Therefore, it will not attempt to store computing results in the local storage of the streaming computing node, and will not fail to store computing results in the local storage of the streaming computing node. It will also not continuously attempt to store computing results in the local storage of the streaming computing node, and thus will not cause the streaming computing node to crash or freeze due to failure to successfully store the results.

[0169] Furthermore, if at least one computation result is subsequently extracted from the local memory of the streaming computing node, such that the utilization rate of the local memory of the streaming computing node is less than a first preset threshold, the streaming computing node then obtains the streaming data to be computed from the data source, computes the streaming data to be computed according to the computation method specified in the streaming computing task, and stores the computation result in the local memory of the streaming computing node. Since the utilization rate of the local memory of the streaming computing node is less than the first preset threshold, that is, there is enough free storage space in the local memory of the streaming computing node, the computation result can be successfully stored in the local memory of the streaming computing node. It will not continuously attempt to store the computation result in the local memory of the streaming computing node due to storage failure, and thus will not cause the streaming computing node to crash or freeze due to multiple unsuccessful storage attempts.

[0170] Furthermore, in scenarios where the tolerance for processing timeliness is lower than a preset tolerance, an allocation request is submitted to the control node in the streaming computing cluster. This allocation request requests more storage space for the local storage of the streaming computing node. If the control node allocates more storage space to the local storage of the streaming computing node according to the allocation request, the streaming data to be computed is obtained from the data source. The streaming data to be computed is computed according to the computation method specified in the streaming computing task. The computation results are stored in the allocated storage space. Since more storage space is allocated to the local storage of the streaming computing node, the computation results can be successfully stored in the local storage of the streaming computing node. This avoids the continuous attempts to store the computation results in the local storage of the streaming computing node due to storage failures, and thus prevents the streaming computing node from crashing or freezing due to multiple unsuccessful storage attempts.

[0171] Furthermore, this application can decide whether to "interrupt the acquisition of streaming data to be computed from the data source and, if the utilization rate of the local storage space of the streaming computing node is less than a first preset threshold, then acquire the streaming data to be computed from the data source" or "request the control node to allocate more storage space for the local storage of the streaming computing node" based on the tolerance for processing timeliness in the scenario, so as to avoid delaying the timely processing of streaming data involved in scenarios with low tolerance for processing timeliness.

[0172] Secondly, see Figure 1 As described in the preceding text of the illustrated embodiments, this application can support a scheme that "executes streaming computing tasks to verify the obtained computing results without involving external storage devices, which can improve the efficiency of verifying computing results, reduce labor costs, reduce hardware costs, and improve the efficiency of officially launching streaming computing tasks."

[0173] Furthermore, in this application, after the streaming computing node writes the calculation results into the local memory of the streaming computing node, the manufacturer can retrieve the calculation results from the local memory of the streaming computing node based on the development node, and verify whether the calculation results are correct and reasonable, for example, displaying the calculation results so that the developers can view the calculation results and evaluate whether the calculation results are correct and reasonable.

[0174] Specifically, in one embodiment, see [link to relevant documentation]. Figure 4 This paper demonstrates a data processing method based on streaming computing, which is applied to... Figure 1 The streaming computing cluster shown includes the following methods:

[0175] In step S301, the development node sends a request to one of the multiple streaming computing nodes in the streaming computing cluster. The request is used to retrieve the computing results stored in the local memory of the streaming computing node.

[0176] The calculation result is as follows: In a scenario where a streaming computing node processes streaming data, if the utilization rate of the storage space in the local memory of the streaming computing node is detected to be greater than or equal to a first preset threshold, and it is determined that the tolerance for processing timeliness in the scenario is greater than or equal to a preset tolerance, the acquisition of the streaming data to be calculated from the data source is interrupted. The acquisition is resumed only when the utilization rate of the storage space in the local memory of the streaming computing node is less than the first preset threshold. The streaming data to be calculated is then processed according to the calculation method specified in the streaming computing task and stored in the free space of the local memory of the streaming computing node. The data source is used to store the streaming data to be calculated, and the streaming data in the data source is used by the streaming computing node to process the data through streaming... The calculation is performed using the calculation method specified in the calculation task; or, the calculation result is that, in a scenario where a streaming computing node processes streaming data, if the utilization rate of the storage space in the local memory of the streaming computing node is detected to be greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario for processing timeliness is lower than a preset tolerance, an allocation request is submitted to the control node in the streaming computing cluster. The allocation request is used to request the allocation of more storage space for the local memory of the streaming computing node; and if the control node allocates more storage space for the local memory of the streaming computing node according to the allocation request, the streaming data to be calculated is obtained from the data source, and the streaming data to be calculated is calculated according to the calculation method specified in the streaming computing task and stored in the allocated storage space.

[0177] For detailed procedures, please refer to [link / reference]. Figure 3 The embodiments shown are not described in detail here.

[0178] The data source is used to store streaming data to be computed. The streaming data in the data source is used by the streaming computing nodes to compute the data using the computing method specified in the streaming computing task.

[0179] In this application, if a manufacturer needs to retrieve computation results from the local storage of a streaming computing node and verify whether the computation results are correct and reasonable, it can control the development node to generate an acquisition request. The acquisition request is used to retrieve the computation results stored in the local storage of the streaming computing node, and then the acquisition request is sent to the streaming computing node.

[0180] In step S302, the streaming computing node receives an acquisition request sent by the development node.

[0181] In step S303, the streaming computing node reads the computation results stored in its local memory.

[0182] In step S304, the streaming computing node sends the read computing results to the development node.

[0183] In step S305, the development node receives the computation results returned by the streaming computing node based on the acquisition request.

[0184] Afterwards, the development node can display the calculation results so that developers can view them and evaluate whether the results are correct and reasonable.

[0185] After the streaming computing node reads the computation results stored in its local memory, or after the streaming computing node sends the read computation results to the development node, the read computation results can be deleted from the streaming computing node's local memory. This frees up the space occupied by the computation results in the streaming computing node's local memory, allowing the streaming computing node to have more storage space. Furthermore, if the utilization rate of the streaming computing node's local memory is less than a first preset threshold, the streaming computing node can continue to execute streaming computing tasks. For example, it can obtain streaming data to be computed from the data source, compute the streaming data to be computed according to the computation method specified in the streaming computing task, obtain the computation results, and store the computation results in the free space of the streaming computing node's local memory.

[0186] In this application, in the scenario of streaming computing nodes processing streaming data, if the utilization rate of the local storage space of the streaming computing node is greater than or equal to a first preset threshold, it indicates that the local storage space of the streaming computing node cannot currently store more computation results. Therefore, in order to avoid the streaming computing node crashing or freezing during the execution of streaming computing tasks as much as possible, it can be determined whether the tolerance for processing timeliness in this scenario is higher than or equal to a preset tolerance. If the tolerance for processing timeliness in this scenario is higher than or equal to the preset tolerance, the streaming computing node may not acquire streaming data. That is, the streaming computing node interrupts the acquisition of streaming data to be computed from the data source. In this way, the streaming computing node will not compute the streaming data according to the computing method specified in the streaming computing task, and will not obtain computing results. Therefore, it will not attempt to store computing results in the local storage of the streaming computing node, and will not fail to store computing results in the local storage of the streaming computing node. It will also not continuously attempt to store computing results in the local storage of the streaming computing node, and thus will not cause the streaming computing node to crash or freeze due to failure to successfully store the results.

[0187] Furthermore, if at least one computation result is subsequently extracted from the local memory of the streaming computing node, such that the utilization rate of the local memory of the streaming computing node is less than a first preset threshold, the streaming computing node then obtains the streaming data to be computed from the data source, computes the streaming data to be computed according to the computation method specified in the streaming computing task, and stores the computation result in the local memory of the streaming computing node. Since the utilization rate of the local memory of the streaming computing node is less than the first preset threshold, that is, there is enough free storage space in the local memory of the streaming computing node, the computation result can be successfully stored in the local memory of the streaming computing node. It will not continuously attempt to store the computation result in the local memory of the streaming computing node due to storage failure, and thus will not cause the streaming computing node to crash or freeze due to multiple unsuccessful storage attempts.

[0188] Furthermore, in scenarios where the tolerance for processing timeliness is lower than a preset tolerance, an allocation request is submitted to the control node in the streaming computing cluster. This allocation request requests more storage space for the local storage of the streaming computing node. If the control node allocates more storage space to the local storage of the streaming computing node according to the allocation request, the streaming data to be computed is obtained from the data source. The streaming data to be computed is computed according to the computation method specified in the streaming computing task. The computation results are stored in the allocated storage space. Since more storage space is allocated to the local storage of the streaming computing node, the computation results can be successfully stored in the local storage of the streaming computing node. This avoids the continuous attempts to store the computation results in the local storage of the streaming computing node due to storage failures, and thus prevents the streaming computing node from crashing or freezing due to multiple unsuccessful storage attempts.

[0189] Furthermore, this application can decide whether to "interrupt the acquisition of streaming data to be computed from the data source and, if the utilization rate of the local storage space of the streaming computing node is less than a first preset threshold, then acquire the streaming data to be computed from the data source" or "request the control node to allocate more storage space for the local storage of the streaming computing node" based on the tolerance for processing timeliness in the scenario, so as to avoid delaying the timely processing of streaming data involved in scenarios with low tolerance for processing timeliness.

[0190] Secondly, see Figure 1 As described in the preceding text of the illustrated embodiments, this application can support a scheme that "executes streaming computing tasks to verify the obtained computing results without involving external storage devices, which can improve the efficiency of verifying computing results, reduce labor costs, reduce hardware costs, and improve the efficiency of officially launching streaming computing tasks."

[0191] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by this application.

[0192] Reference Figure 5This paper illustrates a structural block diagram of a data processing device based on streaming computing, applied to one of multiple streaming computing nodes in a streaming computing cluster. The device includes: a detection module 11, used to detect whether the utilization rate of the local memory of the streaming computing node is greater than or equal to a first preset threshold in a scenario where the streaming computing node processes streaming data; a determination module 12, used to determine whether the tolerance of the scenario for processing timeliness is higher than or equal to a preset tolerance when the utilization rate of the local memory of the streaming computing node is greater than or equal to the first preset threshold; and an interruption module 13, used to interrupt the acquisition of streaming data to be computed from the data source when the tolerance of the scenario for processing timeliness is higher than or equal to the preset tolerance. The data source is used to store the streaming data to be computed, and the streaming data in the data source is used by the streaming computing node to perform streaming computing tasks. The system performs calculations according to the specified calculation method. If the utilization rate of the local storage space of the streaming computing node is less than a first preset threshold, the system then retrieves the streaming data to be calculated from the data source, performs calculations on the streaming data according to the calculation method specified in the streaming computing task, and stores the calculation results in the local storage of the streaming computing node. The first submission module 14 is used to submit an allocation request to the control node in the streaming computing cluster when the tolerance for processing timeliness in the scenario is lower than a preset tolerance. The allocation request is used to request more storage space to be allocated to the local storage of the streaming computing node. If the control node allocates more storage space to the local storage of the streaming computing node according to the allocation request, the system retrieves the streaming data to be calculated from the data source, performs calculations on the streaming data according to the calculation method specified in the streaming computing task, and stores the calculation results in the allocated storage space.

[0193] The device further includes a first submission module that is also used to submit an allocation request to the control node in the streaming computing cluster when the duration of the interruption in obtaining streaming data to be computed from the data source is longer than the preset duration corresponding to the scenario.

[0194] The device further includes a second submission module, used to submit a recycling request to the control node when the utilization rate of the local memory of the streaming computing node is less than a second preset threshold. The recycling request is used to reclaim a portion of the storage space allocated to the local memory of the streaming computing node. The second preset threshold is less than a first preset threshold, so that the control node reclaims a portion of the storage space in the local memory of the streaming computing node according to the recycling request.

[0195] Reference Figure 6This diagram illustrates a structural block diagram of a data processing device based on streaming computing, applied to a control node outside of multiple streaming computing nodes in a streaming computing cluster. The device includes: a first receiving module 21, used to receive allocation requests submitted by streaming computing nodes. The allocation request requests more storage space to be allocated to the local memory of the streaming computing nodes. The allocation request is sent to the control node when, in a scenario where the streaming computing nodes are processing streaming data, the utilization rate of the storage space in the local memory of the streaming computing nodes is detected to be greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario for processing timeliness is lower than a preset tolerance. Alternatively, the allocation request is sent to the control node when it is detected that the utilization rate of the storage space in the local memory of the streaming computing nodes is greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario for processing timeliness is lower than a preset tolerance. The allocation module 22 is used to allocate more storage space to the local memory of the streaming computing node according to the allocation request, so that the streaming computing node can obtain the streaming data to be calculated from the data source, calculate the streaming data to be calculated according to the calculation method specified in the streaming computing task, and store the calculation results in the allocated storage space.

[0196] The device further includes: a second receiving module, used to receive a recycling request submitted by a streaming computing node, the recycling request being used to reclaim a portion of the storage space allocated to the local memory of the streaming computing node, the recycling request being submitted to the control node when the utilization rate of the storage space of the local memory of the streaming computing node is less than a second preset threshold, the second preset threshold being less than a first preset threshold; and a recycling module, used to reclaim a portion of the storage space in the local memory of the streaming computing node according to the recycling request.

[0197] Reference Figure 7This diagram illustrates a structural block diagram of a data processing device based on streaming computing, applied to a development node. The device includes: an acquisition module 31, used to acquire computer program code developed for streaming computing tasks; the computer program code includes at least: a first table creation statement and a write statement; the first table creation statement includes at least a table creation instruction, a first table name, and a first data storage type; the first table creation statement is used to instruct the creation of a first database table with the first table name according to the table creation instruction and to indicate that the applicable data storage type for the first database table is the first data storage type; the first data storage type includes an external storage type; the write statement includes at least a write instruction and the first table name, used to instruct the writing of calculation results to the first database table according to the write instruction and the first table name; a search module 32, used to search for the first data storage type in the first table creation statement; a replacement module 33, used to replace the first data storage type in the first table creation statement with a pre-set third data storage type to obtain new computer program code, the third data storage type including a local storage type; and a generation module 34, used to generate streaming computing tasks based on the new computer program code, the streaming computing tasks being suitable for both offline verification environments and online environments.

[0198] Reference Figure 8This diagram illustrates a structural block diagram of a data processing device based on streaming computing, applied to one of multiple streaming computing nodes in a streaming computing cluster. The device includes: a third receiving module 41, used to receive an acquisition request sent by a development node, the acquisition request being used to acquire computation results stored in the local memory of the streaming computing node; the computation results are obtained when, in a scenario where the streaming computing node processes streaming data, the utilization rate of the storage space in the local memory of the streaming computing node is detected to be greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario for processing timeliness is higher than or equal to a preset tolerance, the acquisition of streaming data to be computed from the data source is interrupted, and the acquisition of streaming data to be computed is resumed when the utilization rate of the storage space in the local memory of the streaming computing node is less than the first preset threshold, the streaming data to be computed is then acquired from the data source, computed according to the computation method specified in the streaming computing task, and stored in the free space of the local memory of the streaming computing node; the data source is used to store the data to be computed. The streaming data in the data source is used by the streaming computing node to perform calculations using the calculation method specified in the streaming computing task; or, the calculation result is submitted to the control node in the streaming computing cluster when the utilization rate of the storage space in the local memory of the streaming computing node is detected to be greater than or equal to a first preset threshold in the scenario of the streaming computing node processing streaming data, and it is determined that the tolerance of the scenario for processing timeliness is lower than a preset tolerance. The allocation request is used to request the allocation of more storage space for the local memory of the streaming computing node; and when the control node allocates more storage space for the local memory of the streaming computing node according to the allocation request, the streaming data to be calculated is obtained from the data source, and the streaming data to be calculated is calculated according to the calculation method specified in the streaming computing task and stored in the allocated storage space; the reading module 42 is used to read the calculation result stored in the local memory of the streaming computing node; the first sending module 43 is used to send the read calculation result to the development node.

[0199] The device further includes: deleting the read computation results from the local memory of the streaming computing node.

[0200] Reference Figure 9This diagram illustrates a structural block diagram of a data processing device based on streaming computing, applied to a development node. The device includes: a second sending module 51, used to send a request to one of multiple streaming computing nodes in a streaming computing cluster. The request is used to retrieve computation results stored in the local memory of the streaming computing node. The computation results are obtained when, in a scenario where the streaming computing node processes streaming data, the utilization rate of the storage space in the local memory of the streaming computing node is detected to be greater than or equal to a first preset threshold, and it is determined that the tolerance for processing timeliness in the scenario is greater than or equal to a preset tolerance. In this case, the retrieval of the streaming data to be computed from the data source is interrupted, and the retrieval is resumed when the utilization rate of the storage space in the local memory of the streaming computing node is less than the first preset threshold. The streaming data to be computed is then computed according to the computation method specified in the streaming computing task and stored in the free space of the local memory of the streaming computing node. The data source is used to store the streaming data to be computed, and the streaming data in the data source is used to be computed by the streaming computing node through the computing method specified in the streaming computing task; or, the computing result is that in the scenario of the streaming computing node processing streaming data, it is detected that the utilization rate of the storage space in the local memory of the streaming computing node is greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario for processing timeliness is lower than the preset tolerance, and an allocation request is submitted to the control node in the streaming computing cluster. The allocation request is used to request the allocation of more storage space for the local memory of the streaming computing node; and when the control node allocates more storage space for the local memory of the streaming computing node according to the allocation request, the streaming data to be computed is obtained from the data source, and the streaming data to be computed is computed according to the computing method specified in the streaming computing task and stored in the allocated storage space; the fourth receiving module 52 is used to receive the computing result returned by the streaming computing node according to the acquisition request.

[0201] against Figure 5-9In the illustrated embodiment, in a scenario where a streaming computing node processes streaming data, if the utilization rate of the local storage space of the streaming computing node is greater than or equal to a first preset threshold, it indicates that the local storage space of the streaming computing node is currently unable to store more computation results. Therefore, to minimize the risk of the streaming computing node crashing or freezing during the execution of streaming computing tasks, it can be determined whether the tolerance for processing timeliness in this scenario is higher than or equal to a preset tolerance. If the tolerance for processing timeliness in this scenario is higher than or equal to the preset tolerance, the streaming computing node may not acquire streaming data; that is, the streaming computing node interrupts the acquisition of streaming data to be computed from the data source. Thus, the streaming computing node will not compute the streaming data according to the computation method specified in the streaming computing task, and therefore will not obtain computation results. Consequently, it will not attempt to store the computation results in the local storage of the streaming computing node, and there will be no failure to store computation results in the local storage of the streaming computing node. This prevents the continuous attempts to store computation results in the local storage of the streaming computing node, thus avoiding the scenario of the streaming computing node crashing or freezing due to unsuccessful storage.

[0202] Furthermore, if at least one computation result is subsequently extracted from the local memory of the streaming computing node, such that the utilization rate of the local memory of the streaming computing node is less than a first preset threshold, the streaming computing node then obtains the streaming data to be computed from the data source, computes the streaming data to be computed according to the computation method specified in the streaming computing task, and stores the computation result in the local memory of the streaming computing node. Since the utilization rate of the local memory of the streaming computing node is less than the first preset threshold, that is, there is enough free storage space in the local memory of the streaming computing node, the computation result can be successfully stored in the local memory of the streaming computing node. It will not continuously attempt to store the computation result in the local memory of the streaming computing node due to storage failure, and thus will not cause the streaming computing node to crash or freeze due to multiple unsuccessful storage attempts.

[0203] Furthermore, in scenarios where the tolerance for processing timeliness is lower than a preset tolerance, an allocation request is submitted to the control node in the streaming computing cluster. This allocation request requests more storage space for the local storage of the streaming computing node. If the control node allocates more storage space to the local storage of the streaming computing node according to the allocation request, the streaming data to be computed is obtained from the data source. The streaming data to be computed is computed according to the computation method specified in the streaming computing task. The computation results are stored in the allocated storage space. Since more storage space is allocated to the local storage of the streaming computing node, the computation results can be successfully stored in the local storage of the streaming computing node. This avoids the continuous attempts to store the computation results in the local storage of the streaming computing node due to storage failures, and thus prevents the streaming computing node from crashing or freezing due to multiple unsuccessful storage attempts.

[0204] Furthermore, this application can decide whether to "interrupt the acquisition of streaming data to be computed from the data source and, if the utilization rate of the local storage space of the streaming computing node is less than a first preset threshold, then acquire the streaming data to be computed from the data source" or "request the control node to allocate more storage space for the local storage of the streaming computing node" based on the tolerance for processing timeliness in the scenario, so as to avoid delaying the timely processing of streaming data involved in scenarios with low tolerance for processing timeliness.

[0205] Secondly, see Figure 1 As described in the preceding text of the illustrated embodiments, this application can support a scheme that "executes streaming computing tasks to verify the obtained computing results without involving external storage devices, which can improve the efficiency of verifying computing results, reduce labor costs, reduce hardware costs, and improve the efficiency of officially launching streaming computing tasks."

[0206] This application also provides a non-volatile readable storage medium storing one or more modules (programs). When these modules are applied to a device, they enable the device to execute the instructions for the method steps in this application.

[0207] This application provides one or more machine-readable media storing instructions that, when executed by one or more processors, cause an electronic device to perform one or more methods as described in the above embodiments. In this application, the electronic device includes a server, a gateway, sub-devices, etc., and the sub-devices are devices such as Internet of Things (IoT) devices.

[0208] Embodiments of this disclosure can be implemented as an apparatus with any suitable hardware, firmware, software, or any combination thereof, configured as desired. This apparatus may include electronic devices such as servers (clusters) and terminal devices such as IoT devices.

[0209] Figure 10 An exemplary apparatus 1300 is schematically shown that can be used to implement the various embodiments of this application.

[0210] In one embodiment, Figure 10 An exemplary device 1300 is shown, which includes one or more processors 1302, a control module (chipset) 1304 coupled to at least one of the processors 1302, a memory 1306 coupled to the control module 1304, a non-volatile memory (NVM) / storage device 1308 coupled to the control module 1304, one or more input / output devices 1310 coupled to the control module 1304, and a network interface 1312 coupled to the control module 1304.

[0211] Processor 1302 may include one or more single-core or multi-core processors, and processor 1302 may include any combination of general-purpose processors or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, device 1300 can function as a server device such as a gateway in the embodiments of this application.

[0212] In some embodiments, apparatus 1300 may include one or more computer-readable media (e.g., memory 1306 or NVM / storage device 1308) having instructions 1314 and one or more processors 1302 that are combined with the one or more computer-readable media and configured to execute the instructions 1314 to implement the module and thus perform the actions in this disclosure.

[0213] In one embodiment, the control module 1304 may include any suitable interface controller to provide any suitable interface to at least one of the processors 1302 and / or any suitable device or component communicating with the control module 1304.

[0214] The control module 1304 may include a memory controller module to provide an interface to the memory 1306. The memory controller module may be a hardware module, a software module, and / or a firmware module.

[0215] Memory 1306 may be used, for example, to load and store data and / or instructions 1314 for device 1300. In one embodiment, memory 1306 may include any suitable volatile memory, such as suitable DRAM. In some embodiments, memory 1306 may include double data rate quad synchronous dynamic random access memory (DDR4 SDRAM).

[0216] In one embodiment, the control module 1304 may include one or more input / output controllers to provide interfaces to the NVM / storage device 1308 and (one or more) input / output devices 1310.

[0217] For example, NVM / storage device 1308 may be used to store data and / or instructions 1314. NVM / storage device 1308 may include any suitable non-volatile memory (e.g., flash memory) and / or may include any suitable (one or more) non-volatile storage devices (e.g., one or more hard disk drives (HDDs), one or more optical disc drives (CDs), and / or one or more digital universal optical disc (DVD) drives).

[0218] NVM / storage device 1308 may include storage resources that are physically part of a device on which device 1300 is mounted, or that can be accessed by the device without needing to be part of the device. For example, NVM / storage device 1308 may be accessed via a network via one or more input / output devices 1310.

[0219] One or more input / output devices 1310 may provide an interface for device 1300 to communicate with any other suitable device. Input / output devices 1310 may include communication components, pinyin components, sensor components, etc. Network interface 1312 may provide an interface for device 1300 to communicate via one or more networks. Device 1300 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols, such as accessing wireless networks based on communication standards, such as WiFi, 2G, 3G, 4G, 5G, etc., or combinations thereof.

[0220] In one embodiment, at least one of the processors 1302 may be logically packaged with one or more controllers (e.g., memory controller modules) of the control module 1304. In one embodiment, at least one of the processors 1302 may be logically packaged with one or more controllers of the control module 1304 to form a system-in-package (SiP). In one embodiment, at least one of the processors 1302 may be integrated with the logic of one or more controllers of the control module 1304 on the same die. In one embodiment, at least one of the processors 1302 may be integrated with the logic of one or more controllers of the control module 1304 on the same die to form a system-on-a-chip (SoC).

[0221] In various embodiments, device 1300 may be, but is not limited to, a server, desktop computing device, or mobile computing device (e.g., laptop computing device, handheld computing device, tablet computer, netbook, etc.). In various embodiments, device 1300 may have more or fewer components and / or different architectures. For example, in some embodiments, device 1300 includes one or more cameras, a keyboard, a liquid crystal display (LCD) screen (including a touchscreen display), a non-volatile memory port, multiple antennas, a graphics chip, an application-specific integrated circuit (ASIC), and a speaker.

[0222] This application provides an electronic device, including: one or more processors; and one or more machine-readable media having instructions stored thereon, which, when executed by the one or more processors, cause the electronic device to perform one or more methods as described in this application.

[0223] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0224] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0225] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable information processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable information processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0226] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable information processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0227] These computer program instructions can also be loaded onto a computer or other programmable information processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0228] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the embodiments of the present application.

[0229] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0230] The data processing method and apparatus based on streaming computing provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A data processing method based on stream computing, characterized in that, A method applied to one of a plurality of stream computing nodes in a stream computing cluster, the method comprising: In a scenario in which the stream computing node processes stream data, detecting whether utilization of storage space of a local memory of the stream computing node is greater than or equal to a first preset threshold; In a case where the utilization of the storage space of the local memory of the stream computing node is greater than or equal to the first preset threshold, determining whether tolerance of the scenario to processing timeliness is higher than or equal to a preset tolerance; In a case where the tolerance of the scenario to processing timeliness is higher than or equal to the preset tolerance, interrupting obtaining, from a data source, stream data to be computed, the data source being configured to store the stream data to be computed, the stream data in the data source being configured to be computed by the stream computing node through a computing manner specified in a stream computing task; and then, in a case where the utilization of the storage space of the local memory of the stream computing node is less than the first preset threshold, obtaining, from the data source, the stream data to be computed, computing the stream data to be computed according to the computing manner specified in the stream computing task, and storing a computation result in the local memory of the stream computing node; In a case where the tolerance of the scenario to processing timeliness is lower than the preset tolerance, submitting, to a control node in the stream computing cluster, an allocation request, the allocation request being configured to request allocation of more storage space for the local memory of the stream computing node; and in a case where the control node allocates more storage space for the local memory of the stream computing node according to the allocation request, obtaining, from the data source, the stream data to be computed, computing the stream data to be computed according to the computing manner specified in the stream computing task, and storing a computation result in the allocated storage space.

2. The method of claim 1, wherein, The method further comprises: In a case where a duration of the interruption of obtaining, from the data source, the stream data to be computed is greater than a preset duration corresponding to the scenario, performing the step of submitting, to the control node in the stream computing cluster, the allocation request.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: In a case where the utilization of the storage space of the local memory of the stream computing node is less than a second preset threshold after the interruption, submitting, to the control node, a recovery request, the recovery request being configured to recover part of the storage space allocated for the local memory of the stream computing node, the second preset threshold being less than the first preset threshold, so that the control node recovers part of the storage space in the local memory of the stream computing node according to the recovery request.

4. A data processing method based on stream computing, characterized in that, A method applied to a control node outside a plurality of stream computing nodes in a stream computing cluster, the method comprising: The allocation request is sent to the control node when it is detected that the utilization of the storage space of the local memory of the stream computing node is greater than or equal to a first preset threshold and it is determined that the tolerance of the scenario to processing timeliness is lower than a preset tolerance; the allocation request is sent to the control node after the stream computing node interrupts the acquisition of the stream data to be calculated from the data source when it is detected that the utilization of the storage space of the local memory of the stream computing node is greater than or equal to the first preset threshold and it is determined that the tolerance of the scenario to processing timeliness is higher than the preset tolerance, and it is determined that the interruption duration of the acquisition of the stream data to be calculated from the data source is greater than a preset duration corresponding to the scenario; According to the allocation request, more storage space is allocated for the local memory of the stream computing node, so that the stream computing node acquires the stream data to be calculated from the data source, calculates the stream data to be calculated according to a calculation manner specified in the stream computing task, and stores the calculation result in the allocated storage space.

5. The method of claim 4, wherein, The method further comprises: receiving a recovery request submitted by the stream computing node, the recovery request being used to recover part of the storage space allocated for the local memory of the stream computing node, the recovery request being submitted to the control node when the utilization of the storage space of the local memory of the stream computing node is less than a second preset threshold, the second preset threshold being less than the first preset threshold; recovering part of the storage space in the local memory of the stream computing node according to the recovery request.

6. A data processing method based on stream computing, characterized in that, The method applied to the development node comprises: acquiring computer program code developed for a stream computing task; the computer program code at least comprises a first table creation statement and a write statement; the first table creation statement at least comprises a table creation instruction, a first table name and a first data storage type; the first table creation statement is used to instruct to create a first database table with the first table name according to the table creation instruction and to instruct that the data storage type applicable to the first database table is the first data storage type; the first data storage type comprises an external storage type; the write statement at least comprises a write instruction and the first table name and is used to instruct to write a calculation result in the first database table according to the write instruction and the first table name; finding the first data storage type in the first table creation statement; replacing the first data storage type in the first table creation statement with a third data storage type set in advance to obtain new computer program code, the third data storage type comprising a local storage type; generating a stream computing task according to the new computer program code, the stream computing task being applicable to an offline verification environment and an online environment; delivering the stream computing task to a stream computing node; The stream computing node is used to: detect whether the utilization of the storage space of the local memory of the stream computing node is greater than or equal to a first preset threshold in a scenario in which the stream computing node processes stream data. In a case where the utilization of the storage space of the local memory of the stream computing node is greater than or equal to a first preset threshold, it is determined whether the tolerance of the scenario to processing timeliness is higher than or equal to a preset tolerance; In a case where the tolerance of the scenario to processing timeliness is higher than or equal to the preset tolerance, the acquisition of the stream data to be computed from the data source is interrupted, the data source is configured to store the stream data to be computed, and the stream data in the data source is configured to be computed by the stream computing node through a computing manner specified in a stream computing task; and then, in a case where the utilization of the storage space of the local memory of the stream computing node is less than the first preset threshold, the stream data to be computed is acquired from the data source again, the stream data to be computed is computed according to the computing manner specified in the stream computing task, and the computing result is stored in the local memory of the stream computing node; In a case where the tolerance of the scenario to processing timeliness is lower than the preset tolerance, an allocation request is submitted to a control node in the stream computing cluster, the allocation request is configured to request more storage space to be allocated to the local memory of the stream computing node; and in a case where the control node allocates more storage space to the local memory of the stream computing node according to the allocation request, the stream data to be computed is acquired from the data source, the stream data to be computed is computed according to the computing manner specified in the stream computing task, and the computing result is stored in the allocated storage space.

7. A data processing method based on stream computing, characterized in that, The method is applied to one of a plurality of stream computing nodes in a stream computing cluster, and the method comprises: receiving an acquisition request sent by a development node, the acquisition request being configured to acquire a computing result stored in a local memory of the stream computing node; The computing result is obtained in a case where, in a scenario of processing stream data by the stream computing node, it is detected that the utilization of the storage space of the local memory of the stream computing node is greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario to processing timeliness is higher than or equal to a preset tolerance, the acquisition of the stream data to be computed from a data source is interrupted, and then, in a case where the utilization of the storage space of the local memory of the stream computing node is less than the first preset threshold, the stream data to be computed is acquired from the data source again, the stream data to be computed is computed according to a computing manner specified in a stream computing task, and the computing result is stored in an idle space of the local memory of the stream computing node; the data source is configured to store the stream data to be computed, and the stream data in the data source is configured to be computed by the stream computing node through the computing manner specified in the stream computing task; The computing result is that, in a scenario of processing stream data by the stream computing node, it is detected that the utilization of the storage space of the local memory of the stream computing node is greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario to processing timeliness is lower than a preset tolerance, a distribution request is submitted to a control node in a stream computing cluster, the distribution request is used to request to allocate more storage space for the local memory of the stream computing node, and in a case where the control node allocates more storage space for the local memory of the stream computing node according to the distribution request, the stream computing node obtains stream data to be computed from a data source, and computes the stream data to be computed according to a computing manner specified in a stream computing task and stores the computing result in the allocated storage space. The computing result stored in the local memory of the stream computing node is read. The read computing result is sent to the development node.

8. The method of claim 7, wherein, The method further includes: The read computing result is deleted from the local memory of the stream computing node.

9. A data processing method based on stream computing, characterized in that, The method applied to the development node includes: An obtaining request is sent to one of a plurality of stream computing nodes in a stream computing cluster, the obtaining request is used to obtain a computing result stored in a local memory of the stream computing node; The computing result is that, in a scenario of processing stream data by the stream computing node, it is detected that the utilization of the storage space of the local memory of the stream computing node is greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario to processing timeliness is lower than a preset tolerance, an obtaining request is sent to one of a plurality of stream computing nodes in a stream computing cluster, the obtaining request is used to obtain a computing result stored in a local memory of the stream computing node, and in a case where the control node allocates more storage space for the local memory of the stream computing node according to the distribution request, the stream computing node obtains stream data to be computed from a data source, and computes the stream data to be computed according to a computing manner specified in a stream computing task and stores the computing result in the allocated storage space. The computing result is that, in a scenario of processing stream data by the stream computing node, it is detected that the utilization of the storage space of the local memory of the stream computing node is greater than or equal to a first preset threshold, and it is determined that the tolerance of the scenario to processing timeliness is lower than a preset tolerance, a distribution request is submitted to a control node in a stream computing cluster, the distribution request is used to request to allocate more storage space for the local memory of the stream computing node, and in a case where the control node allocates more storage space for the local memory of the stream computing node according to the distribution request, the stream computing node obtains stream data to be computed from a data source, and computes the stream data to be computed according to a computing manner specified in a stream computing task and stores the computing result in the allocated storage space. The computing result returned by the stream computing node according to the obtaining request is received.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the steps of the method of any one of claims 1 to 9 when executing the program.

11. A computer readable storage medium, characterized in that, A computer program is stored on a computer-readable storage medium, which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 9.

Images