Python operator scheduling method and device based on streaming computing framework, and storage medium

Abstract

The invention discloses a python operator scheduling method and device based on a streaming computing framework, a storage medium and computer equipment. The method comprises the steps that a task submitting end submits a task execution request to a streaming computing management service; the streaming computing management service responds to the task execution request, generates a target task containing a main program file, and sends the target task to a streaming computing cluster; the streaming computing cluster receives a target task, and executes a main program file through a Python execution environment integrated in a streaming computing framework, so as to call a user-defined data source component and a user-defined data output component running on a Java virtual machine through a communication channel between a Python virtual machine and the Java virtual machine, and constructing a data processing assembly line through the user-defined data source component, the user-defined data output component and the data processing logic defined in the main program file, and triggering the data processing assembly line to be executed in the streaming computing framework.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a Python operator scheduling method and apparatus, storage medium, and computer equipment based on a streaming computing framework. Background Technology

[0002] With the deep integration of big data and artificial intelligence technologies, enterprises are increasingly demanding real-time data processing capabilities. Apache Flink, as an advanced stream and batch processing framework, has become a core engine in the field of real-time computing. Meanwhile, Python, with its rich ecosystem of libraries in data science and machine learning (such as NumPy, Pandas, and TensorFlow) and concise syntax, has become the preferred language for data scientists and algorithm engineers. Therefore, integrating Flink's computing power with Python's ecosystem advantages, and lowering the barrier for Python developers to use enterprise-level stream processing technologies, is of great significance for promoting the integrated application of big data and AI.

[0003] However, under current technology, the integration of Python and Flink faces significant obstacles. First, task submission heavily relies on command lines, requiring users to have a complete Flink and Python runtime environment and to manually manage and upload program files. This process is cumbersome and difficult to integrate with existing scheduling and management systems, resulting in low automation. Second, and more fundamentally, Flink runs on a Java Virtual Machine, while user logic is implemented in Python. The inherent language barrier between these two runtime environments makes it difficult for Python users to leverage the rich Java / Scala data connectors within the Flink ecosystem. The officially provided Python versions of the Source and Sink components are very limited in both functionality and quantity, failing to flexibly adapt to diverse enterprise data systems and severely restricting the application of Python on Flink in real-world production environments. Summary of the Invention

[0004] In view of this, this application provides a Python operator scheduling method and apparatus, storage medium, and computer device based on a streaming computing framework. By introducing a streaming computing management service, it receives task execution requests and automatically submits target tasks to the streaming computing cluster, realizing the service-oriented and automated scheduling of tasks and effectively solving the drawbacks of command-line operations. At the architectural level, by establishing an efficient communication channel between the Python virtual machine and the Java virtual machine, applications written in Python can directly call custom data sources and custom data output components running on the JVM, breaking the input / output flexibility limitations caused by language barriers. This allows Python developers to easily apply the powerful capabilities of the streaming computing framework and freely access various data systems, realizing the flexible and efficient application of the streaming computing framework.

[0005] According to one aspect of this application, a Python operator scheduling method based on a streaming computing framework is provided, comprising: The task submission client submits a task execution request to the streaming computing management service; In response to the task execution request, the streaming computing management service generates a target task containing a main program file and sends the target task to the streaming computing cluster. The main program file is a program file written in Python that defines data processing logic, and the data processing logic includes at least one Python operator. The streaming computing cluster receives the target task and executes the main program file of the target task through the Python execution environment integrated in the streaming computing framework. It calls the custom data source component and custom data output component running on the Java Virtual Machine through the communication channel between the Python virtual machine and the Java Virtual Machine. Through the custom data source component, the custom data output component, and the data processing logic defined in the main program file, a data processing pipeline is constructed and triggered to execute in the streaming computing framework.

[0006] According to another aspect of this application, a Python operator scheduling device based on a streaming computing framework is provided, comprising: The task submission end is used to submit task execution requests to the streaming computing management service; The streaming computing management service is used to respond to the task execution request, generate a target task containing a main program file, and send the target task to the streaming computing cluster. The main program file is a program file written in Python that defines data processing logic, and the data processing logic includes at least one Python operator. The streaming computing cluster is used to receive the target task, execute the main program file of the target task through the Python execution environment integrated in the streaming computing framework, and call the custom data source component and custom data output component running on the Java virtual machine through the communication channel between the Python virtual machine and the Java virtual machine. Through the custom data source component, the custom data output component and the data processing logic defined in the main program file, a data processing pipeline is constructed and the data processing pipeline is triggered to execute in the streaming computing framework.

[0007] According to another aspect of this application, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the above-described Python operator scheduling method based on a streaming computing framework.

[0008] According to another aspect of this application, a computer device is provided, including a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, wherein the processor executes the program to implement the above-described Python operator scheduling method based on a streaming computing framework.

[0009] By employing the above technical solutions, this application provides a Python operator scheduling method and apparatus, storage medium, and computer device based on a streaming computing framework. By introducing a streaming computing management service, it receives task execution requests and automatically submits target tasks to the streaming computing cluster, achieving service-oriented and automated task scheduling and effectively solving the drawbacks of command-line operations. At the architectural level, by establishing an efficient communication channel between the Python virtual machine and the Java virtual machine, applications written in Python can directly call custom data sources and custom data output components running on the JVM, breaking down the input / output flexibility limitations imposed by language barriers. This allows Python developers to easily utilize the powerful capabilities of the streaming computing framework and freely access various data systems, realizing the flexible and efficient application of the streaming computing framework.

[0010] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0011] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1The illustration shows a flowchart of a Python operator scheduling method based on a streaming computing framework provided in an embodiment of this application; Figure 2 This paper illustrates a schematic diagram of the structure of a Python operator scheduling device based on a streaming computing framework, as provided in an embodiment of this application. Figure 3 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0012] The present application will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the embodiments of the present application can be combined with each other.

[0013] This embodiment provides a Python operator scheduling method based on a streaming computing framework, such as... Figure 1 As shown, the method includes: Step 101: The task submitter submits a task execution request to the streaming computing management service.

[0014] Step 102: In response to the task execution request, the streaming computing management service generates a target task containing a main program file and sends the target task to the streaming computing cluster. The main program file is a program file written in Python that defines data processing logic, and the data processing logic includes at least one Python operator.

[0015] Step 103: The streaming computing cluster receives the target task and executes the main program file of the target task through the Python execution environment integrated in the streaming computing framework. It calls the custom data source component and custom data output component running on the Java Virtual Machine through the communication channel between the Python virtual machine and the Java Virtual Machine. Through the custom data source component, the custom data output component, and the data processing logic defined in the main program file, a data processing pipeline is constructed and the data processing pipeline is triggered to execute in the streaming computing framework.

[0016] In this embodiment, the task submission client initiates the entire scheduling process. Specifically, the task submission client can be a client component that, upon user triggering, submits a task execution request to the streaming computing management service via the HTTP protocol. Here, the streaming computing management service can be a self-developed Java service named Flink-Base. The task execution request can encapsulate metadata required to execute the Flink Python task, such as the task identifier and the corresponding main program file path, thereby realizing the transformation from traditional manual command-line submission to service-oriented, automated scheduling.

[0017] Next, upon receiving a task execution request, the streaming computing management service can assemble a specific target task and send it to the streaming computing cluster (i.e., the Flink cluster). Here, the task execution request may include basic task information (such as task name, resource requirements, priority, etc.) and the path to the main program file, while the target task may include the main program file. The main program file, i.e., the user-defined Python script, contains data processing logic (such as calls to one or more Python operators, data transformation rules, etc.).

[0018] Furthermore, once the streaming computing cluster receives the target task, it can launch a Python execution environment to run the main program file. During the execution of this main program file, communication between Python and Java can be established, specifically through the Py4J library. The communication channel established by Py4J allows access to custom Java classes in the JVM, thereby enabling calls to custom data source components and custom data output components. Then, a data processing pipeline can be constructed based on the custom data source components, custom data output components, and the data processing logic defined in the main program file. Specifically, a data source component can be added using `env.add_source(RedisSourceFunction)`. When multiple data source components exist, they can be merged using the `union()` method, and the triggering conditions can be controlled using a custom stream trigger. Next, the core data processing logic, i.e., calling the "Python operator," is added using `processed = windowed_stream.process(BaseProcessFunction(...))`. Finally, a data output component is added using `processed.add_sink(DataCenterHistorySinkFunction)`. This completes the construction of a full data processing pipeline.

[0019] After completing all the steps described above to build the data processing pipeline, you can further submit the pipeline to the streaming computing framework by calling the `env.execute("Flink Python Stream Job")` method. Afterward, the data processing pipeline will begin running continuously. Data will flow in from the data source component, be processed by Python operators, and finally flow out through the data output component, putting the entire target task into a long-term stable execution state.

[0020] By applying the technical solution of this embodiment, the task submission end submits a task execution request to the streaming computing management service via the HTTP protocol. Then, upon receiving the task execution request, the streaming computing management service can assemble a specific target task and send it to the streaming computing cluster. Further, when the streaming computing cluster receives the target task, it can start a Python execution environment to run the main program file. During the execution of this main program file, communication between Python and Java can be established, thereby enabling the invocation of custom data source components and custom data output components. Subsequently, a data processing pipeline can be constructed based on the custom data source components, custom data output components, and the data processing logic defined in the main program file. After completing all the above steps for constructing the data processing pipeline, the data processing pipeline can be submitted to the streaming computing framework. Thereafter, the data processing pipeline begins to run continuously; data begins to flow in from the data source component, is processed by Python operators, and finally flows out through the data output component, and the entire target task enters a long-term stable execution state. This application's embodiments introduce a streaming computing management service to receive task execution requests and automatically submit target tasks to the streaming computing cluster, achieving service-oriented and automated task scheduling and effectively solving the drawbacks of command-line operations. At the architectural level, by establishing an efficient communication channel between the Python virtual machine and the Java virtual machine, applications written in Python can directly call custom data sources and custom data output components running on the JVM, breaking the input / output flexibility limitations imposed by language barriers. This allows Python developers to easily utilize the powerful capabilities of the streaming computing framework and freely access various data systems, realizing the flexible and efficient application of the streaming computing framework.

[0021] In this embodiment of the application, optionally, the step 103 of "constructing a data processing pipeline through the custom data source component, the custom data output component, and the data processing logic defined in the main program file" includes: constructing an input end of the data stream through the custom data source component to access the data stream from an external system; configuring the data processing logic defined in the main program file as a data processing node to process the data stream from the input end; and constructing an output end of the data stream through the custom data output component to output the data stream processed by the data processing node to the target system through the output end.

[0022] In this embodiment, the data processing pipeline can be constructed as follows. First, a custom data source component is used to construct the input end of the data stream, allowing data to be accessed from external systems. In practical applications, external systems can encompass various types, such as database systems storing large amounts of business data, or message queues that continuously push in new data messages. The custom data source component can be customized according to the specific characteristics and access requirements of the external system. For example, if the external system is a MySQL database, the custom data source component can implement a connection driver for MySQL, configure corresponding connection parameters such as database address, port, username, and password, and then read data from the specified table using SQL query statements to form a continuous data stream. If the external system is a Kafka message queue, the custom data source component can pull message data from Kafka according to the Kafka protocol, specifying the topic, partition, and other information to be consumed, and construct a data stream.

[0023] Next, the data processing logic defined in the main program file is configured as a data processing node to process the data stream from the input end. The data processing logic defined in the main program file contains various complex operations, which exist in the form of Python operators. For example, there might be an operator for data cleaning, capable of identifying and removing noisy data, duplicate data, etc., from the data stream; and another operator for data transformation, capable of converting data from one format to another. When building the data processing pipeline, these Python operators can be combined in a certain order and logical relationship according to the definitions in the main program file to form a complete data processing node.

[0024] Furthermore, a custom data output component can be used to construct the output end of the data stream, allowing the data stream processed by the data processing nodes to be output to the target system. The custom data output component can be designed according to the needs of the target system. For example, if the target system is an Elasticsearch search engine, the custom data output component can organize and encapsulate the processed data according to the Elasticsearch index format, and then send the data to the Elasticsearch cluster for storage and indexing via HTTP requests.

[0025] In a specific implementation, in the main program file, a custom data source component can be constructed by calling the execution environment's 'add_source' method and passing in a custom Java data source component. The specific code can be represented as 'ds = env.add_source(RedisSourceFunction)'. Here, 'RedisSourceFunction' is a class defined in Python, but in its 'init' method, it dynamically accesses the Java implementation in the JVM through the Py4J gateway. That is, when the data processing pipeline runs, the actual data reading work is performed by Java code running on the JVM that can interact with external systems such as Redis, thus realizing the function of accessing data streams from external systems.

[0026] In the main program file, data processing nodes can be constructed by calling the `process` method of `DataStream` and passing in a custom Python processing function. Specifically, this can be represented as `processed = windowed_stream.process(BaseProcessFunction(...), output_type=list_row_type)`. Here, `BaseProcessFunction` represents the "data processing logic," configured as a data processing node in the data processing pipeline. Its key implementation lies in the fact that it does not contain fixed business logic code; instead, it acts as a dynamic loader. At runtime (in its `open` method), it dynamically imports and instantiates user-specified Python modules and classes using the lines `module = __import__(self.module_name)` and `cls = getattr(module, self.class_name)`. Subsequently, in the 'process' method, the specific business function is obtained by 'method = getattr(self.instance,self.function_name)' and 'method(values)' is called, thereby handing over the incoming data stream to this dynamically loaded, pure Python-implemented "operator" for processing, achieving a high degree of decoupling and flexible scheduling between business logic and the framework.

[0027] In the main program file, mirroring the input method, the `add_sink` method of `DataStream` is called, passing in a custom Java output component. Specifically, this can be represented as `processed.add_sink(DataCenterHistorySinkFunction)`. Similar to the data source component, `DataCenterHistorySinkFunction` is also a Python wrapper class. Internally, it uses the Py4J gateway to obtain the Java `SinkFunction` implemented in the JVM via the line of code `jvmSinkFunction = get_gateway().jvm.com.jiefanglian.sink.alg.DataCenterHistorySinkFunction`. Therefore, the data stream processed by the Python operators is ultimately passed to this custom Java component running on the JVM, which is responsible for efficiently and reliably writing the data to specified target systems such as databases and message queues, thus forming a complete data processing loop.

[0028] Optionally, in this embodiment, the custom data source component includes multiple components, and the data processing pipeline is further equipped with a data stream merging component and a stream trigger. Each custom data source component is used to access a data stream from a corresponding external system. The data stream merging component is used to perform real-time merging processing on the data streams accessed by each custom data source component to obtain a merged data stream. The stream trigger is used to receive the merged data stream and control its waiting state. When the latest merged data stream meets preset conditions, it triggers the transmission of the merged data stream that meets the preset conditions to the backend data processing node.

[0029] In this embodiment, multiple custom data source components can be created in the main program file, for example, by calling the 'env.add_source' method multiple times or creating multiple instances of 'RedisSourceFunction' and storing them in a list 'data_stream_sources = []'. The data stream merging component is not an independent entity, but refers to the processing logic and operations that merge multiple data streams. The stream trigger is a custom Java class configured as a window trigger in the data processing pipeline. All of the above components together constitute the key control mechanism for processing multi-source asynchronous data.

[0030] For each independent data source component, such as 'RedisSourceFunction', its Java implementation encapsulates the connection and communication protocol with a specific external system (such as Redis), responsible for continuously extracting data from that system and generating an independent data stream. Subsequently, a merge operation is triggered, merging the data streams from various data sources through a data stream merging component. This can be accomplished using the 'union()' method of the Flink DataStream API. This method logically converges multiple independent data streams into a unified "merged data stream," and all subsequent processing operations will be based on this merged data stream, thus achieving real-time integration of data during streaming processing.

[0031] The stream trigger is associated with the merged data stream through a window mechanism, specifically represented by '.trigger(AllStreamsTrigger.AllStreamsTrigger(len(data_stream_sources), timeout_seconds))'. The instantiation parameters clearly specify two key "preset conditions": the first parameter 'len(data_stream_sources)' requires waiting for data to arrive from all N data sources (i.e., meeting the quantity condition), and the second parameter 'timeout_seconds' sets the maximum waiting time (i.e., the timeout condition). The stream trigger continuously monitors the status of the merged data stream in the background. It will only trigger when either of these conditions is met: all data sources have arrived, or the preset timeout period has been reached. This triggers the window to release and transmit the currently cached data set as a whole to downstream data processing nodes for processing, thereby ensuring the integrity of the data processing window or the low latency of the system.

[0032] Optionally, in this embodiment of the application, the preset condition is that the latest merged data stream contains data streams of all custom data source components, and / or the waiting time reaches a preset time threshold.

[0033] In this embodiment, the preset time threshold is the aforementioned maximum waiting time.

[0034] In this embodiment of the application, optionally, the data processing logic defined in the main program file further includes a processing function; the processing function is configured to perform the following steps to dynamically schedule the Python operators: during the operation of the streaming computing cluster, dynamically loading the Python module files of each Python operator included in the data processing logic into the Python execution environment in the streaming computing framework; obtaining a specified class or a specified function from the loaded Python module, and creating an executable instance based on the specified class or specified function; when a data stream is received from the custom data source component, calling the executable instance to perform data processing, and outputting the processed data stream to the custom data output component.

[0035] In this embodiment, the main program file may also include processing functions. During the operation of the streaming computing cluster, these functions can dynamically load the Python module files of each Python operator contained in the data processing logic into the Python execution environment of the streaming computing framework, based on specific scheduling strategies and current data processing needs. In streaming computing scenarios, data flows continuously. To flexibly handle data processing needs of different types and complexities, each Python operator in the data processing logic is encapsulated in different Python module files. When the streaming computing cluster starts running, these Python module files are dynamically loaded into the Python execution environment of the streaming computing framework according to actual data processing needs. For example, in a real-time recommendation system, the data processing logic may contain multiple operators, such as data preprocessing operators, feature extraction operators, and model inference operators, which are located in different Python module files. Based on the needs of the current data processing stage, the corresponding module files can be dynamically loaded, making these operators available in the Python execution environment and preparing for subsequent data processing. This dynamic loading method improves the system's flexibility and scalability, enabling flexible combination and adjustment of operators according to different business scenarios and data processing needs.

[0036] After Python module files are loaded into the Python execution environment, specific classes or functions need to be retrieved from the loaded modules in order to actually execute the data processing logic within these modules. These specified classes or functions are predefined and encapsulate specific data processing operations. For example, in a data cleaning module, there might be a class specifically for handling missing values, containing various methods for handling missing values. Specifically, this class can be retrieved from the loaded modules based on configuration information or preset rules, and then an executable instance can be created based on this class. Similarly, if it is a functional operator, the specified function can be retrieved and wrapped into an executable object. By creating executable instances, abstract data processing logic can be transformed into concrete executable objects, providing a concrete execution entity for subsequent data processing operations.

[0037] In stream computing, custom data source components can continuously receive data streams from external systems and transmit these streams to the stream computing framework. When the Python execution environment receives a data stream from the custom data source component, it can call a previously created executable instance to process the data stream. The executable instance performs corresponding operations on the data stream based on its internally encapsulated data processing logic, such as data transformation, filtering, and aggregation. For example, if the executable instance is a data transformation class, it can convert the format of certain fields in the data stream or recalculate the data according to business rules. After processing, the processed data stream is output to a custom data output component. The custom data output component, according to a preset configuration, transmits the processed data stream to the target system, such as a database, file system, or other data processing system, completing one link in the entire data processing pipeline. In this way, the continuously flowing data stream can be processed in real time, and the processing results can be output promptly, achieving efficient data processing and analysis. In a specific embodiment, the processing function acts as the scheduling core, dynamically planning and allocating Python operators to provide execution logic for the data processing nodes, which then execute the data processing tasks according to the scheduling of the processing function.

[0038] In a specific implementation, the processing function may refer to the 'BaseProcessFunction' class, which is written by the user and inherits from Flink's 'ProcessFunction'. When the data processing pipeline is built based on the main program file, this class is instantiated and configured onto the data stream. It does not directly process the data, but rather acts as a high-level scheduler responsible for dynamically locating, loading, and executing Python operators containing specific business algorithms at runtime, thereby separating the stable process framework from the volatile business logic.

[0039] Once the target task has started in the streaming computing cluster and 'BaseProcessFunction' has been initialized, the loading process is specifically executed in its 'open(self, runtime_context: RuntimeContext)' method. The loading process is completed through standard Python reflection, where the dynamic aspect lies in the fact that the module name 'self.module_name' is not fixed in the code but comes from the "configuration information" passed in when the task is submitted. When this line of code is executed, the Python interpreter can find and load the specified '.py' file from the submitted ZIP dependency package (i.e., the subsequent archive file), importing it as a module into the current Flink Python execution environment, preparing for subsequent instantiation.

[0040] After successfully loading the module, 'BaseProcessFunction' continues to use reflection to retrieve the specific class or function object from the newly loaded module. This can be achieved using 'cls = getattr(module, self.class_name)', where 'self.class_name' also comes from external configuration. Subsequently, a concrete instance of the class is created using the statement 'self.instance = cls()', which is an immediately callable "executable instance". This instance carries the entity of user-specific business logic (such as data cleaning, feature calculation, model inference, etc.), but it is encapsulated and managed by the generalized 'BaseProcessFunction'.

[0041] When data is generated from the upstream data source and flows through 'BaseProcessFunction', its 'process(self, context, elements)' method can be automatically invoked by the Flink framework. At this point, this method does not directly process the data; instead, it acts as a proxy, retrieving the specific method from the previously created executable instance via 'method = getattr(self.instance, self.function_name)', and then calling 'method(values)' to deliver the incoming data 'values' to this dynamically loaded operator for actual computation. After computation, 'BaseProcessFunction' collects the processing results and outputs them via a 'return' statement. These results are then integrated into Flink's data stream, continuing to propagate downstream, and ultimately written to the target system connected by a custom data output component, thus forming a complete closed loop from dynamic loading to actual execution.

[0042] Optionally, in this embodiment of the application, the step 102, "the streaming computing management service responds to the task execution request and generates a target task containing a main program file", includes: the streaming computing management service responds to the task execution request, obtains the main program file and multiple Python module files that the main program file depends on, packages the multiple Python module files to obtain an archive file, and generates a target task based on the archive file and the main program file.

[0043] In this embodiment, the streaming computing management service can be in a constant listening state. Once a task execution request is received, it can obtain all the relevant files required to execute the task. First, it obtains the main program file, which is the core of the entire data processing logic and contains the main data processing flow and framework. Simultaneously, since the main program file may call multiple Python module files to implement specific data processing functions, the streaming computing management service can also obtain these dependent Python module files. After obtaining the multiple Python module files that the main program file depends on, the streaming computing management service can package these Python module files to obtain an archive file for easy management and transmission. The packaging method can be varied, such as using common compression algorithms to compress multiple module files into one file, or organizing these files into an archive package according to a specific format. This archive file can be seen as a collection containing all dependent modules, ensuring that the relevant module files can be loaded and used completely during subsequent task execution. After completing the packaging of the Python module files, the streaming computing management service can combine the main program file and the generated archive file to generate the target task. This target task is a complete task description containing all the information needed for execution, specifying the main program logic to be executed and the required dependent modules. After generating the target task, the streaming computing management service can submit it to the streaming computing cluster.

[0044] In this embodiment of the application, optionally, before step 103, which involves "calling the custom data source component and the custom data output component running on the Java Virtual Machine through the communication channel between the Python Virtual Machine and the Java Virtual Machine," the method further includes: the streaming computing cluster executing the main program file in the target task through the Python execution environment integrated in the streaming computing framework, so as to call the dependency addition method of the streaming computing framework based on the main program file, adding the storage path of the custom Java archive file package to the streaming computing framework through the dependency addition method, and loading the Java archive file package specified by the storage path into the classpath of the Java Virtual Machine through the streaming computing framework, wherein the Java archive file package includes the custom data source component and the custom data output component.

[0045] In this embodiment, before calling the custom data source component and custom data output component running on the Java Virtual Machine through the communication channel between the Python Virtual Machine and the Java Virtual Machine, the streaming computing cluster needs to perform a series of preparatory work to ensure that the subsequent components can be called correctly.

[0046] Specifically, to support the execution of the Python language, the streaming computing framework includes a built-in Python execution environment. When the streaming computing cluster receives a target task, it can submit the main program file of the target task to this Python execution environment for execution. The main program file is the core of the entire data processing flow; it contains the main logic and framework for data processing. Running the main program file in the Python execution environment initiates the entire data processing task. During execution, the main program file can call the dependency addition method provided by the streaming computing framework according to preset logic and requirements. This dependency addition method is an interface designed by the streaming computing framework to facilitate users in adding external dependencies. By calling this method, the main program file can pass the dependency information that needs to be added to the streaming computing framework, preparing for the subsequent loading of custom Java components. For example, in the main program file, when a custom Java data source component needs to be used, the dependency addition method can be called to inform the streaming computing framework that the relevant Java dependency needs to be added.

[0047] When the main program file calls the dependency addition method, the streaming computing framework can receive the storage path information of the custom Java archive file package passed by the method and add this path to the framework's dependency management system. In this way, the streaming computing framework knows where to find this custom Java archive file package, providing path guidance for subsequent loading operations.

[0048] After receiving the storage path of a custom Java archive file package, the streaming computing framework can load the Java archive file package specified by that path into the Java Virtual Machine's classpath according to the framework's internal mechanism. The Java Virtual Machine's classpath is an important path for Java programs to find classes and resources at runtime. Loading the custom Java archive file package into the classpath enables the Java Virtual Machine to recognize and use these custom components.

[0049] In one specific implementation, before the Python code in the main program file can successfully execute a call such as 'get_gateway().jvm.com.jiefanglian.source.alg.RedisSourceFunction' through the Py4J gateway, the dependency loading operation must be completed first. If this is missing, the JVM will throw a 'ClassNotFoundException' because it cannot find the corresponding Java class, causing the entire target task to fail.

[0050] When the streaming computing cluster starts executing the main program file, in the initial stage of building the data processing pipeline, a specific method can be executed, namely the 'add_jars' method provided by the Flink Python API (PyFlink), which is the dependency addition method that is called and belongs to the 'StreamExecutionEnvironment' object.

[0051] When `env.add_jars()` is called with a file path (such as `file: / / / opt / ... / source.jar`), this call is passed to the Flink Java framework in the background via the Py4J communication channel. After receiving this path parameter, the Flink framework (Java side) loads the JAR file pointed to by that path into the classpath of the JVM executing this task, enabling the JVM to recognize and load all the Java classes contained in these JAR packages. Only after these JAR packages containing the specific component implementations are successfully loaded do the aforementioned cross-language dynamic calls have a target object to operate on, and only then can the entire hybrid architecture pipeline be successfully built.

[0052] Furthermore, as Figure 1 In terms of specific implementation, this application provides a Python operator scheduling device based on a streaming computing framework, such as... Figure 2 As shown, the device includes: The task submission end is used to submit task execution requests to the streaming computing management service; The streaming computing management service is used to respond to the task execution request, generate a target task containing a main program file, and send the target task to the streaming computing cluster. The main program file is a program file written in Python that defines data processing logic, and the data processing logic includes at least one Python operator. The streaming computing cluster is used to receive the target task, execute the main program file of the target task through the Python execution environment integrated in the streaming computing framework, and call the custom data source component and custom data output component running on the Java virtual machine through the communication channel between the Python virtual machine and the Java virtual machine. Through the custom data source component, the custom data output component and the data processing logic defined in the main program file, a data processing pipeline is constructed and the data processing pipeline is triggered to execute in the streaming computing framework.

[0053] Optionally, the streaming computing cluster is used for: The custom data source component is used to construct the input end of the data stream, so as to access the data stream from external systems; Configure the data processing logic defined in the main program file as a data processing node that processes the data stream from the input end; The custom data output component is used to construct an output end for the data stream, so that the data stream processed by the data processing node can be output to the target system through the output end.

[0054] Optionally, the custom data source component includes multiple components, and the data processing pipeline is also equipped with a data stream merging component and a stream trigger; Each of the custom data source components is used to access data streams from the corresponding external system. The data stream merging component is used to perform real-time merging processing on the data streams accessed by each custom data source component to obtain a merged data stream. The stream trigger is used to receive the merged data stream and control the waiting state of the merged data stream. When the latest merged data stream meets the preset conditions, it triggers the transmission of the merged data stream that meets the preset conditions to the backend data processing node.

[0055] Optionally, the preset condition is that the latest merged data stream contains data streams from all custom data source components, and / or the waiting time reaches a preset time threshold.

[0056] Optionally, the data processing logic defined in the main program file further includes processing functions; the processing functions are configured to perform the following steps to dynamically schedule the Python operators: During the operation of the streaming computing cluster, the Python module files of each Python operator included in the data processing logic are dynamically loaded into the Python execution environment of the streaming computing framework. Retrieve a specified class or function from a loaded Python module, and create an executable instance based on the specified class or function; When a data stream is received from the custom data source component, the executable instance is invoked to process the data, and the processed data stream is output to the custom data output component.

[0057] Optionally, the streaming computing management service is used for: In response to the task execution request, the main program file and multiple Python module files that the main program file depends on are obtained, the multiple Python module files are packaged to obtain an archive file, and the target task is generated based on the archive file and the main program file.

[0058] Optionally, the streaming computing cluster is further used for: Before invoking the custom data source component and custom data output component running on the Java Virtual Machine via the communication channel between the Python Virtual Machine and the Java Virtual Machine, the main program file in the target task is executed through the Python execution environment integrated in the streaming computing framework. Based on the main program file, the dependency addition method of the streaming computing framework is called. The storage path of the custom Java archive file package is added to the streaming computing framework through the dependency addition method. The streaming computing framework then loads the Java archive file package specified by the storage path into the classpath of the Java Virtual Machine. The Java archive file package includes the custom data source component and the custom data output component.

[0059] It should be noted that other corresponding descriptions of the functional units involved in the Python operator scheduling device based on a streaming computing framework provided in this application embodiment can be found in the following references. Figure 1 The corresponding descriptions in the method will not be repeated here.

[0060] This application also provides a computer device, which may specifically be a personal computer, a server, a network device, etc. Figure 3 As shown, the computer device includes a bus, a processor, memory, and a communication interface, and may also include an input / output interface and a display device. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores location information. The network interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements the steps in the various method embodiments.

[0061] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0062] In one embodiment, a computer-readable storage medium is provided, which may be non-volatile or volatile, having stored thereon a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0063] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0064] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0065] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0066] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0067] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A Python operator scheduling method based on a streaming computing framework, characterized in that, include: The task submission client submits a task execution request to the streaming computing management service; In response to the task execution request, the streaming computing management service generates a target task containing a main program file and sends the target task to the streaming computing cluster. The main program file is a program file written in Python that defines data processing logic, and the data processing logic includes at least one Python operator. The streaming computing cluster receives the target task and executes the main program file of the target task through the Python execution environment integrated in the streaming computing framework. It calls the custom data source component and custom data output component running on the Java Virtual Machine through the communication channel between the Python virtual machine and the Java Virtual Machine. Through the custom data source component, the custom data output component, and the data processing logic defined in the main program file, a data processing pipeline is constructed and triggered to execute in the streaming computing framework.

2. The method according to claim 1, characterized in that, The process of constructing a data processing pipeline using the custom data source component, the custom data output component, and the data processing logic defined in the main program file includes: The custom data source component is used to construct the input end of the data stream, so as to access the data stream from external systems; Configure the data processing logic defined in the main program file as a data processing node that processes the data stream from the input end; The custom data output component is used to construct an output end for the data stream, so that the data stream processed by the data processing node can be output to the target system through the output end.

3. The method according to claim 2, characterized in that, The custom data source component includes multiple components, and the data processing pipeline is also equipped with a data stream merging component and a stream trigger; Each of the custom data source components is used to access data streams from the corresponding external system. The data stream merging component is used to perform real-time merging processing on the data streams accessed by each custom data source component to obtain a merged data stream. The stream trigger is used to receive the merged data stream and control the waiting state of the merged data stream. When the latest merged data stream meets the preset conditions, it triggers the transmission of the merged data stream that meets the preset conditions to the backend data processing node.

4. The method according to claim 3, characterized in that, The preset conditions are that the latest merged data stream contains data streams from all custom data source components, and / or the waiting time reaches a preset time threshold.

5. The method according to claim 1, characterized in that, The data processing logic defined in the main program file also includes processing functions; these processing functions are configured to perform the following steps to dynamically schedule the Python operators: During the operation of the streaming computing cluster, the Python module files of each Python operator included in the data processing logic are dynamically loaded into the Python execution environment of the streaming computing framework. Retrieve a specified class or function from a loaded Python module, and create an executable instance based on the specified class or function; When a data stream is received from the custom data source component, the executable instance is invoked to process the data, and the processed data stream is output to the custom data output component.

6. The method according to claim 5, characterized in that, In response to the task execution request, the streaming computing management service generates a target task containing a main program file, including: In response to the task execution request, the streaming computing management service obtains the main program file and multiple Python module files that the main program file depends on, packages the multiple Python module files to obtain an archive file, and generates the target task based on the archive file and the main program file.

7. The method according to claim 1, characterized in that, Before invoking the custom data source component and custom data output component running on the Java Virtual Machine via the communication channel between the Python Virtual Machine and the Java Virtual Machine, the method further includes: The streaming computing cluster executes the main program file of the target task through a Python execution environment integrated in the streaming computing framework. Based on the main program file, it calls the dependency addition method of the streaming computing framework to add the storage path of the custom Java archive file package to the streaming computing framework. The streaming computing framework then loads the Java archive file package specified by the storage path into the classpath of the Java Virtual Machine. The Java archive file package includes the custom data source component and the custom data output component.

8. A Python operator scheduling device based on a streaming computing framework, characterized in that, include: The task submission end is used to submit task execution requests to the streaming computing management service; The streaming computing management service is used to respond to the task execution request, generate a target task containing a main program file, and send the target task to the streaming computing cluster. The main program file is a program file written in Python that defines data processing logic, and the data processing logic includes at least one Python operator. The streaming computing cluster is used to receive the target task, execute the main program file of the target task through the Python execution environment integrated in the streaming computing framework, and call the custom data source component and custom data output component running on the Java virtual machine through the communication channel between the Python virtual machine and the Java virtual machine. Through the custom data source component, the custom data output component and the data processing logic defined in the main program file, a data processing pipeline is constructed and the data processing pipeline is triggered to execute in the streaming computing framework.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. A computer device, comprising a storage medium, a processor, and a computer program stored on the storage medium and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

Images