Spark operation flow control method, system and device

Abstract

The invention relates to the technical field of traffic processing, in particular to a Spark operation traffic control method, which comprises the following steps: under the condition that a new service container group is created, webhook injects an agent container and a wamplug-in container into the new service container group, collects and manages Pod information of the new service container group through an adapter, and constructs a service container group topological relation; in the operation process, hijacking all network traffic of the service container through the wasm plug-in container, extracting original traffic connection information carried in the network traffic, and sending the original traffic connection information to the agent container; the method comprises the following steps: reconstructing original flow connection information through an agent container to obtain target flow connection information, sending the target flow connection information to an adapter to decide that the adapter generates a flow control strategy based on the target flow connection information, processes the target flow connection information in combination with a current processing stage, generates a flow processing instruction, and sends the flow processing instruction to a server; and the flow processing instruction is transmitted to the Wasm plug-in container through the agent, and a flow processing decision is executed through the Wasm plug-in container to determine whether the flow is allowed to pass.

Description

Technical Field

[0001] The present application relates to the field of traffic processing technology, and in particular to a Spark job traffic control method, system and device. Background Art

[0002] There are many traffic control solutions at present, such as Istio, Kubernetes NetworkPolicy, etc. Istio and Kubernetes NetworkPolicy are two different network control tools, providing different traffic control capabilities. The main difference between them is that Istio is a service mesh solution, while KubernetesNetworkPolicy is the network policy within the Kubernetes cluster. The current mainstream traffic control solutions are mainly used to solve the traffic between services, but they cannot provide solutions for traffic control during the execution of SparkJOB tasks, nor can they solve the fine-grained traffic monitoring and control at different stages of the task execution process.

[0003] The two common ways to execute Spark tasks are Spark-submit and JavaAPI. Spark-submit scripts are usually used for submission during production deployment and release. Spark business jar packages are developed for different business needs and submitted to the Spark cluster environment for execution through Spark-submit scripts. Among them, the development of business jar packages includes the following steps: initializing the Spark basic environment code; reading the data source operation code; business logic conversion code; data result writing operation code; closing the Spark session code.

[0004] However, this operation mode will increase the complexity and labor cost when handling abnormal situations. There are two main types of abnormal situations: business logic problems and Spark basic environment problems. When the abnormality is a business logic problem, it is necessary to check the business logic code and Spark basic environment code, which increases the difficulty and time of troubleshooting. If there are many business packages, such operations will greatly increase the workload of manpower, thereby increasing enterprise costs.

[0005] In view of this, how to provide a Spark job flow control method to solve the separation problem of business code and Spark execution basic code and reduce workload and labor costs has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The embodiments of the present application provide a Spark job flow control method, a Spark job flow control system, a Spark job flow control device, an electronic device and a computer storage medium, which are used to solve the problem of separation of business code and Spark execution basic code.

[0007] In a first aspect of an embodiment of the present application, a Spark job flow control method is provided, comprising:

[0008] When a new service container group is created, webhook will inject agent container and wasmplug-in container into the new service container group, and collect and manage Pod information of the new service container group through adapter to build the service container group topology relationship, wherein the traffic controller includes wehook and adapter;

[0009] During operation, all network traffic of the business container is hijacked through the wasm plug-in container, the original traffic connection information carried in the network traffic is extracted, and sent to the agent container, wherein the original traffic connection information carries the source address and the target address; the data format of the target traffic connection information is unified and contains sufficient context information;

[0010] Reconstruct the original traffic connection information through the agent container to obtain the target traffic connection information, and send the target traffic connection information to the adapter for decision-making;

[0011] The adapter generates a traffic control strategy based on the target traffic connection information, processes the target traffic connection information in combination with the current processing stage, generates a traffic processing instruction, and passes the traffic processing instruction to the wasmplug-in container through the agent, and executes the traffic processing decision through the wasmplug-in container to determine whether to allow the traffic to pass, wherein the traffic control strategy carries the business container group relationship and the whitelist system.

[0012] In a second aspect of an embodiment of the present application, a Spark job flow control system is provided, comprising a flow controller, an agent container and a wasm plug-in container, wherein:

[0013] The flow controller includes wehook and adapter, which are used to inject agent container and wasmplug-in container into the new service container group, generate flow control strategy and manage service container group;

[0014] The agent container is used to receive the original traffic connection information sent by wasmplug-in, reconstruct the original traffic connection information to obtain the target traffic connection information, and send it to the adapter for decision-making; and receive and execute the traffic processing instructions from the adapter, and convert the traffic processing instructions into specific traffic processing decisions;

[0015] The wasm plug-in container is used to hijack all inbound and outbound traffic of the business container through the underlying network hook mechanism, extract the original traffic connection information carried in the network traffic, and send it to the agent container for processing; and execute traffic processing decisions to determine whether to allow traffic to pass.

[0016] In a third aspect of an embodiment of the present application, a Spark job flow control device is provided, including:

[0017] The injection module is configured to inject the agent container and the wasm plug-in container into the new business container group when a new business container group is created, and collect and manage the Pod information of the new business container group through the adapter to build the topological relationship of the business container group, wherein the traffic controller includes the wehook and the adapter;

[0018] The extraction module is configured to hijack all network traffic of the business container through the wasmplug-in container during operation, extract the original traffic connection information carried in the network traffic, and send it to the agent container, wherein the original traffic connection information carries the source address and the target address; the data format of the target traffic connection information is unified and contains sufficient context information;

[0019] A reconstruction module is configured to reconstruct the original flow connection information through an agent container to obtain target flow connection information, and send the target flow connection information to an adapter for decision making;

[0020] The generation module is configured as an adapter, which generates a flow control strategy based on the target flow connection information, processes the target flow connection information in combination with the current processing stage, generates a flow processing instruction, and passes the flow processing instruction to the wasm plug-in container through the agent, and executes the flow processing decision through the wasm plug-in container to determine whether to allow the flow to pass, wherein the flow control strategy carries the business container group relationship and the whitelist system.

[0021] In a fourth aspect of an embodiment of the present application, a computing device is provided, including:

[0022] Memory and processor;

[0023] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the above-mentioned Spark job flow control method are implemented.

[0024] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the above-mentioned Spark job flow control method are implemented.

[0025] The present application provides a Spark job flow control method, comprising: first, when a new business container group is created, webhook injects an agent container and a wasm plug-in container into the new business container group, and collects and manages the Pod information of the new business container group through an adapter to build a business container group topology relationship, wherein the flow controller includes wehook and adapter; then, during operation, through wasm The plug-in container hijacks all network traffic of the business container, extracts the original traffic connection information carried in the network traffic, and sends it to the agent container, wherein the original traffic connection information carries the source address and the destination address; the data format of the target traffic connection information is unified and contains sufficient context information; secondly, through the agent container, the original traffic connection information is reconstructed to obtain the target traffic connection information, and the target traffic connection information is sent to the adapter for decision-making; finally, the adapter generates a traffic control strategy based on the target traffic connection information, processes the target traffic connection information in combination with the current processing stage, generates a traffic processing instruction, and passes the traffic processing instruction to the wasmplug-in container through the agent, and the wasmplug-in container executes the traffic processing decision to determine whether to allow the traffic to pass, wherein the traffic control strategy carries the business container group relationship and the whitelist system.

[0026] The Spark job flow control method provided by the embodiment of the present application has the following beneficial effects:

[0027] On the one hand, it is security and reliability. The security of the system is achieved through a multi-layer protection mechanism. First, container injection ensures the integrity of the control component. Second, the precise management of the Adapter ensures the accuracy of the container group relationship and the whitelist. At the same time, the system can promptly identify and block illegal traffic, especially at the critical stage of data processing. In terms of reliability, the system implements mechanisms such as state persistence, fault recovery, performance optimization, and real-time monitoring to ensure the stability and reliability of traffic control.

[0028] Another aspect is scalability. The embodiments of the present application have good scalability. Through the automatic injection mechanism of webhook, the flow control capability can be easily extended to the newly created business container group. The container group management and whitelist system of Adapter supports dynamic updates and can adapt to changes in business scale. At the same time, the system reserves interfaces for integration with other security systems to support future expansion of functions and evolution of policies. This design ensures that the system can continue to develop as business needs change.

[0029] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, which can be implemented in accordance with the contents of the specification, and in order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0031] Figure 1 A schematic diagram of a Spark JOB framework of a Spark job flow control method provided in an embodiment of the present application;

[0032] Figure 2 A flow chart of a Spark job flow control method provided in an embodiment of the present application;

[0033] Figure 3 A flow hijacking diagram of a Spark job flow control method provided in an embodiment of the present application;

[0034] Figure 4 A result schematic diagram of a Spark job flow control system provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of the structure of a Spark job flow control device provided in an embodiment of the present application;

[0036] Figure 6 A structural block diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0037] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0038] There are many traffic control solutions, such as Istio, Kubernetes Network Policy, etc. Istio and Kubernetes Network Policy are two different network control tools, providing different traffic control capabilities. The main difference between them is that Istio is a service mesh solution, while Kubernetes Network Policy is a network policy within the Kubernetes cluster.

[0039] Among them, the main features of Kubernetes NetworkPolicy include: NetworkPolicy is a built-in function of Kubernetes, which is mainly responsible for network traffic control within the cluster, mainly for network communication between pods; Through NetworkPolicy, you can define allowed ingress (ingress) and egress (egress) traffic, for example, you can define which pods can communicate with other pods; NetworkPolicy is very effective for service-to-service (i.e. pod-to-pod) network isolation, which can enhance the security of the Kubernetes cluster.

[0040] The main features of Istio include: Istio is a service mesh solution that provides more advanced and complex traffic control capabilities, such as request routing, retries, fault injection, circuit breakers, etc.; Istio can control not only network communication between pods, but also communication between services, such as routing requests based on HTTP headers or URL paths; Istio provides comprehensive observation and measurement capabilities, such as request latency, success rate, and distributed tracing, which can help you better understand and monitor interactions between services; Istio provides powerful security features, such as automatic mTLS encryption and refined access control policies.

[0041] Obviously, the current mainstream traffic control solutions are mainly used to solve the traffic between services, but they cannot provide solutions for traffic control during the execution of Spark JOB tasks, nor can they solve the fine-grained traffic monitoring and control at different stages of the task execution process.

[0042] In view of this, in order to meet the needs of fine-grained traffic monitoring and control at different stages of the Spark JOB execution process, this application introduces the Spark JOB framework to output the current stage of the JOB execution process. Only by obtaining the current stage can different traffic control strategies be implemented for different stages.

[0043] See also Figure 1 , Figure 1 A schematic diagram of a Spark JOB framework of a Spark job flow control method provided in an embodiment of the present application. Figure 1 As shown, the Spark task running process executed by the Spark JOB framework is divided into two parts: the first part is to define the workflow on the Web side, schedule the workflow definition through task scheduling, and submit it to the Spark side for operation, wherein the Spark JOB framework includes the Web side and the Spark side; the workflow definition includes the program type, Spark version, business package reflection class of the main function, main program package, deployment method, task name, number of Drive cores, number of Drive memories, number of Executors, number of Executor memories, main program parameters, option parameters and resources; the second part is to execute the corresponding Spark task through the Spark computing engine. That is, the spark side is responsible for parsing parameters, and completes the execution of the spark task through the four steps of parsing parameters, reading data, business package reflection, and outputting data, and monitoring each step in stages. The workflow is as follows Figure 1 shown.

[0044] It should be noted that the program types here support Java, Scala, and Python languages.

[0045] The Spark version here supports Spark 3.

[0046] The main function Class (business package reflection class) here is the full path of the entry of the Spark business package program.

[0047] The main package here executes the jar package of the Spark program (uploaded through the resource center).

[0048] The deployment method here is natively supported by kubernetes spark submit.

[0049] The task name here (optional) is the name of the Spark program.

[0050] The Driver core number here is used to set the number of Driver cores. The corresponding core number can be set according to the actual production environment.

[0051] The Driver memory number here is used to set the Driver memory number. You can set the corresponding memory number according to the actual production environment.

[0052] The number of Executors here is used to set the number of Executors. The corresponding memory amount can be set according to the actual production environment.

[0053] The Executor memory number here is used to set the Executor memory number. The corresponding memory number can be set according to the actual production environment.

[0054] The main program parameters here are the input parameters for setting the Spark program.

[0055] The option parameters here support --jar, --files, and --conf formats.

[0056] If resource files are referenced in other parameters, you need to select and specify them in the resource.

[0057] Specifically, the method of submitting the scheduled workflow definition to the Spark end for operation includes: obtaining basic definition information of the task according to the task number run by the user, assembling parameters based on the basic definition information, and generating a spark-submit command form; executing the assembled spark-submit command through ProcessBuilder, and submitting the spark-submit command to the spark computing engine to execute the corresponding Spark task.

[0058] Specifically, submitting the spark-submit command to the spark computing engine to execute the corresponding Spark task includes:

[0059] The JSON string parameter passed in the spark-submit command is parsed to obtain a spark parameter object, wherein the parameter object carries connection information and table information for reading data, as well as a business package reflection class. The business package reflection class carries the location of the business package during reflection, the location of the business package used when executing the task, the class name, and output result information; the output result information here carries information on which table the output result is output to, connection information, etc.

[0060] According to the connection information and table information set in the input source in the workflow definition, the data is passed to the business package for data processing and data reading;

[0061] According to the business package reflection class set in the workflow definition, determine the fixed calculation method in the business package to execute the corresponding business logic, and send the business logic to the output result end, wherein the data used in the business package comes from the data read from the connection information and table information set in the input source;

[0062] According to the connection information and table information set in the output source in the workflow definition, the results are written to the corresponding library and table, and the execution of the current task is terminated.

[0063] In addition, the Spark task also includes stage monitoring, wherein submitting the spark-submit command to the spark computing engine to execute the corresponding Spark task includes: recording and monitoring the process of parsing parameters, reading data, business package reflection and outputting data respectively.

[0064] After the introduction of the Sparkjob development framework, business personnel only need to care about the business logic of their own data processing, which can reduce the size of the business package and the disk overhead. In the case of a large number of business packages, it can save a lot of storage costs for the enterprise. When a problem occurs during task execution, the problem point can be quickly found, whether it is a problem in the business package or in the spark execution environment package, improving the efficiency of finding problems and the speed of processing.

[0065] The introduction of the traffic control component will give us the ability to deeply monitor the Spark Job traffic, which will greatly enhance our control over the entire execution process. By monitoring and analyzing traffic data, this component can capture, intercept and record abnormal traffic in real time during the execution process, providing us with instant traffic anomaly alarm and recording functions. In addition, this measure greatly improves the security and integrity of the data. Through real-time control of data flow, we can ensure that SparkJob cannot read or transfer unauthorized data during data processing, thereby ensuring data security and compliance.

[0066] See also Figure 2 , Figure 2 The following is a flow chart of a Spark job flow control method provided in an embodiment of the present application. Figure 2 As shown, the specific steps include the following steps.

[0067] Step S202: When a new service container group is created, webhook injects an agent container and a wasm plug-in container into the new service container group, collects and manages the Pod information of the new service container group through the adapter, and builds a service container group topology relationship, wherein the traffic controller includes wehook and adapter;

[0068] Step S204: during operation, all network traffic of the business container is hijacked through the wasm plug-in container, original traffic connection information carried in the network traffic is extracted, and sent to the agent container, wherein the original traffic connection information carries a source address and a target address; the data format of the target traffic connection information is unified and contains sufficient context information;

[0069] Step S206: reconstructing the original flow connection information through the agent container to obtain target flow connection information, and sending the target flow connection information to the adapter for decision-making;

[0070] Step S208: The adapter generates a traffic control strategy based on the target traffic connection information, processes the target traffic connection information in combination with the current processing stage, generates a traffic processing instruction, and passes the traffic processing instruction to the wasmplug-in container through the agent. The wasmplug-in container executes the traffic processing decision to determine whether to allow the traffic to pass, wherein the traffic control strategy carries the business container group relationship and the whitelist system.

[0071] It should be noted that the traffic control strategy here is based on two core dimensions: business container group relationships and the whitelist system. When the data processing phase enters the calculation phase, the system only allows communication between Pods in the same business container group and between IPs on the whitelist. This strategy is executed through the Adapter's decision engine to ensure the security and isolation of the data processing process. When the decision engine receives a connection request, it verifies the relationship between the source address and the destination address, and makes a decision to release or reject it based on the current processing phase and the whitelist situation.

[0072] In the embodiment of the present application, the adapter generates a flow control strategy based on the target flow connection information, processes the target flow connection information in combination with the current processing stage, and generates a flow processing instruction, including:

[0073] When the data processing phase enters the calculation phase, based on the business container group relationship, the Pod information in the business container group is determined, and when all the Pods in the business group belong to the same business container group and exist in the whitelist system, a traffic control strategy is generated, wherein the traffic control strategy includes traffic release and traffic rejection;

[0074] The target traffic connection information is verified to determine a processing stage corresponding to the current stage, and in response to the current stage being in the execution stage, traffic is prohibited from flowing out of the service container group.

[0075] Specifically, the method for determining that all Pods in a business group belong to the same business container group includes:

[0076] When a traffic connection request is received, the relationship between the source address and the target address corresponding to the Pod in the business group is verified. In response to the source address and the target address being consistent, it is determined that the Pods in the business group all belong to the same business container group.

[0077] In the embodiment of the present application, the traffic processing instruction is passed to the wasmplug-in container through the agent, and the traffic processing decision is executed through the wasmplug-in container, including:

[0078] The traffic processing instruction is sent to the agent container, the traffic processing strategy is converted into a traffic processing decision, and is passed to the wasm plug-in container, and the traffic processing decision is executed by the wasm plug-in container.

[0079] In the embodiment of the present application, hijacking all network traffic of the business container through the wasm plug-in container includes: hijacking all network traffic of the business container through the underlying network hook mechanism, wherein the network traffic includes inbound traffic and outbound traffic.

[0080] In another aspect of the embodiment of the present application, by monitoring and analyzing traffic data, Sparkjob abnormal traffic can be captured, intercepted and recorded in real time during the execution process, providing instant traffic abnormality alarm and recording functions. Specifically including: decoupling solution of business execution package and spark basic environment; monitoring of task execution stage status.

[0081] See also Figure 3 , Figure 3 A flow hijacking diagram of a Spark job flow control method provided in an embodiment of the present application.

[0082] like Figure 3As shown in the figure, when the business container group is created, the traffic controller webhook will be injected into the agent and wasm plug-in containers; the wasmplug-in container will hijack all the traffic of the business container and send the connection information of the traffic to the agent, including: source address and destination address information; the agent will reconstruct the traffic connection information and send it to the adapter component for connection information verification; the adapter will verify the current stage, and if it is currently in the execution stage, it will prohibit the traffic from flowing out of the business container group; the adapter will reject illegal traffic and provide the agent to notify wasmplug-in to discard the rejected request data packets.

[0083] See also Figure 4 , Figure 4 A result schematic diagram of a Spark job flow control system provided in an embodiment of the present application.

[0084] like Figure 4 As shown in the figure, the overall architecture of the traffic hijacking and control implementation solution consists of three core components, including: traffic controller (including wehook and adapter), agent container and wasmplug-in container. This architectural design allows precise control and secure management of communication between container groups without modifying the business container.

[0085] Specifically, the traffic controller includes wehook and adapter, which are used to inject agent container and wasmplug-in container into the new business container group, generate traffic control strategy and manage business container group, that is, the traffic controller serves as a central management unit, responsible for container injection, formulation of traffic strategy and management of business container group.

[0086] The agent container is used to receive the original traffic connection information sent by wasmplug-in, obtain the target traffic connection information by reconstructing the original traffic connection information, and send it to the adapter for decision-making; and receive and execute the traffic processing instructions from the adapter, and convert the traffic processing instructions into specific traffic processing decisions.

[0087] That is, the Agent container serves as the collection and processing center of traffic information, and is responsible for receiving the original connection information from the Wasm Plugin, including key data such as the source address and the destination address. It will reconstruct and normalize this information to ensure that the data format passed to the adapter is unified and contains sufficient context information. At the same time, the Agent is also responsible for receiving and executing control instructions from the adapter, and converting these instructions into specific traffic control strategies.

[0088] The wasm plug-in container is used to hijack all inbound and outbound traffic of the business container through the underlying network hook mechanism, extract the original traffic connection information carried in the network traffic, and send it to the agent container for processing; and execute traffic processing decisions to determine whether to allow traffic to pass.

[0089] That is, the Wasm Plugin container is the core component that actually performs traffic hijacking and control. It can intercept all inbound and outbound traffic of the business container through the underlying network hook mechanism. For each network connection, Wasm Plugin extracts the necessary connection information and sends it to the Agent for processing. According to the control instructions received, it can accurately control the traffic, including allowing or directly discarding data packets.

[0090] The entire Spark job traffic control system implements a complete control link from traffic hijacking to policy execution through a layered design.

[0091] More specifically, the adapter component is used to collect and manage the Pod information of the new business container group, build a business container group topology relationship, and maintain an IP whitelist system that allows intercommunication; and when the data processing stage enters the calculation stage, based on the business container group relationship, determine the Pod information in the business container group, and generate a flow control instruction when all Pods in the business group belong to the same business container group and exist in the whitelist system; by verifying the target traffic connection information, determine the processing stage corresponding to the current stage, and in response to the current stage being in the execution stage, prohibit traffic from flowing out of the business container group.

[0092] That is, as the core component of the traffic controller, the Adapter assumes the important responsibilities of business container group management and traffic decision-making. It continuously monitors and collects Pod information in the cluster, builds a complete business container group topology relationship, and maintains an IP whitelist system that allows intercommunication. It ensures the accuracy and security of traffic control through a deep understanding of the container group topology relationship.

[0093] During the data processing phase, the Adapter ensures that only Pods in the same business container group and IPs in the whitelist can communicate through precise phase identification and strict release rules. This design ensures the security of data processing while maintaining the necessary business connectivity.

[0094] The wehook is used to inject the agent container and the wasmplug-in container into the new business container group when the new business container group is created.

[0095] The Spark job flow control method provided by the embodiment of the present application has the following beneficial effects:

[0096] On the one hand, it is security and reliability. The security of the system is achieved through a multi-layer protection mechanism. First, container injection ensures the integrity of the control component. Second, the precise management of the Adapter ensures the accuracy of the container group relationship and the whitelist. At the same time, the system can promptly identify and block illegal traffic, especially at the critical stage of data processing. In terms of reliability, the system implements mechanisms such as state persistence, fault recovery, performance optimization, and real-time monitoring to ensure the stability and reliability of traffic control.

[0097] Another aspect is scalability. The embodiments of the present application have good scalability. Through the automatic injection mechanism of webhook, the flow control capability can be easily extended to the newly created business container group. The container group management and whitelist system of Adapter supports dynamic updates and can adapt to changes in business scale. At the same time, the system reserves interfaces for integration with other security systems to support future expansion of functions and evolution of policies. This design ensures that the system can continue to develop as business needs change.

[0098] Corresponding to the above method embodiment, this specification also provides a Spark job flow control device embodiment, Figure 5 A schematic diagram of the structure of a Spark job flow control device provided in an embodiment of the present application. Figure 5 As shown, the device comprises:

[0099] The injection module 502 is configured to inject the agent container and the wasm plug-in container into the new business container group when a new business container group is created, and collect and manage the Pod information of the new business container group through the adapter to build a business container group topology relationship, wherein the traffic controller includes the wehook and the adapter;

[0100] The extraction module 504 is configured to hijack all network traffic of the business container through the wasmplug-in container during operation, extract the original traffic connection information carried in the network traffic, and send it to the agent container, wherein the original traffic connection information carries the source address and the target address; the data format of the target traffic connection information is unified and contains sufficient context information;

[0101] The reconstruction module 506 is configured to reconstruct the original flow connection information through the agent container to obtain the target flow connection information, and send the target flow connection information to the adapter for decision making;

[0102] The generation module 508 is configured as an adapter to generate a traffic control strategy based on the target traffic connection information, process the target traffic connection information in combination with the current processing stage, generate a traffic processing instruction, and pass the traffic processing instruction to the wasm plug-in container through the agent, and execute the traffic processing decision through the wasm plug-in container to determine whether to allow the traffic to pass, wherein the traffic control strategy carries the business container group relationship and the whitelist system.

[0103] In an optional embodiment, the generating module 508 is further configured to:

[0104] When the data processing phase enters the calculation phase, based on the business container group relationship, the Pod information in the business container group is determined, and when all the Pods in the business group belong to the same business container group and exist in the whitelist system, a traffic control strategy is generated, wherein the traffic control strategy includes traffic release and traffic rejection;

[0105] The target traffic connection information is verified to determine a processing phase corresponding to the current phase, and in response to the current phase being in the execution phase, traffic is prohibited from flowing out of the service container group.

[0106] In an optional embodiment, the generating module 508 is further configured to:

[0107] When a traffic connection request is received, the relationship between the source address and the target address corresponding to the Pod in the business group is verified. In response to the source address and the target address being consistent, it is determined that the Pods in the business group all belong to the same business container group.

[0108] In an optional embodiment, the generating module 508 is further configured to:

[0109] The traffic processing instruction is sent to the agent container, the traffic processing strategy is converted into a traffic processing decision, and is passed to the wasm plug-in container, and the traffic processing decision is executed by the wasm plug-in container.

[0110] In an optional embodiment, the extraction module 504 is further configured to hijack all network traffic of the business container through an underlying network hook mechanism, wherein the network traffic includes inbound traffic and outbound traffic.

[0111] In an optional embodiment, before the injection module 502, it is further configured as follows:

[0112] The submission module is configured to introduce a Spark JOB framework, perform a workflow definition on a Web end of the Spark JOB framework, schedule the workflow definition through task scheduling, and submit it to the Spark end for operation, wherein the Spark JOB framework includes a Web end and a Spark end; the workflow definition includes a program type, a Spark version, a business package reflection class of a main function, a main program package, a deployment method, a task name, a Drive core number, a Drive memory number, an Executor number, an Executor memory number, a main program parameter, an option parameter, and resources;

[0113] The execution module is configured to execute the corresponding Spark task through the Spark computing engine.

[0114] The Spark job flow control device provided by the embodiment of the present application has the following beneficial effects:

[0115] On the one hand, it is security and reliability. The security of the system is achieved through a multi-layer protection mechanism. First, container injection ensures the integrity of the control component. Second, the precise management of the Adapter ensures the accuracy of the container group relationship and the whitelist. At the same time, the system can promptly identify and block illegal traffic, especially at the critical stage of data processing. In terms of reliability, the system implements mechanisms such as state persistence, fault recovery, performance optimization, and real-time monitoring to ensure the stability and reliability of traffic control.

[0116] Another aspect is scalability. The embodiments of the present application have good scalability. Through the automatic injection mechanism of webhook, the flow control capability can be easily extended to the newly created business container group. The container group management and whitelist system of Adapter supports dynamic updates and can adapt to changes in business scale. At the same time, the system reserves interfaces for integration with other security systems to support future expansion of functions and evolution of policies. This design ensures that the system can continue to develop as business needs change.

[0117] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the Spark job flow control device, since it is basically similar to the Spark job flow control method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the Spark job flow control method embodiment.

[0118] Figure 6This is a block diagram of a computing device provided in an embodiment of the present application. The components of the computing device 600 include but are not limited to a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and the database 650 is used to store data.

[0119] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, and a near field communication (NFC).

[0120] In one embodiment of the present specification, the above components of the computing device 600 and Figure 6 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 6 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0121] The computing device 600 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 600 may also be a mobile or stationary server.

[0122] The processor 620 is used to execute the following computer executable instructions, which, when executed by the processor, implement the steps of the above-mentioned Spark job flow control method.

[0123] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the Spark job flow control method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the Spark job flow control method embodiment.

[0124] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned Spark job flow control method.

[0125] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the Spark job flow control method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the Spark job flow control method embodiment.

[0126] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned Spark job flow control method.

[0127] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer program embodiment, since it is basically similar to the Spark job flow control method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the Spark job flow control method embodiment.

[0128] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0129] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0130] It should be noted that the above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present specification.

[0131] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0132] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A Spark job flow control method, characterized in that: include: When a new service container group is created, webhook will inject agent container and wasmplug-in container into the new service container group, and collect and manage Pod information of the new service container group through adapter to build the service container group topology relationship, wherein the traffic controller includes wehook and adapter; During operation, all network traffic of the business container is hijacked through the wasm plug-in container, the original traffic connection information carried in the network traffic is extracted, and sent to the agent container, wherein the original traffic connection information carries the source address and the target address; the data format of the target traffic connection information is unified and contains sufficient context information; Reconstruct the original traffic connection information through the agent container to obtain the target traffic connection information, and send the target traffic connection information to the adapter for decision-making; The adapter generates a traffic control strategy based on the target traffic connection information, processes the target traffic connection information in combination with the current processing stage, generates a traffic processing instruction, and passes the traffic processing instruction to the wasmplug-in container through the agent, and executes the traffic processing decision through the wasmplug-in container to determine whether to allow the traffic to pass, wherein the traffic control strategy carries the business container group relationship and the whitelist system.

2. The method according to claim 1, characterized in that The adapter generates a flow control strategy based on the target flow connection information, processes the target flow connection information in combination with the current processing stage, and generates a flow processing instruction, including: When the data processing phase enters the calculation phase, based on the business container group relationship, the Pod information in the business container group is determined, and when all the Pods in the business group belong to the same business container group and exist in the whitelist system, a traffic control strategy is generated, wherein the traffic control strategy includes traffic release and traffic rejection; The target traffic connection information is verified to determine a processing stage corresponding to the current stage, and in response to the current stage being in the execution stage, traffic is prohibited from flowing out of the service container group.

3. The method according to claim 2, characterized in that Methods for determining that all Pods in a business group belong to the same business container group include: When a traffic connection request is received, the relationship between the source address and the target address corresponding to the Pod in the business group is verified. In response to the source address and the target address being consistent, it is determined that the Pods in the business group all belong to the same business container group.

4. The method according to claim 1, characterized in that: The method of passing the traffic processing instruction to the wasmplug-in container through the agent, and executing the traffic processing decision through the wasm plug-in container, includes: The traffic processing instruction is sent to the agent container, the traffic processing strategy is converted into a traffic processing decision, and is passed to the wasm plug-in container, and the traffic processing decision is executed by the wasm plug-in container.

5. The method according to claim 1, characterized in that The wasmplug-in container hijacks all network traffic of the business container, including: All network traffic of the business container is hijacked through the underlying network hook mechanism, where the network traffic includes inbound traffic and outbound traffic.

6. The method according to claim 1, characterized in that Before the new service container group is created, the following steps are included: Introduce the Spark JOB framework, define the workflow on the Web side of the Spark JOB framework, schedule the workflow definition through task scheduling, and submit it to the Spark side for operation, wherein the Spark JOB framework includes the Web side and the Spark side; the workflow definition includes the program type, Spark version, business package reflection class of the main function, main program package, deployment mode, task name, number of Drive cores, number of Drive memories, number of Executors, number of Executor memories, main program parameters, option parameters and resources; Execute the corresponding Spark tasks through the Spark computing engine.

7. A Spark operation flow control system, characterized in that: The Spark job flow control system includes a flow controller, an agent container and a wasm plug-in container, wherein: The flow controller includes wehook and adapter, which are used to inject agent container and wasmplug-in container into the new service container group, generate flow control strategy and manage service container group; The agent container is used to receive the original traffic connection information sent by wasmplug-in, reconstruct the original traffic connection information to obtain the target traffic connection information, and send it to the adapter for decision-making; and receive and execute the traffic processing instructions from the adapter, and convert the traffic processing instructions into specific traffic processing decisions; The wasm plug-in container is used to hijack all inbound and outbound traffic of the business container through the underlying network hook mechanism, extract the original traffic connection information carried in the network traffic, and send it to the agent container for processing; and execute traffic processing decisions to determine whether to allow traffic to pass.

8. The system according to claim 7, characterized in that Said The adapter component is used to collect and manage the Pod information of the new business container group, build the topological relationship of the business container group, and maintain the IP whitelist system that allows intercommunication; and when the data processing stage enters the calculation stage, based on the business container group relationship, determine the Pod information in the business container group, and generate a flow control instruction when all the Pods in the business group belong to the same business container group and exist in the whitelist system; by verifying the target flow connection information, determine the processing stage corresponding to the current stage, and in response to the current stage being in the execution stage, prohibit the flow from leaving the business container group; The wehook is used to inject the agent container and the wasmplug-in container into the new business container group when the new business container group is created.

9. A Spark job flow control device, characterized in that: include: The injection module is configured to inject the agent container and the wasm plug-in container into the new business container group when a new business container group is created, and collect and manage the Pod information of the new business container group through the adapter to build the topological relationship of the business container group, wherein the traffic controller includes the wehook and the adapter; The extraction module is configured to hijack all network traffic of the business container through the wasmplug-in container during operation, extract the original traffic connection information carried in the network traffic, and send it to the agent container, wherein the original traffic connection information carries the source address and the target address; the data format of the target traffic connection information is unified and contains sufficient context information; A reconstruction module is configured to reconstruct the original flow connection information through an agent container to obtain target flow connection information, and send the target flow connection information to an adapter for decision making; The generation module is configured as an adapter, which generates a flow control strategy based on the target flow connection information, processes the target flow connection information in combination with the current processing stage, generates a flow processing instruction, and passes the flow processing instruction to the wasm plug-in container through the agent, and executes the flow processing decision through the wasm plug-in container to determine whether to allow the flow to pass, wherein the flow control strategy carries the business container group relationship and the whitelist system.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores an implementation program for information transmission, and when the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images