A spark job traffic control method, system and device
By injecting agent and wasm plug-in containers into Spark JOB tasks, and utilizing adapters to build container group topology and traffic control strategies, the problem of traffic management and monitoring of Spark JOB tasks is solved, improving the efficiency of anomaly investigation and data security, and supporting system expansion.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHONGDIAN DATA IND CO LTD
- Filing Date
- 2024-12-11
- Publication Date
- 2026-04-14
Smart Images
Figure CN119906637B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of traffic processing technology, and in particular to a method, system and apparatus for controlling traffic in Spark jobs. Background Technology
[0002] Currently, there are many traffic control solutions available, such as Istio and Kubernetes Network Policy. Istio and Kubernetes Network Policy are two different network control tools, each providing different traffic control capabilities. Their main difference lies in that Istio is a service mesh solution, while Kubernetes Network Policy is a network policy within a Kubernetes cluster. Current mainstream traffic control solutions primarily address traffic between services, but they cannot provide solutions for traffic management during the execution of Spark JOB-like tasks, nor can they address fine-grained traffic monitoring and control at different stages of task execution.
[0003] There are two common ways to execute Spark tasks: Spark-submit and the Java API. For production deployment and release, Spark-submit scripts are typically used for submission. Develop Spark business JAR packages tailored to different business needs, and submit them to the Spark cluster environment for execution via Spark-submit scripts. Developing a business JAR package involves the following steps: initializing the Spark basic environment; reading data source operations; business logic transformation; writing data results; and closing the Spark session.
[0004] However, this operational model increases complexity and labor costs when handling exceptional situations. Exceptions mainly fall into two categories: business logic issues and Spark infrastructure issues. When the exception is a business logic issue, it's necessary to examine both the business logic code and the Spark infrastructure code, which increases the difficulty and time required for troubleshooting. If there are many business packages, this operation will significantly increase the workload, thereby increasing enterprise costs.
[0005] In view of this, how to provide a Spark job flow control method to solve the problem of separating business code from Spark execution base code and reduce workload and labor costs has become an urgent technical problem to be solved. Summary of the Invention
[0006] This application provides a Spark job flow control method, a Spark job flow control system, a Spark job flow control device, an electronic setup, and a computer storage medium to solve the problem of separating business code from Spark execution base code.
[0007] In a first aspect of this application, a Spark job flow control method is provided, comprising:
[0008] When a new business container group is created, the webhook will inject the agent container and the wasm plug-in container into the new business container group, and collect and manage the Pod information of the new business container group through the adapter to build the business container group topology. The traffic controller includes the webhook and the adapter.
[0009] During operation, the wasm plug-in container intercepts all network traffic of the service container, extracts the raw traffic connection information carried in the network traffic, and sends it to the agent container. The raw traffic connection information carries the source address and the destination address. The data format of the destination traffic connection information is uniform and contains sufficient context information.
[0010] The original traffic connection information is reconstructed using the agent container to obtain the target traffic connection information, and the target traffic connection information is sent to the adapter for decision-making.
[0011] Based on the target traffic connection information, the adapter generates a traffic control policy, processes the target traffic connection information in conjunction with the current processing stage, generates a traffic processing instruction, and transmits the traffic processing instruction to the wasm plug-in container through the agent. The wasm plug-in container executes the traffic processing decision to determine whether to allow the traffic to pass. The traffic control policy carries the business container group relationship and whitelist system.
[0012] In a second aspect of this application, a Spark job traffic control system is provided, including a traffic controller, an agent container, and a wasm plug-in container, wherein...
[0013] The traffic controller, including wehook and adapter, is used to inject agent containers and wasm plug-in containers into the new service container group, generate traffic control policies, and manage the service container group.
[0014] The agent container is used to receive raw traffic connection information sent by the wasm plug-in, reconstruct the raw traffic connection information to obtain target traffic connection information, and send it to the adapter for decision-making; and to receive and execute traffic processing instructions from the adapter, and convert the traffic processing instructions into specific traffic processing decisions.
[0015] The wasm plug-in container is used to intercept all inbound and outbound traffic of the service container through the underlying network hook mechanism, extract the original traffic connection information carried in the network traffic and send it to the agent container for processing; and to execute traffic processing decisions to determine whether to allow the traffic to pass.
[0016] In a third aspect of this application, a Spark job flow control device is provided, comprising:
[0017] The injection module is configured so that when a new business container group is created, the webhook will inject an agent container and a wasm plug-in container into the new business container group, and collect and manage the Pod information of the new business container group through the adapter to build the business container group topology. The traffic controller includes the webhook and the adapter.
[0018] The extraction module is configured to, during runtime, intercept all network traffic of the service container through the wasm plug-in container, extract the raw traffic connection information carried in the network traffic, and send it to the agent container. The raw traffic connection information carries the source address and the destination address. The data format of the destination traffic connection information is uniform and contains sufficient context information.
[0019] The reconstruction module is configured to reconstruct the original traffic connection information through the agent container to obtain the target traffic connection information, and send the target traffic connection information to the adapter for decision-making.
[0020] The generation module is configured to generate a traffic control policy based on the target traffic connection information, process the target traffic connection information in conjunction with the current processing stage, generate a traffic processing instruction, and pass the traffic processing instruction to the wasm plug-in container through the agent. The wasm plug-in container executes the traffic processing decision to determine whether to allow the traffic to pass. The traffic control policy carries the business container group relationship and whitelist system.
[0021] In a fourth aspect of this application, a computing device is provided, comprising:
[0022] Memory and processor;
[0023] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the Spark job flow control method described above.
[0024] According to a fifth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the Spark job flow control method described above.
[0025] This application provides a Spark job traffic control method, comprising: First, when a new business container group is created, a webhook injects an agent container and a wasm plug-in container into the new business container group, and collects and manages the Pod information of the new business container group through the adapter to construct the business container group topology, wherein the traffic controller includes a webhook and an adapter; then, during operation, the wasm plug-in container intercepts all network traffic of the business container, extracts the original traffic connection information carried in the network traffic, and sends it to the agent container, wherein the original traffic connection information carries a source address and a destination address; the data format of the target traffic connection information is uniform and contains sufficient context information; second, the agent container reconstructs the original traffic connection information to obtain the target traffic connection information, and sends the target traffic connection information to the adapter for decision-making; finally, the adapter generates a traffic control policy based on the target traffic connection information, processes the target traffic connection information in conjunction with the current processing stage, generates a traffic processing instruction, and passes the traffic processing instruction to the wasm plug-in container through the agent, and then uses wasm plug-in to process the traffic connection information. The plug-in container executes traffic processing decisions to determine whether traffic is allowed to pass through. The traffic control policy carries the business container group relationship and whitelist system.
[0026] The Spark job flow control method provided in this application has the following advantages:
[0027] On one hand, there's security and reliability. System security is achieved through multi-layered protection mechanisms. First, container injection ensures the integrity of the control components. Second, precise adapter management ensures the accuracy of container group relationships and whitelists. Simultaneously, the system can promptly identify and block illegal traffic, especially during critical data processing stages. Regarding reliability, the system implements mechanisms such as state persistence, fault recovery, performance optimization, and real-time monitoring to ensure the stability and reliability of traffic control.
[0028] On the other hand, there is scalability. The embodiments of this application exhibit excellent scalability. Through the automatic injection mechanism of webhooks, traffic control capabilities can be easily extended to newly created business container groups. The adapter's container group management and whitelist system support dynamic updates, adapting to changes in business scale. Simultaneously, the system reserves interfaces for integration with other security systems, supporting future functional expansion and policy evolution. This design ensures that the system can continuously evolve with changing business needs.
[0029] The above description is merely an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description
[0030] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0031] Figure 1 A schematic diagram of the Spark JOB framework for a Spark job flow control method provided in an embodiment of this application;
[0032] Figure 2 A flowchart illustrating a Spark job flow control method provided in an embodiment of this application;
[0033] Figure 3 A schematic diagram illustrating traffic hijacking in a Spark job traffic control method provided in this application embodiment;
[0034] Figure 4 This is a schematic diagram illustrating the results of a Spark job flow control system provided in an embodiment of this application.
[0035] Figure 5 This is a schematic diagram of the structure of a Spark job flow control device provided in an embodiment of this application;
[0036] Figure 6 This is a structural block diagram of a computing device provided in an embodiment of this application. Detailed Implementation
[0037] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.
[0038] Currently, there are many traffic control solutions available, such as Istio and Kubernetes Network Policy. Istio and Kubernetes Network Policy are two different network control tools, each providing different traffic control capabilities. Their main difference lies in the fact that Istio is a service mesh solution, while Kubernetes Network Policy is a network policy within a Kubernetes cluster.
[0039] The main features of Kubernetes NetworkPolicy include: NetworkPolicy is a built-in feature of Kubernetes, mainly responsible for network traffic control within the cluster, primarily targeting network communication between pods; through NetworkPolicy, you can define allowed ingress and egress traffic, for example, you can define which pods can communicate with other pods; NetworkPolicy is very effective for service-to-service (i.e., pod-to-pod) network isolation, which can enhance the security of Kubernetes clusters.
[0040] Key features of Istio include: Istio is a service mesh solution that provides more advanced and sophisticated traffic control capabilities, such as request routing, retries, fault injection, and circuit breakers; Istio can control not only network communication between pods but also communication between services, for example, by routing requests based on HTTP headers or URL paths; Istio offers comprehensive observation and measurement capabilities, such as request latency, success rate, and distributed tracing, which can help you better understand and monitor interactions between services; and Istio provides robust security features, such as automatic mTLS encryption and fine-grained access control policies.
[0041] Clearly, current mainstream traffic control solutions are mainly used to solve traffic between services, but they cannot provide solutions for traffic management during the execution of Spark JOB-type tasks, nor can they solve fine-grained traffic monitoring and control at different stages of task execution.
[0042] In view of this, in order to meet the need for fine-grained traffic monitoring and control at different stages of Spark JOB execution, this application introduces the Spark JOB framework to output the current stage of JOB execution. Only by obtaining the current stage can different traffic control strategies be implemented for different stages.
[0043] See Figure 1 , Figure 1 This is a schematic diagram of the Spark JOB framework for a Spark job flow control method provided in an embodiment of this application. Figure 1 As shown, the Spark task execution flow of this Spark JOB framework is divided into two parts: The first part involves defining the workflow on the web client, scheduling the workflow definition through task scheduling, and submitting it to the Spark client for job execution. The Spark JOB framework includes both a web client and a Spark client. The workflow definition includes the program type, Spark version, the business package reflection class of the main function, the main program package, deployment method, task name, number of Drive cores, number of Drive memory, number of Executors, number of Executor memory, main program parameters, option parameters, and resources. The second part involves executing the corresponding Spark task through the Spark computing engine. Specifically, the Spark client is responsible for parsing parameters, performing parameter parsing, data reading, business package reflection, and data output in four steps, and monitoring each step to complete the execution of the Spark task. The workflow is as follows: Figure 1 As shown.
[0044] It should be noted that the program types supported here are Java, Scala, and Python.
[0045] The Spark version here supports Spark 3.
[0046] The Class (reflection class of the business package) of the main function here is the full path to the entry point of the Spark business package program.
[0047] The main program package here is the JAR file that executes the Spark program (uploaded through the resource center).
[0048] The deployment method here uses Kubernetes Spark submit native support.
[0049] The task name (optional) here is the name of the Spark program.
[0050] The Driver core count here is used to set the number of Driver cores, which can be set according to the actual production environment.
[0051] The Driver memory limit here is used to set the amount of memory for the Driver. You can set the corresponding amount of memory according to the actual production environment.
[0052] The number of Executors here is used to set the number of Executors, and the corresponding memory amount can be set according to the actual production environment.
[0053] The Executor memory limit here is used to set the memory limit for each Executor. You can set the corresponding memory limit according to the actual production environment.
[0054] The main program parameters here are used to set the input parameters for the Spark program.
[0055] The options here support the formats --jar, --files, and --conf.
[0056] If other parameters reference resource files, you need to specify them in the resource selection.
[0057] Specifically, the method for submitting the scheduling workflow definition to Spark for job execution includes: obtaining the basic definition information of the task based on the task number run by the user; assembling parameters based on the basic definition information to generate a spark-submit command; executing the assembled spark-submit command through ProcessBuilder; and submitting the spark-submit command to the Spark computing engine to execute the corresponding Spark task.
[0058] Specifically, submitting the spark-submit command to the Spark computing engine to execute the corresponding Spark task includes:
[0059] The JSON string parameters passed in the spark-submit command are parsed to obtain a Spark parameter object. This parameter object carries connection information and table information for reading data, as well as a business package reflection class. The business package reflection class carries the location of the business package during reflection, the location and class name of the business package used when executing the task, and the output result information. The output result information includes information such as which table the output result will be output to and connection information.
[0060] Based on the connection information and table information set in the input source in the workflow definition, the data will be passed to the business package for data processing and data reading.
[0061] Based on the business package reflection class set in the workflow definition, the fixed calculation method in the business package is determined to execute the corresponding business logic, and the business logic is sent to the output result terminal. The data used in the business package comes from the data read from the connection information and table information set in the input source.
[0062] Based on the connection and table information set in the output source of the workflow definition, the results are written to the corresponding databases and tables, and the execution of the current task is terminated.
[0063] In addition, the Spark task also includes stage monitoring, wherein submitting the spark-submit command to the Spark computing engine to execute the corresponding Spark task includes: recording and monitoring the processes of parsing parameters, reading data, reflecting business packages, and outputting data.
[0064] By introducing the Spark job development framework, business users only need to focus on their specific data processing logic, reducing the size of business packages and disk overhead. For massive amounts of business packages, this can save enterprises significant storage costs. When problems arise during task execution, the source of the issue can be quickly identified—whether it's a problem with the business package or the Spark execution environment—improving the efficiency and speed of problem finding and resolution.
[0065] Introducing a traffic management component grants us deep monitoring capabilities for Spark Job traffic, significantly enhancing our control over the entire execution process. This component monitors and analyzes traffic data, capturing, intercepting, and recording abnormal traffic in real time during execution, providing immediate traffic anomaly alerts and logging. Furthermore, this measure greatly improves data security and integrity. Through real-time control of the data stream, we can ensure that SparkJobs cannot read or transfer unauthorized data during processing, thus guaranteeing data security and compliance.
[0066] See Figure 2 , Figure 2 This is a flowchart illustrating a Spark job traffic control method provided in an embodiment of this application. Figure 2 As shown, the specific steps include:
[0067] Step S202: When a new business container group is created, the webhook will inject the agent container and the wasm plug-in container into the new business container group, and collect and manage the Pod information of the new business container group through the adapter to build the business container group topology relationship. The traffic controller includes the webhook and the adapter.
[0068] Step S204: During operation, all network traffic of the service container is hijacked through the wasm plug-in container, the original traffic connection information carried in the network traffic is extracted and sent to the agent container, wherein the original traffic connection information carries the source address and the destination address; the data format of the destination traffic connection information is uniform and contains sufficient context information;
[0069] Step S206: Reconstruct the original traffic connection information through the agent container to obtain the target traffic connection information, and send the target traffic connection information to the adapter for decision-making;
[0070] Step S208: Based on the target traffic connection information, the adapter generates a traffic control policy, processes the target traffic connection information in conjunction with the current processing stage, generates a traffic processing instruction, and transmits the traffic processing instruction to the wasm plug-in container through the agent. The wasm plug-in container executes the traffic processing decision to determine whether to allow traffic to pass. The traffic control policy carries the business container group relationship and whitelist system.
[0071] It's important to note that the traffic control strategy here is based on two core dimensions: the relationship between business container groups and a whitelist system. When the data processing phase transitions to the computation phase, the system only allows communication between Pods within the same business container group and between IPs on the whitelist. This strategy is executed through the Adapter's decision engine, ensuring the security and isolation of the data processing process. Upon receiving a connection request, the decision engine verifies the relationship between the source and destination addresses and makes a decision to allow or deny the request based on the current processing stage and the whitelist status.
[0072] In this embodiment, the adapter generates a traffic control policy based on the target traffic connection information, processes the target traffic connection information in conjunction with the current processing stage, and generates traffic processing instructions, including:
[0073] When the data processing stage enters the computing stage, the Pod information within the business container group is determined based on the relationship between the business container groups. If all Pods within the business group belong to the same business container group and exist in the whitelist system, a traffic control policy is generated. The traffic control policy includes traffic allowance and traffic rejection.
[0074] By verifying the target traffic connection information, the processing stage corresponding to the current stage is determined. In response to the current stage being in the execution stage, traffic is prohibited from flowing out of the service container group.
[0075] Specifically, methods for determining that all Pods within a business group belong to the same business container group include:
[0076] Upon receiving a traffic connection request, the relationship between the source address and the destination address of the Pod within the business group is verified. If the source address and the destination address are consistent, it is determined that all Pods within the business group belong to the same business container group.
[0077] In this embodiment of the application, the step of transmitting the traffic processing instruction to the wasm plug-in container via the agent, and executing the traffic processing decision via the wasm plug-in container, includes:
[0078] The traffic processing instruction is sent to the agent container, the traffic processing policy is converted into a traffic processing decision, and then passed to the wasm plug-in container, which executes the traffic processing decision.
[0079] In this embodiment of the application, the hijacking of all network traffic of the service container through the wasm plug-in container includes: hijacking all network traffic of the service container through the underlying network hook mechanism, wherein the network traffic includes inbound traffic and outbound traffic.
[0080] In another aspect of this application's embodiments, by monitoring and analyzing traffic data, abnormal Spark job traffic can be captured, intercepted, and recorded in real time during execution, providing immediate traffic anomaly alarms and recording functions. Specifically, this includes: a decoupling scheme between the business execution package and the Spark basic environment; and monitoring of the task execution phase status.
[0081] See Figure 3 , Figure 3 This is a traffic hijacking diagram illustrating a Spark job traffic control method provided in an embodiment of this application.
[0082] like Figure 3As shown, when the business container group is created, the traffic controller webhook injects two containers: agent and wasm plug-in. The wasm plug-in container intercepts all traffic from the business container and sends the traffic connection information, including source and destination address information, to the agent. After reconstructing the traffic connection information, the agent sends it to the adapter component for connection information verification. The adapter verifies the current stage; if it is currently in the execution stage, it will prevent traffic from flowing out of the business container group. The adapter rejects illegal traffic and provides the agent with a notification to the wasm plug-in to discard the rejected request packets.
[0083] See Figure 4 , Figure 4 This is a schematic diagram of the results of a Spark job flow control system provided in an embodiment of this application.
[0084] like Figure 4 As shown, the overall architecture of this traffic hijacking and control solution consists of three core components: a traffic controller (including wehook and adapter), an agent container, and a wasm plug-in container. This architecture design allows for precise control and secure management of inter-container communication without modifying the business containers.
[0085] Specifically, the traffic controller, including wehook and adapter, is used to inject agent containers and wasm plug-in containers into the new service container group, generate traffic control policies, and manage the service container group. That is, the traffic controller acts as a central management unit, responsible for container injection, traffic policy formulation, and management of the service container group.
[0086] The agent container is used to receive raw traffic connection information sent by the wasm plug-in, reconstruct the raw traffic connection information to obtain target traffic connection information, and send it to the adapter for decision-making; and to receive and execute traffic processing instructions from the adapter, and convert the traffic processing instructions into specific traffic processing decisions.
[0087] In essence, the Agent container acts as the central hub for collecting and processing traffic information. It receives raw connection information from the Wasm Plugin, including crucial data such as source and destination addresses. It reconstructs and normalizes this information, ensuring that the data passed to the adapter is formatted consistently and contains sufficient contextual information. Simultaneously, the Agent is also responsible for receiving and executing control commands from the adapter, translating these commands into specific traffic control policies.
[0088] The wasm plug-in container is used to intercept all inbound and outbound traffic of the service container through the underlying network hook mechanism, extract the original traffic connection information carried in the network traffic and send it to the agent container for processing; and to execute traffic processing decisions to determine whether to allow the traffic to pass.
[0089] The Wasm Plugin container serves as the core component for actually executing traffic hijacking and control. Through an underlying network hook mechanism, it can intercept all inbound and outbound traffic from the service container. For each network connection, the Wasm Plugin extracts the necessary connection information and sends it to the Agent for processing. Based on the received control instructions, it can precisely control the traffic, including allowing packets to pass or dropping them directly.
[0090] The entire Spark job traffic control system is designed in a layered manner to achieve a complete control chain from traffic hijacking to policy execution.
[0091] More specifically, the adapter component is used to collect and manage Pod information of the new business container group, construct the business container group topology, maintain an IP whitelist system that allows interconnection; and when the data processing stage enters the computing stage, it determines the Pod information within the business container group based on the business container group relationship, and generates traffic control instructions when all Pods within the business group belong to the same business container group and exist in the whitelist system; by verifying the target traffic connection information, it determines the processing stage corresponding to the current stage, and in response to the current stage being in the execution stage, it prohibits traffic from flowing out of the business container group.
[0092] In short, the Adapter, as the core component of the traffic controller, bears the important responsibility of managing business container groups and making traffic decisions. It continuously monitors and collects Pod information in the cluster, builds a complete business container group topology, and maintains a whitelist system of allowed IP addresses. Through a deep understanding of the container group topology, it ensures the accuracy and security of traffic control.
[0093] During the data processing phase, the Adapter ensures that communication is only possible between Pods within the same business container group and between whitelisted IPs through precise phase identification and strict access rules. This design guarantees both data processing security and maintains necessary business connectivity.
[0094] The wehook is used to inject an agent container and a wasm plug-in container into a new business container group when the new business container group is created.
[0095] The Spark job flow control method provided in this application has the following advantages:
[0096] On one hand, there's security and reliability. System security is achieved through multi-layered protection mechanisms. First, container injection ensures the integrity of the control components. Second, precise adapter management ensures the accuracy of container group relationships and whitelists. Simultaneously, the system can promptly identify and block illegal traffic, especially during critical data processing stages. Regarding reliability, the system implements mechanisms such as state persistence, fault recovery, performance optimization, and real-time monitoring to ensure the stability and reliability of traffic control.
[0097] On the other hand, there is scalability. The embodiments of this application exhibit excellent scalability. Through the automatic injection mechanism of webhooks, traffic control capabilities can be easily extended to newly created business container groups. The adapter's container group management and whitelist system support dynamic updates, adapting to changes in business scale. Simultaneously, the system reserves interfaces for integration with other security systems, supporting future functional expansion and policy evolution. This design ensures that the system can continuously evolve with changing business needs.
[0098] Corresponding to the above method embodiments, this specification also provides an embodiment of a Spark job flow control device. Figure 5 This is a schematic diagram of a Spark job flow control device provided in an embodiment of this application. Figure 5 As shown, the device includes:
[0099] The injection module 502 is configured such that when a new business container group is created, the webhook will inject an agent container and a wasm plug-in container into the new business container group, and collect and manage the Pod information of the new business container group through the adapter to build the business container group topology. The traffic controller includes the webhook and the adapter.
[0100] Extraction module 504 is configured to, during operation, hijack all network traffic of the service container through the wasm plug-in container, extract the original traffic connection information carried in the network traffic, and send it to the agent container. The original traffic connection information carries the source address and the destination address. The data format of the destination traffic connection information is uniform and contains sufficient context information.
[0101] The reconstruction module 506 is configured to reconstruct the original traffic connection information through the agent container to obtain the target traffic connection information, and send the target traffic connection information to the adapter for decision-making;
[0102] The generation module 508 is configured to generate a traffic control policy based on the target traffic connection information, process the target traffic connection information in conjunction with the current processing stage, generate a traffic processing instruction, and pass the traffic processing instruction to the wasm plug-in container through the agent. The wasm plug-in container executes the traffic processing decision to determine whether to allow the traffic to pass. The traffic control policy carries the business container group relationship and whitelist system.
[0103] In one optional embodiment, the generation module 508 is further configured to:
[0104] When the data processing stage enters the computing stage, the Pod information within the business container group is determined based on the relationship between the business container groups. If all Pods within the business group belong to the same business container group and exist in the whitelist system, a traffic control policy is generated. The traffic control policy includes traffic allowance and traffic rejection.
[0105] By verifying the target traffic connection information, the processing stage corresponding to the current stage is determined. In response to the current stage being in the execution stage, traffic is prohibited from flowing out of the service container group.
[0106] In one alternative embodiment, the generation module 508 is further configured to:
[0107] Upon receiving a traffic connection request, the relationship between the source address and the destination address of the Pod within the business group is verified. If the source address and the destination address are consistent, it is determined that all Pods within the business group belong to the same business container group.
[0108] In one optional embodiment, the generation module 508 is further configured to:
[0109] The traffic processing instruction is sent to the agent container, the traffic processing policy is converted into a traffic processing decision, and then passed to the wasm plug-in container, which executes the traffic processing decision.
[0110] In one optional embodiment, the extraction module 504 is further configured to: hijack all network traffic of the service container through an underlying network hook mechanism, wherein the network traffic includes inbound traffic and outbound traffic.
[0111] In one optional embodiment, the injection module 502 is further configured to:
[0112] The submission module is configured to import the Spark Job framework, define the workflow on the web side of the Spark Job framework, schedule the workflow definition through task scheduling, and submit it to the Spark side for job processing. The Spark Job framework includes a web side and a Spark side. The workflow definition includes the program type, Spark version, the business package reflection class of the main function, the main program package, deployment method, task name, number of Drive cores, number of Drive memory, number of Executors, number of Executor memory, main program parameters, option parameters, and resources.
[0113] The execution module is configured to execute corresponding Spark tasks through the Spark computing engine.
[0114] The Spark job flow control device provided in this application has the following advantages:
[0115] On one hand, there's security and reliability. System security is achieved through multi-layered protection mechanisms. First, container injection ensures the integrity of the control components. Second, precise adapter management ensures the accuracy of container group relationships and whitelists. Simultaneously, the system can promptly identify and block illegal traffic, especially during critical data processing stages. Regarding reliability, the system implements mechanisms such as state persistence, fault recovery, performance optimization, and real-time monitoring to ensure the stability and reliability of traffic control.
[0116] On the other hand, there is scalability. The embodiments of this application exhibit excellent scalability. Through the automatic injection mechanism of webhooks, traffic control capabilities can be easily extended to newly created business container groups. The adapter's container group management and whitelist system support dynamic updates, adapting to changes in business scale. Simultaneously, the system reserves interfaces for integration with other security systems, supporting future functional expansion and policy evolution. This design ensures that the system can continuously evolve with changing business needs.
[0117] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the Spark job flow control device is basically similar to the Spark job flow control method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the Spark job flow control method embodiments.
[0118] Figure 6This is a structural block diagram of a computing device provided in an embodiment of this application. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.
[0119] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.
[0120] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.
[0121] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.
[0122] The processor 620 is used to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the Spark job flow control method described above.
[0123] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computing device embodiments are basically similar to the Spark job flow control method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the Spark job flow control method embodiments.
[0124] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the Spark job flow control method described above.
[0125] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the computer-readable storage medium embodiment is described simply because it is substantially similar to the Spark job flow control method embodiment; relevant parts can be referred to in the description of the Spark job flow control method embodiment.
[0126] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the Spark job flow control method described above.
[0127] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the computer program embodiments are relatively simple in description because they are fundamentally similar to the Spark job flow control method embodiments; relevant parts can be referred to in the descriptions of the Spark job flow control method embodiments.
[0128] The foregoing has described specific embodiments of this specification. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.
[0129] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.
[0130] It should be noted that the above description describes specific embodiments of this specification. Furthermore, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous. Secondly, those skilled in the art should also understand that the embodiments described in this specification are preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments described in this specification.
[0131] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0132] The preferred embodiments disclosed above are merely illustrative of this specification. Optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described in this specification. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification.
Claims
1. A Spark job flow control method, characterized in that, include: When a new business container group is created, the webhook will inject the agent container and the wasm plug-in container into the new business container group, and collect and manage the Pod information of the new business container group through the adapter to build the business container group topology. The traffic controller includes the webhook and the adapter. During operation, the wasm plug-in container hijacks all network traffic of the service container, extracts the original traffic connection information carried in the network traffic, and sends it to the agent container. The original traffic connection information carries the source address and the destination address. The original traffic connection information is reconstructed using an agent container to obtain target traffic connection information, which is then sent to the adapter for decision-making. The target traffic connection information has a unified data format and contains sufficient context information. Based on the target traffic connection information, the adapter generates a traffic control policy, processes the target traffic connection information in conjunction with the current processing stage, generates a traffic processing instruction, and transmits the traffic processing instruction to the wasm plug-in container through the agent. The wasm plug-in container executes the traffic processing decision to determine whether to allow the traffic to pass. The traffic control policy carries the business container group relationship and whitelist system.
2. The method according to claim 1, characterized in that, The adapter generates a traffic control policy based on the target traffic connection information, processes the target traffic connection information in conjunction with the current processing stage, and generates traffic processing instructions, including: When the data processing stage enters the computing stage, the Pod information within the business container group is determined based on the relationship between the business container groups. If all Pods within the business group belong to the same business container group and exist in the whitelist system, a traffic control policy is generated. The traffic control policy includes traffic allowance and traffic rejection. By verifying the target traffic connection information, the processing stage corresponding to the current stage is determined. In response to the current stage being in the execution stage, traffic is prohibited from flowing out of the service container group.
3. The method according to claim 2, characterized in that, Methods to determine that all Pods within a business group belong to the same business container group include: Upon receiving a traffic connection request, the relationship between the source address and the destination address of the Pod within the business group is verified. If the source address and the destination address are consistent, it is determined that all Pods within the business group belong to the same business container group.
4. The method according to claim 1, characterized in that, The step of passing the traffic processing instructions to the wasm plug-in container via the agent, and executing the traffic processing decision through the wasm plug-in container, includes: The traffic processing instruction is sent to the agent container, the traffic processing policy is converted into a traffic processing decision, and then passed to the wasm plug-in container, which executes the traffic processing decision.
5. The method according to claim 1, characterized in that, The hijacking of all network traffic of the service container via the wasm plug-in container includes: By using an underlying network hook mechanism, all network traffic of the service container is hijacked, including inbound and outbound traffic.
6. The method according to claim 1, characterized in that, Before the new business container group is created, the following is included: The Spark Job framework is introduced, and the workflow is defined on the web client of the Spark Job framework. The workflow definition is scheduled through task scheduling and submitted to the Spark client for job processing. The Spark Job framework includes a web client and a Spark client. The workflow definition includes program type, Spark version, reflection class of the main function's business package, main program package, deployment method, task name, number of Drive cores, number of Drive memory, number of Executors, number of Executor memory, main program parameters, option parameters, and resources. Execute the corresponding Spark tasks using the Spark computing engine.
7. A Spark job flow control system, characterized in that, The Spark job traffic control system includes a traffic controller, an agent container, and a wasm plug-in container, wherein... The traffic controller, including wehook and adapter, is used to inject agent containers and wasm plug-in containers into the new business container group, generate traffic control policies, and manage the business container group. The agent container is used to receive raw traffic connection information sent by the wasm plug-in, reconstruct the raw traffic connection information to obtain target traffic connection information, and send it to the adapter for decision-making; and to receive and execute traffic processing instructions from the adapter, and convert the traffic processing instructions into specific traffic processing decisions. The wasm plug-in container is used to intercept all inbound and outbound traffic of the service container through the underlying network hook mechanism, extract the original traffic connection information carried in the network traffic and send it to the agent container for processing; and to execute traffic processing decisions to determine whether to allow the traffic to pass. The adapter component is used to collect and manage Pod information of the new service container group, construct the topology relationship of the service container group, maintain the IP whitelist system that allows communication; and when the data processing stage enters the computing stage, it determines the Pod information within the service container group based on the service container group relationship. If all Pods within the service group belong to the same service container group and exist in the whitelist system, it generates a traffic control instruction; by verifying the target traffic connection information, it determines the processing stage corresponding to the current stage, and in response to the current stage being in the execution stage, it prohibits traffic from flowing out of the service container group. The wehook is used to inject an agent container and a wasm plug-in container into a new business container group when the new business container group is created.
8. A Spark job flow control device, characterized in that, include: The injection module is configured so that when a new business container group is created, the webhook will inject an agent container and a wasm plug-in container into the new business container group, and collect and manage the Pod information of the new business container group through the adapter to build the business container group topology. The traffic controller includes the webhook and the adapter. The extraction module is configured to, during operation, hijack all network traffic of the service container through the wasm plug-in container, extract the original traffic connection information carried in the network traffic, and send it to the agent container. The original traffic connection information carries the source address and the destination address. The reconstruction module is configured to reconstruct the original traffic connection information through the agent container to obtain the target traffic connection information, and send the target traffic connection information to the adapter for decision-making; the target traffic connection information has a unified data format and contains sufficient context information; The generation module is configured to generate a traffic control policy based on the target traffic connection information, process the target traffic connection information in conjunction with the current processing stage, generate a traffic processing instruction, and pass the traffic processing instruction to the wasm plug-in container through the agent. The wasm plug-in container executes the traffic processing decision to determine whether to allow the traffic to pass. The traffic control policy carries the business container group relationship and whitelist system.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an implementation program for information transmission, which, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 6.