A distributed computing framework and distributed computing processing method

Abstract

The present application discloses a distributed computing framework and a distributed computing processing method, which includes: a distributed computing engine Spark driver end, multiple Spark slave nodes and a Spring Boot module; the Spring Boot module includes a first object and at least one second object managed by a container in the Spring Boot framework, each second object corresponds to a distributed computing business logic, and the distributed computing business logics corresponding to each second object are different from each other, the first object is obtained by injecting the SparkSession in the Spark driver end into the container; the second object is used to obtain distributed computing information based on its corresponding distributed computing business logic, call the first object, so that the first object uses the driving function of the Spark driver end to send the distributed computing information to each Spark slave node.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a distributed computing framework and a distributed computing processing method. Background Art

[0002] At present, when the scale of enterprise data is huge and the business logic is extremely complex, in order to ensure the timeliness of data calculation, large-scale data processing and reasonable design of complex business logic are generally based on a distributed computing framework.

[0003] Among them, currently, in order to achieve enterprise-level large-scale data processing and complex business logic, developers are generally required to use native Java programming methods to write complex business codes based on the Java language to develop a distributed computing framework. However, the above method has the problems of high development difficulty and low development efficiency. Summary of the Invention

[0004] To solve the above problems, this application provides the following technical solutions:

[0005] On the one hand, the present application provides a distributed computing framework, including: a distributed computing engine Spark driver, multiple Spark slave nodes and a Spring Boot module;

[0006] The Spring Boot module includes a first object and at least one second object managed by a container in the Spring Boot framework, each second object corresponds to a distributed computing business logic, and the distributed computing business logics corresponding to the second objects are different from each other. The first object is obtained by injecting the SparkSession in the Spark driver into the container;

[0007] The second object is used to obtain distributed computing information based on its corresponding distributed computing business logic, and call the first object so that the first object uses the driving function of the Spark driver to send the distributed computing information to each of the Spark slave nodes.

[0008] Optionally, the process of the second object obtaining the distributed computing information based on its corresponding distributed computing business logic specifically includes:

[0009] The second object obtains the data to be broadcast based on its corresponding distributed computing business logic and JPA framework;

[0010] The second object generates a task based on its corresponding distributed computing business logic.

[0011] Optionally, the second object calls the first object so that the first object uses the driving function of the Spark driver to send the distributed computing information to each of the Spark slave nodes, specifically including:

[0012] If the second object obtains the first information sent by other second objects having a dependency relationship with it from the responsibility chain, the first object is called, so that the first object uses the driving function of the Spark driver to send the task to each of the Spark slave nodes. The first information indicates that the tasks of the other second objects have been completed. The responsibility chain is obtained by connecting multiple second objects in series.

[0013] Optionally, the Spark slave node is used to execute the task, obtain an execution result, and persistently store the execution result in the memory or hardware storage device of the server where the Spark slave node is located.

[0014] Optionally, at least a portion of the distributed computing business logic corresponding to each of the second objects is obtained through Aspect Oriented Programming (AOP) processing.

[0015] Another aspect of the present application provides a distributed computing processing method based on a distributed computing framework, wherein the distributed computing framework includes: a distributed computing engine Spark driver, multiple Spark slave nodes, and a Spring Boot module, wherein the Spring Boot module includes a first object and at least one second object managed by a container in the Spring Boot framework, wherein each second object corresponds to a distributed computing business logic, and the distributed computing business logics corresponding to each second object are different from each other, and the first object is obtained by injecting a SparkSession in the Spark driver into the container;

[0016] The second object obtains distributed computing information based on its corresponding distributed computing business logic;

[0017] The second object calls the first object, so that the first object uses the driving function of the Spark driver to send the distributed computing information to each of the Spark slave nodes.

[0018] Optionally, the second object obtains distributed computing information based on its corresponding distributed computing business logic, including:

[0019] The second object obtains the data to be broadcast based on its corresponding distributed computing business logic and JPA framework;

[0020] The second object generates a task based on its corresponding distributed computing business logic.

[0021] Optionally, the second object calls the first object so that the first object uses the driving function of the Spark driver to send the distributed computing information to each of the Spark slave nodes, including:

[0022] If the second object obtains the first information sent by other second objects having a dependency relationship with it from the responsibility chain, the first object is called, so that the first object uses the driving function of the Spark driver to send the task to each of the Spark slave nodes. The first information indicates that the tasks of the other second objects have been completed. The responsibility chain is obtained by connecting multiple second objects in series.

[0023] Optionally, the method further includes:

[0024] The Spark slave node executes the task, obtains an execution result, and persistently stores the execution result in the memory or hardware storage device of the server where the Spark slave node is located.

[0025] Optionally, at least a portion of the distributed computing business logic corresponding to each of the second objects is obtained through Aspect Oriented Programming (AOP) processing.

[0026] In this application, developers can develop distributed computing business logic based on the Spring Boot framework, obtain at least one second object managed by the container in the Spring Boot framework, and each second object corresponds to a distributed computing business logic. The developer can also inject the SparkSession in the Spark driver into the container to obtain the first object, so that the SparkSession in the Spark driver can be managed by the container in the Spring Boot framework, ensuring that the second object can use the driving function of the Spark driver by calling the first object to develop a distributed computing framework. Since the Spring Boot framework can simplify the developer's development process of distributed computing business logic, it can also simplify the development process of the distributed computing framework and improve the development efficiency of the distributed computing framework. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0028] Figure 1 A schematic diagram of the structure of a distributed computing framework provided for this application;

[0029] Figure 2 A schematic diagram of the structure of another distributed computing framework provided for this application;

[0030] Figure 3 A schematic diagram of an implementation scenario of a distributed computing framework provided in this application;

[0031] Figure 4 A schematic diagram of the structure of another distributed computing framework provided in this application;

[0032] Figure 5 A flowchart of a distributed computing processing method provided in this application;

[0033] Figure 6 A flowchart of another distributed computing processing method provided in this application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0035] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0036] Reference Figure 1 , which is a structural diagram of a distributed computing framework provided in Example 1 of the present application, such as Figure 1 As shown, the distributed computing framework may include but is not limited to:

[0037] The Spark (distributed computing engine) driver, multiple Spark slave nodes, and a Spring Boot module. The multiple Spark slave nodes are represented as Spark slave node 1, Spark slave node 2, ..., Spark slave node n.

[0038] A distributed computing engine can be understood as a fast and general computing engine designed specifically for large-scale data processing.

[0039] The Spark driver can at least be used to distribute tasks.

[0040] Spark slave nodes can be used to receive tasks sent by the Spark driver and execute them.

[0041] The Spring Boot module includes a first object managed by the Spring Boot framework container and at least one second object. Each second object corresponds to a distributed computing business logic, and the distributed computing business logic corresponding to each second object is different. The first object is obtained by injecting the SparkSession from the Spark driver into the container. The at least one second object is represented as Bean 1, Bean 2, ..., Bean m.

[0042] The Spring Boot framework can be understood as an application framework on the Java platform that can solve the complexity in enterprise-level programming development.

[0043] In this embodiment, the SparkSession in the Spark driver can be injected into the container based on the IoC (Inversion of Control) technology in the Spring Boot framework. Among them, IoC can reduce the coupling relationship between objects in Java programming.

[0044] SparkSession in the Spark driver provides users with a unified entry point to use various functions of the Spark driver.

[0045] The second object is used to obtain distributed computing information based on its corresponding distributed computing business logic, and call the first object so that the first object uses the driving function of the Spark driver to send the distributed computing information to each of the Spark slave nodes.

[0046] The first object and the second object can be understood as objects that constitute the backbone of the application and are managed by the container in the Spring Boot framework. The first object and the second object are objects that are instantiated, assembled, and managed by the container in the Spring Boot framework.

[0047] like Figure 2 As shown, the process of the second object obtaining the distributed computing information based on its corresponding distributed computing business logic may specifically include:

[0048] The second object obtains the data to be broadcast based on its corresponding distributed computing business logic and JPA (Java Persistence API) framework;

[0049] The second object generates a task based on its corresponding distributed computing business logic.

[0050] Correspondingly, the second object calls the first object so that the first object uses the driving function of the Spark driver to send the distributed computing information to each of the Spark slave nodes. The process may specifically include:

[0051] The second object calls the first object, causing the first object to use the broadcast function of the Spark driver to broadcast the data to each Spark slave node and use the task distribution function of the Spark driver to send the task to each Spark slave node. Each Spark slave node can execute a task based on at least the data to be broadcast. The Spark driver can collect and process the task execution results of each Spark slave node.

[0052] The JPA framework provides an enterprise-level persistence layer specification, and the Spring Boot framework's persistence layer is backed by the JPA framework. This framework facilitates the Spring Boot framework's persistence layer's database operations, reading and writing data. The Spring Boot framework's persistence layer is backed by the JPA framework, making code development simple and easy for developers.

[0053] In this embodiment, the container in the Spring Boot framework can manage each second object based on the chain of responsibility pattern, so that each second object is connected in series to form a chain of responsibility. The chain of responsibility pattern is an object behavior pattern. In this pattern, many objects are connected to form a chain by each object's reference to its next object. Requests are passed along this chain until an object in the chain decides to handle the request. The client issuing the request does not know which object in the chain will ultimately handle the request. This allows the system to dynamically reorganize and redistribute responsibilities without affecting the client.

[0054] In the Spring Boot framework, a container can manage each second object based on the chain of responsibility model. The second object calls the first object so that the first object uses the driving function of the Spark driver to send the distributed computing information to each Spark slave node. The process may specifically include:

[0055] If the second object obtains the first information sent by another second object having a dependency relationship with it in the responsibility chain, the first object is called, so that the first object uses the driving function of the Spark driver to send the task to each of the Spark slave nodes.

[0056] The first information indicates that the tasks of other second objects have been completed, and the responsibility chain is obtained by connecting multiple second objects in series.

[0057] For example, Figure 3 As shown, Bean 1 can generate task A and send task A to each Spark slave node. Each Spark slave node executes task A and obtains execution result A. The first object can use the recycling function of the Spark driver to send the first information returned by each Spark slave node to each second object on the responsibility chain. Since task B generated by Bean 2 needs to depend on result A, Bean 2 calls the first object when receiving the first information and sends task B to each Spark slave node.

[0058] If the corresponding second object obtains the first information sent by other second objects with which it has a dependency relationship from the responsibility chain, the first object is called, so that the first object uses the driving function of the Spark driver to send the task to each of the Spark slave nodes. In order to realize that the task of the next second object in the responsibility chain can be executed by each of the Spark slave nodes, it is necessary to ensure that the execution result obtained by each Spark slave node each time the task is executed is not lost. Therefore, after receiving the task, each Spark slave node can be used to execute the task, obtain the execution result, and persist the execution result to the memory or hardware storage device of the server where the Spark slave node is located.

[0059] like Figure 4 As shown, in this embodiment, at least a portion of the distributed computing business logic corresponding to each second object is processed based on AOP (Aspect Oriented Programming). Using AOP can isolate various parts of the distributed computing business logic, thereby reducing the coupling between them, improving program reusability, and increasing development efficiency.

[0060] The distributed computing processing method provided in this application is described below. The distributed computing processing method described below and the distributed computing framework described above can be referenced to each other.

[0061] The distributed computing processing method provided in this embodiment is based on the distributed computing framework introduced in the above embodiment, such as Figure 5 As shown, the method may include but is not limited to the following steps:

[0062] Step S101: The second object obtains distributed computing information based on its corresponding distributed computing business logic.

[0063] Step S102: The second object calls the first object, so that the first object uses the driving function of the Spark driver to send the distributed computing information to each of the Spark slave nodes.

[0064] In another embodiment of the present application, step S101 is introduced as follows: Figure 6 As shown, step S101 may include:

[0065] Step S1011: The second object obtains the data to be broadcast based on its corresponding distributed computing business logic and JPA framework.

[0066] Step S1012: The second object generates a task based on its corresponding distributed computing business logic.

[0067] Corresponding to steps S1011-S1012, step S102 may include:

[0068] S1021. The second object calls the first object, so that the first object uses the broadcast function of the Spark driver to broadcast the data to be broadcast to each of the Spark slave nodes, and uses the task distribution function of the Spark driver to send the task to each of the Spark slave nodes.

[0069] In this embodiment, step S102 may include:

[0070] S1021. If the second object obtains the first information sent by other second objects having a dependency relationship with it from the responsibility chain, the first object is called, so that the first object uses the driving function of the Spark driver to send the task to each of the Spark slave nodes. The first information indicates that the tasks of the other second objects have been completed. The responsibility chain is obtained by connecting multiple second objects in series.

[0071] In this embodiment, the above method may further include:

[0072] The Spark slave node executes the task, obtains an execution result, and persistently stores the execution result in the memory or hardware storage device of the server where the Spark slave node is located.

[0073] In this embodiment, at least a portion of the distributed computing business logic corresponding to each of the second objects is obtained through Aspect Oriented Programming (AOP) processing.

[0074] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0075] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A distributed computing framework, characterized in that: include: Distributed computing engine Spark driver, multiple Spark slave nodes, and Spring Boot modules; The Spring Boot module includes a first object and at least one second object managed by a container in the Spring Boot framework, each second object corresponds to a distributed computing business logic, and the distributed computing business logics corresponding to the second objects are different from each other. The first object is obtained by injecting the SparkSession in the Spark driver into the container; The second object is configured to obtain distributed computing information based on its corresponding distributed computing business logic, and call the first object so that the first object uses the driving function of the Spark driver to send the distributed computing information to each of the Spark slave nodes; The second objects include second object 1 and second object 2. Second object 1 and second object 2 are connected in series in a responsibility chain. Second object 2 has a dependency relationship with second object 1. Second object 1 generates task A. Task A is executed by each Spark slave node to obtain execution result A. Second object 2 generates task B. The execution of task B depends on execution result A. The responsibility chain is formed by connecting each of the multiple second objects with references to its subordinate objects. The process of the second object calling the first object so that the first object uses the driving function of the Spark driver to send the distributed computing information to each of the Spark slave nodes specifically includes: The second object 1 sends the task A to each Spark slave node; Each Spark slave node executes the task A to obtain an execution result A, and returns first information, where the first information indicates that the second object 1 has completed the execution of task A. The first object uses the recycling function of the Spark driver to send the first information to other second objects in the responsibility chain; In response to receiving the first information returned by the first object, the second object 2 calling the first object; The first object, under the call of the second object 2, uses the driving function of the Spark driver to send the task B to each Spark slave node; Each Spark slave node executes the task B.

2. The distributed computing framework according to claim 1, characterized in that The process of the second object obtaining the distributed computing information based on its corresponding distributed computing business logic specifically includes: The second object obtains the data to be broadcast based on its corresponding distributed computing business logic and JPA framework; The second object generates a task based on its corresponding distributed computing business logic.

3. The distributed computing framework according to claim 1, wherein: The Spark slave node is used to execute the task, obtain the execution result, and persistently store the execution result in the memory or hardware storage device of the server where the Spark slave node is located.

4. The distributed computing framework according to claim 1, wherein: At least a portion of the distributed computing business logic corresponding to each of the second objects is obtained through Aspect Oriented Programming (AOP) processing.

5. A distributed computing processing method, characterized in that: Based on a distributed computing framework, the distributed computing framework includes: a distributed computing engine Spark driver, multiple Spark slave nodes, and a Spring Boot module. The Spring Boot module includes a first object and at least one second object managed by a container in the Spring Boot framework. Each second object corresponds to a distributed computing business logic, and the distributed computing business logics corresponding to each second object are different from each other. The first object is obtained by injecting the SparkSession in the Spark driver into the container. The second object obtains distributed computing information based on its corresponding distributed computing business logic; The second object calls the first object, so that the first object uses the driving function of the Spark driver to send the distributed computing information to each of the Spark slave nodes; The second object includes a second object 1 and a second object 2, the second object 2 has a dependency relationship with the second object 1, the second object 1 generates a task A, the task A is executed by each Spark slave node to obtain an execution result A, the second object 2 generates a task B, and the execution of the task B depends on the execution result A; the second object calls the first object so that the first object uses the driving function of the Spark driver end to send the distributed computing information to each Spark slave node, specifically including: The second object 1 sends the task A to each Spark slave node; Each Spark slave node executes the task A to obtain an execution result A, and returns first information, where the first information indicates that the second object 1 has completed the execution of task A. The first object uses the recycling function of the Spark driver to send the first information to other second objects in the responsibility chain; In response to receiving the first information returned by the first object, the second object 2 calling the first object; The first object, under the call of the second object 2, uses the driving function of the Spark driver to send the task B to each Spark slave node; Each Spark slave node executes the task B.

6. The method according to claim 5, characterized in that The second object obtains distributed computing information based on its corresponding distributed computing business logic, including: The second object obtains the data to be broadcast based on its corresponding distributed computing business logic and JPA framework; The second object generates a task based on its corresponding distributed computing business logic.

7. The method according to claim 5, characterized in that The method further comprises: The Spark slave node executes the task, obtains an execution result, and persistently stores the execution result in the memory or hardware storage device of the server where the Spark slave node is located.

8. The method according to claim 5, characterized in that At least a portion of the distributed computing business logic corresponding to each of the second objects is obtained through Aspect Oriented Programming (AOP) processing.

Images