Pluggable distributed data processing system and method thereof

Abstract

The invention discloses a pluggable distributed data processing system and method, and the system comprises a client which is responsible for the collection and serialization of data; after being serialized, the data are efficiently transmitted to the server side through the Arrow Flight; the server side is used for deploying an Arrow Flight Server on an edge node of the big data cluster; the method comprises the following steps: starting an Arrow Flight Server, monitoring a data processing request of a client, and carrying out data processing; the Arrow Flight Server makes full use of the distributed computing power of the Spark, and the computing power is elastically expanded according to requirements; the Arrow Flight Server realizes dynamic expansion of data processing types through a data processor registration mechanism driven by a configuration file; the newly added data processing type only needs to realize a data processor interface and update configuration. According to the system and the method, the key problems of data exchange efficiency, type security, expansibility and the like in a large-scale heterogeneous data processing scene are solved, and a set of complete technical scheme is provided for enterprise-level data processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of big data processing, and in particular to a pluggable distributed data processing system and method thereof. BACKGROUND

[0002] In the field of big data processing, data transmission between client and server usually relies on traditional REST API or RPC call. These methods produce significant overhead in data serialization and deserialization process, limiting the overall performance of the system. Especially based on HTTP protocol request, in large-scale data transmission, it may cause the request to be reset or interrupted, thereby affecting the stability and reliability of data transmission.

[0003] In addition, when the data transmission logic changes or new transmission requirements are added, the data processing logic of the server often needs to be modified. Such modification usually has a large invasion on the server code, increasing the complexity of maintenance and may adversely affect other business logic. Therefore, there is an urgent need for an efficient, flexible and easy-to-maintain data transmission solution to cope with changing business needs and data processing challenges. SUMMARY

[0004] In view of the above, the present application provides a pluggable distributed data processing system and method thereof based on Apache Arrow Flight, which realizes type safety and dynamic expansion of data processing through configuration-driven processor registration mechanism and fine-grained schema definition, and solves the key problems of data exchange efficiency, type safety and scalability in large-scale heterogeneous data processing scenarios by utilizing Apache Arrow's zero-copy transmission and Spark's distributed computing capabilities, providing a complete technical solution for enterprise-level data processing.

[0005] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0006] In an embodiment of the present application, a pluggable distributed data processing system is provided, which comprises:

[0007] The client is responsible for data collection and serialization. After serialization, the data is transmitted to the server efficiently through Arrow Flight;

[0008] The service end deploys an Arrow Flight Server on an edge node of a big data cluster; the Arrow Flight Server is started, data processing requests from the client are listened to, and data processing is performed; the Arrow Flight Server fully utilizes the distributed computing capability of Spark, and the computing capability is elastically expanded according to requirements; the Arrow Flight Server realizes dynamic expansion of data processing types through a configuration file driven data processor registration mechanism; and a new data processing type only needs to implement a data processor interface and update a configuration.

[0009] Further, the client constructs instructions according to an Arrow Flight protocol, including a data processor type, a specific processing method of a data set and data processor parameters, and converts the instructions into an Apache Arrow format, sends the instructions to the service end through a bidirectional data flow channel, and waits for and acquires a response from the service end.

[0010] Further, the service end starts to listen through the following steps:

[0011] Before starting the Arrow Flight Server, a SparkSession is initialized;

[0012] Through a processor registration center configuration file, metadata information of all data processors is loaded, so that when the client performs a data processing request, the data processing request can be routed to a correct data processor for data processing;

[0013] The Flight Server is started, data processing requests from the client are listened to, and after data processing is completed, the client's request is responded to.

[0014] Further, dynamic expansion of data processing types is realized through the following steps:

[0015] An interface specification class of a data processor is defined, and specific processing logic and a supported method set are implemented in the interface specification class;

[0016] Metadata of the data processor is declared in a processor registration center configuration file, including a data processor type, a processing class path, an input schema, an output schema of a method and specific definitions of required parameters.

[0017] In an embodiment of the application, a pluggable distributed data processing method is further provided, and the method comprises:

[0018] The client constructs instructions according to the Arrow Flight protocol, including a data processor type, a specific processing method of a data set and data processor parameters, and converts the instructions into an Apache Arrow format, and sends the instructions to the server through a bidirectional data flow channel, and waits for and obtains a response from the server;

[0019] After the server is started, a connection request from the client is listened to, once the server and the client establish a connection, instructions from the client are obtained and parsed, according to information contained in the instructions from the client, a corresponding processor instance is obtained, Apache Arrow format data sent by the client is read by the server, and the data is converted into a Spark DataFrame, a processing method of the processor is called to process the data, and after the data processing is completed, the server converts processing results back into the Apache Arrow format and sends the results back to the client.

[0020] Further, the server starts to listen through the following steps:

[0021] Before starting the Arrow Flight Server, a SparkSession is initialized;

[0022] Through a processor registry center configuration file, metadata information of all data processors is loaded, so that when the client makes a data processing request, the data processing request can be routed to a correct data processor for data processing;

[0023] The Arrow Flight Server is started, and data processing requests from the client are listened to, and after the data processing is completed, the request from the client is responded to.

[0024] Further, the Arrow Flight Server fully utilizes the distributed computing capability of Spark, and the computing capability is flexibly expanded according to requirements.

[0025] Further, the Arrow Flight Server realizes dynamic expansion of a data processing type through a data processor registration mechanism driven by a configuration file; when a data processing type needs to be added, an interface specification class of a data processor is defined, and specific processing logic and a supported method set are implemented in the class; metadata of the data processor is declared in a processor registry center configuration file, including a data processor type, a processing class path, an input schema, an output schema of a method and specific definitions of required parameters.

[0026] In an embodiment of the present application, a computer device is also provided, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the foregoing pluggable distributed data processing method when executing the computer program.

[0027] In an embodiment of the present application, a computer readable storage medium is also provided, which stores a computer program for performing the pluggable distributed data processing method.

[0028] Advantages:

[0029] 1. High-performance data exchange

[0030] The system uses the Apache Arrow Flight protocol to realize bidirectional streaming data transmission between the client and the server, significantly reduces the data serialization overhead through the zero-copy mechanism and memory sharing technology, and realizes high-throughput and low-latency data processing services in combination with the columnar storage characteristics and efficient memory layout of Arrow.

[0031] 2. Fine-grained type safety

[0032] An independent schema and parameter constraint are defined for each processing method, and automatic schema verification is performed before and after data processing. Through this fine-grained type constraint mechanism, the type safety and result predictability of the data processing process are ensured, and complete interface self-description capability is provided.

[0033] 3. Distributed computing extension

[0034] Arrow format data is seamlessly converted into a Spark DataFrame, and the distributed computing engine of Spark is fully utilized for data processing. The extension capability of distributed computing is obtained while ensuring the efficiency of data transmission, and efficient processing of large-scale data is realized.

[0035] 4. Configuration-driven extension mechanism

[0036] The data type is dynamically extended through a configuration file-driven data processor registration mechanism. New data types only need to implement the processor interface and update the configuration. The system automatically completes the loading and management of the processor, providing good extensibility and maintainability. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 is the architecture diagram of the pluggable distributed data processing system of the present application;

[0038] Figure 2 is the data processor extension implementation flowchart of an embodiment of the present application;

[0039] Figure 3 is the server start-up monitoring implementation flowchart of an embodiment of the present application;

[0040] Figure 4 is the flowchart of the pluggable distributed data processing method of the present application;

[0041] Figure 5 is a schematic diagram of a computer device structure of the present application. DETAILED DESCRIPTION

[0042] The principles and spirits of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the present application, and in no way limit the scope of the present application. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0043] Those skilled in the art know that the embodiments of the present application can be implemented as a system, a device, an apparatus, a method or a computer program product. Therefore, the present disclosure can be embodied in the form of a complete hardware, a complete software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0044] According to the embodiments of the present application, a pluggable distributed data processing system is proposed, which is based on the Apache Arrow Flight distributed data processing architecture. The type safety and dynamic expansion of data processing are realized through the configuration-driven data processor registration mechanism and fine-grained schema definition. At the same time, the zero-copy transmission of Arrow and the distributed computing capability of Spark are utilized to solve the key problems such as data exchange efficiency, type safety, scalability, etc. in large-scale heterogeneous data processing scenarios, and to provide a complete technical solution for enterprise-level data processing.

[0045] The principles and spirits of the present application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are given only to enable those skilled in the art to better understand and implement the present application, and in no way limit the scope of the present application. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0046] Figure 1 is a schematic diagram of a computer device structure of the present application. Figure 1 As shown in the figure, the system includes:

[0047] On the edge node of the big data cluster, an Arrow Flight Server is deployed to optimize the data processing process. When starting, the Arrow Flight Server will quickly initialize the SparkSession, which is the unified session entry class of Spark, so as to prepare the context environment of Spark in advance. The purpose of this is to ensure that the data processing request of the client can be processed immediately once it is submitted, eliminating the delay in the traditional starting process.

[0048] The client uses Arrow Flight, a unified and efficient data transmission protocol, to significantly reduce the overhead of data serialization. After serialization, the data is transmitted efficiently to the server through Arrow Flight, greatly improving the efficiency of network transmission.

[0049] On the server side, Arrow Flight Server takes full advantage of the distributed computing capabilities of Spark, and the computing capabilities can be flexibly expanded according to demand. This design reduces the data transfer between the server and the client, simplifies the data processing process. The client is therefore lightweight, mainly responsible for data collection and serialization. The server takes on the heavy task of computing, and through a unified data processing process, the maintenance cost is reduced.

[0050] Arrow Flight Server realizes dynamic expansion of data processing types through a configuration-driven data processor registration mechanism. When a new data processing type is needed, only an interface specification class of the data processor needs to be defined, and the specific processing logic and supported method set are implemented in it. Then, the meta information of the data processor is declared in the data processor registration center configuration file (written manually by the developer according to business requirements), including data processor type, processing class path, input schema (data structure definition, used to describe data types, fields and relationships), method output schema and specific definition of required parameters. When the system starts, the data processor registration center will automatically load all data processors according to the configuration information, realizing the unified management and dynamic invocation of data processors. This loose coupling architecture based on configuration makes it easy to access new data processing types, only focusing on the implementation of processing logic and the addition of configuration, without modifying existing code. This not only ensures the stability of the system, but also provides good scalability and maintainability.

[0051] It should be noted that the implementation of the system can be implemented by referring to the implementation of the following method, and the repeated parts will not be described. The term "module" used above can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the above embodiments is preferably implemented in software, hardware or a combination of software and hardware implementation is also possible and contemplated.

[0052] In order to more clearly explain the above-described pluggable distributed data processing system, a specific embodiment will be described below, however, it should be noted that this embodiment is only used to better illustrate the present application, and does not constitute an improper limitation on the present application.

[0053] Embodiment:

[0054] Detailed implementation

[0055] I. Data processor extension

[0056] The data processor is the core implementation of data processing logic. In order to ensure the scalability of new data processing types, an extension mechanism of the data processor is implemented, and the implementation process is as shown in Figure 2

[0057] 1. Define the interface specification class of the data processor

[0058] The interface specification of the data processor specifies the method set that needs to be followed to implement a specific data processor, including the input parameters and return types of each method, etc. The specific methods include:

[0059] (1) Get the processor type.

[0060] This method of getting the processor type has no input parameters. A meaningful name is specified in the method to uniquely identify the specific processor type, and the type is returned. The client submits a data processing request through the type.

[0061] (2) Method of processing data specific implementation.

[0062] The input parameters of this method are as follows:

[0063] ① Dataset, the format of the data is Spark distributed dataset DataFrame.

[0064] ② Method, a specific processing method based on the dataset. The specific method is determined by business requirements. Each data processor can define multiple custom data processing methods as needed. For example, methods for aggregating data, or methods for statistical or filtering data, etc.

[0065] ③ A variable number of keyword arguments. Support for passing in any number of keyword arguments to implement flexible and diverse data processing needs.

[0066] (3) Get the supported method set. List all the processing methods supported by the processor and their descriptions to facilitate unified management and display of the capabilities of the processor.

[0067] The implementation example is as follows:

[0068]

[0069]

[0070] 2. Implement a specific processor

[0071] To implement a specific processor, you need to inherit the interface specification class of the data processor and implement the three methods of the specification class.

[0072] ​As shown in the following code example.

[0073] The processor type is "UserStats".

[0074] Two methods are defined: aggregate, filter. In the method of processing data specific implementation, specific implementation is carried out for each method.

[0075]

[0076]

[0077]

[0078] 3. Configure data processor

[0079] By declaring the metadata information of the data processor in the data processor registry center configuration file, including the data processor type, processing class path, input schema, method output schema and specific definition of required parameters. When the Arrow flight server is started subsequently, the data processor registry center will automatically load all data processors and methods according to the configuration information, realize the unified management and dynamic calling of data processor. Automatic schema verification is carried out before and after data processing, to ensure the type safety and expected result of data processing. At the same time, this design also provides complete self-description ability for the system, supports interface contract management and automatic document generation, and facilitates the cooperation and connection between different teams. Through this fine-grained type constraint mechanism, the type safety and expected result of data processing process are ensured, and complete interface self-description ability is provided.

[0080] The configuration example is as follows:

[0081]

[0082]

[0083]

[0084] II. Server startup and listen

[0085] On the edge node of large data center, start Arrow Flight Server, listen to the data processing request of client, the specific implementation process is as Figure 3 shown:

[0086] (1) Initialize Spark session

[0087] Before starting the Arrow Flight Server, initialize the SparkSession, which is the unified session entry class of Spark, so as to prepare the context environment of Spark in advance. Since the initialization of SparkSession requires the application of resources such as memory and CPU, it consumes a long time. By initializing SparkSession in advance, the subsequent client eliminates the delay in the traditional starting process after submitting a data processing request, and the data can be processed immediately, improving the efficiency of data processing and response.

[0088] (2) Register data processor

[0089] Through the data processor registration center configuration file, according to the specific definition of the information including the data processor type, processing class path, input schema, method output schema and required parameters configured by the configuration file, the metadata information of all data processors is loaded, which is used for the client to route to the correct data processor for data processing when making a data processing request.

[0090] (3) Start Arrow Flight Server

[0091] Start the Arrow Flight Server to listen to the data processing request of the client, and respond to the request of the client after the data processing is completed.

[0092] The code example is as follows:

[0093]

[0094]

[0095] It should be noted that although several modules of the pluggable distributed data processing system are mentioned in the foregoing detailed description, such division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into embodied by multiple modules.

[0096] Based on the same inventive concept, the present application also proposes a pluggable distributed data processing method. As shown in Figure 4 The cooperation process of the client and the server is as follows:

[0097] The client constructs instructions according to the Arrow Flight protocol and sends them to the server. The server listens to the request of the client, receives the data of the client, processes it, and responds to the request of the client.

[0098] The process of the client is as follows:

[0099] 1. Prepare Instructions

[0100] The client first needs to construct instructions based on the Arrow Flight protocol, which is the foundation of the entire data transfer process. The construction of instructions includes the following key elements:

[0101] (1) Data Processor Type: Specifies the type of data operation that the client wants the server to perform, such as SQL query, file reading, etc.

[0102] (2) Specific Processing Method of Data Set: Defines the specific operation performed on the data set, such as filtering, aggregation, etc.

[0103] (3) Data Processor Parameters: Provides parameters required by the data processor, which will affect the specific behavior of data processing.

[0104] 2. Convert Data to Arrow Format

[0105] On the client side, all data that needs to be processed must be converted to Apache Arrow format. Apache Arrow provides a cross-platform columnar data structure that can efficiently process and transfer large amounts of data. The client can use APIs in Java or Python frameworks to achieve this conversion. For example, in Java, the ArrowVectorBuilder class can be used to build Arrow vectors, while in Python, the pyarrow library can be used to convert data.

[0106] 3. Send Data

[0107] The client sends data to the server through the following steps:

[0108] (1) Establish a two-way data flow channel: Establish a two-way data flow channel between the client and the server, which is used not only to send data but also to receive the server's response data. In the process of establishing the channel, processor type, processing method, and other meta-information will be passed.

[0109] (2) Send data: The client sends Arrow format data to the server through the established channel.

[0110] 4. Get the Server's Response

[0111] After sending data, the client needs to wait and get the server's response. After the server finishes processing, it will send the response data back to the client in Arrow format. The client then processes the next step based on the response data. The efficiency of this step directly affects the performance of the entire data exchange process.

[0112] Throughout the client process, data preparation, conversion, and transmission are critical steps that collectively ensure that data is received and processed by the server in an efficient and accurate format. By adhering to the Arrow Flight protocol, which significantly reduces data serialization overhead, the client can effectively exchange data with the server, enabling fast and reliable data processing and analysis.

[0113] The server process is as follows:

[0114] (1) The server starts listening for client connection requests.

[0115] After starting, the primary task of the server is to listen for connection requests from clients. This step is the foundation of communication between the server and the client. The server listens for requests on a specified port, waiting for the client to initiate a connection.

[0116] (2) Obtain the client's instructions

[0117] Once the server and client have established a connection, the next task is to obtain the instructions sent by the client. The server needs to parse these instructions to understand the specific needs of the client, including data processor types, processing methods, and parameters. This step is crucial for the server to correctly respond to client requests, as only by accurately understanding the client's instructions can the server perform the corresponding data processing operations.

[0118] (3) Obtain the processor instance

[0119] According to the information contained in the client's instructions, the server needs to obtain the corresponding processor instance. The processor is a component in the server responsible for specific data processing tasks. The server instantiates the processor to prepare to call its methods to process the data sent by the client. This step involves the server's management of different processors and the dynamic selection of appropriate processors based on client requests.

[0120] (4) Read data and convert to Spark DataFrame

[0121] After receiving the Arrow format data sent by the client, the server needs to read and convert it to a Spark DataFrame. Apache Spark is a powerful distributed data processing framework that can efficiently handle large-scale data sets. The server utilizes Spark's distributed processing capabilities to quickly filter, aggregate, and perform other operations on data. This step is the core of the data processing process and directly affects the efficiency and accuracy of data processing results.

[0122] (5) Process data

[0123] After converting to a Spark DataFrame, the server calls the processing method of the processor to process the data. This step involves specific data processing logic, such as data cleaning, feature extraction, model training, etc. The server performs the corresponding data processing operation according to the client's request to ensure that the processing result meets the client's needs.

[0124] (6) Convert the returned data to Arrow format

[0125] After the data processing is completed, the server needs to convert the processing result back to Arrow format to send it to the client. This step ensures the efficiency and compatibility of data transmission over the network. The server uses the API provided by Apache Arrow to encapsulate the processing result as Arrow format data, preparing for the return of data.

[0126] (7) Return the result

[0127] Finally, the server sends the processed data back to the client in Arrow format. This step is the final link of the server responding to the client's request and the end of the entire data exchange process. The server sends the response data in Arrow format to the client through the established data flow channel, completing the data transmission and processing task.

[0128] It should be noted that although the operations of the method of the present application are described in a specific order in the above embodiments and drawings, this does not require or imply that the operations must be performed in that specific order, or that all of the shown operations must be performed to achieve the desired result. Additionally or alternatively, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be divided into multiple steps.

[0129] Based on the foregoing inventive concept, as shown in Figure 5 The present application also proposes a computer device 200, which includes a memory 210, a processor 220, and a computer program 230 stored in the memory 210 and executable on the processor 220, wherein the processor 220 implements the foregoing pluggable distributed data processing method when executing the computer program 230.

[0130] Based on the foregoing inventive concept, the present application also proposes a computer-readable storage medium, which stores a computer program for executing the foregoing pluggable distributed data processing method.

[0131] The pluggable distributed data processing system and method proposed by the present application have the following highlights:

[0132] 1. High-performance data exchange

[0133] The system uses Apache Arrow Flight protocol to realize the bidirectional streaming data transmission between the client and the server, significantly reduces the data serialization overhead through the zero-copy mechanism and memory sharing technology, and realizes high throughput and low delay data processing service combined with the columnar storage characteristics and efficient memory layout of Arrow.

[0134] 2、Fine-grained type safety

[0135] Define independent schema and parameter constraints for each processing method, and perform automatic schema verification before and after data processing. Through this fine-grained type constraint mechanism, the type safety and result predictability of the data processing process are ensured, and complete interface self-description capability is provided.

[0136] 3、Distributed computing extension

[0137] Seamlessly convert Arrow format data into Spark DataFrame, and fully utilize the distributed computing engine of Spark for data processing. The distributed computing extension capability is obtained while ensuring the data transmission efficiency, and efficient processing of large-scale data is realized.

[0138] 4、Configuration-driven extension mechanism

[0139] Implement dynamic extension of data types through configuration file-driven data processor registration mechanism. New data types only need to implement processor interfaces and update configurations. The system automatically completes processor loading and management, providing good extensibility and maintainability.

[0140] Although the spirit and principles of the present application have been described with reference to several specific embodiments, it should be understood that the present application is not limited to the disclosed specific embodiments, and the division of aspects does not mean that the features in these aspects cannot be combined for benefit. This division is only for the convenience of expression. The present application is intended to cover various modifications and equivalent arrangements contained in the spirit and scope of the appended claims.

[0141] The scope of protection of the present application should be understood by those skilled in the art that various modifications or changes made on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A pluggable distributed data processing system, characterized by, The system comprises: The client is responsible for data collection and serialization. After serialization, the data is transmitted to the server through Arrow Flight in an efficient manner. The server is deployed on the edge node of the big data cluster. The Arrow Flight Server is started, and the data processing request from the client is listened to and processed. The Arrow Flight Server makes full use of the distributed computing capability of Spark, and the computing capability is flexibly expanded according to the demand. The Arrow Flight Server realizes the dynamic expansion of the data processing type through the configuration file driven data processor registration mechanism. The newly added data processing type only needs to implement the data processor interface and update the configuration.

2. The pluggable distributed data processing system of claim 1, wherein, The client constructs instructions according to the Arrow Flight protocol, including the data processor type, the specific processing method of the data set and the data processor parameters, and converts them into the Apache Arrow format, and sends them to the server through the bidirectional data flow channel, and waits for and obtains the response of the server.

3. The pluggable distributed data processing system of claim 1, wherein, The following steps are taken to start the server to listen: Before starting the Arrow Flight Server, initialize the SparkSession. Through the processor registration center configuration file, load the metadata information of all data processors, which is used for the client to route to the correct data processor for data processing when making a data processing request. Start the Flight Server to listen to the data processing request from the client, and respond to the request of the client after the data processing is completed.

4. The pluggable distributed data processing system of claim 1, wherein, The following steps are taken to realize the dynamic expansion of the data processing type: Define an interface specification class of the data processor, and implement specific processing logic and supported method set in it. Declare the meta information of the data processor in the processor registration center configuration file, including the data processor type, the processing class path, the input schema, the output schema of the method and the specific definition of the required parameters.

5. A pluggable distributed data processing method, characterized by, The method comprises: The client constructs instructions according to the Arrow Flight protocol, including the data processor type, the specific processing method of the data set and the data processor parameters, and converts them into the Apache Arrow format, and sends them to the server through the bidirectional data flow channel, and waits for and obtains the response of the server. After the server is started, the connection request from the client is listened to. Once the server and the client establish a connection, the instruction of the client is obtained and parsed, the corresponding processor instance is obtained according to the information contained in the instruction of the client. The server reads the Apache Arrow format data sent by the client, and converts it into a Spark DataFrame, and calls the processing method of the processor to process the data. After the data processing is completed, the server converts the processing result back to the Apache Arrow format and sends it back to the client.

6. The pluggable distributed data processing method of claim 5, wherein, The following steps are taken to start the server to listen: Before starting the Arrow Flight Server, initializing the SparkSession; By the processor registry center configuration file, load all the metadata information of the data processor, for the client in the data processing request, can be routed to the correct data processor for data processing; Start the Arrow Flight Server, listen to the client's data processing request, after the data processing is completed, respond to the client's request.

7. The pluggable distributed data processing method of claim 6, wherein, The ArrowFlight Server makes full use of the distributed computing capability of Spark, and the computing capability is elastically expanded according to the demand.

8. The pluggable distributed data processing method of claim 6, wherein, The ArrowFlight Server realizes the dynamic expansion of data processing type through the configuration file driven data processor registration mechanism; when a new data processing type is needed, define an interface specification class of the data processor, and implement specific processing logic and supported method set in it; declare the meta information of the data processor in the processor registry center configuration file, including the data processor type, the processing class path, the input schema, the output schema of the method and the specific definition of the required parameters.

9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 5-8 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores the computer program for executing the method of any one of claims 5-8.

Images