SparkSql external data source device, implementation method thereof and system
A technology of external data and source devices, which is applied in the field of SparkSql parsing external data sources, can solve problems such as lack of versatility, and achieve the effect of less error prone and short development cycle
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
Method used
Image
Examples
Embodiment 1
[0056] Such as figure 1 As shown, the present invention provides a SparkSql external data source device, and the external data source device includes:
[0057] The registration module 11 is used for registering the Sql query language received and analyzing and obtaining the logical plan;
[0058] A receiving module 12, configured to obtain the business logic code corresponding to the logical plan and transfer it to the physical plan;
[0059]The request data set module 13 is used to call the business logic code to query the corresponding external data source to obtain the data result set;
[0060] The partition conversion module 14 processes the data result set according to the preset number of partitions and converts the processed data result set into an iterative block, specifically Iterator[InternalRow].
[0061] It should be noted that the partition conversion module is optional, and in some embodiments, partitioning may not be performed, but the result set may be stored...
Embodiment 2
[0083] figure 2 It is a flow chart of a method embodiment implemented by the SparkSql external data source installer. The methods include:
[0084] S21. Registering the received Sql query language and analyzing it to obtain a logical plan;
[0085] S22. Obtain the business logic code corresponding to the logical plan and transfer it to the physical plan;
[0086] S23. Call the business logic code to query the corresponding external data source to obtain a data result set;
[0087] S24. Call the running block of SparkCore to format the result set of data and return it to the client.
[0088] The method also includes:
[0089] S210. After performing a recursive operation on the logic plan, inject the logic plan into a business logic interface so that developers can implement the business logic code;
[0090] The S22 includes: obtaining the business logic code corresponding to the logical plan and transferring it to the physical plan.
[0091] Preferably, the method also i...
Embodiment 3
[0186] The present invention also provides a computer system, comprising:
[0187] one or more processors; and
[0188] A memory associated with the one or more processors, the memory is used to store program instructions, and when the program instructions are read and executed by the one or more processors, perform the operations of the above method embodiments, specifically include:
[0189] Register and parse the received Sql query language to obtain a logical plan;
[0190] Obtain the business logic code corresponding to the logical plan and transfer it to the physical plan;
[0191] Call the business logic code to query the corresponding external data source to obtain the data result set;
[0192] Call the running block of SparkCore to format the data result set and return it to the client.
[0193] in, image 3 The architecture of the computer system is exemplarily shown, which may specifically include a processor 1510 , a video display adapter 1511 , a disk drive 1...
PUM
Login to View More Abstract
Description
Claims
Application Information
Login to View More