Distributed SQL processing method and system
A processing method and processing system technology, applied in the field of distributed SQL processing methods and systems, can solve problems such as manual errors, and achieve the effects of sufficient memory for execution, improving performance, and avoiding errors in manual selection.
Active Publication Date: 2018-04-27
HUAZHONG UNIV OF SCI & TECH
View PDF3 Cites 1 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0008] Aiming at the defects of the prior art, the purpose of the present invention is to solve the problem that in the existing distributed SQL processing system, there are often intermediate results that can be reused to avoid repeated calculations between various application programs and within the application programs, but how to screen out these The intermediate results that can be used currently need to be manually determined by programmers, which requires programmers to have deep programming skills, and technical problems that are prone to manual errors
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0029] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0030] figure 1 A schematic flowchart of the distributed SQL processing method provided by the present invention, such as figure 1 shown, including:
[0031] S1: Receive an application program submitted by a user, and store the submitted application program in an application queue.
[0032]S2, analyze the code of each application program in the application queue, each application program includes...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a distributed SQL processing method and system. The method comprises the steps that application programs submitted by users are received, and the submitted application programsare stored into an application queue; the code of each application program in the application queue is analyzed, wherein each application program comprises a plurality of intermediate results, a first marker is made to the intermediate results, needing to be used repeatedly in the application program, in each application program, and a second marker is made to the intermediate results, needing tobe used repeatedly in the following application program, in each application program; while the data consistency is ensured, the application programs in the application queues which run in parallel on each Worker are completed, and according to the first marker and the second marker, running speed is increased by using the intermediate results corresponding to the markers respectively. By means of the method, data needing to be cached is more precisely selected, errors caused by manual selection are avoided, and therefore, the program performance is improved.
Description
technical field [0001] The present invention relates to the technical field of big data processing, and more particularly, to a distributed SQL processing method and system. Background technique [0002] Many real-world applications involve structured query language (SQL, Structured Query Language) data processing. Distributed data processing systems achieve high computing efficiency by using multiple machines to process data in parallel, but there are still problems such as unreasonable utilization of memory resources. In academia, there have been many studies on these issues. [0003] Spark is a research project that originated from the cluster computing platform of the AMP Lab at the University of California, Berkeley, in 2009. After that, the Spark project was open sourced in 2010, became an Apache funded project in 2013, and became an Apache top-level funded project in 2014. It is based on in-memory computing, starts from multi-iteration batch processing, and integra...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
CPCG06F16/2433G06F16/252
Inventor 石宣化金海王斐
Owner HUAZHONG UNIV OF SCI & TECH