Micro-service and big data scheduling method

Abstract

The invention relates to a micro-service and big data scheduling method, which comprises a micro-service scheduling method and a big data scheduling method, and is characterized in that the big data scheduling method comprises the following steps of: 1, extracting data from a database to hadoop by adopting an sqoop program; 2, reading the extracted data by utilizing SparkSQL (Structured Query Language) for calculation; and 3, after the data calculation is completed, storing the data into a database or directly calling a resful service. The big data scheduling method disclosed by the inventionhas the advantages that 1, a data exporting command is generated by a program, so that errors are avoided; 2, a calculation program is executed by a management program, and errors are avoided; 3, allwork is controlled by a program, and the execution modes and methods are relatively unified; 4, a scheduling platform is utilized, and extraction and calculation programs can be executed regularly; and 5, the SparkSQL can be configured, only corresponding SQL statements need to be written for different tasks, and a complete program does not need to be written.

Description

technical field [0001] The invention relates to the field of scheduling methods, in particular to a microservice and big data scheduling method. Background technique [0002] Microservice scheduling. Generally, microservice scheduling is handled by using the service ID to be scheduled to obtain the corresponding service IP through the registration center, and then scheduling, which will result in multiple requests and delays. The request cost is high, the failure rate is high, and the stability is poor. [0003] For big data scheduling, before the present invention, people generally used manually written commands when performing data extraction, used Sqoop to extract data into HDFS, and then wrote a special SparkSQL program to read the extracted data, and based on the SQL already written in the program The statement calculates the data, and finally saves the calculation result to HDFS or the database. Different tasks require writing a complete set of SparkSQL programs. Th...

AI Technical Summary

Problems solved by technology

The disadvantages of this method are: 1. Using manual writing commands to export data, the error rate of command writing is relatively high; 2. The calculation program is also manually executing commands, and the error rate is relatively high; 3. Since it is all done manually, it is disadvantageous due to the need to perform calculations regularly Task; 4. SparkSQL is a special program, and each task needs to be written once, which is not conducive to the situation of changing task requirements, and the program flexibility is not high; 5. All work is done manually, and there are many ways to execute commands , unfavorable by unified management; 6. All commands are operated by the command line interface, and the execution efficiency is not high; 7. The predicate is downloaded, which can speed up data extraction; 8. The program can only be saved with the database and HDFS; 9. There is no unified Service management; 10. Integrate source and destination

Images