Big data computing task development method and device, equipment and medium

Abstract

The invention relates to a big data computing task development method and device, equipment and a medium, and belongs to the technical field of data processing. According to the method, step processes corresponding to a big data calculation task and read-write operation information on a database in each step are determined, the calculation task is summarized as a visual process implementation process, and an execution information packet in a spark system is determined based on a package file of the step processes and the operation information so as to complete big data calculation task development. According to the method, unified abstraction and classified solidification processing is carried out on operation of the spark on various types of data sources in the development process, the data sources are packaged into fixed operation nodes convenient for business personnel to understand and operate, the flexibility of the spark is sacrificed to a certain extent, but big data calculation task development is sufficient to be completed, and more importantly, the operation efficiency of the spark is greatly improved. And the use threshold of the spark in the big data computing task development process is remarkably reduced, so that the big data computing task development difficulty is remarkably reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, equipment and medium for developing big data computing tasks. Background Art

[0002] In projects based on big data environments such as CDH, most big data computing tasks involve multiple data sources, such as Oracle, Hive, HBase, etc., which require reading, writing and computing data in multiple data sources, and then complex Spark program development, task release testing, etc.

[0003] The Spark cluster computing system has high flexibility. For developers with experience in developing in an integrated development environment (IDE), the flexibility of Spark can effectively improve the development efficiency of big data computing tasks. However, precisely because of the high flexibility of the Spark cluster computing system, the Spark cluster computing system requires users to have a good grasp of the meaning and usage of various Spark operators, and learn to fill in a large number of personalized computing parameters for computing operators. Therefore, for developers who do not understand Spark, or even business operation and maintenance personnel who do not have programming development technical capabilities, the high requirements for the use of Spark will make it difficult to develop big data computing tasks, affecting the efficiency of processing and analyzing big data.

[0004] In other words, current big data computing tasks have technical problems that make development difficult. Summary of the invention

[0005] In view of this, the embodiments of the present invention provide a method, device, equipment and medium for developing big data computing tasks to solve the technical problem that the development of big data computing tasks is difficult.

[0006] In a first aspect, a method for developing a big data computing task is provided, the method comprising: Determine the step flow of the computing task according to the big data computing task information, and determine the operation information of reading and writing the database in each step of the step flow; Encapsulate the step flow and the operation information into a task json file; The task json file is configured in a general spark processing program to obtain the task id.jar corresponding to the computing task. The general spark processing program is used to parse the file format and data exchange format information in the task json file to determine the reading and writing information of each database; Respond to the computing task with the task id.jar to complete the big data computing task development.

[0007] In a second aspect, a big data computing task development device is provided, the device comprising: A task flow planning module is used to determine the step flow of the computing task according to the big data computing task information, and determine the operation information of reading and writing the database in each step of the step flow; A process information encapsulation module is used to encapsulate the step process and the operation information into a task json file; A task execution information acquisition module is used to configure the task json file in a general spark processing program to obtain the task id.jar corresponding to the computing task. The general spark processing program is used to parse the file format and data exchange format information in the task json file to determine the reading and writing information of each database; The computing task response module is used to respond to the computing task with the task id.jar to complete the development of big data computing tasks.

[0008] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor implements the big data computing task development method as described in the first aspect when executing the computer program.

[0009] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for developing big data computing tasks as described in the first aspect is implemented.

[0010] Compared with the prior art, the present invention has the following beneficial effects: The method of the present invention summarizes the big data computing task into an intuitive process implementation process by determining the step flow corresponding to the big data computing task and the read and write operation information of each step to the database, and then determines its execution information package in the spark system, namely the task id.jar, based on the json package file of the step flow and the operation information, responds to the big data computing task through the obtained task id.jar, and completes the development of the big data computing task. The big data computing task development process is implemented based on spark and sparkSQL, and the operations of spark on various types of data sources such as hive, hbase, oracle, etc. in the development process are uniformly abstracted and classified and solidified, and encapsulated into fixed operation nodes that are easy for business personnel to understand and operate. Although the flexibility of spark is sacrificed to a certain extent, it is also sufficient to complete the development of the big data computing task, and more importantly, the use threshold of spark in the development process of the big data computing task is significantly reduced, thereby significantly reducing the difficulty of the development of the big data computing task. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0012] Figure 1 This is a schematic diagram of an application environment of a big data computing task development method provided by the first embodiment of the present invention; Figure 2 It is a flowchart of a method for developing a big data computing task provided in the first embodiment of the present invention; Figure 3 It is a schematic diagram of determining a step flow in a visual development page provided by the first embodiment of the present invention; Figure 4 This is a schematic diagram of an operation information interface provided by Embodiment 1 of the present invention; Figure 5 This is a schematic diagram of an interface for defining step flow and operation information in JSON, provided in the first embodiment of the present invention; Figure 6 It is a data interaction structure diagram of a big data computing task development method provided by another preferred embodiment of the present invention; Figure 7 is a structural schematic diagram of a device provided in Embodiment 2 of the present invention; Figure 8 It is a structural diagram of a computer device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0013] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0014] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0015] It should also be understood that the term "and / or" used in the present description and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0016] As used in the present specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]", depending on the context.

[0017] In addition, in the description of the present specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0018] References to "one embodiment" or "some embodiments" etc. described in the present specification mean that one or more embodiments of the present invention include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways.

[0019] The embodiments of the present invention can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0020] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0021] It should be understood that the order of execution of the steps in the following embodiments does not imply a precedence of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0022] In order to illustrate the technical solution of the present invention, specific embodiments are provided below for illustration.

[0023] A big data computing task development method provided in the first embodiment of the present invention can be applied in the following aspects: Figure 1 In the application environment, the client communicates with the server. The client includes but is not limited to PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, personal digital assistants (PDAs), and other terminal devices. The server can be implemented with an independent server or a server cluster consisting of multiple servers.

[0024] See also Figure 2 , is a flow chart of a method for developing a big data computing task provided by the first embodiment of the present invention. The method for developing a big data computing task can be applied to Figure 1 The client in the client, the terminal device corresponding to the client connects to the target database through the preset application programming interface (API). When the target data is driven to execute the corresponding task, the corresponding task log will be generated, and the above task log can be collected through the API.

[0025] Before introducing the contents of the embodiments of the present invention, the following explanations are given to the terms in the art that may be involved in the embodiments of the present invention: Spark: Apache Spark is a fast, multi-purpose cluster computing system. Unlike Hadoop MapReduce, which stores intermediate results on disk, Spark uses memory to store intermediate results and can perform calculations in memory before the data is written to the hard disk. Since Spark is only a computing framework, it does not include a distributed file system and a complete scheduling system like Hadoop. Therefore, if you want to use Spark, you also need to install other file systems and more mature scheduling systems.

[0026] spark SQL: SQL (Structured Query Language) refers to the structured query language, and SparkSQL is the Spark module used by Spark to process structured data.

[0027] Hive: Hive is a data warehouse tool based on Hadoop, which is used for data extraction, transformation and loading. It is a mechanism for storing, querying and analyzing large-scale data stored in Hadoop. The hive data warehouse tool can map structured data files into a database table, and provides SQL query functions, which can convert SQL statements into MapReduce (where Map refers to mapping and Reduce refers to reduction) tasks for execution. The advantage of Hive is that it has a low learning cost and can implement fast MapReduce statistics through similar SQL statements, making MapReduce simpler without having to develop special MapReduce applications. Hive is very suitable for statistical analysis of data warehouses.

[0028] Hbase: HBase is a distributed, column-oriented open source database. It is a subproject of the Apache Hadoop project and aims to solve the storage problem of massive data. HBase is different from general relational databases. It is a database suitable for storing unstructured data.

[0029] DolphinScheduler: DolphinScheduler is a distributed, lightweight big data task scheduling platform designed to help enterprises efficiently manage and schedule complex workflows. It simplifies the process of task definition, dependency setting, and scheduling execution through a graphical interface, greatly reducing the threshold for users to get started. As an Apache top-level project, DolphinScheduler not only supports basic functions such as timed scheduling and dependency scheduling, but also can handle large-scale task scheduling needs in distributed environments.

[0030] YARN: YARN is a distributed resource scheduling system that is dedicated to providing computing resources for distributed computing programs. YARN is only responsible for providing resources and does not care about the logic of the computing program. Therefore, as long as the distributed computing program meets the running requirements of YARN, it can run on it and YARN will perform resource scheduling. Spark, Flink and other distributed computing programs can run on YARN.

[0031] like Figure 2 As shown, the big data computing task development method of this embodiment may include the following steps: Step S101, determining the step flow of the computing task according to the big data computing task information, and determining the operation information of reading and writing the database in each step of the step flow.

[0032] The general process of big data computing tasks is to read, calculate and write data in various databases, among which common databases include hive, oracle, hbase, etc.

[0033] In order to reduce the difficulty of non-professionals such as business operation and maintenance personnel in developing big data computing tasks and improve their intuitiveness, Figure 3 As shown, this embodiment preferably determines the read and write order and read and write category of different databases by dragging pre-set database read and write operation nodes in the visual development page, and uses the read and write order and read and write category as the step flow of the big data computing task.

[0034] Specific as Figure 3 For example, the specific process corresponding to the computing task is to read the device archive information from Oracle, then read the historical daily device frozen data from Hive, associate and calculate the collection success rate and write it to Hive. The corresponding operation on the visual development page is to drag a read oracle node with a read / write category of read, then connect a read hive computing node with a read / write category of read, and finally connect a write hive node with a read / write category of write to represent writing the collection success rate to the Hive table. The read / write order and read / write category of each database constructed by dragging and dropping on the visual development page can determine the step flow of the big data computing task.

[0035] Likewise Figure 3 As can be seen from the example in the figure, since the computing task not only includes the step flow content, but also includes the specific operation information or operation content of the database at each node of the database read and write operation, the specific content of the operation information needs to be clarified. Figure 4 As shown, the operation information of each operation node is specifically determined.

[0036] Based on the read and write categories of the database, this embodiment preferably divides the operation information into read information and write information, wherein the read information includes: database name, user name, read data SQL, that is, read data structured query statement, result set view name, whether the result set is cached, whether it is merged with the obtained result set, and personalized parameters of each database such as the scan start rowkey and end roukey of hbase; the write information includes: database name, user name, target table name, data set view name, append write or overwrite write (that is, write mode), and personalized parameters of each database such as hive partition column and partition value, oracle overwrite partition column, oracle write using insert or merge, and corresponding merge associated columns.

[0037] By obtaining the above step flow and operation information content of this step in this embodiment, the big data computing task is summarized as an intuitive process implementation process, which improves the convenience and understandability of business operation and maintenance personnel in developing big data computing tasks.

[0038] Step S102, encapsulate the step flow and the operation information into a task json file.

[0039] In order to ensure the operability of the big data computing task, after obtaining the step flow and operation information in step S101, it is also necessary to convert the step flow and the operation information into json, as follows: Figure 5 As shown, the step flow and the operation information are defined in json and encapsulated into a task json file corresponding to the computing task.

[0040] At the same time, this embodiment preferably stores the task json file, such as storing it in a mysql database, to facilitate the management of addition, deletion, modification and query operations, thereby facilitating the continuous optimization of big data computing task development.

[0041] Step S103, configuring the task json file in a general spark processing program to obtain the task id.jar corresponding to the computing task, and the general spark processing program is used to parse the file format and data exchange format information in the task json file to determine the reading and writing information of each database.

[0042] Based on the task json file, in order to make the big data computing task callable and executable on a cluster computing system such as spark, this embodiment integrates the task json file configuration into the general spark processing program to generate common-spark-taskid.jar, that is, to generate the task id.jar corresponding to the computing task.

[0043] Among them, the general spark processing program is responsible for parsing json information, that is, parsing the content of reading and writing database tables in json, or parsing the reading and writing information of each database in json, determining the spark computing process corresponding to the step flow and operation information, and executing the computing process of the big data computing task. Its core process includes accepting the configuration file identifier through parameters, parsing the corresponding configuration file; using corresponding algorithms to process different types of step attributes, and converting them into corresponding addition, deletion, modification and query sql; and then executing business calculations based on spark sql. The algorithm for converting sql for each type of step is embedded with monitoring messages that send processing status to the outside, and the external receiving of messages and processing of monitoring display task process.

[0044] Step S104, responding to the computing task with the task id.jar to complete the big data computing task development.

[0045] The task id.jar obtained in the above step S103 is essentially the code content that can be executed in spark based on the step flow and operation information corresponding to the computing task. Therefore, based on the task id.jar, you can respond to the big data computing task and complete the development of the big data computing task.

[0046] Specifically, regarding the method of responding to a computing task with task id.jar, the preferred embodiment is as follows: Use the page publishing function to publish the information of the big data computing task and the task ID.jar to the workflow of the big data task scheduling platform through the interface exposed by the big data task scheduling platform, completing the preparation work before going online; The workflow corresponding to the computing task is scheduled through the interface exposed by the big data task scheduling platform. Then the big data task scheduling platform calls the spark submit task command (spark-submit) to submit the spark computing task corresponding to the task id.jar to the resource management system for execution, completes the response to the big data computing task, and obtains the calculation result of the big data computing task.

[0047] Among them, in this embodiment, the preferred big data task scheduling platform is DolphinScheduler, the resource management system is yarn, and the interface is the rest interface. In other embodiments, the operator can also use any other existing feasible big data task scheduling platform and resource management system.

[0048] At the same time, considering that a considerable part of the calculation results of big data computing tasks need to be updated in real time, in order to make the development of big data computing tasks more reasonable, in a preferred embodiment, the specific method of scheduling the workflow corresponding to the computing task exposed by the interface of the big data task scheduling platform is defined as follows: The periodic execution instruction information of the periodic execution computing task is saved in the big data task scheduling platform through the interface exposed to the outside by the big data task scheduling platform, and the scheduling of the workflow corresponding to the computing task is completed based on the periodic execution instruction information.

[0049] By configuring the periodic execution instruction information in the big data task scheduling platform, the periodic call of the workflow corresponding to the fixed big data computing task can be completed, and the corresponding spark computing task can be executed periodically and continuously, thereby realizing the real-time update of the computing result. Of course, in other embodiments, the operator, such as the business operation and maintenance personnel, can also manually schedule the workflow corresponding to the computing task to complete the execution of the spark computing task corresponding to the big data computing task and obtain the computing result of the big data computing task.

[0050] In another preferred embodiment, in order to further improve the development effect of big data computing tasks, a method is provided as follows: Figure 6 The data interaction structure of the big data computing task development method shown in the figure, in which the big data computing task development method also includes the content of process monitoring and display of the big data computing task running process information, and based on the task json file stored in the above embodiment, the big data computing task development process is continuously debugged.

[0051] Specifically, when the spark computing task program runs in the yarn environment, it relies on the read and write step flow of the big data computing task configured on the visual development page, reads the data table from each type of data repository according to the operation information of the corresponding data repository, and completes the calculation. After each step in the calculation process is completed, the spark computing task program will output the applicationID, read and write data table, read and write data volume, read and write time and other information of the current big data computing task through the kafka message queue. After receiving the message, the background writes the record to the background management database. At the same time, the page process monitoring function can also use the applicationId to display in real time which step the current computing task has reached, the current time consumption, and the amount of data operated, etc.

[0052] The process monitoring function can also obtain the running client logs and each yarn node log from yarn through applicationId and display them on the display page to debug and analyze problems in big data computing tasks. Through log analysis, it can quickly respond to and solve problems in the development process, so as to facilitate the modification, debugging and scheduled launch of big data computing tasks.

[0053] Effects of the embodiments of the present invention: The method of the present invention summarizes the big data computing task into an intuitive process implementation process by determining the step flow corresponding to the big data computing task and the read and write operation information of each step to the database, and then determines its execution information package in the spark system, namely the task id.jar, based on the json package file of the step flow and the operation information, responds to the big data computing task through the obtained task id.jar, and completes the development of the big data computing task. The big data computing task development process is implemented based on spark and sparkSQL, and the operations of spark on various types of data sources such as hive, hbase, oracle, etc. in the development process are uniformly abstracted and classified and solidified, and encapsulated into fixed operation nodes that are easy for business personnel to understand and operate. Although the flexibility of spark is sacrificed to a certain extent, it is also sufficient to complete the development of the big data computing task, and more importantly, the use threshold of spark in the development process of the big data computing task is significantly reduced, thereby significantly reducing the difficulty of the development of the big data computing task.

[0054] Corresponding to the method of the above embodiment, Figure 7 The structural block diagram of a big data computing task development device provided by the second embodiment of the present invention is shown. The development device is applied to a computer device, and the computer device is connected to a target database through a preset application program interface. When the target database is driven to execute a corresponding task, a corresponding task log is generated, and the task log can be collected through the API. For ease of explanation, only the part related to the embodiment of the present invention is shown.

[0055] See also Figure 7 , the development device includes: The task flow planning module 21 is used to determine the step flow of the computing task according to the big data computing task information, and determine the operation information of reading and writing the database in each step of the step flow; A process information encapsulation module 22 is used to encapsulate the step process and the operation information into a task json file; The task execution information acquisition module 23 is used to configure the task json file in the general spark processing program to obtain the task id.jar corresponding to the computing task. The general spark processing program is used to parse the file format and data exchange format information in the task json file to determine the reading and writing information of each database; The computing task response module 24 is used to respond to the computing task with the task id.jar to complete the big data computing task development.

[0056] Optionally, the task process planning module 21 includes: A step flow determination unit is used to determine the read and write order and read and write category of each of the databases based on the big data computing task information by dragging the pre-set read and write operation nodes for each of the databases in a visual development page, and use the read and write order and the read and write category as the step flow of the computing task.

[0057] Optionally, the task process planning module 21 further includes: An operation information limiting unit, used for limiting the operation information to include read-related information and write-related information; The read information includes: database name, user name, structured query statement for reading data, result set view name, whether the result set is cached, whether it is merged with the obtained result set, and personalized parameters of each database; The write-in information includes: database name, user name, target table name, data set view name, write mode, and personalized parameters of each database.

[0058] Optionally, the computing task response module 24 includes: A computing task response unit, used to limit the computing task to be responded to with the task id.jar, includes: The big data computing task information and the task id.jar are published to the workflow of the big data task scheduling platform through the interface exposed by the big data task scheduling platform through the page publishing function; The workflow corresponding to the computing task is scheduled through the interface exposed by the big data task scheduling platform, and the big data task scheduling platform calls the spark submit task command to submit the spark computing task corresponding to the task id.jar to the resource management system for execution.

[0059] Optionally, the computing task response unit includes: The workflow scheduling subunit is used to limit the workflow corresponding to the computing task to be scheduled through the interface exposed by the big data task scheduling platform, including: The periodic execution instruction information of the periodic execution of the computing task is saved in the big data task scheduling platform through the interface exposed to the outside by the big data task scheduling platform, and the scheduling of the workflow corresponding to the computing task is completed based on the periodic execution instruction information.

[0060] Optionally, the computing task response unit further includes: The platform and system definition subunit is used to define the big data task scheduling platform as DolphinScheduler and the resource management system as yarn.

[0061] Optionally, the above device further comprises: The process monitoring module is used to output the application ID, read and write data table information, read and write data volume information and read and write time of the computing task through the message queue during the running of the computing task, and display it through the display page.

[0062] It should be noted that the information interaction, execution process and other contents between the above modules are based on the same concept as the embodiment of the method of the present invention. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0063] Figure 8 This is a schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. Figure 8 As shown, the computer device of this embodiment includes: at least one processor ( Figure 8 Only one is shown), a memory, and a computer program stored in the memory and executable on at least one processor, wherein when the processor executes the computer program, the steps in any of the above-mentioned big data computing task development method embodiments are implemented.

[0064] The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that Figure 8 This is merely an example of a computer device and does not constitute a limitation on the computer device. The computer device may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0065] The processor may be a CPU, or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0066] The memory includes a readable storage medium, an internal memory, etc., wherein the internal memory may be the memory of a computer device, and the internal memory provides an environment for the operation of the operating system and computer-readable instructions in the readable storage medium. The readable storage medium may be a hard disk of a computer device, and in other embodiments, it may also be an external storage device of the computer device, for example, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device. Further, the memory may also include both an internal storage unit of the computer device and an external storage device. The memory is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of a computer program, etc. The memory may also be used to temporarily store data that has been output or is to be output.

[0067] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention. The specific working process of the units and modules in the above-mentioned device can refer to the corresponding process in the above-mentioned method embodiment, which will not be repeated here. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of the above-mentioned method embodiment can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying computer program code, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0068] The present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by a computer program product. When the computer program product runs on a computer device, the computer device can implement the steps in the above-mentioned method embodiment when executing.

[0069] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0070] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0071] In the embodiments provided by the present invention, it should be understood that the disclosed devices / computer equipment and methods can be implemented in other ways. For example, the device / computer equipment embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0072] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0073] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A method for developing big data computing tasks, characterized in that: The method comprises: Determine the step flow of the computing task according to the big data computing task information, and determine the operation information of reading and writing the database in each step of the step flow; Encapsulate the step flow and the operation information into a task json file; The task json file is configured in a general spark processing program to obtain the task id.jar corresponding to the computing task. The general spark processing program is used to parse the file format and data exchange format information in the task json file to determine the reading and writing information of each database; Respond to the computing task with the task id.jar to complete the big data computing task development.

2. The big data computing task development method according to claim 1, characterized in that: The step flow of determining the computing task according to the big data computing task information includes: Based on the big data computing task information, the read and write operation nodes for each of the databases are pre-set by dragging in the visual development page to determine the read and write order and read and write category for each of the databases, and the read and write order and the read and write category are used as the step flow of the computing task.

3. The big data computing task development method according to claim 1, characterized in that: The operation information includes read information and write information; The read information includes: database name, user name, structured query statement for reading data, result set view name, whether the result set is cached, whether it is merged with the obtained result set, and personalized parameters of each database; The write-in information includes: database name, user name, target table name, data set view name, write mode, and personalized parameters of each database.

4. The method for developing big data computing tasks according to claim 1, characterized in that: The step of responding to the computing task with the task id.jar includes: The big data computing task information and the task id.jar are published to the workflow of the big data task scheduling platform through the interface exposed by the big data task scheduling platform through the page publishing function; The workflow corresponding to the computing task is scheduled through the interface exposed by the big data task scheduling platform, and the big data task scheduling platform calls the spark submit task command to submit the spark computing task corresponding to the task id.jar to the resource management system for execution.

5. The method for developing big data computing tasks according to claim 4, characterized in that: The step of scheduling the workflow corresponding to the computing task through the interface exposed by the big data task scheduling platform includes: The periodic execution instruction information of the periodic execution of the computing task is saved in the big data task scheduling platform through the interface exposed to the outside by the big data task scheduling platform, and the scheduling of the workflow corresponding to the computing task is completed based on the periodic execution instruction information.

6. The method for developing big data computing tasks according to claim 4, characterized in that: The big data task scheduling platform is DolphinScheduler, and the resource management system is yarn.

7. The method for developing a big data computing task according to any one of claims 1 to 6, characterized in that: During the running of the computing task, the application ID, read and write data table information, read and write data volume information and read and write time of the computing task are output through the message queue and displayed through the display page.

8. A big data computing task development device, characterized in that: The device comprises: A task flow planning module is used to determine the step flow of the computing task according to the big data computing task information, and determine the operation information of reading and writing the database in each step of the step flow; A process information encapsulation module is used to encapsulate the step process and the operation information into a task json file; A task execution information acquisition module is used to configure the task json file in a general spark processing program to obtain the task id.jar corresponding to the computing task. The general spark processing program is used to parse the file format and data exchange format information in the task json file to determine the reading and writing information of each database; The computing task response module is used to respond to the computing task with the task id.jar to complete the development of big data computing tasks.

9. A computer device, characterized in that: The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the method for developing big data computing tasks as described in any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the big data computing task development method as described in any one of claims 1 to 7 is implemented.

Images