Spark-based target task processing method and device and electronic equipment

Abstract

The disclosure provides a Spark-based target task processing method and device and electronic equipment, relating to the technical field of computers, specifically to the technical field of high-precision maps and big data, and the specific implementation scheme is: obtaining respective processing programs of a plurality of parallel business modules of a target task; based on a structured query language (SQL) statement, the processing programs of the plurality of parallel business modules are combined and programmed to obtain an SQL script; the SQL script is sent to a Spark big data computing engine, and the SQL script is parsed by the Spark big data computing engine to generate a to-be-executed job; and the to-be-executed job is processed by the Spark big data computing engine to obtain a processing result corresponding to the target task. The disclosure improves the ability of the Spark big data computing engine to process jobs submitted in the form of an SQL statement by extending the function of the SQL statement and adding an extended SQL statement parsing mode to the Spark big data computing engine, thereby improving the utilization rate of computing resources.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computers, in particular to the technical field of high-precision maps and big data, and more particularly to a target task processing method and device based on Spark and an electronic device. BACKGROUND

[0002] A production line of a high-precision map needs to go through data solving, algorithm fusion, positioning map generation, quality inspection and other business processes, and the production line is relatively complex. In the mapping production line, a scheduling framework such as airflow is widely used to organize business modules into a directed acyclic graph (DAG), wherein some parallelly executed module instances have relevance in some aspects, for example: the parallelly executed modules depend on the same files or toolkits; the parallelly executed modules read the same data source; and most of the program logic of the parallelly executed modules is the same, and the submission parameters of the spark distributed job are determined by a specific environment variable.

[0003] In the related art, for the parallelly executed business module instances, multiple containers are started, and the process of pulling dependent files and toolkits is repeatedly executed in each container of the running module instance. Moreover, it is difficult to combine multiple tasks when deploying different containers into one, resulting in a large cluster resource overhead, a low utilization rate of computing resources, and a low computing efficiency. Therefore, how to improve the resource utilization rate in the processing of a target task and save the cluster resources in the processing of the target task has become a problem to be solved. SUMMARY

[0004] The present disclosure provides a target task processing method and device based on Spark, an electronic device, a storage medium and a program product.

[0005] According to a first aspect, a target task processing method based on Spark is provided, including: obtaining respective processing programs of a plurality of parallel business modules of a target task; based on a structured query language (SQL) statement, merging and programming the processing programs of the plurality of parallel business modules to obtain an SQL script; sending the SQL script to a Spark big data computing engine, and parsing the SQL script by the Spark big data computing engine to generate a to-be-executed job; and processing the to-be-executed job by the Spark big data computing engine to obtain a processing result corresponding to the target task.

[0006] According to a second aspect, a processing apparatus for a target task based on Spark is provided, comprising: an obtaining module configured to obtain respective processing programs of a plurality of parallel business modules of the target task; a merging module configured to merge program of the processing programs of the plurality of parallel business modules based on a structured query language (SQL) statement to obtain a SQL script; an analyzing module configured to send the SQL script to a Spark big data computing engine, and analyze the SQL script by the Spark big data computing engine to generate a to-be-executed job; and a processing module configured to process the to-be-executed job by the Spark big data computing engine to obtain a processing result corresponding to the target task.

[0007] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the processing method for a target task based on Spark according to the first aspect of the present disclosure.

[0008] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to perform the processing method for a target task based on Spark according to the first aspect of the present disclosure.

[0009] According to a fifth aspect, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the processing method for a target task based on Spark according to the first aspect of the present disclosure.

[0010] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0011] The accompanying drawings are used to better understand the present scheme, and do not limit the present disclosure. Among them:

[0012] Figure 1 is a flow diagram of the processing method for a target task based on Spark according to the first embodiment of the present disclosure;

[0013] Figure 2 is a flow diagram of the processing method for a target task based on Spark according to the first embodiment of the present disclosure;

[0014] Figure 3 is a flow diagram of the processing method for a target task based on Spark according to the first embodiment of the present disclosure;

[0015] Figure 4 is a flowchart of a processing method of a target task based on Spark according to a second embodiment of the present disclosure;

[0016] Figure 5 is a flowchart of a processing method of a target task based on Spark according to a third embodiment of the present disclosure;

[0017] Figure 6 is a block diagram of a processing device of a target task based on Spark for implementing the embodiments of the present disclosure;

[0018] Figure 7 is a block diagram of an electronic device for implementing a processing method of a target task based on Spark according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Thus, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, for the sake of brevity and clarity, descriptions of well-known functions and constructions are omitted from the following description.

[0020] Computer Technology, the content is very extensive, can be roughly divided into computer system technology, computer device technology, computer component technology and computer assembly technology and so on. Computer technology 5 includes: the basic principles of operation method and operation device design, instruction system, central processing unit (CPU) design, pipeline principle and its application in CPU design, storage system, bus and input and output.

[0021] High-precision map (Precision Map) refers to a map with high precision and fine definition. High-precision map is mainly a major tool for assisting driving and automatic driving, and is provided for computer use, and is one of the most core sensing technologies for automatic driving.

[0022] Big data (big data) or huge data refers to data volume so large that it cannot be captured, managed, processed and organized into information to help business decision-making in a reasonable time through mainstream software tools.

[0023] A processing method of a target task based on Spark according to an embodiment of the present disclosure is described below with reference to the accompanying drawings.

[0024] Figure 1 is a flowchart of a processing method of a target task based on Spark according to a first embodiment of the present disclosure.

[0025] As Figure 1 shown, the processing method of the target task based on Spark of the embodiment of the disclosure can specifically include the following steps:

[0026] S101, obtaining respective processing programs of a plurality of parallel business modules of the target task.

[0027] Specifically, the execution subject of the processing method of the target task based on Spark of the embodiment of the disclosure can be a processing device of the target task based on Spark provided by the embodiment of the disclosure, which can be a hardware device with data information processing capability and / or necessary software required to drive the hardware device to work. Alternatively, the execution subject can include workstations, servers, computers, user terminals and other devices. Among them, the user terminal includes but is not limited to mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle-mounted terminals, etc.

[0028] Optionally, the target task can be a mapping task of a high-precision map.

[0029] It should be noted that the target task has a plurality of parallel business modules, wherein different parallel business modules correspond to different business programs, which can be the same or different.

[0030] For example, for the target task A, the processing program corresponding to the parallel business module 1 and the processing program corresponding to the parallel business module 2.

[0031] S102, based on a structured query language (SQL) statement, merging and programming the processing programs of the plurality of parallel business modules to obtain a SQL script.

[0032] Among them, the structured query language (Structured Query Language, SQL) is a database query and program design language, which is used to access data and query, update and manage relational database systems. SQL statement is a language for operating database.

[0033] It should be noted that the function of the SQL language level can be extended, and then based on the SQL statement, the processing programs of the plurality of parallel business modules are merged and programmed to obtain the SQL script.

[0034] Optionally, the pipe function can be embedded into the SQL statement to configure the running environment corresponding to the processing program, and based on the running environment corresponding to the processing program, the processing program is merged and programmed to obtain the SQL script.

[0035] S103, the SQL script is sent to the Spark big data computing engine, and the SQL script is parsed by the Spark big data computing engine to generate a to-be-executed job.

[0036] Wherein, the Spark is a general big data computing engine, which has high performance, easy to use and universality.

[0037] In the embodiment of the present disclosure, after obtaining the SQL script, the SQL script can be sent to the Spark big data computing engine, and the SQL script is parsed by the Spark big data computing engine to generate a to-be-executed job.

[0038] It should be noted that, since the present disclosure extends the function at the SQL language level, the Spark big data computing engine is also extended to adapt to the extended function at the SQL language level, that is, on the basis of the original Spark big data computing engine supporting the SQL language, the SQL script can be parsed into a to-be-executed job, and the original business module can be parallelized and integrated into the parallelization of the job execution stage.

[0039] S104, the to-be-executed job is processed by the Spark big data computing engine to obtain the processing result corresponding to the target task.

[0040] In the embodiment of the present disclosure, after obtaining the to-be-executed job, the Spark big data computing engine can schedule resources for the to-be-executed job, and obtain the data source required by the to-be-executed job, process the to-be-executed job based on the scheduled resources and the data source, and then obtain the processing result corresponding to the target task.

[0041] Wherein, the to-be-executed job includes a physical execution plan corresponding to the processing logic.

[0042] It should be noted that, in the related art, such as Figure 2As shown, in the existing parallel method of distributed computing jobs, the processes of pulling environment variables param#1, param#2, preparing files files and program packages archives are repeated in each container Kubernetes, resulting in large cluster resource overhead, and each parallel business module needs to read the data source hive once. In the case of a large number of business modules, the data source is under a large concurrent access pressure. Meanwhile, in the execution process of multiple parallel business modules, each business module submits a distributed computing job request once, resulting in low resource utilization. In addition, the original Apache-spark framework supports the job submission method, which is difficult to combine multiple jobs receiving input parameters from different container deployments into one job to perform Mapper#1 task, Reducer#1 task, Mapper#2 task and Reducer#2 task respectively.

[0043] It should be noted that the processing method of the target task based on Spark proposed by the present disclosure, as shown in Figure 3 As shown, only one container Kubernetes is started, and the processing programs of multiple parallel business modules are combined and programmed based on structured query language SQL statements to obtain a SQL script. Only the processes of pulling environment variables param#1, environment variables param#2, template shared files shared files and program packages archives need to be performed once, thereby reducing the cluster resource overhead. In addition, the parallel business modules only need to read the data source hive once. In the case of a large number of business modules, the data source is under a large concurrent access pressure. Meanwhile, in the execution process of multiple parallel business modules, the parallel business modules submit a distributed computing job request once. In addition, the original Spark distributed job submission method is improved to implement business logic and submit a job through a SQL script. Multiple jobs receiving input parameters from different container deployments are combined into one job to perform Mapper#1 task, Reducer#1 task, Mapper#2 task and Reducer#2 task in the job internal stage level.

[0044] In summary, the processing method of the target task based on Spark in the embodiment of the disclosure comprises the following steps:

[0045] Figure 4 is a flowchart of the processing method of the target task based on Spark according to the second embodiment of the disclosure.

[0046] As shown in Figure 4 the processing method of the target task based on Spark in the embodiment of the disclosure can comprise the following steps based on the embodiment shown in Figure 1

[0047] S401, obtaining the processing program of each of the plurality of parallel business modules of the target task.

[0048] Specifically, step S401 in the embodiment is the same as step S101 in the above-mentioned embodiment, which will not be described here.

[0049] Step S102 in the above-mentioned embodiment, that is, "merging and programming the processing program of the plurality of parallel business modules based on the structured query language (SQL) statement to obtain a SQL script" can specifically comprise the following steps S402-S404.

[0050] S402, obtaining the processing program of the plurality of parallel business modules by using a pipe function in the SQL statement.

[0051] It should be noted that the pipe function can be embedded into the SQL statement to obtain the processing program of the plurality of parallel business modules.

[0052] It should be noted that the processing program of the parallel business module can be constructed by using a plurality of languages through the pipe function.

[0053] Optionally, the processing program of the parallel business module can be constructed based on C++, Shell, and Python. ​

[0054] S403, configuring the running environment corresponding to the processing program.

[0055] It should be noted that, since the running environment required by different processing programs can be different, the running environment corresponding to each processing program can be configured respectively.

[0056] Optionally, the environment variable can be configured separately for the execution process of each processing program in the corresponding field in the SQL statement.

[0057] S404, based on the running environment corresponding to the processing program, the processing program is programmed to obtain a SQL script.

[0058] In the embodiments of the present disclosure, by configuring the running environment corresponding to the processing program, the processing programs depending on different environment variables can be combined into the same SQL script, that is, the processing programs can be programmed to obtain a SQL script.

[0059] In the embodiments of the present application, a file acquisition request can be sent to the local distributed file system of the Spark big data computing engine, and according to the file acquisition request, the shared files and program packages corresponding to the multiple parallel business modules are acquired from the local distributed file system.

[0060] It should be noted that, in the related art, when submitting a spark job, a mapper and a reducer field need to be added in the command of submitting the spark job, that is, one spark job corresponds to one task, in this case, there is only one mapper and one reducer under one spark job.

[0061] However, by programming the processing programs to obtain a SQL script, the mapper and reducer fields in the spark job submission command are not required, only a SQL script field needs to be submitted, and the SQL script is sent to the spark job submission command, wherein by editing the SQL statement in the SQL script, one SQL statement corresponds to the logic executed by one mapper or one reducer, then two SQL statements can correspond to one processing task previously, further, when programming multiple processing programs in the SQL script, the running environment corresponding to the processing program is configured, for example: if three processing tasks are combined, six SQL statements can be programmed and combined, and the environment variables corresponding to the six SQL statements are set to form a SQL script.

[0062] S405, send the SQL script to the Spark big data computing engine, and parse the SQL script by the Spark big data computing engine to generate a to-be-executed job.

[0063] It should be noted that after the SQL script is obtained, the SQL script can be sent to the Spark big data computing engine, and the SQL script is parsed by the Spark big data computing engine. When the SQL script is parsed, each SQL statement in the SQL script can be parsed, that is, the underlying logic is parsed. After each SQL statement is parsed, it exists in the form of a temporary table. When all SQL statements in the SQL script are parsed, all temporary tables are concatenated according to the logical relationship between the temporary tables, that is, the mapper and the reducer can be concatenated two by two, and then a to-be-executed job is formed.

[0064] S406, resource scheduling of the to-be-executed job is performed by the Spark big data computing engine.

[0065] Optionally, the computing resources such as central processing units (CPUs) and memories can be scheduled.

[0066] S407, a data source required by the to-be-executed job is obtained.

[0067] It should be noted that the specific manner of obtaining the data source required by the to-be-executed job is not limited in the present application, and can be selected according to actual conditions.

[0068] Optionally, a data source obtaining request can be sent to a local distributed file storage system of the Spark big data computing engine, and the data source is obtained from the local distributed file storage system.

[0069] For example, the data source can be obtained from the HDFS and the Hive distributed file storage system.

[0070] Optionally, the data source can be extended, a data source obtaining request can be sent to an external distributed file storage system of the Spark big data computing engine, and the data source is obtained from the external distributed file storage system.

[0071] For example, the data source can be obtained from the distributed file storage system in the enterprise.

[0072] S408, the to-be-executed job is processed based on the scheduled resources and the data source to obtain a processing result corresponding to the target task.

[0073] As a possible implementation manner, as shown in Figure 5As shown, on the basis of the above embodiment, the specific process of processing the to-be-executed job based on the scheduled resource and the data source in step S408 to obtain the processing result corresponding to the target task includes the following steps:

[0074] S501, obtaining the physical execution plan corresponding to the processing logic in the to-be-executed job.

[0075] It should be noted that the to-be-executed job includes the physical execution plans corresponding to the processing logics of multiple parallel business modules.

[0076] S502, based on the program package, obtaining the dependency file of the multiple parallel business modules and obtaining the metadata file of the data source.

[0077] The metadata (Metadata) is also called intermediate data and relay data, which is data about data, mainly information describing data properties (property), and is used to support functions such as indicating storage location, historical data, resource search, file record, etc.

[0078] It should be noted that the metadata file can specify the range of the data source read by the to-be-executed job.

[0079] S503, determining the dependency relationship of the physical execution plan based on the dependency file.

[0080] S504, reading the data source based on the metadata file, and performing the physical execution plan on the data source according to the dependency relationship to obtain the processing result corresponding to the target task.

[0081] It should be noted that after reading the data source based on the metadata file, the read data source can be cached in the distributed cluster executor memory, so as to avoid reading the same data source multiple times and reduce the data source access hotspot problem caused by too many parallel business modules.

[0082] In the embodiment of the present disclosure, after reading the data source according to the metadata file, the physical execution plan can be performed on the data source according to the dependency relationship to obtain the processing result corresponding to the target task.

[0083] In summary, the processing method of the target task based on Spark provided in the embodiments of the present disclosure obtains the respective processing programs of the plurality of parallel business modules of the target task, obtains the processing programs of the plurality of parallel business modules through the pipe function in the SQL statement, configures the running environment corresponding to the processing programs, merges and programs the processing programs based on the running environment corresponding to the processing programs, obtains the SQL script, sends the SQL script to the Spark big data computing engine, parses the SQL script by the Spark big data computing engine to generate a to-be-executed job, schedules the resources of the to-be-executed job by the Spark big data computing engine, obtains the data source required by the to-be-executed job, processes the to-be-executed job based on the scheduled resources and the data source, and obtains the processing result corresponding to the target task. Thus, the present disclosure improves the flexibility and scalability of the business module processing program by parsing the SQL script to obtain the to-be-executed job, reduces the learning cost, improves the visibility of the process, and at the same time, can parallelize the original container-level modules into the parallelization of the job execution stage, which can significantly improve the utilization rate of computing resources such as CPU and memory, and is beneficial to maintaining the health status of the entire computing cluster.

[0084] The processing method of the target task based on Spark will be explained below taking the production line scene of high-definition maps as an example.

[0085] The processing method of the target task based on Spark provided in the present disclosure obtains the respective processing programs of the plurality of parallel business modules of the production task of high-definition maps based on the production scene of high-definition maps, merges and programs the processing programs of the plurality of parallel business modules based on the structured query language (SQL) statement, obtains the SQL script, sends the SQL script to the Spark big data computing engine, parses the SQL script by the Spark big data computing engine to generate a to-be-executed job, schedules the resources of the to-be-executed job by the Spark big data computing engine, obtains the data source required by the production task of high-definition maps, processes the to-be-executed job based on the scheduled resources and the data source, and obtains the processing result corresponding to the target task. In summary, the processing method of the target task based on Spark provided in the embodiments of the present disclosure improves the flexibility and scalability of the business module processing program by parsing the SQL script to obtain the to-be-executed job, reduces the learning cost, improves the visibility of the process, and at the same time, can parallelize the original container-level modules into the parallelization of the job execution stage, which can significantly improve the utilization rate of computing resources such as CPU and memory, and is beneficial to maintaining the health status of the entire computing cluster.

[0086] It should be noted that the acquisition, storage and application of user personal information involved in the technical solutions of the present disclosure comply with relevant laws and regulations and do not violate public order and good customs.

[0087] Figure 6 is a structural schematic diagram of a Spark-based target task processing apparatus according to one embodiment of the present disclosure.

[0088] As shown in Figure 6 The Spark-based target task processing apparatus 600 includes an acquisition module 610, a merging module 620, an analysis module 630, and a processing module 640. Wherein:

[0089] The acquisition module 610 is configured to acquire respective processing programs of a plurality of parallel business modules of a target task;

[0090] The merging module 620 is configured to perform merging programming on the processing programs of the plurality of parallel business modules based on a structured query language (SQL) statement, to obtain an SQL script;

[0091] The analysis module 630 is configured to send the SQL script to a Spark big data computing engine, and perform analysis on the SQL script by the Spark big data computing engine, to generate a to-be-executed job;

[0092] The processing module 640 is configured to perform processing on the to-be-executed job by the Spark big data computing engine, to obtain a processing result corresponding to the target task.

[0093] The processing module 640 is further configured to:

[0094] perform resource scheduling on the to-be-executed job by the Spark big data computing engine;

[0095] acquire a data source required by the to-be-executed job;

[0096] perform processing on the to-be-executed job based on the scheduled resources and the data source, to obtain the processing result corresponding to the target task.

[0097] The merging module 620 is further configured to:

[0098] acquire the processing programs of the plurality of parallel business modules through a pipe function in the SQL statement;

[0099] configure a running environment corresponding to the processing programs;

[0100] perform merging programming on the processing programs based on the running environment corresponding to the processing programs, to obtain the SQL script.

[0101] The apparatus 600 is further configured to:

[0102] sending a file acquisition request to a local distributed file system of the Spark big data computing engine;

[0103] acquiring, according to the file acquisition request, the shared files and the program package corresponding to the plurality of concurrent business modules from the local distributed file system.

[0104] The processing module 640 is further configured to:

[0105] acquiring a physical execution plan corresponding to the processing logic in the to-be-executed job;

[0106] acquiring, based on the program package, dependent files of the plurality of concurrent business modules and metadata files of a data source;

[0107] determining a dependency relationship of the physical execution plan based on the dependent files;

[0108] reading the data source based on the metadata files and performing processing on the data source according to the dependency relationship of the physical execution plan, to obtain a processing result corresponding to the target task.

[0109] The processing module 640 is further configured to:

[0110] sending a data source acquisition request to an external distributed file storage system of the Spark big data computing engine;

[0111] acquiring the data source from the external distributed file storage system.

[0112] The processing module 640 is further configured to:

[0113] sending a data source acquisition request to a local distributed file storage system of the Spark big data computing engine;

[0114] acquiring the data source from the local distributed file storage system.

[0115] The processing module 640 is further configured to:

[0116] acquiring, based on the shared files, indication information of the data source;

[0117] generating the data source acquisition request according to the indication information of the data source.

[0118] It should be noted that the above explanation and description of the processing method for the target task based on Spark also applies to the processing apparatus for the target task based on Spark of the present embodiment, and the specific process will not be repeated here.

[0119] In summary, the processing apparatus for a target task based on Spark in the embodiment of the disclosure obtains the respective processing programs of the plurality of parallel business modules of the target task, and merges and programs the processing programs of the plurality of parallel business modules based on a structured query language (SQL) statement to obtain a SQL script, sends the SQL script to a Spark big data computing engine, parses the SQL script by the Spark big data computing engine to generate a to-be-executed job, and processes the to-be-executed job by the Spark big data computing engine to obtain a processing result corresponding to the target task. The disclosure improves the ability of the Spark big data computing engine to process jobs submitted in the form of a SQL statement by extending the function of the SQL statement and adding an extended SQL statement parsing manner to the Spark big data computing engine, reduces the cluster resource overhead and the concurrent access pressure of the data source, and improves the utilization rate of the computing resource.

[0120] According to embodiments of the disclosure, the disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0121] Figure 7 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptops, desktops, tablets, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present disclosure described and / or claimed in this document.

[0122] As shown in Figure 7 The device 700 includes a computing unit 701 that can perform various appropriate actions and processes in accordance with a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 707 into a random access memory (RAM) 703. Various programs and data required for the operation of the device 700 can also be stored in the RAM 703. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other through a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0123] A plurality of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0124] The computing unit 701 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 701 performs various methods and processes described above, such as the Spark-based target task processing method. For example, in some embodiments, the Spark-based target task processing method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded onto the RAM 703 and executed by the computing unit 701, one or more steps of the Spark-based target task processing method described above can be performed. Alternatively, in other embodiments, the computing unit 701 can be configured to perform the Spark-based target task processing method by other any appropriate means, such as by means of firmware.

[0125] The various implementations of the systems and techniques described above herein can be realized in a digital electronic circuit system, an integrated circuit system, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip system (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0126] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0127] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0128] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0129] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0130] The computer system can include clients and servers. The clients and the servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The servers can be cloud servers, servers of a distributed system, or servers combined with a blockchain.

[0131] The present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the processing method for a target task based on Spark as described above.

[0132] It should be understood that various forms of flow shown above can be used with orders of steps re-ordered, added to, or removed. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions disclosed in the present disclosure are achieved, and the present disclosure is not limited herein.

[0133] The specific implementation described above does not constitute a limitation on the protection scope of the present disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements within the spirit and principles of the present disclosure should be included in the protection scope of the present disclosure.

Claims

1. A Spark-based processing method of a target task, comprising: obtaining respective processing programs of a plurality of parallel business modules of the target task; merging programming of the processing programs of the plurality of parallel business modules based on a structured query language (SQL) statement to obtain an SQL script; sending the SQL script to a Spark big data computing engine, and parsing the SQL script by the Spark big data computing engine to generate a to-be-executed job; scheduling resources of the to-be-executed job by the Spark big data computing engine; obtaining data sources required by the to-be-executed job; processing the to-be-executed job based on the scheduled resources and the data sources to obtain a processing result corresponding to the target task; the method further comprises: sending a file obtaining request to a local distributed file system of the Spark big data computing engine; obtaining shared files and program packages corresponding to the plurality of parallel business modules from the local distributed file system according to the file obtaining request; the processing the to-be-executed job based on the scheduled resources and the data sources to obtain the processing result corresponding to the target task comprises: obtaining a physical execution plan corresponding to processing logic in the to-be-executed job; obtaining dependent files of the plurality of parallel business modules and metadata files of the data sources based on the program packages; determining a dependency relationship of the physical execution plan based on the dependent files; reading the data sources based on the metadata files, and processing the data sources according to the physical execution plan based on the dependency relationship to obtain the processing result corresponding to the target task.

2. The treatment method of claim 1, wherein, the merging programming of the processing programs of the plurality of parallel business modules based on the SQL statement to obtain the SQL script comprises: obtaining the processing programs of the plurality of parallel business modules through a pipe function in the SQL statement; configuring a running environment corresponding to the processing programs; merging programming of the processing programs based on the running environment corresponding to the processing programs to obtain the SQL script.

3. The treatment method of claim 1, wherein, the obtaining the data sources required by the to-be-executed job comprises: sending a data source obtaining request to an external distributed file storage system of the Spark big data computing engine; obtaining the data sources from the external distributed file storage system.

4. The treatment method of claim 1, wherein, the obtaining the data sources required by the to-be-executed job comprises: sending a data source obtaining request to a local distributed file storage system of the Spark big data computing engine; obtaining the data sources from the local distributed file storage system.

5. The treatment method according to claim 3 or 4, wherein, the obtaining process of the data source obtaining request comprises: obtaining indication information of the data sources based on the shared files; generating the data source obtaining request according to the indication information of the data sources.

6. A Spark-based processing apparatus of a target task, comprising: an obtaining module configured to obtain respective processing programs of a plurality of parallel business modules of the target task; The merging module is configured to perform merging programming on the processing programs of the plurality of concurrent business modules based on a structured query language (SQL) statement, to obtain an SQL script. The parsing module is configured to send the SQL script to a Spark big data computing engine, and perform parsing on the SQL script by the Spark big data computing engine to generate a to-be-executed job. The processing module is configured to perform resource scheduling on the to-be-executed job by the Spark big data computing engine. The processing module is configured to obtain a data source required by the to-be-executed job. The processing module is configured to perform processing on the to-be-executed job based on the scheduled resources and the data source, to obtain a processing result corresponding to the target task. The apparatus is further configured to: send a file acquisition request to a local distributed file system of the Spark big data computing engine; acquire the shared files and program packages corresponding to the plurality of concurrent business modules from the local distributed file system according to the file acquisition request. The processing module is further configured to: obtain a physical execution plan corresponding to a processing logic in the to-be-executed job; obtain dependent files of the plurality of concurrent business modules and metadata files of the data source based on the program packages; determine a dependency relationship of the physical execution plan based on the dependent files; read the data source based on the metadata files, and perform processing on the data source according to the physical execution plan and the dependency relationship, to obtain the processing result corresponding to the target task. 7.The Spark-based target task processing apparatus of claim 6, wherein, The merging module is further configured to: acquire the processing programs of the plurality of concurrent business modules through a pipe function in the SQL statement; configure a running environment corresponding to the processing programs; perform merging programming on the processing programs based on the running environment corresponding to the processing programs, to obtain the SQL script. 8.The Spark-based target task processing apparatus of claim 6, wherein, The processing module is further configured to: send a data source acquisition request to an external distributed file storage system of the Spark big data computing engine; acquire the data source from the external distributed file storage system. 9.The Spark-based target task processing apparatus of claim 6, wherein, The processing module is further configured to: send a data source acquisition request to a local distributed file storage system of the Spark big data computing engine; acquire the data source from the local distributed file storage system.

10. The Spark-based target task processing apparatus according to claim 8 or 9, wherein, The processing module is further configured to: acquire indication information of the data source based on the shared files; generate the data source acquisition request according to the indication information of the data source.

11. An electronic device, comprising: The apparatus includes a processor and a memory; The processor runs a program corresponding to executable program code stored in the memory by reading the executable program code, to implement the method of claims 1-5.

12. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the method of claims 1-5.

13. A computer program product comprising a computer program which, when executed by a processor, implements the method of claims 1-5.

Images