A cloud computing-based big data processing method and system

Abstract

This application discloses a cloud computing-based big data processing method and system. The method involves a server acquiring a job start instruction, which includes at least a job identifier. The server retrieves job metadata corresponding to the job identifier from a pre-set database. Based on the job metadata, the server determines multiple tool metadata for executing the current job and establishes an execution order among these tool metadata. This allows the cloud computing management platform to sequentially call the corresponding tool image files from an image file repository based on the call commands generated from the tool metadata, and run the tool image files to perform phased processing on the data to be processed in the current job until the current job is completed. Each call command corresponds one-to-one with the tool metadata. This approach improves big data processing speed while reducing costs.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a big data processing method and system based on cloud computing. Background Technology

[0002] Data governance is a complex process. The content and quality of the data being processed differ, requiring different processing steps and logic. Faced with massive amounts of data that are rich in content, diverse in format, and inconsistent in quality, numerous processing tools need to be developed to improve efficiency. However, these tools vary in size and complexity. Larger, more complex tools contain more logic and have greater data processing capabilities. However, data is diverse, so no matter how complex a single tool is, it cannot fully meet all data processing needs. Furthermore, large, complex tools lack flexibility and have high maintenance costs. Smaller, simpler tools are relatively more flexible and have lower maintenance costs, but their data processing capabilities are weaker and their application of data is limited.

[0003] Therefore, how to process big data quickly and efficiently while reducing the cost of big data processing has become an urgent technical problem to be solved. Summary of the Invention

[0004] This specification provides a cloud computing-based big data processing method and system platform, equipment, and medium for full-element quality data management, which addresses the following technical problem in the prior art: how to perform big data processing quickly and efficiently while reducing the cost of big data processing.

[0005] The embodiments in this specification adopt the following technical solutions:

[0006] A cloud computing-based big data processing method, the method comprising:

[0007] The server obtains a job start instruction, which includes at least: a job identifier;

[0008] The server retrieves the job metadata corresponding to the job identifier from a preset database based on the job identifier.

[0009] The server determines multiple tool metadata for executing the current job based on the job metadata, and determines the execution order among the multiple tool metadata. This allows the cloud computing management platform to sequentially call the tool image files corresponding to the tool metadata from the image file repository according to the execution order, based on the call commands generated according to the tool metadata, and run the tool image files to perform phased processing on the data to be processed corresponding to the current job until the current job is completed.

[0010] The invocation command corresponds one-to-one with the tool metadata.

[0011] In some embodiments of this application, the method further includes, before obtaining the job start instruction:

[0012] The server receives tool metadata sent by the management client and saves the tool metadata to the corresponding database;

[0013] The server receives the tool source package corresponding to the tool metadata uploaded by the management client, and generates a deployment tool command based on the tool source package. This allows the management client to package the tool source package into a corresponding tool image file and upload the tool image file to the image repository for storage.

[0014] In some embodiments of this application, the method further includes:

[0015] The server generates and stores the association between the tool's metadata and the tool's image file.

[0016] In some embodiments of this application, obtaining the job start instruction specifically includes:

[0017] The server generates the job start instruction based on the data storage path data sent by the external system; or

[0018] The server generates the job start instruction according to a preset time interval;

[0019] The data storage path is generated by the external system when it detects that a corresponding storage device stores data to be processed, based on the storage path of the data to be processed.

[0020] In some embodiments of this application, before retrieving the job metadata corresponding to the job identifier from a preset database based on the job identifier, the method further includes:

[0021] The server receives the job created by the management client based on the user's second operation, as well as the job metadata corresponding to the job;

[0022] The server determines and stores the association between the job metadata and the job, and stores the job metadata in the corresponding database;

[0023] The job metadata includes at least: a job identifier, wherein the job is composed of multiple tools connected in sequence; and the job metadata corresponds one-to-one with the job.

[0024] In some embodiments of this application, retrieving job metadata corresponding to the job identifier from a preset database based on the job identifier specifically includes:

[0025] The server determines whether job metadata corresponding to the job identifier exists in the preset database;

[0026] If no job metadata corresponding to the job identifier exists, the server sends a prompt message to the management client;

[0027] The prompt message is used to prompt the administrator to create a database corresponding to the job identifier;

[0028] The management client receives job metadata corresponding to the job identifier based on the prompt information and stores it in the corresponding database.

[0029] In some embodiments of this application, the method further includes:

[0030] After receiving the job creation command, the management client displays the pre-stored tool identifiers to the administrator and determines the tool identifier selected by the administrator based on the administrator's actions;

[0031] The management client creates jobs based on the selected tool identifier and the execution order of each tool;

[0032] The management client determines the job metadata corresponding to the job based on the user's corresponding operation and stores it in the corresponding database.

[0033] In some embodiments of this application, the step of processing the data to be processed corresponding to the current job to obtain the stage processing result specifically includes:

[0034] The K8S-MASTER service in the cloud computing management platform sends a scheduling command generated based on the received call command to the K8S-NODE service in the cloud data management platform.

[0035] The K8S-NODE service in the cloud data management platform pulls the tool image file corresponding to the tool metadata from the image repository according to the tool metadata in the scheduling command, and runs the tool image file in the form of a container to perform staged processing on the data to be processed.

[0036] In some embodiments of this application, the method further includes:

[0037] The current job's running status is acquired in real time and stored in the corresponding database, so that the management client can view the job's running status through the database;

[0038] The job operation status includes at least: startup, tool identifier of the running tool, normal termination, and abnormal termination.

[0039] A cloud computing-based big data processing system, the system comprising: a job module; the job module comprising:

[0040] A job initiation management unit is used to obtain a job initiation instruction, wherein the job initiation instruction includes at least: a job identifier;

[0041] The job operation management unit is used to retrieve job metadata corresponding to the job identifier from a preset database based on the job identifier; and

[0042] It is also used to determine multiple tool metadata for executing the current job based on the job metadata, and to determine the execution order among the multiple tool metadata, so that the cloud computing management platform, in accordance with the execution order, sequentially calls the tool image file corresponding to the tool metadata from the image file repository based on the call command generated according to the tool metadata, and runs the tool image file to perform stage processing on the data to be processed corresponding to the current job until the current job processing is completed;

[0043] The invocation command corresponds one-to-one with the tool metadata.

[0044] The at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects: By connecting multiple small tools in the form of jobs to participate in data governance, existing tool resources can be fully utilized, redundant development can be avoided, and manpower and material resources can be saved. Furthermore, any combination of tools can form a variety of comprehensive capabilities, which are powerful and can quickly adapt to various data governance requirements without the need to develop new programs, thus possessing high economic value. Moreover, by scheduling the operation of each tool in the job through a cloud computing management platform (i.e., a cloud computing engine), large-scale computing, storage, and network resources can be managed, forming large-scale parallel computing big data processing capabilities. This provides strong fault tolerance and error recovery capabilities, making data processing more stable. Attached Figure Description

[0045] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0046] Figure 1 A flowchart illustrating a cloud computing-based big data processing method provided in an embodiment of this application;

[0047] Figure 2Another flowchart of a cloud computing-based big data processing method provided in an embodiment of this application;

[0048] Figure 3 A schematic diagram of a cloud computing-based big data processing method provided in an embodiment of this application;

[0049] Figure 4 Another flowchart of a cloud computing-based big data processing method provided in the embodiments of this application;

[0050] Figure 5 Another flowchart of a cloud computing-based big data processing method provided in this application embodiment;

[0051] Figure 6 A flowchart illustrating a cloud computing-based big data processing method provided in an embodiment of this application;

[0052] Figure 7 A schematic diagram of a cloud computing-based big data processing system provided in an embodiment of this application;

[0053] Figure 8 A schematic diagram of a cloud computing-based big data processing system provided in an embodiment of this application;

[0054] Figure 9 A schematic diagram of a cloud computing-based big data processing system provided in an embodiment of this application;

[0055] Figure 10 This is a schematic diagram of a cloud computing-based big data processing system provided in an embodiment of this application. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0057] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0058] Figure 1 This is a flowchart illustrating a cloud computing-based big data processing method provided in an embodiment of this application. Figure 1 As shown, the cloud computing-based big data processing method provided in this application embodiment may include the following steps:

[0059] S101, the server receives the job start instruction.

[0060] The job start instruction includes at least the job identifier.

[0061] In some embodiments of this application, the job startup methods include two types: external system service startup and internal system startup. The two different startup methods generate corresponding startup instructions.

[0062] In other words, there are two methods for obtaining job start instructions: one is for the server to generate job start instructions based on data storage path data sent by an external system; the other is for the server to generate job start instructions at preset time intervals.

[0063] The data storage path is generated by the external system when it detects that the corresponding storage device contains data to be processed, based on the storage path of the data to be processed.

[0064] In some embodiments of this application, before executing step S101, i.e., before obtaining the job start instruction, it is also necessary to be able to create and manage tools, such as... Figure 2 As shown, the method provided in this application embodiment further includes:

[0065] The server receives tool metadata sent by the management client and saves the tool metadata to the corresponding database.

[0066] The tool metadata may include: English name, Chinese name, language, version, packaging command, and function description.

[0067] In this embodiment, the tool administrator creates a tool on the management client and fills in the corresponding tool metadata. The management client uploads the tool metadata to the server, and the server's tool management service saves the received tool metadata to the corresponding database.

[0068] The management client sends the tool metadata package, which corresponds to the tool metadata uploaded by the tool administrator, to the server.

[0069] The server receives the tool source package corresponding to the tool metadata uploaded by the management client, generates a deployment tool command based on the tool source package, and sends the deployment command to the management client.

[0070] The management client retrieves the auxiliary package corresponding to the tool's meta-package based on the deployment tool command, packages the tool's source package into a corresponding tool image file, and uploads the tool image file to the corresponding image repository on the server for storage.

[0071] In addition, the server can generate and store the association between tool metadata and tool image files, so that the tool image file corresponding to the tool metadata can be quickly found through this association.

[0072] S102, the server retrieves the job metadata corresponding to the job identifier from the preset database based on the job identifier.

[0073] Before executing step S102, the job needs to be defined, which can be done in the following way:

[0074] First, the server receives the job created by the management client based on the user's second operation, along with the job metadata corresponding to the job.

[0075] The server determines and stores the association between job metadata and the job, and stores the job metadata in the corresponding database.

[0076] The job metadata includes at least: job identifier; a job is composed of multiple tools strung together in sequence; and job metadata corresponds one-to-one with a job.

[0077] In addition to the job identifier, job metadata may also include: execution order, tool name in English, tool image name, input data source type, and input data source information.

[0078] A job is a data processing task divided into several processing stages, each handled by a tool with corresponding functions. From another perspective, a job is a workflow composed of multiple tools connected together. During job execution, the tools are executed sequentially to process the data. For example... Figure 3 As shown, Task 1 is divided into several processing stages, and each processing stage has a corresponding tool.

[0079] By associating the created job with the job metadata using the above method, the server can determine the job to be executed (i.e., the current job) and the job metadata corresponding to the job identifier based on the job identifier in the obtained job start instruction.

[0080] In some embodiments of this application, the server determines whether job metadata corresponding to the job identifier exists in a preset database. If no job metadata corresponding to the job identifier exists, the server sends a prompt message to the management client. The prompt message prompts the administrator to create the job metadata corresponding to the job identifier and store it in the aforementioned preset database; the server also receives the job metadata corresponding to the job identifier sent by the management client based on the prompt message and stores it in the corresponding database.

[0081] Because jobs and their corresponding metadata need to be created in advance, there might be situations where the job identifier in the job startup command does not have corresponding job metadata. Therefore, if the server determines that the job metadata corresponding to the job identifier in the preset database does not exist, it will send a prompt message to the management client. This prompt message may indicate to the administrator that the corresponding job metadata could not be found, prompting the administrator to create the corresponding data source; or it may prompt the administrator to check whether there is an error in the job identifier in the job startup command.

[0082] In some embodiments of this application, after receiving a job creation command, the management client displays a pre-stored tool identifier to the administrator and determines the tool identifier selected by the administrator based on the administrator's operation; wherein, the tool identifier and the management client create a job according to the selected tool identifier and the execution order of each tool; the management client determines the job metadata corresponding to the job based on the user's corresponding operation and stores it in the corresponding database.

[0083] S103, the server determines the execution order of multiple tool metadata for the current job based on the job metadata.

[0084] The above scheme determines the metadata of multiple tools corresponding to the current job and the execution order among the metadata of each tool, that is, it determines the tools included in the current job and the execution order among the tools.

[0085] S104, the cloud computing management platform sequentially calls the tool image file corresponding to the tool metadata from the image file repository based on the call command generated according to the tool metadata, in the execution order.

[0086] S105, the cloud computing platform runs the above tool image file to perform phased processing on the pending data corresponding to the current job until the current job is completed.

[0087] In some embodiments of this application, such as Figure 4 As shown, the processing of the pending data corresponding to the current job yields the stage processing results. Specifically, this includes: the K8S-MASTER service in the cloud computing management platform sends a scheduling command generated based on the received call command to the K8S-NODE service in the cloud data management platform. The K8S-NODE service in the cloud data management platform then pulls the tool image file corresponding to the tool metadata from the image repository based on the tool metadata in the scheduling command, and runs the tool image file as a container to perform stage processing on the pending data.

[0088] In the embodiments of this application, such as Figure 5As shown, the server locates the metadata of the first tool to be executed in the current job based on the job identifier, generates a call command that the K8S-MASTER service can recognize, and sends a scheduling command to the K8S-MASTER service. The call command includes: job identifier, execution order, tool name (English), tool image name, input data source type, input data source information, output data source type, and output data source information.

[0089] The K8S-MASTER service sends scheduling commands to the K8S-NODE service to run tool image files. These scheduling commands include: job identifier, execution order, tool name (English), tool image name, input data source type, input data source information, output data source type, and output data source information.

[0090] The K8S-NODE service, based on the scheduling commands of the K8S-MASTER service, pulls the specified tool image file from the image repository, runs the processing tool image file as a container, forms a job processing service, and passes the job ID, execution order, tool name, tool image name, input data source type, input data source information, output data source type, and output data source information as parameters to the job processing service to complete the current job.

[0091] In some embodiments of this application, a target report can be generated after the current job processing section. The report may include: job identifier, execution order, processing results, etc.

[0092] Connecting multiple small tools through jobs allows for collaborative data governance, fully utilizing existing resources, avoiding redundant development, and saving manpower and resources. Furthermore, arbitrary combinations of tools can create diverse and powerful capabilities, quickly adapting to various data governance requirements without requiring new program development, thus possessing significant economic value. Moreover, scheduling the execution of tools within a job through a cloud computing management platform (i.e., a cloud engine) manages large-scale computing, storage, and network resources, enabling large-scale parallel computing for big data processing. This provides strong fault tolerance and error recovery capabilities, making data processing more stable.

[0093] In some embodiments of this application, such as Figure 6As shown, after each tool in the current job has completed processing the input source data, a corresponding phase report is generated. This phase report includes at least: job identifier, execution order, and processing result. The server can use the job ID and execution order from the phase report to find the metadata of the next tool to be executed from a preset database, thereby retrieving the metadata of the next tool to be executed to obtain the current tool's phase report. The phase report is then stored so that it can be viewed by the management client.

[0094] The above processing results include normal and abnormal. If the processing result is abnormal, the metadata of the next tool to be executed will not be searched, and the job will end.

[0095] In some embodiments of this application, the method provided in this application may further include: the server acquiring the current job's running status in real time and storing it in a corresponding database, so that the management client can view the job's running status through the database.

[0096] The job operation status includes at least: startup, tool identifier of the running tool, normal termination, and abnormal termination.

[0097] The above methods can be used to monitor each operation, ensuring its normal operation.

[0098] In addition to monitoring the job's running status, it can also obtain the input metadata and output metadata for each job. The output metadata includes: regular output data, error output data, log files, and completion reports.

[0099] The management client can view and download the input data, normal output data, error output data, log files, etc. of the job processing.

[0100] Based on the same idea, some embodiments of this application also provide systems corresponding to the above methods.

[0101] Figure 7 This is a schematic diagram of a cloud computing-based big data processing system provided in an embodiment of this application. Figure 7 As shown in the illustration, the cloud-based big data processing system provided in this application includes a security module, a tool module, and a job module. The security module includes a user management unit, a role management unit, and a permission management unit. The tool module includes a tool category management unit and a tool management unit. The job module includes a job definition management unit, a job startup management unit, a job execution management unit, and a job monitoring unit.

[0102] like Figure 8As shown, the tool management module can be implemented through the following devices: PC (i.e., management client), web server, application server, database server, file server, image repository, and other components. Figure 5 As shown, the tool module includes: a tool category management unit and a tool management unit.

[0103] The tool management unit is used to receive tool metadata sent by the management client and save the tool metadata to the corresponding database; it is also used to receive the tool source package corresponding to the tool metadata uploaded by the management client and generate deployment tool commands based on the tool source package; and it is used to upload the tool image file corresponding to the tool source package uploaded by the management client to the image repository for storage.

[0104] The tool category management unit is used to categorize created tools, including operations such as adding, deleting, modifying, and querying tool categories. Nested tool categories are possible, meaning a major tool category can include multiple subcategories. Each tool category includes: Parent Category ID, Category ID, Category Name, and Category Description. This allows for organized and efficient tool management, facilitating easy retrieval.

[0105] like Figure 2 As shown, the tool administrator initiates a command to create a tool from the management client and fills in the tool's metadata. The tool management service saves the tool metadata to the database. The tool administrator uploads the tool's source code or image file from the management client. The tool management service saves the tool's source code to the file server; if the tool type is an image file, the tool management service saves the image file to the image repository. The tool administrator issues a command to deploy the tool from the management client. The tool management service packages the tool's source code uploaded by the tool administrator into an image file according to the packaging command and uploads the image file to the image repository.

[0106] like Figure 5 As shown, the job module includes: a job definition management unit, a job initiation management unit, a job execution management unit, and a job monitoring management unit. Among them, as... Figure 9 As shown, the job definition management unit is implemented through the following devices: PC (i.e., management client), WEB server, application server, database server and other components.

[0107] The job definition management unit receives a job created by the management client based on a second user operation, along with the corresponding job metadata. The server determines and stores the association between the job metadata and the job, and stores the job metadata in a corresponding database. The job metadata includes at least a job identifier, and the job is composed of multiple tools linked together in sequence. Each job metadata corresponds one-to-one with a specific job. This enables the creation of jobs.

[0108] A job initiation management unit is used to obtain a job initiation instruction, wherein the job initiation instruction includes at least: a job identifier;

[0109] The job execution management unit is configured to retrieve job metadata corresponding to the job identifier from a preset database based on the job identifier; and to determine multiple tool metadata for executing the current job based on the job metadata, and to determine the execution order among the multiple tool metadata, so that the cloud computing management platform, in accordance with the execution order, sequentially calls the tool image file corresponding to the tool metadata from the image file repository based on the call command generated according to the tool metadata, and runs the tool image file to perform staged processing on the pending data corresponding to the current job until the current job processing is completed; wherein, the call command corresponds one-to-one with the tool metadata.

[0110] The job monitoring and management unit is used to obtain the job running status of the current job in real time and store it in the corresponding database so that the management client can view the job running status through the database;

[0111] The job operation status includes at least: startup, tool identifier of the running tool, normal termination, and abnormal termination.

[0112] The job monitoring and management unit is also used to obtain periodic reports generated after each tool runs during the job execution process.

[0113] From the perspective of security and division of labor, the big data processing system provided in this application embodiment can be abstractly set up as multiple roles, each role completing different tasks. For example... Figure 10 As shown, the roles include: Super Administrator, User Administrator, Tool Category Administrator, Tool Administrator, Job Administrator, and Job Run Administrator.

[0114] The super administrator is responsible for role management and access control. Role management includes adding, deleting, modifying, and querying roles; access control involves establishing the relationship between roles and functions, and includes adding, deleting, modifying, and querying permissions.

[0115] User administrators are responsible for user management, including adding, deleting, modifying, and querying users.

[0116] The tool category administrator is responsible for the management of tool categories, including adding, deleting, modifying, and querying tool categories.

[0117] The tool administrator is responsible for tool management, including adding, deleting, modifying, and querying tool metadata, as well as uploading and deploying tool files.

[0118] The job administrator is responsible for job creation and management, connecting processing tools to form a job workflow. Specific operations include adding, deleting, modifying, and querying jobs, as well as adding and deleting processing tools for jobs.

[0119] The job execution administrator is responsible for monitoring and executing jobs. Specific operations include: checking job execution status, checking log files, downloading dirty data or failed files, uploading modified data files, re-executing jobs, or processing them.

[0120] like Figure 5 As shown, the security module includes a user management unit, a role management unit, and a permission management unit to manage the roles in this system.

[0121] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0122] The systems and methods provided in this application are one-to-one correspondences. Therefore, the system also has similar beneficial technical effects as its corresponding method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the device and medium will not be repeated here.

[0123] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0124] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0125] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0126] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0127] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0128] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0129] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0130] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0131] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A cloud computing-based big data processing method, characterized in that, The method comprises: The server receives tool metadata sent by the management client and a tool source package corresponding to the tool metadata, saves the tool metadata to a corresponding database, and generates a deployment tool command according to the tool source package, so that the management client packs the tool source package into a corresponding tool image file according to the deployment tool command and then uploads the tool image file to an image warehouse for storage, and meanwhile, an association between the tool metadata and the tool image file is generated and stored; The server generates a job starting instruction according to a data storage path sent by an external system or according to a preset time interval; the data storage path is generated by the external system according to a storage path of to-be-processed data when the external system monitors that a corresponding storage device stores the to-be-processed data; the job starting instruction at least comprises a job identifier; the job is a plurality of tools connected in sequence according to an order; The server receives a job created by the management client based on a second operation of a user and job metadata corresponding to the job, determines an association between the job metadata and the job and stores the association, and stores the job metadata to a corresponding database; the job metadata at least comprises a job identifier; the job metadata and the job one-to-one correspond; The server obtains job metadata corresponding to the job identifier from a preset database according to the job identifier; the server determines whether the preset database stores job metadata corresponding to the job identifier; if not, the server sends prompt information to the management client, the prompt information is used to prompt an administrator to create a database corresponding to the job identifier, receives job metadata corresponding to the job identifier sent by the management client based on the prompt information, and stores the job metadata to a corresponding database; further comprising: the management client shows pre-stored tool identifiers to the administrator after receiving a job creation command, determines tool identifiers selected by the administrator based on an operation of the administrator, creates a job according to the selected tool identifiers and an execution order of the tools, and stores job metadata corresponding to the job to a corresponding database based on a corresponding operation of a user. The server determines a plurality of tool metadata for executing the current job according to the job metadata, and determines an execution order among the plurality of tool metadata, so that the cloud computing management platform invokes tool image files corresponding to the tool metadata from the image file warehouse in sequence according to the invocation commands generated according to the tool metadata according to the execution order, and runs the tool image files to perform stage processing on the to-be-processed data corresponding to the current job until the current job processing ends, including: the K8S-MASTER service in the cloud computing management platform sends a scheduling command generated based on the invocation command to the K8S-NODE service in the cloud data management platform according to the received invocation command, and the K8S-NODE service in the cloud data management platform pulls the tool image file corresponding to the tool metadata from the image warehouse according to the tool metadata in the scheduling command, and runs the tool image file in the form of a container to perform stage processing on the to-be-processed data to obtain stage processing results. The invocation command corresponds to the tool metadata one by one. The method further includes: acquiring a job running state of the current job in real time, and storing to a corresponding database, so that the management client can view the job running state through the database, and the job running state at least includes: starting, tool identifier of a tool being running, normal ending, and abnormal ending.

2. A cloud computing-based big data processing system, characterized by, The system includes: a job module; the job module includes: A job start management unit is configured to receive tool metadata and tool source program packages corresponding to the tool metadata sent by a management client, save the tool metadata to a corresponding database, and generate a deployment tool command according to the tool source program packages, so that the management client packs the tool source program packages into corresponding tool image files according to the deployment tool command and uploads the tool image files to an image warehouse for storage, and at the same time, stores an association relationship between the tool metadata and the tool image files; Further configured to generate a job start instruction according to a data storage path sent by an external system, or according to a preset time interval; the data storage path is generated by the external system according to a to-be-processed data storage path when the external system monitors that a corresponding storage device stores the to-be-processed data, and the job start instruction at least includes a job identifier, and the job is a plurality of tools connected in sequence according to an order; Further configured to receive a job and job metadata corresponding to the job created by the management client based on a second operation of a user, determine an association relationship between the job metadata and the job and store the association relationship, and store the job metadata to a corresponding database; the job metadata at least includes a job identifier, and the job metadata corresponds to the job one by one; Further configured to receive a job and job metadata corresponding to the job created by the management client based on a second operation of a user, determine an association relationship between the job metadata and the job and store the association relationship, and store the job metadata to a corresponding database; the job metadata at least includes a job identifier, and the job metadata corresponds to the job one by one; The job running management unit is configured to obtain job metadata corresponding to the job identifier from a preset database according to the job identifier, including: determining, by the server, whether the job metadata corresponding to the job identifier exists in the preset database, and in the case that the job metadata corresponding to the job identifier does not exist, sending, by the server, prompt information to the management client, the prompt information being used to prompt an administrator to create a database corresponding to the job identifier, receiving, by the server, the job metadata corresponding to the job identifier sent by the management client based on the prompt information, and storing the job metadata to the corresponding database; and further including: after receiving a job creation command, the management client shows the administrator the tool identifiers stored in advance, and determines the tool identifier selected by the administrator based on the administrator's operation, and the management client creates a job according to the selected tool identifier and the execution order of each tool, and the management client determines the job metadata corresponding to the job based on the corresponding operation of the user and stores the job metadata in the corresponding database. The cloud computing management platform is further configured to determine a plurality of tool metadata for executing a current job according to the job metadata, and determine the execution order between the plurality of tool metadata, so that the cloud computing management platform sequentially calls tool image files corresponding to the tool metadata from the image file warehouse based on calling commands generated according to the tool metadata according to the execution order, and runs the tool image files to perform stage processing on the to-be-processed data corresponding to the current job until the current job processing is completed, including: the K8S-MASTER service in the cloud computing management platform sends a scheduling command generated based on the calling command to the K8S-NODE service in the cloud data management platform according to the received calling command, and the K8S-NODE service in the cloud data management platform pulls the tool image file corresponding to the tool metadata from the image warehouse according to the tool metadata in the scheduling command, and runs the tool image file in the form of a container to perform stage processing on the to-be-processed data to obtain stage processing results. The calling command and the tool metadata correspond to each other. The cloud computing management platform is further configured to obtain a job running state of the current job in real time and store the job running state in the corresponding database, so that the management client can view the job running state through the database, and the job running state at least includes: starting, tool identifier of a tool being running, normal ending, and abnormal ending.

Images