Big data framework compiling method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a big data framework compiling method and device, equipment and a storage medium. The method comprises the following steps: establishing a compiling environment mirror image for the big data framework through a container technology to obtain a containerized big data framework compiling environment; receiving a construction task submitted by a user, and distributing the construction task to a corresponding target node in the big data framework compiling environment according to a preset task scheduling rule; wherein the construction task at least comprises a compiling task; and controlling the target node to execute the construction task by using a preset script so as to compile the big data framework to obtain a compilation result. The embodiment of the invention aims at improving the compiling efficiency and accuracy of the big data framework.

Description

Technical Field

[0001] The present application relates to the field of big data technology, is applicable to the financial field, and in particular to a compilation method of a big data framework, a compilation device of a big data framework, a computer device, and a computer-readable storage medium. Background Art

[0002] Spark is a big data framework widely used in the financial field. With the rapid growth of data volume and business scale in the financial field, the demand for Spark R&D iteration continues to increase. Currently, the compilation process of Spark is mainly completed manually. For example, developers need to install various compilation environments and dependencies on their local hosts, and then manually execute compilation commands; or the development team configures a public compilation machine for members, and developers uniformly complete manual compilation on the compilation machine.

[0003] The above compilation method is prone to compilation failures due to large differences in the local environments of developers and the large number of different versions of software packages and libraries that Spark relies on, and the cause of the error is difficult to locate. In addition, since the development team shares a compiler machine, it cannot support multiple R&D personnel compiling at the same time, so the compilation efficiency is low. Summary of the invention

[0004] The present application provides a compilation method for a big data framework, a compilation device for a big data framework, a computer device and a computer-readable storage medium, aiming to improve the compilation efficiency and accuracy of the big data framework.

[0005] To achieve the above purpose, the present application also provides a method for compiling a big data framework, including:

[0006] Building a compilation environment image for the big data framework by using container technology to obtain a containerized big data framework compilation environment;

[0007] Receive a construction task submitted by a user, and assign the construction task to a corresponding target node in the big data framework compilation environment according to a preset task scheduling rule; wherein the construction task at least includes a compilation task;

[0008] The target node is controlled to execute the construction task using a preset script to compile the big data framework to obtain a compilation result.

[0009] To achieve the above purpose, the present application also provides a compilation device for a big data framework, including:

[0010] A compilation environment building module, wherein the compilation environment building module is used to build a compilation environment image for the big data framework through container technology to obtain a containerized big data framework compilation environment;

[0011] A task allocation module, the task allocation module is used to receive the construction task submitted by the user, and allocate the construction task to the corresponding target node in the big data framework compilation environment according to the preset task scheduling rules; wherein the construction task at least includes the compilation task;

[0012] A compilation module is used to control the target node to execute the construction task using a preset script to compile the big data framework to obtain a compilation result.

[0013] In addition, to achieve the above-mentioned purpose, the present application also provides a computer device, which includes a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program and implement the steps of the big data framework compilation method provided in any one of the embodiments of the present application when executing the computer program.

[0014] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor implements the steps of the big data framework compilation method described in any one of the embodiments of the present application.

[0015] The embodiments of the present application disclose a compilation method of a big data framework, a compilation device of a big data framework, a computer device, and a computer-readable storage medium. The method includes building a compilation environment image for the big data framework through container technology to obtain a containerized big data framework compilation environment;

[0016] Receive a construction task submitted by a user, and assign the construction task to a corresponding target node in the big data framework compilation environment according to a preset task scheduling rule; wherein the construction task at least includes a compilation task;

[0017] The target node is controlled to execute the construction task using a preset script to compile the big data framework to obtain a compilation result.

[0018] The present application provides a compilation method of a big data framework, a compilation device of a big data framework, a computer device and a computer-readable storage medium. By using container technology to build a compilation environment image for the big data framework, a containerized big data framework compilation environment is obtained, and a construction task submitted by a user is received, and the construction task is assigned to the corresponding target node in the big data framework compilation environment according to a preset task scheduling rule; wherein the construction task at least includes a compilation task. Thus, the target node can be controlled to execute the construction task using a preset script to compile the big data framework to obtain a compilation result. In the above manner, the compilation environment image is created by container technology, which can ensure the environmental consistency during the compilation process of the big data framework and avoid compilation problems caused by differences in environmental configuration. According to the preset task scheduling rules, the construction task is assigned to the target node in the big data framework compilation environment, so that the compilation task can be efficiently allocated and executed, thereby improving the overall compilation efficiency. Through the containerized compilation environment and the preset script, the target node can be automatically controlled to execute the construction task, reducing the need for manual intervention and reducing human errors in the compilation process. Thus, the compilation efficiency and accuracy of the big data framework can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 It is a scenario diagram of a method for compiling a big data framework provided in an embodiment of the present application;

[0021] Figure 2 It is a flowchart of a method for compiling a big data framework provided in an embodiment of the present application;

[0022] Figure 3 It is a schematic block diagram of a compiling device for a big data framework provided in an embodiment of the present application;

[0023] Figure 4 It is a schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions. In addition, although the functional modules are divided in the device schematic, in some cases, the module division can be different from that in the device schematic.

[0026] The term "and / or" as used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0027] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0028] like Figure 1 As shown, the compilation method of the big data framework provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. The application environment includes a terminal device 110 and a server 120, wherein the terminal device 110 can communicate with the server 120 through a network. Specifically, the server 120 can build a compilation environment image for the big data framework through container technology, obtain a containerized big data framework compilation environment, receive the build task submitted by the user, and assign the build task to the corresponding target node in the big data framework compilation environment according to the preset task scheduling rules; wherein the build task at least includes a compilation task. Finally, the target node can be controlled to execute the build task using a preset script to compile the big data framework to obtain a compilation result, thereby sending the compilation result to the terminal device 110. Among them, the server 120 can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal device 110 can be a smart phone, a tablet computer, a laptop, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this. The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in this application.

[0029] Please continue reading Figure 2 , Figure 2 is a flow chart of a method for compiling a big data framework provided in an embodiment of the present application. Figure 2As shown, the compilation method of the big data framework can be implemented through steps S11 to S13.

[0030] Step S11: Build a compilation environment image for the big data framework through container technology to obtain a containerized big data framework compilation environment.

[0031] Among them, container technology is a lightweight and convenient virtualization technology that aims to provide a lightweight isolation environment so that applications and their dependencies can run seamlessly in different computing environments. It can package applications and their dependencies into a portable container so that applications can run consistently in any environment that supports container technology, without relying on the specific configuration of the underlying operating system. Container technology can ensure the consistency and portability of applications in development, testing and production environments, and container technology supports rapid expansion, and can easily deploy, upgrade and reduce application instances, thereby achieving flexible resource management.

[0032] For example, a base image can be selected, for example, the base image is a general operating system image (such as Ubuntu) or an image specifically used for development. Thus, all tools and dependencies required for the compilation of the big data framework can be installed in the base image. Furthermore, environment variables, paths, and other necessary configurations can be set in the base image so that the compilation process can run smoothly, thereby achieving the construction of a compilation environment image for the big data framework through container technology, and obtaining a containerized big data framework compilation environment.

[0033] For example, big data frameworks include Spark, which is an open source distributed computing system that can be used to process large-scale data. Spark reduces disk I / O operations by storing data sets in memory and can perform many data operations in memory, thereby significantly speeding up processing. It is suitable for various application scenarios from batch processing to real-time processing.

[0034] Step S12: receiving the construction task submitted by the user, and assigning the construction task to the corresponding target node in the big data framework compilation environment according to the preset task scheduling rules.

[0035] For example, a build task generally refers to a task submitted by a developer during the development process to convert source code into an executable file, a library file, or other deployment artifacts. The build task at least includes a compile task, which is the process of converting source code into a binary file.

[0036] Optionally, the task scheduling rules include but are not limited to time triggering rules and code submission triggering rules.

[0037] For example, task scheduling rules refer to a series of preset logic and conditions used to determine when, how, and on which nodes a task is executed after a build task is received. These rules are intended to optimize resource utilization, shorten build time, and improve overall system performance.

[0038] It should be noted that time-triggered rules mean that the execution of tasks is triggered based on time conditions, which may include scheduled tasks (such as executing a build at midnight every day) or triggered according to specific time intervals (such as executing a build once an hour). Time-triggered rules are suitable for projects that need to be built and released regularly to ensure that the build task is automatically executed within the specified time.

[0039] Code submission trigger rules refer to automatically triggering a build task whenever a developer submits new code to the code repository. They are usually integrated with version control systems (such as Git) to execute builds instantly when the code changes. They are suitable for agile development environments or CI / CD environments to ensure that the latest build is automatically generated after each code change, and potential compilation or testing problems are discovered and fixed in a timely manner.

[0040] The above method starts with the build task submitted by the user and intelligently allocates the task to the appropriate computing node for execution through a series of preset scheduling rules, ensuring efficient use of system resources and timely processing of build tasks.

[0041] Step S13: Control the target node to execute the construction task using the preset script to compile the big data framework to obtain the compilation result.

[0042] For example, the target node is a computing resource or machine in the big data framework environment, which is responsible for actually executing the assigned build task. Each target node may be an independent server, virtual machine, or containerized instance. The target node can be controlled by remote call or script to execute the build task using a preset script to compile the big data framework and obtain the compilation result.

[0043] For example, the preset script refers to an automated script executed on the target node, which is used to execute specific operation steps in the build task. The preset script may include a Jenkins Pipeline script, for example.

[0044] It should be noted that Jenkins Pipeline is a code-based continuous integration and continuous delivery (CI / CD) tool that allows the entire process of building, testing, and deployment to be defined and automated through code. Through the Jenkins Pipeline script, the entire build process can be automated, reducing human intervention and improving efficiency and consistency.

[0045] For example, when the target node receives the build task, the preset script will be executed on the target node. This process usually includes the following steps: prepare the compilation environment and load the necessary dependencies; run the compilation command to convert the source code into binary files or other output formats; run unit tests, integration tests, etc. to ensure the correctness and stability of the code; if the compilation and testing are successful, the output files are packaged and may be deployed to a specific environment.

[0046] For example, in a big data framework environment, compilation tasks may include compiling core framework code, customized modules or plug-ins, or other related services and tools. The compilation process may involve processing a large number of source files and generating binary files or libraries required for distributed computing. These files will be distributed among cluster nodes to process large-scale data tasks. The ultimate goal of compilation is to generate executable binary files, library files, Docker images, etc., which will be used in subsequent production environments or test environments.

[0047] For example, after the compilation is completed, the results will be fed back to the scheduling system or user. If the compilation is successful, the system will continue to execute the subsequent deployment or release steps; if it fails, the system will notify the relevant personnel to handle it.

[0048] The above method is implemented through automated scripts to ensure the efficiency, consistency and repeatability of compilation.

[0049] The present application provides a compilation method for a big data framework, which builds a compilation environment image for the big data framework through container technology, obtains a containerized big data framework compilation environment, receives a construction task submitted by a user, and assigns the construction task to the corresponding target node in the big data framework compilation environment according to a preset task scheduling rule; wherein the construction task at least includes a compilation task. Thus, the target node can be controlled to execute the construction task using a preset script to compile the big data framework to obtain a compilation result. In the above manner, the compilation environment image is created through container technology, which can ensure the environmental consistency during the compilation process of the big data framework and avoid compilation problems caused by differences in environmental configuration. According to the preset task scheduling rules, the construction task is assigned to the target node in the big data framework compilation environment, so that the compilation task can be efficiently assigned and executed, thereby improving the overall compilation efficiency. Through the containerized compilation environment and the preset script, the target node can be automatically controlled to execute the construction task, reducing the need for manual intervention and reducing human errors in the compilation process. Thus, the compilation efficiency and accuracy of the big data framework can be improved.

[0050] Optionally, a compilation environment image is built for the big data framework through container technology to obtain a containerized big data framework compilation environment, including: determining a basic image, wherein the basic image includes at least a preset compilation tool; setting environment variables in the basic image to build a compilation environment image for the data framework to obtain a big data framework compilation environment; wherein the environment variables include at least location information of the preset compilation.

[0051] The base image is the starting point of the container, which includes the operating system and some basic tools and libraries. For the compilation environment of the big data framework, the base image needs to include at least the preset compilation tools, such as compilers such as GCC and Clang. Environment variables control the behavior of the compilation process in the containerized compilation environment and indicate the location of tools and libraries.

[0052] For example, after determining the base image and setting the environment variables, you can start building a specific big data framework compilation environment image. For example, it includes adding dependencies, that is, installing the dependent libraries and tools required for the big data framework compilation into the image; adding source code, that is, copying the source code of the big data framework to the specified working directory in the container for compilation; compiling and building, that is, executing the compilation command of the big data framework to generate the final executable file or library. Finally, through the above steps, a containerized big data framework compilation environment image can be generated. The environment image not only contains compilation tools and environment variables, but also contains the compiled results, so that it can be reused in different environments to compile and deploy the big data framework at any time.

[0053] The above method uses container technology to build a compilation environment image for the big data framework, which can create a unified and isolated compilation environment to ensure consistency and repeatability. Determining the base image, setting the necessary environment variables, and building the final compilation environment image can provide a reliable foundation for the compilation of the big data framework. This method not only simplifies the management of the compilation environment, but also improves the efficiency of distributed development and deployment.

[0054] Optionally, controlling the target node to execute the construction task using a preset script also includes: obtaining the code number of the code repository corresponding to the big data framework, and downloading the code number to the target node; controlling the target node that has downloaded the code number to execute the construction task using a preset script.

[0055] The code number usually refers to a unique identifier of a specific submission or version in the code repository. The code number represents the code status at a certain point in time in the code repository. It can ensure that the obtained code is accurate and repeatable.

[0056] For example, before the build task starts, you need to obtain the current version number of the code that needs to be built from the code repository. Once the code number is obtained, the next step is to download the version of the code from the code repository to the target node. This process ensures that the code running on the target node is the specified version of the code, avoiding build failures or other problems caused by code inconsistencies. After the code is downloaded, the target node can execute the build task. At this time, the preset script will start and perform a series of compilation, testing, packaging and other operations to ensure that the final generated build product corresponds to the specified code number.

[0057] In the above method, by downloading the code of the specified code number on the target node and executing the preset script for construction, it can ensure that the construction task is based on the determined code version, ensuring the stability and consistency of the construction results. This process realizes the automation, repeatability and traceability of the construction task through the combination of version control system and automation script, greatly improving the development and deployment efficiency of large-scale distributed systems.

[0058] Optionally, controlling the target node to execute the build task using a preset script includes: obtaining the version number of the big data framework, and controlling the target node to execute the build task according to the version number; after obtaining the compilation result, it also includes: pushing the compilation result to the product library corresponding to the big data framework.

[0059] The version number is usually used to identify the specific release status of a software project. It is usually defined by the development team and complies with certain version naming rules, such as semantic versioning, such as 1.0.0, 2.1.3, etc. In addition, the artifact library usually provides version control functions, allowing you to view, download, or delete artifacts of a certain version, making the entire build process more controllable.

[0060] For example, after obtaining the version number, the target node (usually a build server or compile node) can be controlled to perform specific build tasks according to the version number. This process is usually automated through scripts, including the following steps: setting up the target node environment, ensuring that the target node has the necessary compilation tools and environment variable settings, usually through `ssh` or similar remote command tools to configure the node environment; pulling code, pulling the corresponding version of the code on the target node, ensuring that the source code required for the build task is the latest or specified version; executing the build command, executing the build command on the target node, usually calling the compilation tool chain (such as Maven, Gradle, Make, etc.) to compile, package and generate products; specifying the product library address, defining the address and access credentials of the product library in the script to ensure that the products can be uploaded correctly.

[0061] For example, after the build task is completed, the compilation results usually include binary files, library files or other deployable artifacts. In order to facilitate distribution and use, these artifacts need to be pushed to an artifact repository (ArtifactRepository), such as Nexus, Artifactory, etc. The push process includes: uploading artifacts, uploading the artifacts generated by the build to the artifact repository through preset upload commands or APIs. During the upload process, it is usually necessary to specify the version number and storage path of the artifact; confirm the upload result, the last step is to confirm that the artifact has been successfully uploaded, which can be verified by checking the corresponding path in the artifact repository or through the API.

[0062] For example, in order to ensure the traceability of the entire process and the convenience of troubleshooting, build and upload logs are usually recorded, and a feedback mechanism (such as email notification or Webhook) is provided to inform the build status.

[0063] For example, after the compilation results are pushed to the artifact library, these artifacts can be managed according to the version number. For example, the build products of a specific version can be retrieved, rolled back, or released at any time. Therefore, container technology and automated scripts can be used to efficiently control the target node to execute the build task of the big data framework and push the compilation results to the artifact library. This not only improves the build efficiency, but also ensures the rigor of version management and the standardization of artifact distribution.

[0064] The above method ensures the accuracy and repeatability of the build process by obtaining the version number of the big data framework and executing the build task based on the version number. After obtaining the compilation result, it is pushed to the artifact library to further ensure the orderliness of version management and the traceability of artifacts. This process not only improves the automation level of software development and release, but also improves the stability and maintainability of the entire system.

[0065] Optionally, controlling the target node to execute the build task using a preset script also includes: obtaining queue information of the task cache queue, and determining whether there is a code compilation task in the task cache queue based on the queue information; when there is no code compilation task in the task cache queue, controlling the target node to execute the build task using the preset script.

[0066] Optionally, after determining whether there is a code compilation task in the task cache queue according to the queue information, the method further includes: when there is a code compilation task in the task cache queue, adding the construction task to the task cache queue.

[0067] The task cache queue is a queue that temporarily stores tasks and is used to manage build tasks that need to be executed. When the resources in the system (such as the computing resources of the target node) are limited, the task cache queue can be used to schedule the execution order of tasks.

[0068] Queue information includes the task list in the queue, task priority, task execution status, etc.

[0069] For example, before executing a new build task, the status of the task cache queue is queried first. By reading the queue information, you can understand the tasks currently waiting to be executed.

[0070] For example, if it is confirmed that there are no other code compilation tasks in the task cache queue, the current build task can be assigned to the target node for execution. In this way, the rational use of resources can be ensured, and multiple tasks can be prevented from competing for the same resources at the same time, thereby improving the efficiency and stability of the system.

[0071] For example, if it is confirmed that there is a pending code compilation task in the task cache queue, the system will not immediately execute the new compilation task, but add it to the task cache queue. In this way, the system can be prevented from being overloaded and all compilation tasks can be executed in order, thereby improving the overall performance of the system and the orderliness of task processing.

[0072] In the above method, by introducing the task cache queue and the judgment of queue information, the system can more intelligently manage and schedule the execution of build tasks to avoid resource conflicts and task overload. Through this process, the system can effectively improve the stability and efficiency of task execution, ensure the rational use of resources, and manage tasks in an orderly manner according to task priority or other scheduling rules.

[0073] See also Figure 3 , Figure 3 1 is a schematic block diagram of a big data framework compilation device provided in an embodiment of the present application. The big data framework compilation device can be configured in a server to execute the aforementioned big data framework compilation method.

[0074] like Figure 3 As shown, the compilation device 200 of the big data framework includes: a compilation environment building module 201, a task allocation module 202 and a compilation module 203.

[0075] A compilation environment building module 201 is used to build a compilation environment image for the big data framework through container technology to obtain a containerized big data framework compilation environment;

[0076] The task allocation module 202 is used to receive the construction task submitted by the user, and allocate the construction task to the corresponding target node in the big data framework compilation environment according to the preset task scheduling rules; wherein the construction task at least includes the compilation task;

[0077] The compilation module 203 is used to control the target node to execute the construction task using a preset script to compile the big data framework to obtain a compilation result.

[0078] The compilation environment building module 201 is also used to determine a basic image, wherein the basic image includes at least a preset compilation tool; setting environment variables in the basic image to build the compilation environment image for the data framework to obtain the big data framework compilation environment; wherein the environment variables include at least the location information of the preset compilation.

[0079] The compilation module 203 is also used to obtain the code number of the code warehouse corresponding to the big data framework, and download the code number to the target node; control the target node downloaded with the code number to use the preset script to execute the construction task.

[0080] The compilation module 203 is also used to obtain the version number of the big data framework, and control the target node to execute the construction task according to the version number; and push the compilation result to the product library corresponding to the big data framework.

[0081] The compilation module 203 is also used to obtain queue information of the task cache queue, and determine whether there is a code compilation task in the task cache queue according to the queue information; when there is no code compilation task in the task cache queue, control the target node to execute the construction task using the preset script.

[0082] The compiling module 203 is further configured to add the building task to the task cache queue when the code compiling task exists in the task cache queue.

[0083] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0084] The method and apparatus of the present application can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer terminal devices, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices.

[0085] Exemplarily, the above method and apparatus may be implemented in the form of a computer program. The computer program may be implemented in the form of a computer program. Figure 4 Runs on the computer device shown.

[0086] See also Figure 4 , Figure 4 1 is a schematic diagram of a computer device provided in an embodiment of the present application. The computer device may be a server.

[0087] like Figure 4 As shown, the computer device includes a processor, a memory and a network interface connected through a system bus, wherein the memory may include a volatile storage medium, a non-volatile storage medium and an internal memory.

[0088] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any one of the compilation methods of the big data framework.

[0089] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0090] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any compilation method of the big data framework.

[0091] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that the structure of the computer device is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0092] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0093] Among them, in some embodiments, the processor is used to build a compilation environment image for the big data framework through container technology to obtain a containerized big data framework compilation environment; receive a build task submitted by a user, and assign the build task to a corresponding target node in the big data framework compilation environment according to a preset task scheduling rule; wherein the build task includes at least a compilation task; control the target node to execute the build task using a preset script to compile the big data framework to obtain a compilation result.

[0094] In some embodiments, the processor is also used to determine a base image, wherein the base image includes at least a preset compilation tool; setting environment variables in the base image to build the compilation environment image for the data framework to obtain the big data framework compilation environment; wherein the environment variables include at least the location information of the preset compilation.

[0095] In some embodiments, the processor is further used to obtain the code number of the code repository corresponding to the big data framework, and download the code number to the target node; control the target node downloaded with the code number to execute the construction task using the preset script.

[0096] In some implementations, the processor is further used to obtain a version number of the big data framework, and control the target node to execute the build task according to the version number; and push the compilation result to a product library corresponding to the big data framework.

[0097] In some embodiments, the processor is also used to obtain queue information of the task cache queue, and determine whether there is a code compilation task in the task cache queue based on the queue information; when there is no code compilation task in the task cache queue, control the target node to execute the construction task using the preset script.

[0098] In some implementations, the processor is further configured to add the construction task to the task cache queue when the code compilation task exists in the task cache queue.

[0099] An embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed, a compilation method of any big data framework provided in the embodiment of the present application is implemented.

[0100] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.

[0101] Furthermore, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application program required for at least one function, and the like.

[0102] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for compiling a big data framework, characterized in that: The method comprises: Building a compilation environment image for the big data framework by using container technology to obtain a containerized big data framework compilation environment; Receive a construction task submitted by a user, and assign the construction task to a corresponding target node in the big data framework compilation environment according to a preset task scheduling rule; wherein the construction task at least includes a compilation task; The target node is controlled to execute the construction task using a preset script to compile the big data framework to obtain a compilation result.

2. The method according to claim 1, characterized in that The method of building a compilation environment image for the big data framework by using container technology to obtain a containerized big data framework compilation environment includes: Determine a base image, wherein the base image includes at least a preset compilation tool; Environment variables are set in the basic image to build the compilation environment image for the data framework to obtain the big data framework compilation environment; wherein the environment variables at least include the location information of the preset compilation.

3. The method according to claim 1, characterized in that The controlling the target node to execute the construction task using a preset script also includes: Obtaining a code number of a code warehouse corresponding to the big data framework, and downloading the code number to the target node; The target node downloaded with the code number is controlled to execute the construction task using the preset script.

4. The method according to claim 1, characterized in that: The controlling the target node to execute the construction task using a preset script includes: Obtaining a version number of the big data framework, and controlling the target node to execute the construction task according to the version number; After obtaining the compilation result, the following steps are also included: The compilation result is pushed to the product library corresponding to the big data framework.

5. The method according to claim 4, characterized in that The controlling the target node to execute the construction task using a preset script also includes: Obtaining queue information of a task cache queue, and determining whether there is a code compilation task in the task cache queue according to the queue information; When there is no code compilation task in the task cache queue, the target node is controlled to execute the construction task using the preset script.

6. The method according to claim 5, characterized in that After determining whether there is a code compilation task in the task cache queue according to the queue information, the method further includes: When the code compiling task exists in the task cache queue, the building task is added to the task cache queue.

7. The method according to claim 1, characterized in that The task scheduling rules include but are not limited to time triggering rules and code submission triggering rules.

8. A compiling device for a big data framework, characterized in that: The compiling device comprises: A compilation environment building module, wherein the compilation environment building module is used to build a compilation environment image for the big data framework through container technology to obtain a containerized big data framework compilation environment; A task allocation module, the task allocation module is used to receive the construction task submitted by the user, and allocate the construction task to the corresponding target node in the big data framework compilation environment according to the preset task scheduling rules; wherein the construction task at least includes the compilation task; A compilation module is used to control the target node to execute the construction task using a preset script to compile the big data framework to obtain a compilation result.

9. A computer device, characterized in that: include: A memory and a processor; wherein the memory is connected to the processor and is used to store a program; the processor is used to implement the steps of the compiling method of the big data framework as described in any one of claims 1 to 7 by running the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, enables the processor to implement the steps of the method for compiling a large data framework as described in any one of claims 1 to 7.

Images