Big Data Application Lifecycle Management

Abstract

The present disclosure relates to systems, methods, devices, and other aspects for creating an application lifecycle management platform for big data applications. In one embodiment, the lifecycle management platform may include a multi-layer container file that integrates multiple big data tools / platforms. The system may create a generic template application, create a build environment for the generic template application, create a test environment for the generic template application, and run the constructed generic template application in the test environment before the user writes any new code in the generic template application. In one embodiment, the test environment includes a container management system or a virtual machine that starts a big data application on a separate big data server cluster, which may be a generic template application before a developer edits a file.

Description

[0001] Cross-cutting areas of related applications

[0002] This application is a continuation of and claims priority to U.S. Patent Application No. 16,440,827, filed on June 13, 2019, which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates generally to big data application development, and more particularly, to big data application lifecycle management. Background Art

[0004] Application lifecycle management is the technology used to control applications from the conception stage through development, deployment, upgrades, and removal from service. Tools used to manage the application lifecycle enable deployment, troubleshooting, management governance, and testing of applications in a structured approach.

[0005] Digital devices and services generate trillions, petabytes, and exabytes of data every day. Big data is the field of analyzing large amounts of data that are too large for traditional software to analyze. The data can include archived documents, files, and websites, media files (images, videos, audio files), data storage systems (repositories, databases, file systems), business applications, public and government data sources, social media activity, machine log data, and sensor data (from media devices, IoT devices, appliances, farm equipment, vehicle sensors). This data can be analyzed or mined to perform predictions, analysis, modeling, and training for artificial intelligence. However, the volume, quality, accuracy, and velocity of the data received can pose challenges to monitoring and analyzing the data, as can the use of a large number of systems and the large number of applications that need to be completed.

[0006] Due to the hardware and software requirements of the systems that run and store big data applications, tools used to manage the life cycle of non-big data applications cannot be used to manage big data applications. Therefore, tools are needed to manage the life cycle of these applications (including the creation, construction, testing, deployment, and maintenance of big data applications). BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 A block diagram of a big data application lifecycle management system according to one embodiment is shown.

[0008] Figure 2 A block diagram of an exemplary template structure according to one embodiment is shown.

[0009] Figure 3A A flowchart for performing big data application lifecycle management according to one embodiment is shown.

[0010] Figure 3B A flowchart for performing big data application lifecycle management according to one embodiment is shown.

[0011] Figure 4 Shown is a block diagram illustrating a big data application lifecycle management system according to one embodiment.

[0012] Figure 5 An example block diagram of a computer system according to one embodiment is shown. The computer system is suitable for implementing Figure 1-4 One or more devices in a big data application lifecycle management system.

[0013] Embodiments of the present disclosure and their advantages may be best understood by referring to the following detailed description. It should be understood that similar reference numerals are used to identify similar elements shown in one or more of the accompanying drawings, which are shown to illustrate embodiments of the present disclosure, but not to limit them. DETAILED DESCRIPTION

[0014] In the following description, the specific details of some embodiments consistent with the present disclosure are set forth. However, it will be apparent to those skilled in the art that some embodiments may be practiced without some or all of these specific details. The specific embodiments disclosed herein are intended to be illustrative and not restrictive. Those skilled in the art will appreciate that other elements, although not specifically described herein, are within the scope and spirit of the present disclosure. In addition, in order to avoid unnecessary repetition, unless specifically described in addition or if one or more features will make the embodiment inoperative, one or more features shown and described in association with an embodiment may be incorporated into other embodiments.

[0015] Various aspects of the present disclosure relate to systems, methods, and devices for creating an application lifecycle management platform for big data applications. Big data applications include applications in which the data cannot be accommodated on a single computer / server or a single transaction cannot be processed by a single computer. The process of managing the lifecycle of big data applications includes creating, building, testing, deploying, analyzing, and maintaining big data applications. In some systems, lifecycle management has many steps, many of which are manual and require direct contact and communication with many teams / platforms and services.

[0016] The features for managing the application lifecycle are built on multiple development and management platforms. These platforms may include, for example, an orchestration platform for code development, construction, deployment and monitoring. The platform may also include a container management platform, Apache Mesos, Github, Jenkins, Twistlock, an assembler for creating low-level code from a high-level language, a Hadoop ecosystem, a Kerberos security focused network authentication protocol, LIVY and other related RESTful interfaces for submitting and managing Apache Spark jobs on a server cluster, a credential and secret management platform for storing encrypted keys and passwords so that they do not need to be stored on disk, and a GIMEL unified API for accessing data stored in any data storage area and platform to write SQL to access data on any data storage area.

[0017] According to the embodiments of this document, these services can be modified and improved to adapt to big data applications. In one embodiment, a multi-layer container file (e.g., DOCKER image) integrated with the above-mentioned tool / platform can be created. Each layer can add the ability to work with one or more of the platforms. The multi-layer container can allow multiple platform teams to create such common components: each platform team can build these common components for their developers, and these common components can then be placed on the base code. This can allow increased code reusability and standardization between development teams in a single organization. In one embodiment, four layers of containers are described. The first layer describes the base layer, which contains tools, platforms, and configurations for operating environments, networking, and security. The first layer can also include other core software and core libraries, such as python, curl, clib, and many other libraries and software. In some examples, the first layer includes an operating system. The operating system can be a custom operating system for programmers to use and run applications. For example, the operating system can include software and libraries for creating and building programs. The second layer describes the big data layer, which includes features for accessing big data frameworks (e.g., the Apache HADOOP ecosystem, which includes Apache PIG, Apache HIVE, Apache HBASE, Apache PHOENIX, Apache SPARK, Apache ZOOKEEPER, CLOUDERA IMPALA, Apache FLUME, Apache SQOOP, Apache OOZIE, and Apache STORM) and uploading application dependencies to a distributed file system (e.g., the Hadoop Distributed File System (HDFS)). The third layer describes the platform / team layer, which includes features required for each specific platform or team, such as templates for operating parameters and dependencies. The fourth layer describes the application layer, which includes features used or required by applications, such as application-specific dependencies.

[0018] In an embodiment of the present disclosure, a user can instruct the system to create a new big data application for development. The user can be a single developer, a team of developers, or an organization or sub-organization of developers. The system can create a generic template application (in some embodiments, the generic application is specific to a developer group or team), create a build environment for the generic template application, create a test environment for the generic template application, and run the built generic template application in the test environment before the user writes any new code in the generic template application. The advantage of this embodiment is that it can enable developers / development teams to focus on the code of their specific applications without having to worry about the environment, infrastructure complexity, connectivity, etc. Compared with the traditional relational database management system (RDBMS)-based application world, this problem is more difficult in the big data world due to the number of components, infrastructure complexity, and connectivity. In one embodiment, the test environment includes a virtual machine / container application that starts a big data application on a separate big data server cluster (the big data application can be a generic template application before the developer edits the file). In some environments, it may be difficult to create a test or pre-release (stage) environment for data applications (especially big data applications) because the environment needs to be composed of various dependencies, and the API and interaction may be picky even for minor version differences. Embodiments of the present disclosure alleviate this situation by testing common components in a build and test environment before testing the application code to provide a simplified debugging process. In some examples, the virtual machine / container application is started on a cluster management system and the big data application runs on a big data cluster. After building and testing the common template application, a user or user group can update the common template application with application-specific code in a code repository. Updating / saving the application or upgrading the main branch or version of the big data application in the code repository can trigger the build environment to rebuild the big data application. The test environment can then automatically run the new version of the big data application.

[0019] Automated tools for big data application lifecycle management can improve the developer experience of creating and managing applications and can improve operations for viewing and controlling running applications. An automated big data application lifecycle management system can minimize the amount of manual data entry when setting up and building big data applications, create a standardized environment for creating and testing big data applications, and allow scheduling, testing, and management of multiple big data applications. The automated big data application lifecycle management system allows integration with third-party (open source and proprietary) tools and platforms. A big data application lifecycle management platform can scan applications for vulnerabilities, enforce coding standards, and provide monitoring and alerts, among other things.

[0020] Therefore, in one embodiment, a system 100 for performing application lifecycle management is introduced, which is configured to manage the lifecycle of big data applications. Figure 1 A system 100 for performing lifecycle management of a big data application is shown. The lifecycle management platform 102 may include an interface for a user device 104 to connect to the lifecycle management platform 102 directly or via a network 106. The system 100 may also include a build environment 108, a code repository 110, and a configuration and settings server 112. The system 100 may include a test environment 114 and a production environment 116. Each of the test environment 114 and the production environment 116 may include one or more scheduling and management servers 118 and a big data server cluster 120.

[0021] The user device 104 may be implemented using any suitable combination of hardware and / or software configured for wired and / or wireless communication over the network 106. For example, in one embodiment, the user device 104 may include a personal computer device. The personal computer may include a desktop computing system, a laptop / notebook computing system, a tablet computing system, a mobile phone, an application specific integrated circuit (ASIC) computing system, an Internet of Things (IoT) device, and / or other computing devices known in the art. The personal computer may include software (e.g., an application or browser interface) for executing or connecting to a device (e.g., the lifecycle management platform 102) configured to perform big data application lifecycle management.

[0022] User device 104 may include one or more applications (e.g., browser applications) that may be used, for example, to provide an interface to allow a user to browse information available via network 106. For example, in one embodiment, the application may be a web browser configured to view information available via the Internet.

[0023] The lifecycle management platform 102 is configured to manage the lifecycle of the big data application. The lifecycle management platform 102 can be configured to interact with the user device 104. For example, the lifecycle management platform 102 can have an interface for the following items: setting up a new big data application; checking the current state of the big data application, including whether the application is new, whether it has been built, whether it has been tested in a test environment (e.g., test environment 114), or whether the application is running in a production environment (e.g., production environment 116). In addition, the lifecycle management platform 102 can provide an interface for the following items: viewing logs and monitoring server cluster settings and resource usage, governance, access control, and any errors or warnings. The lifecycle management platform 102 can be connected to the code repository 110 to store and manage code.

[0024] In addition, the lifecycle management platform 102 can provide data analysis and integration, enabling users to register their big data applications and deploy them in production. The lifecycle management platform 102 enables the ability to view available data sets, view schemas, and view system and object properties. The lifecycle management platform 102 can automatically discover data sets in all data storage areas. The lifecycle management platform 102 can provide dashboards and warnings including operational metrics, including statistical information, refresh time and trends, visibility of approvals and audits, and administrative warnings (e.g., capacity issues, data access violations, and data classification violations) and user warnings (e.g., refresh delays and profile anomalies). In addition, the lifecycle management platform 102 can include query and integration features designed for integration of notebooks and tools. In order to provide these features, the lifecycle management platform 102 can provide classification, grading, analysis, and mechanisms for running and storing big data.

[0025] The lifecycle management platform 102 may be a platform designed to provide a unified user experience for any computer engine. That is, the lifecycle management platform 102 may be an analysis platform or framework that enables users to run one or more big data applications. For example, the lifecycle management platform 102 may enable users to run big data applications / data processing frameworks (including but not limited to SPARK, HIVE, PRESTO, etc.). Big data applications may be run through software architecture styles or other application programming interfaces (APIs) based on interface protocols. For example, the API may include but is not limited to an API based on representational state transfer (REST), an API based on Thrift, and an API based on a simple object access protocol (SOAP). In addition, the lifecycle management platform 102 may be designed to run without installation, setup, and / or configuration. Therefore, the lifecycle management platform 102 may be used to provide a complete set of tools and techniques for application development, execution, logging, detection, warning, security, workload management, performance tuning, etc. In addition, the lifecycle management platform 102 may be used to support general computing engines (e.g., SPARK) for large-scale data processing and running (one or more) interactive codes, scheduling jobs, and machine learning analysis. For example, interactive SPARK can be used with interactive shells, JUPYTER Notebooks, APACHE ZEPPELIN, and Squirrel / DBVISUALIZER SQL clients. As an example of scheduling jobs, the lifecycle management platform 102 can be used to collaborate with one or more scheduling and management servers 118 to schedule jobs on low-latency applications, batch-heavy applications, and streaming media applications. The advantages of the lifecycle management platform 102 include, but are not limited to, improvements in management (e.g., less maintenance, deployment of software stacks, and the ability to manage system configurations in one place), operations / security (e.g., through the execution of a single job, coding standards, logging, monitoring and alerting, auditing, and complete statement-level history and metrics), development (e.g., through modularity of applications, ease of recoverability, reduced latency, cache sharing, etc.), and analysis (e.g., for direct SQL execution, multi-user support for notebook capabilities, user-friendly interactive applications, and authentication integration).

[0026] The lifecycle management platform 102 can provide a unified access API for any data storage device. In particular, the lifecycle management platform 102 can provide scalable platform services. Such scalable data services can include data integration. For example, the lifecycle management platform 102 can facilitate the orchestration of data acquisition and conversion, and provide reliable and secure means of delivering data to various destinations through streaming or batch processing. The core data platform can also be used as an analytical data processing platform to access data located in big data applications (such as HADOOP) and data located in data and analysis platforms. In addition, the lifecycle management platform 102 can also provide access to data storage devices to perform self-service lifecycle management of a single clustered data storage area and management based on commodity storage devices. In addition, the lifecycle management platform 102 can be used to learn, optimize, build, deploy and run various applications and changes. For example, the data application lifecycle on the lifecycle management platform 102 can include: loading big data applications and managing computing engine changes, computing version changes, storage API changes, storage connector upgrades, storage host migrations and storage device changes.

[0027] The build environment 108 can be configured to convert source code files into independent software programs (project templates (artifacts), binary files, executable files), or files that are ready to be interpreted and can be run on a computer / server cluster. The build environment 108 can be configured to collect relevant software codes for building big data applications and libraries including these codes. The build environment 108 can be an independent server, or use shared resources. The build environment 108 can use automated tools to trigger builds and schedule regular builds, or can build applications based on triggering events, such as a submission item in the code repository 110. In one example, the build environment 108 can use a continuous delivery development tool such as JENKINS.

[0028] The code repository 110 may be configured to provide storage and development tools for big data applications. The code repository 110 may provide developers with collaboration tools, version control, issue tracking, and documentation. The code repository 110 may be directly integrated into the lifecycle management platform 102, or the code repository 110 may be a third-party server / application (e.g., GITHUB using a version control system such as GIT) that can be connected to the lifecycle management platform 102 via a network.

[0029] The configuration and settings server 112 may contain and manage configurations and settings for various server environments, such as the test environment 114 and the production environment 116. The lifecycle management platform 102 may connect to the configuration and settings server 112 and retrieve settings, configuration files, and templates for developing big data applications. The settings may include the address of the resource manager and the address of the stored data.

[0030] The test environment 114 and the production environment 116 are two independent server environments. The test environment 114 and the production environment 116 may have the same or similar settings, however, the data used in the test environment 114 is test data, and these data are not configured to be accessed by end users, while the data used in the production environment 116 is live data, and the production environment 116 will run applications for end users to process operational data. The test environment 114 and the production environment 116 may include one or more scheduling and management servers 118. Each scheduling and management server 118 may be associated with a big data server cluster 120. In another embodiment, the scheduling and management server 118 may be associated with multiple big data server clusters 120.

[0031] The scheduling and management server 118 may be configured to manage big data applications on the big data server cluster 120. In some examples, a virtual machine or container application is created on the scheduling and management server 118 to start the big data application on the big data server cluster 120. The scheduling and management server 118 may provide scheduling of server resources between multiple applications (or multiple versions of the same application) running on the big data server cluster 120. The scheduling and management server 118 may trigger the operation of the big data application after another application completes or fails or when new data is available for processing. The scheduling and management server 118 may also trigger the big data application in other cases. As an example, the scheduling and management server 118 may trigger the second application at the beginning of the first application, regardless of whether the first application ultimately succeeds or fails. As another example, the scheduling and management server 118 may trigger the second application 30 minutes after application B is started. As another example, if the first application fails multiple times in a row (for example, the first application fails three times), the second application may be triggered. The scheduling and management server 118 may trigger the operation of the big data application after a specific time period (daily, hourly, etc.). At the start of the application, the big data application can be throttled by the scheduling and management server 118 to spread it out. The scheduling and management server 118 can collect resource usage metrics and statistics. Recommendations on when to run big data applications can be provided to the lifecycle management platform 102. Other analyses performed by the scheduling and management server include timing to run big data applications (and timing to run the application in relation to other applications), forecasting, risk assessment, resource utilization, etc.

[0032] The big data server cluster 120 may include a group of servers that work together to run one or more applications. The big data server cluster 120 may include a scalable big data storage device (e.g., HDFS) storage pool on multiple commodity computer systems. In some embodiments, dedicated hardware for big data applications is used.

[0033] Figure 2 An exemplary block diagram of a template container used by the lifecycle management platform 102 is shown. The build environment 108 and / or the lifecycle management platform 102 can generate and use a template file or template 200 to organize and automate big data lifecycle management. The template 200 is a multi-layer container file that contains all or most of the settings, tools, and utilities for testing, building, running, managing, and analyzing big data applications. The template 200 can include a four-level container file having a base layer 202, a big data layer 204, a group layer 206, and an application layer 208.

[0034] An organization may have multiple development groups / teams. In some examples, different groups may contribute and develop different layers of template 200. For example, base layer 202 may be developed by a base platform team and a security team. A big data platform team and a big data team may develop big data layer 204. Developers may develop application layer 208. Developers may develop application layer 208 with a source code management team and a build and test team. The group to which the developer belongs may develop group layer 206. The overall template functionality may be overseen by an application lifecycle team, an application repository team, a container management team, a hardware provisioning team, and / or a support team.

[0035] In some examples, template 200 is a DOCKER image / container file. A DOCKER image is a file (which may contain multiple layers) used to execute code in a DOCKER container. A DOCKER image is essentially built from instructions for a complete and executable version of an application, which may depend on the host operating system (OS) kernel. In some examples, template 200 is a DOCKER image having one or more read-only layers and a readable / writable layer (for big data applications) above these read-only layers. These layers (also called intermediate images) may be generated when commands in a container / DOCKER file are executed during container / DOCKER image construction.

[0036] The base layer 202 may include settings, utilities and tools for the operating environment, networking and security of the application. The settings in the base layer 202 may include settings suitable for big data and non-big data applications, including operating system 210 (e.g., LINUX distribution, WINDOWS, APPLE OS, etc.), platform tools and configuration 212 (including low-level network topology settings and server cluster configuration and settings), and platform security tools and configuration 214. The security tools may include connections to security keychain applications to provide access permissions to resources and protect user and application credentials, passwords and other secrets.

[0037] The big data layer 204 may include features for accessing a big data ecosystem (e.g., a HADOOP ecosystem) and uploading dependencies to a big data file system (e.g., HDFS). The big data layer 204 may include a big data configuration and library 216, which may include information about how to connect to one or more big data clusters 120 in a test environment 114 or a production environment 116. The orchestration, monitoring, and alerting client 218 is an interface that can be configured to retrieve information about logging, monitoring, governance, access control, and alerting from the scheduling and management server 118. The launch, termination, update, and recovery client 220 may be an interface that can be configured to launch and manage big data applications running on one or more big data clusters 120. The big data file system client 222 may include a HADOOP HDFS client. The big data file system client 222 may include an interface that is used to communicate with the big data file system and perform file-related tasks (such as reading and writing block data). The job server client 224 (e.g., an APACHE Livy client) may be used to interact with one or more scheduling and management servers 118 to indicate or receive information about job scheduling. The job server client 224 may include an API or REST interface.

[0038] The group layer 206 may include user-specific features, wherein the user includes platform / development team / group-specific features that are specifically needed by a specific team or are useful to that specific team. These features may include templates for run parameters and dependencies. A team / group includes a group of developers who create similar applications with the same endpoints or library dependencies. The group layer 206 may include shared libraries 226. Shared libraries 226 may include shared libraries that will be included in each application developed by a specific team. The group layer 206 may include shared test environment settings 228 and shared production environment settings 230 to interact with a big data ecosystem (e.g., a HADOOP ecosystem) on a big data server cluster 120, because a group may have one or more dedicated data server clusters 120 for testing or production.

[0039] The application layer 208 may include features required by the application, such as application-specific dependencies, tools, or utilities. This includes dependencies that are not typically discovered or used by a specific development team (dependencies that are typically discovered or used by a specific development team may be placed in the group layer 206). The application layer 208 includes user libraries and configurations 232, which include all the code that the user application 234 (to be built) depends on. The user application 234 may initially be a stub or template application. In some examples, the user application 234 may be a specific stub or template file for a team / group. The template file may be able to be built and run in the test environment 114 (or production environment 116).

[0040] Reference now Figure 3A , Figure 3A 350 is a flowchart illustrating the use of the system 100 to manage the lifecycle of a big data application according to one embodiment. The user device 104 can be connected to the lifecycle management platform 102 via the network 106. The lifecycle management platform 102 can present a user interface to the user device 104 to create a new big data application. The user interface can request certain information to generate the new big data application. In one example, the lifecycle management platform 102 requests a team or development group of a big data application to create a big data application.

[0041] At block 352, the lifecycle management platform 102 can create a source code template. The source code template can be specific to a particular development or user group. In some examples, the group can be specified by user input. In other embodiments, the group can be implicitly expressed by the user credentials connected to the lifecycle management platform 102. At block 354, the lifecycle management platform 102 can create a container instance. The container instance can store the source code template.

[0042] At block 356, the lifecycle management platform 102 may set up a build environment 108 for building a lifecycle management application and a test environment 114 for running the big data application on the big data server cluster 120. The scheduling and management server 118 may be configured by the lifecycle management platform 102 for testing the big data application.

[0043] At block 358, the lifecycle management platform 102 can build and test the stub application in the instructing build environment 102. The stub application can include only automatically generated files (based on, for example, a source code template) and not any files modified or created by a user. In some examples, the build creates a self-contained container and / or a set of container combination files and can include all library dependencies for running the stub application.

[0044] If the build and test are successful, the user can modify the big data application through the source code template. Modifying the source code can be done using the code repository 110. When the test of the code is completed (or before the test of the code is completed), the user can submit the code or finalize the code. The process may trigger the source code and dependencies to be transmitted to the lifecycle management platform 102 at block 360. In other examples, the user can manually provide the updated source code to the lifecycle management platform 102. The updated source code can be placed in the container instance by the lifecycle management platform 102. At block 362, the lifecycle management platform 102 can rebuild and retest the newly updated code / application.

[0045] Reference now Figure 3B , Figure 3B 3 is a flow chart 300 illustrating the use of the system 100 to manage the lifecycle of a big data application according to one embodiment. A user device 104 can be connected to a lifecycle management platform 102 via a network 106. The lifecycle management platform 102 can present a user interface to the user device 104 to create a new big data application. The user interface can request certain information to generate the new big data application. In one example, the lifecycle management platform 102 requests a team or development group of a big data application to create a big data application.

[0046] Users and groups can generate template or stub application source code files and configuration files before using the lifecycle management platform 102. The lifecycle management platform 102 can use these pre-generated application source code files and configuration files to generate self-contained container files.

[0047] At block 302, the user device 104 may submit, and the lifecycle management platform 102 may receive, an instruction to create a new application, the instruction specifying a user group. At block 304, the lifecycle management platform 102 may create a source code template specific to the submitted user group. At block 306, the lifecycle management platform 102 may create a container instance (e.g., DOCKERFile). The container instance may store the source code template specific to the user group created at block 304.

[0048] At block 308 , the lifecycle management platform 102 can set up a build environment 108 for building a lifecycle management application. At block 310 , the lifecycle management platform 102 can set up a test environment, wherein a scheduling and management server 118 of the test environment 114 runs a big data application on a big data server cluster 120 .

[0049] The lifecycle management platform 102 can instruct the build environment 102 to build a stub application. In some examples, the build creates a self-contained container (e.g., a DOCKER image) and a set of container (e.g., DOCKER) combination files. In one example, self-contained means: the DOCKER image contains all or substantially all binary files required to run the application. In some examples, the binary file is a java archive (.jar) file. In addition, the DOCKER image can contain all library dependencies for running the application. In some cases, the library dependencies are managed by a dependency manager (such as MAVEN, SBT, and / or GRADEL). The DOCKER image can contain multiple sets of configurations to run applications in four different environments. These environments can include a basic / development environment, a quality assurance (QA) environment, a sandbox environment (for third-party use), and a production environment, and the quality assurance environment can include a user stage of a cloud platform, such as the GOOGLE CLOUD platform (GCP). Multiple combinations of environments and configurations can be used for the same application. For example, a single application can have one configuration set for a development environment, three configuration sets for a QA environment, one configuration set for a sandbox environment, and five configuration sets for a production environment.

[0050] The DOCKER image may contain scripts that connect to the keychain application to obtain the secrets / passwords required to run the application securely from the keychain application. The DOCKER image may contain scripts for placing all the files required to run the application on a big data file system (e.g., HDFS). The DOCKER image may contain scripts for submitting new jobs to a job allocation / scheduling (e.g., Livy) server. The DOCKER image may contain scripts that are configured to periodically check the status of big data applications / jobs (e.g., Apache Spark jobs) submitted on a big data cluster (e.g., Apache HADOOP "Yet Another Resource Negotiator" (YARN)) to perform pull-based scheduling and resource management. The DOCKER image may contain scripts for cleaning up uploaded files into a big data file system (e.g., HDFS) after the application ends (or fails).

[0051] A container (e.g., DOCKER) compose file may contain all the parameters required to run an application in an environment, and may constitute a command executed when submitting a big data application on a computing engine (e.g., an APACHE SPARK computing engine). Some of these parameters may be provided by a big data lifecycle management platform, while other parameters may be defined by application developers. In some examples, application developers may define their own parameters and add them to their big data applications. Some of the following parameters may have the same value in all environments, but other parameters may vary from environment to environment. These parameters may include: SPARK_USER, HDFS_SERVER_ADDRESS, WEBHDFS_SERVER_PORT, CLUSTER_ENV, AVAILABILITY_ZONE, APP_MAIN_CLASS, APP_MAIN_JAR, EXTRA_JARS, EXTRA_FILES, APP_NAME, VERSION, LIVY_SERVER, TYPE, and KEYPOINTA.

[0052] Depending on the specific environment, one or more types of container orchestration files, such as docker-compose files, can be generated for each environment or environment group. For example, for the base / development environment, a docker-compose-base.yml file can be generated; for the QA / GCP user stage, a docker-compose-qa.yml file can be generated; for the sandbox environment, a docker-compose-sandbox.yml file can be generated; for production, a docker-compose-production.yml file can be generated. In one example, the big data application lifecycle management platform can support any desired number of environments at any level of granularity. These environments can be fine-grained or coarse-grained as needed. For example, one parameter can be configured for all test environments. However, another parameter can be configured only for production environments located in a city (e.g., Salt Lake City) or a region (e.g., the southwestern United States) or a country (e.g., the United States). The naming of the orchestration file can be based on the environment being configured.

[0053] At block 312, the newly created container can be run in a test environment 114 (which may include one of the aforementioned test environments). The test environment 114 can manage the application separately and run the application on a specific server cluster. The application can be run in a virtual machine or container in the test environment 114. Once the application is fully built and tested, the user can update the source code of the new application with the new code written for the application to supplement or replace the built and tested application. The new code can be sent to the code repository 110. When the user submits the code in the code repository 110, the code repository 110 can send a trigger event to the lifecycle management platform 102. At block 314, the lifecycle management platform 102 can receive a trigger event that warns that the source code has been updated. At block 316, the lifecycle management platform 102 can rebuild the build and retest the application.

[0054] When the application is complete and has been tested, at block 318, the user can indicate to the lifecycle management platform 102 that the application is ready for production. The user can provide confirmation that the application is ready for production, or the process can be automated. At block 320, the lifecycle management platform 102 can perform additional tests (e.g., security scans, resource usage tests) to determine whether the application is ready to run in the production environment 106. The test environment 114 can manage the application separately and run the application on a specific server cluster during testing and / or during the entire life cycle of the application. After completing the test, at block 322, the lifecycle management platform 102 can set up a production environment (e.g., production environment 116) to run the application. At block 324, the lifecycle management platform 102 can collect metrics (logs, monitoring) about the application, which are then analyzed. The lifecycle management platform 102 can provide automatic approval for using the application for production. At block 326, the user can provide instructions to confirm that the application is ready for production. After receiving the instructions, at block 328, the lifecycle management platform 102 can run the application in the production environment 116. In some embodiments, starting the application in the production environment includes staggering the deployment of the big data application to multiple big data server clusters 120. For example, the application may be deployed in a first big data server cluster at a first time, and if the application is stable, the application may be deployed in a second big data server cluster at a second later time.

[0055] The application may be run in a production environment. At block 330, data about the application, resource usage statistics, error logs, etc. may be collected and stored. At block 332, the application may be scheduled (via the scheduling and management server 118) to run at specific times to coordinate with one or more other applications (e.g., if there are data dependencies between the applications), or may be scheduled to run more or less frequently based on the collected, stored, and analyzed data.

[0056] During the life cycle of an application, at block 334, the software may be modified, managed, tested, and updated. Once an application is no longer useful, has been replaced by another application, or has completed its set functionality, the application may be phased out at block 336. This may include deleting it from the system, moving production data to a different location, or deleting production data. The proposed system may manage the life cycle of a single application, or may be used to manage the life cycles of multiple applications in parallel. Thus, there may be multiple applications being built, being tested, or in production, or a combination of the three may be in progress.

[0057] Reference now Figure 4 , Figure 4 A system 400 for performing lifecycle management of a big data application is shown. The lifecycle management platform 102 may include an interface for a user device. Application developers may interact with the lifecycle management platform 102 and the code repository 110. In some embodiments, the code repository 110 may include a user configuration 402 and application source code 404 that can be edited by a user.

[0058] Configuration preparer server 406 may receive user configuration 402 and application source code 404 from code repository 110, environment configuration 408, and environment template 410. Each of configuration 408 and template 410 may be designed to work with a specific big data server cluster 120.

[0059] When building a big data application, library dependencies can be collected from the library repository 412. The application library 414 in the library repository 412 can be combined and compiled with the application source code 404 to build a big data application. The application source code 404 and the application library 414 are linked and / or compiled together by the automatic build environment server 416 to form an application binary file 418. The automatic build environment server 416 can run an automatic build application (e.g., JENKINS). The application binary file 418 is placed in the template container image 420 of the application. Other library dependencies and application configurations can be placed in the template container image 420 of the application. Once the application binary file (as well as the library and configuration) is placed in the template container image, the resulting container image is the application. The container image 420 of the application can then be moved to the project template repository 422, which can include a DOCKERHUB or ARTIFACTORY server.

[0060] The application secrets 424 may include passwords, encryption keys, etc. and may be stored in a secure keychain 426. The secure keychain 426 may be used to launch a big data application on the server cluster 120. The container image 420 is sent to a job server 434 (e.g., a LIVY server) running a scheduler application that is configured to schedule the big data application to run on the big data server cluster 120. The job server 434 may be configured to schedule multiple applications and may be configured to schedule applications based on data dependencies between multiple applications. The job server 434 is also configured to track statistics and health information about the big data server cluster 120, including logging 436, monitoring 438, governance 440, access control 442, and alerts 444. Information 436-444 is provided to the lifecycle management platform 102.

[0061] The application binary file 418 from the automated build environment server 416 and the configuration and parameters of the application 428 (from the configuration preparer server 406) can be transmitted to a cluster management system 430 running a big data management application, which is configured to manage a computer cluster (e.g., APACHE MESOS). The cluster management system 430 may include an application launcher 432, which is configured to launch the template container image 420 of the application.

[0062] Reference now Figure 5 , Figure 5 An embodiment of a computer system 500 is shown, which is suitable for implementing, for example, user devices, platforms, and servers (including combined server clusters). It should be understood that other devices used in a lifecycle management system can be implemented as the computer system 500 in the following manner.

[0063] According to various embodiments of the present disclosure, a computer system 500 (e.g., a computer and / or a network server) includes a bus 502 or other communication mechanism for transmitting information, which connects subsystems and components such as a processor 504 (e.g., a processor, a microcontroller, a digital signal processor DSP, etc.), a system memory component 506 (e.g., a RAM), a static storage component 508 (e.g., a ROM), a disk drive component 510 (e.g., a magnetic, optical, flash memory, or solid state), a network interface component 512 (e.g., a modem or an Ethernet card), a display component 514 (e.g., a CRT or LCD), an input component 518 (e.g., a keyboard, a keypad or a virtual keyboard, a microphone), a cursor control component 520 (e.g., a mouse, a pointer, or a trackball), and / or a position determination component 522 (e.g., a global positioning system (GPS) device as shown, a cellular tower triangulation device, and / or various other position determination devices known in the art). In one embodiment, the disk drive component 510 may include a database having one or more disk drive components.

[0064] According to an embodiment of the present disclosure, the computer system 500 performs specific operations, such as the operations described herein with respect to user devices, server devices (including payment provider servers, merchant servers, and identity verification servers), data storage areas, and nodes, by executing one or more instruction sequences contained in the memory component 506 through the processor 504. These instructions can be read into the system memory component 506 from another computer-readable medium (such as a static storage component 508 or a disk drive component 510). In other embodiments, hard-wired circuits can replace software instructions or be used in combination with software instructions to implement the present disclosure.

[0065] Logic may be encoded in a computer readable medium, which may refer to any medium that participates in providing instructions to processor 504 for execution. Such media may take a variety of forms, including, but not limited to, non-volatile media, volatile media, and transmission media. In one embodiment, the computer readable medium is non-transitory. In various implementations, non-volatile media include optical or magnetic disks (e.g., disk drive component 510), volatile media include dynamic memory (e.g., system memory component 508), and transmission media include coaxial cables, copper wire, and optical fiber, including the wires that make up bus 502. In one example, the transmission medium may take the form of sound or light waves such as those generated during radio wave and infrared data communications.

[0066] Some common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, a solid state drive (SSD), a magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, a punch card, a paper tape, any other physical medium with holes, a flash memory, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other memory chip or cassette, a carrier wave, or any other medium suitable for a computer to read. In one embodiment, the computer readable medium is non-transitory.

[0067] In various embodiments of the present disclosure, the execution of the instruction sequences implementing the present disclosure may be performed by a computer system 500. In various other embodiments of the present disclosure, multiple computer systems 500 coupled to a network (e.g., a LAN, WLAN, PSTN, and / or various other wired or wireless networks, including telecommunications, mobile, and cellular telephone networks) via a communication link 518 may execute the instruction sequences to implement the present disclosure in coordination with each other.

[0068] The network interface 512 of the computer system 500 may also include a short-range communication interface. Therefore, in various embodiments, the network interface 512 may include a transceiver circuit, an antenna, and / or a waveguide. The network interface 512 may use one or more short-range wireless communication technologies, protocols, and / or standards (e.g., Wi-Fi, Bluetooth, Bluetooth Low Energy (BLE), infrared, NFC, etc.).

[0069] In various embodiments, the network interface 512 can be configured to detect other systems, devices, peripherals, and data storage with short-range communication technology near the computer system 500. The network interface 512 can create a communication area for detecting other devices with short-range communication capabilities. When other devices with short-range communication capabilities are placed in the communication area of ​​the network interface 512, the network interface 512 can detect these other devices and exchange data with these other devices. When close enough, the network interface 512 can receive identifier packets from other devices. The identifier packet may include one or more identifiers, which may be operating system registry entries, cookies associated with an application, identifiers associated with the hardware of another device, and / or various other appropriate identifiers.

[0070] In some embodiments, network interface 512 can identify a local area network using a short-range communication protocol such as Wi-Fi and join the local area network. In some examples, computer system 500 can use network interface 512 to discover and / or communicate with other devices that are part of a local area network.

[0071] Computer system 500 may send and receive messages, data, information, and instructions, including one or more programs (i.e., application code), via communication link 524 and network interface component 512. Network interface component 512 may include a separate or integrated antenna to enable transmission and reception via communication link 524. Received program code may be executed by processor 504 upon receipt and / or stored in storage drive component 510 or some other non-volatile storage component for execution.

[0072] Where applicable, hardware, software, or a combination of hardware and software may be used to implement the various embodiments provided by the present disclosure. In addition, where applicable, the various hardware components and / or software components recorded herein may be combined into composite components including software, hardware, and / or both, without departing from the scope of the present disclosure. Where applicable, the various hardware components and / or software components recorded herein may be separated into subcomponents including software, hardware, and / or both, without departing from the scope of the present disclosure. In addition, where applicable, it is contemplated that software components may be implemented as hardware components, and vice versa.

[0073] Software (e.g., program code and / or data) according to the present disclosure may be stored on one or more computer-readable media. It is also contemplated that the software identified herein may be implemented using one or more general or special-purpose computers and / or computer systems that are networked and / or not networked. Where applicable, the ordering of the various steps described herein may be changed, combined into composite steps, and / or separated into sub-steps to provide the features described herein.

[0074] The foregoing disclosure is not intended to limit the disclosure to the exact form or specific field of use disclosed. Therefore, various alternative embodiments and / or modifications to the disclosure are contemplated, whether explicitly described herein or implied, and are possible in light of the disclosure. Having thus described the embodiments of the disclosure, one of ordinary skill in the art will recognize that changes in form and detail may be made without departing from the scope of the disclosure. Thus, the disclosure is limited only by the claims.

Claims

1. A method for managing the life cycle of a big data application, comprising: receiving instructions for creating a big data application; In response to receiving the indication, generating an application instance using a large data container template, wherein the large data container template includes a multi-layer container configured to manage the life cycle of the large data application, wherein the multi-layer container includes: The first container layer includes operating system settings and security settings; A second container layer, including big data settings, the big data settings including settings and libraries shared by multiple big data applications; A third container layer, including settings associated with the user and a template test application; and a fourth container layer, including big data application source code and settings specific to the big data application; and Configure a build environment and a test environment for the application instance.

2. The method of claim 1, wherein: The large data container template includes a DOCKER image.

3. The method of claim 1, wherein: The first container layer includes at least one of an operating system, a platform tool, or a platform security tool; and The second container layer includes an orchestration monitoring client and a big data file system client.

4. The method of claim 1, wherein: The operations also include: receiving a trigger event indicating that the big data application source code has been updated; and In response to receiving the trigger event: Rebuilding the big data application using the updated big data application source code; and The application instance is updated with the restructured big data application.

5. The method of claim 1, wherein: The first layer includes a customized operating system configured with one or more specialized developer tools.

6. The method of claim 4, further comprising: After rebuilding the big data application, retest the big data application.

7. The method of claim 6, further comprising: configuring the big data application to process the operational data; as well as The big data application is placed in one or more production environments.

8. The method of claim 7, wherein: The placement includes deploying the big data application to multiple data clusters in a staggered manner in time.

9. The method of claim 1, further comprising: The big data application is scheduled to run periodically on the production server cluster.

10. The method of claim 9, wherein: The scheduling is based on managing data dependencies between two or more big data applications.

11. The method of claim 4, further comprising: The big data application source code is stored in a source code repository, wherein the trigger event is received from the source code repository.

12. The method of claim 1, further comprising: Collecting statistical information about resources and user usage of the big data application; as well as The timing of running the big data application is determined based on the resource and user usage statistics.

13. The method of claim 1, wherein: The test environment includes the following equipment: the equipment runs a virtual machine or a container to start the big data application on a server cluster.

14. The method of claim 13, further comprising: Each of the multiple versions of the big data application is started using the virtual machine or container to test the multiple versions of the big data application.

15. A non-transitory computer-readable medium having instructions stored thereon, the instructions being executable by a computer system to cause the computer system to perform operations comprising: receiving instructions for creating a big data application; In response to receiving the indication, generating an application instance using a large data container template, wherein the large data container template includes a multi-layer container configured to manage the life cycle of the large data application, wherein the multi-layer container includes: The first container layer includes operating system settings and security settings; A second container layer, including big data settings, the big data settings including settings and libraries shared by multiple big data applications; A third container layer, including settings associated with the user and a template test application; and a fourth container layer, including big data application source code and settings specific to the big data application; and Configure a build environment and a test environment for the application instance.

16. The non-transitory computer readable medium of claim 15, wherein: The large data container template includes a DOCKER image.

17. The non-transitory computer readable medium of claim 15, wherein: The first container layer includes at least one of an operating system, a platform tool, or a platform security tool; and The second container layer includes an orchestration monitoring client and a big data file system client.

18. The non-transitory computer readable medium of claim 15, wherein: The operations also include: receiving a trigger event indicating that the big data application source code has been updated; and In response to receiving the trigger event: Rebuilding the big data application using the updated big data application source code; and The application instance is updated with the restructured big data application.

19. The non-transitory computer readable medium of claim 15, wherein: The first layer includes a customized operating system configured with one or more specialized developer tools.

20. A system for managing the life cycle of a big data application, comprising: a first non-transitory memory; as well as One or more first hardware processors coupled to the first non-transitory memory and configured to execute instructions from the first non-transitory memory to cause the system to perform operations including: receiving instructions for creating a big data application; In response to receiving the indication, generating an application instance using a large data container template, wherein the large data container template includes a multi-layer container configured to manage the life cycle of the large data application, wherein the multi-layer container includes: The first container layer includes operating system settings and security settings; A second container layer, including big data settings, the big data settings including settings and libraries shared by multiple big data applications; A third container layer, including settings associated with the user and a template test application; and a fourth container layer, including big data application source code and settings specific to the big data application; and Configure a build environment and a test environment for the application instance.

21. The system of claim 20, wherein: The large data container template includes a DOCKER image.

22. The system of claim 20, wherein: The first container layer includes at least one of an operating system, a platform tool, or a platform security tool; and The second container layer includes an orchestration monitoring client and a big data file system client.

23. The system of claim 20, wherein: The operations also include: receiving a trigger event indicating that the big data application source code has been updated; and In response to receiving the trigger event: Rebuilding the big data application using the updated big data application source code; and The application instance is updated with the restructured big data application.

24. The system of claim 20, wherein: The first layer includes a customized operating system configured with one or more specialized developer tools.

Images