A self-developed software integration method for domestic big data platform based on mpack

Through metadata management, the independent software image warehouse mpack is built, which solves the problems of self-developed software integration and domestic compatibility of domestic big data platforms, and realizes rapid and decoupled software integration and management, which is suitable for different architectures and operating systems. Reduces cost and complexity.

CN114385200BActive Publication Date: 2024-11-26HAOHAN DATA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210050894.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-17
Publication Date
2024-11-26
Estimated Expiration
2042-01-17

AI Technical Summary

Technical Problem

The domestic big data industry faces the closed source problem of CDH and HDP big data platforms, resulting in backward technology. In the process of domestic production, self-developed software integration and management are difficult to compatible with domestic architectures and operating systems. The existing solutions have service interruptions and high migration costs. Migration adaptation is not universal, etc.

Method used

Using metadata management method, build a software mirror warehouse mpack independent of the software stack, add, upgrade and remove self-developed software by operating metadata information, and automatically select and install adapted software according to the server architecture to realize the software Fast, decoupled integration and management.

Benefits of technology

It realizes the rapid, decoupled integration and management of self-developed software, reduces server costs, supports the automatic deployment and management of heterogeneous servers and operating systems, and reduces the complexity of ordinary software personnel.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114385200B_ABST
    Figure CN114385200B_ABST
Patent Text Reader

Abstract

The present invention discloses a self-developed software integration method of a domestic big data platform based on mpack, comprising the following steps: S1, compile and publish the self-developed software installation package, and migrate it to the domestic big data platform; S2, the domestic big data platform classifies and stores the migrated and adapted self-developed software and the original software according to the yum installation form; S3, use createrepo to build a software image warehouse mpack, the software image warehouse mpack includes the life cycle information and metadata information of the original software and the self-developed software in the domestic big data platform software stack stack; S4, specify the mapping information of the software image warehouse mpack and the domestic big data platform software stack stack. The present invention adopts the method of metadata management to construct a software image warehouse mpack that is independently externalized to the existing software stack. By operating the metadata information, the software outside the existing software stack of the platform can be independently operated, and the self-developed software can be integrated and managed in a fast and decoupled manner without affecting the online service.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of big data platforms, and in particular to a self-developed software integration method for a domestic big data platform based on mpack. Background Art

[0002] CDH and HDP, two major big data platform products widely used in the domestic big data industry, have successively announced paid subscriptions and closed source. Many related companies in the domestic big data industry are facing the problem that the big data platforms originally built based on CDH and HDP can no longer be updated. If this continues, it will inevitably lead to technological backwardness and be eliminated by the times. In addition, with the trend of localization, the country calls for technological independence and solves "bottleneck" technologies. In view of the closed source problem of the two major big data platforms CDH and HDP, they are not compatible with domestic architecture chips such as Kunpeng and the domestic Euler operating system, and are not technically independent. We have developed a new domestic big data platform based on the secondary development route of the open source product Ambari, but how to integrate and manage self-developed software and how to be compatible and adapt to various domestic architectures and domestic operating systems is an urgent problem to be solved.

[0003] Regarding the problem of integrating self-developed software, there are currently two main solutions for integrating self-developed software in the original way of Ambari:

[0004] The first is to deploy the self-developed software directly to the Ambari software stack, which can be deployed and managed as a regular software. However, in this solution, the self-developed software will be bundled into Ambari's own software stack. Once the stack software is upgraded, other non-related software will need to be migrated, and the platform service will be interrupted during the migration.

[0005] The second solution is to build a new cluster based on the new stack based on the first solution, gradually migrate the original data to the new version of the cluster, and finally complete the switch. This solution allows the platform to continue to provide services in a smooth manner without downtime, but it places high demands on the personnel responsible for the migration and how to coordinate upstream and downstream applications. In addition, additional servers are required for migration, resulting in additional server costs for the enterprise.

[0006] In addition, regarding the issue of domestic compatibility and adaptation, as the localization process continues to advance, the same set of software needs to be deployed on servers with different operating systems and even different hardware architectures at the same time. The current solution only refers to the migration and adaptation solutions provided by domestic server manufacturers and operating system manufacturers, and professionals perform migration and adaptation according to the official solutions. However, both domestic servers and operating systems are in a process of rapid iterative development. The migration and adaptation of certain software still requires professional practitioners to spend a lot of energy to complete, and the migration and adaptation solutions in different environments are not necessarily universal, which is usually unsolvable by ordinary software users. This has brought great resistance to the internal promotion of domestically adapted software by enterprises. At the same time, how to integrate the migrated and adapted software with the existing x86 software for unified deployment and management is also a major challenge. Summary of the invention

[0007] The purpose of the present invention is to provide a self-developed software integration method for a domestic big data platform based on mpack. Based on the domestic big data platform and using metadata management, a method for independently externalizing software to an existing software stack is proposed. By operating metadata information, the software outside the existing software stack of the platform can be independently added, upgraded, and removed, and the adapted software can be automatically selected and installed according to the server architecture.

[0008] To achieve the above purpose, the present invention provides the following technical solution: a self-developed software integration method for a domestic big data platform based on mpack, the self-developed software integration method comprising the following steps:

[0009] S1. Compile and release the self-developed software installation package, and migrate it to the domestic big data platform;

[0010] S2. The domestic big data platform classifies and stores the migrated and adapted self-developed software and the original software according to the yum installation method;

[0011] S3. Use createrepo to build a software image repository mpack, where the software image repository mpack includes the life cycle information and metadata information of the original software in the domestic big data platform software stack and the self-developed software;

[0012] S4. Specify the mapping information between the software image repository mpack and the domestic big data platform software stack stack.

[0013] Preferably, the life cycle information includes software installation information, software configuration information, software startup information and software stop process information, the software installation information describes the software installation method, the software configuration information describes the modification method of the software configuration file, the software startup information describes different startup methods according to different software, and the software stop information describes different stop methods according to different software.

[0014] Preferably, the metadata information includes software name, software version, software notes, software dependent components and software cross-platform information, and the software cross-platform information describes the location of the software adapted to the current platform in the software image repository mpack.

[0015] Preferably, S1 includes professionals compiling and publishing the installable software package in accordance with the migration adaptation rules of each server manufacturer and operating system manufacturer.

[0016] Preferably, S3 comprises the following steps:

[0017] S301, compile a service package, wherein the service package includes life cycle information and metadata information of the software;

[0018] S302, organize one or more service packages into the extension version directory;

[0019] S302, organize one or more extensions into the extensions directory;

[0020] S304. Organize one or more extensions into the mpack directory of the software image repository, and define mpack metadata information, where the mpack metadata information includes the mpack name, mpack version, and the source of the mpack extensions.

[0021] Preferably, S3 further includes:

[0022] S305. Compress the software image repository mpack and mpack metadata information into a zip file and publish it as a distribution.

[0023] Preferably, S4 includes: the user downloads the mpack distribution, imports it into the domestic big data platform, establishes a metadata database on the domestic big data platform, and mounts all service packages under the software image warehouse mpack to the domestic big data platform software stack stack.

[0024] Preferably, the update steps of the software image repository mpack include:

[0025] A1. Professionals release new versions of software packages and update the software in the software image repository mpack according to the migration and adaptation software process.

[0026] A2. Based on the original service package, modify the software cross-platform information, software version information and dependent component information in its metadata information.

[0027] A3. Add the new version of the service package generated in step A2 to the original extension. The platform software update is complete.

[0028] Preferably, the steps of uninstalling the software image warehouse mpack include:

[0029] a: Suspend software services on domestic big data platforms;

[0030] b: Delete the corresponding software services contained in the software image repository mpack;

[0031] c: Delete the mapping information of the software image repository mpack and the domestic big data platform software stack stack in the metadata database.

[0032] Compared with the prior art, the present invention has the following beneficial effects:

[0033] The self-developed software integration method of the present invention is based on a domestic big data platform and adopts a metadata management method to construct a software image warehouse mpack that is independently external to the existing software stack. By operating metadata information, it is possible to independently perform add, upgrade, and remove operations on software outside the platform's existing software stack. It is possible to integrate and manage self-developed software in a fast, decoupled manner that does not affect online services, and automatically select and install adapted software according to the server architecture. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is an operation structure diagram of a self-developed software integration method of a domestic big data platform based on mpack in the present invention;

[0035] Figure 2 This is a packaging structure diagram of the software image warehouse mpack in a self-developed software integration method of a domestic big data platform based on mpack in the present invention;

[0036] Figure 3 This is an architecture diagram of the migration and adaptation software process in the self-developed software integration method of the domestic big data platform based on mpack in the present invention;

[0037] Figure 4 This is a structural block diagram of a service package in a self-developed software integration method for a domestic big data platform based on mpack in the present invention;

[0038] Figure 5 It is a structural block diagram of the platform software stack in the self-developed software integration method of the domestic big data platform based on mpack of the present invention;

[0039] Figure 6 This is a structural block diagram of the extension in the self-developed software integration method of the domestic big data platform based on mpack in the present invention;

[0040] Figure 7 It is a structural block diagram of extensions in a self-developed software integration method of a domestic big data platform based on mpack in the present invention;

[0041] Figure 8 It is a structural block diagram of the software image warehouse mpack in a self-developed software integration method of a domestic big data platform based on mpack in the present invention;

[0042] Fig. 9 This is a flow chart corresponding to step S305 in a self-developed software integration method for a domestic big data platform based on mpack in the present invention;

[0043] Fig.10 This is a process architecture diagram corresponding to the software image warehouse mpack uninstallation step in the self-developed software integration method of the domestic big data platform based on mpack in the present invention. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] like Figure 1 As shown, a self-developed software integration method operation structure diagram of a domestic big data platform based on mpack, an embodiment provided by the present invention, the self-developed software integration method comprises the following steps:

[0046] S1. Compile and release the self-developed software installation package, and migrate it to the domestic big data platform;

[0047] S2. The domestic big data platform classifies and stores the migrated and adapted self-developed software and the original software according to the yum installation method;

[0048] S3. Use createrepo to build a software image repository mpack, where the software image repository mpack includes the life cycle information and metadata information of the original software in the domestic big data platform software stack and the self-developed software;

[0049] S4. Specify the mapping information between the software image repository mpack and the domestic big data platform software stack stack.

[0050] Among them, the life cycle information includes software installation information (describing the software installation method), software configuration information (describing the modification method of the software configuration file), software startup information (describing different startup methods according to different software) and software stop process information (describing different stop methods according to different software). The metadata information includes software name, software version, software notes, software dependency components and software cross-platform information, and the software cross-platform information describes the location of the software adapted by the current platform in the software image warehouse mpack. Under this embodiment, the self-developed software integration method is based on the self-developed domestic big data platform, and adopts the metadata management method to propose a method for software to be independently externalized to an existing software stack. By operating the metadata information, the software outside the existing software stack of the platform can be independently added, upgraded, and removed, and the adapted software can be automatically selected and installed according to the server architecture. It can integrate self-developed software under servers / operating systems of different architectures (such as existing Intel servers + domestic Huawei Kunpeng arm servers, or CentOS + Huawei Euler operating system) and ensure that the migrated and adapted software can be uniformly deployed and managed with the existing software (and ensure that the software of heterogeneous servers and operating systems can be automatically deployed). Under this solution, software integration, upgrading, and uninstalling no longer require direct operation within the software stack. It supports external upgrading software to replace the software within the software stack, and supports automatic deployment and management of software on heterogeneous servers and operating systems, reducing the cost of use for ordinary software personnel.

[0051] like Figure 2As shown, it is a packaging structure diagram of the software image warehouse mpack. The figure shows that the software image warehouse mpack describes a combination of one or more extensions. In this embodiment, mpack is the top-level packaging, which contains one or more extensions; each extension contains one or more extensions; each extension contains one or more versions. Extension describes a combination of one or more services, which supports multiple versions and has the same platform software stack stack organizational structure, but is independent of the platform software stack stack. Extensions describe a combination of one or more extensions. Compared with the prior art, the present invention is based on the mpack method. Through the management of metadata, it can directly expand and integrate self-developed software without involving stack internal operations and service restarts of the stack software stack; it can automatically select software packages according to the server and uniformly manage deployment through the platform; ordinary application personnel directly introduce the software image warehouse mpack without paying attention to software migration and service writing.

[0052] like Figure 3 As shown in the figure, it is an architectural diagram of the migration adaptation software process. The migration adaptation software process is a prerequisite step for integrating software using the mpack method on the big data platform.

[0053] The migration and adaptation software process is as follows:

[0054] Step 1: Professionals compile and publish installable software packages according to the migration adaptation manuals of each server manufacturer and operating system manufacturer.

[0055] Step 2: Classify and store the migrated and adapted ARM software and the original x86 software according to the server architecture.

[0056] Step 3: Use createrepo to build a software image repository; createrepo is open source software used to build metadata for the software image repository.

[0057] The migration and adaptation software process uses the yum installation method. To install with yum, you need to store the software versions of different architectures in different directories and follow the yum path specification.

[0058] The second step of classification storage refers to following the yum path specification and organizing the directory according to the default variables of the operating system. For example, the following variables:

[0059] $releasever: operating system version, such as centos7.6 is 7.6

[0060] $basearch: server architecture name, such as aarch64 for arm 64-bit

[0061] For example:

[0062] Software A has two packages for x86 and arm, a.x86_64.rpm and a.aarch64.rpm, so the directory structure is

[0063] / var / www / html / 7.6 / aarch64 /

[0064] / var / www / html / 7.6 / x86_64 /

[0065] Configuration in yum's repo file:

[0066] [a]

[0067] name=a repo

[0068] baseurl=http: / / {{yum_repo_host}} / $releasever / $basearch

[0069] enabled=1

[0070] Then, using yum install a can automatically identify different versions of the installation package of software a.

[0071] like Figure 4 and Figure 5 The following are the structural diagrams of the service package and the platform software stack stack. The service package describes the life cycle and metadata information of the software. The life cycle includes software installation information, software configuration information, software startup information, and software stop information. The metadata information includes software name, software version, software notes, and software dependent components. The platform software stack stack describes the platform software stack structure, which is the most primitive software organization form of Ambari. The platform software stack has multiple iterative versions, and the iterative version is a combination of one or more services.

[0072] like Figure 6-Figure 8 As shown, the mpack integration process in S3 includes the following steps:

[0073] S301, compile a service package, wherein the service package includes life cycle information and metadata information of the software;

[0074] S302, organize one or more service packages into the extension version directory;

[0075] S302, organize one or more extensions into the extensions directory;

[0076] S304. Organize one or more extensions into the mpack directory of the software image repository, and define mpack metadata information, where the mpack metadata information includes the mpack name, mpack version, and the source of the mpack extensions.

[0077] like Fig. 9 As shown, a process architecture diagram corresponding to step S305 is shown. Preferably, the S3 also includes: S305, compressing the software image warehouse mpack and mpack metadata information into a zip file, publishing it as a distribution version, ordinary users download the mpack distribution version, import it into the domestic big data platform, establish a metadata database on the domestic big data platform, and all service packages under the software image warehouse mpack are attached to the domestic big data platform software stack stack, specify the mapping information between mpack and stack, and users install and deploy software through the big data platform, and the operation method remains unchanged.

[0078] Preferably, the update steps of the software image repository mpack include:

[0079] A1. Professionals release new versions of software packages and update the software in the software image repository mpack according to the migration and adaptation software process.

[0080] A2. Based on the original service package, modify the software cross-platform information, software version information and dependent component information in its metadata information.

[0081] A3. Add the new version of the service package generated in step A2 to the original extension. The platform software update is complete.

[0082] Preferably,

[0083] like Fig.10 As shown in the figure, the specific steps for uninstalling the software image repository mpack include:

[0084] a: Suspend software services on domestic big data platforms;

[0085] b: Delete the corresponding software services contained in the software image repository mpack;

[0086] c: Delete the mapping information of the software image repository mpack and the domestic big data platform software stack stack in the metadata database.

[0087] The self-developed software integration method of the present invention is based on the management method of the metadata library adopted by the domestic big data platform to manage different versions of the software on the big data platform, and realize the rapid integration and free switching of versions through metadata. In order to take into account the software of servers with different architectures, the form of a mirror warehouse is adopted, and a software mirror warehouse mpack that is independently external to the existing software stack is constructed. By operating the metadata information, the software outside the existing software stack of the platform can be independently added, upgraded, and removed. It can integrate and manage self-developed software in a fast, decoupled way that does not affect online services, and automatically select and install adapted software according to the server architecture. Provides a fast, decoupled way that does not affect online services, a method for integrating and managing different versions of software, and a method for integrating software of different server architectures / operating systems.

[0088] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A self-developed software integration method for a domestic big data platform based on mpack, It is characterized in that The self-developed software integration method comprises the following steps: S1. Compile and release the self-developed software installation package, and migrate it to the domestic big data platform; S2. The domestic big data platform classifies and stores the migrated and adapted self-developed software and the original software according to the yum installation method; S3. Use createrepo to build a software image repository mpack, which includes the life cycle information and metadata information of the original software in the domestic big data platform software stack and the self-developed software; S4, specifying mapping information between the software image repository mpack and the domestic big data platform software stack stack; said S3 includes the following steps: S301, compile a service package, wherein the service package includes life cycle information and metadata information of the software; S302, organize one or more service packages into the extension version directory; S302, organize one or more extensions into the extensions directory; S304: Organize one or more extensions to the mpack directory of the software image repository, and define mpack metadata information, wherein the mpack metadata information includes the mpack name, mpack version, and the source of the mpack extensions. S3 also includes: S305, compressing the software image warehouse mpack and mpack metadata information into a zip file and publishing it as a distribution version, said S4 includes: the user downloads the mpack distribution version, imports it into the domestic big data platform, establishes a metadata database on the domestic big data platform, and all service packages under the software image warehouse mpack are attached to the domestic big data platform software stack stack, and the update steps of the software image warehouse mpack include: A1. Professionals release new versions of software packages and update the software in the software image repository mpack according to the migration and adaptation software process; A2. Based on the original service package, modify the software cross-platform information, software version information and dependent component information in its metadata information; A3. Add the new version of the service package generated in step A2 to the original extension. The platform software update is complete.

2. According to the self-developed software integration method of the domestic big data platform based on mpack according to claim 1, It is characterized in that The life cycle information includes software installation information, software configuration information, software startup information and software stop process information. The software installation information describes the software installation method, the software configuration information describes the modification method of the software configuration file, the software startup information describes different startup methods according to different software, and the software stop process information describes different stop methods according to different software.

3. According to the self-developed software integration method of the domestic big data platform based on mpack according to claim 2, It is characterized in that The metadata information includes software name, software version, software notes, software dependent components and software cross-platform information. The software cross-platform information describes the location of the software adapted to the current platform in the software image repository mpack.

4. According to the self-developed software integration method of the domestic big data platform based on mpack as claimed in claim 3, It is characterized in that S1 includes professionals compiling and publishing an installable software package in accordance with the migration adaptation rules of each server manufacturer and operating system manufacturer.

5. According to the self-developed software integration method of the domestic big data platform based on mpack according to claim 4, It is characterized in that The steps to uninstall the software image repository mpack include: a: Suspend software services on domestic big data platforms; b: Delete the corresponding software services contained in the software image repository mpack; c: Delete the mapping information of the software image repository mpack and the domestic big data platform software stack stack in the metadata database.