Big data platform construction method and device, computer equipment and storage medium

Abstract

The invention discloses a big data platform construction method and device, computer equipment and a storage medium. The method comprises the steps that a first server cluster and a second server cluster are configured; configuring corresponding software for the first server cluster and the second server cluster respectively, and obtaining first software corresponding to the first server cluster and second software corresponding to the second server cluster; if the first software and the second software have the same software type and different versions, the first server cluster, the first software and the first operating system form a first cluster, and the second server cluster, the second software and the second operating system form a second cluster; therefore, a credential big data platform can be constructed on a hybrid cluster composed of the first cluster and the second cluster. According to the method and the device, the non-creative server cluster can be added on the basis of the creative server cluster, so that the creative big data platform can be constructed on the hybrid cluster formed by the creative servers and the non-creative servers, and the flexibility of server configuration is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to a method, apparatus, computer equipment and storage medium for building a big data platform. Background Art

[0002] In order to meet the needs of information and innovation, the big data platform should be deployed on a physical machine whose CPU (Central Processing Unit), operating system, and basic software all meet the requirements of information and innovation.

[0003] At present, the big data platform is built on the trusted server cluster, and the trusted server cluster is installed with the corresponding trusted operating system and trusted software layer to ensure that the big data platform as a whole meets the trusted requirements. For example, MRS (MapReduce Service) is a big data basic platform with a relatively high market share in private deployment. MRS can be configured on cloud servers with x86 processors or ARM processor CPUs, CentOS / KYLIN / UOS and other operating systems.

[0004] However, the above-mentioned trusted server cluster can only expand the capacity of trusted servers, but cannot add non-trusted servers, which makes it difficult to support the construction of a trusted big data platform on a hybrid cluster composed of trusted servers and non-trusted servers. Summary of the invention

[0005] The main purpose of this application is to provide a big data platform construction method, device, computer equipment and storage medium, which can add a non-trusted innovation server cluster on the basis of the trusted innovation server cluster to support the construction of a trusted innovation big data platform on a hybrid cluster composed of trusted innovation servers and non-trusted innovation servers, thereby improving the flexibility of server configuration and reducing platform construction costs.

[0006] In order to achieve the above objectives, in a first aspect, the present application provides a method for constructing a big data platform, comprising:

[0007] Configure a first server cluster and a second server cluster, wherein the first server cluster is a cluster consisting of ICT servers, and the second server cluster is a cluster consisting of servers other than ICT servers;

[0008] Configure corresponding software for the first server cluster and the second server cluster respectively, and obtain first software corresponding to the first server cluster and second software corresponding to the second server cluster;

[0009] If the first software and the second software are of the same software type but different versions, the first cluster is composed of the first server cluster, the first software and the first operating system, and the second cluster is composed of the second server cluster, the second software and the second operating system, to support the construction of an innovative big data platform on the hybrid cluster composed of the first cluster and the second cluster.

[0010] In an embodiment, corresponding software is configured for the first server cluster and the second server cluster respectively, and the first software corresponding to the first server cluster and the second software corresponding to the second server cluster are obtained, including:

[0011] Acquire first software information corresponding to the first server cluster and second software information corresponding to the second server cluster through a configured intelligent management algorithm;

[0012] respectively generating a first installation instruction corresponding to the first software information and a second installation instruction corresponding to the second software information;

[0013] Sending the first installation instruction to the first server cluster, and causing the servers in the first server cluster to perform software installation according to the first installation instruction and the first software information, and obtaining the first software corresponding to the first server cluster;

[0014] The second installation instruction is sent to the second server cluster, and the servers in the second server cluster are enabled to perform software installation according to the second installation instruction and the second software information, so as to obtain the second software corresponding to the second server cluster.

[0015] In an embodiment, obtaining first software information corresponding to the first server cluster through a configured intelligent management algorithm includes:

[0016] Obtaining and identifying a cluster type of the first cluster;

[0017] determining a server type in the first server cluster based on the cluster type of the first cluster;

[0018] First software information corresponding to the first server cluster is generated based on the server types in the first server cluster.

[0019] In one embodiment, the first software includes basic software and / or service software, and the first software information includes the basic software package, the service software package name and the download URL;

[0020] Sending a first installation instruction to a first server cluster, and causing a server in the first server cluster to perform software installation according to the first installation instruction and the first software information, and obtaining the first software corresponding to the first server cluster, includes:

[0021] Sending a first installation instruction to the first server cluster;

[0022] Obtain the service package by using the service package name and download URL;

[0023] Based on the first installation instruction, the servers in the first server cluster are enabled to install the basic software package and / or the service software package, and obtain the basic software and / or service software corresponding to the first server cluster.

[0024] In an embodiment, based on the first installation instruction, the server in the first server cluster executes the installation of the basic software package and / or the service software package, and obtains the basic software and / or service software corresponding to the first server cluster, including:

[0025] Initializing the servers in the first server cluster;

[0026] If the servers in the first server cluster are successfully initialized, the servers in the first server cluster are enabled to install the basic software package and / or the service software package based on the first installation instruction, and obtain the basic software and / or service software corresponding to the first server cluster.

[0027] In an embodiment, configuring corresponding software for a first server cluster and obtaining first software corresponding to the first server cluster includes:

[0028] Determining whether the servers in the first server cluster are servers of the same type;

[0029] If the servers in the first server cluster are of the same type, obtaining a code of each server in the first server cluster;

[0030] Combining the codes of the servers in the first server cluster to obtain an unlocking code;

[0031] The first software corresponding to the first server cluster is obtained from the software configuration file through the unlocking code.

[0032] In one embodiment, obtaining the first software corresponding to the first server cluster from the software configuration file by using the unlock code includes:

[0033] Unlock the software configuration file using the unlock code to obtain the software set;

[0034] Obtaining digital information in the name of each software in the software collection, and concatenating the digital information in the name of each software in the software collection with the unlock code to obtain a concatenated code;

[0035] The first software corresponding to the first server cluster is obtained from the software configuration file by concatenating the code and the codes of the servers in the first server cluster.

[0036] In an embodiment, obtaining the first software corresponding to the first server cluster from the software configuration file by concatenating the code and the codes of each server in the first server cluster includes:

[0037] Obtaining an identifier of each server in the first server cluster from the concatenated code based on the code size of each server in the first server cluster;

[0038] Obtaining preset identification differences, wherein each preset identification in the preset identification interval is matched with corresponding software;

[0039] If the identifiers of the servers in the first server cluster are in the preset identifier interval, obtaining software matching the identifiers of the servers from the software configuration file based on the identifiers of the servers in the first server cluster;

[0040] The first software corresponding to the first server cluster is determined by the software matched by the identifiers of the servers.

[0041] In one embodiment, the method further includes:

[0042] If the initialization of the servers in the first server cluster fails, the servers in the first server cluster are checked and the process returns to execute the step of initializing the servers in the first server cluster.

[0043] In one embodiment, obtaining a service software package through the service software package name and download URL includes:

[0044] Concatenate the service software package name and the download URL to obtain the concatenated download address;

[0045] Obtain the service software package through the concatenated download address.

[0046] In one embodiment, the method further includes:

[0047] A basic software package is obtained based on the server type in the first server cluster.

[0048] In a second aspect, an embodiment of the present application provides a big data platform construction device, including:

[0049] A configuration module, used to configure a first server cluster and a second server cluster, wherein the first server cluster is a cluster composed of ICT servers, and the second server cluster is a cluster composed of servers other than ICT servers;

[0050] A software acquisition module, used to configure corresponding software for the first server cluster and the second server cluster respectively, and acquire first software corresponding to the first server cluster and second software corresponding to the second server cluster;

[0051] The platform building module is used to support the construction of an information and communication technology big data platform on a hybrid cluster consisting of the first cluster and the second cluster, if the first software and the second software are of the same software type but different versions, by forming a first cluster consisting of a first server cluster, a first software and a first operating system, and a second cluster consisting of a second server cluster, a second software and a second operating system.

[0052] In a third aspect, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0053] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0054] The embodiment of the present application provides a method, device, computer equipment and storage medium for building a big data platform, including: first configuring a first server cluster and a second server cluster, wherein the first server cluster is a cluster composed of trusted servers, and the second server cluster is a cluster composed of servers other than trusted servers, and then configuring corresponding software for the first server cluster and the second server cluster respectively, obtaining the first software corresponding to the first server cluster and the second software corresponding to the second server cluster, if the software types of the first software and the second software are the same but the versions are different, the first cluster is composed of the first server cluster, the first software and the first operating system, and the second cluster is composed of the second server cluster, the second software and the second operating system, so as to support the construction of a trusted big data platform on a hybrid cluster composed of the first cluster and the second cluster. The present application can add a non-trusted server cluster on the basis of the trusted server cluster to support the construction of a trusted big data platform on a hybrid cluster composed of trusted servers and non-trusted servers, thereby improving the flexibility of server configuration. In addition, the trusted big data platform is constructed by a hybrid cluster composed of trusted servers and non-trusted servers, thereby avoiding the configuration of a large number of the same servers, reducing the number of servers and reducing the cost of platform construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings constituting a part of this application are used to provide a further understanding of this application, so that other features, purposes and advantages of this application become more obvious. The schematic embodiment drawings and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application. In the drawings:

[0056] Figure 1 is a schematic diagram of a big data platform provided in an embodiment of the present application;

[0057] Figure 2 It is a flowchart of a method for building a big data platform provided in an embodiment of the present application;

[0058] Figure 3 is a schematic diagram of another big data platform provided in an embodiment of the present application;

[0059] Figure 4 is a schematic diagram of a big data platform construction device provided in an embodiment of the present application;

[0060] Figure 5 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0062] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein.

[0063] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0064] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0065] It should be understood that in the present application, "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0066] It should be understood that in the present application, "plurality" refers to two or more than two. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0067] It should be understood that in this application, "B corresponding to A", "B corresponding to A", "A corresponds to B" or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based only on A, but B can also be determined based on A and / or other information. A and B match when the similarity between A and B is greater than or equal to a preset threshold.

[0068] Depending on the context, "if" as used herein may be interpreted as "when" or "when" or "in response to determining" or "in response to detecting."

[0069] The data involved in this application may be data authorized by the tester or fully authorized by all parties. The collection, dissemination, and use of the data shall comply with the relevant laws, regulations, and standards of the relevant countries and regions. The implementation methods / examples of this application may be combined with each other.

[0070] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0071] To facilitate understanding of this application, the key words in this application are explained as follows:

[0072] Xinchuang: Innovation in information technology applications, aimed at achieving independent and controllable information technology and avoiding risks.

[0073] MRS: MapReduce Service (MRS for short) is a service for deploying and managing Hadoop systems on Huawei Cloud. It is currently a big data infrastructure platform with a relatively high market share in private deployment.

[0074] x86 processor: A processor architecture based on CISC (Complex Instruction Set Computer), which has a complex instruction set and longer instructions, thereby allowing greater flexibility and wider software support.

[0075] ARM processor: A processor architecture based on RISC (Reduced Instruction Set Computer). The ARM processor has a simpler instruction set and shorter instruction length, so it can achieve higher performance and lower power consumption.

[0076] Next, the present application scheme will be described through specific embodiments in conjunction with the accompanying drawings.

[0077] In order to meet the needs of information and innovation, the big data platform should be deployed on a physical machine whose CPU (Central Processing Unit), operating system, and basic software all meet the requirements of information and innovation.

[0078] At present, the big data platform is built on a trusted computing server cluster, and the trusted computing server cluster is installed with the corresponding trusted computing operating system and trusted computing software layer to ensure that the big data platform as a whole meets the trusted computing requirements.

[0079] For example, MRS (MapReduce Service) is a big data infrastructure platform with a relatively high market share in private deployment. MRS can be configured on cloud servers with x86 processors or ARM processor CPUs, CentOS / KYLIN / UOS and other operating systems. Figure 1 In the MRS trusted computing deployment architecture diagram shown, the big data platform is built on a trusted computing server cluster, and the trusted computing server is installed with a trusted computing operating system and trusted computing software layer to ensure that the big data platform as a whole meets the trusted computing requirements.

[0080] However, the above-mentioned trusted server cluster can only expand the capacity of trusted servers, but cannot add non-trusted servers, which makes it difficult to support the construction of a trusted big data platform on a hybrid cluster composed of trusted servers and non-trusted servers.

[0081] See also Figure 2 , Figure 2 A flowchart of a method for building a big data platform provided in an embodiment of the present application. The method includes the following steps:

[0082] Step S201: configuring a first server cluster and a second server cluster.

[0083] Among them, the first server cluster is a cluster composed of trusted computing servers.

[0084] The second server cluster is a cluster composed of servers other than the trusted servers. The server types in the second server cluster include entry-level servers, workgroup-level servers, department-level servers, enterprise-level servers, Web servers, mail servers, DNS servers, proxy servers, FTP servers, and original servers, etc.

[0085] Among them, entry-level servers usually use only one CPU, and configure corresponding memory and large-capacity IDE hard disk as needed. IDE RAID technology will also be used for data protection when necessary. This type of server is mainly aimed at users based on network operating systems such as Windows NT and NetWare, meeting the needs of office-type small and medium-sized network users for file sharing, printing services, data processing, Internet access and simple database applications. Workgroup-level servers generally support 1 to 2 PⅢ processors or a single P4 processor, can support large-capacity ECC memory, and are comprehensive in functions. Department-level servers and enterprise-level servers provide higher performance and reliability to meet the needs of large enterprises or departments. Web servers are used to host websites and applications and process HTTP requests and responses. Mail servers are used to handle email communications, including sending, receiving and storing emails. DNS servers are used to resolve domain names and map domain names to IP addresses so that users can access websites and services. Proxy servers are used to proxy client requests and communicate with other servers. FTP servers are used for file transfer and sharing using the File Transfer Protocol (FTP). An origin server is typically a server used to host content such as a website, video, or application. It provides the actual content and can be cached and delivered through a proxy server or content delivery network (CDN).

[0086] The above-mentioned trusted computing servers and non-trusted computing servers can be divided according to different application levels and functions to meet various needs from personal office to large-scale enterprise-level applications.

[0087] This application aims to address the limitation that existing big data platforms only support deployment on trusted computing server clusters, and proposes a technology that can efficiently deploy and run a big data basic platform on a hybrid environment cluster that includes non-trusted computing servers and trusted computing servers.

[0088] Step S202: corresponding software is configured for the first server cluster and the second server cluster respectively, and first software corresponding to the first server cluster and second software corresponding to the second server cluster are obtained.

[0089] like Figure 3As shown, the big data platform is generally divided into a management and control platform (Manager) layer and a service component layer (Service). The present application mainly executes the steps of configuring corresponding software for the first server cluster and the second server cluster respectively through the management and control module and the automated operation and maintenance module in the management and control platform, and obtaining the first software corresponding to the first server cluster and the second software corresponding to the second server cluster.

[0090] In one embodiment, corresponding software is configured for the first server cluster and the second server cluster respectively, and the first software corresponding to the first server cluster and the second software corresponding to the second server cluster are obtained, including: obtaining the first software information corresponding to the first server cluster and the second software information corresponding to the second server cluster through the configured intelligent management algorithm, and then generating a first installation instruction corresponding to the first software information and a second installation instruction corresponding to the second software information respectively, and then sending the first installation instruction to the first server cluster, and making the servers in the first server cluster perform software installation according to the first installation instruction and the first software information, and obtain the first software corresponding to the first server cluster, and finally sending the second installation instruction to the second server cluster, and making the servers in the second server cluster perform software installation according to the second installation instruction and the second software information, and obtain the second software corresponding to the second server cluster.

[0091] Among them, obtaining the first software information corresponding to the first server cluster through the configured intelligent management algorithm includes: obtaining and identifying the cluster type of the first cluster; determining the server type in the first server cluster based on the cluster type of the first cluster; and generating the first software information corresponding to the first server cluster based on the server type in the first server cluster.

[0092] The management and control module in the management and control platform is configured with an intelligent management algorithm. This intelligent management algorithm is a specific judgment logic, that is, first identify the cluster type of the first server cluster in the first cluster. Since this application includes two cluster types, one is a trusted server cluster and the other is a non-trusted server cluster, the corresponding server type within the cluster can be determined according to the cluster type, that is, information servers are configured in the trusted server cluster, and non-trusted servers are configured in the non-trusted server cluster.

[0093] Specifically, the server type in the server cluster may be determined based on the following function:

[0094] function determineMachineType(CPUModel,OSVersion): Define a function to determine the server type

[0095] if CPUModel belongs to the Xinchuang CPU list and OSVersion belongs to the Xinchuang OS list:

[0096] return"Xinchuang"

[0097] else:

[0098] return"Non-trusted innovation"

[0099] Through the above function, you can get from the CPU list whether the server is a trusted or non-trusted server.

[0100] Since different server clusters correspond to different software information, it is necessary to obtain corresponding software information based on the server type, that is, to generate first software information corresponding to the first server cluster based on the server type in the first server cluster.

[0101] After obtaining the first software information through the control module in the management control platform, a first installation instruction corresponding to the first software information will be generated, and then the control module will send the first installation instruction to the automated operation and maintenance module. The automated operation and maintenance module will send the received first installation instruction to the first server cluster, and enable the servers in the first server cluster to perform software installation according to the first installation instruction and the first software information, and obtain the first software corresponding to the first server cluster.

[0102] It should be noted that the method of obtaining the second software corresponding to the second server cluster is similar to the above-mentioned method of obtaining the first software, which will not be repeated here.

[0103] The first software includes basic software and / or service software, and the first software information includes the basic software package, the service software package name and the download URL.

[0104] Specifically, a first installation instruction is sent to a first server cluster, and servers in the first server cluster are enabled to perform software installation according to the first installation instruction and the first software information to obtain the first software corresponding to the first server cluster, including: sending the first installation instruction to the first server cluster; obtaining the service software package through the service software package name and the download URL; based on the first installation instruction, servers in the first server cluster are enabled to perform the installation of the basic software package and / or the service software package, and obtain the basic software and / or service software corresponding to the first server cluster.

[0105] After the first installation instruction is sent to the first server cluster, the service software package will be obtained through the service software package name and the download URL.

[0106] In one embodiment, a function may be defined to automatically concatenate the service package name and download URL functionbuildPackageInfo(machineType):

[0107] if machineType == "Xinchuang":

[0108] packageName = "Xinchuang Edition Service Software Package Name"

[0109] downloadURL = "Xinchuang Edition Service Software Package Download URL Address / Xinchuang Edition Service Software Package Name"

[0110] else: / / Non - Xinchuang

[0111] packageName = "Non - Xinchuang Edition Service Software Package Name"

[0112] downloadURL = "Non - Xinchuang Edition Service Software Package Download URL Address / Non - Xinchuang Edition Service Software Package Name return {packageName, downloadURL}

[0113] The corresponding service software package can be obtained by concatenating the service software package name and the download URL included in the software information. That is, the corresponding first service software package can be obtained by concatenating the service software package name and the download URL in the first software information, and the corresponding second service software package can be obtained by concatenating the service software package name and the download URL in the second software information.

[0114] Specifically, to obtain the service software package through the service software package name and the download URL, it includes: concatenating the service software package name and the download URL to get the concatenated download address, and obtaining the service software package through the concatenated download address.

[0115] After that, based on the installation instruction, the servers in the server cluster execute the installation of the basic software package and / or the service software package, and obtain the corresponding basic software and / or service software of the server cluster.

[0116] Among them, based on the first installation instruction, the servers in the first server cluster execute the installation of the basic software package and / or the service software package, and obtain the corresponding basic software and / or service software of the first server cluster, including: initializing the servers in the first server cluster; if the initialization of the servers in the first server cluster is successful, based on the first installation instruction, the servers in the first server cluster execute the installation of the basic software package and / or the service software package, and obtain the corresponding basic software and / or service software of the first server cluster.

[0117] If the initialization of the servers in the first server cluster is not successful, troubleshoot the servers in the first server cluster and return to the step of initializing the servers in the first server cluster.

[0118] Among them, troubleshooting for servers in the server cluster includes but is not limited to hardware inspection, system log analysis, network connection testing, etc.

[0119] Similarly, the method of obtaining the basic software and / or service software corresponding to the second server cluster is similar to the method of obtaining the basic software and / or service software corresponding to the first server cluster, which will not be repeated here.

[0120] Among them, the present application can obtain the basic software package based on the server type in the first server cluster.

[0121] In another embodiment, corresponding software is configured for the first server cluster. To obtain the first software corresponding to the first server cluster, it is necessary to first determine whether the servers in the first server cluster are of the same type. If the servers in the first server cluster are of the same type, the codes of the servers in the first server cluster are obtained, the codes of the servers in the first server cluster are combined to obtain an unlocking code, and then the first software corresponding to the first server cluster is obtained from the software configuration file through the unlocking code.

[0122] Since different types of servers are configured with different software, it is necessary to identify and judge the types of servers configured in the server cluster to ensure the accuracy of the corresponding software installation.

[0123] When the servers in the first server cluster are identified as servers of the same type, the code of the server can be obtained. For example, if three trusted computing servers are configured in the first server cluster, their codes will be obtained respectively. The codes can be represented by at least one of numbers, letters, etc. For example, the codes of the three trusted computing servers are 1, 2 and 3 respectively.

[0124] After obtaining the codes of the three trusted computing servers, the codes of each server need to be combined to obtain an unlocking code. The combination method can be permutation, calculation, etc. For example, the unlocking code obtained by directly combining is 123.

[0125] Among them, obtaining the first software corresponding to the first server cluster from the software configuration file through the unlocking code includes: unlocking the software configuration file using the unlocking code to obtain a software set; obtaining digital information in the name of each software in the software set, and splicing the digital information in the name of each software in the software set with the unlocking code to obtain a splicing code; obtaining the first software corresponding to the first server cluster from the software configuration file through the splicing code and the encoding of each server in the first server cluster.

[0126] The software configuration file contains software corresponding to various server types, and the software corresponding to all server types is configured in the software collection.

[0127] After obtaining the unlock code, the software configuration file can be unlocked with the unlock code to obtain the software collection. Since the software collection is configured with software of various server types, in order to match each server to the corresponding software, the digital information in the name of each software in the software collection needs to be spliced ​​with the unlock code to obtain the splicing code.

[0128] For example, the software collection is configured with three software corresponding to the trusted computing server, which are called XC1, XC2 and XC3 respectively, and the software collection is configured with three software corresponding to the non-trusted computing server, which are called FXC4, FXC5 and FXC6 respectively.

[0129] When there are three trusted computing servers in the first server cluster and their codes are 1, 2 and 3 respectively, the unlocking code 123 is obtained by combining the codes of each server, and then the unlocking code 123 is spliced ​​with the numbers in each software in the software set to obtain splicing codes 1231, 1232, 1233, 1234, 1235 and 1236.

[0130] Among them, the first software corresponding to the first server cluster is obtained from the software configuration file through the splicing code and the encoding of each server in the first server cluster, including: obtaining the identification of each server in the first server cluster from the splicing code based on the encoding size of each server in the first server cluster; obtaining a preset identification interval, wherein each preset identification in the preset identification interval matches a corresponding software; if the identification of each server in the first server cluster is in the preset identification interval, obtaining the software matching the identification of each server from the software configuration file based on the identification of each server in the first server cluster; determining the first software corresponding to the first server cluster through the software matching the identification of each server.

[0131] After obtaining the splicing codes 1231, 1232, 1233, 1234, 1235 and 1236, the identification of the trusted server with the code 3 is obtained from the splicing code starting from the largest code 3 of the three trusted servers. Here, it is only necessary to check whether the last digit of the splicing code is the same as the trusted server code. If they are the same, the splicing code is selected as the identification of this trusted server. Specifically, the identification of the trusted server with the code 2 is 1232, and the identification of the trusted server with the code 1 is 1231.

[0132] Since the server identifier has a corresponding identifier interval (i.e., a preset identifier interval), if the server identifier is not within the identifier interval, the software corresponding to the server cannot be obtained and the server needs to be replaced; if the server identifier is within the identifier interval, the software matching the identifier of each server can be obtained from the software configuration file based on the identifier of each server in the first server cluster.

[0133] For example, the preset identification interval is [0, 25550]. It can be seen that the identifications 1232, 1231 and 1233 of the above-mentioned trusted computing server are all within this interval. Therefore, the matching software can be obtained in the software configuration file based on the identifications 1232, 1231 and 1233 of the trusted computing server.

[0134] Specifically, when the ID of the trusted server is 1231, its digits are added in pairs in a loop, and only the unit digit is retained each time, and the corresponding software is determined by the parity of the last obtained number. For example, if the ID of the trusted server is 1231, the first round of addition of two digits is 354, and then the two digits are added in pairs to get 89, and finally the odd number 7 is added. Since the last number is an odd number, any of the trusted servers XC1 and XC3 can be selected from the software configuration file as the matching software.

[0135] Through the above method, each trusted computing server can be matched to the corresponding software, and then all the acquired software can be aggregated to obtain the first software corresponding to the first server cluster.

[0136] Similarly, the method for obtaining the second software corresponding to the second server cluster is similar to the method for obtaining the first software, which will not be described in detail here.

[0137] Step S203: If the first software and the second software are of the same software type but different versions, the first cluster is composed of the first server cluster, the first software and the first operating system, and the second cluster is composed of the second server cluster, the second software and the second operating system, so as to support the construction of an innovative big data platform on the hybrid cluster composed of the first cluster and the second cluster.

[0138] Among them, software types include but are not limited to basic software, middleware, application software and embedded software.

[0139] After obtaining the first software and the second software through the above embodiment, since it is necessary to build and run the trusted big data platform on the hybrid cluster, it is also necessary to consider whether the software types of the first software and the second software are the same and whether the software versions are the same. Only when the software types of the first software and the second software are the same but the versions are different, can the trusted big data platform be implemented on the hybrid cluster.

[0140] In order to determine whether the software types and versions of the first software and the second software are the same, first identify whether the types of the first software and the second software are the same. If the types of the first software and the second software are the same, then identify whether the versions of the first software and the second software are the same. If the versions of the first software and the second software are also the same, the first cluster is composed of the first server cluster, the first software and the first operating system, and the second cluster is composed of the second server cluster, the second software and the second operating system, so as to support the construction of a big data platform for information innovation on the hybrid cluster composed of the first cluster and the second cluster. Among them, the software version can be distinguished by the version number.

[0141] The services provided by the big data platform can be divided into two types according to the source code type: C / C++ and Java / Scala. Among them, the services developed based on C / C++ need to be recompiled to generate binary executable programs under different CPU architectures because of incompatible instruction sets. The services developed based on Java / Scala adopted in this application have cross-platform compatibility and can be run directly without recompilation.

[0142] The embodiment of the present application provides a method for building a big data platform, including: first configuring a first server cluster and a second server cluster, wherein the first server cluster is a cluster composed of trusted servers, and the second server cluster is a cluster composed of servers other than trusted servers, and then respectively configuring corresponding software for the first server cluster and the second server cluster, obtaining the first software corresponding to the first server cluster and the second software corresponding to the second server cluster, if the software types of the first software and the second software are the same but the versions are different, the first cluster is composed of the first server cluster, the first software and the first operating system, and the second cluster is composed of the second server cluster, the second software and the second operating system, so as to support the construction of a trusted big data platform on a hybrid cluster composed of the first cluster and the second cluster. The present application can add a non-trusted server cluster on the basis of the trusted server cluster to support the construction of a trusted big data platform on a hybrid cluster composed of trusted servers and non-trusted servers, thereby improving the flexibility of server configuration. In addition, the trusted big data platform is constructed by a hybrid cluster composed of trusted servers and non-trusted servers, thereby avoiding the configuration of a large number of the same servers, reducing the number of servers, and reducing the cost of platform construction.

[0143] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0144] The following is an embodiment of the device of the present application. For details not described in detail, please refer to the corresponding method embodiment described above.

[0145] Figure 4 A schematic diagram of the structure of a big data platform construction device provided in an embodiment of the present application is shown. For the sake of convenience, only the parts related to the embodiment of the present application are shown. A big data platform construction device includes a configuration module 401, a software acquisition module 402 and a platform construction module 403, which are as follows:

[0146] Configuration module 401, used to configure a first server cluster and a second server cluster, wherein the first server cluster is a cluster composed of ICT servers, and the second server cluster is a cluster composed of servers other than ICT servers;

[0147] The software acquisition module 402 is used to configure corresponding software for the first server cluster and the second server cluster respectively, and acquire the first software corresponding to the first server cluster and the second software corresponding to the second server cluster;

[0148] The platform construction module 403 is used to support the construction of an innovative big data platform on a hybrid cluster consisting of the first cluster and the second cluster, if the first software and the second software are of the same software type but different versions, by forming a first cluster consisting of a first server cluster, a first software and a first operating system, and forming a second cluster consisting of a second server cluster, a second software and a second operating system.

[0149] In an embodiment, the software acquisition module 402 is further used to acquire first software information corresponding to the first server cluster and second software information corresponding to the second server cluster through a configured intelligent management algorithm;

[0150] respectively generating a first installation instruction corresponding to the first software information and a second installation instruction corresponding to the second software information;

[0151] Sending the first installation instruction to the first server cluster, and causing the servers in the first server cluster to perform software installation according to the first installation instruction and the first software information, and obtaining the first software corresponding to the first server cluster;

[0152] The second installation instruction is sent to the second server cluster, and the servers in the second server cluster are enabled to perform software installation according to the second installation instruction and the second software information, so as to obtain the second software corresponding to the second server cluster.

[0153] In one embodiment, the software acquisition module 402 is further used to acquire and identify the cluster type of the first cluster;

[0154] determining a server type in the first server cluster based on the cluster type of the first cluster;

[0155] First software information corresponding to the first server cluster is generated based on the server types in the first server cluster.

[0156] In one embodiment, the first software includes basic software and / or service software, and the first software information includes the basic software package, the service software package name and the download URL;

[0157] The software acquisition module 402 is also used to send the first installation instruction to the first server cluster;

[0158] Obtain the service package by using the service package name and download URL;

[0159] Based on the first installation instruction, the servers in the first server cluster are enabled to install the basic software package and / or the service software package, and obtain the basic software and / or service software corresponding to the first server cluster.

[0160] In one embodiment, the software acquisition module 402 is further used to initialize the servers in the first server cluster;

[0161] If the servers in the first server cluster are successfully initialized, the servers in the first server cluster are enabled to install the basic software package and / or the service software package based on the first installation instruction, and obtain the basic software and / or service software corresponding to the first server cluster.

[0162] In one embodiment, the software acquisition module 402 is further used to determine whether the servers in the first server cluster are servers of the same type;

[0163] If the servers in the first server cluster are of the same type, obtaining a code of each server in the first server cluster;

[0164] Combining the codes of the servers in the first server cluster to obtain an unlocking code;

[0165] The first software corresponding to the first server cluster is obtained from the software configuration file through the unlocking code.

[0166] In one embodiment, the software acquisition module 402 is further used to unlock the software configuration file using the unlock code to obtain the software set;

[0167] Obtaining digital information in the name of each software in the software collection, and concatenating the digital information in the name of each software in the software collection with the unlock code to obtain a concatenated code;

[0168] The first software corresponding to the first server cluster is obtained from the software configuration file by concatenating the code and the codes of the servers in the first server cluster.

[0169] In an embodiment, the software acquisition module 402 is further used to acquire the identifier of each server in the first server cluster from the splicing code based on the code size of each server in the first server cluster;

[0170] Obtaining preset identification differences, wherein each preset identification in the preset identification interval is matched with corresponding software;

[0171] If the identifiers of the servers in the first server cluster are in the preset identifier interval, obtaining software matching the identifiers of the servers from the software configuration file based on the identifiers of the servers in the first server cluster;

[0172] The first software corresponding to the first server cluster is determined by the software matched by the identifiers of the servers.

[0173] In one embodiment, the device further includes a troubleshooting module, which is used to troubleshoot the servers in the first server cluster and return to the step of initializing the servers in the first server cluster if the initialization of the servers in the first server cluster fails.

[0174] In one embodiment, the software acquisition module 402 is further used to concatenate the service software package name and the download URL to obtain a concatenated download address;

[0175] Obtain the service software package through the concatenated download address.

[0176] In an embodiment, the device further includes a basic software package acquisition module, which is used to acquire the basic software package based on the server type in the first server cluster.

[0177] The embodiment of the present application provides a big data platform construction device, which can be used to: first configure the first server cluster and the second server cluster, wherein the first server cluster is a cluster composed of trusted servers, and the second server cluster is a cluster composed of servers other than trusted servers, and then respectively configure the corresponding software for the first server cluster and the second server cluster, obtain the first software corresponding to the first server cluster and the second software corresponding to the second server cluster, if the software types of the first software and the second software are the same but the versions are different, the first cluster is composed of the first server cluster, the first software and the first operating system, and the second cluster is composed of the second server cluster, the second software and the second operating system, so as to support the construction of a trusted big data platform on a hybrid cluster composed of the first cluster and the second cluster. The present application can add a non-trusted server cluster on the basis of the trusted server cluster to support the construction of a trusted big data platform on a hybrid cluster composed of trusted servers and non-trusted servers, thereby improving the flexibility of server configuration. In addition, the trusted big data platform is constructed by a hybrid cluster composed of trusted servers and non-trusted servers, thereby avoiding the configuration of a large number of the same servers, reducing the number of servers, and reducing the cost of platform construction.

[0178] This application Figure 5 A schematic diagram of a computer device is provided. Figure 5 As shown, the computer device 5 of this embodiment includes: a processor 501, a memory 502, and steps in the big data platform construction method embodiment stored in the memory 502 and executable on the processor 501, such as Figure 2 Alternatively, when the processor 501 executes the computer program 503, the functions of each module / unit in the above-mentioned big data platform construction device embodiments are implemented, for example Figure 4 Functionality of modules / units 401 to 403 shown.

[0179] The present application also provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the big data platform construction method provided by the various embodiments described above.

[0180] Among them, the readable storage medium can be a computer storage medium or a communication medium. The communication medium includes any medium that facilitates the transmission of a computer program from one place to another. The computer storage medium can be any available medium that can be accessed by a general or special-purpose computer. For example, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application-specific integrated circuit (ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the readable storage medium can also exist in a communication device as discrete components. The readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0181] The present application also provides a program product, which includes an execution instruction, which is stored in a readable storage medium. At least one processor of the device can read the execution instruction from the readable storage medium, and at least one processor executes the execution instruction so that the device implements the big data platform construction method provided by the various embodiments described above.

[0182] In the embodiments of the above-mentioned devices, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0183] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for constructing a big data platform, characterized in that: include: Configure a first server cluster and a second server cluster, wherein the first server cluster is a cluster consisting of ICT servers, and the second server cluster is a cluster consisting of servers other than the ICT servers; configuring corresponding software for the first server cluster and the second server cluster respectively, and obtaining first software corresponding to the first server cluster and second software corresponding to the second server cluster; If the first software and the second software are of the same software type but different versions, the first cluster is composed of the first server cluster, the first software and the first operating system, and the second cluster is composed of the second server cluster, the second software and the second operating system, so as to support the construction of an innovative big data platform on a hybrid cluster composed of the first cluster and the second cluster.

2. The method for constructing a big data platform according to claim 1, characterized in that: The configuring corresponding software for the first server cluster and the second server cluster respectively, and obtaining first software corresponding to the first server cluster and second software corresponding to the second server cluster, includes: Acquire first software information corresponding to the first server cluster and second software information corresponding to the second server cluster through a configured intelligent management algorithm; respectively generating a first installation instruction corresponding to the first software information and a second installation instruction corresponding to the second software information; Sending the first installation instruction to the first server cluster, and causing the servers in the first server cluster to perform software installation according to the first installation instruction and the first software information, to obtain the first software corresponding to the first server cluster; The second installation instruction is sent to the second server cluster, and the servers in the second server cluster are enabled to perform software installation according to the second installation instruction and the second software information, so as to obtain the second software corresponding to the second server cluster.

3. The method for constructing a big data platform as claimed in claim 2, characterized in that: The acquiring the first software information corresponding to the first server cluster by using the configured intelligent management algorithm includes: Obtaining and identifying a cluster type of the first cluster; determining a server type in the first server cluster based on a cluster type of the first cluster; First software information corresponding to the first server cluster is generated based on the server types in the first server cluster.

4. The method for constructing a big data platform as claimed in claim 3, characterized in that: The first software includes basic software and / or service software, and the first software information includes the name of the basic software package, the service software package and a download URL; The sending of the first installation instruction to the first server cluster, and causing the servers in the first server cluster to perform software installation according to the first installation instruction and the first software information, to obtain the first software corresponding to the first server cluster, includes: Sending the first installation instruction to the first server cluster; Obtain the service software package through the service software package name and download URL; Based on the first installation instruction, the servers in the first server cluster are enabled to install the basic software package and / or the service software package, and obtain the basic software and / or service software corresponding to the first server cluster.

5. The method for constructing a big data platform as claimed in claim 4, characterized in that: The step of causing the servers in the first server cluster to execute the installation of the basic software package and / or the service software package based on the first installation instruction and obtaining the basic software and / or service software corresponding to the first server cluster includes: Initializing the servers in the first server cluster; If the servers in the first server cluster are successfully initialized, the servers in the first server cluster execute the installation of the basic software package and / or the service software package based on the first installation instruction, and obtain the basic software and / or service software corresponding to the first server cluster.

6. The method for constructing a big data platform according to claim 1, characterized in that: The configuring corresponding software for the first server cluster and obtaining first software corresponding to the first server cluster includes: Determining whether the servers in the first server cluster are servers of the same type; If the servers in the first server cluster are of the same type, obtaining a code of each server in the first server cluster; Combining the codes of the servers in the first server cluster to obtain an unlocking code; The first software corresponding to the first server cluster is obtained from a software configuration file through the unlock code.

7. The method for constructing a big data platform as claimed in claim 6, characterized in that: The obtaining the first software corresponding to the first server cluster from the software configuration file by using the unlock code includes: Unlocking the software configuration file using the unlock code to obtain a software set; Obtaining digital information in the name of each software in the software collection, and concatenating the digital information in the name of each software in the software collection with the unlock code to obtain a concatenated code; The first software corresponding to the first server cluster is obtained from the software configuration file through the splicing code and the codes of the servers in the first server cluster.

8. The method for constructing a big data platform as claimed in claim 7, characterized in that: The obtaining, from the software configuration file, the first software corresponding to the first server cluster by using the splicing code and the codes of each server in the first server cluster, comprises: Obtaining an identifier of each server in the first server cluster from the splicing code based on the code size of each server in the first server cluster; Obtaining preset identification differences, wherein each preset identification in the preset identification interval is matched with corresponding software; If the identifiers of the servers in the first server cluster are in a preset identifier interval, obtaining software matching the identifiers of the servers from the software configuration file based on the identifiers of the servers in the first server cluster; The first software corresponding to the first server cluster is determined by the software matched by the identifiers of the servers.

9. The method for constructing a big data platform as claimed in claim 5, characterized in that: The method further comprises: If the initialization of the servers in the first server cluster fails, the servers in the first server cluster are checked and the process returns to execute the step of initializing the servers in the first server cluster.

10. The method for constructing a big data platform according to claim 4, characterized in that: The obtaining of the service software package by using the service software package name and the download URL includes: Concatenate the service software package name and the download URL to obtain a concatenated download address; The service software package is obtained through the combined download address.

11. The method for constructing a big data platform according to claim 4, characterized in that: The method further comprises: The basic software package is obtained based on the server type in the first server cluster.

12. A big data platform construction device, characterized in that: include: A configuration module, used to configure a first server cluster and a second server cluster, wherein the first server cluster is a cluster composed of ICT servers, and the second server cluster is a cluster composed of servers other than the ICT servers; A software acquisition module, configured to configure corresponding software for the first server cluster and the second server cluster respectively, and acquire first software corresponding to the first server cluster and second software corresponding to the second server cluster; A platform building module is used to support the construction of an information and communication technology big data platform on a hybrid cluster consisting of the first cluster and the second cluster, if the first software and the second software are of the same software type but different versions, by forming a first cluster consisting of the first server cluster, the first software and the first operating system, and forming a second cluster consisting of the second server cluster, the second software and the second operating system.

13. A computer device, characterized in that: comprising a memory, and one or more processors communicatively connected to the memory; The memory stores instructions that can be executed by the one or more processors, and the instructions are executed by the one or more processors so that the one or more processors implement the method for building a trusted big data platform as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that: It includes programs or instructions, which, when executed on a computer, implement the method for constructing an innovative big data platform as described in any one of claims 1 to 11.

Images