Method and device for building a power big data analysis platform outside the MRS cluster

Abstract

The present invention discloses a method and device for building a power big data analysis platform outside the MRS cluster. The method includes: preparing the basic environment for the power big data analysis platform according to the cloud environment where the MRS cluster is deployed; synchronizing the time of the power big data analysis platform with the time of the Master node; installing the MRS cluster client on the nodes outside the cluster, creating cluster users in the management console and granting corresponding permissions, and authorizing the users to log in to the client on the power big data analysis platform for mutual identity verification; analyzing the node environment configuration and development environment deployment of the analysis platform, and installing and deploying the integrated development environment. The analysis platform built by using this method is general and replicable, and can be packaged into an image with Docker containers and directly used on other nodes outside the cluster. Through this method, developers can quickly realize the ability to access all resources of the data middle platform without cumbersome environment deployment, reducing the learning and development costs.

Description

Technical Field

[0001] The present invention belongs to the technical field of electric power, and in particular relates to a method and device for building a power big data analysis platform outside the MRS cluster. Background Art

[0002] Power big data is an important means for the State Grid Corporation of China to seize the opportunities of digital new infrastructure construction and promote the digital transformation of the power grid. The company and provincial power companies have all built data middle platforms, which almost collect all data in various fields within the company, including equipment, operation, human resources, etc., and have built an MRS cluster. The management mode of power grid data is unified management and application on demand, that is, the company uniformly stores and operates and maintains the data in the middle platform, and each branch company applies to the data middle platform management department for access to the corresponding data tables according to its own needs.

[0003] The data middle platform operation and maintenance management department is only responsible for storing, operating and maintaining, and managing data. That is, when data users submit access applications, corresponding table access permissions are granted to users, and it is not responsible for data access of users, data table access, and data development environment deployment. On the other hand, due to the complex data storage structure of the data middle platform, there is data stored in the traditional relational database PostgresSql, and there is also data stored in the Hadoop distributed file system HDFS (Hadoop Distributed File System). Moreover, the access of each grass-roots unit is outside the big data cluster of the data middle platform. It is extremely difficult to deploy service clients, configure the environment, authenticate the identity, and develop data outside the cluster, which is also the main difficulty restricting the grass-roots units from carrying out big data analysis work. Summary of the Invention

[0004] The present invention provides a method and device for building a power big data analysis platform outside the MRS cluster, which is used to solve at least one of the above technical problems.

[0005] In the first aspect, the present invention provides a method for building a power big data analysis platform outside the MRS cluster, including: preparing the basic environment for the power big data analysis platform according to the cloud environment where the MRS cluster is deployed, and the basic environment preparation includes the selection and installation of the operating system image, the configuration of the basic development and operation environment, and the configuration of the component installation tool; synchronizing the time of the power big data analysis platform with the time of the Master node, and the synchronization includes installing the ntp service of the power big data analysis platform according to the command yum - y install ntp; modifying server xx.xx.xx.xx prefer in the ntp.conf file according to the command vi / etc / ntp.conf, and pointing the server to the master node, where xx.xx.xx.xx is the IP address of the time synchronization server of the Mater node; starting the ntp service according to the command service ntpd start, completing the time synchronization between the analysis platform and the master node, and checking the time synchronization status according to ntpstat to confirm the successful time synchronization; installing the MRS cluster client on the external node of the cluster, creating a cluster user in the management console and granting corresponding permissions, and authorizing the user to log in to the client on the power big data analysis platform for mutual authentication; configuring the node environment and deploying the development environment of the analysis platform, and installing and deploying the integrated development environment.

[0006] In the second aspect, the present invention provides a device for building a power big data analysis platform outside the MRS cluster, including: a preparation module configured to prepare the basic environment for the power big data analysis platform according to the cloud environment where the MRS cluster is deployed, and the basic environment preparation includes the selection and installation of the operating system image, the configuration of the basic development and operation environment, and the configuration of the component installation tool; a synchronization module configured to synchronize the time of the power big data analysis platform with the time of the Master node, and the synchronization includes installing the ntp service of the power big data analysis platform according to the command yum - y install ntp; modifying server xx.xx.xx.xx prefer in the ntp.conf file according to the command vi / etc / ntp.conf, and pointing the server to the master node, where xx.xx.xx.xx is the IP address of the time synchronization server of the Mater node; starting the ntp service according to the command service ntpd start, completing the time synchronization between the analysis platform and the master node, and checking the time synchronization status according to ntpstat to confirm the successful time synchronization; a verification module configured to install the MRS cluster client on the external node of the cluster, create a cluster user in the management console and grant corresponding permissions, and authorize the user to log in to the client on the power big data analysis platform for mutual authentication; an integration module configured to configure the node environment and deploy the development environment of the analysis platform, and install and deploy the integrated development environment.

[0007] In a third aspect, an electronic device is provided, which includes: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the steps of the method for building a power big data analysis platform outside the MRS cluster according to any embodiment of the present invention.

[0008] In a fourth aspect, the present invention further provides a computer-readable storage medium, on which a computer program is stored, and the computer program includes program instructions, and when the program instructions are executed by a computer, the computer is enabled to execute the steps of the method for building a power big data analysis platform outside the MRS cluster according to any embodiment of the present invention.

[0009] The method and device for building a power big data analysis platform outside the MRS cluster of the present application adopt steps of preparing the basic environment, installing and authenticating the cluster client, and deploying the development tools to build the power big data analysis platform, making the power big data analysis platform general and replicable, and it can be packaged into an image with Docker containers and directly used on other nodes outside the cluster. Through this method, developers can quickly realize the access capabilities of all resources in the data middle platform without cumbersome environment deployment, reducing the learning and development costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0011] Figure 1 It is a flowchart of a method for building a power big data analysis platform outside the MRS cluster provided by an embodiment of the present invention;

[0012] Figure 2 It is a flowchart of another method for building a power big data analysis platform outside the MRS cluster provided by an embodiment of the present invention;

[0013] Figure 3 It is a structural block diagram of a device for building a power big data analysis platform outside the MRS cluster provided by an embodiment of the present invention;

[0014] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0016] Please refer to Figure 1 , which shows a flowchart of a method for building a power big data analysis platform outside the MRS cluster of this application.

[0017] As Figure 1 shown, the method for building a power big data analysis platform outside the MRS cluster specifically includes the following steps:

[0018] Step S101, perform basic environment preparation for the power big data analysis platform according to the cloud environment where the MRS cluster is deployed.

[0019] It should be noted that the basic environment preparation includes the installation of the operating system, the selective installation of development language packages (including JDK or Python), and the installation of component installation tools. The operating system version is Linux, Redhat 7.6 or above; the server side and the client side only support the built-in OpenJDK, with the version of 1.8.0_272, and replacement is not allowed; the Python environment supports Python / Python3, where the Python version requirement is not lower than 2.6.6 and the highest does not exceed 2.7.13. In addition, for the preparation of the component installation tool setuptools, it should be noted that the basic configuration of the Python3 development environment is required, with the version requirement of 47.3.1.

[0020] Step S102, synchronize the time of the power big data analysis platform with the time of the Master node.

[0021] In this embodiment, the synchronization includes installing the ntp service of the power big data analysis platform according to the command yum - y install ntp; modifying server xx.xx.xx.xx prefer in the ntp.conf file according to the command vi / etc / ntp.conf to point the server to the master node, where xx.xx.xx.xx is the IP address of the time synchronization server of the Mater node; starting the ntp service according to the command service ntpd start to complete the time synchronization between the analysis platform and the master node, and checking the time synchronization status according to ntpstat to confirm the successful time synchronization.

[0022] It should be noted that achieving the time synchronization between the power big data analysis platform and the Master node can ensure that there is no lag phenomenon when nodes outside the cluster and the cluster execute the same task.

[0023] Step S103: Install the MRS cluster client on the nodes outside the cluster, create a cluster user in the management console and grant corresponding permissions, and authorize the user to log in to the client on the power big data analysis platform for mutual identity verification.

[0024] In this embodiment, the authorization for users to log in to the client on the power big data analysis platform for mutual identity verification specifically includes: Step 1: User preparation and authorization: In the MRS (MapReduce Service) cluster, use the cluster administrator account to create a user group in the console and grant the corresponding permissions for the MRS service; create a user in the console and add it to the created and authorized user group; the user logs in and verifies the permissions, logs in to the console with the newly created user, switches to the authorized area, and verifies the permissions. Step 2: Client installation and environment variable configuration: Download the MRS client in the console, send it to the specified server in the analysis platform server, and after installation, switch to the client installation directory and configure the environment variables at one key through the command "source bigdata_env". Step 3: Through Kerberos security authentication: Use Kerberos authentication to adopt a "client / server" structure and encryption technologies such as AES, so that both the client and the server can authenticate each other, and verify through the command "kinit MRS cluster user" (for example, "kinit admin"). In this way, developers outside the cluster can log in using the user account and access the data and resources of the data middle platform after passing the security authentication.

[0025] Step S104: Configure the node environment of the analysis platform and deploy the development environment, and install and deploy the integrated development environment.

[0026] It should be noted that the power big data analysis platform supports multiple development environments with strong compatibility, including Java, Python, etc. Learners and developers can choose to install Java, scala, Python and development tools according to their own needs. According to the components installed in the client in Step 3, big data components such as zookeeper, hadoop, spark, and Hbase can be started at one key.

[0027] In summary, the method of this application prepares the basic environment according to the cloud environment in which the MRS cluster is deployed; installs the NTP (Network Time Protocol) service, configures the network time synchronization configuration file, and realizes network time synchronization with the Master node server in the cluster; installs the MRS cluster client on the external nodes of the cluster, creates cluster users in the management console and grants corresponding permissions, and authorizes the users to log in to the client in the analysis platform for mutual authentication; configures the node environment of the analysis platform and deploys the development environment, and installs and deploys the integrated development environment (IDE). In this way, by using the steps of preparing the basic environment, installing and authenticating the cluster client, and deploying the development tools to build the power big data analysis platform, the power big data analysis platform is universal and replicable, and can be packaged into an image with Docker containers and directly used on other external nodes of the cluster. Developers can quickly realize the access capabilities of all resources in the data middle platform through this method, without the need for cumbersome environment deployment, reducing the learning and development costs.

[0028] Please refer to Figure 2 , which shows the flowchart of another method for building a power big data analysis platform outside the MRS cluster of this application.

[0029] As Figure 2 shown, in step S201, obtain the data of the current big data middle platform, and the data includes dws library data and hive library data;

[0030] In step S202, perform missing value processing and noise data processing on the dws library data and the hive library data;

[0031] In step S203, perform correlation analysis on the processed dws library data and / or the hive library data according to the preset statistical method and the preset machine learning model.

[0032] In this embodiment, after the basic environment is set up, the data in the power big data data center is acquired and preprocessed. The data acquisition includes reading data from the DWS library and acquiring data from the Hive library. The data preprocessing includes missing value processing and noise data processing. For the processing of missing value data, three processing methods are preset, namely forward filling, backward filling, and mean filling. For the processing of noise data, common power data business rules are established to process the noise data that does not conform to the business rules. For example, the normal value of the distribution transformer operating voltage is about 220V. If noise data such as 10V or 2000V is collected for the distribution transformer voltage data, it will be automatically excluded. The preprocessed data is calculated, and multiple pieces of acquired data are correlated. The data codes are converted into easy-to-understand languages. For example, the voltage level obtained in the data table is AC00101, and its actual meaning is the AC 10kV voltage level. The preset statistical methods and preset machine learning models include linear regression, random forest, k-means algorithm, etc. Users can select the methods they need according to their own application analysis needs, which is very simple and efficient.

[0033] Please refer to Figure 3 , which shows the structural block diagram of a power big data analysis platform building device outside the MRS cluster of the present application.

[0034] As Figure 3 shown, the power big data analysis platform building device 300 includes a preparation module 230, a synchronization module 320, a verification module 330, and an integration module 340.

[0035] Among them, the preparation module 310 is configured to prepare the basic environment of the power big data analysis platform according to the cloud environment where the MRS cluster is deployed. The basic environment preparation includes the selection and installation of the operating system image, the configuration of the basic development and operation environment, and the configuration of the component installation tool; the synchronization module 320 is configured to synchronize the time of the power big data analysis platform with the time of the Master node. The synchronization includes installing the ntp service of the power big data analysis platform according to the command yum - y install ntp; modifying server xx.xx.xx.xxprefer in the ntp.conf file according to the command vi / etc / ntp.conf to point the server to the master node, where xx.xx.xx.xx is the IP address of the time synchronization server of the Mater node; starting the ntp service according to the command service ntpd start to complete the time synchronization between the analysis platform and the master node, and checking the time synchronization status according to ntpstat to confirm the successful time synchronization; the verification module 330 is configured to install the MRS cluster client on the external nodes of the cluster, create a cluster user in the management console and grant corresponding permissions, and authorize the user to log in to the client on the power big data analysis platform for mutual authentication; the integration module 340 is configured to configure the node environment of the analysis platform and deploy the development environment, and install and deploy the integrated development environment.

[0036] It should be understood that Figure 3 the modules described in Figure 1 correspond to the respective steps in the method described in Figure 3 Therefore, the operations, features, and corresponding technical effects described above for the method also apply to

[0037] In some other embodiments, the embodiments of the present invention further provide a computer - readable storage medium. The computer - readable storage medium stores computer - executable instructions, and the computer - executable instructions can execute the method for building a power big data analysis platform outside the MRS cluster in any of the above - mentioned method embodiments;

[0038] As an implementation manner, the computer - readable storage medium of the present invention stores computer - executable instructions, and the computer - executable instructions are set as:

[0039] Prepare the basic environment of the power big data analysis platform according to the cloud environment where the MRS cluster is deployed. The basic environment preparation includes the selection and installation of the operating system image, the configuration of the basic development and operation environment, and the configuration of the component installation tool;

[0040] Synchronize the time of the power big data analysis platform with the time of the Master node. The synchronization includes installing the ntp service of the power big data analysis platform according to the command yum - y install ntp; modifying server xx.xx.xx.xx prefer in the ntp.conf file according to the command vi / etc / ntp.conf, and pointing the server to the master node, where xx.xx.xx.xx is the IP address of the time synchronization server of the Mater node; starting the ntp service according to the command service ntpd start to complete the time synchronization between the analysis platform and the master node, and checking the time synchronization status according to ntpstat to confirm the successful time synchronization;

[0041] Install the MRS cluster client on the external nodes of the cluster, create cluster users in the management console and grant corresponding permissions, and authorize the users to log in to the client on the power big data analysis platform for mutual authentication;

[0042] Configure the node environment of the analysis platform and deploy the development environment, and install and deploy the integrated development environment.

[0043] The computer-readable storage medium may include a storage program area and a storage data area. Among them, the storage program area may store an operating system and application programs required for at least one function; the storage data area may store data created according to the use of the power big data analysis platform building device outside the MRS cluster, etc. In addition, the computer-readable storage medium may include high-speed random access memory, and may also include memories, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the computer-readable storage medium may optionally include memories remotely provided with respect to the processor, and these remote memories may be connected to the power big data analysis platform building device outside the MRS cluster through a network. Examples of the above network include but are not limited to the Internet, enterprise intranets, local area networks, mobile communication networks, and combinations thereof.

[0044] Figure 4 It is a schematic structural diagram of an electronic device provided by an embodiment of the present invention, as Figure 4 shown. The device includes: a processor 410 and a memory 420. The electronic device may further include: an input device 440 and an output device 440. The processor 410, the memory 420, the input device 440, and the output device 440 may be connected through a bus or other means, Figure 4Take the bus connection as an example. The memory 420 is the computer-readable storage medium described above. The processor 410 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 420, that is, implements the method for building a power big data analysis platform outside the MRS cluster in the above method embodiment. The input device 440 can receive input digital or character information, and generate key signal inputs related to user settings and function controls of the device for building a power big data analysis platform outside the MRS cluster. The output device 440 may include display devices such as a display screen.

[0045] The above electronic device can execute the method provided by the embodiment of the present invention, and has corresponding functional modules and beneficial effects for executing the method. For technical details not described in detail in this embodiment, reference can be made to the method provided by the embodiment of the present invention.

[0046] As an implementation manner, the above electronic device is applied to a device for building a power big data analysis platform outside the MRS cluster and is used for a client, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to:

[0047] Prepare the basic environment for the power big data analysis platform according to the cloud environment deployed by the MRS cluster, and the basic environment preparation includes mirror selection and installation of the operating system, configuration of the basic development and running environment, and configuration of component installation tools;

[0048] Synchronize the time of the power big data analysis platform with the time of the Master node, and the synchronization includes installing the ntp service of the power big data analysis platform according to the command yum - y install ntp; modifying server xx.xx.xx.xx prefer in the ntp.conf file according to the command vi / etc / ntp.conf to point the server to the master node, where xx.xx.xx.xx is the IP address of the time synchronization server of the Mater node; starting the ntp service according to the command service ntpd start to complete the time synchronization between the analysis platform and the master node, and checking the time synchronization status according to ntpstat to confirm the successful time synchronization;

[0049] Install the MRS cluster client on the out-of-cluster node, create cluster users in the management console and grant corresponding permissions, and authorize users to log in to the client on the power big data analysis platform for mutual authentication;

[0050] Configure the node environment of the analysis platform and deploy the development environment, and install and deploy the integrated development environment.

[0051] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of each embodiment or some parts of the embodiments.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. And these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for building an electric power big data analysis platform outside the MRS cluster, characterized in that: include: Based on the cloud environment deployed by the MRS cluster, the basic environment preparation for the power big data analysis platform is carried out. The basic environment preparation includes the selection and installation of the operating system image, the configuration of the basic development and operation environment, and the configuration of the component installation tool. After the basic environment preparation for the power big data analysis platform is carried out based on the cloud environment deployed by the MRS cluster, the following steps are also included: Obtain data from the current big data platform, including DWS database data and Hive database data; Perform missing value processing and noise data processing on the DWS database data and the Hive database data; Performing correlation analysis on the processed DWS database data and / or the Hive database data according to a preset statistical method and a preset machine learning model; Synchronize the time of the power big data analysis platform with the time of the Master node. The synchronization includes completing the installation of the NTP service of the power big data analysis platform using the command yum -y install ntp; modifying the server xx.xx.xx.xx prefer in the ntp.conf file using the command vi / etc / ntp.conf to point the server to the Master node, where xx.xx.xx.xx is the IP address of the Master node time synchronization server; starting the NTP service using the command service ntpd start to complete the time synchronization between the analysis platform and the Master node, and checking the time synchronization status using ntpstat to confirm that the time synchronization is successful; Install the MRS cluster client on nodes outside the cluster, create cluster users in the management console and grant corresponding permissions, and authorize users to log in to the client on the power big data analysis platform for mutual authentication; Analyze platform node environment configuration and development environment deployment, and installation and deployment of integrated development environment.

2. The method for building an electric power big data analysis platform outside the MRS cluster according to claim 1, characterized in that: The steps of installing the MRS cluster client on nodes outside the cluster, creating a cluster user in the management console and granting corresponding permissions, and authorizing the user to log in to the client on the power big data analysis platform for mutual authentication include: Create a user in the management console based on the cluster administrator account of the MRS cluster and grant corresponding permissions to the MRS service. Log in to the console with the newly created user, switch to the authorized area, and verify the permissions. Download the MRS client in the management console and send it to the designated server in the power big data analysis platform server. After the installation is complete, switch to the client installation directory and configure the environment variables using the command source bigdata_env. Perform bidirectional identity authentication on the client and server using Kerberos authentication, and use the kinit command to control MRS cluster users for verification.

3. The method for building an electric power big data analysis platform outside an MRS cluster according to claim 1, characterized in that: The operating system version is Linux, Redhat 7.6 and above.

4. A device for building a power big data analysis platform outside the MRS cluster, characterized in that: include: The preparation module is configured to prepare the basic environment for the power big data analysis platform based on the cloud environment deployed by the MRS cluster. The basic environment preparation includes the selection and installation of the operating system image, the configuration of the basic development and operation environment, and the configuration of the component installation tool. After the basic environment preparation for the power big data analysis platform is performed based on the cloud environment deployed by the MRS cluster, the following steps are also included: Obtain data from the current big data platform, including DWS database data and Hive database data; Perform missing value processing and noise data processing on the DWS database data and the Hive database data; Performing correlation analysis on the processed DWS database data and / or the Hive database data according to a preset statistical method and a preset machine learning model; The synchronization module is configured to synchronize the time of the power big data analysis platform with the time of the master node. The synchronization includes completing the installation of the ntp service of the power big data analysis platform according to the command yum -y install ntp; modifying the server xx.xx.xx.xx prefer in the ntp.conf file according to the command vi / etc / ntp.conf to point the server to the master node, where xx.xx.xx.xx is the IP address of the master node time synchronization server; starting the ntp service according to the command service ntpd start to synchronize the time of the analysis platform with the master node, and checking the time synchronization status according to ntpstat to confirm that the time synchronization is successful; The authentication module is configured to install the MRS cluster client on nodes outside the cluster, create cluster users in the management console and grant corresponding permissions, and authorize users to log in to the client on the power big data analysis platform for mutual authentication; The integration module is configured to analyze the platform node environment configuration and development environment deployment, and the installation and deployment of the integrated development environment.

5. An electronic device, characterized in that: include: At least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 3 is implemented.

Images