Access method and device for multiple Hadoop clusters and computer equipment

A Hadoop cluster and access method technology, applied in the field of computer equipment, can solve problems such as disallowing mutual trust configuration, cumbersome kerberos mutual trust process, increasing insecurity, etc., to achieve the effect of improving access efficiency

Pending Publication Date: 2022-06-03
杭州玳数科技有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] However, the process of configuring kerberos mutual trust in related technologies is cumbersome and may also increase insecurity, and in some cases, mutual trust configuration between clusters is not allowed, so the same client cannot safely and simultaneously access multiple kerberos-enabled and Hadoop cluster without mutual trust configured

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Access method and device for multiple Hadoop clusters and computer equipment
  • Access method and device for multiple Hadoop clusters and computer equipment
  • Access method and device for multiple Hadoop clusters and computer equipment

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0039] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0040] Currently, there is a lack of flexibility and scalability in parameter processing, resulting in poor task submission effects. For example, when the types of tasks gradually increase, the parameter processing of each task is different, and the corresponding processing logic becomes difficult to maintain. Especially when the task needs to construct task running parameter information according to different cluster environments. F...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides an access method and device for multiple Hadoop clusters and computer equipment, and the method comprises the steps: obtaining a krb5. Conf file corresponding to the kerberos authentication under the condition of determining that each Hadoop cluster has started the kerberos authentication and the KDC of each Hadoop cluster is different; carrying out merging processing on the plurality of krb5. Conf files; the method comprises the following steps of: determining a host name of each YARN ResourceManger component and a host name of each HDFS (Hadoop Distributed File System) NameNode component, and determining a domain Realm corresponding to kerberos authentication; determining a plurality of mappings between the hostname and the Realm, and setting the mappings between the hostname and the Realm in a domain real column; the method comprises the following steps of: judging whether a UserGroup Information object exists in a local cache or not; and if the UserGroup Information object exists in the local cache, the UserGroup Information object is obtained, and the plurality of Hadoop clusters are accessed to the UserGroup Information object. Therefore, one client can simultaneously access a plurality of Hadoop clusters which open the Kerberos and have different KDCs, and the access efficiency can be improved.

Description

technical field [0001] The invention relates to the technical field of big data, in particular to an access method for multiple Hadoop clusters, an access device for multiple Hadoop clusters, and a computer device. Background technique [0002] For multiple Hadoop (Hadoop is a distributed system infrastructure) clusters, the clusters cannot access each other, especially when kerberos is enabled in each cluster and each cluster has a different KDC (KeyDistribution Center, key distribution center) Center) In this scenario, the same client cannot access such multiple Hadoop clusters at the same time. Among them, Kerberos is a computer network authorization protocol used to conduct personal communication in a secure way in a non-secure network. Identity authentication, in order to access the cluster, the common practice in related technologies is to configure mutual trust between multiple Hadoop cluster kerberos. [0003] However, the configuration of kerberos mutual trust in t...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/13G06F16/182H04L9/40
CPCG06F16/13G06F16/182H04L63/10
Inventor 郝卫亮
Owner 杭州玳数科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap