Hive data source hot loading method based on Dores and related device

Abstract

The embodiment of the invention relates to a Hive data source hot loading method based on Dores and a related device, and the method comprises the steps: monitoring and consuming a message of a target theme in a kafka message system by using program packages respectively deployed by a front-end node and a rear-end node in a Dores cluster, analyzing the message, obtaining a configuration file of a Hive data source, and storing the configuration file of the Hive data source in a server; and writing a configuration file of the hive data source into a specified path of the front-end node and the rear-end node, receiving a message for creating a target hive data source, and registering the target hive data source into the Dores cluster. By the adoption of the technical scheme, automatic deployment of hive data source configuration is achieved, manual maintenance of the configuration file of the hive data source is not needed, the operation and maintenance cost is greatly reduced, manpower expenditure is saved, and errors caused by manual operation are reduced.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of data processing, and particularly relates to a hive data source hot loading method based on Doris and a related device. BACKGROUND

[0002] When the Doris cluster is widely applied to big data analysis and data governance and the like, it is necessary to dynamically add a hive data source and register it to the Doris cluster. When the Doris cluster adds the hive data source, the configuration file of the hive data source needs to be manually maintained in the front-end node and the back-end node of the Doris cluster, which results in high operation and maintenance cost, large manpower expenditure, and configuration abnormality caused by uncontrollability of manual operation. SUMMARY

[0003] In order to solve the above technical problems, the present disclosure provides a hive data source hot loading method based on Doris and a related device.

[0004] The present disclosure provides a hive data source hot loading method based on Doris, which comprises the following steps: A program package deployed in a front-end node and a back-end node in the Doris cluster is used to listen to and consume messages of a target topic in a kafka message system; The messages are parsed to obtain a configuration file of a hive data source; The configuration file of the hive data source is written to a specified path of the front-end node and the back-end node; A message for creating a target hive data source is received, and the target hive data source is registered to the Doris cluster.

[0005] The present disclosure also provides a hive data source hot loading device based on Doris, which comprises the following modules: A listening and consuming module is configured to use a program package deployed in a front-end node and a back-end node in the Doris cluster to listen to and consume messages of a target topic in a kafka message system; An obtaining module is configured to parse the messages to obtain a configuration file of a hive data source; A writing module is configured to write the configuration file of the hive data source to a specified path of the front-end node and the back-end node; A registering module is configured to receive a message for creating a target hive data source, and register the target hive data source to the Doris cluster.

[0006] The embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory for storing executable instructions of the processor; the processor is used for reading the executable instructions from the memory and executing the executable instructions to realize the Doris-based hive data source hot loading method provided by the embodiment of the present disclosure.

[0007] The embodiment of the present disclosure further provides a computer readable storage medium, which stores a computer program, and the computer program is used for executing the Doris-based hive data source hot loading method provided by the embodiment of the present disclosure.

[0008] The technical scheme provided by the embodiment of the present disclosure has the following advantages compared with the prior art: the Doris-based hive data source hot loading scheme provided by the embodiment of the present disclosure uses the program package respectively deployed in the front-end node and the back-end node in the Doris cluster to listen to and consume the messages of the target topic in the kafka message system, analyzes the messages, obtains the configuration file of the hive data source, writes the configuration file of the hive data source into the specified path of the front-end node and the back-end node, receives the message of creating the target hive data source, and registers the target hive data source to the Doris cluster. By using the above technical scheme, the automatic deployment of the hive data source configuration is realized, manual maintenance of the configuration file of the hive data source is not needed, and the operation and maintenance cost is greatly reduced, the labor cost is saved, and the errors caused by human operation are reduced. BRIEF DESCRIPTION OF DRAWINGS

[0009] The above and other features, advantages, and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the original and elements are not necessarily drawn according to the scale.

[0010] Figure 1 A flowchart of a Doris-based hive data source hot loading method provided by the embodiment of the present disclosure is shown in the figure; Figure 2 A flowchart of a jar package generation provided by the embodiment of the present disclosure is shown in the figure; Figure 3 A flowchart of an image construction provided by the embodiment of the present disclosure is shown in the figure; Figure 4 A workflow diagram of a Doris cluster deployment and startup provided by the embodiment of the present disclosure is shown in the figure; Figure 5 A structure diagram of a Doris-based hive data source hot loading device provided by the embodiment of the present disclosure is shown in the figure; Figure 6 A structural schematic diagram of an electronic device is provided for the embodiments of the present disclosure. DETAILED DESCRIPTION

[0011] Embodiments of the present disclosure will be described in more detail with reference to the drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be interpreted as being limited to the embodiments set forth herein; rather, these embodiments are provided so that the present disclosure can be more thoroughly and completely understood. It should be understood that the drawings and embodiments of the present disclosure are only for illustrative purposes and are not intended to limit the scope of protection of the present disclosure.

[0012] It should be understood that each of the steps recited in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0013] The term "comprising" and variations thereof as used in the present disclosure are open-ended, that is, "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related definitions will be given in the description below.

[0014] It should be noted that the concepts of "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0015] It should be noted that the adjectives "one", "multiple" mentioned in the present disclosure are illustrative and not limiting, and those skilled in the art should understand that, unless otherwise explicitly stated in the context, it should be understood as "one or more".

[0016] The names of the messages or information exchanged between the multiple devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0017] When the Doris cluster is widely used in big data analysis and data governance applications, it is necessary to dynamically add a hive data source and register it to the Doris cluster. When the Doris cluster adds the hive data source, the configuration file of the hive data source needs to be manually maintained in the front-end node and the back-end node of the Doris cluster. If the hive data source starts Kerberos authentication, the authentication files krb5.conf and keytab also need to be manually maintained. The existing scheme has the following defects and deficiencies: The Doris official does not provide a dynamic loading scheme for the configuration of the hive data source, and the configuration file of the hive data source needs to be manually maintained, which has low flexibility and high operation and maintenance cost. If the hive data source configuration changes, it cannot be updated in a timely manner, and it is too dependent on the operation and maintenance personnel, and cannot guarantee the timeliness and accuracy of the change. If the Doris engine node fails and is redeployed, the already configured hive configuration file will disappear. Manual repeated operations are required to ensure the availability of the hive data source.

[0018] Therefore, the above problems result in high operation and maintenance cost, large manpower expenditure, and configuration abnormalities caused by uncontrollability of manual operation.

[0019] To solve the above problems, the present embodiment provides a hot loading method of a hive data source based on Doris, which will be introduced below in combination with specific embodiments.

[0020] In order to facilitate the understanding of the present embodiment, the terms appearing in the present embodiment are first explained.

[0021] Doris: Apache Doris is a high-performance, real-time analysis database based on MPP architecture. Doris can better meet the use scenarios of report analysis, instant query, unified data warehouse construction, data lake federation query acceleration, etc.

[0022] fe node: fe is the front-end component of Doris, responsible for receiving client requests, parsing SQL queries, and generating execution plans, and also responsible for communicating with be to coordinate the execution of queries.

[0023] be node: be is the back-end component of Doris, responsible for executing the execution plan generated by fe. The be node is responsible for data storage and calculation, and is the worker node in the Doris cluster.

[0024] Hive: Hive is a data warehouse tool based on Hadoop, used for storing, querying and analyzing large-scale data stored in Hadoop.

[0025] hosts: The hosts file is a file used to map hostnames to IP addresses, its main function is to associate the commonly used website domain name with its corresponding IP address, so that when the user inputs the website, the system can quickly find the corresponding IP address.

[0026] kerberos: Kerberos is a network authentication protocol proposed by MIT, which provides a method for identifying user identity information in an open and insecure network, aiming to provide strong authentication for client / server applications using key encryption technology.

[0027] keytab: The keytab file is a file containing multiple principal (user, group, service or server) identity authentication information, which is functionally equivalent to a password.

[0028] krb5.conf: The krb5.conf file is a key configuration file in the Kerberos authentication system, which contains the configuration information of Kerberos.

[0029] spring-boot: Spring-boot is an open-source framework that can simplify the development process of Java programs.

[0030] jar package: A jar archive package is a platform-independent file format that can combine multiple files into a package.

[0031] kafka: Kafka is an open-source distributed event stream platform widely used in data pipelines, streaming analysis and data integration.

[0032] catalog: Used to describe the attributes of a data source.

[0033] base64: A base64 encoding method that represents binary data based on 64 printable characters (letters, numbers, +, / , etc.).

[0034] Dockerfile: A text file containing all instructions for building a Docker image.

[0035] Figure 1 A flowchart of a Doris-based hive data source hot loading method provided by the embodiments of the present disclosure, which can be executed by a Doris-based hive data source hot loading device, wherein the device can be implemented by software and / or hardware, and can be integrated in an electronic device. As shown in Figure 1 The method comprises: In step 101, a program package deployed on a front-end node and a back-end node in the Doris cluster is used to listen to and consume messages of a target topic in the kafka message system.

[0036] The Doris cluster refers to a Doris running environment formed by multiple nodes working cooperatively. The front-end node can be the fe node described above. The back-end node can be the be node described above. The program package refers to an executable program set developed based on a programming language and directly deployed and run on a server node. Specifically, the program package can be a jar package. The target topic is a pre-set message classification container in the kafka message system. Specifically, the target topic is used to receive messages added by a configuration file of a hive data source or messages of configuration file changes of the hive data source.

[0037] The configuration file of the hive data source refers to a set of key configuration information used to enable the Doris cluster to identify, connect to, and access the hive data source. Specifically, the configuration file includes hosts information and a Kerberos authentication file, and the Kerberos authentication file includes a Kerberos 5 configuration file and a key table file. The Kerberos 5 configuration file is a krb5.conf file, and the key table file is a keytab file. The message added by the configuration file refers to a message of creating a configuration file for a newly added hive data source that needs to access the Doris cluster. The message of configuration file changes refers to a message generated when a configuration file of a hive data source that has accessed the Doris cluster is modified (such as host address change, authentication key update, port adjustment, etc.).

[0038] In the embodiments of the present disclosure, the program package is deployed on the front-end node and the back-end node of the Doris cluster. The program package continuously establishes a connection with the kafka message system and listens to a pre-agreed target topic. When new messages such as messages of adding a configuration file of a hive data source are listened to in the target topic, the program package reads and processes these messages (i.e., consumes), thereby realizing real-time perception of configuration changes of the hive data source by the Doris cluster.

[0039] In an optional implementation, the program package is generated based on a java project. The java project adds a dependent configuration of the kafka message system, sets the target topic in the kafka message system as a communication topic for communication between the Doris cluster and a business party, and configures the java project to read kafka environment variables when starting, load a kafka listening function, and listen to the target topic. In the listening callback, the messages are serialized into objects to obtain the configuration file of the hive data source, and the configuration file of the hive data source is written to a specified path of the front-end node and the back-end node.

[0040] Among them, the java project adds a dependency configuration of the kafka message system, which is used to enable the program to connect to the kafka message system. The business party refers to a subject that initiates a hive data source registration process and provides a required configuration file to complete the registration due to the need to operate the hive data source in the doris cluster. The kafka environment variable refers to a key-value pair defined in advance in the program running environment, which is used to store the core parameters required for connection and interaction with the kafka message system. The kafka listening function refers to a mechanism that continuously establishes a connection with the kafka message system after the program is started, and monitors whether new messages are generated in the target topic in real time. The listening callback refers to a processing function triggered automatically when new messages are detected in the target topic, which is used for subsequent processing of the received messages. The serialized message as an object refers to converting the original message received by the target topic in the kafka message system into an object that can be directly operated by the java program, so as to extract key information such as hosts information and base64 encoded information of the file from the message.

[0041] In the embodiments of the present disclosure, the program package is generated by packaging the java project, and the java project has made two key configurations in advance. One is to introduce the dependency of the kafka message system, and set the target topic of the kafka message system as the communication channel between the doris cluster and the business party. The second is to set the program to automatically read the kafka environment variable when starting, so as to load the listening function and continuously listen to the target topic. When there is a new message in the target topic, the program will convert the message into a java object through the listening callback function, extract the configuration file of the hive data source from the object, and write these configuration files to the specified path of the front-end node and the back-end node of the doris cluster.

[0042] In order to facilitate the understanding of the embodiments of the present disclosure, taking the jar package as an example, the process of generating the jar package is described in detail, as shown in Figure 2 , which is a flowchart of generating a jar package provided by the embodiments of the present disclosure. Figure 2

[0043] Firstly, based on JDK17, a java project is constructed. Specifically, a spring-boot framework is added as a basic configuration, a dependency configuration of the kafka message system is added to enable communication with the kafka message system, the hive-topic topic (i.e. the above target topic) is agreed as the communication topic of the doris cluster and the business party, and a packaging plug-in is added in the maven configuration to package the java project into a jar package.

[0044] ​At the start of the project, read the kafkaEnv environment variable, which has the connection information of the kafka, load the kafka listener, listen to the hive-topic topic (i.e. the target topic), serialize the message to an object in the listener callback, get the hosts information and the base64 encoded information of the file.

[0045] Write the hosts information obtained above into the / etc / hosts file in the container. If the same hosts are encountered, de-duplication processing needs to be done. Reverse parse the base64 encoding of the keytab file into a file and save it under the / opt / apache-doris directory, and reverse parse the base64 encoding of the krb5.conf file into a file and save it under the / etc / directory. If the krb5.conf file already exists, merge processing will be done. Package the above java project into a jar package.

[0046] Further, as shown in Figure 3 , Figure 3 A flowchart for building an image is provided for the embodiments of the present disclosure. In the code of Doris, modify the Dockerfile files of the fe image and the be image, copy the jar package to the fe image and the be image, modify the Entrypoint initialization instruction, add the startup command of the jar package, build the fe image and the be image, and push them to the image repository. When deploying the Doris engine, mount the kafkaEnv environment variable to connect the kafka message system, start the fe node and the be node, and enter the container. The customized jar package has already been started.

[0047] Thus, the jar packages respectively deployed by the fe nodes and the be nodes in the Doris cluster can be used to listen to and consume the messages of the hive-topic topic in the kafka message system.

[0048] It should be noted that the generation process of the jar package can be completed by a programmer through a build tool of the java project.

[0049] When the business end adds or changes the hive data source, it sends a message to the hive-topic topic of the kafka message system. The jar packages in the fe nodes and the be nodes listen to the message and, according to the above logic, add the hosts value locally and parse, merge and write to disk the krb5.conf and keytab files.

[0050] In an alternative embodiment, a program package respectively deployed by the front-end nodes and the back-end nodes in the Doris cluster is used to listen to and consume the messages of the target topic in the kafka message system, comprising: If the Doris cluster is redeployed, the program package deployed on the front-end node and the back-end node in the Doris cluster is used to listen to and consume all historical messages of the target topic in the kafka message system from the message starting offset, so as to regenerate the configuration file of the historical hive data source.

[0051] The Doris cluster redeployment refers to the process of uninstalling the Doris cluster from the original environment and then reinstalling, configuring and starting the Doris cluster in a new environment due to reasons such as version upgrade, environment migration, fault reconstruction and the like. At this time, the configuration file of the original hive data source on the node will be lost. Each message in the kafka message system has a unique offset, and the message starting offset refers to the offset of the earliest message. Starting from the starting offset, all historical messages in the target topic of the kafka message system are consumed. Regenerating the configuration file of the historical hive data source refers to that when the Doris cluster is redeployed, the historical messages of the target topic in the kafka message system are consumed to regenerate the configuration file of all past hive data sources by parsing the historical messages, and the configuration state before redeployment is restored.

[0052] In the embodiment of the present disclosure, when the Doris cluster causes the configuration file of the original hive data source to be lost due to redeployment, a special program package is redeployed on the newly deployed fe node and be node. After the program package is started, all historical messages stored in the target topic are consumed from the message starting offset (i.e. the earliest message) of the target topic in the kafka message system, and the configuration file of the historical hive data source is regenerated by parsing the historical messages. It can be seen that in the embodiment of the present disclosure, if the Doris cluster is redeployed, the fe node and the be node are started at the same time, and the messages in the kafka are consumed from the earliest offset, and the historical hive configuration is regenerated to ensure that the redeployment does not cause the loss of the hive configuration.

[0053] In another optional implementation, the program package deployed on the front-end node and the back-end node in the Doris cluster is used to listen to and consume the messages of the target topic in the kafka message system, including: If the Doris cluster is not redeployed, the program package deployed on the front-end node and the back-end node in the Doris cluster is used to listen to and consume the new messages of the target topic in the kafka message system from the current message offset.

[0054] The Doris cluster without redeployment refers to that the Doris cluster is in a normal running state, and no unloading, reloading, environment migration or the like is performed, and the configuration file of the existing hive data source is complete and available. The current message offset refers to the next sequence number of the last consumed message recorded after the program package last consumes the kafka message, for example, the program package last consumes the message with the offset of 100, and the current offset is 101, which represents that the new message will be read from the offset of 101 subsequently. The new message refers to the message in the target topic of the kafka message system, which is generated after the program package last consumption ends, that is, the message with the offset greater than the current offset, and the message usually corresponds to the latest initiated hive data source configuration operation of the business party.

[0055] In the embodiment of the present disclosure, when the Doris cluster does not need to be redeployed, the program package deployed on the fe node and the be node will adopt the incremental consumption mode to connect the target topic in the kafka message system. After the program package is started, the current message offset recorded by the program package, that is, the last consumption stop position, is read first, and then the new message in the target topic is continuously listened to and consumed from the offset, without the need to repeatedly process the historical configuration synchronized, so as to avoid redundancy and ensure real-time performance.

[0056] In addition, if the Doris cluster is not redeployed, the program package deployed on the front-end node and the back-end node in the Doris cluster can also be used to listen to and consume all historical messages of the target topic in the kafka message system from the message starting offset, which is not limited in the embodiment of the present disclosure.

[0057] Step 102, the message is parsed to obtain the configuration file of the hive data source.

[0058] The configuration file includes hosts information and a Kerberos authentication file, and the Kerberos authentication file includes a Kerberos5 configuration file and a key table file.

[0059] Specifically, the hosts information is the network positioning information of the hive data source, including the mapping and association of the host name and the corresponding IP address of the hive data source, for the Doris cluster to locate the hive data source. The Kerberos authentication file refers to the file used for identity verification by the Doris cluster when the hive data source enables Kerberos. The Kerberos5 configuration file is the krb5.conf file. The key table file is the keytab file.

[0060] In an alternative embodiment, the message is parsed to obtain the configuration file of the hive data source, including: serializing the message into an object in the listening callback, obtaining the hosts information and base64 encoded information of the file of the hive data source, and inversely parsing the base64 encoded information into a file to obtain the Kerberos 5 configuration file and the key table file.

[0061] The listening callback refers to a processing function automatically triggered when a new message is listened to in the target topic, and is used for subsequent processing of the received message. The serialized message into an object refers to converting the original message received by the target topic in the kafka message system into an object that can be directly operated by the java program, so as to facilitate extraction of key information in the message, such as hosts information and base64 encoded information of the file. The base64 encoded information refers to converting the Kerberos authentication file into an encoded string in text format. The inversely parsed into a file refers to restoring the base64 encoded string into the original file.

[0062] In the embodiments of the present disclosure, when the program package receives the kafka message through the listening callback, the message is first converted into an operable object, from which the hosts information of the hive data source and the base64 encoded content of the Kerberos authentication file (i.e. the base64 encoded information of the file) are extracted; then the base64 encoding is inversely parsed, i.e. decoded into the original file, and finally the usable Kerberos 5 configuration file and the key table file are obtained, which provide necessary network positioning information and security authentication credentials for the Doris cluster to connect the hive data source.

[0063] Further, in the process of parsing the message, if there is the same hosts information, a deduplication process is performed, and if the Kerberos 5 configuration file already exists in the corresponding directory of the front-end node and the back-end node, a merging process is performed.

[0064] Step 103, write the configuration file of the hive data source into the specified path of the front-end node and the back-end node.

[0065] In the embodiments of the present disclosure, in the callback, the message is serialized into an object, the hosts information and the base64 encoded information of the file are obtained, the obtained hosts information is written into the / etc / hosts file of the front-end node and the back-end node. If there is the same hosts information, the de-duplication processing is performed. The base64 encoding of the keytab file (i.e., the key table file) is inversely parsed into a file and saved under the / opt / apache-doris directory, and the base64 encoding of the krb5.conf file (i.e., the Kerberos 5 configuration file) is inversely parsed into a file and saved under the / etc / directory. If the corresponding directory of the front-end node and the back-end node already exists the Kerberos 5 configuration file (i.e., the krb5.conf file), the merging processing is performed.

[0066] In step 104, a message of creating a target hive data source is received, and the target hive data source is registered to the Doris cluster.

[0067] In the embodiments of the present disclosure, after the configuration file of the hive data source is written into the specified path of the front-end node and the back-end node, the message of creating the target hive data source by the user in the Doris client is received, that is, the request of creating the hive-catalog data source, the target hive data source to be accessed can be determined through the message, and then the subsequent operation is performed, the target hive data source is registered to the Doris cluster, so that the Doris cluster can identify and interface the target hive data source.

[0068] In actual application, an example of the message of creating the target hive data source is as follows: When the user creates the hive-catalog data source in the Doris client, the above configuration file can be used. As shown in the following example, the / opt / apache-doris / your.keytab path is the above configuration: CREATE CATALOG hive_krb PROPERTIES ( 'type'='hms', 'hive.metastore.uris' = 'thrift: / / 172.0.0.1:9083' 'hive.metastore.kerberos.principal' = 'your-hms-principal', 'hadoop.security.authentication' = 'kerberos', ' hadoop. kerberos. keytab'= ' / opt / apache-doris / your.keytab', The above code is a CREATE CATALOG statement executed by a user on a Doris client, which is used to create a hive data source named hive_krb in the Doris cluster, that is, the target hive data source.

[0069] As shown in Figure 4 As shown in Figure 4 A Doris cluster deployment and startup workflow diagram provided by an embodiment of the present disclosure.

[0070] First, the deployment of the Doris cluster is completed, and it is ensured that the fe node (that is, the front-end node) and the be node (that is, the back-end node) are normally ready. Then, a jar package is started on the fe node and the be node. After the jar package is started, it will first read the earliest message in the kafka topic (that is, the target topic in the kafka message system) to start consumption, so as to load the configuration file of the existing historical hive data source in the target topic. After the historical configuration loading is completed, the kafka message listening function is continuously started. When the business end has a demand for adding or modifying the hive data source, the configuration information of the corresponding hive data source will be encapsulated into a message and sent to the target topic. After the jar package listens to the new message, it immediately consumes and deserializes the message content into an object, parses the hive data source configuration, and persists it to the disk. Finally, the user executes the CREATE CATALOG statement on the Doris client, creates the hive data source in the Doris cluster based on the persisted configuration, and can subsequently operate the data in the hive through the hive data source.

[0071] The hive data source hot loading scheme based on Doris provided by the embodiment of the present disclosure uses the program packages respectively deployed on the front-end node and the back-end node in the Doris cluster to listen to and consume the messages of the target topic in the kafka message system, parse the messages, obtain the configuration file of the hive data source, write the configuration file of the hive data source to the specified path of the front-end node and the back-end node, receive the message of creating the target hive data source, and register the target hive data source to the Doris cluster. By using the above technical scheme, the automatic deployment of the hive data source configuration is realized, and manual maintenance of the configuration file of the hive data source is not required, which greatly reduces the operation and maintenance cost, saves the labor cost, and reduces the errors caused by human operation.

[0072] In actual application, a java project is constructed, packaged into a jar package, and added to the be image and fe image of Doris when the be image and fe image of Doris are constructed, and a startup script is added. When the Doris engine is deployed, the be node and the fe node are started, and the jar package is also started. The project reads the connection information of the kafka from the environment variable, starts continuous listening to a specific topic, and if a message is listened to, the hosts, the krb5 file and the keytab file are parsed from the message and written into the corresponding directory in sequence, so as to realize dynamic loading of the hive data source configuration.

[0073] Specifically, first, a java project is constructed, business logic is written, and the project is constructed into a jar package. Then, a Dockerfile file is configured, the jar package is packed into the fe image and the be image, and the jar package is loaded in the startup item to construct the image. Finally, the Doris cluster is deployed using the above image, the fe node and the be node are started, listening is started, and the hive data source configuration is dynamically updated.

[0074] It can be seen that the customized jar package in the embodiment of the disclosure is automatically deployed in the Doris cluster, and the operation and maintenance personnel are not aware. The kafka message system and the outside are decoupled through the listening and transmission of the hive configuration in the Doris cluster. In actual project deployment and operation, the difficulty of automatic registration of the hive data source can be solved, the customized development based on the Doris engine is ingeniously implanted into the customized jar package, the jar package is the link between the business end and the hive data source configuration, and the addition, deletion and modification of the business end hive data source configuration will be timely listened to and persisted in the Doris cluster. The patent can be applied to data governance and big data analysis, and is an important upgrade and function perfecting case of the Doris engine. Compared with manually maintaining the configuration of the hive data source, the scheme can greatly reduce the operation and maintenance cost, save labor cost, and reduce errors caused by human operation. The scheme provides a timely updating method for changes in the hive data source configuration based on the business end, accurately covers the old configuration, realizes no delay synchronization of the hive configuration between the business end and the Doris cluster, and ensures the timeliness of dynamic refreshing of the configuration.

[0075] Figure 5 A structure diagram of a Doris-based hive data source hot loading device provided by the embodiment of the disclosure is shown in the figure. The device can be implemented by software and / or hardware, and can be integrated in an electronic device. As shown in the figure, the device comprises: Figure 5 A listening and consuming module 501 is configured to listen to and consume messages of a target topic in a kafka message system by using program packages respectively deployed in front-end nodes and back-end nodes in a Doris cluster. ​The acquisition module 502 is configured to parse the message and acquire a configuration file of a hive data source. The writing module 503 is configured to write the configuration file of the hive data source to a specified path of the front-end node and the back-end node. The registration module 504 is configured to receive a message of creating a target hive data source, and register the target hive data source in the Doris cluster.

[0076] In an optional implementation, the program package is generated based on a java project, the java project is added with a dependent configuration of the kafka message system, a target topic in the kafka message system is set as a communication topic for communication between the Doris cluster and the business party, and the java project is configured to read a kafka environment variable when starting, load a kafka listening function, and listen to the target topic, serialize a message into an object in a listening callback, acquire a configuration file of a hive data source, and write the configuration file of the hive data source to a specified path of the front-end node and the back-end node.

[0077] In an optional implementation, the configuration file includes hosts information and a Kerberos authentication file, the Kerberos authentication file includes a Kerberos 5 configuration file and a key table file, and the device further includes: The deduplication module is configured to perform deduplication processing if there is the same hosts information. The merging module is configured to perform merging processing if the Kerberos 5 configuration file already exists in a corresponding directory of the front-end node and the back-end node.

[0078] In an optional implementation, the listening and consuming module 501 is configured to: If the Doris cluster is redeployed, the program package deployed in the front-end node and the back-end node in the Doris cluster is used to listen to and consume all historical messages of a target topic in a kafka message system from a message starting offset, so as to regenerate a configuration file of a historical hive data source.

[0079] In an optional implementation, the listening and consuming module 501 is configured to: If the Doris cluster is not redeployed, the program package deployed in the front-end node and the back-end node in the Doris cluster is used to listen to and consume new messages of a target topic in a kafka message system from a current message offset.

[0080] In an optional implementation, the acquisition module 502 includes: The acquisition sub-module is configured to serialize the message into an object in a callback, acquire hosts information and base64 encoding information of a file of the hive data source; The reverse analysis module is configured to reversely analyze the base64 encoding information into a file to obtain the Kerberos 5 configuration file and the key table file.

[0081] In an optional implementation, the target topic is configured to receive a message of adding a configuration file of the hive data source or a message of changing the configuration file of the hive data source.

[0082] The Doris-based hive data source hot loading apparatus provided by the embodiments of the present disclosure can execute the Doris-based hive data source hot loading method provided by any of the embodiments of the present disclosure, and has the corresponding function modules and beneficial effects of the execution method.

[0083] The embodiments of the present disclosure further provide a computer program product, which comprises computer programs / instructions, and the computer programs / instructions are executed by a processor to implement the Doris-based hive data source hot loading method provided by the embodiments of the present disclosure.

[0084] Figure 6 A structural schematic diagram of an electronic device provided by the embodiments of the present disclosure is provided.

[0085] Reference will be made to the following specific description Figure 6 , which shows a structural schematic diagram of an electronic device 600 suitable for implementing the electronic device in the embodiments of the present disclosure. The electronic device 600 in the embodiments of the present disclosure can include, but is not limited to, a mobile terminal such as a mobile phone, a notebook computer, a digital broadcast receiver, a Personal Digital Assistant (PDA), a Personal Access Device (PAD), a Portable Media Player (PMP), a vehicle-mounted terminal (for example, a vehicle-mounted navigation terminal), and the like, and a fixed terminal such as a digital TV, a desktop computer, and the like. Figure 6 The electronic device shown is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0086] As Figure 6As shown, the electronic device 600 can include a processing device 601 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to programs stored in a Read-Only Memory (ROM) 602 or loaded into a Random Access Memory (RAM) 603 from a storage device 608. Various programs and data required for operation of the electronic device 600 are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other through a bus 604. An Input / Output (I / O) interface 605 is also connected to the bus 604.

[0087] Generally, the following devices can be connected to the I / O interface 605: input devices 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 607 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, etc.; storage devices 608 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 609. The communication devices 609 can allow the electronic device 600 to communicate wirelessly or wired with other devices to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that all of the illustrated devices are not required, and fewer or different devices can alternatively be implemented.

[0088] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 609, or installed from the storage devices 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the Doris-based hive data source hot loading method of embodiments of the present disclosure are performed.

[0089] It is noted that the aforementioned computer-readable medium of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example and without limitation, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus, or device. In the present disclosure, the computer-readable signal medium can include a computer-readable program code transmitted by a computer-readable storage medium or carried by a carrier wave in a baseband or as part of a carrier wave. Such a propagated computer-readable signal medium can take various forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that can be used to carry or store a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to wire, cable, radio frequency (RF), or any suitable combination thereof.

[0090] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as the HyperText Transfer Protocol (HTTP), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a Local Area Network (LAN), a Wide Area Network (WAN), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed network.

[0091] The aforementioned computer-readable medium can be included in the aforementioned electronic device; or can exist separately from the electronic device and can be accessed by the electronic device through a network.

[0092] The computer readable medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: use program packages respectively deployed by a front-end node and a back-end node in a Doris cluster, listen to and consume messages of a target topic in a kafka message system, parse the messages, obtain a configuration file of a hive data source, write the configuration file of the hive data source to a specified path of the front-end node and the back-end node, receive a message of creating a target hive data source, and register the target hive data source to the Doris cluster.

[0093] Computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ as well as conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network or a wide area network, or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0094] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or in the opposite order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware-based systems and computer instructions.

[0095] The units described in the embodiments of the present disclosure can be implemented by software, or by hardware. In some cases, the name of the unit does not constitute a limitation on the unit itself.

[0096] The functionality described above in this document can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application-Specific Standard Products (ASSPs), System-on-a-Chip (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0097] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more of: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0098] It can be understood that, before using the technical solutions disclosed by the embodiments of the present disclosure, the type of information involved in the present disclosure, the use range, the use scenario, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0099] The above description is merely preferred embodiments of the present disclosure and a description of the principles of the technology used. Those skilled in the art should understand that the disclosed scope of the present disclosure is not limited to the technical solutions formed by the specific combinations of the above technical features, and should also cover other technical solutions formed by any combination of the above technical features or equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features disclosed in the present disclosure (but not limited to) having similar functions to form technical solutions.

[0100] Moreover, while operations are depicted in a particular order, this should not be understood as requiring such an order nor infringing on the scope of the disclosure. Certain of the operations described in the discussion are combinable into a single operation, and certain operations can be separated into several operations. In some embodiments, the operations described in the discussion can be performed in an order different than presented in the discussion. In some embodiments, the operations described in the discussion can be performed concurrently. Also, while several specific implementation details are discussed in the discussion, these should not be interpreted as limiting the scope of the disclosure. Rather, certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0101] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A method for hot-loading Hive data sources based on Doris, characterized in that, The method includes: Using packages deployed on the front-end and back-end nodes of the Doris cluster, listen to and consume messages on the target topic in the Kafka messaging system; The message is parsed to obtain the configuration file of the Hive data source; Write the configuration file of the Hive data source to the specified path of the front-end node and the back-end node; Receive the message to create a target Hive data source and register the target Hive data source to the Doris cluster.

2. The method according to claim 1, characterized in that, The package is generated based on a Java project. The Java project includes the dependency configuration of the Kafka messaging system, sets the target topic in the Kafka messaging system to the communication topic between the Doris cluster and the business party, and configures the Java project to read Kafka environment variables at startup, load the Kafka listening function and listen to the target topic. In the listening callback, the message is serialized into an object to obtain the Hive data source configuration file, and the Hive data source configuration file is written to the specified path of the front-end node and the back-end node.

3. The method according to claim 1, characterized in that, The configuration file includes hosts information and a Kerberos authentication file, wherein the Kerberos authentication file includes a Kerberos 5 configuration file and a key table file. The method further includes: If duplicate hosts information exists, deduplication will be performed. If the Kerberos 5 configuration file already exists in the corresponding directories of the front-end node and the back-end node, then a merging process is performed.

4. The method according to claim 1, characterized in that, The method of using packages deployed on the front-end and back-end nodes of the Doris cluster to listen to and consume messages from the target topic in the Kafka messaging system includes: If the Doris cluster is redeployed, the packages deployed on the front-end and back-end nodes in the Doris cluster are used to listen for and consume all historical messages of the target topic in the Kafka messaging system from the message start offset, in order to regenerate the configuration file of the historical Hive data source.

5. The method according to claim 1, characterized in that, The method of using packages deployed on the front-end and back-end nodes of the Doris cluster to listen to and consume messages from the target topic in the Kafka messaging system includes: If the Doris cluster is not redeployed, the packages deployed on the front-end and back-end nodes in the Doris cluster will be used to listen for and consume new messages on the target topic in the Kafka messaging system, starting from the current message offset.

6. The method according to claim 3, characterized in that, The step of parsing the message to obtain the Hive data source configuration file includes: In the listener callback, the message is serialized into an object to obtain the hosts information of the Hive data source and the base64 encoded information of the file; The base64 encoded information is reverse-parsed into a file to obtain the Kerberos 5 configuration file and the key table file.

7. The method according to claim 1, characterized in that, The target topic is used to receive messages about adding to the configuration file of the Hive data source or changing the configuration file of the Hive data source.

8. A Doris-based Hive data source hot-loading device, characterized in that, The device includes: The listening and consuming module is used to listen to and consume messages on target topics in the Kafka messaging system using packages deployed on the front-end and back-end nodes of the Doris cluster. The acquisition module is used to parse the message and obtain the configuration file of the Hive data source; The writing module is used to write the configuration file of the Hive data source to the specified path of the front-end node and the back-end node; The registration module is used to receive messages about creating a target Hive data source and register the target Hive data source to the Doris cluster.

9. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the Doris-based hive data source hot-loading method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the Doris-based hive data source hot-loading method described in any one of claims 1-7.

Images