A data writing method and related device

Abstract

The embodiment of the present application provides a data writing method, including: obtaining data storage information and permission information associated with a target application, the target application is a target application developed based on a target stream processing framework; obtaining the data to be written collected by the target application; determining whether the target file system and the target application are running in the same cluster based on the data storage information; if the target file system and the target application are running in different clusters, establishing a data connection with the target file system based on the permission information; storing the data to be written to the target file system. This solution determines whether the data needs to be written across clusters by identifying the data storage location recorded in the data storage information associated with the target application. If necessary, the corresponding permission information is used to pass the authentication of different clusters, and then a data transmission channel is established and data storage is completed, thereby achieving the effect of using a set of general solutions to solve the problem of two clusters having different authentication systems when writing across clusters.

Description

Technical Field

[0001] The present application belongs to the field of data storage, and in particular, relates to a data writing method and related equipment. Background Art

[0002] A cluster is a group of independent computers connected via a high-speed network. They form a group and are managed as a single system. When a client interacts with the cluster, the cluster acts like an independent server. Cluster configuration is used to improve server availability and scalability.

[0003] Flink is an open source stream processing framework developed by the Apache Software Foundation. Its core is a distributed stream data flow engine written in Java and Scala. Flink executes any stream data program in a data-parallel and pipelined manner. Flink's pipeline runtime system can execute batch and stream processing programs. In addition, Flink's runtime itself also supports the execution of iterative algorithms.

[0004] When a program developed based on Flink is running, some data that needs to be stored will be generated accordingly, such as log files and cache files. The data generated during the running process is stored by default in the file storage system of the cluster where the program is running. However, when analyzing the data, the analysis work may be completed by a service running in another cluster. In this case, the files need to be transferred from the file storage system of one cluster to the file storage system of another cluster, which is inconvenient. Summary of the invention

[0005] A first aspect of an embodiment of the present application provides a data writing method, comprising:

[0006] Acquire data storage information and permission information associated with a target application, wherein the target application is a target application developed based on a target stream processing framework;

[0007] Acquire the data to be written that is collected by running the target application;

[0008] Determining whether the target file system and the target application are running in the same cluster according to the data storage information;

[0009] If the target file system and the target application are running in different clusters, establishing a data connection with the target file system according to the permission information;

[0010] The data to be written is stored in the target file system.

[0011] Based on the data writing method provided in the first aspect of the embodiment of the present application, optionally, the target stream processing framework is a Flink framework.

[0012] Based on the data writing method provided in the first aspect of the embodiment of the present application, optionally, the target file system is an HDFS file system.

[0013] Based on the data writing method provided in the first aspect of the embodiment of the present application, optionally, the target file system uses a Kerberos authentication system;

[0014] The step of establishing a data connection with the target file system according to the permission information includes:

[0015] The permission information is used to establish a data connection with the target file system through the keytab security verification protocol included in the Kerberos authentication system.

[0016] Based on the data writing method provided in the first aspect of the embodiment of the present application, optionally, the method further includes:

[0017] A checkpoint directory is generated on the target file system.

[0018] Based on the data writing method provided in the first aspect of the embodiment of the present application, optionally, the method further includes:

[0019] If the target file system and the target application are running in the same cluster, the data to be written is stored in the target file system.

[0020] A second aspect of an embodiment of the present application provides a data writing device, including:

[0021] A first acquisition unit, the first acquisition unit is used to acquire data storage information and permission information associated with a target application, wherein the target application is a target application developed based on a target stream processing framework;

[0022] A second acquisition unit, the second acquisition unit is used to acquire the to-be-written data collected by the target application during operation;

[0023] a determination unit, the determination unit being configured to determine, based on the data storage information, whether the target file system and the target application are running in the same cluster;

[0024] an establishing unit, the establishing unit being configured to establish a data connection with the target file system according to the permission information if the target file system and the target application are running in different clusters;

[0025] A third aspect of an embodiment of the present application provides a data writing device, including:

[0026] CPU, memory, input and output interfaces, wired or wireless network interfaces, and power supply;

[0027] The memory is a transient storage memory or a persistent storage memory;

[0028] The central processing unit is configured to communicate with the memory and execute instruction operations in the memory on the device to perform the method described in any one of the first aspects of the embodiments of the present application.

[0029] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enable the computer to execute a method as described in any one of the first aspects of the embodiment of the present application.

[0030] A fifth aspect of the embodiments of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute a method as described in any one of the first aspects of the embodiments of the present application.

[0031] It can be seen from the above technical scheme that the embodiment of the present application has the following advantages: The embodiment of the present application provides a data writing method, including: obtaining data storage information and permission information associated with a target application, wherein the target application is a target application developed based on a target stream processing framework; obtaining the data to be written collected by the operation of the target application; determining whether the target file system and the target application are running in the same cluster based on the data storage information; if the target file system and the target application are running in different clusters, establishing a data connection with the target file system based on the permission information; storing the data to be written to the target file system. Based on this scheme, it can be seen that this scheme determines whether the data needs to be written across clusters by identifying the data storage location recorded in the data storage information associated with the target application. If necessary, the corresponding permission information is used to pass the authentication of different clusters, and then a data transmission channel is established and data storage is completed, thereby achieving the effect of using a set of general solutions to solve the problem of two clusters having different authentication systems when writing across clusters. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. It is understood that the drawings provided in this section are only used to better understand the present solution and do not constitute a limitation of the present application.

[0033] Figure 1 A schematic diagram of a flow chart of an embodiment of a data writing method provided by the present application;

[0034] Figure 2 Another flowchart of the data writing method embodiment provided by the present application;

[0035] Figure 3 Another flowchart of the data writing method embodiment provided by the present application;

[0036] Figure 4 A structural schematic diagram of an embodiment of a data writing device provided by the present application;

[0037] Figure 5 This is another structural schematic diagram of an embodiment of the data writing device provided in this application. DETAILED DESCRIPTION

[0038] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application are clearly and completely described below. Obviously, the described embodiments are only embodiments of a part of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should belong to the scope of protection of the present application. At the same time, for the sake of clarity and simplicity, the description of well-known functions and structures is omitted in the following description.

[0039] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0040] A cluster is a group of independent computers connected via a high-speed network. They form a group and are managed as a single system. When a client interacts with the cluster, the cluster acts like an independent server. Cluster configuration is used to improve server availability and scalability.

[0041] Flink is an open source stream processing framework developed by the Apache Software Foundation. Its core is a distributed stream data flow engine written in Java and Scala. Flink executes any stream data program in a data-parallel and pipelined manner. Flink's pipeline runtime system can execute batch and stream processing programs. In addition, Flink's runtime itself also supports the execution of iterative algorithms.

[0042] When a program developed based on Flink is running, some data that needs to be stored will be generated accordingly, such as log files and cache files. The data generated during the running process is stored by default in the file storage system of the cluster where the program is running. However, when analyzing the data, the analysis work may be completed by a service running in another cluster. In this case, the files need to be transferred from the file storage system of one cluster to the file storage system of another cluster, which is inconvenient.

[0043] Specifically, if there are two types of jobs, offline computing and real-time computing, to ensure the stability of the two jobs, offline computing and real-time computing (Flink tasks) are currently performed on two different Hadoop clusters. The offline computing Hadoop cluster currently deploys the Kerberos authentication system. The client must pass the Kerberos authentication before communicating with HDFS; the real-time computing cluster uses the simple authentication method. However, data developers have the need to directly write to the offline large cluster HDFS through real-time computing tasks. Currently, this is done by writing to Kafka through Flink, and then writing to the offline cluster from Kafka. The data link is long and the cost is relatively high. Therefore, it is necessary to open up the channel for offline computing cluster and real-time computing cluster data.

[0044] To solve the above problems, this application provides a new data writing method, please refer to Figure 1 , an embodiment of the data writing method provided in the present application includes: step 101-step 105.

[0045] 101. Obtain data storage information and permission information associated with the target application.

[0046] Specifically, the data storage information and permission information associated with the target application are obtained. The target application is a target application developed based on the target stream processing framework, wherein the target stream processing framework may be a Flink framework, or may be other processing frameworks that support the development and operation of the target application, which is not specifically limited here. The target application may be an application used to implement any function, such as instant messaging applications, shopping applications, and other types of applications, which are not specifically limited here. The target application may run on a mobile terminal or a personal computer or other device. The data storage information is used to describe the storage location of the data generated by the target application. Specifically, the storage location described by the data storage information may be a cluster or other cluster running the target application, which may be determined according to the actual situation, which is not limited here. The permission information is used to establish a data connection with the file system, and records login-related information such as a user name and password, which is not specifically limited here. It is understandable that if the file system adopts simple authentication, the permission information may also be empty, which is not specifically limited here.

[0047] 102. Obtain the data to be written that is collected by running the target application.

[0048] Specifically, the data to be written collected by the target application is obtained. The data to be written is the data that the application needs to store, which can be other categories such as log data or cache data. The specific data is not limited here. The data to be written needs to be stored in the corresponding target file system according to the data storage information associated with the target application.

[0049] 103. Determine, according to the data storage information, whether the target file system and the target application are running in the same cluster.

[0050] Specifically, it is determined whether the target file system and the target application are running in the same cluster based on the data storage information. The data storage information records the path information of the target file system. It is possible to determine whether the target file system and the target application are running in the same cluster based on the path. Specifically, it is possible to determine whether the target file system and the target application are running in the same cluster based on information such as the server address of the target file system. If the target file system and the target application are running in the same cluster, it means that the file system of the cluster itself can be used without further adjustment to complete the data writing task according to the normal data storage process. If the target file system and the target application are running in different clusters, it is necessary to execute step 104. If the target file system and the target application are running in different clusters, a data connection with the target file system is established based on the permission information.

[0051] 104. Establish a data connection with the target file system according to the permission information.

[0052] Specifically, if the target file system and the target application are running in different clusters, a data connection with the target file system is established based on the permission information, that is, the permission information is used to pass the authentication process of the target file system, so that data can be exchanged between two different clusters, ensuring that the data to be written can be transmitted to the target file system through the data connection.

[0053] 105. Store the data to be written into the target file system.

[0054] Specifically, the data to be written is stored in the target file system, transmitted through the data connection path described in step 104, and stored in the corresponding position of the target file system for subsequent calling process, thereby completing the data writing process between different clusters.

[0055] It can be seen from the above technical scheme that the embodiment of the present application has the following advantages: The embodiment of the present application provides a data writing method, including: obtaining data storage information and permission information associated with a target application, wherein the target application is a target application developed based on a target stream processing framework; obtaining the data to be written collected by the operation of the target application; determining whether the target file system and the target application are running in the same cluster based on the data storage information; if the target file system and the target application are running in different clusters, establishing a data connection with the target file system based on the permission information; storing the data to be written to the target file system. Based on this scheme, it can be seen that this scheme determines whether the data needs to be written across clusters by identifying the data storage location recorded in the data storage information associated with the target application. If necessary, the corresponding permission information is used to pass the authentication of different clusters, and then a data transmission channel is established and data storage is completed, thereby achieving the effect of using a set of general solutions to solve the problem of two clusters having different authentication systems when writing across clusters.

[0056] In order to facilitate the use of this method in the actual data writing process, the present application also provides a more detailed embodiment that can be optionally implemented, please refer to Figure 2 , an embodiment of the data writing method provided in the present application includes: step 201-step 207.

[0057] 201. Obtain data storage information and permission information associated with a target application.

[0058] Specifically, the data storage information and permission information associated with the target application are obtained, and the target application is a target application developed based on the target stream processing framework, wherein the target stream processing framework is the Flink framework. Flink is an open source stream processing framework developed by the Apache Software Foundation, and its core is a distributed stream data flow engine written in Java and Scala. Flink executes any stream data program in a data parallel and pipeline manner, and Flink's pipeline runtime system can execute batch processing and stream processing programs. The target application can be an application used to implement any function, such as instant messaging applications, shopping applications and other categories of applications, which are not specifically limited here. The target application can run on devices such as mobile terminals or personal computers. The data storage information is used to describe the storage location of the data generated by the target application. For example, the storage location described by the data storage information can be the HDFS file system under the Hadoop cluster. Hadoop is a distributed system infrastructure developed by the Apache Foundation. Users can develop distributed programs without understanding the underlying details of the distribution. Make full use of the power of the cluster for high-speed computing and storage. Hadoop implements a distributed file system (Distributed File System), one component of which is HDFS (Hadoop Distributed File System). HDFS is highly fault-tolerant and designed to be deployed on low-cost hardware; it provides high throughput to access application data and is suitable for applications with large datasets. HDFS relaxes POSIX requirements and can access data in the file system in a streaming manner.

[0059] Specifically, the storage location described by the data storage information can be the cluster running the target application or other clusters. For example, offline computing and real-time computing are performed on two different Hadoop clusters. The offline computing Hadoop cluster uses the Kerberos authentication system, while the real-time computing cluster uses the simple authentication method. The permission information is used to establish a data connection with the file system. It records login-related information such as user name and password, and can be used to pass the Kerberos authentication system. If the file system uses the simple authentication method, the permission information can also be empty. There is no specific limitation here.

[0060] 202. Obtain the data to be written that is collected by running the target application.

[0061] Specifically, this step is similar to the above Figure 1The corresponding step 102 in the embodiment is similar and will not be described in detail here.

[0062] 203. Determine whether the target file system and the target application run in the same cluster according to the data storage information.

[0063] Specifically, it is determined whether the target file system and the target application are running in the same cluster based on the data storage information. In this embodiment, the target file system running in the same cluster is an HDFS file system authenticated by simple method, and the target file system running in different clusters is an HDFS file system authenticated by kerberos authentication system as an example. If the target file system and the target application are running in the same cluster, step 207 is executed to store the data to be written to the target file system. If the target file system and the target application are running in different clusters, step 204 is executed to use the permission information to pass the keytab security verification protocol included in the kerberos authentication system, and then establish a data connection with the target file system.

[0064] 204. Use the permission information to establish a data connection with the target file system through the keytab security verification protocol included in the Kerberos authentication system.

[0065] Specifically, the permission information is used to pass the keytab security verification protocol included in the kerberos authentication system, and then a data connection is established with the target file system. If the target file system and the target application run in the same cluster, the corresponding permission information needs to be used to pass the keytab security verification protocol, where the permission information is pre-written by the user. Specifically, the Flink security module can be modified. The original Flink security module related to Hadoop will not log in from the keytab when the running cluster does not have kerberos enabled. After the modification, the user can specify whether to log in through the keytab so that data can be output to the cluster with kerberos authentication.

[0066] 205. Store the data to be written into the target file system.

[0067] Specifically, the data to be written is stored in the target file system. To support the execution of this process, the Flink file system module can be expanded. The original Flink file system uses the HDFS of the running cluster as the default file system, and all data output to HDFS will be landed in the running cluster. By expanding this module, the function of dynamically determining the cluster location to store data based on the HDFS path is achieved. Specifically, the storage process can be referred to Figure 3 , the unit structure used in the data transmission process includes:

[0068] Streaming platform, on which users can freely configure, deploy, and save their own Flink applications.

[0069] The Yarn platform is a server that manages computing resources for the Hadoop cluster. Applications are submitted to Yarn, which then starts the cluster.

[0070] The HDFS platform is the file system for storing all data in the Hadoop cluster. The HDFS platform of cluster 1 is a local file system, that is, the HDFS platform in the cluster where the target application runs, and the HDFS platform of cluster 2 is a cross-cluster HDFS platform, that is, the HDFS platform running in a different cluster from the target application.

[0071] Flink-JobManager is the control center of the Flink cluster and is responsible for scheduling cluster tasks.

[0072] Flink-TaskManager is the working node of the Flink cluster, which performs specific computing tasks.

[0073] As an application running on Yarn, the Flink cluster needs to interact with the Yarn platform all the time. At the same time, Flink's unique checkpoint mechanism also requires periodic (user-specified intervals) generation of a directory on the HDFS platform. For TaskManager, it only needs to interact with the HDFS platform to write data.

[0074] The platform submits tasks to Yarn in the mode of cluster 1, and Yarn starts Flink-JobManager. After Flink-JobManager is started, it will switch the authentication mode according to the previous configuration. First, it will apply for computing resources and confirm file resources in the authentication mode of cluster 1. After these steps are completed, it will switch to the authentication mode of cluster 2. After Flink-TaskManager is started, it will also identify the configuration in the settings, and then switch to the authentication mode of cluster 2 to write data to the HDFS of cluster 2.

[0075] 206. Generate a checkpoint directory on the target file system.

[0076] Specifically, a checkpoint directory is generated on the target file system, and a checkpoint directory is periodically generated on the HDFS of cluster 2 for recording.

[0077] 207. Store the data to be written to the target file system.

[0078] Specifically, in the case where the target file system and the target application run in the same cluster, the data to be written is stored in the target file system. The storage process is similar to the prior art and will not be described in detail here.

[0079] It can be seen from the above technical scheme that the embodiment of the present application has the following advantages: The embodiment of the present application provides a data writing method, including: obtaining data storage information and permission information associated with a target application, wherein the target application is a target application developed based on a target stream processing framework; obtaining the data to be written collected by the operation of the target application; determining whether the target file system and the target application are running in the same cluster based on the data storage information; if the target file system and the target application are running in different clusters, establishing a data connection with the target file system based on the permission information; storing the data to be written to the target file system. Based on this scheme, it can be seen that this scheme determines whether the data needs to be written across clusters by identifying the data storage location recorded in the data storage information associated with the target application. If necessary, the corresponding permission information is used to pass the authentication of different clusters, and then a data transmission channel is established and data storage is completed, thereby achieving the effect of using a set of general solutions to solve the problem of two clusters having different authentication systems when writing across clusters.

[0080] The above content describes the data writing method provided by the present application. To support the implementation of the above embodiment, the present application also provides a data writing device. Figure 4 , an embodiment of the data writing device provided by the present application includes:

[0081] A first acquisition unit 401, the first acquisition unit 401 is used to acquire data storage information and permission information associated with a target application, the target application being a target application developed based on a target stream processing framework;

[0082] A second acquisition unit 402, the second acquisition unit 402 is used to acquire the to-be-written data collected by the target application during operation;

[0083] A determination unit 403, the determination unit 403 is used to determine whether the target file system and the target application are running in the same cluster according to the data storage information;

[0084] An establishing unit 404, the establishing unit 404 is used to establish a data connection with the target file system according to the permission information if the target file system and the target application are running in different clusters;

[0085] A storage unit, wherein the storage unit is used to store the data to be written into the target file system.

[0086] In this embodiment, the process executed by each unit in the data writing device is the same as the above Figure 1 or Figure 2 The method flow described in the corresponding embodiment is similar and will not be repeated here.

[0087] Figure 5 It is a structural diagram of a data writing device provided in an embodiment of the present application. The data writing device server 500 may include one or more central processing units (CPU) 501 and a memory 505. The memory 505 stores one or more application programs or data.

[0088] In this embodiment, the specific functional module division in the central processing unit 501 can be the same as the above Figure 4 The functional module division method of each unit described in is similar and will not be repeated here.

[0089] The memory 505 may be a volatile storage or a persistent storage. The program stored in the memory 505 may include one or more modules, each of which may include a series of instruction operations on the server. Furthermore, the central processor 501 may be configured to communicate with the memory 505 and execute a series of instruction operations in the memory 505 on the server 500.

[0090] The server 500 may also include one or more power supplies 502, one or more wired or wireless network interfaces 503, one or more input and output interfaces 504, and / or one or more operating systems, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0091] The CPU 501 can execute the aforementioned Figure 1 The operations performed by the data writing method in the illustrated embodiment will not be described in detail here.

[0092] An embodiment of the present application also provides a computer storage medium, which is used to store computer software instructions used for the above-mentioned data writing method, including a program designed for executing the data writing method.

[0093] The data writing method can be as described above. Figure 1 The data writing method described in .

[0094] The present application also provides a computer program product, which includes computer software instructions, which can be loaded by a processor to implement the above Figure 1 Figure 2The process of any one of the data writing methods.

[0095] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the equivalent transformation of the circuit, the division of the unit, is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0096] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0097] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0098] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent substitution or improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data writing method, characterized in that: include: Acquire data storage information and permission information associated with a target application, wherein the target application is a target application developed based on a target stream processing framework; Acquire the data to be written collected by the target application during operation, wherein the data to be written is log data; Determining whether the target file system and the target application are running in the same cluster according to the data storage information; If the target file system and the target application are running in different clusters, establishing a data connection with the target file system according to the permission information; The data to be written is stored in the target file system.

2. The data writing method according to claim 1, characterized in that: The target stream processing framework is the Flink framework.

3. The data writing method according to claim 1, characterized in that: The target file system is the HDFS file system.

4. The data writing method according to claim 1, characterized in that: The target file system uses the kerberos authentication system; The step of establishing a data connection with the target file system according to the permission information includes: The permission information is used to establish a data connection with the target file system through the keytab security verification protocol included in the Kerberos authentication system.

5. The data writing method according to claim 1, characterized in that: The method further comprises: A checkpoint directory is generated on the target file system.

6. The data writing method according to claim 1, characterized in that: The method further comprises: If the target file system and the target application are running in the same cluster, the data to be written is stored in the target file system.

7. A data writing device, characterized in that: include: A first acquisition unit, the first acquisition unit is used to acquire data storage information and permission information associated with a target application, wherein the target application is a target application developed based on a target stream processing framework; A second acquisition unit, the second acquisition unit is used to acquire the data to be written collected by the target application during operation, wherein the data to be written is log data; a determination unit, the determination unit being configured to determine, based on the data storage information, whether the target file system and the target application are running in the same cluster; an establishing unit, the establishing unit being configured to establish a data connection with the target file system according to the permission information if the target file system and the target application are running in different clusters; A storage unit, wherein the storage unit is used to store the data to be written into the target file system.

8. A data writing device, characterized in that: include: CPU, memory, input and output interfaces, wired or wireless network interfaces, and power supply; The memory is a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory, and execute instruction operations in the memory on the data writing device to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The method comprises instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 6.

10. A computer program product comprising instructions, characterized in that When the method is executed on a computer, the computer is enabled to execute the method according to any one of claims 1 to 6.

Images