Data processing method, device and electronic equipment

Abstract

The present invention discloses a data processing method, device, and electronic device. The method comprises: determining, based on a task identifier carried in a task to be processed, a target file storage system corresponding to the task identifier and the Kerberos authentication status of the target file storage system; when the Kerberos authentication status indicates authentication passed, creating a target connection pool corresponding to the target file storage system and creating at least one data thread in the target connection pool; and obtaining target data corresponding to the task to be processed from the target file storage system based on each data thread. This method solves the problem of low efficiency during data transmission and achieves the effect of significantly reducing performance consumption during data transmission.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a data processing method, device, and electronic device. Background Art

[0002] Hadoop is a recognized standard open source software for big data. Its distributed architecture and huge advantages in data extraction, conversion, and loading have made it increasingly important to ensure data security under the Hadoop architecture as Hadoop is widely used in data analysis and processing platforms.

[0003] Currently, Kerberos authentication is commonly used in Hadoop data sources to ensure data security. Kerberos authentication is a computer network authorization protocol used for identity authentication. Kerberos authentication primarily establishes a shared key between the client and server. Users enter their authentication information once and, upon successful verification, gain access to all data within the Hadoop data source platform. However, this approach poses certain data security risks, potentially leading to data leaks. Furthermore, it is impossible to perform two or more Kerberos authentications simultaneously, resulting in inefficient data transmission.

[0004] In order to solve the above problems, the Kerberos authentication method needs to be improved. Summary of the Invention

[0005] The present invention provides a data processing method, device and electronic equipment to reduce performance consumption in data transmission.

[0006] In a first aspect, an embodiment of the present invention provides a data processing method, including:

[0007] Determine, according to the task identifier carried in the task to be processed, the target file storage system corresponding to the task identifier and the Kerberos authentication status of the target file storage system;

[0008] When the Kerberos authentication status is authentication passed, a target connection pool corresponding to the target file storage system is created, and at least one data thread is created in the target connection pool;

[0009] Based on each data thread, target data corresponding to the task to be processed is obtained from the target file storage system.

[0010] In a second aspect, an embodiment of the present invention further provides a data processing device, including:

[0011] A Kerberos authentication status determination module is used to determine, based on a task identifier carried in a task to be processed, a target file storage system corresponding to the task identifier and the Kerberos authentication status of the target file storage system;

[0012] A data thread creation module is used to create a target connection pool corresponding to the target file storage system when the Kerberos authentication status is authentication passed, and to create at least one data thread in the target connection pool;

[0013] The target data feedback module is used to obtain target data corresponding to the task to be processed from the target file storage system based on each data thread.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising:

[0015] one or more processors;

[0016] a storage device for storing one or more programs,

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of the embodiments of the present invention.

[0018] The technical solution of this embodiment is to determine the target file storage system corresponding to the task identifier carried in the pending task and the Kerberos authentication status of the target file storage system. When the pending task is detected, the target encryption value of the target file storage system is compared with the pre-stored target encryption value to be verified to determine whether the target file storage system has passed the Kerberos authentication. When the Kerberos authentication status is authentication passed, a target connection pool corresponding to the target file storage system is created, and at least one data thread is created in the target connection pool. Based on each created data thread, a pending task request is sent to the target file storage system so that the target file storage system obtains the target data corresponding to the pending task based on the pending task request. Based on each data thread, the target data corresponding to the pending task is obtained from the target file storage system to feed the target data back to the client. By setting a Kerberos authentication cycle and keeping it within the cycle, the data threads of each file storage system that has completed Kerberos authentication can be reused or a new data thread can be re-established. The problem of low efficiency in the data transmission process is solved, and the effect of reducing the performance consumption in data transmission is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] To more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings introduced here only illustrate some of the embodiments to be described by the present invention, and are not exhaustive. A person skilled in the art can derive other drawings based on these drawings without inventive effort.

[0020] Figure 1 A flowchart of a data processing method provided in the first embodiment of the present invention;

[0021] Figure 2 A flowchart of a data processing method provided in the second embodiment of the present invention;

[0022] Figure 3 A flowchart of a data processing method provided in Embodiment 3 of the present invention;

[0023] Figure 4 A schematic structural diagram of a data processing device provided in a fourth embodiment of the present invention;

[0024] Figure 5 This is a structural diagram of an electronic device provided in Example 5 of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0026] Before elaborating on this technical solution in detail, the application scenarios of this technical solution are first introduced in order to have a clearer understanding of this technical solution. In the era of big data, data security is an issue that needs to be considered in the process of collecting, acquiring or analyzing massive amounts of data. In order to ensure that data is not leaked during acquisition, access or transmission, it is usually necessary to first perform security verification on the file system when data is transmitted between application software and file storage system, and establish a data transmission connection between the application software and the file storage system when the file storage system passes the security verification, and perform data transmission and other tasks based on the established data connection.

[0027] Example 1

[0028] Figure 1This is a flow chart of a data processing method provided in the first embodiment of the present invention. This embodiment is applicable to the situation where, when a pending task is detected, it is determined whether Kerberos authentication is required for the target file storage system based on the Kerberos authentication status of the file storage system corresponding to the pending task, and after completing Kerberos authentication, a data thread is created between the application software and the target file storage system for data transmission. The method can be executed by a data processing device, which can be implemented in the form of software and / or hardware. The hardware can be an electronic device, such as a mobile terminal or a PC.

[0029] like Figure 1 As shown, the method includes:

[0030] S110: Determine, according to the task identifier carried in the task to be processed, a target file storage system corresponding to the task identifier and a Kerberos authentication status of the target file storage system.

[0031] Among them, the pending tasks can be understood as tasks such as accessing, querying, modifying or obtaining data in the file storage system. The task identifier can be understood as identification information corresponding to each pending task, such as the task name or task number of the pending task. The target file storage system can be understood as the file storage system that needs to be accessed, such as a distributed file storage system. Exemplarily, each client can connect to multiple distributed file storage systems, and use the file storage system that needs to be accessed as the target file storage system. The Kerberos authentication status can be understood as whether the target file storage system has passed the Kerberos authentication, which can include an authenticated state and an unauthenticated state.

[0032] Specifically, when operations such as accessing, modifying, retrieving, or transferring data in a file storage system are required, corresponding pending tasks are determined based on business needs. When generating pending tasks, the pending tasks include the task type, task name, task identifier, and the file storage system to which the pending tasks correspond. Based on the task identifier in the pending tasks, the target file storage system corresponding to the pending tasks can be determined. The Kerberos authentication status of the target file storage system is then queried to determine whether the target file storage system has passed Kerberos authentication.

[0033] It should be noted that before performing operations such as data transfer from each file storage system, the file system must undergo Kerberos authentication. Only after passing Kerberos authentication can a data thread be established between the application software and the file storage system, and data transmission be performed based on the established data thread to ensure the security of the data transmission process. In the prior art, Kerberos authentication is usually performed on the file storage system when the application software is initialized. However, due to the global limitations of Kerberos authentication, operations can only be performed on the data of the file storage system that has currently passed Kerberos authentication. If data operations need to be performed from other file storage systems during use, Kerberos authentication cannot be performed again, or Kerberos authentication must be performed again on the corresponding file storage system. This process will cause high traffic volume and consume the performance of the application software. In contrast, in this technical solution, Kerberos authentication is performed on the corresponding file storage system only when data operations need to be performed on a certain file storage system, and the reusability of the data thread is guaranteed within a certain period, which can reduce the consumption of application software performance and thereby improve the execution efficiency of the system.

[0034] Optionally, determining the target file storage system corresponding to the task identifier and the Kerberos authentication status of the target file storage system includes: determining a target encryption value corresponding to the target file storage system, and retrieving a target verification encryption value corresponding to the target file storage system from the target mapping table; and determining the Kerberos authentication status corresponding to the target file storage system by comparing the target encryption value with the target verification encryption value.

[0035] Among them, when determining the Kerberos authentication status of the target file storage system, it can be detected through the target encryption value corresponding to the target file storage system. The target encryption value can be understood as the value obtained after the target file storage system is encrypted. For example, it can be a hash value corresponding to the target file storage system obtained by encrypting the target file storage system based on a hash encryption algorithm. In this case, the hash value is the target encryption value corresponding to the target file storage system. The target mapping table can be understood as a file that records the correspondence between each file storage system and the corresponding encryption value. The target verification encryption value can be understood as the encryption value corresponding to the target file storage system recorded in the target mapping table, which can be used to compare with the target encryption value of the target file storage system obtained.

[0036] Specifically, to determine whether the target file storage system has passed Kerberos authentication, after determining the target file storage system, the target encryption value corresponding to the target file storage system is obtained. Then, based on the file identifier of the target file storage system, the target verification encryption value corresponding to the target file storage system is determined from the target mapping table. Generally speaking, the target encryption value and target verification encryption value corresponding to the same target file storage system are the same. By comparing the target encryption value and the target verification encryption value, and based on the comparison result, it can be determined whether the target file storage system has passed Kerberos authentication.

[0037] Optionally, the Kerberos authentication status corresponding to the target file storage system is determined by comparing the target encryption value with the target verification encryption value, including: when the target encryption value is consistent with the target verification encryption value, the authentication status of the Kerberos authentication is determined to be passed; when the target encryption value is inconsistent with the target verification encryption value, the authentication status of the Kerberos authentication is determined to be failed.

[0038] Exemplarily, if the target file storage system is encrypted based on a hash algorithm, a correspondence is established between the obtained hash value and the target file storage system to determine the target encryption value corresponding to the target file storage system based on the correspondence. At the same time, the hash value corresponding to the target file storage system is stored in the target mapping table as the target verification encryption value corresponding to the target file storage system. When it is necessary to confirm the Kerberos authentication status of the target file storage system, the target encryption value corresponding to the target file storage system is obtained, and the target verification encryption value corresponding to the target file storage system is retrieved from the target mapping table, and the two encryption values ​​are compared. If the comparison result is the same, it indicates that the target file storage system has passed the Kerberos authentication; conversely, if the comparison result is different, it indicates that the target file storage system has not passed the Kerberos authentication.

[0039] S120. When the Kerberos authentication status is authentication passed, a target connection pool corresponding to the target file storage system is created, and at least one data thread is created in the target connection pool.

[0040] The connection pool can be used to create and manage data threads between application software and various file storage systems. Each file storage system can correspond to a connection pool. The target connection pool can be understood as the connection pool corresponding to the target file storage system. The data thread can be understood as the thread used for data transmission.

[0041] Specifically, when the Kerberos authentication status corresponding to the target file storage system is "authenticated passed," it indicates that the target file storage system has been granted permission for data transmission, and data transmission in this state is relatively secure. Based on this, a target connection pool can be established between the application software and the target file storage system, and at least one data thread can be created based on the target connection pool for data transmission between the application software and the target file storage system.

[0042] For example, when a pending task is received and the Kerberos authentication status of the target file storage system corresponding to the pending task is authenticated, a target connection pool corresponding to the target file storage system can be established, and then at least one data thread can be created in the target connection pool, and then data transmission can be performed based on at least one data thread according to the task information of the pending task.

[0043] S130. Acquire target data corresponding to the task to be processed from the target file storage system based on each data thread.

[0044] The target data may be understood as data corresponding to the task to be processed.

[0045] Specifically, after creating a data thread, the application software can send pending tasks to the target file storage system, and then the target file storage system obtains the target data corresponding to the pending tasks based on the received pending tasks, and feeds the target data back to the application software through each data thread.

[0046] Optionally, based on each data thread, obtaining the target data corresponding to the pending task from the target file storage system includes: generating a corresponding pending task request based on the pending task information corresponding to the pending task; wherein the pending task information includes at least one of a task name, a task identifier, and a task attribute; based on each data thread corresponding to the target file storage system, sending the pending task request to each target file storage system to obtain the target data corresponding to the pending task.

[0047] The pending task information may be understood as information corresponding to the pending task, including the pending task name, type, task identifier, and corresponding target file storage system. The pending task request may be understood as a request generated based on the pending task information.

[0048] Specifically, the application software displays an interface that allows users to enter task information corresponding to the pending task. Clicking a button to confirm the entered information generates a pending task request. Each pending task request is then sent to the target file storage system via a created data thread. The target file storage system retrieves the target data corresponding to the pending task based on the pending task request and feeds the target data back to the application software via each data thread.

[0049] Optionally, the sending of the pending task request to each target file storage system based on each data transmission connection corresponding to the target file storage system includes: monitoring each data thread, and sending the pending task request to the target file storage system based on an idle data thread; wherein, the idle data thread is a data thread that is not performing data transmission at the current moment; if the idle data thread does not perform data transmission within a preset time period, the idle data thread will be closed.

[0050] Specifically, the target connection pool can also manage each data thread created. When creating each data thread, the execution time of each data thread can be pre-set, such as one hour. Within the preset time, the target connection pool can monitor each data thread in the connection pool to obtain the task status of each data thread. When receiving a pending task request, the working status of each data thread is obtained, and the pending task request is preferentially sent to the target file storage system through an idle data thread. During the monitoring process, if a data thread is found to be consistently idle within the preset time, the idle data thread can be closed. If the number of pending task requests is greater than the number of data threads in the same time period, the target connection pool can also create a new data thread to meet the data transmission needs.

[0051] It should be noted that when performing Kerberos authentication on the target file storage system, a validity period for the target file storage system can be pre-set. During this validity period, each data thread connected to the target file storage system can be used multiple times, or new data threads can be created in the target connection pool, without the need for re-Kerberos authentication on the data threads in the target connection pool. When the validity period expires, Kerberos authentication can be performed again on the target file storage system to allow data transfer and other operations to be performed again on the target file storage system.

[0052] The technical solution of this embodiment is to determine the target file storage system corresponding to the task identifier carried in the pending task and the Kerberos authentication status of the target file storage system. When the pending task is detected, the target encryption value of the target file storage system is compared with the pre-stored target encryption value to be verified to determine whether the target file storage system has passed the Kerberos authentication. When the Kerberos authentication status is authentication passed, a target connection pool corresponding to the target file storage system is created, and at least one data thread is created in the target connection pool. Based on each created data thread, a pending task request is sent to the target file storage system so that the target file storage system obtains the target data corresponding to the pending task based on the pending task request. Based on each data thread, the target data corresponding to the pending task is obtained from the target file storage system to feed the target data back to the client. By setting a Kerberos authentication cycle and keeping it within the cycle, the data threads of each file storage system that has completed Kerberos authentication can be reused or a new data thread can be re-established. The problem of low efficiency in the data transmission process is solved, and the effect of reducing the performance consumption in data transmission is achieved.

[0053] Example 2

[0054] As an alternative embodiment of the above embodiment, Figure 2 A flow chart of a data processing method provided in the second embodiment of the present invention, optionally, before determining the target file storage system corresponding to the task identifier carried in the task to be processed and the Kerberos authentication status of the target file storage system, Kerberos authentication is performed on each file storage system to be processed based on the pre-verified credential ticket corresponding to each file system to be processed, and when the target file storage system fails the Kerberos authentication, Kerberos authentication is performed on the target file storage system.

[0055] like Figure 2 As shown, the method includes:

[0056] S210: Determine the credential ticket to be verified corresponding to each file storage system to be processed.

[0057] The pending file storage systems can be understood as the various file storage systems connected to the application software. The pending credential tickets can be understood as the credential tickets required to authenticate the pending file storage systems, including a key table and configuration file corresponding to Kerberos authentication. The key table includes authentication user information, access addresses corresponding to the pending tasks, and access keys corresponding to each access address. The configuration file includes the authentication addresses corresponding to Kerberos authentication.

[0058] Specifically, the application software can simultaneously connect to multiple pending file storage systems. Each pending file storage system must pass Kerberos authentication before a connection pool can be established with the application software, and data transmission can then be performed based on the data thread created in the connection pool. When performing Kerberos authentication on each pending file storage system, the pending verification credential tickets corresponding to each pending file system are different. When a pending file storage system is selected for Kerberos authentication, the pending verification credential ticket corresponding to the pending file storage system is first determined. For example, this can be determined based on the correspondence between the pending file storage system and each pending verification credential ticket.

[0059] S220: Encrypt each credential ticket to be verified to obtain an encrypted value to be verified corresponding to each credential ticket to be verified, and store each encrypted value to be verified in a target mapping table to determine the Kerberos authentication status of the target file storage system based on the target mapping table.

[0060] The encrypted value to be verified can be understood as an encrypted value obtained by encrypting each file storage system to be processed based on an encryption algorithm. The target mapping table includes each file storage system to be processed and the corresponding encrypted value to be verified.

[0061] Specifically, based on a preset encryption algorithm, the to-be-verified credential ticket corresponding to each pending file storage system is encrypted to obtain the to-be-verified encrypted value corresponding to each pending file storage system, and then each to-be-verified encrypted value is stored in a target mapping table. When a pending task is detected, the target file storage system corresponding to the pending task is determined, and the target to-be-verified encrypted value corresponding to the target file storage system is retrieved from the target mapping table. The obtained target encrypted value corresponding to the target file storage system is compared with the target to-be-verified encrypted value to determine the Kerberos authentication status of the target file storage system.

[0062] S230: Determine, according to the task identifier carried in the task to be processed, the target file storage system corresponding to the task identifier and the Kerberos authentication status of the target file storage system.

[0063] S240: When the Kerberos authentication status is not authenticated, obtain the to-be-authenticated credential ticket corresponding to the target file storage system, and perform Kerberos authentication on the target file system based on the to-be-authenticated credential ticket.

[0064] The credential ticket to be authenticated can be understood as the credential ticket corresponding to the target file storage system when performing Kerberos authentication on each file storage system to be processed.

[0065] Specifically, when the authentication status of the Kerberos authentication corresponding to the target file storage system is failed, the target file storage system needs to be Kerberos authenticated. At this time, the authentication credential ticket corresponding to the target file storage system is obtained, and then the target file storage system is Kerberos authenticated based on the authentication credential ticket.

[0066] Optionally, performing Kerberos authentication on the target file system based on the credential ticket to be authenticated includes: obtaining a target authentication credential ticket corresponding to the target file storage system, and generating a corresponding authentication request based on the target authentication credential ticket; sending the authentication request to a Kerberos authentication server to perform Kerberos authentication on the target file storage system based on the Kerberos authentication server, and feeding back the Kerberos authentication status corresponding to the target file storage system; receiving the Kerberos authentication status corresponding to the target file storage system sent by the Kerberos server.

[0067] The target authentication credential ticket can be understood as the credential ticket corresponding to the target file storage system. The authentication request can be understood as a request for authentication to a Kerberos authentication server generated based on the target authentication credential ticket. The Kerberos authentication server can be understood as the server that performs Kerberos authentication on each file storage system.

[0068] Specifically, when performing Kerberos authentication on the target file storage system, the authentication ticket corresponding to the target file storage system is obtained. Based on the key table and configuration file in the authentication ticket, a corresponding authentication request is generated. The authentication request is then sent to the Kerberos authentication server, which authenticates the target file storage system and determines the Kerberos authentication status of the target file storage system.

[0069] In actual applications, the application software obtains the corresponding credential ticket to be authenticated from the target file storage system and generates a corresponding authentication request. The authentication request is sent to the Kerberos authentication server, which includes a key table and configuration file, as well as information about the target file storage system and the application software that communicates with it. After receiving the authentication request, the Kerberos authentication server parses it and randomly generates a password and two identical credential tickets. These two credential tickets are sent to the application software, one of which is for the application software and is encrypted using the application software's password; the other is for the target file storage system and is encrypted using the target file storage system's password. However, the Kerberos authentication server does not send the credential ticket directly to the target file storage system; instead, the application software hands it over to the target file storage system. Because the credential ticket is encrypted by the Kerberos authentication server, it cannot be forged or tampered with during transmission. After obtaining the two tickets, the application software first decrypts the corresponding credential ticket using its own password to generate an authentication factor. It then sends this authentication factor and the other credential ticket to the target file storage system, which then decrypts the corresponding credential ticket using its own password to obtain the second authentication factor. If the two authentication factors are identical, the target file storage system has passed Kerberos authentication. Otherwise, the target file storage system has failed Kerberos authentication.

[0070] S250: When the Kerberos authentication status is authentication passed, a target connection pool corresponding to the target file storage system is created, and at least one data thread is created in the target connection pool.

[0071] S260: Acquire target data corresponding to the task to be processed from the target file storage system based on each data thread.

[0072] The technical solution of this embodiment determines the credential ticket to be verified corresponding to each file storage system to be processed, encrypts each credential ticket to be verified, obtains the encrypted value to be verified corresponding to each credential ticket to be verified, and stores each encrypted value to be verified in a target mapping table, so that when a task to be processed is detected, the target encryption value of the target file storage system corresponding to the task to be processed is compared with the target verification encryption value, and the Kerberos authentication status of the target file storage system is determined based on the target mapping table. When the Kerberos authentication status is not authenticated, the credential ticket to be authenticated corresponding to the target file storage system is obtained, and the target file system is Kerberos authenticated based on the credential ticket to be authenticated to complete the Kerberos authentication of the target storage system, so as to create a connection between the application software and the target storage system and complete the data transmission. The problem of low efficiency in the data transmission process is solved, and the effect of reducing the performance consumption in data transmission is achieved.

[0073] Example 3

[0074] In a specific example, Figure 3 As shown, when a pending task is detected, the task identifier corresponding to the pending task is determined, the data type corresponding to the pending task is determined based on the task identifier, and then the data source connection corresponding to the pending task (i.e., the target file storage system) is filled in, so that when the target file storage system needs to be processed, a data thread is established between the application software and the target file storage system to perform data transmission based on the established data thread.

[0075] First, determine whether the target file storage system for data transmission requires authentication. When the Kerberos authentication status of the target file storage system is passed, there is no need to perform Kerberos authentication on the target file storage system again. A data thread can be directly created between the application software and the target file storage system, and the driver connection can be tested (i.e., data transmission is performed based on the data thread). If the Kerberos authentication status of the target file storage system is failed, it is necessary to fill in the authentication information, generate a corresponding authentication request based on the pending task information of the pending task, upload the authentication credentials (i.e., authentication request), and send the authentication request corresponding to the pending task to the Kerberos authentication server. The Kerberos authentication authenticates the corresponding authentication credentials of the target file storage system and determines whether the target file storage system has passed the Kerberos authentication.

[0076] Exemplarily, when a pending task is detected, the target file storage system corresponding to the pending task is determined, and Kerberos authentication is performed on the target file storage system. The process of Kerberos authentication is: setting the attribute hadoop.security.auth_to_local of the org.apache.hadoop.conf.Configuration object, setting its rules for domain name resolution; setting the system context environment variable java.security.krb5.conf in the synchronization block; refreshing sun.security.krb5.Config.refresh() in the synchronization block; setting Configuration, UserGroupInformation.setConfiguration(conf) in the synchronization block; completing Kerberos authentication in the synchronization block.

[0077] If the target file storage system can pass Kerberos authentication, a connection pool can be created between the application software and the target file storage system. For example, when creating a connection pool corresponding to the target file storage system, a connection can be established using the authenticated UserGroupInformation and a timer can be set based on the server's key table refresh to periodically refresh the Kerberos key table. At least one data thread is created in the connection pool to perform data transmission based on each data thread. The connection pool can also manage each created data thread. When creating each data thread, a pre-set execution time for each data thread can be set, such as one hour. During this pre-set time, the connection pool can monitor each data thread in the connection pool to obtain the task status of each data thread. Upon receiving pending task requests, the connection pool obtains the working status of each data thread and preferentially sends the pending task requests to the target file storage system via idle data threads. During the monitoring process, if a data thread is consistently idle for the pre-set time, the idle data thread can be closed. If the number of pending task requests exceeds the number of data threads within the same time period, the target connection pool can also create a new data thread to meet the data transmission needs.

[0078] The technical solution of this embodiment determines the target file storage system corresponding to the task identifier and the Kerberos authentication status of the target file storage system based on the task identifier carried in the pending task. When a pending task is detected, the target encryption value of the target file storage system is compared with the pre-stored target encryption value to be verified to determine whether the target file storage system has passed the Kerberos authentication. When the Kerberos authentication status is authentication passed, a target connection pool corresponding to the target file storage system is created, and at least one data thread is created in the target connection pool. Based on each created data thread, a pending task request is sent to the target file storage system, so that the target file storage system obtains the target data corresponding to the pending task based on the pending task request. Based on each data thread, the target data corresponding to the pending task is obtained from the target file storage system to feed the target data back to the client. This solves the problem of low efficiency in the data transmission process when multiple Kerberos authentications are required, and achieves a significant effect of reducing performance consumption in data transmission.

[0079] Example 4

[0080] Figure 4 A data processing device is provided in accordance with a fourth embodiment of the present invention. The device includes a Kerberos authentication state determination module 310 , a data thread creation module 320 , and a target data feedback module 330 .

[0081] The Kerberos authentication status determination module 310 is configured to determine, based on a task identifier carried in a task to be processed, a target file storage system corresponding to the task identifier and the Kerberos authentication status of the target file storage system;

[0082] The data thread creation module 320 is configured to create a target connection pool corresponding to the target file storage system and create at least one data thread in the target connection pool when the Kerberos authentication status is authentication passed;

[0083] The target data feedback module 330 is configured to obtain target data corresponding to the task to be processed from the target file storage system based on each data thread.

[0084] The technical solution of this embodiment determines the target file storage system corresponding to the task identifier and the Kerberos authentication status of the target file storage system based on the task identifier carried in the pending task. When a pending task is detected, the target encryption value of the target file storage system is compared with the pre-stored target encryption value to be verified to determine whether the target file storage system has passed the Kerberos authentication. When the Kerberos authentication status is authentication passed, a target connection pool corresponding to the target file storage system is created, and at least one data thread is created in the target connection pool. Based on each created data thread, a pending task request is sent to the target file storage system, so that the target file storage system obtains the target data corresponding to the pending task based on the pending task request. Based on each data thread, the target data corresponding to the pending task is obtained from the target file storage system to feed the target data back to the client. This solves the problem of low efficiency in the data transmission process when multiple Kerberos authentications are required, and achieves a significant effect of reducing performance consumption in data transmission.

[0085] Based on any optional technical solution in the embodiments of the present invention, optionally, the data processing device further includes:

[0086] a module for determining a credential ticket to be verified, configured to determine a credential ticket to be verified corresponding to each file storage system to be processed; wherein each credential ticket to be verified includes a key table and a configuration file corresponding to Kerberos authentication, wherein the key table includes authentication user information, an access address corresponding to the task to be processed, and an access key corresponding to each access address, and the configuration file includes an authentication address corresponding to Kerberos authentication;

[0087] An encryption module is used to encrypt each credential ticket to be verified, obtain the encrypted value to be verified corresponding to each credential ticket to be verified, and store each encrypted value to be verified in a target mapping table to determine the Kerberos authentication status of the target file storage system based on the target mapping table; wherein the target mapping table includes each file storage system to be processed and the corresponding encrypted value to be verified.

[0088] Based on any optional technical solution in the embodiment of the present invention, optionally, the Kerberos authentication status determination module includes:

[0089] a target verification encryption value retrieving submodule, configured to determine a target encryption value corresponding to the target file storage system and retrieve the target verification encryption value corresponding to the target file storage system from the target mapping table;

[0090] The Kerberos authentication status determination submodule is used to determine the Kerberos authentication status corresponding to the target file storage system by comparing the target encryption value with the target verification encryption value.

[0091] Based on any optional technical solution in the embodiment of the present invention, optionally, the Kerberos authentication status determination submodule includes:

[0092] an authentication pass unit, configured to determine that the authentication status of the Kerberos authentication is passed when the target encryption value is consistent with the target verification encryption value;

[0093] an authentication failure unit, configured to determine that the authentication status of the Kerberos authentication is failed when the target encryption value is inconsistent with the target verification encryption value;

[0094] Based on any optional technical solution in the embodiments of the present invention, optionally, the data processing device further includes:

[0095] a credential ticket to be authenticated acquiring unit, configured to acquire the credential ticket to be authenticated corresponding to the target file storage system when the Kerberos authentication status is unauthenticated;

[0096] A Kerberos authentication unit is configured to perform Kerberos authentication on the target file system based on the credential ticket to be authenticated.

[0097] Based on any optional technical solution in the embodiment of the present invention, optionally, the Kerberos authentication unit includes:

[0098] an authentication request generating subunit, configured to obtain a target authentication credential ticket corresponding to the target file storage system, and generate a corresponding authentication request based on the target authentication credential ticket;

[0099] an authentication request sending subunit, configured to send the authentication request to a Kerberos authentication server, so as to perform Kerberos authentication on the target file storage system based on the Kerberos authentication server, and to feed back a Kerberos authentication status corresponding to the target file storage system;

[0100] The Kerberos authentication status feedback subunit is used to receive the Kerberos authentication status corresponding to the target file storage system sent by the Kerberos server.

[0101] Based on any optional technical solution in the embodiments of the present invention, optionally, the target data feedback module includes:

[0102] a pending task request generation submodule, configured to generate a corresponding pending task request based on pending task information corresponding to the pending task; wherein the pending task information includes at least one of a task name, a task identifier, and a task attribute;

[0103] The target data acquisition submodule is used to send the pending task request to each target file storage system based on each data thread corresponding to the target file storage system, and acquire the target data corresponding to the pending task.

[0104] Based on any optional technical solution in the embodiments of the present invention, optionally, the target data acquisition submodule includes:

[0105] a data thread monitoring unit, configured to monitor each data thread and send the pending task request to the target file storage system based on an idle data thread; wherein the idle data thread is a data thread that is not currently transmitting data;

[0106] The data thread closing unit is configured to close the idle data thread if the idle data thread does not perform data transmission within a preset time period.

[0107] The data processing device provided by the embodiment of the present invention can execute the data processing method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0108] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the embodiments of the present invention.

[0109] Example 5

[0110] Figure 5 This is a structural diagram of an electronic device provided in Example 5 of the present invention. Figure 5 A block diagram of an exemplary electronic device 40 suitable for implementing exemplary embodiments of the present invention is shown. Figure 5 The electronic device 40 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention.

[0111] like Figure 5 As shown, electronic device 40 is a general-purpose computing device. Components of electronic device 40 may include, but are not limited to, one or more processors or processing units 401, system memory 402, and a bus 403 connecting various system components (including system memory 402 and processing unit 401).

[0112] Bus 403 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0113] The electronic device 40 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 40, including volatile and non-volatile media, removable and non-removable media.

[0114] System memory 402 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 404 and / or cache memory 405. Electronic device 40 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 406 may be used to read and write non-removable, non-volatile magnetic media ( Figure 5 Not shown, often called a "hard drive"). Although Figure 5 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 403 via one or more data medium interfaces. Memory 402 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0115] A program / utility 408 having a set (at least one) of program modules 407 may be stored, for example, in memory 402. Such program modules 407 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 407 generally perform the functions and / or methods of the embodiments described herein.

[0116] The electronic device 40 may also communicate with one or more external devices 409 (e.g., keyboard, pointing device, display 410, etc.), and may also communicate with one or more devices that enable a user to interact with the electronic device 40, and / or any device that enables the electronic device 40 to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through an input / output (I / O) interface 411. Furthermore, the electronic device 40 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 412. As shown, the network adapter 412 communicates with other modules of the electronic device 40 via the bus 403. It should be understood that although Figure 5 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 40, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0117] The processing unit 401 executes various functional applications and data processing by running programs stored in the system memory 402, such as implementing the data processing method provided by the embodiment of the present invention.

[0118] Example 6

[0119] Embodiment 6 of the present invention further provides a storage medium containing computer-executable instructions, wherein the computer-executable instructions, when executed by a computer processor, are used to perform a data processing method, the method comprising:

[0120] Determine, according to the task identifier carried in the task to be processed, the target file storage system corresponding to the task identifier and the Kerberos authentication status of the target file storage system;

[0121] When the Kerberos authentication status is authentication passed, a target connection pool corresponding to the target file storage system is created, and at least one data thread is created in the target connection pool;

[0122] Based on each data thread, target data corresponding to the task to be processed is obtained from the target file storage system.

[0123] The computer storage medium of the embodiment of the present invention may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0124] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0125] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0126] The computer program code for performing the operations of the embodiments of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0127] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data processing method, characterized in that: include: Determine, according to the task identifier carried in the task to be processed, the target file storage system corresponding to the task identifier and the Kerberos authentication status of the target file storage system; When the Kerberos authentication status is authentication passed, a target connection pool corresponding to the target file storage system is created, and at least one data thread is created in the target connection pool; Acquiring target data corresponding to the task to be processed from the target file storage system based on each data thread; Wherein, before determining the target file storage system corresponding to the task identifier carried in the task to be processed and the Kerberos authentication status of the target file storage system, the method further includes: determining the credential ticket to be verified corresponding to each file storage system to be processed; wherein each credential ticket to be verified includes a key table and a configuration file corresponding to the Kerberos authentication, wherein the key table includes authentication user information, an access address corresponding to the task to be processed and an access key corresponding to each access address, and the configuration file includes an authentication address corresponding to the Kerberos authentication; encrypting each credential ticket to be verified to obtain an encrypted value to be verified corresponding to each credential ticket to be verified, and storing each encrypted value to be verified in a target mapping table to determine the Kerberos authentication status of the target file storage system based on the target mapping table; wherein the target mapping table includes each file storage system to be processed and the corresponding encrypted value to be verified.

2. The method according to claim 1, characterized in that The determining of the target file storage system corresponding to the task identifier and the Kerberos authentication status of the target file storage system includes: Determining a target encryption value corresponding to the target file storage system, and retrieving a target verification encryption value corresponding to the target file storage system from the target mapping table; The Kerberos authentication status corresponding to the target file storage system is determined by comparing the target encryption value with the target verification encryption value.

3. The method according to claim 2, characterized in that The determining of the Kerberos authentication status corresponding to the target file storage system by comparing the target encryption value with the target verification encryption value includes: When the target encryption value is consistent with the target verification encryption value, determining that the authentication status of the Kerberos authentication is passed; When the target encryption value is inconsistent with the target verification encryption value, it is determined that the authentication status of the Kerberos authentication is failed.

4. The method according to claim 3, characterized in that Also includes: When the Kerberos authentication state is not authenticated, a to-be-authenticated credential ticket corresponding to the target file storage system is obtained, and Kerberos authentication is performed on the target file storage system based on the to-be-authenticated credential ticket.

5. The method according to claim 4, characterized in that The performing Kerberos authentication on the target file storage system based on the credential ticket to be authenticated includes: Acquire a target authentication credential ticket corresponding to the target file storage system, and generate a corresponding authentication request based on the target authentication credential ticket; Sending the authentication request to a Kerberos authentication server to perform Kerberos authentication on the target file storage system based on the Kerberos authentication server, and feeding back a Kerberos authentication status corresponding to the target file storage system; Receive the Kerberos authentication status corresponding to the target file storage system sent by the Kerberos server.

6. The method according to claim 1, characterized in that The acquiring, based on each data thread, target data corresponding to the to-be-processed task from the target file storage system includes: Generate a corresponding pending task request based on pending task information corresponding to the pending task; wherein the pending task information includes at least one of a task name, a task identifier, and a task attribute; Based on each data thread corresponding to the target file storage system, the to-be-processed task request is sent to each target file storage system to obtain target data corresponding to the to-be-processed task.

7. The method according to claim 6, characterized in that The step of sending the pending task request to each target file storage system based on each data thread corresponding to the target file storage system and acquiring target data corresponding to the pending task includes: Monitoring each data thread and sending the pending task request to the target file storage system based on an idle data thread; wherein the idle data thread is a data thread that is not currently transmitting data; If the idle data thread does not transmit data within a preset time period, the idle data thread will be closed.

8. A data processing device, characterized in that: include: A Kerberos authentication status determination module is used to determine, based on a task identifier carried in a task to be processed, a target file storage system corresponding to the task identifier and the Kerberos authentication status of the target file storage system; A data thread creation module is used to create a target connection pool corresponding to the target file storage system when the Kerberos authentication status is authentication passed, and to create at least one data thread in the target connection pool; A target data feedback module is used to obtain target data corresponding to the task to be processed from the target file storage system based on each data thread; Wherein, the data processing device also includes: a credential ticket determination module for determining the credential ticket to be verified corresponding to each file storage system to be processed; wherein, each credential ticket to be verified includes a key table and a configuration file corresponding to Kerberos authentication, the key table includes authentication user information, an access address corresponding to the task to be processed, and an access key corresponding to each access address, and the configuration file includes an authentication address corresponding to Kerberos authentication; an encryption module for encrypting each credential ticket to be verified, obtaining an encrypted value to be verified corresponding to each credential ticket to be verified, and storing each encrypted value to be verified in a target mapping table, so as to determine the Kerberos authentication status of the target file storage system based on the target mapping table; wherein, the target mapping table includes each file storage system to be processed and the corresponding encrypted value to be verified.

9. An electronic device, characterized in that: The electronic device comprises: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method according to any one of claims 1 to 7.

Images