Data processing method, device, electronic device and computer readable storage medium

By implementing the method of dynamically obtaining authentication information in a distributed cluster, the problem of high maintenance cost of authentication information in the prior art is solved, and the degree of automation of data processing is improved.

CN114417306BActive Publication Date: 2024-12-03BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210002087.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-01-04
Publication Date
2024-12-03
Estimated Expiration
2042-01-04

AI Technical Summary

Technical Problem

In the prior art, technical personnel need to manually issue authentication information for all data to be processed to each work node in the distributed cluster, resulting in an increase in maintenance cost of authentication information.

Method used

When it is necessary to perform data processing tasks, the driver node sends an information acquisition request to the information management node, receives and stores authentication information, and the computing node sends an authentication request to the authentication management node based on the acquired authentication information, obtains target data from the first database, and performs Data processing to achieve dynamic and automatic acquisition of authentication information.

Benefits of technology

It reduces the maintenance cost of authentication information, avoids the need for technicians to issue authentication information for all data on each node, and improves the degree of automation of data processing.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114417306B_ABST
    Figure CN114417306B_ABST
Patent Text Reader

Abstract

The embodiment of the present invention provides a data processing method, device, electronic device and computer-readable storage medium, which relates to the field of database technology. The method includes: when a data processing task needs to be executed, a driving node sends an information acquisition request to an information management node for acquiring authentication information of target data to be processed by the data processing task; receiving authentication information of the target data sent by the information management node; storing configuration information and keys in a specified directory in file format respectively, and a computing node that executes the data processing task acquires configuration information and keys from the specified directory; the driving node sends an entity name to the computing node, and the computing node sends an authentication request to the authentication management node according to the authentication information, acquires target data from a first database if the authentication is passed, and processes the target data according to the data processing task. Based on this, the maintenance cost of authentication information can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of big data technology, and in particular to a data processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] A distributed cluster contains multiple worker nodes, in which drivers and executors can be deployed. The driver is used to convert the code logic submitted by the technician into data processing tasks (for example, Spark tasks) and dispatch them to the executor. The executor is used to execute data processing tasks and obtain processing results.

[0003] In the related technology, technicians manually send the authentication information of all data to be processed to each working node in the cluster. Then, when the executor deployed in a working node needs to run a data processing task, it can obtain the authentication information of the target data currently to be processed from the working node. After passing the Kerberos authentication based on the authentication information, the executor can obtain the target data from the database and perform the data processing task.

[0004] It can be seen that in the related technology, the technical personnel need to send the authentication information of all the data to be processed to each working node in advance, which will increase the maintenance cost of the authentication information. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a data processing method, device, electronic device and computer-readable storage medium, which can reduce the maintenance cost of authentication information. The specific technical solution is as follows:

[0006] In a first aspect of the implementation of the present invention, a data processing method is first provided, the method is applied to a driver node, and the method comprises:

[0007] When a data processing task needs to be executed, an information acquisition request is sent to the information management node to obtain authentication information of the target data to be processed by the data processing task;

[0008] Receiving authentication information of the target data sent by the information management node; wherein the authentication information includes configuration information, a key and an entity name;

[0009] The configuration information and the key are stored in a specified directory in a file format, respectively, so that a computing node for executing the data processing task obtains the configuration information and the key from the specified directory;

[0010] The entity name is sent to the computing node so that the computing node sends an authentication request to the authentication management node according to the authentication information, obtains the target data from the first database if the authentication is successful, and processes the target data according to the data processing task.

[0011] Optionally, storing the configuration information and the key in a file format to a specified directory respectively includes:

[0012] The configuration information and key are stored in a specified directory of the distributed file system in file format.

[0013] Optionally, after storing the configuration information and the key in a specified directory in file format respectively, the method further includes:

[0014] Sending to a resource management node a resource allocation request for acquiring computing resources required by the computing node to execute the data processing task, so as to allocate computing resources to the computing node;

[0015] The data processing task is issued to the computing node so that the computing node executes the data processing task according to the allocated computing resources.

[0016] Optionally, the method further includes:

[0017] After the data processing task is completed, releasing the computing resources allocated to the computing node;

[0018] Delete the configuration information and key recorded in the specified directory.

[0019] In a second aspect of the present invention, a data processing method is provided, the method being applied to a computing node, the method comprising:

[0020] Obtain configuration information and keys in the authentication information of the target data to be processed by the data processing task from the designated directory; wherein the configuration information and keys are obtained from the information management node and stored in the designated directory when the driver node needs to execute the data processing task;

[0021] Obtaining the entity name in the authentication information sent by the driver node;

[0022] Sending an authentication request to the authentication management node according to the configuration information, key and entity name;

[0023] When the authentication is passed, the target data is obtained from the first database, and the target data is processed according to the data processing task.

[0024] Optionally, the configuration information includes the address of the authentication management node;

[0025] The sending an authentication request to the authentication management node according to the configuration information, the key and the entity name includes:

[0026] According to the address of the authentication management node, sending an authentication request carrying the key and the entity name to the authentication management node to obtain an authentication ticket;

[0027] The step of obtaining the target data from the first database when the authentication is successful includes:

[0028] A data read request carrying the authentication ticket is sent to the first database, so as to obtain the target data from the first database when the first database verifies that the authentication ticket is legal.

[0029] Optionally, the method further includes:

[0030] A data write request carrying the authentication ticket is sent to the second database, so that when the second database verifies that the authentication ticket is legitimate, a processing result of processing the target data is written into the second database.

[0031] In a third aspect of the implementation of the present invention, a data processing device is provided, the device is applied to a driver node, and the device includes:

[0032] An information acquisition request sending module is used to send, when a data processing task needs to be executed, to the information management node an information acquisition request for acquiring authentication information of target data that needs to be processed by the data processing task;

[0033] An authentication information receiving module, used to receive the authentication information of the target data sent by the information management node; wherein the authentication information includes configuration information, a key and an entity name;

[0034] A storage module, used to store the configuration information and the key in a specified directory in a file format, so that a computing node used to execute the data processing task obtains the configuration information and the key from the specified directory;

[0035] An entity name sending module is used to send the entity name to the computing node, so that the computing node sends an authentication request to the authentication management node according to the authentication information, obtains the target data from the first database if the authentication is passed, and processes the target data according to the data processing task.

[0036] In a fourth aspect of the implementation of the present invention, a data processing device is provided, the device is applied to a computing node, and the device includes:

[0037] A data acquisition module is used to acquire configuration information and keys in the authentication information of the target data to be processed by the data processing task from a specified directory; wherein the configuration information and keys are acquired from the information management node and stored in the specified directory when the driver node needs to execute the data processing task;

[0038] An entity name acquisition module, used to acquire the entity name in the authentication information sent by the driver node;

[0039] An authentication request sending module, used to send an authentication request to an authentication management node according to the configuration information, key and entity name;

[0040] The task execution module is used to obtain the target data from the first database and process the target data according to the data processing task when the authentication is passed.

[0041] An embodiment of the present invention further provides an electronic device, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0042] Memory, used to store computer programs;

[0043] The processor is used to implement any of the above-mentioned data processing methods when executing the program stored in the memory.

[0044] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned data processing methods is implemented.

[0045] An embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned data processing methods.

[0046] Beneficial effects of the embodiments of the present invention:

[0047] The data processing method provided by the embodiment of the present invention, when it is necessary to execute a data processing task, the driving node sends an information acquisition request to the information management node for acquiring the authentication information of the target data that needs to be processed by the data processing task; receives the authentication information of the target data sent by the information management node; stores the configuration information and the key in a specified directory in a file format, respectively, and the computing node that executes the data processing task acquires the configuration information and the key from the specified directory; the driving node sends the entity name to the computing node, and the computing node sends an authentication request to the authentication management node according to the authentication information, acquires the target data from the first database if the authentication is passed, and processes the target data according to the data processing task.

[0048] Based on the above processing, the driver node and the computing node can automatically obtain the authentication information of the target data that needs to be processed when they need to run data processing tasks. That is, they can dynamically and automatically obtain the authentication information without the need for technical personnel to pre-distribute the authentication information of all data in each node, thereby reducing the maintenance cost of the authentication information.

[0049] Of course, it is not necessary to achieve all of the advantages described above at the same time to implement any product or method of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0051] Figure 1 A signaling interaction diagram of a data processing method provided by an embodiment of the present invention;

[0052] Figure 2 A signaling interaction diagram of another data processing method provided by an embodiment of the present invention;

[0053] Figure 3 A signaling interaction diagram of another data processing method provided by an embodiment of the present invention;

[0054] Figure 4 A signaling interaction diagram of another data processing method provided by an embodiment of the present invention;

[0055] Figure 5 A schematic diagram of a data processing principle provided by an embodiment of the present invention;

[0056] Figure 6 A structural diagram of a data processing device provided by an embodiment of the present invention;

[0057] Figure 7 A structural diagram of a data processing device provided by an embodiment of the present invention;

[0058] Figure 8 A structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention belong to the scope of protection of the present invention.

[0060] In the related art, the technical personnel send the authentication information of all the data to be processed to each working node in the cluster, which will increase the maintenance cost of the authentication information.

[0061] In order to solve the above problems, an embodiment of the present invention provides a data processing method, which can be applied to nodes in a distributed cluster. For example, the distributed cluster can be a YARN cluster. The distributed cluster can include at least one worker node (Worker), and a driver (Driver) and an executor (Executor) can be deployed in the worker node. The driver and the executor can be deployed in the same worker node, or they can be deployed in different worker nodes respectively. The driver and the executor can run in the worker node as processes respectively. Among them, the driver is used to convert the code logic submitted by the technician into a data processing task, and schedule the data processing task to the executor. The executor is used to execute the data processing task and obtain the processing result.

[0062] See also Figure 1 , Figure 1 A signaling interaction diagram of a data processing method provided in an embodiment of the present invention, the method may include the following steps:

[0063] S101: When a data processing task needs to be executed, the driver node sends an information acquisition request to the information management node for acquiring authentication information of target data required to be processed by the data processing task.

[0064] S102: The driving node receives the authentication information of the target data sent by the information management node.

[0065] The authentication information includes configuration information, key and entity name.

[0066] S103: The driver node stores the configuration information and the key in a file format to a designated directory.

[0067] S104: The driver node sends the entity name to the computing node.

[0068] S105: The computing node obtains the configuration information and the key in the authentication information of the target data to be processed by the data processing task from the designated directory.

[0069] S106: The computing node obtains the entity name in the authentication information sent by the driving node.

[0070] S107: The computing node sends an authentication request to the authentication management node according to the configuration information, the key and the entity name.

[0071] S108: When the authentication is successful, the computing node obtains the target data from the first database and processes the target data according to the data processing task.

[0072] Based on the data processing method provided by the embodiment of the present invention, the driving node and the computing node can automatically obtain the authentication information of the target data that currently needs to be processed when it needs to run a data processing task. That is, it is possible to dynamically and automatically obtain the authentication information without the need for technical personnel to pre-distribute the authentication information of all data in each node, thereby reducing the maintenance cost of the authentication information.

[0073] The authentication performed in the above process may be Kerberos authentication. In addition, the method provided by the embodiment of the present invention is also applicable to non-Kerberos data sources, that is, when Kerberos authentication is not required, the computing node can directly obtain the target data from the database to perform the data processing task.

[0074] In one embodiment, the driver node may be a working node in a distributed cluster where a driver is deployed, and the computing node may be a working node in a distributed cluster where an executor is deployed.

[0075] With respect to step S101, in a distributed cluster, a data processing task may also be referred to as a Spark task, which means processing the data in the database to obtain the corresponding processing result. For example, a user may determine to process the data in the database (i.e., the target data) as required, and then submit the code logic for processing the target data to the distributed cluster, and the distributed cluster may determine the data processing task corresponding to the code logic. The driver node may convert the code logic submitted by the user into a data processing task.

[0076] For example, the target data includes the hospitalization information of patients, and the corresponding data processing task can be to count the number of patients older than 40. For another example, the target data includes the browsing history of users browsing videos, and the corresponding data processing task can be to count the number of users who browsed video A within a specified time period.

[0077] The driver can run in any worker node of the distributed cluster, and is used to run the main() method of SparkApplication. For example, you can build the operating environment of Spark Application in the worker node, start Spark Context, and start the process corresponding to the driver.

[0078] In one implementation, a component for managing authentication information of data to be processed (which may be referred to as a metadata management system) may also be deployed in the nodes of the distributed cluster, and the node may be referred to as an information management node. The driver node may send an HTTP (Hypertext Transfer Protocol) request to the information management node for obtaining authentication information of the target data. For example, the HTTP request may carry a data identifier of the target data, and accordingly, the information management node may return the authentication information of the target data to the driver node.

[0079] In one embodiment, the authentication information includes configuration information, a key, and an entity name. In one implementation, for Kerberos authentication, the authentication information includes: configuration information of Kerberos authentication, a key of Kerberos authentication, and an entity name of Kerberos authentication.

[0080] For example, the configuration information of Kerberos authentication may include the krb5.conf configuration file. The key of Kerberos authentication may include the encryption key recorded in Keytab. The entity name of Kerberos authentication may include Principal. In Kerberos authentication, Principal is the basic entity for authentication, that is, the user name in Kerberos authentication.

[0081] With respect to step S103, in the embodiment of the present invention, the computing node may obtain the authentication information by calling a corresponding API (Application Programming Interface).

[0082] When obtaining authentication information through the API, the authentication configuration information and key can be called in a file format (for example, TXT format). Therefore, the driver node can store the authentication configuration information and key in a specified directory in a file format, and correspondingly, the computing node can obtain the authentication configuration information and key from the specified directory.

[0083] In one embodiment, the above step S103 may include the following steps: storing the configuration information and the key in a file format in a designated directory of the distributed file system respectively.

[0084] In one embodiment, the driver node can store the authentication configuration information and key in a file format in a designated directory of HDFS (Hadoop Distributed File System). Correspondingly, the computing node can obtain the authentication configuration information and key from the designated directory.

[0085] For example, a designated directory corresponding to the Spark Application may be set in the distributed file system, and the driver node may store the authentication configuration information and the key in the designated directory in file formats.

[0086] When obtaining authentication information through the API, the authenticated entity name can be called in the form of a string, so in step S104, the driver node can directly send the authenticated entity name to the computing node. There is no limitation on the execution order between step S103 and step S104.

[0087] For step S107, the authentication management node may be a node in the distributed cluster that is deployed with a component for authenticating the computing node. After the authentication information sent by the computing node passes the authentication, the authentication management node may generate an authentication ticket based on the authentication information and send it to the computing node.

[0088] For example, after receiving the authentication information sent by the computing node, the authentication management node generates an authentication ticket based on the authentication information, encrypts the authentication ticket with a preset key, and sends the encrypted authentication ticket to the computing node; the computing node uses the preset key to decrypt the encrypted authentication ticket to obtain the authentication ticket.

[0089] With respect to step S108 , the first database is a database storing target data, which may be HBase.

[0090] In one embodiment, when it is necessary to read target data from the first database, the computing node may send a data reading request carrying an authentication ticket to the first database, and then the first database may verify whether the authentication ticket sent by the computing node is legal. If the authentication ticket is legal, it indicates that the computing node has the authority to read the target data in the first database.

[0091] In one embodiment, see Figure 2 ,exist Figure 1 On the basis of, after the above step S103, the method may further include the following steps:

[0092] S109: The driver node sends a resource allocation request to the resource management node for acquiring computing resources required by the computing node to execute the data processing task, so as to allocate computing resources to the computing node.

[0093] S1010: The driver node sends a data processing task to the computing node, so that the computing node executes the data processing task according to the allocated computing resources.

[0094] In the embodiment of the present invention, the distributed cluster may also include a node (ie, a resource management node) on which a manager (Cluster Manager) is deployed. For example, for a YARN cluster, the manager is a Resource Manager.

[0095] The driver node can request the resource management node to allocate computing resources (e.g., CPU, memory), that is, the computing resources required by the computing node to perform the data processing task. Among them, the computing node that performs the data processing task can be one or more. For example, the number of computing nodes can be determined according to the computing amount of the data processing task.

[0096] In one embodiment, the configuration information includes the address of the authentication management node, correspondingly, see Figure 3 ,exist Figure 1 On the basis of, the above step S107 may include the following steps:

[0097] S1071: The computing node sends an authentication request carrying a key and an entity name to the authentication management node according to the address of the authentication management node to obtain an authentication ticket.

[0098] Furthermore, step S108 may include the following steps:

[0099] S1081: The computing node sends a data read request carrying an authentication ticket to the first database, so as to obtain target data from the first database when the first database verifies that the authentication ticket is legitimate, and processes the target data according to the data processing task.

[0100] In the embodiment of the present invention, after obtaining the authentication information, the computing node can extract the configuration information, the key and the entity name from the authentication information. Then, the computing node can parse the configuration information to obtain the address of the authentication management node, and then, the computing node sends an authentication request carrying the authentication key and the entity name to the authentication management node according to the address.

[0101] The authentication management node can verify the key and entity name sent by the computing node to determine whether the computing node has the authority to access the target data. If so, the authentication management node can return an authentication ticket to the computing node.

[0102] In one embodiment, the method may further include the following steps: after the data processing task is completed, the driver node releases the computing resources allocated to the computing node; and deletes the configuration information and keys recorded in the specified directory.

[0103] In the embodiment of the present invention, after the computing node finishes processing the data processing task, the driving node can release the resources occupied by the computing node, realize resource recovery, and avoid resource waste.

[0104] In addition, after the computing node completes the data processing task, the driver node can also delete the data recorded in the specified directory of the distributed file system (including authentication configuration information and keys) to avoid the generation of dirty data.

[0105] In one embodiment, see Figure 4 ,exist Figure 3 On the basis of, the method may further include the following steps:

[0106] S1011: The computing node sends a data write request carrying an authentication ticket to the second database, so as to write the processing result of the target data into the second database when the second database verifies that the authentication ticket is legal.

[0107] The second database may be the same as or different from the first database, and the second database may be HBase.

[0108] In the embodiment of the present invention, after the processing of the target data is completed, the computing node may also store the processing result in the second database.

[0109] When data (i.e., processing results) needs to be written in the second database, the computing node can send a data writing request carrying an authentication ticket to the second database, and then the second database can verify whether the authentication ticket sent by the computing node is legal. If the authentication ticket is legal, it indicates that the computing node has the authority to write the target data in the second database.

[0110] See also Figure 5 , Figure 5 A schematic diagram of a data processing principle provided by an embodiment of the present invention.

[0111] The driver in the driver node can obtain the Kerberos authentication configuration information (krb5.conf), key (Keytab) and entity name (Principal) corresponding to the target data to be processed by the data processing task, and store the configuration information and key in a file format to the specified directory of the distributed file system (i.e. hdfs: / / user / tmp), and send the data processing task to the computing node.

[0112] The executor in the computing node can directly obtain the entity name from the driver node, and obtain the configuration information and key from the specified directory. Then, based on the obtained authentication information, it can perform Kerber os authentication and obtain the authentication ticket. Furthermore, the computing node obtains the target data (i.e. HBaseKbsInput) from HBase1 based on the authentication ticket, processes the target data according to the data processing task (i.e. Transfor m), and writes the processing result (i.e. HBaseKbsOutput) to HBase2 based on the authentication ticket.

[0113] Based on the same inventive concept, the embodiment of the present invention further provides a data processing device, see Figure 6 , Figure 6 A structural diagram of a data processing device provided by an embodiment of the present invention, wherein the device is applied to a driver node, and the device includes:

[0114] An information acquisition request sending module 601 is used to send, when a data processing task needs to be executed, to an information management node an information acquisition request for acquiring authentication information of target data that needs to be processed by the data processing task;

[0115] An authentication information receiving module 602 is used to receive the authentication information of the target data sent by the information management node; wherein the authentication information includes configuration information, a key and an entity name;

[0116] The storage module 603 is used to store the configuration information and the key in a specified directory in a file format, so that the computing node for executing the data processing task obtains the configuration information and the key from the specified directory;

[0117] The entity name sending module 604 is used to send the entity name to the computing node, so that the computing node sends an authentication request to the authentication management node according to the authentication information, obtains the target data from the first database if the authentication is passed, and processes the target data according to the data processing task.

[0118] In one embodiment, the storage module 603 is specifically used to store the configuration information and the key in a file format in a designated directory of the distributed file system.

[0119] In one embodiment, the apparatus further comprises:

[0120] The task issuing module is used to send a resource allocation request for obtaining the computing resources required by the computing node to execute the data processing task to the resource management node, so as to allocate computing resources to the computing node;

[0121] The data processing task is issued to the computing node so that the computing node executes the data processing task according to the allocated computing resources.

[0122] In one embodiment, the apparatus further comprises:

[0123] A resource recovery module, used for releasing the computing resources allocated to the computing node after the processing of the data processing task is completed;

[0124] Delete the configuration information and key recorded in the specified directory.

[0125] Based on the same inventive concept, the embodiment of the present invention further provides a data processing device, see Figure 7 , Figure 7 A structural diagram of a data processing device provided in an embodiment of the present invention, wherein the device is applied to a computing node, and the device includes:

[0126] The data acquisition module 701 is used to acquire configuration information and keys in the authentication information of the target data to be processed by the data processing task from the specified directory; wherein the configuration information and keys are acquired from the information management node and stored in the specified directory when the driver node needs to execute the data processing task;

[0127] An entity name acquisition module 702 is used to acquire the entity name in the authentication information sent by the driver node;

[0128] An authentication request sending module 703 is used to send an authentication request to an authentication management node according to the configuration information, key and entity name;

[0129] The task execution module 704 is used to obtain the target data from the first database and process the target data according to the data processing task when the authentication is passed.

[0130] In one embodiment, the configuration information includes an address of the authentication management node;

[0131] The authentication request sending module 703 is specifically configured to send an authentication request carrying the key and the entity name to the authentication management node according to the address of the authentication management node to obtain an authentication ticket;

[0132] The task execution module 704 is specifically configured to send a data reading request carrying the authentication ticket to the first database, so as to obtain the target data from the first database when the first database verifies that the authentication ticket is legal.

[0133] The embodiment of the present invention further provides an electronic device, such as Figure 8As shown, it includes a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804.

[0134] Memory 803, used for storing computer programs;

[0135] The processor 801 is used to implement the steps of any of the above data processing methods when executing the program stored in the memory 803.

[0136] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0137] The communication interface is used for communication between the above electronic device and other devices.

[0138] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0139] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0140] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above data processing methods are implemented.

[0141] In another embodiment of the present invention, a computer program product including instructions is provided. When the computer program product is run on a computer, the computer executes any one of the data processing methods in the above embodiments.

[0142] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0143] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0144] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0145] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: The method is applied to a driver node, and the method comprises: When a data processing task needs to be executed, an information acquisition request is sent to the information management node to obtain authentication information of the target data to be processed by the data processing task; Receiving authentication information of the target data sent by the information management node; wherein the authentication information includes configuration information, a key and an entity name, and the configuration information includes an address of the authentication management node; The configuration information and the key are stored in a specified directory in a file format, respectively, so that a computing node for executing the data processing task obtains the configuration information and the key from the specified directory; The entity name is sent to the computing node so that the computing node sends an authentication request carrying the key and the entity name to the authentication management node according to the address of the authentication management node to obtain an authentication ticket, and sends a data reading request carrying the authentication ticket to the first database, so that when the first database verifies that the authentication ticket is legal, the computing node obtains the target data from the first database and processes the target data according to the data processing task.

2. The method according to claim 1, characterized in that The step of storing the configuration information and the key in a file format to a specified directory includes: The configuration information and key are stored in a specified directory of the distributed file system in file format.

3. The method according to claim 1, characterized in that: After storing the configuration information and the key in a specified directory in file format respectively, the method further includes: Sending to a resource management node a resource allocation request for acquiring computing resources required by the computing node to execute the data processing task, so as to allocate computing resources to the computing node; The data processing task is issued to the computing node so that the computing node executes the data processing task according to the allocated computing resources.

4. The method according to claim 3, characterized in that The method further comprises: After the data processing task is completed, releasing the computing resources allocated to the computing node; Delete the configuration information and key recorded in the specified directory.

5. A data processing method, characterized in that: The method is applied to a computing node, and the method comprises: Obtain configuration information and keys in the authentication information of the target data to be processed by the data processing task from the designated directory; wherein the configuration information and keys are obtained from the information management node and stored in the designated directory when the driver node needs to execute the data processing task; Obtaining the entity name in the authentication information sent by the driver node; Sending an authentication request to the authentication management node according to the configuration information, key and entity name; If the authentication is successful, the target data is obtained from the first database, and the target data is processed according to the data processing task; The configuration information includes the address of the authentication management node; The sending an authentication request to the authentication management node according to the configuration information, the key and the entity name includes: According to the address of the authentication management node, sending an authentication request carrying the key and the entity name to the authentication management node to obtain an authentication ticket; The step of obtaining the target data from the first database when the authentication is successful includes: A data read request carrying the authentication ticket is sent to the first database, so as to obtain the target data from the first database when the first database verifies that the authentication ticket is legal.

6. The method according to claim 5, characterized in that The method further comprises: A data write request carrying the authentication ticket is sent to the second database, so that when the second database verifies that the authentication ticket is legitimate, a processing result of processing the target data is written into the second database.

7. A data processing device, characterized in that: The device is applied to a driving node, and comprises: An information acquisition request sending module is used to send, when a data processing task needs to be executed, to the information management node an information acquisition request for acquiring authentication information of target data that needs to be processed by the data processing task; An authentication information receiving module, used to receive the authentication information of the target data sent by the information management node; wherein the authentication information includes configuration information, a key and an entity name, and the configuration information includes the address of the authentication management node; A storage module, used to store the configuration information and the key in a specified directory in a file format, so that a computing node used to execute the data processing task obtains the configuration information and the key from the specified directory; An entity name sending module is used to send the entity name to the computing node, so that the computing node sends an authentication request carrying the key and the entity name to the authentication management node according to the address of the authentication management node to obtain an authentication ticket, and sends a data reading request carrying the authentication ticket to the first database, so that when the first database verifies that the authentication ticket is legal, the computing node obtains the target data from the first database and processes the target data according to the data processing task.

8. A data processing device, characterized in that: The device is applied to a computing node, and comprises: A data acquisition module is used to acquire configuration information and keys in the authentication information of the target data to be processed by the data processing task from a specified directory; wherein the configuration information and keys are acquired from the information management node and stored in the specified directory when the driver node needs to execute the data processing task; An entity name acquisition module, used to acquire the entity name in the authentication information sent by the driver node; An authentication request sending module, used to send an authentication request to an authentication management node according to the configuration information, key and entity name; A task execution module, used for obtaining the target data from the first database and processing the target data according to the data processing task if the authentication is passed; The configuration information includes the address of the authentication management node; The authentication request sending module is specifically used to: send an authentication request carrying the key and the entity name to the authentication management node according to the address of the authentication management node to obtain an authentication ticket; The task execution module is specifically used to send a data reading request carrying the authentication ticket to the first database, so as to obtain the target data from the first database when the first database verifies that the authentication ticket is legal.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1-4, or 5 or 6 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 4, or 5 or 6 are implemented.