Task access system, method and device for big data platform and electronic equipment

Abstract

The invention provides a task access system, method and device for a big data platform and electronic equipment, and the system comprises a client, a decision service module and a task registration service module, and is characterized in that related information of a target big data task transmitted by a user is received through the client; the decision service module receives and analyzes the related information of the target big data task, obtains the related information of the data task pre-stored in the registration service module from the task registration service module, and verifies the sub-information of the target related information. And a task decision result is transmitted back to the client in real time. If the task decision result is that the check is passed, the target big data task is allowed to be executed in the target big data cluster, and if the task decision result is not passed, the target big data task is rejected to be executed in the target big data cluster, so that sufficient processing resources of the big data task admission check can be ensured, and the real-time performance of the big data task admission check can be ensured.

Description

Technical Field

[0001] The present application relates to the field of big data, and in particular to a task access system, method, device and electronic equipment for a big data platform. Background Art

[0002] In the field of big data, a big data platform refers to an infrastructure built to store, manage and analyze massive amounts of data. Common big data platforms include: data platforms based on different big data computing engines such as Hadoop, Spark, Storm, Flink, Flume / Kafka, etc. Major data platforms provide real-time data support and can handle massive amounts of data storage and computing tasks. They have a wide range of application scenarios, such as finance, retail, healthcare, etc. The deployment methods of major data platforms are very flexible and can be deployed on both private and public clouds.

[0003] In the financial field, big data platforms can help deploy enterprises or departments to conduct risk assessment, draw customer portraits, and conduct precision marketing, etc. However, as the amount of data on big data platforms continues to grow, access verification for big data tasks on major data platforms has become the key to ensuring data security on big data platforms. Existing big data task access verification methods usually use big data clusters to perform access verification on big data tasks. Since big data clusters carry a large number of data tasks and computing resources are relatively tight, if access verification is performed on the entire process of big data tasks, the real-time performance of access verification is likely to be poor. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a task access system, method, device and electronic device for a big data platform to ensure the real-time access verification of big data tasks.

[0005] In a first aspect, an embodiment of the present application provides a task access system for a big data platform, wherein the task access system includes: a client, a decision service module, and a task registration service module, wherein:

[0006] The client is used to receive target-related information of a target big data task input by a user, and transmit the target-related information of the target big data task to the decision service module, wherein the relevant information of the big data task at least includes: production IP information of the big data task, cluster information to which the big data task belongs, and registration information of the big data task;

[0007] The decision service module is used to receive and parse the target related information of the target big data task, and obtain the task information of the data task pre-stored in the task registration service module from the task registration service module, verify the target related information of the target big data task, and return the task decision result to the client;

[0008] The client is also used to allow the target big data task to be executed in the target big data cluster if the task decision result is verification passed, and to refuse to execute the target big data task in the target big data cluster if the task decision result is verification failed.

[0009] Optionally, the relevant information of the big data task also includes: the type of execution command of the big data task and the authority level of the user; the decision service module includes: a first verification module;

[0010] The first verification module is used to verify the command type, user authority level and production IP information of the target big data task.

[0011] Optionally, the relevant information of the big data task also includes: the task type of the big data task, the task name of the big data task, and the task registration information of the big data task; the decision service module also includes: a second verification module;

[0012] The second verification module is used to verify the task type of the big data task, the task name of the big data task, and the task registration information of the big data task.

[0013] Optionally, the relevant information of the big data task also includes: production and research isolation parameters, and the first verification module is further used to:

[0014] If the production and R&D represented by the production-research isolation parameter have not been isolated, a task decision result indicating that the verification failed is output.

[0015] Optionally, the task registration service module includes: a first task registration service module and a second task registration service module, the first task registration service module is used to store IP information of the data task and cluster information of the data task, and the second task registration service module is used to store task registration information of the data task;

[0016] The first verification module is specifically used to receive target related information of the target data task, and obtain production IP information or cluster information from the first task registration service module, verify the command type, user authority level and production IP information of the target big data task to obtain a first verification result, and pass the first verification result and related information of the target big data task to the second verification module;

[0017] The second verification module is used to receive the first verification result and the relevant information of the target big data task, obtain the task registration information from the second task registration service module, verify the task type of the big data task, the task name of the big data task, and the task registration information of the big data task, obtain the task decision result, and return the task decision result to the client.

[0018] Optionally, the client is further used for:

[0019] If the task decision result is verification failure, record the task decision result and output a prompt message

[0020] In a second aspect, an embodiment of the present application provides a task access method for a big data platform, wherein the method is applied to the task access system for a big data platform described in the first aspect, and the method includes:

[0021] Receive target related information of the target big data task input by the user, wherein the related information of the big data task at least includes: production IP information of the big data task, cluster information to which the big data task belongs, and registration information of the big data task;

[0022] Parsing target related information of the target big data task, and verifying target related information of the target big data task based on task information of pre-stored data tasks;

[0023] If the task decision result is that the verification is passed, the target big data task is allowed to be executed in the target big data cluster. If the task decision result is that the verification is not passed, the target big data task is refused to be executed in the target big data cluster.

[0024] In a third aspect, an embodiment of the present application provides a task access device for a big data platform, wherein the device includes:

[0025] The device comprises:

[0026] A receiving module is used to receive target related information of a target big data task input by a user, wherein the related information of the big data task includes at least: production IP information of the big data task, cluster information to which the big data task belongs, and registration information of the big data task;

[0027] A verification module, used to parse the target related information of the target big data task, and verify the target related information of the target big data task based on the task information of the pre-stored data task;

[0028] The execution module is used to allow the target big data task to be executed in the target big data cluster if the task decision result is that the verification is passed, and to refuse the target big data task to be executed in the target big data cluster if the task decision result is that the verification is not passed.

[0029] In a fourth aspect, an embodiment of the present application provides an electronic device, wherein the electronic device includes:

[0030] Processor; and

[0031] Memory for storing programs,

[0032] Among them, the program includes instructions, and when the instructions are executed by the processor, the processor executes the task access method for the big data platform described in the second aspect.

[0033] In a fifth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, characterized in that the computer instructions are used to enable a computer to execute the task access method for a big data platform described in the second aspect.

[0034] Beneficial effects of this application:

[0035] The present application provides a task access system, method, device and electronic device for a big data platform, wherein the system includes a client, a decision service module, and a task registration service module. The client receives relevant information of a target big data task input by a user and inputs it into the decision service module. The decision service module receives and parses the relevant information of the target big data task, and then obtains relevant information of the data task pre-stored in the registration service module from the task registration service module. The decision service module verifies various sub-information of the target related information and returns the task decision result to the client in real time. If the task decision result is a verification pass, the target big data task is allowed to be executed in the target big data cluster. If it fails, the target big data task is refused to be executed in the target big data cluster.

[0036] By adopting the task access system for the big data platform provided in the embodiment of the present application, the traditional method of performing access verification on big data tasks in the big data cluster is abandoned. Before the big data task enters the big data cluster, the big data task is verified with the help of the task access system provided by the present application. If the verification passes, the big data task is allowed to be executed in the big data cluster. If the verification fails, the big data task is refused to be executed in the big data cluster. In this way, the task access system can perform special processing on the access verification of big data tasks, ensure sufficient processing resources for the access verification of big data tasks, and further help to ensure the real-time performance of the access verification of big data tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0038] Figure 1 A schematic diagram of a logical architecture of a task access system for a big data platform provided in an embodiment of the present application is shown;

[0039] Figure 2 A schematic diagram of a verification process performed by a decision service module provided in an embodiment of the present application is shown;

[0040] Figure 3 A schematic diagram showing a verification process performed by another decision service module provided in an embodiment of the present application

[0041] Figure 4 A schematic diagram of a process flow of a task access method for a big data platform provided in an embodiment of the present application is shown;

[0042] Figure 5 A schematic diagram of a logical structure of a task access device for a big data platform provided in an embodiment of the present application is shown;

[0043] Figure 6 A schematic diagram of a logical structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0045] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0046] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0047] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0048] As described in the background technology, the big data platform is responsible for storing and processing massive amounts of data. The data in the big data platform is an indispensable cornerstone for various business activities of the enterprise's production and operation. How to ensure the security and privacy of the data in the big data platform data has become the key to ensuring the stability of the enterprise's production and operation. The relevant technology proposes to perform access verification on each data task of the data in the big data platform, where each data task refers to a data processing request initiated by the task demander (or the task applicant mentioned below), such as a data read request corresponding to a data read task, and a data write request corresponding to a data write task. The specific type of data task can be limited according to the actual business, and the specific type of data task is not strictly limited in this article.

[0049] Among them, in the embodiment of the present application, the verification content of the access verification for the data task may include the following parts:

[0050] 1. Data permission verification: For each data task, verify the identity information of the task applicant, the permission level of the task applicant and other permission information to determine whether the task applicant's permissions are consistent with the permissions assigned to the task applicant by the big data platform.

[0051] 2. Data format check: For each data task, verify whether the format of the data task entered by the task applicant meets the format requirements of the data platform system. Specific checks: data validity, check whether the data type meets the data type specified by the data platform system, etc.

[0052] 3. Data integrity check: For each data task, verify the integrity of the data entered by the task applicant.

[0053] 4. Data task target verification: For each data task, it is necessary to ensure that the execution target object of the data task is consistent with the execution target object of the data platform system. This is achieved by verifying the description content of the data task, the execution target data object of the data task, and the expected execution result of the data task.

[0054] 5. Data task resource evaluation: For each data task, it is necessary to evaluate whether the computing resources required for each data task are available and ensure that the allocated resources and the resources used are reasonable, including evaluating computing resources, storage resources, and network bandwidth.

[0055] 6. Data task security verification: For each data task, it is necessary to ensure that the operation and data processed by the data task are safe and comply with the corresponding security policies and regulations. This is achieved by conducting security risk assessments on task applicants and related data.

[0056] 7. Data task priority assessment: For each data task, it is necessary to decide whether to allow the data task to access the big data platform system based on its priority. The specific assessment can be based on the usage of system resources, business data requirements, and the urgency of the data task.

[0057] The existing data task access verification method of the big data platform needs to use the resources of the big data cluster to conduct a comprehensive analysis and evaluation of the data tasks submitted by the task applicant, that is, to perform the above 7 parts of task verification one by one for each data task. Since the big data cluster resources carry a large number of data task calculations, when the computing resources are relatively tight, sufficient resources need to be allocated to perform the above 7 parts of task verification one by one. This takes a long time and easily leads to delays in the execution of data tasks, further delaying the start time of data tasks, resulting in poor real-time response of data tasks on the entire big data platform.

[0058] In view of this, the present application provides a task access system, method, device and electronic device for a big data platform, wherein, in the first aspect, the present application provides a task access system for a big data platform, and in the second aspect, the present application provides a task access method for a big data platform. As an implementation mode, the task access method for a big data platform can be applied to any electronic device that deploys the task access system for a big data platform provided by the present application. The electronic device is any device with a task access function for a big data platform, including but not limited to a personal mobile terminal, a computer or a server.

[0059] In some embodiments, Figure 1 As shown, the system includes the following parts:

[0060] Client 11, decision service module 12, task registration service module 13.

[0061] The client 11 is used to receive target-related information of a target big data task input by a user, and transmit the target-related information of the target big data task to the decision service module 12. Specifically, the client 11 is essentially a software program for connecting to a big data cluster and submitting big data tasks.

[0062] Among them, in the embodiments of the present application, data tasks refer to: a series of data processing and analysis activities, which are specifically scheduled and executed according to predetermined rules and processes. Depending on the actual application scenario, the corresponding data task types are different. For example, in the big data platform in the financial field, data tasks usually involve responsible data analysis and processing activities, which are used to extract valuable information from huge data sets. For example, if the data task is customer management, at this time, the customer group to which the target customer belongs can be analyzed from the social media, transaction records, customer interactions and other data stored in the big data platform through pre-designed customer information processing rules. In more detail, the potential overdue probability of the target customer can be analyzed.

[0063] As an implementation method, the client 11 is deployed on an execution machine, which is a node in the big data platform that is actually responsible for data processing and computing tasks, and generally refers to a server or virtual service instance that undertakes data storage, processing and computing tasks. The client 11 is the core of interaction with the user. The system receives a task request initiated by the user through the client 11, wherein the task request carries relevant information of the task (such as the production IP information of the big data task, the cluster information to which the big data task belongs, the registration information of the big data task, etc.), and then passes the relevant information carried by the task request to the decision service module 12 and the task registration service module 13, and performs access verification on the data task based on the relevant information carried by the task request.

[0064] The relevant information of the big data task is the general term for the content information of the big data task, including at least: the production IP information of the big data task, the cluster information to which the big data task belongs, and the registration information of the big data task.

[0065] IP information refers to the IP address of the server where the client is located. Based on this, production IP information refers to the IP address of the server where the client responsible for submitting the big data task is located. The cluster information to which the big data task belongs includes the unique ID information of the big data cluster that executes the big data task. The registration information of the big data task includes the execution command submitted by the client, the identity information of the client, etc. The relevant information of the big data task is a concrete description of what kind of big data execution command is executed in what location, with what identity, and in what cluster.

[0066] The decision service module 12 is used to receive and parse the target-related information of the target big data task, and obtain the task information of the data task pre-stored in the task registration service module from the task registration service module, and verify the target-related information of the target big data task. If all sub-information in the target-related information of the target data task passes the verification, the task decision result is returned to the client. The specific steps performed by the decision service module 12 include:

[0067] S1-1. Receive target-related information of the target big data task transmitted by the client.

[0068] S1-2, parsing the received target related information of the big data task, and obtaining various sub-information in the target related information, such as obtaining the production IP information of the target big data task, the cluster information to which the big data task belongs, and the registration information of the big data task.

[0069] S1-3, obtaining the required task information from the task registration service module 13. The required task information is data pre-stored in the task registration service module 13 for data task verification, and there is a mapping relationship between the task information and the relevant information of the big data task. Specifically, the corresponding mapping relationship can be determined based on different data task types and task names.

[0070] For example, if the data task is to draw a user portrait, the task information of the data task describes the device IP information of the specific execution device for drawing the user portrait, and the device IP information corresponds to the production IP information in the relevant information of the big data task. The task information of the data task describes that the cluster information to which the user portrait is drawn is cluster A, and cluster A corresponds to the cluster information to which the big data task belongs in the relevant information of the big data task. In other words, the relevant information of different types of big data tasks corresponds to different task information of the data tasks.

[0071] Specifically, in the embodiment of the present application, the decision service module 12 is essentially a software program that encapsulates the business logic corresponding to the actual business needs according to the actual business needs. The decision service module 12 stores the task information of each pre-registered data task. Among them, the task information includes: task name, task type, task command, absolute path of the task file, MD5 (Message-Digest Algorithm 5) value of the task file, cluster CPU resource parameter value, cluster memory resource parameter value, etc.

[0072] The MD5 value of a task file refers to a fixed-length hash value obtained after the file or data of a data task is encrypted by the MD5 information digest algorithm. For example, data of any length can be converted into a fixed-length hash value through a hash algorithm, and the hash value of each task file is the MD5 value of the corresponding task file.

[0073] The cluster CPU resource parameter value refers to the CPU usage of the cluster by the data task, which is specifically reflected by the two parameter values ​​of CPURequest and CPU Limit. CPU Request reflects the minimum CPU computing resources required by the container corresponding to the data task, and CPU Limit reflects the maximum CPU computing resources that the container corresponding to the data task will occupy.

[0074] In some embodiments, the relevant information of the big data task also includes: the type of execution command of the big data task and the authority level of the user; the decision service module includes: a first verification module.

[0075] The first verification module is used to verify the command type, user authority level and production IP information of the target big data task.

[0076] As an implementation mode, the first verification module is used to output a task decision result of failed verification when any one of the command type, user authority level and production IP information of the target big data task fails to pass the verification.

[0077] The types of execution commands for big data tasks can be specifically divided into: system commands, execution commands for various big data computing engines, among which system commands refer to software programs or software instructions used to query client instructions, and execution commands for various big data computing engines include commands for Hive computing engines, structured statements for Presto cluster engines, execution commands for Hadoop cluster engines, execution commands for Spark computing engines, etc. Among them, system commands are used to view how the execution commands of various computing engines are executed, which will not consume system resources or cause security issues. Therefore, the verification of the command type for the target big data task can be specifically:

[0078] Determine whether the execution command of the big data task sent by the client is a system command. If so, determine that the command type of the target big data task passes the verification. If not, determine that the command type of the target big data task fails the verification.

[0079] In addition, the client can also be verified to determine the security of the client. Specifically, the first verification module verifies the version number of the client and the production cluster to which the client belongs, determines whether the version number of the client meets the preset version regulations, and determines whether the production cluster to which the client belongs belongs to the data cluster in the backend big data cluster. If the version number meets the preset version number regulations and the client belongs to the data cluster in the backend big data cluster, it is determined that the client has passed the verification.

[0080] In the embodiment of the present application, the user's permission level is determined according to the permission group to which the user belongs. Specifically, it can be determined according to the user group to which the user belongs. That is to say, different user groups have different access rights to the big data cluster. For example, the access rights to the big data cluster corresponding to users in a high-priority user group can be higher than the access rights to the big data cluster corresponding to users in a low-priority user group. Based on this, the verification of the user's permission level for the target big data task can be:

[0081] Determine whether the user group of the user who passed the target big data task belongs to the compliant user group. If so, the user's permission level check is passed; if not, the user's permission level check is not passed.

[0082] Based on the above description, the production IP information refers to the IP address of the server where the client that submits the big data production task is located. The verification of the production IP information of the target big data task can be specifically:

[0083] Determine whether the production IP information is compliant IP information. Compliant IP information refers to IP information that complies with preset IP regulations.

[0084] If so, it is determined that the production IP information has passed the verification, and if not, it is determined that the production IP information has not passed the verification.

[0085] In the embodiment of the present application, the relevant information of the big data task also includes: production and research isolation parameters, which are used to characterize whether the big data platform is a system architecture that isolates production and research and development. Specifically, production refers to normal data processing, handling and analysis based on the actual needs of users, and research and development refers to experimental data processing, handling and analysis based on the research and development needs of researchers. Production and research isolation can effectively avoid the impact of data between production and research and development processes on each other's data, avoid data confusion, and facilitate data management. Based on this, the first verification module is also used to:

[0086] If the production and R&D represented by the production-research isolation parameter have not been isolated, a task decision result indicating that the verification failed is output.

[0087] Specifically, assuming that the production and research isolation parameter is 1, it indicates that production and research and development are isolated from each other, and the production and research isolation parameter is 0, it indicates that production and research and development have not yet been isolated from each other. At this time, if the production and research isolation parameter is 0, it is considered that the production and research isolation parameter verification fails, and the first verification module outputs the task decision result that the verification fails.

[0088] Based on the above specific verification content, as a possible application scenario, the specific verification process of the first verification module can be as follows: Figure 2 As shown, the target big data task submitted by the client is received, and then the execution command of the target big data task is determined to determine whether the execution command of the target big data task is a system command. If so, the execution command verification result is output to the second verification module. If not, continue to determine whether the cluster requested to be used by the target big data task is a production cluster. If so, perform the first conditional verification, wherein the first conditional verification means: the target big data task submitted by the client must be a target big data task submitted by a user belonging to the production user group, and the computing engine that executes the target big data task is a computing engine in the production cluster, and the IP address of the server to which the client belongs belongs to the production IP. If not, perform the second conditional verification, wherein the second conditional verification means: the target big data task submitted by the client is a target big data task submitted by a user belonging to the development user group, and the IP address of the server to which the client belongs belongs to the development IP address.

[0089] If the verification result of the first condition verification is passed, the first condition verification result is output to the second verification module. If not, it is further determined whether the big data platform has enabled production / R&D isolation. If not, the production / R&D isolation verification result is determined as: production / R&D isolation has not yet been enabled, and the production / R&D isolation verification result is output to the second verification module. If production / R&D isolation has been enabled, the user's authority level is further determined, specifically whether the user is a member of the whitelist. The whitelist refers to each user group that meets the preset authority level. If yes, the user authority level verification result is output to the second verification module. If not, the task decision result of the target big data task is considered to be failed.

[0090] In some embodiments, the relevant information of the big data task also includes: the task type of the big data task, the task name of the big data task, and the task registration information of the big data task; the decision service module also includes: a second verification module;

[0091] The second verification module is used to verify the task type of the big data task, the task name of the big data task, and the task registration information of the big data task.

[0092] In the embodiment of the present application, the task type of a big data task depends on the type of the computing engine that executes the big data task. For example, if the computing engine that executes the big data task is a Spark computing engine, then the task type of the big data task is a Spark type. Task types may include: Hive type, Hadoop type, Presto type, Tez type, etc.

[0093] The task name of a big data task refers to the name or code used to represent the execution command, which helps users identify what the task is for. For example, HR_zhangsan_count_interviewees_20000101 indicates the interview data generated by HR Zhangsan's account on January 1, 2000.

[0094] The task registration information of the big data task includes the client IP address, Rcc user group, cluster ID information, big data task command, task name of the big data task, etc. Among them, the registration information corresponding to the task name can be determined by parsing the task name set in the target big data task and querying the task registration data table in the task registration service module.

[0095] In some embodiments, the task registration service module includes: a first task registration service module and a second task registration service module, the first task registration service module is used to store IP information of the data task and cluster information of the data task, and the second task registration service module is used to store task registration information of the data task;

[0096] The first verification module is specifically used to receive target related information of the target data task, and obtain production IP information or cluster information from the first task registration service module, verify the command type, user authority level and production IP information of the target big data task to obtain a first verification result, and pass the first verification result and related information of the target big data task to the second verification module;

[0097] The second verification module is used to receive the first verification result and the relevant information of the target big data task, obtain the task registration information from the second task registration service module, verify the task type of the big data task, the task name of the big data task, and the task registration information of the big data task, obtain the task decision result, and return the task decision result to the client.

[0098] The verification of the task type for big data tasks specifically includes:

[0099] Determine whether the target big data task is a production task, wherein a production task means that the target cluster is a big data task corresponding to the production cluster, and the type of the production task here depends on the target cluster that executes the big data task.

[0100] If the target big data task is not a production task, it is determined that the verification of the task type for the big data task has passed. The reason is that since the production task is submitted to the production cluster, the production cluster needs to ensure the timeliness of the production task output, the stability and security of the cluster resources, etc. When the task is submitted to the cluster, it may have a negative impact on timeliness, stability, security, etc., so it needs to be verified to reduce the negative impact of the production task on timeliness, stability, and security.

[0101] Furthermore, it can be determined again whether the type of the execution command of the big data task is a system command. If so, the verification of the type of the execution command is passed. Specifically, the execution command can be string-segmented according to the preset special characters to obtain each field in the execution command of the big data task, and then the string array corresponding to each field is verified to determine whether the execution command contains each element in the segmented string array. If so, the verification is passed, and if not, the verification fails.

[0102] The verification of the task name of the big data task specifically includes: according to the preset task name specification, verifying the task name of the target big data task to determine whether the task name of the target big data task meets the preset task name specification. If so, the task name verification passes; if not, the task name verification fails, and the task decision result can be output as verification failure.

[0103] The verification of task registration information for big data tasks specifically includes:

[0104] In the process of sending relevant information of the big data task through the client, the MD5 value calculation can be performed for the big data task. Specifically, the absolute path of the task file of the big data task on the execution machine can be determined, and then when the client submits the big data task, the MD5 value of the task file under the absolute path is calculated. In the subsequent task registration information verification process, the MD5 value of the task file at the time of task registration can be obtained from the task registration table in the task registration service module, and the two are compared. If they are equal, the task registration information verification passes, and if they are not equal, the task registration information verification fails.

[0105] In addition, the relevant information of the big data task also includes the cluster resource parameter value. As an optional implementation, the second verification module can also determine the cluster resource parameter value set in the execution command based on the relevant information of the big data task, and generate an execution parameter list based on the cluster resource parameter value. When verifying the task registration information, the registration parameter list in the registration information pre-stored in the task registration service module can be obtained, and the execution parameter list can be compared with the registration parameter list in the task registration service module. If the execution parameter value is less than or equal to the registration parameter value, the verification is passed, and if the execution parameter value is greater than the registration parameter value, the verification fails. In this way, when registering a big data task, the impact of the resource size required by the big data task on the big data cluster is manually evaluated. The smaller the registration parameter value, the smaller the resources consumed, and the smaller the resource burden on the cluster.

[0106] Based on the above specific verification content, as a possible application scenario, the specific verification process of the second verification module can be as follows: Figure 3 As shown, the initial verification result transmitted by the first verification module and the target related information of the target big data task are received, and then the following verification is performed on the target big data task:

[0107] Determine whether it is a production task. If not, determine that the task decision result is a production task verification. If yes, then continue to determine whether the execution command of the target big data task is a system command. If yes, then determine that the execution command type verification has passed. If not, further verify the task name of the target big data task to determine whether the task name of the target big data task meets the task name specification. If not, determine that the task name verification has failed. If yes, then further verify the task registration information.

[0108] During the process, the job name + task type can be used as the query condition to obtain the task registration information corresponding to the target big data task from the task registration service module. In addition, during the process, the number of task registration information can be verified according to the number of task registration information queried. If the number of task registration information is 0, it is determined that the task decision result of the target big data task is verification failure. If the number of task registration information is greater than 2, log monitoring reminders are required to remind manual processing, and manual task registration information verification is performed. The benefit of performing log monitoring reminders to remind manual processing when the number of task registration information is greater than 2 is that since the task registration is uniquely registered by task name + task type, only one registration situation will be queried under normal circumstances. If there are two registration situations, it can be considered that there is a vulnerability in the verification module, and the verification module needs to be functionally repaired, which is a processing method of abnormal downgrade. The reason for performing abnormal downgrade processing is that since most big data tasks will not actually have a particularly negative impact on the cluster, and the cluster has a certain fault tolerance, in order to ensure the user experience, downgrade processing is performed at this time, and manual intervention is performed to troubleshoot specific faults.

[0109] If the number of task registration information is 1, the target big data task is further carefully verified for the task registration information, including the verification of the following three conditions:

[0110] Condition 1: Determine whether the execution command contains a registration command;

[0111] Condition 2: Determine whether the MD5 value of the execution file is equal to the MD5 value of the registration file;

[0112] Condition 3: Determine whether the execution parameter value is less than or equal to the registration parameter value.

[0113] If any of the above conditions 1 to 3 fails to pass the verification, it is necessary to further verify the user's permissions to determine whether the user is a member of the whitelist. If so, output the task decision result that passed the verification. If not, further determine whether to enable registration verification, where registration verification refers to verifying whether the target big data task is a legal big data task. Here, the legality of the target big data task is verified by enabling registration verification. If the target big data task is an illegal big data task, it needs to be intercepted. If the registration verification has been enabled, output the task decision result that failed the verification. If the registration verification has not been enabled, further determine whether the user is a member of the preset blacklist. If not, output the task decision result that passed the verification. If yes, output the task decision result that failed the verification.

[0114] By selecting the embodiment of the present application, it is possible to determine whether the big data tasks are of the same type or solutions to the same goal in a big data environment, wherein each big data task is a computing logic initiated by a user according to actual needs and needs to be executed using big data. For example, Zhang San from the Human Resources Department needs to analyze the relevant information of the interviewees for each month of this year. At this time, the variables in the big data task may include the month, and the interviewee-related information analysis tasks corresponding to each month belong to the same type of big data tasks, and then determine whether the big data task is allowed to be submitted to the big data cluster for execution. For tasks of the same type or the same goal, the characters of specific variables can be set in the task command, and then the decision service module divides the task command to determine whether the execution task of the target big data task is consistent with the registration task at the time of registration, so that the verification of the big data task can be quickly realized without occupying the processing resources of the big data cluster. If the verification passes, the big data task is allowed to be connected to the big data cluster for execution, which effectively guarantees the efficiency of the big data task verification.

[0115] As a possible implementation, if the task decision result is verification failure, the task decision result is recorded and a prompt message is output. This helps subsequent staff to trace back the task verification failure based on the recorded results and determine the reason for the task verification failure, and helps users to change the big data tasks they submitted based on the output prompt message.

[0116] In a second aspect, the present application provides a task access method for a big data platform, wherein the method is applied to the task access system for a big data platform provided in the first aspect, such as Figure 4 As shown, the method comprises the following steps:

[0117] S41. Receive target-related information of the target big data task input by the user.

[0118] The relevant information of the big data task includes at least: the production IP information of the big data task, the cluster information to which the big data task belongs, and the registration information of the big data task;

[0119] S42, parsing target related information of the target big data task, and verifying target related information of the target big data task based on task information of pre-stored data tasks;

[0120] S43. If the task decision result is that the verification is passed, the target big data task is allowed to be executed in the target big data cluster. If the task decision result is that the verification is not passed, the target big data task is refused to be executed in the target big data cluster.

[0121] Among them, step S41 is implemented by the client 11 mentioned in the first aspect, and step S42 is implemented by the decision service module 12 mentioned in the first aspect in cooperation with the task registration service module 13, wherein the task registration service module 13 pre-stores task information of the data task, and the task information is the registration information registered when the big data task is registered, and step S43 is implemented by the client 11 mentioned in the first aspect. For specific descriptions, please refer to the relevant description of the first aspect, which will not be repeated here.

[0122] In a third aspect, the present application provides a task access device for a big data platform, which is also applied to the task access system for a big data platform provided in the first aspect, wherein, Figure 5 As shown, the device 50 comprises:

[0123] The receiving module 501 is used to receive target related information of a target big data task input by a user, wherein the related information of the big data task at least includes: production IP information of the big data task, cluster information to which the big data task belongs, and registration information of the big data task;

[0124] Verification module 502, used to parse the target related information of the target big data task, and verify the target related information of the target big data task based on the task information of the pre-stored data task;

[0125] The execution module 503 is used to allow the target big data task to be executed in the target big data cluster if the task decision result is that the verification is passed, and to refuse the target big data task to be executed in the target big data cluster if the task decision result is that the verification is not passed.

[0126] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0127] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0128] In a fourth aspect, the exemplary embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory in communication with the at least one processor. The memory stores a computer program that can be executed by the at least one processor, and when the computer program is executed by the at least one processor, the electronic device performs a method according to an embodiment of the present application.

[0129] In a fifth aspect, an exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0130] In a sixth aspect, an exemplary embodiment of the present application further provides a computer program product, comprising a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0131] refer to Figure 6 , the structural block diagram of the electronic device 600 that can be used as the server or client of the present application will now be described, which is an example of a hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0132] like Figure 6 As shown, the electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the electronic device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0133] A plurality of components in the electronic device 600 are connected to the I / O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the electronic device 600, and the input unit 606 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 607 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 608 may include, but is not limited to, a disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0134] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the aforementioned task access method for a big data platform may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via ROM 602 and / or a communication unit 609. In some embodiments, the computing unit 601 may be configured to perform the aforementioned task access method for a big data platform in any other appropriate manner (e.g., by means of firmware).

[0135] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0136] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0139] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0140] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A task access system for a big data platform, characterized in that: The task access system includes: a client, a decision service module and a task registration service module, wherein: The client is used to receive target-related information of a target big data task input by a user, and transmit the target-related information of the target big data task to the decision service module, wherein the relevant information of the big data task at least includes: production IP information of the big data task, cluster information to which the big data task belongs, and registration information of the big data task; The decision service module is used to receive and parse the target related information of the target big data task, and obtain the task information of the data task pre-stored in the task registration service module from the task registration service module, verify the target related information of the target big data task, and return the task decision result to the client; The client is also used to allow the target big data task to be executed in the target big data cluster if the task decision result is verification passed, and to refuse to execute the target big data task in the target big data cluster if the task decision result is verification failed.

2. The system according to claim 1, characterized in that The relevant information of the big data task also includes: the type of execution command of the big data task and the authority level of the user; the decision service module includes: a first verification module; The first verification module is used to verify the command type, user authority level and production IP information of the target big data task.

3. The system according to claim 2, characterized in that The relevant information of the big data task also includes: the task type of the big data task, the task name of the big data task, and the task registration information of the big data task; the decision service module also includes: a second verification module; The second verification module is used to verify the task type of the big data task, the task name of the big data task, and the task registration information of the big data task.

4. The system according to claim 2, characterized in that The relevant information of the big data task also includes: production and research isolation parameters, and the first verification module is also used to: If the production and R&D represented by the production-research isolation parameter have not been isolated, a task decision result indicating that the verification failed is output.

5. The system according to claim 3, characterized in that The task registration service module includes: a first task registration service module and a second task registration service module, wherein the first task registration service module is used to store IP information of data tasks and cluster information of data tasks, and the second task registration service module is used to store task registration information of data tasks; The first verification module is specifically used to receive target related information of the target data task, and obtain production IP information or cluster information from the first task registration service module, verify the command type, user authority level and production IP information of the target big data task to obtain a first verification result, and pass the first verification result and related information of the target big data task to the second verification module; The second verification module is used to receive the first verification result and the relevant information of the target big data task, obtain the task registration information from the second task registration service module, verify the task type of the big data task, the task name of the big data task, and the task registration information of the big data task, obtain the task decision result, and return the task decision result to the client.

6. The system according to claim 1, characterized in that The client is also used to: If the task decision result is that the verification fails, the task decision result is recorded and a prompt message is output.

7. A task access method for a big data platform, characterized in that: The method is applied to the task access system for a big data platform as claimed in any one of claims 1 to 6, and the method comprises: Receive target related information of the target big data task input by the user, wherein the related information of the big data task at least includes: production IP information of the big data task, cluster information to which the big data task belongs, and registration information of the big data task; Parsing target related information of the target big data task, and verifying target related information of the target big data task based on task information of pre-stored data tasks; If the task decision result is that the verification is passed, the target big data task is allowed to be executed in the target big data cluster. If the task decision result is that the verification is not passed, the target big data task is refused to be executed in the target big data cluster.

8. A task access device for a big data platform, characterized in that: The device comprises: A receiving module is used to receive target related information of a target big data task input by a user, wherein the related information of the big data task includes at least: production IP information of the big data task, cluster information to which the big data task belongs, and registration information of the big data task; A verification module, used to parse the target related information of the target big data task, and verify the target related information of the target big data task based on the task information of the pre-stored data task; The execution module is used to allow the target big data task to be executed in the target big data cluster if the task decision result is that the verification is passed, and to refuse the target big data task to be executed in the target big data cluster if the task decision result is that the verification is not passed.

9. An electronic device, characterized in that: The electronic device comprises: Processor; and Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to claim 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to claim 7.

Images