Big data management method, device, equipment and product

Abstract

The application discloses a big data management method, device, equipment and product, relates to the technical field of big data, discloses a big data management method, which comprises the following steps: in the development process of a target data task, performing quality detection on the code corresponding to the target data task to obtain a code quality detection result; before deployment and configuration of the target data task which passes the code quality detection result, performing data blood relationship analysis and task dependency management on the target data task to determine the task scheduling time of the target data task; after the deployment and configuration, in the process of data quality monitoring on the target data task, constructing a knowledge graph based on a defined quality rule template, and filling an auditing table corresponding to the target data task based on the matched target quality rule of the knowledge graph, so that the code development efficiency and the code online efficiency are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a big data management method, device, equipment and product. BACKGROUND

[0002] In the field of big data, for big data task code development and automatic testing, mainly covers code development and review, code deployment and configuration management, data quality monitoring, and data blood relationship and dependency management. However, these technical solutions are implemented by different tools, and involve data interaction between multiple independent tools, the development process is relatively cumbersome, which seriously affects the code development efficiency and code online efficiency. SUMMARY

[0003] The main purpose of the present application is to provide a big data management method, device, equipment and product, which aims to improve the code development efficiency and code online efficiency.

[0004] To achieve the above purpose, the present application provides a big data management method, which comprises:

[0005] In the target data task development process, the code corresponding to the target data task is subjected to quality detection, and the code quality detection result is obtained;

[0006] Before the deployment configuration of the target data task whose code quality detection result is passed, the target data task is subjected to data blood relationship analysis and task dependency management, and the task scheduling time of the target data task is determined;

[0007] After the deployment configuration, in the data quality monitoring process of the target data task, the knowledge graph is constructed based on the defined quality rule template, and the target quality rule is matched based on the knowledge graph to fill the audit table corresponding to the target data task.

[0008] In an embodiment, the code quality detection result obtained by subjecting the code corresponding to the target data task to quality detection comprises:

[0009] The code corresponding to the target data task is subjected to quality detection operation by using a syntax parser, and the code quality detection result is obtained, wherein the quality detection operation comprises at least one of the following: reference entity verification operation, syntax verification operation, optimization improvement suggestion operation.

[0010] In an embodiment, in the target data task development process, the code quality detection result obtained by subjecting the code corresponding to the target data task to quality detection operation by using a syntax parser comprises:

[0011] The code is parsed by using a syntax parser to obtain a parsing result, and a syntax rule library is constructed;

[0012] According to the analysis result, it is determined whether the entity referenced in the code exists;

[0013] And / or, the analysis result is compared with the syntax rules in the syntax rule library to determine the code position where the syntax error exists and the error code type;

[0014] And / or, the optimization improvement suggestion corresponding to the code position where the syntax error exists and the error code type is obtained from the syntax rule library.

[0015] In an embodiment, data blood relationship analysis and task dependency management are performed on the target data task, and the task scheduling time of the target data task is determined by:

[0016] The target data task is analyzed by using a syntax parser to determine the pre-data task on which the target data task depends;

[0017] The task scheduling time of the target data task is determined according to the task execution data of the pre-data task.

[0018] In an embodiment, after the step of analyzing the target data task by using a syntax parser to determine the pre-data task on which the target data task depends, the method further comprises:

[0019] The access permission situation and the task execution data of the pre-data task are obtained;

[0020] If the pre-data task does not have corresponding access permission, a permission warning information is generated, and a permission application request is generated to request to open the access permission of the pre-data task.

[0021] In an embodiment, the step of determining the task scheduling time of the target data task according to the task execution data of the pre-data task comprises:

[0022] The task execution data is analyzed by using a time series model to obtain regular information and cycle information of the pre-data task execution;

[0023] The task scheduling time of the target data task is predicted according to the regular information and the cycle information.

[0024] In an embodiment, the step of constructing a knowledge graph based on the defined quality rule template and filling the audit table corresponding to the target data task based on the knowledge graph matching the target quality rule comprises:

[0025] The quality rule template is defined, wherein the quality rule template includes different types of verification rules;

[0026] The quality rules in the quality rule template are data configured to form a knowledge graph;

[0027] vector representations of the nodes of the knowledge graph are determined;

[0028] a similarity between the to-be-recommended rule and the vector representations of the nodes of the knowledge graph is determined, and a target quality rule is determined according to the similarity;

[0029] The audit table corresponding to the target data task is filled with data corresponding to the target quality rule.

[0030] In addition, to achieve the above-mentioned purpose, the present application also provides a big data management device, which comprises:

[0031] The quality detection module is configured to perform quality detection on the code corresponding to the target data task during development of the target data task, and obtain a code quality detection result.

[0032] The management module is configured to perform data blood relationship analysis and task dependency management on the target data task before deployment and configuration of the target data task for which the code quality detection result is passed, and determine a task scheduling time of the target data task.

[0033] The monitoring module is configured to, after the deployment and configuration, perform data quality monitoring on the target data task, construct a knowledge graph based on a defined quality rule template, and fill an audit table corresponding to the target data task with a target quality rule based on the knowledge graph.

[0034] In addition, to achieve the above-mentioned purpose, the present application also provides a big data management device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the above-mentioned big data management method.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which comprises a computer program, the computer program being executable on a processor to implement the steps of the above-mentioned big data management method.

[0036] The present application introduces an integrated agile development and automatic testing tool, and integrates data development, online configuration, and quality monitoring into one tool. Through integration, compliance, integrity, accuracy, and maintainability of the code can be detected in one key during data development; data blood relationship analysis and task dependency management are performed on the target data task before online deployment, a task scheduling time of the target data task is determined, and overall task loading efficiency is ensured; in data quality monitoring, a knowledge graph is constructed based on a defined quality rule template, an audit table corresponding to the target data task is filled with a target quality rule based on the knowledge graph, quality rule setting is efficiently completed, development efficiency is greatly improved, and online quality is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0037] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate embodiments consistent with the present application and, together with the description, further serve to explain the principles of the application.

[0038] To more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the accompanying drawings required by the embodiments or prior art description will be briefly introduced as follows. Obviously, for those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative labor.

[0039] Figure 1 The flowchart provided for the first embodiment of the big data management method of the present application;

[0040] Figure 2 The module structure diagram of the big data management device of the embodiment of the present application;

[0041] Figure 3 The device structure diagram of the hardware running environment involved in the big data management method in the embodiment of the present application.

[0042] The purpose implementation, functional characteristics and advantages of the present application will be further explained with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0043] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and do not limit the present application.

[0044] In order to better understand the technical solutions of the present application, the following will be described in detail in combination with the drawings of the specification and specific embodiments.

[0045] In the related art, in the existing technology in the field of big data, there are various technical solutions for agile development and automatic testing. The related solutions mainly cover code development, deployment and configuration management, data quality monitoring and other aspects, but have not formed a highly integrated solution. The following is the specific technical solution content of the prior art:

[0046] 1. Code development and review.

[0047] Development environment: existing big data project code development mainly relies on various integrated development environments and code editors. The related tools provide code writing, debugging and basic version control functions, but usually lack specific support for big data characteristics.

[0048] Coding standards: Development teams follow certain coding standards to ensure code readability, maintainability, and consistency. However, the implementation of relevant standards often depends on the self-consciousness of developers and the internal audit mechanism of the team.

[0049] Code review: To improve code quality, code review tools are introduced in the prior art. These tools support the review process before code submission, checking the correctness and potential errors of the code through manual or semi-automatic methods. However, this review method is inefficient and easily affected by human factors.

[0050] 2. Deployment and configuration management.

[0051] Deployment tools: In terms of deployment, existing technologies use automated deployment tools to support customized build, test, and deployment processes. However, for the specific deployment needs of big data projects, these tools need a lot of custom configuration when facing data development, including online task dependencies, runtime, etc.

[0052] Configuration management: Configuration management is usually implemented through configuration files or configuration management tools. However, the configuration information of big data projects is complex and variable, making management difficult.

[0053] 3. Data quality monitoring.

[0054] Quality monitoring platform: In the prior art, data quality monitoring usually relies on data quality management systems or data governance platforms. The system can define data quality rules, such as integrity, accuracy, consistency, and other verification rules, and automatically or manually perform data quality check tasks. However, the automation level and comprehensiveness of data verification of related systems need to be improved.

[0055] 4. Data lineage and dependency management.

[0056] Data lineage analysis: In the prior art, data lineage analysis is mainly implemented through data asset management tools or data governance platforms. Related tools can track the source, processing process, and final destination of data, but often lack automation and real-time performance, and are difficult to cope with complex data flow relationships in a big data environment.

[0057] Dependency management: The management of dependency relationships between data is also a difficulty in the prior art. Although some tools support the visualization of data dependency relationships, it is often difficult to automatically identify and handle these dependencies during development, deployment, and testing.

[0058] From the above content, it can be seen that the technical solutions of different links are implemented by different tools, and data interaction between multiple independent tools is involved, making the development process cumbersome, which seriously affects the code development efficiency and code online efficiency.

[0059] To solve the above problems, the application provides a big data management method, and the main technical solution comprises the following steps: in the target data task development process, the quality of the code corresponding to the target data task is detected to obtain a code quality detection result; before the target data task passing the code quality detection result is deployed, the target data task is subjected to data blood relationship analysis and task dependency management to determine the task scheduling time of the target data task; and in the data quality monitoring process of the target data task, a knowledge graph is constructed based on a defined quality rule template, and an audit table corresponding to the target data task is filled based on the matched target quality rule and the knowledge graph.

[0060] The application integrates agile development and automatic testing tools to integrate data development, online configuration and quality monitoring into one tool. Through integration, the compliance, integrity, accuracy and maintainability of the code can be detected in one key during data development; before online deployment, data blood relationship analysis and task dependency management are performed on the target data task to determine the task scheduling time of the target data task and ensure the overall task loading efficiency; in data quality monitoring, a knowledge graph is constructed based on a defined quality rule template, and an audit table corresponding to the target data task is filled based on the matched target quality rule and the knowledge graph, so that the quality rule setting is efficiently completed, the development efficiency is greatly improved, and the online quality is ensured.

[0061] It should be noted that the execution subject of the embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a big data management device capable of realizing the above functions. The following takes the big data management device as an example to describe the embodiment and the following embodiments.

[0062] Based on this, the application provides a big data management method, which is described below with reference to Figure 1 , Figure 1 The flowchart of the first embodiment of the big data management method of the application is shown in the figure. In this embodiment, the big data management method comprises steps S10-S30.

[0063] Step S10: in the target data task development process, the quality of the code corresponding to the target data task is detected to obtain a code quality detection result;

[0064] It should be noted that the target data task can be a data task developed by a tenant on a big data development platform according to its own needs, and each data task has corresponding code.

[0065] Further, step S10 comprises:

[0066] In step S11, the code corresponding to the target data task is subjected to quality detection operation by the syntax parser to obtain code quality detection result, wherein the quality detection operation includes at least one of reference entity checking operation, syntax checking operation, and optimization improvement suggestion operation.

[0067] It should be noted that the syntax parser is JSqlParser SQL syntax parser, and the syntax checking and code quality module can be constructed based on the JSqlParser SQL syntax parser to perform reference entity checking operation, SQL syntax checking operation, and optimization improvement suggestion operation on the SQL code, so as to ensure the correctness of the code corresponding to the target data task. Through the checking, the execution failure or performance problem of the code corresponding to the target data task caused by syntax error can be avoided.

[0068] In an available implementation, step S11 includes:

[0069] In step S111, the code is parsed by the syntax parser to obtain a parsing result, and a syntax rule library is constructed.

[0070] Optionally, parsing the code by the syntax parser to obtain a parsing result includes: using the SQL syntax parsing capability of the JSqlParser syntax parser to parse and traverse the SQL statement of the data task; and parsing the SQL statement in a programming manner to analyze the structure of the SQL query and the table name, column name, and condition expression involved in the data task. The structure of the SQL query and the table name, column name, and condition expression involved in the data task are the parsing result.

[0071] It should be noted that the JSqlParser syntax parser is a SQL parser written in Java, which can parse and convert the SQL statement into an operable Java object.

[0072] Optionally, constructing the syntax rule library includes: classifying the SQL syntax rules of the database and designing the grammar.

[0073] The above-mentioned classification of the SQL syntax rules of the database includes: expanding different database systems by deeply understanding the details of the SQL syntax and the selected technologies such as ANSI SQL, Gbase, SQL Server, and the like, wherein the database systems include the official documents of SQL: 2019, MySQL, PostgreSQL, Oracle, and the like.

[0074] The design grammar includes lexical analysis and syntax analysis. The lexical analysis defines basic symbols in the SQL statement, such as keywords, identifiers, numbers, strings, etc. The syntax analysis defines the structure of the SQL statement, uses the Backus-Naur Form or the Extended Backus-Naur Form to describe the syntax rules of the SQL statement, provides an operation interface, and adds or modifies the performance analysis rules according to actual needs.

[0075] In step S112, it is determined whether the referenced entity in the code exists according to the analysis result.

[0076] Optionally, it is checked whether the referenced entity name such as the table name and the field name in the SQL code exists in the registered information in the enterprise data asset library, and an error is reported if the entity does not exist, which helps to avoid execution errors caused by referencing non-existent entities.

[0077] And / or, in step S113, the analysis result is compared with the syntax rules in the syntax rule library to determine the code position of the syntax error and the error code type.

[0078] Optionally, the SQL code of the target data task is traversed, and in the traversal process, if a syntax error is found, the code position of the syntax error and the error code type are recorded. For the code position of the syntax error, the JSqlParser syntax parser highlights the specific position of the error in the SQL code. For the error agent type, the syntax error type is explained so that the developer can more easily understand and fix the problem.

[0079] And / or, in step S114, the optimization improvement suggestion corresponding to the code position of the syntax error and the error code type is obtained from the syntax rule library.

[0080] Optionally, the optimization SQL code suggestion is provided to improve the code execution performance. The optimization improvement suggestion can be provided according to the error code according to the rule library.

[0081] In this embodiment, the JSqlParser SQL syntax parser is used to realize the deep analysis and quality check of the SQL code. By constructing a detailed SQL syntax rule library, the system can automatically detect syntax errors, non-standard referenced entities and potential performance problems in the code, and provide optimization suggestions. The syntax checking and code quality checking module constructed based on the JSqlParser SQL syntax parser can not only perform detailed syntax checking and referenced entity checking on the SQL code, but also provide code optimization suggestions according to the preset rules and self-learning mode. Compared with the manual or semi-automatic review tools in the prior art, this intelligent checking method is more efficient and less affected by human factors.

[0082] In another possible implementation, the developer needs to select a code quality detection tool that is suitable for their programming language and project requirements. These tools can be static code analysis tools that check for potential issues in the code without running it, or dynamic analysis tools that monitor the performance and resource usage of the code while it is running. Depending on the specific requirements of the project, the developer needs to configure the quality detection tool, including setting up the inspection rules, defining the code style guidelines, specifying the range of code to analyze, and so on. These configurations can ensure that the quality detection tool can accurately identify the issues in the code and provide useful feedback. After the configurations are completed, the developer needs to run the quality detection tool to analyze the code. This can be done through plugins in the integrated development environment, command-line tools, or automated tasks in the continuous integration / continuous deployment pipeline. Running the quality detection tool generates one or more reports that list the issues, warnings, and suggestions in the code. The developer needs to carefully review the quality detection report to understand the problems in the code and their potential impact. This may require certain expertise and experience to judge the severity of the issues and the methods to fix them. Based on the quality detection report, the developer needs to fix the issues in the code. This may include correcting syntax errors, optimizing algorithm performance, improving code structure, following coding standards, and so on. The fixing process may require multiple iterations until the code quality meets the project requirements. After fixing the issues in the code, the developer needs to re-run the quality detection tool to ensure that all issues have been resolved. This helps to verify the effectiveness of the fixing work and avoid missing any potential issues. Finally, the developer needs to record and maintain the quality detection results for future reference and tracking of the changes in code quality. This can be done by storing the quality detection reports in the version control system, recording the issue tracking information in the project management tools, and so on. Through the above steps, the developer can perform code quality detection on the target data task during the development process and obtain detailed detection results. These results help the developer to discover issues in a timely manner, optimize the code, and improve the quality and reliability of the data task.

[0083] In step S20, before deploying the target data task whose code quality detection result is passed, data blood relationship analysis and task dependency management are performed on the target data task to determine the task scheduling time of the target data task.

[0084] In this embodiment, the pre-tables relied on by the target data task are automatically detected before the deployment of the target data task; the SQL statements are analyzed by a syntax parser to identify the data flow, automatically prewarn the pre-tables lacking use permissions, and initiate a permission application; and a time series model is used to automatically predict the task scheduling time of the target data task.

[0085] It should be noted that the data blood relationship refers to the record of each system and processing process through which the data flows in its life cycle, including the source of the data, the processing process, the final destination, etc. Through data blood analysis, the flow path of the data can be tracked, and the generation and change process of the data can be understood; by automatically and quickly generating a data blood atlas, the labor cost is saved. The analysis content includes the pre-tables on which the task depends, the data authority information corresponding to the pre-tables is searched and obtained by interfacing the data assets, and the task completion time and other information. The blood node metadata is uniformly arranged in the form of: node ID, node type, data ID, the blood relationship data is arranged in the form of: source node ID, source node type, target node ID, target node type, and the data structure. In the application aspect of blood data, depth-first search (Depth-First Search, DFS for short) is mainly used. DFS algorithm is to realize traversal of each node by recursion, collect and obtain the running information of each node, and adopt the pruning optimization mode to cut the data in advance. The overall data amount is reduced to improve the traversal efficiency of the algorithm.

[0086] It should be noted that task dependency management refers to identifying, tracking and managing the dependency relationship between target data tasks, including direct dependency and indirect dependency. Alternatively, a project management tool or a graphical tool can be used to establish a task dependency relationship diagram to show the sequence and dependency relationship between tasks. According to the dependency relationship diagram, a task execution plan is formulated to ensure the coordination and consistency between tasks.

[0087] It should be noted that the task scheduling time refers to determining the start time and end time of each task according to the dependency relationship between tasks and resource availability.

[0088] Further, a syntax parser can be used to analyze the target data task to determine the pre-data task on which the target data task depends; the task scheduling time of the target data task is determined according to the task execution data of the pre-data task.

[0089] It should be noted that the pre-data task refers to the task on which the target data task depends, and the target data task needs to depend on the pre-data task for development.

[0090] It should be noted that the task execution data includes execution time, execution duration, resource consumption, etc.

[0091] In a feasible implementation mode, the task scheduling time of the target data task can also be: according to the content and complexity of the task, the resources such as CPU, memory, storage space required by each task are evaluated, according to the resource demand and task dependency relationship, a task scheduling plan is formulated, and the start time and end time of each task are determined. Considering the parallelism and priority between tasks, the scheduling plan is optimized to improve resource utilization and task execution efficiency.

[0092] In another possible implementation, the task execution data can be analyzed by a time series model to obtain regularity information and periodicity information of the pre-data task execution; and the task scheduling time of the target data task is predicted according to the regularity information and the periodicity information.

[0093] It should be noted that the time series model is a statistical method for predicting the numerical change at a future time point. In data task management, the time series model can be used to predict the task scheduling time of the target data task. By collecting the task execution data of the pre-data task such as the execution time, the execution duration, the resource consumption and the like, the time series model can analyze the regularity and periodicity of the task execution. Based on the related regularity and periodicity information, the time series model can predict the best opportunity for task start at a future time point, that is, the task scheduling time of the target data task. This helps to optimize the overall allocation of resources, reduce the waiting time and improve the task execution efficiency.

[0094] Specifically, the following steps are included:

[0095] First, data collection and preprocessing.

[0096] (1) Data range definition: clearly define the range of data collection, covering the historical execution records of all related tasks, including but not limited to task start time, execution duration, resource usage such as CPU, memory, I / O, task status including success, failure, delay and any external factors such as system maintenance window, data availability time.

[0097] (2) Data cleaning and integration: pre-process the collected data, eliminate outliers, fill in missing values, and standardize or normalize the data for model training. Integrate multi-dimensional data to build a comprehensive time series feature set for model input.

[0098] The input data mainly includes the following information:

[0099] Field name Task name Task timing time Task execution time Task execution duration Task memory occupation Task execution status Task CPU occupation duration Task I / O consumption Task end time Number of concurrent tasks running

[0100] Second, feature engineering.

[0101] Feature construction: based on historical data, construct features that help the model learn, such as average execution time of tasks, periodicity indicators of resource consumption such as day of the week, time period of the day, time delay features of task dependency relationships, etc.

[0102] Third, model selection and training, model evaluation and selection: use the deep learning model long short-term memory network LSTM, compare the model performance by cross-validation and other methods, and select the optimal model.

[0103] The operation can be divided into four main steps, which are performed at each time step, as follows:

[0104] 1. Input gate.

[0105] Purpose: Determine how much new information at the current time step needs to be saved into the memory cell.

[0106] Operation: First, the network processes the input x_t and the hidden state at the previous time step h_{t-1} using a sigmoid function to get a vector i_t between 0 and 1, which is called the input gate. A value close to 1 indicates that the information is important and should be allowed to pass through; a value close to 0 indicates that it is not important and should be filtered out. At the same time, a tanh layer called "input adjustment" is used to transform the input x_t to generate a candidate cell state g_t, which contains new information that may be added to the memory cell.

[0107] 2. Forget gate.

[0108] Purpose: Determine which information previously stored in the memory cell should be forgotten.

[0109] Operation: Similarly, use the sigmoid function to process the input and the previous hidden state to get a forget gate vector f_t. A value close to 1 means retaining the corresponding information, and a value close to 0 means forgetting.

[0110] 3. Memory cell update.

[0111] Purpose: Combine the forgetting and input processes to update the memory cell to reflect the integration of new and old information.

[0112] Operation: Use the forget gate f_t to element-wise multiply the memory cell C{t-1} at the previous time step to determine which information needs to be forgotten. Use the input gate i_t to element-wise multiply the candidate cell state g_t to determine which new information needs to be added. Add the results of the above two steps to update the memory cell C_t at the current time step.

[0113] 4. Output gate.

[0114] Purpose: Determine which information in the memory cell should be output to form the hidden state at the current time step.

[0115] Operation: Apply the sigmoid function to determine which part of the memory cell is suitable for output to get the output gate vector o_t. Apply the tanh function to the memory cell C_t for nonlinear transformation, and then multiply it element-wise with the output gate o_t to generate the hidden state h_t at the current time step, which reflects both past information and the influence of current input, and can be directly used for the next step of calculation or as the final output.

[0116] This series of steps enables LSTM to selectively remember long-term information while forgetting unimportant details, making it excel in time series data processing.

[0117] Model Training: Train the selected model using the preprocessed dataset. Adjust model parameters to optimize prediction accuracy while monitoring the risk of overfitting, and introduce regularization or adjust model complexity as necessary.

[0118] Fourth, prediction and verification.

[0119] (1) Generate predictions: Apply the trained model to predict the optimal timing of task initiation for a specified future period. The prediction results should include specific time points, expected resource consumption, and predicted confidence intervals.

[0120] (2) Verification and calibration: Compare the model's prediction results with actual execution data, calculate prediction errors, including mean squared error (MSE), mean absolute error (MAE), etc. Based on error feedback, adjust model parameters or feature selection, and iterate model optimization.

[0121] Fifth, integration and application.

[0122] Decision support system integration: Integrate the optimized prediction model into the data task management system as a decision support tool for task scheduling. The system should automatically receive model output and dynamically adjust task queue and resource allocation strategies accordingly.

[0123] Continuous monitoring and feedback loop: Continuously monitor model performance and system operation effects after deployment, collect new data to continuously update the model, form a closed-loop feedback mechanism, and ensure that prediction accuracy improves over time, better adapting to system changes and external environmental fluctuations.

[0124] Model output:

[0125] Field name Task name Task suggested running time interval range Task expected memory occupation (G) Task expected CPU occupation duration Task expected I / O consumption (G)

[0126] Through strict implementation of the above steps, time series models can not only accurately predict the task scheduling time of target data tasks, but also help efficient resource allocation, significantly improving the overall efficiency and stability of big data processing platforms.

[0127] In this embodiment, the time series model accurately predicts the task scheduling time of the target data task, helping enterprises more reasonably allocate system resources and avoid resource idleness and overload. This prediction capability has higher accuracy and flexibility compared to static configuration or experience-based judgment in existing technologies.

[0128] In another possible implementation, the target data task is analyzed by using a syntax parser to determine a preceding data task on which the target data task depends; the access permission situation and the task execution data of the preceding data task are obtained; if the preceding data task does not have a corresponding access permission, a permission warning information is generated, and a permission application request is generated to request to open the access permission of the preceding data task.

[0129] Optionally, through data permission comparison, it can be detected in advance whether the preceding table corresponding to the target data task has obtained relevant access information. If a missing permission application is found, a permission warning information is sent in time, a permission application work order is generated, and relevant personnel are reminded to handle it as soon as possible, so as to avoid data task failure caused by permission problems.

[0130] In this embodiment, since the dependency relationship can be automatically detected before the data task is deployed, the data flow is identified, and the missing permission application is automatically warned. This automatic management method not only improves the smoothness of the data processing process, but also effectively reduces the risk of data task failure caused by permission problems.

[0131] In step S30, during the data quality monitoring of the target data task, a knowledge graph is constructed based on the defined quality rule template, and the audit table corresponding to the target data task is filled based on the knowledge graph matching the target quality rule.

[0132] In this embodiment, after the quality rule template is defined, the quality rule template is imported into the knowledge graph; the metadata asset information of the target audit table is filled by selecting the quality rule with the highest similarity through the vector matching method, so that the user can configure the quality rule of the target audit table with one key, greatly reducing the burden of the developer, and ensuring the comprehensiveness and accuracy of the data quality rule setting.

[0133] In a possible implementation, step S30 includes:

[0134] In step S31, a quality rule template is defined, wherein the quality rule template includes different types of check rules.

[0135] In this embodiment, the quality rule template includes different types of check rules; the quality rules in the quality rule template are configured to form a knowledge graph; each node of the knowledge graph is represented by a vector; the similarity between the to-be-recommended rule and the vector representation of each node of the knowledge graph is determined, and the target quality rule is determined according to the similarity; and the audit table corresponding to the target data task is filled with the data corresponding to the target quality rule.

[0136] It should be noted that the quality rule template covers multiple levels from basic data type verification to complex business logic verification, including but not limited to non-empty verification, format verification, range verification, consistency verification, primary key uniqueness verification, specific field uniqueness verification, and dependency verification. These rules can be flexibly combined and extended according to actual needs, and the specific details are as follows:

[0137] Integrity verification: (1) Non-empty verification: Ensure that key fields such as ID, name, date, etc. are not empty. (2) Mandatory item verification: Check if a specific field has been filled in, which is applicable to information that all or part of the records must contain.

[0138] Accuracy verification: (1) Format verification: Verify that the field value conforms to the expected format such as date format, email format, phone number format, etc. (2) Range verification: Check if the numerical field is within the expected range such as age, price, score, etc. (3) Consistency verification: Verify the consistency of data across fields or records such as whether the ID number and name match, whether the information of the same person is consistent in different records, etc.

[0139] Uniqueness verification: (1) Primary key uniqueness: Ensure that the value of the primary key field is unique in the database. (2) Specific field uniqueness: Verify the uniqueness of other fields such as email address, mobile phone number, etc. within a specific range such as the entire table or a certain group.

[0140] Logical verification: (1) Dependency verification: Check if there is a correct dependency relationship between fields such as the value of the status field determines the visibility or validity of other fields. (2) Business rule verification: Verify the validity of data according to business logic such as order amount should be greater than 0, inventory quantity should not be negative, etc.

[0141] Time sensitivity verification: (1) Time effectiveness verification: Check if the data is valid within the specified time range such as the end date of the activity should be after the start date of the activity.

[0142] Data update verification: (1) Ensure that data is updated according to predetermined frequency or conditions such as daily, weekly or monthly updates. Duplicate data verification: Record deduplication: Identify and handle duplicate records in data to ensure data uniqueness. Foreign key constraint verification: Ensure that the value of the foreign key field has a corresponding primary key value in the associated table.

[0143] Step S32, data configuration is performed on the quality rules in the quality rule template to form a knowledge graph;

[0144] It should be noted that the quality rule template covers multi-dimensional data, including entity information, field information, and rule information, etc. For each type of information, a plurality of specific fields are included. For example, for the field SUM_DATE, the rule configuration not only involves timeliness verification, but also includes format verification and other types. How to efficiently match these high-quality templates is a key challenge.

[0145] Optionally, the knowledge graph can be used in the form of a template, which is connected to other information through various relationships, and can be further expanded. Through this logical knowledge graph, the accuracy of vector matching is higher. The process of building a knowledge graph to optimize template matching involves multiple steps, from data collection to model training, to final application. The following is the construction process:

[0146] (1) Data collection and preprocessing: First, all relevant entity information, field information, rule information, task running information, etc. should be collected. Related information can be obtained from data assets and scheduling logs. Data preprocessing includes cleaning, such as removing duplicates and handling missing values.

[0147] (2) Define ontology and relationships: In a knowledge graph, entities, attributes, and relationships are core components. You need to define an ontology that clearly defines entity categories, attributes, and relationship types between entities. For example, an entity can be "rule", "attribute is state", and a relationship can be "belongs to", "applied to", etc.

[0148] (3) Build graph: Use a graph database to construct a graph based on the preprocessed data according to the defined ontology and relationship patterns. Ensure the connectivity and integrity of the graph, avoiding isolated nodes or incorrect connections.

[0149] Step S33, vector representation of each node of the knowledge graph is performed;

[0150] Step S34, similarity between the rule to be recommended and the vector representation of each node of the knowledge graph is determined, and the target quality rule is determined according to the similarity;

[0151] Optionally, after the knowledge graph is constructed, graph embedding is performed, and each node in the knowledge graph is represented by a vector. In a higher-dimensional space, each node can find its own position, and then the vector angle between the rule to be recommended and the knowledge graph node in the vector space is compared through mathematical vector operation. The smaller the angle, the higher the similarity between the two, and the target quality rule is selected by comparing the similarity.

[0152] Note that graph embedding is the process of converting nodes and edges in a graph structure into low-dimensional vector space representation, which enables us to more effectively utilize graph data in machine learning and deep learning algorithms. Graph embedding is a kind of graph neural network that iteratively updates node representations through graph convolution layers, taking advantage of the local neighborhood structure of nodes. The implementation steps include:

[0153] (1) Initialization: Assign an initial feature vector to each node.

[0154] (2) Convolution operation: Perform convolution operation on the information of nodes and their neighbors through graph convolution layers, update node representations.

[0155] (3) Output: After multiple layers of convolution operation, get the final node vector representation.

[0156] Note that regardless of the method used, the key to graph embedding is to select appropriate parameters such as the length of random walk, window size, number of layers, etc., and ensure that the model can fully learn the structure and attribute information of the graph. In addition, in order to speed up the training process, GPU resources or distributed computing framework are used.

[0157] In addition, data preprocessing is also needed to ensure the cleanliness and normalization of graph data, remove isolated points and redundant edges. In addition, according to the characteristics and application scenarios of graph data, select the most suitable graph embedding method. And use appropriate indicators such as link prediction, node classification accuracy to evaluate the effect of graph embedding.

[0158] Step S35, filling the target data task corresponding to the audit table with the data corresponding to the target quality rule.

[0159] Note that the audit table is a report or tool that lists items or issues that need to be audited in a table format for systematic checking and evaluation.

[0160] Optionally, after determining the target quality rule, the target data task corresponding to the audit table is filled with the data corresponding to the target quality rule, so that the user can configure the quality rule with one key. Developers do not need to manually write tedious quality rule scripts, but only need to perform a simple one-key operation to automatically generate and apply the corresponding data quality rule. This process greatly simplifies the configuration process and improves work efficiency.

[0161] In this embodiment, by introducing integrated agile development and automatic testing tools, data development, online configuration and quality monitoring are integrated into one tool. Through integration, compliance, integrity, accuracy and maintainability of the code can be detected in one key during data development; before online deployment, data blood relationship analysis and task dependency management are performed on the target data task to determine the task scheduling time of the target data task, thereby ensuring the overall task loading efficiency; in data quality monitoring, a knowledge graph is constructed based on the defined quality rule template, and the audit table corresponding to the target data task is filled based on the knowledge graph matching target quality rules, thereby efficiently completing quality rule setting, greatly improving development efficiency and ensuring online quality.

[0162] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the management method of big data of the present application. More forms of simple transformation based on this technical concept are within the protection scope of the present application.

[0163] The present application also provides a big data management device, please refer to Figure 2 The big data management device comprises:

[0164] The quality detection module 10 is configured to perform quality detection on the code corresponding to the target data task during development of the target data task, and obtain a code quality detection result.

[0165] The management module 20 is configured to perform data blood relationship analysis and task dependency management on the target data task before deployment and configuration of the target data task for which the code quality detection result is passed, and determine the task scheduling time of the target data task.

[0166] The monitoring module 30 is configured to construct a knowledge graph based on a defined quality rule template, and fill the audit table corresponding to the target data task based on the knowledge graph matching target quality rules, during data quality monitoring of the target data task after deployment and configuration.

[0167] Optionally, the quality detection module 10 is further configured to perform quality detection on the code corresponding to the target data task by using a syntax parser, and obtain a code quality detection result, wherein the quality detection operation includes at least one of reference entity verification operation, syntax verification operation, optimization improvement suggestion operation.

[0168] Optionally, the quality detection module 10 is further configured to parse the code by using a syntax parser to obtain a parsing result, and construct a syntax rule library; determine whether an entity referenced in the code exists according to the parsing result; and / or compare the parsing result with syntax rules in the syntax rule library to determine a code position where a syntax error exists and an error code type; and / or obtain an optimization improvement suggestion corresponding to the code position where the syntax error exists and the error code type from the syntax rule library.

[0169] Optionally, the management module 20 is configured to analyze the target data task by using a syntax parser to determine a preceding data task on which the target data task depends; and determine a task scheduling time of the target data task according to task execution data of the preceding data task.

[0170] Optionally, the management module 20 is further configured to obtain access permission information and the task execution data of the preceding data task; if the preceding data task does not have corresponding access permission, generate permission warning information, and generate a permission application request to request to open the access permission of the preceding data task.

[0171] Optionally, the management module 20 is further configured to analyze the task execution data by using a time series model to obtain regularity information and cycle information of the preceding data task execution; and predict the task scheduling time of the target data task according to the regularity information and the cycle information.

[0172] Optionally, the monitoring module 30 is further configured to define a quality rule template, wherein the quality rule template includes different types of verification rules; perform data configuration on quality rules in the quality rule template to form a knowledge graph; perform vector representation on each node of the knowledge graph; determine a similarity between a rule to be recommended and the vector representation of each node of the knowledge graph, and determine a target quality rule according to the similarity; and fill in an audit table corresponding to the target data task by using data corresponding to the target quality rule.

[0173] The management device of big data provided in the present application adopts the management method of big data in the above-mentioned embodiments, which can improve the code development efficiency and the code online efficiency. Compared with the prior art, the management device of big data provided in the present application has the same beneficial effects as the management method of big data provided in the above-mentioned embodiments, and other technical features in the management device of big data are the same as the features disclosed in the above-mentioned embodiments, which will not be repeated here.

[0174] The present application provides a management device of big data, which comprises at least one processor and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the management method of big data in the above-mentioned embodiments.

[0175] Reference will now be made to Figure 3 FIG. 1 shows a structural diagram of a big data management device suitable for implementing embodiments of the present application. The big data management device in embodiments of the present application can include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), car terminals (e.g., car navigation terminals), and the like, as well as fixed terminals such as digital TVs, desktop computers, and the like. Figure 3 The big data management device shown is merely an example and should not impose any limitation on the functions and use range of embodiments of the present application.

[0176] As shown in Figure 3 the big data management device can include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, or the like) that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1002 or programs loaded from a storage device 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the big data management device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. In general, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, and the like; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, and the like; the storage device 1003 including, for example, a magnetic tape, a hard disk, and the like; and a communication device 1009. The communication device 1009 can allow the big data management device to communicate wirelessly or wired with other devices to exchange data. Although the big data management device having various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be alternatively implemented or provided.

[0177] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0178] The management device for big data provided in the present application adopts the management method for big data in the above-mentioned embodiments, and can improve the code development efficiency and the code online efficiency. Compared with the prior art, the management device for big data provided in the present application has the same beneficial effects as the management method for big data provided in the above-mentioned embodiments, and other technical features in the management device for big data are the same as the features disclosed in the above-mentioned embodiments, which will not be repeated here.

[0179] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above-mentioned embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0180] The above is merely specific embodiments of the present application, and the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0181] The present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the management method for big data as described above.

[0182] The computer program product provided in the present application can improve the code development efficiency and the code online efficiency. Compared with the prior art, the computer program product provided in the present application has the same beneficial effects as the management method for big data provided in the above-mentioned embodiments, which will not be repeated here.

[0183] The above is merely part of the embodiments of the present application, and does not limit the patent scope of the present application, and any equivalent structural transformation made by using the contents of the present application specification and drawings, or direct / indirect application in other related technical fields is included in the patent protection scope of the present application.

Claims

1. A big data management method, characterized in that, The method includes: During the development of the target data task, the code corresponding to the target data task is subjected to quality inspection, and the code quality inspection results are obtained. Before deploying and configuring the target data task that has passed the code quality inspection, the target data task is analyzed using a syntax parser to determine the preceding data tasks that the target data task depends on; the task execution data is analyzed using a time series model to obtain the regularity and periodicity information of the preceding data task execution; and the task scheduling time of the target data task is predicted based on the regularity and periodicity information. After deployment and configuration, during the data quality monitoring of the target data task, a knowledge graph is constructed based on the defined quality rule template, and the audit table corresponding to the target data task is populated based on the target quality rule matched with the knowledge graph.

2. The method as described in claim 1, characterized in that, The code quality inspection of the target data task and the resulting code quality inspection results include: The code corresponding to the target data task is subjected to quality inspection using a syntax parser to obtain the code quality inspection result. The quality inspection operation includes at least one of the following: reference entity verification operation, syntax verification operation, and optimization and improvement suggestion operation.

3. The method as described in claim 2, characterized in that, The step of performing quality checks on the code corresponding to the target data task using a syntax parser during the development process of the target data task, and obtaining the code quality check results, includes: The code is parsed using a syntax parser to obtain the parsing results and to build a syntax rule base. Based on the parsing results, determine whether the entity referenced in the code exists; And / or, compare the parsing results with the syntax rules in the syntax rule base to determine the location of the code containing the syntax error and the error code type; And / or, obtain optimization and improvement suggestions corresponding to the code location and error code type where the syntax error exists from the syntax rule base.

4. The method as described in claim 1, characterized in that, After the step of analyzing the target data task using a syntax parser to determine the prerequisite data tasks on which the target data task depends, the method further includes: Obtain the access permissions and task execution data of the aforementioned pre-processing data task; If the preceding data task does not have corresponding access permissions, a permission warning message is generated, and a permission request is generated to request the opening of access permissions for the preceding data task.

5. The method as described in claim 1, characterized in that, The steps of constructing a knowledge graph based on a defined quality rule template and filling the audit table corresponding to the target data task with the target quality rule based on the knowledge graph include: Define a quality rule template, wherein the quality rule template includes different types of verification rules; The quality rules in the quality rule template are configured with data to form a knowledge graph; Each node of the knowledge graph is represented by a vector. Determine the similarity between the rule to be recommended and the vector representation of each node in the knowledge graph, and determine the target quality rule based on the similarity. The audit table corresponding to the target data task is populated with the data corresponding to the target quality rule.

6. A big data management device, characterized in that, The device includes: The quality inspection module is used to perform quality inspection on the code corresponding to the target data task during the development process of the target data task and obtain the code quality inspection results. The management module is used to analyze the target data task using a syntax parser before deploying and configuring the target data task that has passed the code quality inspection, to determine the preceding data tasks that the target data task depends on; to analyze the task execution data using a time series model to obtain the regularity and periodicity information of the preceding data task execution; and to predict the task scheduling time of the target data task based on the regularity and periodicity information. The monitoring module is used, after deployment and configuration, to construct a knowledge graph based on a defined quality rule template during the data quality monitoring process of the target data task, and to populate the audit table corresponding to the target data task based on the target quality rule matching of the knowledge graph.

7. A big data management device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the big data management method as described in any one of claims 1 to 5.

8. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the big data management method as described in any one of claims 1 to 5.

Images