Big data management method and device, equipment and product

Abstract

The invention discloses a big data management method and device, equipment and a product, relates to the technical field of big data, and discloses a big data management method which comprises the steps that in the development process of a target data task, quality detection is conducted on a code corresponding to the target data task, and a code quality detection result is obtained; before a target data task whose code quality detection result is passing is deployed and configured, data blood relationship analysis and task dependency management are performed on the target data task, and task scheduling time of the target data task is determined; after deployment and configuration, in the process of performing data quality monitoring on the target data task, the knowledge graph is constructed based on the defined quality rule template, and the target quality rule is matched based on the knowledge graph to fill the auditing table corresponding to the target data task, so that the code development efficiency and the code online efficiency are improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to big data management methods, devices, equipment and products. Background Art

[0002] In the field of big data, code development and automated testing for big data tasks mainly cover code development and review, code deployment and configuration management, data quality monitoring, and data lineage and dependency management. However, these technical solutions are implemented by different tools and involve data interaction between multiple independent tools. The development process is cumbersome, which seriously affects the efficiency of code development and code launch. Summary of the invention

[0003] The main purpose of this application is to provide a big data management method, device, equipment and product, aiming to improve the efficiency of code development and code launch.

[0004] To achieve the above objectives, this application proposes a big data management method, which includes:

[0005] During the development of the target data task, the code corresponding to the target data task is quality tested to obtain the code quality test result;

[0006] Before deploying and configuring the target data task whose code quality test result is passed, perform data lineage analysis and task dependency management on the target data task to determine the task scheduling time of the target data task;

[0007] After deployment and configuration, during the data quality monitoring process for the target data task, a knowledge graph is built based on the defined quality rule template, and the audit table corresponding to the target data task is filled based on the knowledge graph matching the target quality rules.

[0008] In one embodiment, the quality of the code corresponding to the target data task is tested, and the code quality test results obtained include:

[0009] A syntax parser is used to perform a quality inspection operation on the code corresponding to the target data task to obtain a code quality inspection result, wherein the quality inspection operation includes: at least one of a reference entity verification operation, a syntax verification operation, and an optimization and improvement suggestion operation.

[0010] In one embodiment, during the development of the target data task, the steps of using a syntax parser to perform a quality detection operation on the code corresponding to the target data task and obtaining the code quality detection result include:

[0011] Use the syntax parser to parse the code, obtain the parsing results, and build a syntax rule library;

[0012] According to the parsing results, determine whether the entity referenced in the code exists;

[0013] and / or, comparing the parsing result with the grammar rules in the grammar rule base to determine the code position and error code type of the grammar error;

[0014] And / or, obtain optimization improvement suggestions corresponding to the code position with syntax errors and the error code type from the syntax rule library.

[0015] In one embodiment, performing data lineage analysis and task dependency management on target data tasks and determining the task scheduling time of the target data tasks includes:

[0016] Use the syntax parser to analyze the target data task and determine the predecessor data task that the target data task depends on;

[0017] The task scheduling time of the target data task is determined according to the task execution data of the preceding data task.

[0018] In one embodiment, after the step of analyzing the target data task by using a syntax parser to determine the preceding data task on which the target data task depends, the following steps are further included:

[0019] Obtain access rights to pre-process data tasks and task execution data;

[0020] If the corresponding access rights do not exist for the preceding data task, a permission warning message is generated, and a permission application request is generated to request to open the access rights for the preceding data task.

[0021] In one embodiment, the step of determining the task scheduling time of the target data task according to the task execution data of the preceding data task includes:

[0022] Analyze the task execution data through the time series model to obtain the regularity and cycle information of the previous data task execution;

[0023] According to the regularity information and cycle information, the task scheduling time of the target data task is predicted.

[0024] In one embodiment, the steps of constructing a knowledge graph based on a defined quality rule template and filling an audit table corresponding to a target data task based on matching the target quality rule with the knowledge graph include:

[0025] defining a quality rule template, wherein the quality rule template includes different types of verification rules;

[0026] Perform data configuration on the quality rules in the quality rule template to form a knowledge graph;

[0027] Vectorize each node of the knowledge graph;

[0028] Determine the similarity between the rule to be recommended and the vector representation of each node of the knowledge graph, and determine the target quality rule based on the similarity;

[0029] The audit table corresponding to the target data task is filled with the data corresponding to the target quality rules.

[0030] In addition, to achieve the above purpose, the present application also proposes a big data management device, which includes:

[0031] The quality detection module is used to perform quality detection on the code corresponding to the target data task during the development process of the target data task and obtain the code quality detection result;

[0032] A management module is used to perform data lineage analysis and task dependency management on target data tasks before deploying and configuring target data tasks with passed code quality detection results, and determine the task scheduling time of target data tasks;

[0033] The monitoring module is used to build a knowledge graph based on the defined quality rule template during data quality monitoring of the target data task after deployment and configuration, and to fill in the audit table corresponding to the target data task based on the knowledge graph matching the target quality rules.

[0034] In addition, to achieve the above objectives, the present application also proposes a big data management device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the big data management method as described above.

[0035] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the big data management method as described above.

[0036] This application introduces an integrated agile development and automated testing tool, integrating data development, online configuration and quality monitoring into one tool. Through an integrated approach, the compliance, completeness, accuracy and maintainability of the code can be checked with one click in data development; before online deployment, the target data tasks are analyzed for data lineage and task dependency management, the task scheduling time of the target data tasks is determined, and the overall task loading efficiency is guaranteed; in data quality monitoring, a knowledge graph is constructed based on the defined quality rule template, and the audit table corresponding to the target data task is filled based on the knowledge graph matching the target quality rules, the quality rule setting is completed efficiently, the development efficiency is greatly improved, and the online quality is ensured. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0039] Figure 1 A flowchart diagram of the first embodiment of the big data management method of the present application is provided;

[0040] Figure 2 This is a schematic diagram of the module structure of the big data management device of the embodiment of the present application;

[0041] Figure 3 This is a schematic diagram of the device structure of the hardware operating environment involved in the big data management method in the embodiment of the present application.

[0042] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0043] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0044] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0045] In the related technologies, in the existing technologies in the field of big data, there are many technical solutions for agile development and automated testing. The related solutions mainly cover code development, deployment and configuration management, data quality monitoring and other aspects, but a highly integrated solution has not yet been formed. The following are the specific technical solutions of the existing technologies:

[0046] 1. Code development and review.

[0047] Development environment: Existing big data project code development mainly relies on various integrated development environments and code editors. The related tools provide code writing, debugging and basic version control functions, but usually lack specific support for big data features.

[0048] Coding standards: Development teams follow certain coding standards to ensure code readability, maintainability, and consistency. However, the implementation of relevant standards often depends on the self-consciousness of developers and the team's internal review mechanism.

[0049] Code review: In order to improve code quality, code review tools have been introduced in existing technologies. These tools support the review process before code submission, and check the correctness and potential errors of the code in a manual or semi-automatic way. However, this review method is inefficient and easily affected by human factors.

[0050] 2. Deployment and configuration management.

[0051] Deployment tools: In terms of deployment, existing technologies use automated deployment tools to support customized build, test, and deployment processes. However, in response to the specific deployment requirements of big data projects, these tools require a lot of custom configuration when developing data, including online task dependencies, runtime, etc.

[0052] Configuration management: Configuration management is usually implemented through configuration files or configuration management tools. However, the configuration information of big data projects is complex and changeable, making it difficult to manage.

[0053] 3. Data quality monitoring.

[0054] Quality monitoring platform: In existing technologies, data quality monitoring usually relies on a data quality management system or a data governance platform. The system can define data quality rules, such as completeness, accuracy, consistency and other verification rules, and automatically or manually perform data quality inspection tasks. However, the automation level of the relevant system and the comprehensiveness of data verification need to be improved.

[0055] 4. Data lineage and dependency management.

[0056] Data lineage analysis: In existing technologies, data lineage analysis is mainly implemented through data asset management tools or data governance platforms. Relevant tools can track the source, processing process and final destination of data, but they often lack automation and real-time performance, and are difficult to cope with the complex data flow relationships in a big data environment.

[0057] Dependency management: Dependency management between data is also a difficulty in existing technologies. Although some tools support the visualization of data dependencies, it is often difficult to automatically identify and handle these dependencies during development, deployment, and testing.

[0058] From the above content, we can see that the technical solutions for different links are implemented by different tools and involve data interaction between multiple independent tools. The development process is relatively cumbersome, which seriously affects the efficiency of code development and code launch.

[0059] In response to the above problems, the present application proposes a big data management method, and the main technical solutions include: in the process of developing the target data task, performing quality inspection on the code corresponding to the target data task to obtain the code quality inspection result; before deploying the target data task with a passed code quality inspection result, performing data lineage analysis and task dependency management on the target data task to determine the task scheduling time of the target data task; and, in the process of data quality monitoring of the target data task, building a knowledge graph based on the defined quality rule template, and filling the audit table corresponding to the target data task based on matching the target quality rules on the knowledge graph.

[0060] This application introduces an integrated agile development and automated testing tool, integrating data development, online configuration and quality monitoring into one tool. Through an integrated approach, the compliance, completeness, accuracy and maintainability of the code can be checked with one click in data development; before online deployment, the target data tasks are analyzed for data lineage and task dependency management, the task scheduling time of the target data tasks is determined, and the overall task loading efficiency is guaranteed; in data quality monitoring, a knowledge graph is constructed based on the defined quality rule template, and the audit table corresponding to the target data task is filled based on the knowledge graph matching the target quality rules, the quality rule setting is completed efficiently, the development efficiency is greatly improved, and the online quality is ensured.

[0061] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or a big data management device capable of realizing the above functions, etc. The following takes the big data management device as an example to illustrate this embodiment and the following embodiments.

[0062] Based on this, the present application embodiment provides a big data management method, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the big data management method of the present application. In this embodiment, the big data management method includes steps S10 to S30:

[0063] Step S10, during the development process of the target data task, performing quality inspection on the code corresponding to the target data task to obtain a code quality inspection result;

[0064] It should be noted that the target data task may be a data task developed by a tenant on the big data development platform according to his own needs, and each data task has a corresponding code.

[0065] Further, step S10 includes:

[0066] Step S11, using a syntax parser to perform a quality check operation on the code corresponding to the target data task to obtain a code quality check result, wherein the quality check operation includes: at least one of a reference entity check operation, a syntax check operation, and an optimization and improvement suggestion operation.

[0067] It should be noted that the syntax parser is JSqlParser SQL syntax parser. A syntax verification and code quality module can be built based on JSqlParserSQL syntax parser to perform reference entity verification, SQL syntax verification and optimization and improvement suggestion operations on the SQL code to ensure the correctness of the code corresponding to the target data task. Through verification, execution failure or performance problems caused by syntax errors in the code corresponding to the target data task can be avoided.

[0068] In a feasible implementation manner, step S11 includes:

[0069] Step S111, using a syntax parser to parse the code, obtain parsing results, and construct a syntax rule library;

[0070] Optionally, the code is parsed using a syntax parser to obtain parsing results including: using the SQL syntax parsing capability of the JSqlParser syntax parser to parse and traverse the SQL statements of the data task; parsing the SQL statements programmatically to analyze the structure of the SQL query, and the data task involved such as table names, column names, conditional expressions, etc. Here, the structure of the SQL query, and the data task involved such as table names, column names, and conditional expressions are the parsing results.

[0071] It should be noted that the JSqlParser syntax parser is a SQL parser written in Java, which can parse SQL statements and convert them into operable Java objects.

[0072] Optionally, building a grammar rule base includes: classifying SQL grammar rules based on a database and designing a grammar.

[0073] The above-mentioned database-based SQL grammar rule classification includes: through in-depth understanding of SQL grammar and details of selected technologies such as ANSI SQL, Gbase, SQL Server, etc., it is expanded according to different database systems, among which the database systems include SQL: 2019; official documents of database vendors such as MySQL, PostgreSQL, Oracle, etc.

[0074] The above-mentioned design grammar includes: lexical analysis and syntax analysis. Among them, lexical analysis defines the basic symbols in SQL statements, such as keywords, identifiers, numbers, strings, etc. Syntax analysis defines the structure of SQL statements, uses Backus-Naur form or extended Backus-Naur form to describe the syntax rules of SQL statements, provides an operation interface, and adds or modifies performance analysis rules according to actual needs.

[0075] Step S112, determining whether the entity referenced in the code exists according to the parsing result;

[0076] Optionally, check whether the entity names referenced in the SQL code, such as table names, field names, and registered information in the enterprise data asset library exist. If not, an error is reported, which helps avoid execution errors caused by referencing non-existent entities.

[0077] And / or, step S113, comparing the parsing result with the grammar rules in the grammar rule library to determine the code position and error code type where the grammar error exists;

[0078] Optionally, perform SQL code traversal on the target data task code. During the traversal, if a syntax error is found, record the code location of the syntax error and the error code type. For the code location of the syntax error, the JSqlParser syntax parser highlights the specific location of the error in the SQL code provided. For the error proxy type, the syntax error type can be described to make it easier for developers to understand and fix the problem.

[0079] And / or, step S114, obtaining optimization improvement suggestions corresponding to the code position with grammatical errors and the error code type from the grammar rule library.

[0080] Optionally, provide suggestions for optimizing SQL codes to improve code execution performance. Optimization and improvement suggestions can be provided based on the error code and the rule base.

[0081] In this embodiment, the JSqlParser SQL syntax parser is used to implement in-depth analysis and quality inspection of SQL code. By building a detailed SQL syntax rule library, the system can automatically detect syntax errors, irregular reference entities, and potential performance problems in the code, and provide optimization suggestions. The syntax check and code quality inspection module built based on the JSqlParser SQL syntax parser can not only perform detailed syntax check and reference entity check on the SQL code, but also provide code optimization suggestions based on preset rules and self-learning mode. This intelligent inspection method is more efficient than manual or semi-automatic review tools in the prior art and is not easily affected by human factors.

[0082] In another feasible implementation, developers need to select code quality detection tools that are suitable for their programming language and project requirements. These tools can be static code analysis tools that are used to check potential problems in the code without running the code; or dynamic analysis tools that are used to monitor the performance and resource usage of the code when it is running. According to the specific requirements of the project, developers need to configure the quality detection tool, including setting inspection rules, defining code style guidelines, specifying the code range to be analyzed, etc. These configurations can ensure that the quality detection tool can accurately identify problems in the code and provide useful feedback. After the configuration is completed, the developer needs to run the quality detection tool to analyze the code. This can be achieved through plug-ins in the integrated development environment, command line tools, or automated tasks in the continuous integration / continuous deployment pipeline. Running the quality detection tool will generate one or more reports that list problems, warnings, and suggestions in the code. Developers need to carefully review the quality detection report to understand the problems in the code and their potential impact. This may require certain expertise and experience to judge the severity of the problem and how to fix it. Based on the quality detection report, developers need to fix the problems in the code. This may include correcting syntax errors, optimizing algorithm performance, improving code structure, following coding standards, etc. The repair process may require multiple iterations until the code quality meets the project requirements. After fixing the code issues, developers need to rerun the quality inspection tool to ensure that all issues have been resolved. This helps to verify the effectiveness of the repair work and avoid missing any potential problems. Finally, developers need to record and maintain the quality inspection results so that they can refer to and track changes in code quality during the subsequent development process. This can be achieved by storing quality inspection reports in the version control system and recording issue tracking information in the project management tool. Through the above steps, developers can perform quality inspections on the code during the development of the target data task and obtain detailed inspection results. These results help developers to find problems in a timely manner, optimize the code, and improve the quality and reliability of data tasks.

[0083] Step S20, before deploying the target data task whose code quality detection result is passed, perform data lineage analysis and task dependency management on the target data task to determine the task scheduling time of the target data task;

[0084] In this embodiment, the dependent pre-tables of the target data task are automatically detected before it is deployed; the SQL statement is analyzed by the syntax parser to identify the data flow, and an automatic pre-warning is issued for the pre-tables that lack usage permissions, and an application for permissions is initiated; and the task scheduling time of the target data task is automatically predicted using the time series model.

[0085] It should be noted that data lineage refers to the records of various systems and processing processes that data flows through during its life cycle, including the source, processing process, and final destination of the data. Through data lineage analysis, the data flow path can be tracked and the data generation and change process can be understood; data lineage maps can be quickly generated through automation to save labor costs. The analysis content includes the pre-table that the task depends on, the data permission information corresponding to the pre-table searched and obtained by docking with data assets, and the task running completion time and other information. The lineage node metadata is uniformly organized into the form of: node ID, node type, and data ID, and the lineage relationship data is organized into the data structure of: source node ID, source node type, target node ID, and target node type. In the application of lineage data, depth-first search (DFS) is mainly used. The DFS algorithm uses recursion to traverse each node, collect and obtain the running information of each node, and uses pruning optimization to prune the data in advance. Reducing the overall data volume improves the efficiency of algorithm traversal.

[0086] It should be noted that task dependency management refers to identifying, tracking, and managing the dependencies between target data tasks, including direct dependencies and indirect dependencies. Optionally, a project management tool or graphical tool can be used to create a task dependency diagram to show the sequence and dependencies between tasks. According to the dependency diagram, a task execution plan is formulated to ensure coordination and consistency between tasks.

[0087] It should be noted that task scheduling time refers to determining the start time and end time of each task based on the dependencies between tasks and resource availability.

[0088] Furthermore, a syntax parser may be used to analyze the target data task to determine the predecessor data task on which the target data task depends; and the task scheduling time of the target data task may be determined based on the task execution data of the predecessor data task.

[0089] It should be noted that the predecessor data task refers to the task that the target data task depends on, and the target data task needs to rely on the predecessor data task for development.

[0090] It should be noted that task execution data includes execution time, execution duration, resource consumption, etc.

[0091] In a feasible implementation, the task scheduling time of the target data task can also be determined by: evaluating the resources required for each task such as CPU, memory, storage space, etc. according to the content and complexity of the task, formulating a task scheduling plan according to the resource requirements and task dependencies, and determining the start time and end time of each task. Considering the parallelism and priority between tasks, the scheduling plan is optimized to improve resource utilization and task execution efficiency.

[0092] In another feasible implementation, the task execution data can be analyzed through a time series model to obtain regularity information and periodicity information when the predecessor data task is executed; based on the regularity information and periodicity information, the task scheduling time of the target data task is predicted.

[0093] It should be noted that the time series model is a statistical method for predicting changes in values ​​at future time points. In data task management, the time series model can be used to predict the task scheduling time of the target data task. By collecting the task execution data of the preceding data tasks, such as execution time, execution duration, resource consumption, etc., the time series model can analyze the regularity and periodicity of task execution. Based on the relevant regularity and periodicity information, the time series model can predict the best time to start a task at a certain point in the future, that is, the task scheduling time of the target data task. This helps to optimize the overall allocation of resources, reduce waiting time, and improve task execution efficiency.

[0094] Specifically, the following steps are included:

[0095] First, data collection and preprocessing.

[0096] (1) Data scope definition: Clarify the scope of data collection, covering the historical execution records of all relevant tasks, including but not limited to task start time, execution duration, resource usage such as CPU, memory, I / O, etc., task status including success, failure, delay, and any external factors such as system maintenance windows and data availability time.

[0097] (2) Data cleaning and integration: Preprocess the collected data, remove outliers, fill in missing values, and standardize or normalize the data to facilitate model training. Integrate multi-dimensional data and build a comprehensive time series feature set to prepare for model input.

[0098] The input data mainly includes the following information:

[0099] Field Name Task Name Task timing time Task execution time Task execution time Task memory usage Task execution status Task CPU usage time Task I / O consumption Task end time Number of tasks running at the same time

[0100] Second, feature engineering.

[0101] Feature construction: Based on historical data, construct features that help model learning, such as the average execution time of tasks, periodic indicators of resource consumption such as day of the week, time of day, and delay characteristics of dependencies between tasks.

[0102] Third, model selection and training, model evaluation and selection: Use the deep learning model long short-term memory network LSTM, compare model performance through cross-validation and other methods, and select the optimal model.

[0103] Its operation process can be divided into four main steps, which are performed at each time step, as follows:

[0104] 1. Input gate.

[0105] Purpose: Determine how much new information at the current time step needs to be saved in the memory cell.

[0106] Operation: First, the network uses a sigmoid function to process the input x_t and the hidden state h_{t-1} of the previous time step to obtain a vector i_t between 0 and 1, which is called the input gate. Values ​​close to 1 indicate that the information is important and should be allowed to pass; values ​​close to 0 indicate that it is unimportant and should be filtered out. At the same time, the input x_t is transformed through a tanh layer called "input adjustment" to generate a candidate cell state g_t, which contains new information that may be added to the memory cell.

[0107] 2. Forget gate.

[0108] Purpose: Decide which information previously stored in a memory cell should be forgotten.

[0109] Operation: Similarly, the sigmoid function is used to process the input and the previous hidden state to obtain a forget gate vector f_t. A value close to 1 means that the corresponding information is retained, and a value close to 0 means forgetting.

[0110] 3. Memory unit update.

[0111] Purpose: Combines the processes of forgetting and input to update memory cells to reflect a combination of new and old information.

[0112] Operation: Use the forget gate f_t to multiply the memory cell C{t-1} of the previous time step element by element to determine which information needs to be forgotten. Use the input gate i_t to multiply the candidate cell state g_t element by element to determine which new information needs to be added. Add the results of the above two steps to update the memory cell C_t of the current time step.

[0113] 4. Output gate.

[0114] Purpose: Determine which memory units’ information should be output to form the hidden state of the current time step.

[0115] Operation: Apply the sigmoid function to determine which parts of the memory cells are suitable for output, and obtain the output gate vector o_t. Apply the tanh function to the memory cell C_t for nonlinear transformation, and then multiply it element by element with the output gate o_t to generate the hidden state h_t of the current time step, which reflects both past information and the influence of the current input, and can be directly used for the next step of calculation or as the final output.

[0116] This series of steps enables LSTM to selectively remember long-term information while forgetting unimportant details, thus performing well in time series data processing.

[0117] Model training: Use the selected model and preprocessed dataset for training. Pay attention to adjusting model parameters to optimize prediction accuracy, monitor overfitting risks, and introduce regularization or adjust model complexity when necessary.

[0118] Fourth, prediction and verification.

[0119] (1) Generate predictions: Use the trained model to predict the best time to start a task within a specified time period in the future. The prediction results should include the specific time point, expected resource consumption, and the confidence interval of the prediction.

[0120] (2) Backtesting and calibration: Compare the model prediction results with the actual execution data and calculate the prediction error, including indicators such as mean square error (MSE) and mean absolute error (MAE). Based on the error feedback, adjust the model parameters or feature selection, and perform iterative model optimization.

[0121] Fifth, integration and application.

[0122] Decision support system integration: Integrate the optimized prediction model into the data task management system as a support tool for task scheduling decisions. The system should be able to automatically receive the model output and dynamically adjust the task queue and resource allocation strategy accordingly.

[0123] Continuous monitoring and feedback loop: After deployment, the model performance and system operation results are continuously monitored, new data is collected to continuously iterate and update the model, forming a closed-loop feedback mechanism to ensure that the prediction accuracy improves over time and better adapts to system changes and external environment fluctuations.

[0124] Model output:

[0125] Field Name Task Name Recommended task running time range Expected memory usage of the task (G) Expected CPU usage time of the task Expected I / O consumption of the task (G)

[0126] Through the strict implementation of the above steps, the time series model can not only accurately predict the task scheduling time of the target data task, but also help to efficiently allocate resources and significantly improve the overall performance and stability of the big data processing platform.

[0127] In this embodiment, the time series model is used to accurately predict the task scheduling time of the target data task, which can help enterprises allocate system resources more reasonably and avoid resource idleness and overload. This prediction capability has higher accuracy and flexibility than the static configuration or experience judgment in the prior art.

[0128] In another feasible implementation, a syntax parser is used to analyze the target data task to determine the predecessor data task on which the target data task depends; the access rights of the predecessor data task and the task execution data are obtained; if the predecessor data task does not have corresponding access rights, a permission warning message is generated, and a permission application request is generated to request to open the access rights of the predecessor data task.

[0129] Optionally, by comparing data permissions, it is possible to detect in advance whether the front table corresponding to the target data task has obtained relevant access information. If missing permission applications are found, permission warning information will be issued in a timely manner, permission application work orders will be generated, and relevant personnel will be reminded to handle them as soon as possible to avoid data task failures due to permission issues.

[0130] In this embodiment, since it is possible to automatically detect dependencies, identify data flows, and automatically warn of missing permission applications before data tasks are deployed, this automated management method not only improves the smoothness of the data processing process, but also effectively reduces the risk of data task failures due to permission issues.

[0131] Step S30, during the data quality monitoring process for the target data task, a knowledge graph is constructed based on the defined quality rule template, and the audit table corresponding to the target data task is filled based on the knowledge graph matching the target quality rules.

[0132] In this embodiment, after the quality rule template is defined, the quality rule template is imported into the knowledge graph; the metadata asset information of the target audit table is filled with the quality rule with the highest similarity selected through the vector matching method, so that the user can configure the target audit table quality rules with one click, which greatly reduces the burden on developers and ensures the comprehensiveness and accuracy of data quality rule settings.

[0133] In a feasible implementation manner, step S30 includes:

[0134] Step S31, defining a quality rule template, wherein the quality rule template includes different types of verification rules;

[0135] In this embodiment, the quality rule template includes different types of verification rules; data configuration is performed on the quality rules in the quality rule template to form a knowledge graph; each node of the knowledge graph is represented by a vector; the similarity between the rule to be recommended and the vector representation of each node of the knowledge graph is determined, and the target quality rule is determined based on the similarity; the data corresponding to the target quality rule is used to fill the audit table corresponding to the target data task.

[0136] It should be noted that the quality rule template covers multiple levels from basic data type verification to complex business logic verification, including but not limited to non-null verification, format verification, range verification, consistency verification, primary key uniqueness verification, specific field uniqueness verification and dependency verification, etc. These rules can be flexibly combined and expanded according to actual needs, the specific details are as follows;

[0137] Completeness check: (1) Non-empty check: Ensure that key fields such as ID, name, date, etc. are not empty. (2) Required field check: Check whether specific fields are filled in, applicable to information that must be included in all or part of the record.

[0138] Accuracy check: (1) Format check: Verify whether the field value conforms to the expected format, such as date format, email format, phone number format, etc. (2) Range check: Check whether the numeric field is within the expected range, such as age, price, score, etc. (3) Consistency check: Verify the consistency of data across fields or records, such as whether the ID number and name match, whether the information of the same person is consistent in different records, etc.

[0139] Uniqueness check: (1) Primary key uniqueness: Ensure that the value of the primary key field is unique in the database. (2) Specific field uniqueness: Verify the uniqueness of other fields other than the primary key, such as email address, mobile phone number, etc., within a specific range, such as the entire table or a group.

[0140] Logical verification: (1) Dependency verification: Check whether there is a correct dependency relationship between fields, such as the value of the status field determines the visibility or validity of other fields. (2) Business rule verification: Verify the validity of data based on business logic, such as the order amount should be greater than 0, the inventory quantity should not be negative, etc.

[0141] Time sensitivity check: (1) Timeliness check: Check whether the data is valid within the specified time range, such as the activity end date should be after the activity start date.

[0142] Data update verification: (1) Ensure that data is updated according to a predetermined frequency or condition, such as daily, weekly or monthly updates. Duplicate data verification: Record deduplication: Identify and process duplicate records in the data to ensure data uniqueness. Foreign key constraint verification: Ensure that the value of the foreign key field has a corresponding primary key value in the associated table.

[0143] Step S32, performing data configuration on the quality rules in the quality rule template to form a knowledge graph;

[0144] It should be noted that quality rule templates cover multi-dimensional data, including entity information, field information, and rule information. For each type of information, multiple specific fields are included. For example, for the focus field SUM_DATE, its rule configuration involves not only timeliness verification, but also multiple types such as format verification. How to efficiently match these high-quality templates is a key challenge.

[0145] Alternatively, a knowledge graph can be used, with the template as the center and other information connected through various relationships, and it can be further expanded. This logical knowledge graph can achieve higher accuracy when matching vectors. The process of building a knowledge graph to optimize template matching involves multiple steps, from data collection to model training to final application. The following is the construction process:

[0146] (1) Data collection and preprocessing: First, all relevant entity information, field information, rule information, task operation information, etc. should be collected. Relevant information can be obtained from data assets and scheduling logs. Data preprocessing includes cleaning, such as removing duplicates and processing missing values.

[0147] (2) Define ontology and relationships: In a knowledge graph, entities, attributes, and relationships are the core components. You need to define an ontology to clarify the entity categories, attributes, and relationship types between entities. For example, an entity can be a "rule" or "attribute is a state", and a relationship can be "belongs to" or "applies to", etc.

[0148] (3) Graph construction: Use a graph database to construct a graph based on the preprocessed data and the defined ontology and relationship model. Ensure the connectivity and integrity of the graph and avoid isolated nodes or incorrect connections.

[0149] Step S33, vectorizing each node of the knowledge graph;

[0150] Step S34, determining the similarity between the rule to be recommended and the vector representation of each node of the knowledge graph, and determining the target quality rule according to the similarity;

[0151] Optionally, after the knowledge graph is constructed, graph embedding is performed to represent each node in the knowledge graph with a vector. In a higher-dimensional space, each node can find a position that suits it, and then compare the vector angle between the recommended rule and the knowledge graph node in the vector space through mathematical vector operations. The smaller the angle, the higher the similarity between the two. The target quality rule is selected through similarity comparison.

[0152] It should be noted that graph embedding is the process of converting nodes and edges in a graph structure into low-dimensional vector space representations. This process enables us to more effectively use graph data in machine learning and deep learning algorithms. Graph embedding is a type of graph neural network that iteratively updates node representations through graph convolutional layers, taking advantage of the local neighborhood structure of nodes. The implementation steps include:

[0153] (1) Initialization: Assign an initial feature vector to each node.

[0154] (2) Convolution operation: The information of the node and its neighbors is convolved through the graph convolution layer to update the node representation.

[0155] (3) Output: After multiple layers of convolution operations, the final node vector representation is obtained.

[0156] It should be noted that no matter which method is used, the key to graph embedding is to select appropriate parameters such as the length of random walks, window size, number of layers, etc., and ensure that the model can fully learn the structure and attribute information of the graph. In addition, in order to accelerate the training process, GPU resources or distributed computing frameworks are used.

[0157] In addition, data preprocessing is required to ensure the cleanliness and normalization of graph data and remove isolated points and redundant edges. In addition, the most suitable graph embedding method is selected according to the characteristics of graph data and application scenarios. Appropriate indicators such as link prediction and node classification accuracy are used to evaluate the effect of graph embedding.

[0158] Step S35, using the data corresponding to the target quality rule to fill the audit table corresponding to the target data task.

[0159] It should be noted that an audit form is a report or tool that lists the items or issues that need to be audited in a table for systematic inspection and evaluation.

[0160] Optionally, after determining the target quality rule, the audit table corresponding to the target data task is filled with the data corresponding to the target quality rule, so that the user can configure the quality rule with one click. Developers do not need to manually write cumbersome quality rule scripts, but can automatically generate and apply the corresponding data quality rules with a simple one-click operation. This process greatly simplifies the configuration process and improves work efficiency.

[0161] In this embodiment, by introducing integrated agile development and automated testing tools, data development, online configuration and quality monitoring are integrated into one tool. Through an integrated approach, the compliance, integrity, accuracy and maintainability of the code can be checked with one click in data development; before online deployment, the target data tasks are analyzed for data lineage and task dependency management, the task scheduling time of the target data tasks is determined, and the overall task loading efficiency is guaranteed; in data quality monitoring, a knowledge graph is constructed based on the defined quality rule template, and the audit table corresponding to the target data task is filled based on the knowledge graph matching the target quality rules, the quality rule setting is completed efficiently, the development efficiency is greatly improved, and the online quality is ensured.

[0162] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the big data management method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0163] This application also provides a big data management device, please refer to Figure 2 , the big data management device includes:

[0164] The quality detection module 10 is used to perform quality detection on the code corresponding to the target data task during the development process of the target data task to obtain the code quality detection result;

[0165] The management module 20 is used to perform data lineage analysis and task dependency management on the target data task before deploying and configuring the target data task whose code quality detection result is passed, and determine the task scheduling time of the target data task;

[0166] The monitoring module 30 is used to build a knowledge graph based on the defined quality rule template during data quality monitoring of the target data task after deployment and configuration, and to fill in the audit table corresponding to the target data task based on the knowledge graph matching the target quality rules.

[0167] Optionally, the quality detection module 10 is also used to use a syntax parser to perform a quality detection operation on the code corresponding to the target data task to obtain a code quality detection result, wherein the quality detection operation includes: at least one of a reference entity verification operation, a syntax verification operation, and an optimization and improvement suggestion operation.

[0168] Optionally, the quality detection module 10 is also used to parse the code using a grammar parser, obtain parsing results, and build a grammar rule library; determine whether the entity referenced in the code exists based on the parsing results; and / or compare the parsing results with the grammar rules in the grammar rule library to determine the code location where the grammatical error exists and the type of error code; and / or obtain optimization improvement suggestions corresponding to the code location where the grammatical error exists and the type of error code from the grammar rule library.

[0169] Optionally, the management module 20 is used to analyze the target data task using a syntax parser to determine the predecessor data task on which the target data task depends; and determine the task scheduling time of the target data task according to the task execution data of the predecessor data task.

[0170] Optionally, the management module 20 is also used to obtain the access rights of the preceding data task and the task execution data; if the preceding data task does not have the corresponding access rights, a permission warning message is generated, and a permission application request is generated to request to open the access rights of the preceding data task.

[0171] Optionally, the management module 20 is further used to analyze the task execution data through a time series model to obtain regularity information and cycle information when the preceding data task is executed; and predict the task scheduling time of the target data task based on the regularity information and cycle information.

[0172] Optionally, the monitoring module 30 is also used to define a quality rule template, wherein the quality rule template includes different types of verification rules; perform data configuration on the quality rules in the quality rule template to form a knowledge graph; perform vector representation on each node of the knowledge graph; determine the similarity between the rule to be recommended and the vector representation of each node of the knowledge graph, and determine the target quality rule based on the similarity; and use the data corresponding to the target quality rule to fill the audit table corresponding to the target data task.

[0173] The big data management device provided by the present application adopts the big data management method in the above embodiment, which can improve the efficiency of code development and code launch efficiency. Compared with the prior art, the beneficial effects of the big data management device provided by the present application are the same as the beneficial effects of the big data management method provided by the above embodiment, and other technical features in the big data management device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0174] The present application provides a big data management device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the big data management method in the above-mentioned embodiment one.

[0175] Reference below Figure 3 , which shows a schematic diagram of the structure of a big data management device suitable for implementing the embodiment of the present application. The big data management device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The big data management device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0176] like Figure 3 As shown, the big data management device may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the big data management device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the big data management device to communicate with other devices wirelessly or by wire to exchange data. Although the diagram shows a big data management device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0177] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0178] The big data management device provided by the present application adopts the big data management method in the above embodiment, which can improve the efficiency of code development and code launch. Compared with the prior art, the beneficial effects of the big data management device provided by the present application are the same as the beneficial effects of the big data management method provided by the above embodiment, and the other technical features in the big data management device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0179] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0180] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0181] The present application also provides a computer program product, including a computer program, which implements the steps of the big data management method as described above when executed by a processor.

[0182] The computer program product provided by this application can improve the efficiency of code development and code launch. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the big data management method provided by the above embodiment, which will not be repeated here.

[0183] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for managing big data, characterized in that: The method comprises: During the development of the target data task, the code corresponding to the target data task is quality tested to obtain the code quality test result; Before deploying and configuring the target data task whose code quality detection result is passed, performing data lineage analysis and task dependency management on the target data task to determine the task scheduling time of the target data task; After deployment and configuration, during the data quality monitoring process for the target data task, a knowledge graph is constructed based on the defined quality rule template, and the audit table corresponding to the target data task is filled based on matching the target quality rules with the knowledge graph.

2. The method according to claim 1, characterized in that The code quality detection is performed on the code corresponding to the target data task to obtain the code quality detection result, which includes: A syntax parser is used to perform a quality check operation on the code corresponding to the target data task to obtain the code quality check result, wherein the quality check operation includes: at least one of a reference entity check operation, a syntax check operation, and an optimization and improvement suggestion operation.

3. The method according to claim 2, characterized in that In the process of developing the target data task, the step of using a syntax parser to perform a quality detection operation on the code corresponding to the target data task to obtain the code quality detection result includes: Parsing the code using a grammar parser to obtain parsing results and construct a grammar rule library; Determine, based on the parsing result, whether the entity referenced in the code exists; and / or, comparing the parsing result with the grammar rules in the grammar rule base to determine the code position and error code type of the grammar error; And / or, obtaining optimization improvement suggestions corresponding to the code position with syntax errors and the error code type from the syntax rule library.

4. The method according to claim 1, characterized in that The performing data lineage analysis and task dependency management on the target data task to determine the task scheduling time of the target data task includes: Analyze the target data task using a syntax parser to determine the preceding data task on which the target data task depends; The task scheduling time of the target data task is determined according to the task execution data of the preceding data task.

5. The method according to claim 4, characterized in that After the step of analyzing the target data task by using a syntax parser to determine the preceding data task on which the target data task depends, the method further includes: Obtaining access rights and task execution data for the pre-set data task; If the corresponding access rights do not exist for the preceding data task, a permission warning message is generated, and a permission application request is generated to request to open the access rights for the preceding data task.

6. The method according to claim 4, characterized in that The step of determining the task scheduling time of the target data task according to the task execution data of the preceding data task comprises: Analyze the task execution data through a time series model to obtain regularity information and periodicity information when the preceding data task is executed; The task scheduling time of the target data task is predicted according to the regularity information and the cycle information.

7. The method according to claim 1, characterized in that The step of constructing a knowledge graph based on a defined quality rule template and filling the audit table corresponding to the target data task based on matching the target quality rule with the knowledge graph includes: Defining a quality rule template, wherein the quality rule template includes different types of verification rules; Performing data configuration on the quality rules in the quality rule template to form a knowledge graph; Performing vector representation on each node of the knowledge graph; Determine the similarity between the rule to be recommended and the vector representation of each node of the knowledge graph, and determine the target quality rule according to the similarity; The audit table corresponding to the target data task is filled with the data corresponding to the target quality rule.

8. A big data management device, characterized in that: The device comprises: The quality detection module is used to perform quality detection on the code corresponding to the target data task during the development process of the target data task and obtain the code quality detection result; A management module is used to perform data lineage analysis and task dependency management on the target data task before deploying and configuring the target data task whose code quality detection result is passed, and determine the task scheduling time of the target data task; The monitoring module is used to build a knowledge graph based on the defined quality rule template during the data quality monitoring of the target data task after deployment and configuration, and fill the audit table corresponding to the target data task based on the knowledge graph matching the target quality rules.

9. A big data management device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the big data management method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the big data management method according to any one of claims 1 to 7 are implemented.

Images