Data processing method and system, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and system, a storage medium and electronic equipment, and the method comprises the steps: carrying out the data integration processing of obtained to-be-processed data in a pre-deployed environment, obtaining a data synchronization report, obtaining pre-configured metadata from the data synchronization report, and generating a data quality report according to the metadata and a data blood relationship. According to the scheme, in the aspect of deployment, operation and maintenance, a unified scripted deployment mode is provided for various deployment environments, manual operation configuration during deployment is reduced, the deployment efficiency in the aspect of deployment, operation and maintenance is improved, and rapid implementation and deployment of a lightweight platform are achieved. According to the plug-in type configuration mode of the components, the components and versions thereof can be flexibly selected, and the requirements of the platform for professional technicians are reduced. Besides, through a configuration mode, an application process from basic data to data service is quickly realized, and the integrity, correctness, consistency and availability of the data are ensured, so that a data value is generated.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and more specifically, to a data processing method, system, storage medium and electronic device. Background Art

[0002] As the information age moves towards the data age, many companies have a large amount of data, and the value of data is increasingly valued. Traditional data processing, whether in terms of storage capacity, read and write speed, computing efficiency or cost, is increasingly unable to meet users' needs for processing large amounts of data, and the distributed system infrastructure (Hadoop) came into being.

[0003] Hadoop manages a cluster of hundreds or even thousands of servers, and the computing environment is relatively complex, resulting in high complexity of computing resources, which in turn leads to low deployment efficiency in terms of deployment and operation and maintenance.

[0004] Therefore, how to improve the deployment efficiency in deployment and operation is an urgent problem to be solved in this application. Summary of the invention

[0005] In view of this, the present application discloses a data processing method, system, storage medium and electronic device, which aim to reduce manual operation configuration during deployment, improve deployment efficiency in deployment and operation and maintenance, and realize the rapid implementation and deployment of a lightweight platform.

[0006] In order to achieve the above purpose, the disclosed technical solution is as follows:

[0007] The first aspect of the present application discloses a data processing method, the method comprising:

[0008] In a pre-deployed environment, the acquired data to be processed is subjected to data integration processing to obtain a data synchronization report; the data to be processed represents multi-source data that has not been subjected to data integration processing; the data integration processing is used to verify data consistency, improve data quality and ensure data availability;

[0009] Obtaining pre-configured metadata from the data synchronization report;

[0010] A data quality report is generated based on the metadata and data lineage; the data lineage is determined by the dependency relationship between data scripts and data tasks; the data tasks represent tasks that provide service information that upstream data and downstream data depend on; the data quality report is used to record the integrity of the data, the correctness of the data, the consistency of the same attributes of the same data in different data sets, and whether there is duplication of duplicate data.

[0011] Preferably, the process of environment deployment includes:

[0012] Select a deployment environment, and determine the corresponding service software according to the deployment environment; the service software at least includes a service component and a service version;

[0013] Checking the system environment and generating environment configuration suggestions based on the system environment;

[0014] Perform environment configuration according to the environment configuration suggestions;

[0015] After the environment configuration is completed, the software package corresponding to the service software is installed, and the configuration REST interface is called according to the dependency sequence of the service software and the preset components;

[0016] According to the preset component dependency order, start the configured REST interface, and start all installed components through the configured REST interface;

[0017] After starting all installed components, check the status of the service components. If the status of the service components is normal, the environment deployment is completed.

[0018] Preferably, checking the system environment and generating an environment configuration suggestion based on the system environment includes:

[0019] Establish mutual trust among users based on system IP;

[0020] Under the condition of mutual trust among users, obtain the system hardware, software environment, system information, network information and system variables in the system environment;

[0021] An environmental configuration suggestion is generated according to the system hardware, the software environment, the system information, the network information and the system variables.

[0022] Preferably, it also includes:

[0023] If the state of the service component is in an abnormal state, output configuration information and an abnormal log of the service component in the abnormal state;

[0024] The abnormal point is checked according to the configuration information of the service component in the abnormal state and the abnormal log.

[0025] Preferably, in the pre-deployed environment, the acquired data to be processed is subjected to data integration processing to generate a data synchronization report, including:

[0026] In a pre-deployed environment, determine the data integration method;

[0027] In the data integration mode, determining a target storage for synchronizing the acquired data to be processed;

[0028] Generate a data table according to the data source of the data to be processed, the data integration method and the target storage;

[0029] Create a data synchronization task for the data table, and configure data quality inspection indicators corresponding to the data synchronization task;

[0030] The data quality check is performed using the data quality check indicators to generate a data synchronization report.

[0031] Preferably, the process of configuring metadata includes:

[0032] Check metadata availability;

[0033] If the metadata is available, the metadata is parsed to obtain information; the information represents service information on which the upstream data and the downstream data depend; the information includes at least a data table and a data field;

[0034] Determine data task dependencies;

[0035] Creating a data lineage relationship according to the data script, the data task dependency and the information;

[0036] A data map is generated through the data blood relationship.

[0037] Preferably, it also includes:

[0038] During the data integration process, check the integrity of metadata;

[0039] If the metadata has changed, update the version corresponding to the changed metadata;

[0040] Check if there are any changes in dependent tasks;

[0041] If there are changes in the dependent tasks, the data landscape is updated to generate a new version of data asset information; the data asset information is used to provide unified data sharing services, data usage and data sharing functions.

[0042] A second aspect of the present application discloses a data processing system, the system comprising:

[0043] A processing unit is used to perform data integration processing on the acquired data to be processed in a pre-deployed environment to obtain a data synchronization report; the data to be processed represents multi-source data that has not been subjected to data integration processing; the data integration processing is used to verify data consistency, improve data quality and ensure data availability;

[0044] An acquisition unit, configured to acquire pre-configured metadata from the data synchronization report;

[0045] A generation unit is used to generate a data quality report based on the metadata and data lineage; the data lineage is determined by the dependency relationship between the data script and the data task; the data task represents the task of providing service information on which upstream data and downstream data depend; the data quality report is used to record the integrity of the data, the correctness of the data, the consistency of the same attributes of the same data in different data sets, and whether there is duplication of duplicate data.

[0046] A third aspect of the present application discloses a storage medium, which includes stored instructions, wherein when the instructions are executed, a device where the storage medium is located is controlled to execute the data processing method as described in any one of the first aspects.

[0047] The fourth aspect of the present application discloses an electronic device, comprising a memory and one or more instructions, wherein the one or more instructions are stored in the memory and are configured to be executed by one or more processors to perform a data processing method as described in any one of the first aspects.

[0048] Through the above technical scheme, it can be known that the present application discloses a data processing method, system, storage medium and electronic device. In a pre-deployed environment, the acquired data to be processed is subjected to data integration processing to obtain a data synchronization report. The data to be processed represents multi-source data that has not been subjected to data integration processing. The data integration processing is used to verify data consistency, improve data quality and ensure data availability. The pre-configured metadata is obtained from the data synchronization report. According to the metadata and data lineage, a data quality report is generated. The data lineage is determined by the dependency relationship between the data script and the data task. The data task represents the task of providing service information on which the upstream data and the downstream data depend. The data quality report is used to record the integrity of the data, the correctness of the data, the consistency of the same attributes of the same data in different data sets, and whether there is duplication of duplicate data. Through the above scheme, in terms of deployment and operation, a unified scripted deployment method is provided for a variety of deployment environments, reducing manual operation configuration during deployment, improving deployment efficiency in terms of deployment and operation, and realizing the rapid implementation and deployment of a lightweight platform. According to the component plug-in configuration method, components and their versions can be flexibly selected to reduce the requirements of the platform for professional and technical personnel. In addition, through a configurable approach, the application process from basic data to data services can be quickly implemented to ensure the integrity, correctness, consistency and availability of the data, thereby generating data value. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0050] Figure 1 This is an architecture diagram of the lightweight big data platform disclosed in the embodiments of this application;

[0051] Figure 2 A flowchart of a data processing method disclosed in an embodiment of the present application;

[0052] Figure 3 A schematic diagram of the structure of a data processing system disclosed in an embodiment of the present application;

[0053] Figure 4 A schematic diagram of the structure of an electronic device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0056] As can be seen from the background technology, Hadoop manages a cluster consisting of hundreds or thousands of servers, and the computing environment is relatively complex, which leads to high complexity of computing resources, resulting in low deployment efficiency in deployment and operation. Therefore, how to improve the deployment efficiency in deployment and operation is a problem that needs to be solved urgently in this application.

[0057] In order to solve the above problems, the embodiments of the present application disclose a data processing method, system, storage medium and electronic device. In terms of deployment and maintenance, a unified scripted deployment method is provided for a variety of deployment environments, which reduces manual operation configuration during deployment, improves deployment efficiency in terms of deployment and maintenance, and realizes the rapid implementation and deployment of a lightweight platform. According to the component plug-in configuration method, components and their versions can be flexibly selected to reduce the platform's requirements for professional and technical personnel. In addition, through a configured method, the application process from basic data to data services can be quickly realized to ensure the integrity, correctness, consistency and availability of the data, thereby generating data value. The specific implementation method is illustrated by the following embodiments.

[0058] refer to Figure 1 As shown, it is an architecture diagram of a lightweight big data platform disclosed in an embodiment of the present application. The following data processing method and system are applied to the lightweight big data platform. The lightweight big data platform includes a data application component 11, a data processing component 12, a data computing component 13, a data storage component 14, a data integration component 15 and a platform management component 16.

[0059] The lightweight big data platform provides complete big data core functions. It integrates offline and real-time computing frameworks to meet data needs in data storage, data computing, data query, data modeling, etc. The lightweight big data platform can be used as an independent platform or as a data platform for intelligent supply chains to support the realization of data service value in the supply chain direction.

[0060] The data application component 11 includes a real-time large screen, a logistics control tower, etc.

[0061] The data processing component 12 provides data requirements in terms of data usage, data operations, data governance, and data asset management.

[0062] Specifically, data usage and data operations are implemented through data development workbenches, data labeling tools, interactive queries (ad hoc queries / interactive data queries (OLAP)) and business intelligence (BI) large screen tools. Data operations can support OLAP functions as well as other data calculation functions.

[0063] Data computing supports various computing methods commonly used in the industry, such as batch processing, stream processing, batch-stream integrated processing, etc. It provides computing engines for data warehouse Hive (programming model (MapReduce, MR), open source computing framework Tez), computing framework Spark, computing framework Flink and other data computing frameworks. The system provides a data development workbench, which can directly submit database (StructuredQueryLanguage, SQL) jobs for data computing and realize OLAP functions. Through a visual development process, offline data computing can realize functions such as data exploration, data development, data script saving and submission, and historical execution task viewing. Offline data computing can access various storage types such as distributed storage HDFS and object storage OSS.

[0064] Real-time data computing uses the computing model FlinkSQL to develop real-time data, meeting common data development methods. For complex real-time data development scenarios, you can submit a FlinkJar package to complete real-time data development. Real-time data computing supports storage methods such as Kafka, Hudi, Hbase, and ES. In real-time computing development, the highly available Flink cluster task gateway (FlinkSQLGateway) converts the FlinkSQL submitted by the user on the platform into a FlinkJob. After the job is submitted, the resource manager Yarn manages the resources. In terms of resource management, users bind tenants, apply for the required memory, central processing unit (CentralProcessingUnit, CPU) and other computing resources, bind to the user's project, limit data computing resources, and ensure isolation from offline computing resources to ensure the normal operation of data computing tasks.

[0065] The lightweight big data platform can configure scheduling tasks and implement the scheduling development of data tasks through a low-code drag-and-drop method. The system provides full-stack visual programming for data development, covering the entire life cycle of data development, including data exploration, data design, development, testing, online, operation and maintenance, etc. Non-professional developers can quickly get started with data development. In the data development stage, you can select the tables and fields that need to be calculated through the synchronized metadata, and perform data calculations according to the rules. The system will generate the corresponding data calculation SQL to simplify the data development process. In the workflow configuration stage, you can select the saved data calculation SQL to generate a workflow for data calculation, which supports functions such as data calculation and data push.

[0066] The lightweight big data platform can import data reports into the BI system and support BI large-screen display. The data labeling tool brings together multi-source data. After data calculation, integration and processing, it supports adding data label annotation functions to the data layer through visual operations, building a data label portrait system, and integrating data assets. The system can group and profile users based on labels to realize data value more quickly. The lightweight big data platform can also provide the core capabilities of big data on a miniaturized platform, which can help companies lower the threshold and cost of using data, make full use of data, and make data use a skill that companies can easily acquire. By quickly verifying the value of data through the lightweight big data platform, companies no longer have to worry about the cost of trial and error in the direction of big data, and improve the commercial value of big data projects.

[0067] Data governance and data asset management include metadata management, data quality, lineage management, data asset management, data landscape visualization, data lifecycle management, etc.

[0068] Data asset management is a set of business functions for planning, controlling and providing data and information assets, including developing, executing and supervising plans, policies, programs, projects, processes, methods and procedures related to data, so as to control, protect, deliver and improve the value of data assets. Through the metadata configured in data integration, data lineage relationships are automatically generated according to data operations. For the data quality reports generated by data integration tasks, users can improve data quality.

[0069] The lightweight big data platform provides a unified visual data catalog and metadata management method to improve the value of data assets. It can automatically generate data lineage relationships based on data operations, provide data asset management capabilities, and facilitate data query and use. Through the visual data asset catalog construction process, it supports data asset labeling and provides data with labels that are easier for businesses to understand to organize and define data assets. The data asset catalog provides a unified data sharing service for generating data service functions to facilitate data use and sharing.

[0070] The value of data is affected by two factors: data quality and data application. The intrinsic value of data is the embodiment of the value of data quality. The evaluation dimensions of the intrinsic value of data include data integrity, correctness, consistency, and repeatability. The lightweight big data platform provides a data quality report to record whether there are missing records or missing fields (completeness), whether the data conforms to a specific data format or value range (correctness), whether the same attributes of the same data entity are consistent in different data sets (consistency), and whether there is duplicate data (repeatability).

[0071] The data computing component 13 includes batch processing, streaming computing, permission center (multi-tenant), data security / data auditing, permission security and other functions.

[0072] Among them, permission security:

[0073] The lightweight big data platform provides a secure and flexible access control mechanism to achieve secure isolation of platform services, isolate big data cluster services and user development platforms, and fully ensure data security in the system. Through multi-tenant management, virtual resources between tenants are completely isolated; sub-account permission control enables refined management of account permissions, and permissions can be set for database tables, partitions, and fields to ensure data security;

[0074] The operation log records the user's operation behavior on resources and information about logging in and out of the platform, provides operation record query and screening functions, and realizes compliance audit, data security analysis, resource change tracking and other functions of the lightweight big data platform.

[0075] The data storage component 14 includes distributed intelligent storage such as file storage, column storage, and object storage; the data storage component 14 also includes functions such as intelligent compression and multi-cloud storage. Intelligent compression and multi-cloud storage can improve storage efficiency.

[0076] The lightweight big data platform supports distributed storage solutions such as HDFS file storage, column storage, and object storage in data storage. It supports access and interoperability of multiple data storages, meets the needs of cross-domain data access, reduces data storage redundancy, and reduces data storage costs. It adds small file detection functions to storage, optimizes small file merging strategies, and improves the availability and utilization of data storage.

[0077] The lightweight big data platform supports data fusion of multi-source heterogeneous data, which can eliminate data silos and realize data heterogeneous systems and cross-organization applications. Through system configuration, users do not need to pay attention to the specific storage type of data and can access data conveniently and quickly. For data stored across clouds, data storage and usage can be checked, and data with high usage frequency will be migrated to reduce network overhead during use.

[0078] The lightweight big data platform checks the stored data and issues an early warning for uncompressed data, reminding users to compress the data. Users can compress data with one click to reduce data storage pressure. As the number of jobs and data volume increases, the number of small files will also increase rapidly. Too many small files will increase the pressure on HDFS metadata and increase the number of read requests when the computing engine reads, thus affecting data reading efficiency and reducing platform availability. The lightweight big data platform adds a small file detection function, configures file inspection strategies and file merging strategies, and regularly organizes small files according to strategy requirements.

[0079] The lightweight big data platform can provide hot and cold data storage clusters. The system uses the execution records of data tasks to determine whether the data is hot or cold. For hot data, the standard HDFS default 3-copy strategy is adopted. For cold data, the erasure code (EC) is used to replace the copy strategy. The erasure code provides the same fault tolerance as the copy strategy while reducing the space for data storage. If cold data is frequently called, it will be migrated to the hot data area to speed up data calculation. For temporary files generated during data calculation, different levels of temporary directory recycling strategies are provided, and users can clean up temporary data according to the recycling strategy.

[0080] The data integration component 15 includes access mode management and access data source management. The access mode management includes offline data synchronization, real-time data synchronization, and API access functions; the access data source management includes multi-source support, whole database migration, and one-click access functions.

[0081] The platform management component 16 includes functions such as resource management, component management, and operation and maintenance automation. Among them, resource management is implemented through Docker / multi-cloud / hybrid cloud / IDC physical machine, etc.; component management is implemented through Hadoop ecosystem component / directory configuration, etc.; and operation and maintenance automation is implemented through operation and maintenance dashboard / monitoring warning / deployment, etc.

[0082] The specific interaction process between the data processing component 12, the data integration component 15 and the platform management component 16 is as follows:

[0083] The data integration component 15 performs data integration processing on the acquired data to be processed in a pre-deployed environment to obtain a data synchronization report; the data to be processed represents multi-source data that has not been subjected to data integration processing; data integration processing is used to verify data consistency, improve data quality and ensure data availability.

[0084] Among them, data integration processing includes data integration functions, supporting offline synchronization, real-time synchronization, API synchronization and other methods; data access supports rich data sources, including structured data, semi-structured data, unstructured data, etc. Offline synchronization supports relational database management system MySQL, database Oracle, open source distributed relational database TiDB, etc. Real-time synchronization supports distributed publish-subscribe messaging system Kafka, binary format log file MySQLBinlog, analysis tool Oracle Logminer, etc.

[0085] Functional integration provides one-click access, whole database migration, offline + real-time synchronization integration and other functions, which facilitates the rapid import of enterprise data in different storages into the lightweight big data platform. When heterogeneous systems perform data integration, dirty data will inevitably be generated. Error data such as type conversion errors, null pointers, and primary key conflicts that affect synchronization tasks will be written into the dirty data. For huge amounts of data, flow control management can be used to limit the speed of data reading and writing to avoid excessive load on the business database. A wealth of monitoring indicators are provided in the data integration task to verify data consistency in real time and ensure data quality.

[0086] The specific process of deploying the data integration component 15 environment is shown in (1) to (12):

[0087] (1) The lightweight big data platform provides the dependencies and configurations of all versions of Hadoop ecosystem components, and provides a set of standard REST interfaces for all components. The interfaces support functions such as deployment, configuration, start, stop, and status monitoring. All components can be deployed and configured based on the Representational State Transfer (REST) ​​interface.

[0088] Among them, all components include data storage HDFS, resource management (YARN), open source distributed resource management framework (Mesos), data warehouse (Hive), general distributed parallel computing framework (Spark), distributed processing engine (Flink), data stream Kafka, data storage (HBase), scheduling (dolphinscheduler), data governance (Knox), coordination service (zookeeper), etc.

[0089] (2) Select a deployment environment and provide corresponding service software dependencies according to different environments.

[0090] (3) Check the system environment and generate environmental configuration suggestions.

[0091] Among them, environmental configuration recommendations include host name recommendations, matching of network information and host information, firewall check recommendations, system SELinux, transparent large pages and other kernel parameters, Java language software development kit (JDK), Python version, database version, YUM source recommendations, etc.

[0092] (4) Configure the environment according to the environment configuration recommendations to ensure that the basic environment is stable and available. After the configuration is completed, check the relevant system environment status, network environment, etc.

[0093] (5) Select the service components and versions to be installed, which can adapt to the version dependencies of the user's existing components to ensure the availability of services (various service systems deployed on the lightweight big data platform).

[0094] (6) One-click deployment based on the selected service components and installation of related software packages.

[0095] (7) Based on the selected components, call the configuration REST interface to configure the service in the order of preset component dependencies.

[0096] Among them, the preset component dependency order means configuring the data storage HDFS first, then configuring resource management, and then configuring the data warehouse, data computing, data flow, data storage, scheduling, data governance, etc. in sequence.

[0097] Data storage and resource management are required items and are managed first. The order of other components can be freely controlled according to specific selections. If no selection is made, a default order is provided according to the above results.

[0098] (8) Call the startup REST interface in the order of preset component dependencies to start all installed components.

[0099] (9) After the component is started, the status monitoring REST interface is called to check the status of the service components. If all components are in normal status, the entire process deployment is completed.

[0100] (10) If there is an abnormality in component status monitoring, output the component configuration information and abnormality log.

[0101] (11) The operation and maintenance personnel review the output configuration information and exception logs to identify abnormal points.

[0102] (12) Call the startup REST interface to start the abnormal service. After completion, call the status monitoring REST interface and recheck according to the status.

[0103] In terms of platform operation and maintenance, a visual monitoring and early warning platform is provided, which can monitor the status of hosts, components, systems and other dimensions in real time. For abnormal service status, real-time early warning can be provided through SMS, email, phone and other means, which facilitates operation and maintenance personnel to quickly handle abnormal problems and ensure the stability and reliability of the system.

[0104] The data integration processing flow of the data integration component 15 facilitates developers to quickly import data into the lightweight big data platform through a visual operation flow. The specific flow is shown in 1) to 8):

[0105] 1) Select the data source to be imported and configure the basic information of the data source, such as database connection, user name, password, etc. The system will check the availability of the data source and synchronize the metadata of the data source after completion to facilitate subsequent processes.

[0106] The metadata includes the data source, which includes the data table information, field information in the data table, field type, field description, etc.

[0107] 2) Select the data integration method, such as offline, real-time, etc. Offline synchronization will create an offline task, which requires the configuration of a scheduled task. Real-time synchronization does not require the configuration of a scheduled task.

[0108] 3) Select the target storage that needs to be synchronized for data integration.

[0109] 4) Generate the data tables that need to be created based on the data source, synchronization method, target storage and other configurations of data integration (such as offline synchronization to Hive, create Hive data tables; if real-time synchronization of data to Kafka, create real-time tables with Kafka as storage).

[0110] 5) Synchronize data with one click to create data synchronization tasks for all tables. Or select the tables to be synchronized and create data synchronization tasks for some tables.

[0111] 6) Configure data quality check indicators for data synchronization tasks. After data synchronization is completed, data quality checks will be performed according to the indicators, and tasks with abnormal data quality will be warned.

[0112] 7) Real-time synchronization can start synchronization tasks, while offline synchronization requires the configuration of scheduled tasks.

[0113] 8) After data synchronization is completed, perform a data quality check and generate a data synchronization report. For abnormal indicators, developers check and repair them to improve data quality and ensure data availability.

[0114] The data processing component 12 obtains pre-configured metadata from the data synchronization report obtained by the data integration component 15 .

[0115] The configuration process of metadata and data assets is shown in steps A and B:

[0116] Step A:

[0117] First, configure the data source in Data Integration and check the data source availability.

[0118] Secondly, parse the data source metadata to generate data tables, data fields and other information.

[0119] Then, check the data task dependencies and create data lineage relationships based on the data scripts.

[0120] Among them, the data task can provide service information of upstream and downstream dependencies, and the current node can obtain the ID and data script of the upstream data service.

[0121] Finally, generate the data map.

[0122] Step B:

[0123] First, metadata is checked periodically to check the integrity of the metadata.

[0124] Secondly, if the metadata changes, the corresponding version of the metadata needs to be updated.

[0125] Then, check whether the dependent tasks have changed and update the version of the change metadata.

[0126] Finally, for any changed information, update the data map and generate a new version of the data asset information.

[0127] The data processing component 12 generates a data quality report based on metadata and data lineage; data lineage is determined by the dependency between data scripts and data tasks; data tasks represent tasks that provide service information that upstream data and downstream data depend on; data quality reports are used to record data integrity, data correctness, consistency of the same attributes of the same data in different data sets, and whether there is duplication of duplicate data.

[0128] Platform Management Component 16 In terms of data platform management, the lightweight big data platform can be deployed in a variety of environments such as physical machines, cloud environments, virtual machines, Docker, etc., to meet the company's needs for big data infrastructure environments in different business scenarios. The Hadoop ecosystem component provides multiple versions of directory configurations, and you can choose a version that is compatible with existing system components to ensure that the system is stable and available. At the same time, it supports plug-in configurable deployment methods, and adds a unified REST interface to the components to meet the needs of operation and maintenance deployment such as deployment, configuration, startup, and monitoring, thereby improving the platform's deployment performance. For a variety of deployment environments, a unified scripted deployment method is provided to reduce manual configuration during deployment, greatly improving deployment efficiency.

[0129] This solution implements the construction method of the entire process from data integration, data computing, data storage to data services, builds an enterprise-level big data platform with a distributed computing architecture based on Hadoop, provides standardized and reusable system components, and provides a more economical and efficient solution for small and medium-sized big data platforms.

[0130] In terms of deployment and operation, a unified scripted deployment method is provided for various deployment environments, which reduces manual configuration during deployment, greatly improves deployment efficiency, and realizes the rapid implementation and deployment of lightweight platforms. According to the component plug-in configuration method, components and their versions can be flexibly selected, which reduces the platform's requirements for professional technicians. Enterprises no longer need professional technicians to deploy and implement.

[0131] In terms of data application development, developers can quickly implement the application process from basic data to data services in a configurable manner, ensure the timeliness of data, and generate data value as soon as possible. In terms of functions, it provides complete system functions, integrates a variety of components and their versions, and can adapt to common business scenario requirements.

[0132] In the embodiments of the present application, in terms of deployment and maintenance, a unified scripted deployment method is provided for a variety of deployment environments, which reduces manual operation configuration during deployment, improves deployment efficiency in terms of deployment and maintenance, and realizes the rapid implementation and deployment of a lightweight platform. According to the component plug-in configuration method, components and their versions can be flexibly selected to reduce the platform's requirements for professional and technical personnel. In addition, through a configured method, the application process from basic data to data services can be quickly realized to ensure the integrity, correctness, consistency and availability of the data, thereby generating data value.

[0133] refer to Figure 2 FIG. 1 is a flow chart of a data processing method disclosed in an embodiment of the present application, and the data processing method is applied to the above-mentioned embodiment. Figure 1 A lightweight big data platform, the data processing method mainly includes the following steps:

[0134] S201: In a pre-deployed environment, the acquired data to be processed is subjected to data integration processing to obtain a data synchronization report; the data to be processed represents multi-source data that has not been subjected to data integration processing; the data integration processing is used to verify data consistency, improve data quality and ensure data availability.

[0135] Among them, the environment includes cloud environment, physical machine, Docker deployment and other environments.

[0136] Data integration processing provides functions such as one-click access, whole database migration, offline + real-time synchronization integration, etc., which facilitates the rapid import of enterprise data in different storages into the lightweight big data platform. When heterogeneous systems perform data integration, dirty data will inevitably be generated. Error data such as type conversion errors, null pointers, and primary key conflicts that affect synchronization tasks will be written into the dirty data. For data with huge amounts of data, flow control management can be used to limit the speed of data reading and writing to avoid excessive load on the business database. A wealth of monitoring indicators are provided in the data integration task to verify data consistency in real time and ensure data quality.

[0137] Specifically, in a pre-deployed environment, the acquired data to be processed is subjected to data integration processing to obtain a data synchronization report as shown in A1-A5.

[0138] A1: Determine the data integration method in a pre-deployed environment.

[0139] Among them, data integration methods include offline and real-time methods.

[0140] A2: In the data integration mode, determine the target storage for synchronizing the acquired data to be processed.

[0141] Among them, target storage includes Hive storage, Kafka storage, etc.

[0142] A3: Generate a data table based on the data source, data integration method, and target storage of the data to be processed.

[0143] For example, if you synchronize data to a data warehouse tool (Hive) offline, create a Hive data table; if you synchronize data to Kafka in real time, create a real-time table for Kafka storage.

[0144] A4: Create a data synchronization task for the data table and configure the data quality check indicators corresponding to the data synchronization task.

[0145] Among them, data quality inspection indicators are configured for data synchronization tasks. After data synchronization is completed, data quality inspection will be performed according to the indicators, and tasks with abnormal data quality will be warned.

[0146] A5: Perform data quality checks using data quality check indicators and generate data synchronization reports.

[0147] After data synchronization is completed, a data quality check is performed and a data synchronization report is generated. For abnormal indicators, developers check and repair them to improve data quality and ensure data availability.

[0148] The specific process of environment deployment is shown in B1-B6.

[0149] B1: Select a deployment environment and determine the corresponding service software based on the deployment environment; the service software at least includes service components and service versions.

[0150] B2: Check the system environment and generate environment configuration suggestions based on the system environment.

[0151] Among them, environmental configuration recommendations include host name recommendations, matching of network information and host information, firewall check recommendations, system SELinux, limits, transparent large pages and other kernel parameters, JDK, Python version, database version, YUM source recommendations, etc.

[0152] Specifically check the system environment and generate the following environment configuration suggestions based on the system environment:

[0153] First, establish user mutual trust based on system IP.

[0154] Among them, user mutual trust refers to a cross-system user mutual trust authentication method, which facilitates deployment and operation.

[0155] Then, under the condition of mutual trust among users, the system hardware, software environment, system information, network information and system variables in the system environment are obtained.

[0156] Finally, environmental configuration recommendations are generated based on system hardware, software environment, system information, network information, and system variables.

[0157] B3: Configure the environment according to the recommended environment configuration.

[0158] Among them, configure the environment according to the environment configuration recommendations to ensure that the basic environment is stable and available. After the configuration is completed, check the relevant system environment status, network environment, etc.

[0159] B4: After the environment configuration is completed, install the software package corresponding to the service software, and call the configuration REST interface according to the service software and the preset component dependency sequence.

[0160] Among them, the preset component dependency order is to configure the data storage HDFS first, then configure resource management, and then configure the data warehouse, data computing, data flow, data storage, scheduling, data governance, etc.

[0161] Data storage and resource management are required items and should be managed first. The order of other components can be freely controlled according to specific selections. If no selection is made, the default order provided above can be followed.

[0162] B5: According to the preset component dependency order, start the configured REST interface, and start all installed components through the configured REST interface.

[0163] B6: After starting all installed components, check the status of the service components. If the status of the service components is normal, the environment deployment is completed.

[0164] If the status of the service component is in an abnormal state, the configuration information and the abnormal log of the service component in the abnormal state are output, and the abnormal point is checked according to the configuration information and the abnormal log of the service component in the abnormal state.

[0165] S202: Acquire pre-configured metadata from the data synchronization report.

[0166] Among them, metadata includes data source, data table information, field information in the data table, field type, field description, etc.

[0167] The process of configuring metadata is shown in C1-C5.

[0168] C1: Check metadata availability.

[0169] C2: If metadata is available, parse the metadata to obtain information; the information represents service information on which upstream data and downstream data depend; the information includes at least data tables and data fields.

[0170] C3: Determine data-task dependencies.

[0171] Among them, the data task dependency relationship can provide the service information of upstream and downstream dependencies, and the current node can obtain the ID of the upstream data service and the dependency relationship of the data script.

[0172] C4: Create data lineage based on data scripts, data task dependencies, and information.

[0173] Among them, data lineage refers to the relationship formed between data during the process of data generation, processing, circulation and extinction, which is similar to the blood relationship in human society.

[0174] C5: Generate data map through data blood relationship.

[0175] Among them, the data map is used to facilitate users to view what data is available.

[0176] During the data integration process, check the integrity of the metadata. If there are any changes in the metadata, update the version corresponding to the changed metadata.

[0177] Check whether there are any changes in dependent tasks. If there are any changes in dependent tasks, update the data map and generate a new version of data asset information; data asset information is used to provide unified data sharing services, data usage and data sharing functions.

[0178] Among them, dependent tasks are used to generate data maps.

[0179] S203: Generate a data quality report based on metadata and data lineage; data lineage is determined by the dependency between data scripts and data tasks; data tasks represent tasks that provide service information that upstream data and downstream data depend on; data quality reports are used to record data integrity, data correctness, consistency of the same attributes of the same data in different data sets, and whether there is duplication of duplicate data.

[0180] Among them, the data quality report records whether there are missing records or missing fields (completeness), whether the data conforms to a specific data format or value range (correctness), whether the same attributes of the same data entity are consistent in different data sets (consistency), and whether there is duplicate data (repeatability).

[0181] In the embodiments of the present application, in terms of deployment and maintenance, a unified scripted deployment method is provided for a variety of deployment environments, which reduces manual operation configuration during deployment, improves deployment efficiency in terms of deployment and maintenance, and realizes the rapid implementation and deployment of a lightweight platform. According to the component plug-in configuration method, components and their versions can be flexibly selected to reduce the platform's requirements for professional and technical personnel. In addition, through a configured method, the application process from basic data to data services can be quickly realized to ensure the integrity, correctness, consistency and availability of the data, thereby generating data value.

[0182] Based on the above embodiments Figure 2 A data processing method disclosed in the present application is also disclosed in the embodiment of the present application. Figure 3 As shown, the data processing system includes a processing unit 301 , an acquisition unit 302 and a generation unit 303 .

[0183] The processing unit 301 is used to perform data integration processing on the acquired data to be processed in a pre-deployed environment to obtain a data synchronization report; the data to be processed represents multi-source data that has not been processed by data integration; the data integration processing is used to verify data consistency, improve data quality and ensure data availability.

[0184] The acquisition unit 302 is used to acquire pre-configured metadata from the data synchronization report.

[0185] The generation unit 303 is used to generate a data quality report based on metadata and data lineage; the data lineage is determined by the dependency relationship between data scripts and data tasks; the data task represents the task of providing service information on which upstream data and downstream data depend; the data quality report is used to record the integrity of the data, the correctness of the data, the consistency of the same attributes of the same data in different data sets, and whether there is duplication of duplicate data.

[0186] Furthermore, the processing unit 301 of the environment deployment process includes:

[0187] The first determination module is used to select a deployment environment and determine the corresponding service software according to the deployment environment; the service software at least includes a service component and a service version.

[0188] The first generation module is used to check the system environment and generate environment configuration suggestions based on the system environment.

[0189] The first configuration module is used to perform environment configuration according to the environment configuration suggestion.

[0190] The calling module is used to install the software package corresponding to the service software after the environment configuration is completed, and call the configuration REST interface according to the service software and the preset component dependency sequence.

[0191] The startup module is used to start the configured REST interface according to the preset component dependency sequence, and start all installed components through the configured REST interface.

[0192] The first checking module is used to check the status of the service component after starting all installed components. If the status of the service component is normal, the environment deployment is completed.

[0193] Furthermore, the generating unit 302 includes:

[0194] Establish a module for establishing user mutual trust based on system IP.

[0195] The acquisition module is used to obtain system hardware, software environment, system information, network information and system variables in the system environment under the condition of mutual trust between users.

[0196] The second generation module is used to generate environmental configuration suggestions based on system hardware, software environment, system information, network information and system variables.

[0197] Furthermore, the data processing system also includes an output unit and a troubleshooting unit.

[0198] The output unit is used to output the configuration information and the abnormal log of the service component in the abnormal state if the state of the service component is in the abnormal state.

[0199] The troubleshooting unit is used to troubleshoot abnormal points based on the configuration information and abnormal logs of the service components in the abnormal state.

[0200] Furthermore, the generating unit 303 includes:

[0201] The second determination module is used to determine the data integration method in a pre-deployed environment.

[0202] The third determination module is used to determine the target storage for synchronization of the acquired data to be processed in a data integration mode.

[0203] The third generation module is used to generate a data table according to the data source, data integration method and target storage of the data to be processed.

[0204] The second configuration module is used to create a data synchronization task for a data table and configure data quality inspection indicators corresponding to the data synchronization task.

[0205] The fourth generation module is used to perform data quality inspection through data quality inspection indicators and generate a data synchronization report.

[0206] Furthermore, the generating unit 302 of the process of configuring metadata includes:

[0207] Check module, used to check metadata availability;

[0208] The parsing module is used to parse the metadata to obtain information if the metadata is available; the information represents the service information on which the upstream data and the downstream data depend; the information includes at least a data table and a data field.

[0209] The fourth determination module is used to determine data task dependencies.

[0210] A creation module is used to create data lineage relationships based on data scripts, data task dependencies, and information.

[0211] The fifth generation module is used to generate a data map through data blood relationship.

[0212] Furthermore, the data processing system also includes:

[0213] The first checking unit is used to check the integrity of the metadata during the data integration process.

[0214] The update unit is used to update the version corresponding to the changed metadata if there is a change in the metadata.

[0215] The second checking unit is used to check whether there is any change in the dependent task.

[0216] The update generation unit is used to update the data map and generate a new version of data asset information if there are changes in dependent tasks; the data asset information is used to provide unified data sharing services, data usage and data sharing functions.

[0217] In the embodiments of the present application, in terms of deployment and maintenance, a unified scripted deployment method is provided for a variety of deployment environments, which reduces manual operation configuration during deployment, improves deployment efficiency in terms of deployment and maintenance, and realizes the rapid implementation and deployment of a lightweight platform. According to the component plug-in configuration method, components and their versions can be flexibly selected to reduce the platform's requirements for professional and technical personnel. In addition, through a configured method, the application process from basic data to data services can be quickly realized to ensure the integrity, correctness, consistency and availability of the data, thereby generating data value.

[0218] An embodiment of the present application further provides a storage medium, which includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the above-mentioned data processing method.

[0219] The present application also provides an electronic device, whose structural diagram is as follows: Figure 4 As shown, it specifically includes a memory 401 and one or more instructions 402, wherein the one or more instructions 402 are stored in the memory 401 and are configured to be executed by one or more processors 403 to execute the one or more instructions 402 to perform the above-mentioned data processing method.

[0220] The specific implementation processes and derivative methods of the above-mentioned embodiments are all within the protection scope of this application.

[0221] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as clustering components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without creative work.

[0222] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0223] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0224] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, It is characterized in that The method comprises: In a pre-deployed environment, the acquired data to be processed is subjected to data integration processing to obtain a data synchronization report; the data to be processed represents multi-source data that has not been subjected to data integration processing; the data integration processing is used to verify data consistency, improve data quality and ensure data availability; Obtaining pre-configured metadata from the data synchronization report; A data quality report is generated based on the metadata and data lineage; the data lineage is determined by the dependency relationship between data scripts and data tasks; the data tasks represent tasks that provide service information that upstream data and downstream data depend on; the data quality report is used to record the integrity of the data, the correctness of the data, the consistency of the same attributes of the same data in different data sets, and whether there is duplication of duplicate data.

2. The method according to claim 1, It is characterized in that The process of environment deployment includes: Select a deployment environment, and determine the corresponding service software according to the deployment environment; the service software at least includes a service component and a service version; Checking the system environment and generating environment configuration suggestions based on the system environment; Perform environment configuration according to the environment configuration suggestions; After the environment configuration is completed, the software package corresponding to the service software is installed, and the configuration REST interface is called according to the dependency sequence of the service software and the preset components; According to the preset component dependency order, start the configured REST interface, and start all installed components through the configured REST interface; After starting all installed components, check the status of the service components. If the status of the service components is normal, the environment deployment is completed.

3. The method according to claim 2, It is characterized in that The checking of the system environment and generating an environment configuration suggestion based on the system environment includes: Establish mutual trust among users based on system IP; Under the condition of mutual trust among users, obtain the system hardware, software environment, system information, network information and system variables in the system environment; An environmental configuration suggestion is generated according to the system hardware, the software environment, the system information, the network information and the system variables.

4. The method according to claim 2, It is characterized in that Also includes: If the state of the service component is in an abnormal state, output configuration information and an abnormal log of the service component in the abnormal state; The abnormal point is checked according to the configuration information of the service component in the abnormal state and the abnormal log.

5. The method according to claim 1, It is characterized in that In the pre-deployed environment, the acquired data to be processed is processed for data integration and a data synchronization report is generated, including: In a pre-deployed environment, determine the data integration method; In the data integration mode, determining a target storage for synchronizing the acquired data to be processed; Generate a data table according to the data source of the data to be processed, the data integration method and the target storage; Create a data synchronization task for the data table, and configure data quality inspection indicators corresponding to the data synchronization task; The data quality check is performed using the data quality check indicators to generate a data synchronization report.

6. The method according to claim 1, It is characterized in that The process of configuring metadata includes: Check metadata availability; If the metadata is available, the metadata is parsed to obtain information; the information represents service information on which the upstream data and the downstream data depend; the information includes at least a data table and a data field; Determine data task dependencies; Creating a data lineage relationship according to the data script, the data task dependency and the information; A data map is generated through the data blood relationship.

7. The method according to claim 6, It is characterized in that Also includes: During the data integration process, check the integrity of metadata; If the metadata has changed, update the version corresponding to the changed metadata; Check if there are any changes in dependent tasks; If there is a change in the dependent task, update the data map and generate a new version of data asset information; The data asset information is used to provide unified data sharing services, data usage and data sharing functions.

8. A data processing system, It is characterized in that The system comprises: A processing unit is used to perform data integration processing on the acquired data to be processed in a pre-deployed environment to obtain a data synchronization report; the data to be processed represents multi-source data that has not been subjected to data integration processing; the data integration processing is used to verify data consistency, improve data quality and ensure data availability; An acquisition unit, configured to acquire pre-configured metadata from the data synchronization report; A generation unit is used to generate a data quality report based on the metadata and data lineage; the data lineage is determined by the dependency relationship between the data script and the data task; the data task represents the task of providing service information on which upstream data and downstream data depend; the data quality report is used to record the integrity of the data, the correctness of the data, the consistency of the same attributes of the same data in different data sets, and whether there is duplication of duplicate data.

9. A storage medium, It is characterized in that The storage medium includes stored instructions, wherein when the instructions are executed, the device where the storage medium is located is controlled to execute the data processing method according to any one of claims 1 to 7.

10. An electronic device, It is characterized in that The device comprises a memory and one or more instructions, wherein the one or more instructions are stored in the memory and configured to be executed by one or more processors to perform the data processing method according to any one of claims 1 to 7.

Images