Big data multi-cluster data sharing and transfer methods, devices, equipment and media

Abstract

This application discloses a data sharing and transfer method, apparatus, device, and medium for large-scale multi-cluster data. In this application, after receiving data sharing and transfer request information from the source end, and obtaining the destination end identifier information and the data file from the source end carried in the data sharing and transfer request information, in a data exchange business scenario, the data file is sent to the destination end according to the destination end identifier information, thereby realizing a data copy transfer mode; in a data sharing business scenario, a shared view is created based on the data file, and the shared view is sent to the destination end, thereby realizing a view sharing transfer mode; and after receiving data loading request information from the destination end, the data file obtained by the destination end is loaded into the database, thereby realizing a data loading transfer mode. This solves the problem that existing data transmission schemes only address a single scenario and lack other data transfer modes such as data loading and view sharing.

Description

Technical Field

[0001] This application relates to the field of computer data transmission technology, and in particular to methods, apparatus, equipment and media for data sharing and transfer in big data multi-cluster systems. Background Technology

[0002] With the development and promotion of big data technology, the value of big data has been widely recognized. Today, with the increasing prevalence of big data, more and more users are analyzing and modeling the massive amounts of data stored in big data clusters to uncover the intrinsic value of that data.

[0003] The rapid growth of data and the proliferation of applications have rendered the early single big data cluster model increasingly inadequate for handling complex business scenarios. Existing data transmission solutions primarily focus on the implementation and optimization of data synchronization. By employing techniques such as master-slave clustering or optimizing data storage structures, they can initiate and monitor point-to-point data synchronization between clusters, fulfilling some data flow requirements. However, these solutions are mostly limited to single scenarios and lack other data flow modes such as data loading and view sharing; they are merely simple data copying solutions. Summary of the Invention

[0004] This application provides a data sharing and transfer method, apparatus, device, and medium for big data multi-cluster data transfer, which solves the problem that existing data transfer schemes are only for a single scenario and lack other data transfer modes such as data loading and view sharing.

[0005] This application provides a method for data sharing and transfer across multiple big data clusters, the method comprising:

[0006] Receive data sharing and transfer request information sent by the source end, and obtain the destination end identification information and the data file of the source end carried in the data sharing and transfer request information;

[0007] In a data exchange scenario, the data file is sent to the destination based on the destination identifier information; in a data sharing scenario, a shared view is created based on the data file, and the shared view is sent to the destination based on the destination identifier information, so that the destination can obtain the data file based on the shared view.

[0008] The system receives a data loading request from the destination and loads the data file obtained by the destination into the database, so that the tenant can use the database to perform data queries based on the content of the data file.

[0009] Further, receiving the data sharing transfer request information sent by the source end, and obtaining the destination end identification information and the data file of the source end carried in the data sharing transfer request information includes:

[0010] By polling at intervals to check if the data ready flag file exists, it is determined whether the data file of the source end is ready. If it is, the data sharing transfer request information sent by the source end is received, and the destination end identification information and the data file of the source end carried in the data sharing transfer request information are obtained.

[0011] Furthermore, the method also includes:

[0012] Determine whether the current business scenario is one where the execution of downstream tasks depends on the readiness of upstream data. If so, when the data file at the source end is not ready, no error processing is performed, and the destination end is instructed to continue executing the business.

[0013] Furthermore, the method also includes:

[0014] The data processing method is determined based on the source data type and data location, as well as the destination data type and data location; and the data sharing and transfer processing is performed according to the operator task corresponding to the data processing method.

[0015] Furthermore, the method also includes:

[0016] When performing a data synchronization task, the partition field information of the database is obtained and compared with the field information specified by the data synchronization task. If the database is a multi-level partition and the field information specified by the data synchronization task is not the top-level partition field information, then multi-level partition synchronization processing is performed.

[0017] Furthermore, the multi-level partition synchronization process includes:

[0018] Generate a filter configuration file, in which the partition filter information to be synchronized is written using reverse regular expression matching; perform multi-level partition synchronization processing based on the filter configuration file.

[0019] Furthermore, the method also includes:

[0020] When the data takes effect, a deep search traversal of the file path is performed on the input database and partition field information to obtain a multi-level tree structure of all folder names existing under the file path; then, the table is modified and partitions are added to each node of the multi-level tree structure in sequence.

[0021] Furthermore, the method also includes:

[0022] The cluster is divided into a public area and a tenant area; wherein, the public area includes a public exchange area, a public data area, and a public shared area; the tenant area includes a tenant exchange area and a tenant data area;

[0023] The public exchange area stores the data files exchanged between the public area and external systems; the public data area stores entity table data of basic data and shared data; the public shared area stores views corresponding to the data shared with tenants.

[0024] The tenant exchange area stores the data files exchanged between the tenant area and the outside world; the tenant data area stores the tenant's private data and the shared data accessed by the tenant.

[0025] On the other hand, this application provides a data sharing and transfer device for big data multi-cluster systems, the device comprising:

[0026] The receiving module is used to receive data sharing and transfer request information sent by the source end, and to obtain the destination end identification information and the data file of the source end carried in the data sharing and transfer request information;

[0027] The sending module is used to send the data file to the destination end according to the destination end identification information in a data exchange service scenario; and to create a shared view according to the data file and send the shared view to the destination end according to the destination end identification information in a data sharing service scenario, so that the destination end can obtain the data file according to the shared view.

[0028] The loading module is used to receive data loading request information sent by the destination end, load the data file obtained by the destination end into the database, so that the tenant can use the database to perform data queries based on the content of the data file.

[0029] Furthermore, the receiving module is specifically used to determine whether the data file of the source end is ready by polling at intervals to see if the data ready marker file exists. If it is ready, the module receives the data sharing transfer request information sent by the source end and obtains the destination end identifier information and the data file of the source end carried in the data sharing transfer request information.

[0030] Furthermore, the device also includes:

[0031] The instruction module is used to determine whether the current business scenario is one in which the execution of downstream tasks depends on the data readiness flag of the upstream. If so, when the data file at the source end is not ready, no error processing is performed, and the destination end is instructed to continue executing the business.

[0032] Furthermore, the device also includes:

[0033] The determination module is used to obtain and determine the data processing method based on the source data type and data location, as well as the destination data type and data location; and to perform data sharing and transfer processing according to the operator task corresponding to the data processing method.

[0034] Furthermore, the device also includes:

[0035] The synchronization processing module is used to obtain the partition field information of the database when executing the data synchronization task, and compare the partition field information with the field information specified by the data synchronization task; if the database is multi-level partitioned and the field information specified by the data synchronization task is not the top level in the partition field information, then multi-level partition synchronization processing is performed.

[0036] Furthermore, the synchronization processing module is specifically used to generate a filter configuration file, in which the partition filter information to be synchronized is written by reverse matching of regular expressions; and to perform multi-level partition synchronization processing based on the filter configuration file.

[0037] Furthermore, the device also includes:

[0038] The data activation module is used to perform a deep search traversal of the file path on the input database and partition field information when the data is activated, to obtain a multi-level tree structure of all folder names existing under the file path; and to add partitions to each node of the multi-level tree structure in turn.

[0039] Furthermore, the device also includes:

[0040] The cluster partitioning module is used to divide the cluster into a public area and a tenant area. The public area includes a public exchange area, a public data area, and a public shared area. The tenant area includes a tenant exchange area and a tenant data area. The public exchange area stores data files exchanged between the public area and external systems. The public data area stores entity table data for basic data and shared data. The public shared area stores views corresponding to data shared with tenants. The tenant exchange area stores data files exchanged between the tenant area and external systems. The tenant data area stores tenants' private data and shared data accessed by tenants.

[0041] In another aspect, this application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0042] Memory, used to store computer programs;

[0043] When a processor executes a program stored in memory, it implements the steps of the method described in any of the preceding statements.

[0044] In another aspect, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in any of the above claims.

[0045] This application provides a method, apparatus, device, and medium for data sharing and transfer across multiple big data clusters. The method includes: receiving data sharing and transfer request information sent by a source end; obtaining destination end identification information and a data file from the source end carried in the data sharing and transfer request information; in a data exchange service scenario, sending the data file to the destination end according to the destination end identification information; in a data sharing service scenario, creating a shared view based on the data file; sending the shared view to the destination end according to the destination end identification information, so that the destination end can obtain the data file based on the shared view; receiving data loading request information sent by the destination end; loading the data file obtained by the destination end into a database, so that tenants can use the database to perform data queries based on the content of the data file.

[0046] The above technical solution has the following advantages or beneficial effects:

[0047] In this application, after receiving a data sharing and transfer request from the source end, and obtaining the destination end identifier information and the data file from the source end carried in the data sharing and transfer request information, the data file is sent to the destination end according to the destination end identifier information in a data exchange service scenario, thereby realizing a data copy transfer mode; in a data sharing service scenario, a shared view is created based on the data file, and the shared view is sent to the destination end so that the destination end can obtain the data file according to the shared view, thereby realizing a view sharing transfer mode; and the data loading request information sent by the destination end is received, and the data file obtained by the destination end is loaded into the database so that the tenant can use the database to perform data queries based on the content of the data file, thereby realizing a data loading transfer mode. This solves the problem that existing data transmission solutions are only for a single scenario and lack other data transfer modes such as data loading and view sharing. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This application provides a schematic diagram of the data sharing and transfer process across multiple big data clusters.

[0050] Figure 2 The data sharing and flow architecture diagram for big data multi-cluster provided in this application;

[0051] Figure 3 The overall architecture diagram of the data distribution center provided in this application;

[0052] Figure 4 The data sharing and workflow diagram for big data multi-cluster provided in this application;

[0053] Figure 5 The deployment architecture diagram of the solution provided in this application;

[0054] Figure 6 A schematic diagram of the data sharing and transfer device for big data multi-cluster provided in this application;

[0055] Figure 7 A schematic diagram of the electronic device structure provided in this application. Detailed Implementation

[0056] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0057] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0058] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0059] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0060] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0061] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0062] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

[0063] Figure 1 The data sharing and transfer process for big data multi-cluster provided in this application includes the following steps:

[0064] S101: Receive data sharing and transfer request information sent by the source end, and obtain the destination end identification information and the data file of the source end carried in the data sharing and transfer request information.

[0065] S102: In a data exchange service scenario, the data file is sent to the destination end according to the destination end identification information; in a data sharing service scenario, a shared view is created according to the data file, and the shared view is sent to the destination end according to the destination end identification information, so that the destination end can obtain the data file according to the shared view.

[0066] S103: Receive the data loading request information sent by the destination terminal, load the data file obtained by the destination terminal into the database, so that the tenant can use the database to perform data queries based on the content of the data file.

[0067] The big data multi-cluster data sharing and transfer method provided in this application is applied to electronic devices, such as computers and servers.

[0068] The electronic device receives a data sharing and transfer request information sent by the source end. This request information carries the destination end identifier and the source end's data file. The electronic device can identify the destination end based on the destination end identifier. It should be noted that the source end and destination end in this application can be a tenant, a cluster, or an external system, respectively.

[0069] In data exchange scenarios, such as data exchange between clusters or between a cluster and an external system, electronic devices send data files to the destination. In data sharing scenarios, such as data sharing between tenants, electronic devices create a shared view based on the data file, send the shared view to the destination, and the destination can retrieve the data file based on the shared view.

[0070] The electronic device receives a data loading request from the destination and loads the data file obtained by the destination into the database. The tenant then uses the database to perform data queries based on the content of the data file.

[0071] The database in this application can be a Hive database. Data loading refers to loading data files from the cluster into the Hive database, allowing tenants to directly use the Hive database to perform SQL queries based on the data file content. Shared views are operations based on the Hive database. Specifically, the operation involves storing the result of a query in Hive as a virtual table, which can be used by other tenants in other Hive queries. Shared views are similar to physical tables, but they do not store data; instead, they are based on a query statement and are always up-to-date. This type of shared view allows multiple tenants to share the same dataset in the Hive database without needing to repeatedly execute queries.

[0072] In this application, after receiving a data sharing and transfer request from the source end, and obtaining the destination end identifier information and the data file from the source end carried in the data sharing and transfer request information, the data file is sent to the destination end according to the destination end identifier information in a data exchange service scenario, thereby realizing a data copy transfer mode; in a data sharing service scenario, a shared view is created based on the data file, and the shared view is sent to the destination end so that the destination end can obtain the data file according to the shared view, thereby realizing a view sharing transfer mode; and the data loading request information sent by the destination end is received, and the data file obtained by the destination end is loaded into the database so that the tenant can use the database to perform data queries based on the content of the data file, thereby realizing a data loading transfer mode. This solves the problem that existing data transmission solutions are only for a single scenario and lack other data transfer modes such as data loading and view sharing.

[0073] In this application, receiving data sharing and transfer request information sent by the source end, and obtaining the destination end identifier information and the data file of the source end carried in the data sharing and transfer request information include:

[0074] By polling at intervals to check if the data ready flag file exists, it is determined whether the data file of the source end is ready. If it is, the data sharing transfer request information sent by the source end is received, and the destination end identification information and the data file of the source end carried in the data sharing transfer request information are obtained.

[0075] Electronic devices can configure the `source_data_flag` field in the task list and periodically poll the `source_data_flag` field to determine if a ready flag file exists. If a ready flag file exists in the `source_data_flag` field, the source data file is considered ready; otherwise, it is considered not ready. After determining that the source data file is ready, the electronic device receives a data sharing transfer request from the source and retrieves the destination identifier information and the source data file carried in the request.

[0076] In this application, the method further includes:

[0077] Determine if the current business scenario is one where the execution of downstream tasks depends on the readiness of upstream data. If so, do not process errors when the source data file is not ready, and instruct the destination to continue executing the business.

[0078] In some specific business scenarios, upstream data may not be provided consistently every day. However, the execution of downstream tasks depends on the upstream data readiness flag. Electronic devices employ a special "silent firing" strategy to address this: a `mis_status` field can be added during task entry. The `mis_status` field is used to determine if the current scenario is one where the downstream task's execution depends on the upstream data readiness flag. If so, no error is reported when the source data file is not ready, and the destination is instructed to continue executing the business logic. For example, if the `mis_status` field is not `{mis_status":"ES"}`, it indicates that the current scenario is one where the downstream task's execution depends on the upstream data readiness flag.

[0079] In this application, the method further includes:

[0080] The data processing method is determined based on the source data type and data location, as well as the destination data type and data location; and the data sharing and transfer processing is performed according to the operator task corresponding to the data processing method.

[0081] Electronic devices configure the `dataType` and `clusterEn` fields in the task list, using these two fields to determine the operation to be performed. The `dataType` field indicates the data type, categorized as `HDFS_FILE`, `NFS_FILE`, and `HIVE_TABLE`. These represent HDFS cluster files, NFS external disk files, and Hive data tables, respectively. The `ClusterEn` field is the cluster identifier, such as `BS` for offline primary clusters and `BSA` for offline backup clusters.

[0082] For example, if both the source and destination dataType are HDFS paths, and ClusterEn indicates different clusters, then the cross-cluster data synchronization operator task will be executed. If the dataType field is configured with one NFS path and the other a Hive database name, then the data ingestion operator task will be executed, meaning the NFS path will first be uploaded to the HDFS path, and then the HDFS path will take effect with the data.

[0083] In this application, the method further includes:

[0084] When performing a data synchronization task, the partition field information of the database is obtained and compared with the field information specified by the data synchronization task. If the database is a multi-level partition and the field information specified by the data synchronization task is not the top-level partition field information, then multi-level partition synchronization processing is performed.

[0085] Typical Hive databases are single-partitioned or non-partitioned, with a few using two-level partitions. To ensure future scalability and compatibility, electronic devices support N-level partition data synchronization and activation. At the start of synchronization, the partition field information of the Hive database is obtained via HiveMetaStoreClient and compared with the partition field information to be synchronized. If the Hive database has multiple partitions, and the partition field to be synchronized is located in one of the partitions (excluding the topmost partition), then multi-level partition synchronization is required.

[0086] The multi-level partition synchronization process includes:

[0087] Generate a filter configuration file, in which the partition filter information to be synchronized is written using reverse regular expression matching; perform multi-level partition synchronization processing based on the filter configuration file.

[0088] The electronic device generates a filter.txt configuration file, in which the partition filtering information to be synchronized is written using reverse regular expression matching. When distcp is executed, this txt file is loaded to ensure that only path files containing intermediate partition fields are synchronized.

[0089] Electronic devices connect to the Hive database via HiveMetaStoreClient, and the database obtains the table structure via desc. If two partition fields are detected, it is considered a multi-level partition. A filter.txt file is generated for distcp.

[0090] Distcp is a distributed data replication tool for Hadoop. In this tool, "filter" is an operation that filters files, allowing data to be selected according to specific rules and ignoring unwanted data during the replication process. For example, you can filter by filename extension or creation date to copy only files containing specific data.

[0091] In this application, the method further includes:

[0092] When the data takes effect, a deep search traversal of the file path is performed on the input database and partition field information to obtain a multi-level tree structure of all folder names existing under the file path; then, the table is modified and partitions are added to each node of the multi-level tree structure in sequence.

[0093] During data activation, partition information is activated by monitoring the HDFS file layout. The program performs a deep search traversal of the file path based on the input Hive database and partition field information, obtaining a multi-level tree structure of all folder names existing under the path. Partitions are added to each node of the multi-level tree sequentially, ensuring that the multi-level partitions are activated quickly and completely. The HDFS file system is then deeply traversed, recording the folder names. Hive partitions are automatically loaded by matching folder names with Hive partitions.

[0094] In this application, the method further includes:

[0095] The cluster is divided into a public area and a tenant area; wherein, the public area includes a public exchange area, a public data area, and a public shared area; the tenant area includes a tenant exchange area and a tenant data area;

[0096] The public exchange area stores the data files exchanged between the public area and external systems; the public data area stores entity table data of basic data and shared data; the public shared area stores views corresponding to the data shared with tenants.

[0097] The tenant exchange area stores the data files exchanged between the tenant area and the outside world; the tenant data area stores the tenant's private data and the shared data accessed by the tenant.

[0098] Figure 2The big data multi-cluster data sharing and flow architecture diagram provided in this application includes cluster X, cluster Y, a data distribution center, and business systems. The data distribution center is equivalent to the processing software of the electronic device in this application.

[0099] Cluster X and Cluster Y each include a public area and a tenant area. The public area includes a public exchange area, a public data area, and a public shared area; the tenant area includes a tenant exchange area and a tenant data area. The public exchange area stores data files exchanged between the public area and external systems; the public data area stores entity table data for basic data and shared data; the public shared area stores views corresponding to data shared with tenants; the tenant exchange area stores data files exchanged between the tenant area and external systems; and the tenant data area stores tenants' private data and shared data accessed by tenants.

[0100] The public data area stores entity table data for basic and shared data, and does not directly provide services to the outside world. Data shared with tenants is called shared data. Everything else is basic data.

[0101] The public shared area stores data that needs to be shared with other tenants and can provide continuous and stable services; it contains views.

[0102] The public exchange area is a region for exchanging data files between the public area and external entities, and continuous and stable service is not guaranteed.

[0103] The tenant data area stores tenants' private data and shared data that tenants access; it does not directly provide services to the outside world.

[0104] The tenant exchange area is a region where tenants exchange data files with external entities, and continuous and stable service is not guaranteed.

[0105] This application utilizes a data distribution center to connect to external business systems, handle data synchronization between clusters, and standardize data sharing within clusters. It comprehensively solves the problems of data sharing and flow. The data distribution center employs various triggering strategies, such as scheduled tasks and file dependency marking, and utilizes multi-virtual machine deployment and multi-task parallelism to ensure timely and accurate data synchronization. The task activation method using Excel parameter configuration reduces development difficulty and improves overall requirement implementation efficiency, enabling more efficient and rapid data flow between clusters. The data distribution center is a full-scenario, end-to-end data sharing and flow solution, resolving the traceability issues of where data comes from, where it passes through, and where it goes in future multi-cluster environments.

[0106] This application addresses the limitations of existing technologies by proposing the construction of a big data distribution center to achieve data sharing within big data clusters and data exchange across clusters, standardizing data flow and promoting open data sharing and application. The big data distribution center will enable data flow and sharing management between tenants, business clusters and the full big data cluster, as well as external data and the cluster. A web platform will be established to provide a user-friendly interface for operation and management. A big data cluster data sharing zone will be established. A unified view of the open data will be created, allowing various users to easily access detailed information about the open data in the big data cluster. The big data distribution center will integrate a generalized configuration tool for data inflow and outflow modes, eliminating repetitive and redundant code development work and enabling configuration-based development. Based on the big data distribution center, a data sharing and exchange model will be established, ensuring that the data sharing and exchange process is completed based on the concept of "zones".

[0107] The specific application scenarios of the big data multi-cluster data sharing and transfer method provided in this application are shown in the table below:

[0108]

[0109]

[0110] To support the above application scenarios, Figure 3 The data distribution center provided in this application has an overall architecture diagram. The overall architecture of the data distribution center consists of a management module, a scheduling module, and an execution module.

[0111] Management module:

[0112] Management Service: Responsible for processing requests and persisting data, forming various parameter resources required by the backend. Also responsible for executing view creation / deletion actions. This is a persistent service.

[0113] Query Service: Responds to resource query requests and returns resource information. This is a persistent service.

[0114] Web page: Provides an external page for viewing task status and entering job information.

[0115] Scheduling module:

[0116] Notification Service: Responsible for receiving external notifications. When an external system backs up data into the big data cluster, it notifies this service that the resources are ready, and this service generates a resource readiness task in its daily tasks. This is a persistent service.

[0117] Task Management Service: Daily, it transforms periodic atomic operations into a specific task list. Simultaneously, it executes resource-ready tasks and updates resource status.

[0118] Execution module:

[0119] Task scheduling: Arrange the tasks in the task list, formulate execution strategies, and initiate tasks in sequence according to the strategies.

[0120] Task Execution Job: The actual executor of tasks, implementing the execution flow for various task types. It executes tasks based on parameters passed by the task orchestrator. Tasks include DISTCP execution, SQL execution, and data import into offline cluster Hive tables. During task execution, different operators such as data synchronization, SQL read / write, and data insertion are designed to achieve specific execution effects.

[0121] Figure 4 The data sharing and workflow diagram for big data multi-cluster provided in this application is as follows: Figure 4 As shown, after the process begins, it receives the task execution ID and queries the database for the task to be executed based on the ID, obtains the detailed parameters of the task, and calls different engines according to the task type. These different engines include data synchronization, SQL execution, data ingestion engine, and table structure mirroring request. It then determines whether the task executed successfully. If not, it updates the task status to EF; if so, it updates the task status and output resource status to ES, and writes the resource status flag file.

[0122] Figure 5 The deployment architecture diagram provided in this application shows that the management module uses two virtual machines for active-active deployment, with load balancing via F5. The management module is primarily responsible for inputting task parameter information into the UPSQL database. The scheduling module is responsible for generating specific task information from the input task parameters daily, and promptly querying the status of upstream resources and updating the task status. A master-slave deployment is adopted. The execution module uses multiple virtual machines, with each virtual machine capable of running multiple concurrent executions of specific tasks.

[0123] Task parameter information includes, for example, the following fields:

[0124] Task type, execution center, task cycle, task date, source data type, source data, list of clusters to which the source belongs, source partition field, source data flag file, external notification date, destination data date offset, data batch, destination data type, destination data, destination file permissions, cluster to which the destination belongs, destination partition field, destination data flag file, task node group name, task priority, scheduling time, precondition deadline, fire suppression policy, timeout, number of retries, use of external configuration, additional execution parameters, description, whether to discard, etc.

[0125] The input task parameters are generated into specific task information. For example, the task parameters are input into the tbl_bigdata_distributor_task_config table in the MySQL database, and a record in the tbl_bigdata_distributor_task_config table is instantiated into tbl_bigdata_distributor_task_status, which is the task generation.

[0126] It's important to note that we typically refer to a cluster user as a tenant. Tenants may be grouped by project or different application. Each tenant is essentially a resource pool; the administrator configures certain cluster resources for each tenant, and all users under a tenant share the resources within that pool. Multiple tenants share a single Hadoop big data cluster.

[0127] Figure 6 The schematic diagram of the data sharing and transfer device for big data multi-cluster provided in this application includes:

[0128] The receiving module 61 is used to receive data sharing and transfer request information sent by the source end, and to obtain the destination end identification information and the data file of the source end carried in the data sharing and transfer request information;

[0129] The sending module 62 is used to send the data file to the destination end according to the destination end identification information in a data exchange service scenario; and to create a shared view according to the data file in a data sharing service scenario, and send the shared view to the destination end according to the destination end identification information, so that the destination end can obtain the data file according to the shared view.

[0130] The loading module 63 is used to receive the data loading request information sent by the destination end, load the data file obtained by the destination end into the database, so that the tenant can use the database to perform data queries based on the content of the data file.

[0131] The receiving module 61 is specifically used to determine whether the data file of the source end is ready by polling at intervals to see if the data ready marker file exists. If it is ready, it receives the data sharing transfer request information sent by the source end and obtains the destination end identifier information and the data file of the source end carried in the data sharing transfer request information.

[0132] The device further includes:

[0133] The instruction module 64 is used to determine whether the current business scenario is a business scenario in which the operation of the downstream task depends on the data readiness flag of the upstream. If so, when the data file at the source end is not ready, no error processing is performed, and the destination end is instructed to continue executing the business.

[0134] The device further includes:

[0135] The determination module 65 is used to obtain and determine the data processing method based on the source data type and data location, as well as the destination data type and data location; and to perform data sharing and transfer processing according to the operator task corresponding to the data processing method.

[0136] The device further includes:

[0137] The synchronization processing module 66 is used to obtain the partition field information of the database when executing the data synchronization task, and compare the partition field information with the field information specified by the data synchronization task; if the database is multi-level partitioned and the field information specified by the data synchronization task is not the top level in the partition field information, then multi-level partition synchronization processing is performed.

[0138] The synchronization processing module 66 is specifically used to generate a filter configuration file, in which the partition filter information to be synchronized is written by reverse matching of regular expressions; and to perform multi-level partition synchronization processing based on the filter configuration file.

[0139] The device further includes:

[0140] The data activation module 67 is used to perform a deep search traversal of the file path on the input database and partition field information when the data is activated, to obtain a multi-level tree structure of all folder names existing under the file path; and to add partitions to each node of the multi-level tree structure in turn.

[0141] The device further includes:

[0142] The cluster partitioning module 68 is used to divide the cluster into a public area and a tenant area; wherein, the public area includes a public exchange area, a public data area, and a public shared area; the tenant area includes a tenant exchange area and a tenant data area; the public exchange area stores the data files exchanged between the public area and external entities; the public data area stores entity table data of basic data and shared data; the public shared area stores views corresponding to the data shared with tenants; the tenant exchange area stores the data files exchanged between the tenant area and external entities; and the tenant data area stores the tenant's private data and the shared data accessed by the tenant.

[0143] This application also provides an electronic device, such as Figure 7 As shown, it includes: processor 71, communication interface 72, memory 73 and communication bus 74, wherein processor 71, communication interface 72 and memory 73 communicate with each other through communication bus 74;

[0144] The memory 73 stores a computer program, which, when executed by the processor 71, causes the processor 71 to perform any of the above method steps.

[0145] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0146] Communication interface 72 is used for communication between the above-mentioned electronic device and other devices.

[0147] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0148] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0149] This application also provides a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform any of the above method steps.

[0150] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0151] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for data sharing and transfer across multiple big data clusters, characterized in that, The method includes: Receive data sharing and transfer request information sent by the source end, and obtain the destination end identification information and the data file of the source end carried in the data sharing and transfer request information; In a data exchange scenario, the data file is sent to the destination based on the destination identifier information; in a data sharing scenario, a shared view is created based on the data file, and the shared view is sent to the destination based on the destination identifier information, so that the destination can obtain the data file based on the shared view. The system receives a data loading request from the destination terminal and loads the data file obtained by the destination terminal into the database, so that the tenant can use the database to perform data queries based on the content of the data file. The process involves transforming periodic atomic operations into a specific task list, executing resource-ready tasks, updating resource status, orchestrating tasks in the task list, formulating execution strategies, initiating tasks sequentially according to the strategies, and achieving specific execution effects based on different operators. The method further includes: The data processing method is determined based on the source data type and data location, as well as the destination data type and data location; and the data sharing and transfer processing is performed according to the operator task corresponding to the data processing method.

2. The method as described in claim 1, characterized in that, The process of receiving data sharing and transfer request information sent by the source end, and obtaining the destination end identification information and the source end's data file carried in the data sharing and transfer request information, includes: By polling at intervals to check if the data ready flag file exists, it is determined whether the data file of the source end is ready. If it is, the data sharing transfer request information sent by the source end is received, and the destination end identification information and the data file of the source end carried in the data sharing transfer request information are obtained.

3. The method as described in claim 1, characterized in that, The method further includes: Determine whether the current business scenario is one where the execution of downstream tasks depends on the readiness of upstream data. If so, when the data file at the source end is not ready, no error processing is performed, and the destination end is instructed to continue executing the business.

4. The method as described in claim 1, characterized in that, The method further includes: When performing a data synchronization task, the partition field information of the database is obtained and compared with the field information specified by the data synchronization task. If the database is a multi-level partition and the field information specified by the data synchronization task is not the top-level partition field information, then multi-level partition synchronization processing is performed.

5. The method as described in claim 4, characterized in that, The multi-level partition synchronization process includes: Generate a filter configuration file, in which the partition filter information to be synchronized is written using reverse regular expression matching; perform multi-level partition synchronization processing based on the filter configuration file.

6. The method as described in claim 1, characterized in that, The method further includes: When the data takes effect, a deep search traversal of the file path is performed on the input database and partition field information to obtain a multi-level tree structure of all folder names existing under the file path; then, the table is modified and partitions are added to each node of the multi-level tree structure in sequence.

7. The method as described in claim 1, characterized in that, The method further includes: The cluster is divided into a public area and a tenant area; wherein, the public area includes a public exchange area, a public data area, and a public shared area; the tenant area includes a tenant exchange area and a tenant data area; The public exchange area stores the data files exchanged between the public area and external systems; the public data area stores entity table data of basic data and shared data; the public shared area stores views corresponding to the data shared with tenants. The tenant exchange area stores the data files exchanged between the tenant area and the outside world; the tenant data area stores the tenant's private data and the shared data accessed by the tenant.

8. A data sharing and transfer device for big data multi-cluster systems, characterized in that, The device includes: The receiving module is used to receive data sharing and transfer request information sent by the source end, and to obtain the destination end identification information and the data file of the source end carried in the data sharing and transfer request information; The sending module is used to send the data file to the destination end according to the destination end identification information in a data exchange service scenario; and to create a shared view according to the data file and send the shared view to the destination end according to the destination end identification information in a data sharing service scenario, so that the destination end can obtain the data file according to the shared view. The loading module is used to receive the data loading request information sent by the destination end, load the data file obtained by the destination end into the database, so that the tenant can use the database to perform data queries based on the content of the data file; The scheduling module is used to transform periodic atomic operations into a specific task list, execute resource-ready tasks, and update resource status. The execution module is used to orchestrate tasks in the task list, formulate execution strategies, initiate tasks sequentially according to the strategies, and achieve specific execution effects based on different operators. The device further includes: The determination module is used to obtain and determine the data processing method based on the source data type and data location, as well as the destination data type and data location; and to perform data sharing and transfer processing according to the operator task corresponding to the data processing method.

9. The apparatus as claimed in claim 8, characterized in that, The receiving module is specifically used to determine whether the data file of the source end is ready by polling at intervals to see if the data ready marker file exists. If it is ready, the module receives the data sharing transfer request information sent by the source end and obtains the destination end identifier information and the data file of the source end carried in the data sharing transfer request information.

10. The apparatus as claimed in claim 8, characterized in that, The device further includes: The instruction module is used to determine whether the current business scenario is one in which the execution of downstream tasks depends on the data readiness flag of the upstream. If so, when the data file at the source end is not ready, no error processing is performed, and the destination end is instructed to continue executing the business.

11. The apparatus as claimed in claim 8, characterized in that, The device further includes: The synchronization processing module is used to obtain the partition field information of the database when executing the data synchronization task, and compare the partition field information with the field information specified by the data synchronization task; if the database is multi-level partitioned and the field information specified by the data synchronization task is not the top level in the partition field information, then multi-level partition synchronization processing is performed.

12. The apparatus as claimed in claim 11, characterized in that, The synchronization processing module is specifically used to generate a filter configuration file, in which the partition filter information to be synchronized is written by reverse matching of regular expressions; and to perform multi-level partition synchronization processing based on the filter configuration file.

13. The apparatus as claimed in claim 8, characterized in that, The device further includes: The data activation module is used to perform a deep search traversal of the file path on the input database and partition field information when the data is activated, to obtain a multi-level tree structure of all folder names existing under the file path; and to add partitions to each node of the multi-level tree structure in turn.

14. The apparatus as claimed in claim 8, characterized in that, The device further includes: The cluster partitioning module is used to divide the cluster into a public area and a tenant area. The public area includes a public exchange area, a public data area, and a public shared area. The tenant area includes a tenant exchange area and a tenant data area. The public exchange area stores data files exchanged between the public area and external systems. The public data area stores entity table data for basic data and shared data. The public shared area stores views corresponding to data shared with tenants. The tenant exchange area stores data files exchanged between the tenant area and external systems. The tenant data area stores tenants' private data and shared data accessed by tenants.

15. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the steps of the method described in any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method described in any one of claims 1-7.

Images