Cross-cluster data modification method, device, apparatus and storage medium

Abstract

This application discloses a cross-cluster data modification method, apparatus, device, and storage medium, relating to the field of data management technology. The method includes: responding to a user-submitted data modification request and determining the data distribution corresponding to the data to be modified in the data modification request; if the data distribution involves an archived data cluster, obtaining the syntax mapping relationship between the archived data cluster and the current data cluster; mapping the data modification request based on the syntax mapping relationship to obtain a data modification task that matches the syntax of the archived data cluster; and modifying the data in the archived data cluster according to the data modification task. Compared to existing methods involving multiple cross-cluster synchronizations, this application can directly modify data in the archived data cluster according to a data modification task with an adapted syntax, reducing resource consumption caused by frequent cross-cluster synchronization and thus improving data modification efficiency.

Description

Technical Field

[0001] This application relates to the field of data management technology, and in particular to a method, apparatus, device and storage medium for cross-cluster data modification. Background Technology

[0002] Currently, in data asset management scenarios within a data lake architecture (e.g., enterprises divide data into online zones, near-line zones, and historical archive zones for separate storage), to manage data assets across multiple storage areas throughout their entire lifecycle, improve job batch processing efficiency, and reduce unit storage costs, the online zone, near-line zone, and historical data archive zone are typically stored in different databases. The online zone and near-line zone are generally stored in computing clusters with data processing capabilities, such as the Gauss database with a Massively Parallel Processing (MPP) architecture; the historical data archive zone is typically stored in a lower-cost archive cluster, such as the Hive database with a Hadoop architecture.

[0003] The existing method for modifying data in the archive area is to first backfetch the historical data to be modified from the archive area to the Gauss database in the online area, then perform insert, delete, and update operations on the Gauss database, and finally resynchronize the data back to the Hive database in the archive area to overwrite the data that needs to be modified in the Hive database.

[0004] However, this data modification method requires multiple synchronizations, leading to frequent cross-cluster data synchronization and consuming significant network bandwidth and storage resources. For example, even if only one row of records needs to be modified, the entire partition's data must be migrated back and forth, greatly wasting computing and storage resources and resulting in low data modification efficiency. Summary of the Invention

[0005] The main purpose of this application is to provide a method, apparatus, device and storage medium for cross-cluster data modification, which aims to solve the technical problem of low data modification efficiency in the existing data lake architecture, where modifying archive data requires multiple cross-cluster data synchronizations.

[0006] To achieve the above objectives, this application proposes a cross-cluster data modification method, the method comprising:

[0007] In response to a user's data modification request, the data distribution corresponding to the data to be modified in the data modification request is determined; If the data distribution involves an archived data cluster, then obtain the syntax mapping relationship between the archived data cluster and the current data cluster, where the current data cluster is a massively parallel processing cluster and the archived data cluster is a Hive cluster. Based on the grammar mapping relationship, the data modification request is mapped to obtain a data modification task that matches the grammar of the archived data cluster; The data in the archived data cluster is modified according to the data modification task.

[0008] In one embodiment, the step of responding to a user-submitted data modification request and determining the data distribution corresponding to the data to be modified in the data modification request includes: In response to a user's data modification request, the target table identifier involved in the data modification request is extracted; The table slice information is obtained by querying the preset meta-database according to the target table identifier. The preset meta-database stores the mapping relationship between partition data and each data cluster. Based on the table slice information, determine the data distribution of the data to be modified in each data cluster involved in the data modification request.

[0009] In one embodiment, the step of obtaining the syntax mapping relationship between the archived data cluster and the current data cluster if the data distribution involves an archived data cluster includes: The data modification request is parsed to obtain the data partition field, and the modification date of the data partition field is extracted. Determine the data date associated with the archived data cluster in the data distribution, and compare the data date with the data revision date; If there is an overlap between the data date and the data modification date, it is determined that the data to be modified involves the archived data cluster, and the syntax mapping relationship between the archived data cluster and the current data cluster is obtained.

[0010] In one embodiment, the step of mapping the data modification request based on the syntax mapping relationship to obtain a data modification task that matches the syntax of the archived data cluster includes: Determine the data modification type in the data modification request, and if the data modification type is laboratory data modification, generate a cross-cluster synchronization instruction; Based on the cross-cluster synchronization command, the current laboratory table is synchronized to the archived data cluster to obtain a temporary laboratory table; Extract the field types from the temporary laboratory table; Based on the syntax mapping relationship and the field type, the data modification request is mapped to obtain a data modification task that matches the syntax of the archived data cluster.

[0011] In one embodiment, the step of modifying the data in the archived data cluster according to the data modification task includes: Extract the data modification operation instructions from the data modification task, and determine the operation type of the data modification operation instructions; When the operation type is data insertion, the partition field of the temporary laboratory table is written into the data partition of the archived data cluster according to the data modification operation instruction; When the operation type is data deletion, the data modification operation instruction is converted into an overwrite instruction, and the data in the archived data cluster is modified according to the overwrite instruction; When the operation type is data update, the data modification operation instruction is converted into a data merge instruction, and the temporary laboratory table is merged into the archived data cluster according to the data merge instruction.

[0012] In one embodiment, the step of converting the data modification operation instruction into an overwrite instruction and modifying the data in the archived data cluster according to the overwrite instruction when the operation type is data deletion includes: When the operation type is data deletion, the archived data cluster is matched to determine the first partition of the archived data cluster to be processed. The data to be deleted in the first partition to be processed is located using the data modification operation command; Filter the data in the first partition to be processed that is not related to the data to be deleted to obtain the first unmatched data; The data modification operation instruction is converted into an overwrite instruction, and the first unmatched data is overwritten into the first partition to be processed according to the overwrite instruction.

[0013] In one embodiment, the step of converting the data modification operation instruction into a data merge instruction when the operation type is data update, and merging the temporary laboratory table into the archived data cluster according to the data merge instruction, includes: When the operation type is data update, the archived data cluster and the temporary laboratory table are matched by the data revision date to determine the second partition to be processed of the archived data cluster and the third partition to be processed of the temporary laboratory table. The data in the second partition to be processed that are not related to the temporary laboratory table are filtered to obtain the second unmatched data; Extract the data to be updated from the third partition to be processed, and convert the data editing operation instruction into a data merging instruction; According to the data merging instruction, the second unmatched data and the data to be updated are merged into the second partition to be processed.

[0014] Furthermore, to achieve the above objectives, this application also proposes a cross-cluster data modification apparatus, the apparatus comprising: The distribution module is used to respond to a user's data modification request and determine the data distribution corresponding to the data to be modified in the data modification request. The mapping module is used to obtain the syntax mapping relationship between the archived data cluster and the current data cluster if the data distribution involves an archived data cluster, wherein the current data cluster is a massively parallel processing cluster and the archived data cluster is a Hive cluster. The conversion module is used to map the data modification request based on the syntax mapping relationship to obtain a data modification task that matches the syntax of the archived data cluster. The data modification module is used to modify the data in the archived data cluster according to the data modification task.

[0015] In addition, to achieve the above objectives, this application also proposes a cross-cluster data modification device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the cross-cluster data modification method as described above.

[0016] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the cross-cluster data modification method described above.

[0017] One or more technical solutions proposed in this application have at least the following technical effects: The cross-cluster data modification method of this application includes: responding to a data modification request submitted by a user and determining the data distribution corresponding to the data to be modified involved in the data modification request; if the data distribution involves an archived data cluster, obtaining the syntax mapping relationship between the archived data cluster and the current data cluster, wherein the current data cluster is a massively parallel processing cluster and the archived data cluster is a Hive cluster; mapping the data modification request based on the syntax mapping relationship to obtain a data modification task that matches the syntax of the archived data cluster; and modifying the data in the archived data cluster according to the data modification task.

[0018] This application first responds to user-submitted data modification requests by determining the data distribution corresponding to the data to be modified. If an archived data cluster is involved, it obtains the syntax mapping relationship between the cluster and the current massively parallel processing cluster. Then, based on this syntax mapping relationship, it maps the data modification request to obtain a data modification task adapted to the syntax of the archived data cluster, and directly modifies the data in the archived data cluster according to the data modification task. Compared with the existing method of multiple cross-cluster synchronizations, this application can directly modify data in the archived data cluster according to the data modification task adapted to the syntax, reducing the resource consumption caused by frequent cross-cluster synchronization and thus improving the efficiency of data modification. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the cross-cluster data modification method of this application in Embodiment 1. Figure 2 A flowchart illustrating the process architecture for determining whether data to be modified involves an archived data cluster, provided in Embodiment 1 of this application. Figure 3 This is a flowchart illustrating Embodiment 2 of the cross-cluster data modification method of this application; Figure 4 This is an overall architecture diagram for performing bilateral data modification provided in Embodiment 2 of this application; Figure 5 This is a flowchart illustrating Embodiment 3 of the cross-cluster data modification method of this application; Figure 6 This is a block diagram of the module structure of the cross-cluster data modification device according to an embodiment of this application; Figure 7 This is a schematic diagram of the hardware operating environment involved in the cross-cluster data modification device in this application embodiment.

[0022] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0023] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0024] The main solution proposed in this application is as follows: Currently, in data asset management scenarios within a data lake architecture, to meet different data processing needs and rationally allocate storage costs, data is typically divided into three zones: the online zone, the near-line zone, and the archive zone. The online zone, also known as the Operational Data Store (ODS), is primarily used to support daily enterprise operations. Data in the online zone is usually more recent, such as data from the most recent week or month, to support batch processing calculations and timely business decisions. The near-line zone, also known as the Data Mart or Operational Data Store (ODM), is primarily used to support the enterprise's business analysis needs. The near-line zone contains more historical data than the online zone, but its data retention period is shorter compared to the historical data archive zone. Data in the near-line zone typically includes data from the most recent two or three years to support more in-depth data analysis. The archive zone, also known as Data Archiving Storage (DAS), is primarily used for long-term retention of historical data.

[0025] To manage data assets across multiple storage regions throughout their entire lifecycle, improve job batch processing efficiency, and reduce unit storage costs, online zones, near-line zones, and historical data archives are typically stored in different databases. Online zones and near-line zones are generally stored in computing clusters with data processing capabilities, such as the Gauss database with a Massively Parallel Processing (MPP) architecture; historical data archives are typically stored in lower-cost archive clusters, such as the Hive database with a Hadoop architecture.

[0026] The existing method for modifying data in the archive area is to first backfetch the historical data to be modified from the archive area to the Gauss database in the online area, then perform insert, delete, and update operations on the Gauss database, and finally resynchronize the data back to the Hive database in the archive area to overwrite the data that needs to be modified in the Hive database.

[0027] However, this data modification method requires multiple synchronizations, leading to frequent cross-cluster data synchronization and consuming significant network bandwidth and storage resources. For example, when modifying an erroneous record in an order table from three years ago, the entire table or partition needs to be exported from the Hive database to the Gauss database, modified, and then the entire partition's data is overwritten back to the Hive database. However, since the archive typically uses a Hadoop architecture, its underlying HDFS file system does not support efficient row-level random modification. Therefore, even if only one record needs to be modified, the entire partition's data must be migrated back and forth, greatly wasting computing and storage resources and resulting in inefficient data modification.

[0028] To address the aforementioned issues, this application provides a cross-cluster data modification method. First, in response to a user-submitted data modification request, the data distribution corresponding to the data to be modified is determined. If an archived data cluster is involved, the syntax mapping relationship between it and the current massively parallel processing cluster is obtained. Then, based on this syntax mapping relationship, the data modification request is mapped to obtain a data modification task adapted to the syntax of the archived data cluster. Finally, the data in the archived data cluster is directly modified according to the data modification task. Compared to existing methods involving multiple cross-cluster synchronizations, this application can directly modify data in the archived data cluster according to the syntax-adapted data modification task, reducing resource consumption caused by frequent cross-cluster synchronization and thus improving data modification efficiency.

[0029] It should be noted that the executing entity of this application embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a personal computer, a server, etc., or an electronic device capable of realizing the above functions, a cross-cluster data modification device executing the cross-cluster data modification method of this application, etc. This embodiment does not limit this. The following uses a cross-cluster data modification device (hereinafter referred to as the device) as an example to describe this embodiment and the following embodiments.

[0030] Based on this, this application proposes a cross-cluster data modification method according to a first embodiment, referring to... Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the cross-cluster data modification method of this application. In this embodiment, the cross-cluster data modification method may include steps S10 to S40: Step S10: In response to the data modification request submitted by the user, determine the data distribution corresponding to the data to be modified involved in the data modification request.

[0031] It should be noted that a data modification request can be a user-initiated instruction to modify data, also known as a data modification command using Structured Query Language (SQL). This data modification request may include the location of the data to be modified, the content to be modified, etc., but this embodiment does not impose any limitations on this.

[0032] It should also be noted that the data to be modified can be the specific data that needs to be modified based on the data modification request submitted by the user. This data may be distributed across different data clusters and is the target object of this data modification operation. Data distribution can refer to the storage location and distribution of the data to be modified in various data clusters. For example, the data to be modified may be the most recent month's data in the online area and historical data from three years ago in the archive area.

[0033] In practical use, the device can display an SQL production template to the user. The user fills in the data location and modification content of the data to be modified (i.e., the data to be modified) in the SQL production template, generating a data modification SQL. The device responds to the data modification SQL, parses the table name, partition, field and other identification information, and then queries the storage location, cluster and partitioning strategy of the data to be modified based on the identification information, thereby confirming the distribution status of the data to be modified in a cross-cluster scenario and obtaining the data distribution corresponding to the data to be modified.

[0034] Furthermore, in order to obtain the above-mentioned data distribution, in this embodiment, the step of responding to a user-submitted data modification request and determining the data distribution corresponding to the data to be modified involved in the data modification request includes: Step S11: In response to the data modification request submitted by the user, extract the target table identifier involved in the data modification request.

[0035] Understandably, the target table identifier can be a unique identifier for the specific database table involved in the data modification request. Through this target table identifier, the device can accurately point to the table whose data needs to be modified.

[0036] Step S12: Query the preset metadata database according to the target table identifier to obtain table slice information. The preset metadata database stores the mapping relationship between partition data and each data cluster.

[0037] Step S13: Based on the table slice information, determine the data distribution of the data to be modified in each data cluster involved in the data modification request.

[0038] It is also understandable that the preset metadata database can be a pre-built database that stores the mapping relationship between partitioned data and each data cluster, specifying which data clusters the data of different partitions are stored in. For example, order data of a certain time partition is stored in the online area of ​​the Gauss database, while order data of an earlier time partition is stored in the archive area of ​​the Hive database. Before the device modifies the data, it can use the preset metadata database to locate the storage situation of the data to be modified in different clusters.

[0039] It should be understood that table slicing information can be detailed partitioning information about the target table obtained from a preset metadata database based on the target table identifier. It describes the partitioning of the target table in different dimensions (such as time, business type, etc.) and the correspondence between each partition and each data cluster.

[0040] In this embodiment, the device can parse the data modification request submitted by the user and extract the target table identifier. Then, it performs a correlation query based on the partition-cluster mapping table in the preset metadata database. For example, if a user requests to modify a record in the order table for January 2023, the device first locates the metadata information of the table in the metadata database, queries its time partition and the storage cluster corresponding to each partition, and thus obtains the table slice information. Next, based on the partitioning strategy and cluster mapping relationship in the table slice information, the device accurately determines the distribution status of the data to be modified in a cross-cluster scenario. Thus, the data distribution can be quickly located through the metadata database mapping and table slice information to avoid full data synchronization across clusters.

[0041] Step S20: If the data distribution involves an archived data cluster, then obtain the syntax mapping relationship between the archived data cluster and the current data cluster, wherein the current data cluster is a massively parallel processing cluster and the archived data cluster is a Hive cluster.

[0042] It should be noted that an archived data cluster can be a data storage cluster used to retain historical data for a long time. It typically uses a low-cost storage architecture, such as a Hive database based on the Hadoop architecture (i.e., a Hive cluster). This Hive cluster is mainly used to store large amounts of historical data at low cost, but the underlying file system does not support efficient row-level random modification.

[0043] It should also be noted that the current data cluster can be the data cluster that acts as the main entity for operation execution during data modification. In this embodiment, it refers to a Massively Parallel Processing (MPP) cluster. For example, the Gauss database with a massively parallel processing architecture has data processing capabilities and can perform various data operation tasks.

[0044] Understandably, a syntax mapping relationship can be seen as a correspondence between the archived data cluster (i.e., the Hive cluster) and the current data cluster (i.e., the massively parallel processing cluster) in terms of data operation syntax. Since different clusters of database systems use different syntax rules, by establishing a syntax mapping relationship, the syntax of the Hive cluster can be converted into a syntax that the massively parallel processing cluster can recognize, thereby enabling cross-cluster data operations.

[0045] In practical use, refer to Figure 2 , Figure 2 This document presents a flowchart illustrating the process for determining whether data to be modified involves an archived data cluster, as provided in Embodiment 1 of this application. When a user submits a data modification request, the device first parses the table metadata and the relevant time range in the request to determine whether the data to be modified involves an archived data cluster. If the data to be modified does not involve an archived data cluster, only the MPP cluster data modification is performed. If the data to be modified involves an archived data cluster, both sides are executed, meaning that the MPP cluster data modification is performed simultaneously with the Hive cluster data modification. During the Hive cluster data modification process, the device can obtain the syntax mapping relationship between the archived data cluster and the current data cluster, which can be used for subsequent modifications to the archived data cluster.

[0046] Furthermore, in order to determine whether the above data distribution involves an archived data cluster, in this embodiment, the step of obtaining the syntax mapping relationship between the archived data cluster and the current data cluster if the data distribution involves an archived data cluster includes: Step S21: Parse the data modification request to obtain the data partition field, and extract the modification date of the data partition field.

[0047] It should be understood that the data partitioning field can be a field used in a data modification request to partition the data, which allows for quick location and access to data within different clusters.

[0048] It should also be understood that the data modification date can be date information related to the data to be modified, which can be parsed from the data modification request. It is used to reflect which dates the user wants to modify and serves as the basis for determining the scope of data modification in the time dimension.

[0049] Step S22: Determine the data date associated with the archived data cluster in the data distribution, and compare the data date with the data revision date.

[0050] Step S23: If there is an overlap between the data date and the data modification date, it is determined that the data to be modified involves the archived data cluster, and the syntax mapping relationship between the archived data cluster and the current data cluster is obtained.

[0051] In the archived data cluster, the stored data also has associated data dates, which identify the time range of data in different partitions within the archived data cluster. Comparing the data dates associated with the archived data cluster with the modification dates extracted from the data modification request can determine whether there is any overlap in the time dimension, thereby determining whether the data to be modified involves the archived data cluster.

[0052] If the same or overlapping dates are found when comparing the data date with the data revision date (i.e., coincident dates), it means that the data to be modified contains data from the archived data cluster, and cross-cluster data modification operations can be performed.

[0053] In this embodiment, the device can determine whether an archived data cluster is involved based on the date range of the data partition field in the data modification SQL. If there is an overlap between the data modification date and the data date of the archived data cluster (for example, the data modification dates from July 15, 2023 to June 31, 2024 overlap with the archived partition dates from January 1, 2023 to December 31, 2023), a syntax mapping instruction is triggered to load the pre-stored syntax mapping relationship between Hive and the MPP cluster. This allows for precise triggering of cross-cluster data modifications through date comparison, improving the efficiency of data modification.

[0054] Step S30: Based on the syntax mapping relationship, map the data modification request to obtain a data modification task that matches the syntax of the archived data cluster.

[0055] Step S40: Modify the data in the archived data cluster according to the data modification task.

[0056] It should be noted that data modification tasks can be obtained through syntax mapping, which clarifies the specific data modification operations to be performed on the archived data cluster, and are used to modify the data in the archived data cluster.

[0057] For example, if the submitted SQL for data modification is related to the Gauss database syntax, the database name and field types can be mapped to SQL related to the Hive cluster syntax to obtain the data modification task.

[0058] For example, the SQL for modifying numbers submitted to the online area is: update acos_data.acos_e_06_mkt_zdsj_udef_ele_s; set fld_nm='test'; Where fld_id='123' and dw_snsh_dt>='2023-11-30'; (The meaning of this SQL modification is: Update the acos_e_06_mkt_zdsj_udef_ele_s table under the acos_data schema, and change the value of the fld_nm field to 'test' in the records that meet the conditions fld_id='123' and dw_snsh_dt (data warehouse refresh date) ≥ 2023-11-30.) After acos_data is synchronized to the archive data cluster, the database name will be changed to acos_tdw, and the field types will also be changed accordingly, with type conversion performed according to the specific mapping. The partition field will become the default prt_d.

[0059] The generated SQL related to Hive cluster syntax is as follows: update acos_tdw.acos_e_06_mkt_zdsj_udef_ele_s; set fld_nm='test'; where fld_id='123' and prt_d>='2023-11-30'.

[0060] In practical use, the device first parses the SQL syntax structure of the data modification request, and then converts it into Hive-compatible syntax using pre-stored syntax mapping relationships in the syntax mapping rule base. The converted data modification task is then submitted to the archived data cluster for execution, while the modification results are captured in real time using the metadata synchronization mechanism of the MPP data cluster. After execution, the device verifies data consistency through a dual-write verification mechanism, ensuring that the data in the archived cluster and the MPP cluster remain consistent, thus preventing data corruption or logical anomalies caused by syntax conversion errors.

[0061] This application provides a cross-cluster data modification method. First, in response to a user-submitted data modification request, the data distribution corresponding to the data to be modified is determined. If an archived data cluster is involved, the syntax mapping relationship between it and the current massively parallel processing cluster is obtained. Then, based on this syntax mapping relationship, the data modification request is mapped to obtain a data modification task adapted to the syntax of the archived data cluster. The data in the archived data cluster is then directly modified according to the data modification task. Compared to existing methods that involve multiple cross-cluster synchronizations, this embodiment can directly modify data in the archived data cluster according to the syntax-adapted data modification task, reducing resource consumption caused by frequent cross-cluster synchronization and thus improving data modification efficiency.

[0062] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the above embodiment can be referred to the above description, and will not be repeated hereafter. On this basis, a second embodiment of the dialogue method of this application is proposed, please refer to... Figure 3 , Figure 3 This is a flowchart illustrating a second embodiment of the cross-cluster data modification method of this application. To obtain the data modification task that matches the syntax of the archived data cluster, as described above... Figure 3 As shown, in this embodiment, the step of mapping the data modification request based on the syntax mapping relationship to obtain a data modification task that matches the syntax of the archived data cluster includes: Step S31: Determine the data modification type in the data modification request, and if the data modification type is laboratory data modification, generate a cross-cluster synchronization instruction.

[0063] It should be noted that the data modification type can be the specific operation category to which the user's data modification request belongs, such as routine production environment data modification or laboratory data modification. Among them, laboratory data modification is usually carried out in a simulation or test environment to verify the data modification logic and test new functions. Compared with direct data modification in the production environment, it can reduce the risk of impacting actual business data.

[0064] It should also be noted that the cross-cluster synchronization command can be a command generated by the device to trigger the synchronization operation of data between different clusters when the data repair type is determined to be laboratory data repair.

[0065] Step S32: Based on the cross-cluster synchronization instruction, synchronize the current laboratory table to the archived data cluster to obtain a temporary laboratory table.

[0066] Understandably, the current lab table can be a table used to store relevant data in a lab environment, containing data records that need to be modified or tested. A temporary lab table, on the other hand, can be a table generated from the current lab table in an archive data cluster through cross-cluster synchronization operations. It retains the data structure and content of the current lab table but exists within the archive data cluster environment.

[0067] Step S33: Extract the field types of the temporary laboratory table.

[0068] Step S34: Map the data modification request according to the syntax mapping relationship and the field type to obtain a data modification task that matches the syntax of the archived data cluster.

[0069] It is also understandable that the field type can be the data type defined by each field in the temporary laboratory table, such as integer, string, date, etc., which determines the format and range of data that the field can store.

[0070] For example, for ease of understanding, refer to Figure 4 , Figure 4 This is an overall architecture diagram for performing two-sided data modification as provided in Embodiment 2 of this application. When the data distribution involves an archived data cluster, the device enters the two-sided execution phase. First, it determines the data modification type in the data modification request and then judges whether the data modification request is a laboratory data modification type. If it is determined to be a laboratory data modification, the current laboratory table needs to be synchronized to the archived data cluster first, and then the MPP syntax of the data modification request is converted into a data modification task adapted to Hive syntax; if it is not a laboratory data modification, the syntax conversion is performed directly. Subsequently, the converted data modification tasks are concurrently scheduled and executed in parallel on the MPP cluster and the Hive cluster, respectively. Finally, the results of the two-sided execution are judged. If the data modification tasks of both the MPP and Hive clusters are successfully executed, the task is considered complete; if either side fails, the device will trigger a retry mechanism or manual intervention to ensure the integrity of the data modification operation and avoid data inconsistency problems caused by single point of failure.

[0071] In practical use, by clearly defining the data modification type and generating cross-cluster synchronization instructions in the laboratory data modification scenario, the laboratory table is synchronized to the archive cluster to form a temporary table; then, the field types are extracted and combined with the syntax mapping relationship to generate data modification tasks adapted to the syntax of the archive cluster, which can ensure the accurate execution of data modification in the laboratory environment in the archive cluster and avoid errors caused by syntax differences.

[0072] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to that in embodiments one and two above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 5 , Figure 5 This is a flowchart illustrating a third embodiment of the cross-cluster data modification method of this application. During the data modification process in the archived data cluster, such as... Figure 5 As shown, in this embodiment, the step of modifying the data in the archived data cluster according to the data modification task includes: Step S41: Extract the data modification operation instructions from the data modification task and determine the operation type of the data modification operation instructions.

[0073] It should be noted that data modification instructions can be the specific commands within a data modification task, clearly indicating what specific operation to be performed on the data; they are the specific execution statements of the data modification task. The operation type can be a classification and definition of the nature of the data modification instruction, used to clarify whether the instruction is to perform different types of data modification operations such as data insertion, data deletion, or data update.

[0074] Step S421: When the operation type is data insertion, the partition field of the temporary laboratory table is written into the data partition of the archived data cluster according to the data modification operation instruction.

[0075] It should also be noted that data insertion can be an operation type that adds new data records to a specified location in the archive data cluster, thereby increasing the amount of data in the cluster. The partition field can be a field in a temporary laboratory table used to divide data into partitions, determining the storage location of data within the archive data cluster's data partitions.

[0076] For example, when inserting a temporary lab table into a specified table in the archive area, partitioning can be performed based on the partition field prt_dt, allowing the Hive cluster to automatically assign partition values ​​based on the partition field to complete the partition management for data insertion.

[0077] Furthermore, since there may be two-sided execution, the same principle applies to data insertion in the MPP cluster. For example, data from the temporary laboratory table that meets the condition that the partition field is greater than or equal to November 30, 2023, can be inserted into the target table in the online zone; then, data from the same temporary laboratory table that meets the condition that the partition field is less than November 30, 2023, can be inserted into the target table in the near-line zone.

[0078] Step S422: When the operation type is data deletion, the data modification operation instruction is converted into an overwrite instruction, and the data in the archived data cluster is modified according to the overwrite instruction.

[0079] Understandably, data deletion can be an operation that removes data records from an archived data cluster, cleaning up useless data within the cluster. An overwrite command can be a modified version of the data deletion operation; its function is to delete existing data by rewriting it.

[0080] Data deletion can be categorized into several types, such as row deletion, full table deletion, and partial deletion.

[0081] For example, for partial deletion of archived data clusters, data that does not need to be deleted can be filtered from a specified table and the deletion effect can be achieved by overwriting; for full deletion of archived data clusters, all partitions that meet the partition field are deleted directly, and the partition information in the metadata and the corresponding files in the storage layer are also deleted, thereby meeting the data deletion needs of different ranges.

[0082] For example, partial and full table deletions in an MPP cluster differ from those in an archived data cluster. Data can be directly deleted from the target tables in the online and near-line zones using SQL.

[0083] Furthermore, for row deletion in the archived data cluster, the step of converting the data modification operation instruction into an overwrite instruction and modifying the data in the archived data cluster according to the overwrite instruction when the operation type is data deletion includes: Step S4221: In the case that the operation type is data deletion, the archived data cluster is matched to determine the first partition to be processed in the archived data cluster.

[0084] It should be noted that the first partition to be processed can be the partition that needs to be deleted from the many data partitions in the archive data cluster after matching. It is the specific area in the archive data cluster where this data deletion operation is performed.

[0085] Step S4222: Locate the data to be deleted in the first partition to be processed using the data modification operation command.

[0086] Step S4223: Filter the data in the first partition to be processed that is unrelated to the data to be deleted to obtain the first unmatched data.

[0087] Step S4224: Convert the data modification operation instruction into an overwrite instruction, and overwrite the first unmatched data into the first partition to be processed according to the overwrite instruction.

[0088] It should be understood that the data to be deleted can be the data records in the first partition to be processed that need to be deleted according to the data editing operation instructions. The first unmatched data can be data records in the first partition to be processed that are unrelated to the data to be deleted after filtering. These data will be retained and subsequently re-stored in the first partition to be processed by overwriting to ensure the integrity and accuracy of the data in the partition, and only the unnecessary data to be deleted will be removed.

[0089] In practical use, since Hive clusters do not support efficient deletion via subqueries, it is necessary to filter out the data that needs to be retained from the first partition to be processed, obtaining the first unmatched data. Then, by overwriting the first partition with the first unmatched data, the data deletion operation in the archive data cluster can be achieved. This avoids full table scans and large-scale data migrations in the Hive cluster, effectively reducing the amount of data processing while ensuring the accuracy and efficiency of the data deletion operation.

[0090] Furthermore, due to the potential for dual-sided execution, data deletion in an MPP cluster differs from that in a Hive cluster. In an MPP cluster, data deletion can be performed directly on target tables in both online and near-line zones. Records that meet specific time criteria and have a completely matching combination of fields in the source table can be deleted from the target table, thus achieving precise removal of specific data from target tables in different regions.

[0091] Step S423: When the operation type is data update, the data modification operation instruction is converted into a data merge instruction, and the temporary laboratory table is merged into the archived data cluster according to the data merge instruction.

[0092] It should be understood that a data update can be an operation that updates the current state of existing data in an archived data cluster or corrects previously erroneous data. A data merge instruction can be a command derived from a data update operation, used to merge data in a temporary laboratory table with data in the archived data cluster, thereby updating the data in the archived data cluster.

[0093] The data update process is divided into partial updates and full table updates. For full table updates, in both archived data clusters and MPP clusters, the corresponding target table is simply cleared, and the data to be updated is directly written to the target table.

[0094] Furthermore, during the data partial update process, the step of converting the data modification operation instruction into a data merge instruction when the operation type is data update, and merging the temporary laboratory table into the archived data cluster according to the data merge instruction, includes: Step S231: When the operation type is data update, match the archived data cluster and the temporary laboratory table by the data revision date to determine the second partition to be processed of the archived data cluster and the third partition to be processed of the temporary laboratory table.

[0095] It should be noted that the second partition to be processed can be a partition that needs data update processing, determined by matching the archived data cluster based on the revision date; it is the data area in the archived data cluster that will be operated on. The third partition to be processed can be a partition determined by matching the temporary laboratory table based on the revision date; the data in this third partition will participate as the data to be updated in the subsequent data merge and update operation with the archived data cluster.

[0096] Step S232: Filter the data in the second partition to be processed that are not related to the temporary laboratory table to obtain the second unmatched data.

[0097] It should also be noted that the second unmatched data can be data that is unrelated to the temporary laboratory table, obtained after filtering the second partition to be processed.

[0098] Step S233: Extract the data to be updated from the third partition to be processed, and convert the data editing operation instruction into a data merging instruction.

[0099] Step S234: According to the data merging instruction, merge the second unmatched data and the data to be updated into the second partition to be processed.

[0100] Understandably, the data to be updated can be data extracted from the third partition to be processed that needs to be updated. This data will be merged into the second partition to be processed in the archived data cluster according to the data merge instruction, thereby updating the original data.

[0101] For example, suppose the temporary laboratory table is 'b', and the table in the archived data cluster that needs data updates is 'a'. During the data update process, a left join (i.e., connecting the target table a as the main table with the temporary laboratory table b, setting a join condition where all four fields match) is used, along with filtering conditions. Records from the target table a that cannot be matched in the temporary laboratory table b and meet specific time partitioning requirements (e.g., times earlier than 2023-11-30) are selected, retaining the data that does not need updating, resulting in the second set of unmatched data. Then, the values ​​of all fields in the target table a are replaced with empty strings, and the new data from the temporary laboratory table b with times earlier than 2023-11-30 and the second set of unmatched data are merged into the target table a, thus achieving targeted update processing of the target table data.

[0102] Furthermore, due to the potential for two-sided execution, the data update process in the MPP cluster differs from that in the Hive cluster. In the MPP cluster, the data to be updated can be directly matched against the target table in the online or near-line area using field combinations. If a match is found and the time condition is greater than or equal to the time partition (e.g., 2023-11-30), the fields of the matching records in the target table are updated; this achieves the data update process for the target table in the online or near-line area.

[0103] In this embodiment, data deletion in the archive area can accurately locate the partition to be processed and the data to be deleted, reducing unnecessary data processing; in data updates, the partition to be processed is determined by date matching, and data is reasonably filtered and merged, ensuring the accuracy and integrity of the data.

[0104] It should be noted that the above examples are for understanding this application only and do not constitute a limitation on the cross-cluster data modification method of this application. Any simple modifications based on this technical concept are within the scope of protection of this application. All actions involving the acquisition of signals, information, or data in this application are performed in accordance with the relevant data protection regulations of the country where the application is located and with authorization from the owner of the relevant device.

[0105] This application also provides a cross-cluster data modification device, please refer to... Figure 6 , Figure 6 This is a block diagram of the module structure of the cross-cluster data modification device according to an embodiment of this application; in this embodiment, the cross-cluster data modification device includes: The distribution module 601 is used to respond to a data modification request submitted by a user and determine the data distribution corresponding to the data to be modified involved in the data modification request; The mapping module 602 is used to obtain the syntax mapping relationship between the archived data cluster and the current data cluster if the data distribution involves an archived data cluster, wherein the current data cluster is a massively parallel processing cluster and the archived data cluster is a Hive cluster. The conversion module 603 is used to map the data modification request based on the syntax mapping relationship to obtain a data modification task that matches the syntax of the archived data cluster. The data modification module 604 is used to modify the data in the archived data cluster according to the data modification task.

[0106] This embodiment first responds to a user-submitted data modification request by determining the data distribution corresponding to the data to be modified. If an archived data cluster is involved, it obtains the syntax mapping relationship between the cluster and the current massively parallel processing cluster. Then, based on this syntax mapping relationship, it maps the data modification request to obtain a data modification task adapted to the syntax of the archived data cluster, and directly modifies the data in the archived data cluster according to the data modification task. Compared with the existing method of multiple cross-cluster synchronizations, this embodiment can directly modify the data in the archived data cluster according to the data modification task adapted to the syntax, reducing the resource consumption caused by frequent cross-cluster synchronization and thus improving the data modification efficiency.

[0107] In one implementation, the distribution module 601 is further configured to respond to a data modification request submitted by a user and extract the target table identifier involved in the data modification request; query a preset metadata database based on the target table identifier to obtain table slice information, wherein the preset metadata database stores the mapping relationship between partitioned data and each data cluster; and determine the data distribution of the data to be modified involved in the data modification request in each data cluster based on the table slice information.

[0108] In one implementation, the mapping module 602 is further configured to parse the data modification request, obtain a data partition field, and extract the data modification date of the data partition field; determine the data date related to the archived data cluster in the data distribution, and compare the data date with the data modification date; if there is an overlap between the data date and the data modification date, it is determined that the data to be modified involves the archived data cluster, and the syntax mapping relationship between the archived data cluster and the current data cluster is obtained.

[0109] In one implementation, the conversion module 603 is further configured to determine the data modification type in the data modification request, and if the data modification type is laboratory data modification, generate a cross-cluster synchronization instruction; based on the cross-cluster synchronization instruction, synchronize the current laboratory table to the archived data cluster to obtain a temporary laboratory table; extract the field types of the temporary laboratory table; and map the data modification request according to the syntax mapping relationship and the field types to obtain a data modification task that matches the syntax of the archived data cluster.

[0110] In one implementation, the data editing module 604 is further configured to extract data editing operation instructions from the data editing task and determine the operation type of the data editing operation instructions; if the operation type is data insertion, write the partition field of the temporary laboratory table into the data partition of the archived data cluster according to the data editing operation instructions; if the operation type is data deletion, convert the data editing operation instructions into overwrite instructions and modify the data in the archived data cluster according to the overwrite instructions; if the operation type is data update, convert the data editing operation instructions into data merge instructions and merge the temporary laboratory table into the archived data cluster according to the data merge instructions.

[0111] In one implementation, the data editing module 604 is further configured to, when the operation type is data deletion, match the archived data cluster to determine the first partition to be processed in the archived data cluster; locate the data to be deleted in the first partition to be processed through the data editing operation instruction; filter the data in the first partition to be processed that is unrelated to the data to be deleted to obtain the first unmatched data; convert the data editing operation instruction into an overwrite instruction, and overwrite the first unmatched data into the first partition to be processed according to the overwrite instruction.

[0112] In one implementation, the data editing module 604 is further configured to, when the operation type is data update, match the archived data cluster and the temporary laboratory table according to the data editing date to determine the second partition to be processed in the archived data cluster and the third partition to be processed in the temporary laboratory table; filter the data in the second partition to be processed that is unrelated to the temporary laboratory table to obtain the second unmatched data; extract the data to be updated in the third partition to be processed and convert the data editing operation instruction into a data merging instruction; and merge the second unmatched data and the data to be updated into the second partition to be processed according to the data merging instruction.

[0113] Other embodiments or specific implementations of the cross-cluster data modification device of this application can be found in the above-described method embodiments, and will not be repeated here.

[0114] The cross-cluster data modification apparatus provided in this application, employing the cross-cluster data modification method described in the above embodiments, can solve the technical problem in existing data lake architectures where modifying archived data requires multiple cross-cluster data synchronizations, resulting in low data modification efficiency. Compared with the prior art, the beneficial effects of the cross-cluster data modification apparatus provided in this application are the same as those of the cross-cluster data modification method provided in the above embodiments, and other technical features in the cross-cluster data modification apparatus are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0115] This application provides a cross-cluster data modification device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the cross-cluster data modification methods in the above embodiments.

[0116] The following is for reference. Figure 7 , Figure 7 This is a schematic diagram of the hardware operating environment involved in the cross-cluster data modification device in the embodiments of this application, showing a structural schematic diagram suitable for implementing the cross-cluster data modification device in the embodiments of this application. The cross-cluster data modification device in the embodiments of this application may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 7The cross-cluster data modification device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0117] like Figure 7 As shown, the cross-cluster data modification device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in read-only memory 1002 or a program loaded from storage device 1003 into random access memory 1004. Random access memory 1004 also stores various programs and data required for the operation of the cross-cluster data modification device. The processing unit 1001, read-only memory 1002, and random access memory 1004 are interconnected via bus 1005. Input / output interface 1006 is also connected to the bus. Typically, the following systems can be connected to input / output interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows cross-cluster data modification devices to communicate wirelessly or wiredly with other devices to exchange data. While the figures show cross-cluster data modification devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0118] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0119] The cross-cluster data modification device provided in this application, employing the cross-cluster data modification method described in the above embodiments, can solve the technical problem in existing data lake architectures where modifying archived data requires multiple cross-cluster data synchronizations, resulting in low data modification efficiency. Compared with the prior art, the beneficial effects of the cross-cluster data modification device provided in this application are the same as those of the cross-cluster data modification method provided in the above embodiments, and other technical features of this cross-cluster data modification device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0120] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0121] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0122] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the cross-cluster data modification method in the above embodiments.

[0123] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0124] The aforementioned computer-readable storage medium may be included in a cross-cluster data modification device; or it may exist independently and not be assembled into a cross-cluster data modification device.

[0125] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by a cross-cluster data modification device, the cross-cluster data modification device: responds to a data modification request submitted by a user and determines the data distribution corresponding to the data to be modified involved in the data modification request; if the data distribution involves an archived data cluster, it obtains a syntax mapping relationship between the archived data cluster and the current data cluster, wherein the current data cluster is a massively parallel processing cluster and the archived data cluster is a Hive cluster; based on the syntax mapping relationship, it maps the data modification request to obtain a data modification task that matches the syntax of the archived data cluster; and modifies the data in the archived data cluster according to the data modification task.

[0126] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0128] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0129] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described cross-cluster data modification method. This solves the technical problem in existing data lake architectures where modifying archived data requires multiple cross-cluster data synchronizations, resulting in low data modification efficiency. Compared to the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the cross-cluster data modification method provided in the above embodiments, and will not be elaborated upon here.

[0130] The above description is only a part of the embodiments of this application and does not limit the scope of protection of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included in the scope of protection of this application.

Claims

1. A method for modifying data across clusters, characterized in that, The method includes: In response to a user's data modification request, the data distribution corresponding to the data to be modified in the data modification request is determined; If the data distribution involves an archived data cluster, then obtain the syntax mapping relationship between the archived data cluster and the current data cluster, where the current data cluster is a massively parallel processing cluster and the archived data cluster is a Hive cluster. Based on the grammar mapping relationship, the data modification request is mapped to obtain a data modification task that matches the grammar of the archived data cluster; The data in the archived data cluster is modified according to the data modification task.

2. The method as described in claim 1, characterized in that, The step of responding to a user-submitted data modification request and determining the data distribution corresponding to the data to be modified in the data modification request includes: In response to a user's data modification request, the target table identifier involved in the data modification request is extracted; The table slice information is obtained by querying the preset meta-database according to the target table identifier. The preset meta-database stores the mapping relationship between partition data and each data cluster. Based on the table slice information, determine the data distribution of the data to be modified in each data cluster involved in the data modification request.

3. The method as described in claim 1, characterized in that, If the data distribution involves an archived data cluster, the step of obtaining the syntax mapping relationship between the archived data cluster and the current data cluster includes: The data modification request is parsed to obtain the data partition field, and the modification date of the data partition field is extracted. Determine the data date associated with the archived data cluster in the data distribution, and compare the data date with the data revision date; If there is an overlap between the data date and the data modification date, it is determined that the data to be modified involves the archived data cluster, and the syntax mapping relationship between the archived data cluster and the current data cluster is obtained.

4. The method according to any one of claims 1 to 3, characterized in that, The step of mapping the data modification request based on the syntax mapping relationship to obtain a data modification task that matches the syntax of the archived data cluster includes: Determine the data modification type in the data modification request, and if the data modification type is laboratory data modification, generate a cross-cluster synchronization instruction; Based on the cross-cluster synchronization command, the current laboratory table is synchronized to the archived data cluster to obtain a temporary laboratory table; Extract the field types from the temporary laboratory table; Based on the syntax mapping relationship and the field type, the data modification request is mapped to obtain a data modification task that matches the syntax of the archived data cluster.

5. The method as described in claim 4, characterized in that, The step of modifying the data in the archived data cluster according to the data modification task includes: Extract the data modification operation instructions from the data modification task, and determine the operation type of the data modification operation instructions; When the operation type is data insertion, the partition field of the temporary laboratory table is written into the data partition of the archived data cluster according to the data modification operation instruction; When the operation type is data deletion, the data modification operation instruction is converted into an overwrite instruction, and the data in the archived data cluster is modified according to the overwrite instruction; When the operation type is data update, the data modification operation instruction is converted into a data merge instruction, and the temporary laboratory table is merged into the archived data cluster according to the data merge instruction.

6. The method as described in claim 5, characterized in that, The step of converting the data modification operation instruction into an overwrite instruction when the operation type is data deletion, and modifying the data in the archived data cluster according to the overwrite instruction, includes: When the operation type is data deletion, the archived data cluster is matched to determine the first partition of the archived data cluster to be processed. The data to be deleted in the first partition to be processed is located using the data modification operation command; Filter the data in the first partition to be processed that is not related to the data to be deleted to obtain the first unmatched data; The data modification operation instruction is converted into an overwrite instruction, and the first unmatched data is overwritten into the first partition to be processed according to the overwrite instruction.

7. The method as described in claim 5, characterized in that, The step of converting the data modification operation instruction into a data merge instruction when the operation type is data update, and merging the temporary laboratory table into the archived data cluster according to the data merge instruction, includes: When the operation type is data update, the archived data cluster and the temporary laboratory table are matched by the data revision date to determine the second partition to be processed of the archived data cluster and the third partition to be processed of the temporary laboratory table. The data in the second partition to be processed that are not related to the temporary laboratory table are filtered to obtain the second unmatched data; Extract the data to be updated from the third partition to be processed, and convert the data editing operation instruction into a data merging instruction; According to the data merging instruction, the second unmatched data and the data to be updated are merged into the second partition to be processed.

8. A cross-cluster data modification device, characterized in that, The device includes: The distribution module is used to respond to a user's data modification request and determine the data distribution corresponding to the data to be modified in the data modification request. The mapping module is used to obtain the syntax mapping relationship between the archived data cluster and the current data cluster if the data distribution involves an archived data cluster, wherein the current data cluster is a massively parallel processing cluster and the archived data cluster is a Hive cluster. The conversion module is used to map the data modification request based on the syntax mapping relationship to obtain a data modification task that matches the syntax of the archived data cluster. The data modification module is used to modify the data in the archived data cluster according to the data modification task.

9. A cross-cluster data modification device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the cross-cluster data modification method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the cross-cluster data modification method as described in any one of claims 1 to 7.

Images