Partition migration method and device, equipment and storage medium
By implementing partition migration methods in the data lake, the problem of data duplication and low migration efficiency caused by partition field changes in the data lake is solved, and automated migration is achieved, improving efficiency and stability.
Patent Information
- Application Number
- CN202411724699.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-27
- Publication Date
- 2025-03-21
Smart Images
Figure CN119669188A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology and can be applied to the field of big data technology. In particular, the present invention relates to a partition migration method, device, equipment and storage medium. Background Art
[0002] When the partition field in the data lake changes, the change in the partition field will change the organization and indexing method of the data lake, causing duplicate data records stored in the initial partition. This duplication will lead to data anomalies, making data analysis and business decisions unreliable. To avoid this situation, data migration is required for the data records stored in the initial partition.
[0003] However, the migration process of the data records stored in the initial partition is cumbersome, which is not conducive to improving the migration efficiency of the data records. The reason is that the existing technology lacks a partition migration solution for the data lake, and currently can only rely on cumbersome manual operations to complete the migration of the data records stored in the initial partition. The manual operation method consumes a lot of human resources and time resources, and increases the migration time of the data records stored in the initial partition. Therefore, it is not conducive to improving the migration efficiency of the data records. Summary of the invention
[0004] The present invention provides a partition migration method, device, computer equipment and storage medium to solve the technical problem that the migration process of data records stored in the initial partition is cumbersome and is not conducive to improving the migration efficiency of the data records.
[0005] In a first aspect, a partition migration method is provided, including:
[0006] Connecting to a data source, and obtaining incremental data from the data source;
[0007] Obtain a target partition field in the incremental data, obtain a metadata table in the data lake, and obtain an initial partition field stored in the metadata table;
[0008] When the target partition field and the initial partition field meet a preset condition, obtaining an initial partition;
[0009] Acquire the data records stored in the initial partition;
[0010] Based on the target partition field and a preset update method, the data record is updated to obtain the updated data record;
[0011] A target partition corresponding to the target partition field is obtained, and the updated data record is migrated from the initial partition to the target partition.
[0012] Further, the obtaining of the target partition field in the incremental data, obtaining the metadata table in the data lake, and obtaining the initial partition field stored in the metadata table include:
[0013] Obtain the primary key and target partition field in the incremental data, and obtain the metadata table in the data lake through the query interface provided by the data lake;
[0014] A query request for the primary key is obtained, and whether the metadata table contains the primary key is queried according to the query request; when the metadata table contains the primary key, an initial partition field stored in the metadata table is obtained.
[0015] Further, when the target partition field and the initial partition field meet a preset condition, obtaining the initial partition includes:
[0016] Obtaining the field value in the target partition field, and obtaining the field value in the initial partition field;
[0017] When the field value in the target partition field is inconsistent with the field value in the initial partition field, the initial partition corresponding to the field value in the initial partition field is obtained.
[0018] Further, the obtaining of the data records stored in the initial partition includes:
[0019] Access the initial partition through an access interface provided by the data lake;
[0020] A read operation is performed on the data block of the initial partition to obtain the data record stored in the initial partition.
[0021] Further, the updating of the data record based on the target partition field and a preset updating method to obtain the updated data record includes:
[0022] Obtain an update instruction, execute the update instruction, and update the initial partition field in the data record to the target partition field;
[0023] The data record using the target partition field is selected as the updated data record.
[0024] Further, the obtaining of the target partition corresponding to the target partition field and migrating the updated data record from the initial partition to the target partition includes:
[0025] Obtain the target partition corresponding to the target partition field, and access the target partition corresponding to the target partition field;
[0026] A migration instruction is generated, and the migration instruction is executed to migrate the updated data record from the initial partition to the target partition.
[0027] Further, after obtaining the target partition corresponding to the target partition field and migrating the updated data record from the initial partition to the target partition, the partition migration method includes:
[0028] A deletion instruction is generated, and the deletion instruction is executed to delete the data records stored in the initial partition.
[0029] In a second aspect, a partition migration device is provided, including:
[0030] A first acquisition module is used to connect to a data source and acquire incremental data from the data source;
[0031] A second acquisition module is used to acquire a target partition field in the incremental data, acquire a metadata table in the data lake, and acquire an initial partition field stored in the metadata table;
[0032] A third acquisition module, configured to acquire an initial partition when the target partition field and the initial partition field meet a preset condition;
[0033] A fourth acquisition module, used to acquire the data records stored in the initial partition;
[0034] An updating module, used for updating the data record based on the target partition field and a preset updating method to obtain the updated data record;
[0035] The migration module is used to obtain the target partition corresponding to the target partition field and migrate the updated data record from the initial partition to the target partition.
[0036] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned partition migration method when executing the computer program.
[0037] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned partition migration method are implemented.
[0038] The present application provides a partition migration method, device, computer equipment and storage medium, which connect to a data source, obtain incremental data from the data source; obtain a target partition field in the incremental data, obtain a metadata table in a data lake, and obtain an initial partition field stored in the metadata table; when the target partition field and the initial partition field meet a preset condition, obtain the initial partition; obtain the data record stored in the initial partition; based on the target partition field and a preset update method, update the data record to obtain the updated data record; obtain the target partition corresponding to the target partition field, and migrate the updated data record from the initial partition to the target partition, which has two beneficial effects. On the one hand, the target partition corresponding to the target partition field is obtained, and the updated data record is migrated from the initial partition to the target partition. Since manual migration is not required, the migration time of the data record stored in the initial partition is reduced, which is conducive to improving the migration efficiency of the data record stored in the initial partition; on the other hand, the automatic migration of the data record stored in the initial partition is realized, which will not be affected by manual intervention, so it is conducive to enhancing the stability of the updated data record during the migration process. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.
[0040] Figure 1 is a schematic diagram of an application environment of a partition migration method in an embodiment of the present invention;
[0041] Figure 2 A schematic diagram of a flow chart of a partition migration method provided by an embodiment of the present invention;
[0042] Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S23;
[0043] Figure 4 yes Figure 1 A schematic flow chart of a specific implementation of step S25;
[0044] Figure 5 yes Figure 1 A schematic flow chart of a specific implementation of step S26;
[0045] Figure 6 is a schematic diagram of a structure of a partition migration device in one embodiment of the present invention;
[0046] Figure 7 is a schematic diagram of a structure of a computer device in one embodiment of the present invention;
[0047] Figure 8 It is another structural schematic diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION
[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0049] See also Figure 1 , Figure 1 FIG. 1 is a schematic diagram of an application environment of a partition migration method according to an embodiment of the present invention. The partition migration method provided by the embodiment of the present invention can be applied in the following embodiments: Figure 1 In an application environment, a client communicates with a server through a network.
[0050] The server connects to the data source through the client and obtains incremental data from the data source;
[0051] Obtain a target partition field in the incremental data, obtain a metadata table in the data lake, and obtain an initial partition field stored in the metadata table;
[0052] When the target partition field and the initial partition field meet a preset condition, obtaining an initial partition;
[0053] Acquire the data records stored in the initial partition;
[0054] Based on the target partition field and a preset update method, the data record is updated to obtain the updated data record;
[0055] A target partition corresponding to the target partition field is obtained, and the updated data record is migrated from the initial partition to the target partition.
[0056] In the scheme implemented by the above-mentioned partition migration method, device, equipment and medium, the beneficial effects are in two aspects. On the one hand, the target partition corresponding to the target partition field is obtained, and the updated data record is migrated from the initial partition to the target partition. Since no manual migration is required, the migration time of the data records stored in the initial partition is reduced, which is conducive to improving the migration efficiency of the data records stored in the initial partition; on the other hand, the automatic migration of the data records stored in the initial partition is realized, which will not be affected by manual intervention, so it is conducive to enhancing the stability of the updated data records during the migration process.
[0057] The device running the client is referred to as a client device.
[0058] The device running the server is referred to as the server device.
[0059] The client devices may include but are not limited to various personal computers, laptops, smart phones, tablet computers, and portable wearable devices.
[0060] The server device can be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.
[0061] See also Figure 2 , Figure 2 A schematic flow chart of a partition migration method provided in an embodiment of the present invention includes the following steps:
[0062] S21, connecting to a data source, and obtaining incremental data from the data source;
[0063] Exemplarily, connecting to a data source and obtaining incremental data from the data source includes:
[0064] Obtain connection information of the data source, connect to the data source according to the connection information by controlling the data lake tool, and obtain a connection result;
[0065] When the connection result is successful, incremental data is obtained from the data source.
[0066] Data lake tools refer to a series of software and tools used to build, manage and utilize data lakes. Data lakes are centralized data storage and processing platforms that can store massive amounts of data from multiple sources and types, including structured data, semi-structured data and unstructured data.
[0067] Among them, data lake tools include but are not limited to Hudi and Apache Spark.
[0068] Preferably, the data lake tool adopts Hudi.
[0069] Among them, Hudi is an open source data lake storage framework. Hudi provides efficient management, update, deletion, and incremental processing capabilities for large analytical data sets.
[0070] Among them, the full English name of Hudi is: Hadoop Upserts Deletes and Incrementals. Hudi supports distributed file systems and provides three logical views for query access, making it easier and more efficient to manage data in the data lake.
[0071] Among them, Apache Spark is an open source distributed processing engine that provides fast and general large-scale data processing capabilities. Apache Spark allows users to easily read, transform, aggregate data, and train and deploy complex statistical models.
[0072] S22, obtaining a target partition field in the incremental data, obtaining a metadata table in the data lake, and obtaining an initial partition field stored in the metadata table;
[0073] The step of obtaining the target partition field in the incremental data, obtaining the metadata table in the data lake, and obtaining the initial partition field stored in the metadata table includes:
[0074] Obtain the primary key and target partition field in the incremental data, and obtain the metadata table in the data lake through the query interface provided by the data lake;
[0075] Obtain a query request for the primary key, and query whether the primary key exists in the metadata table according to the query request;
[0076] When the metadata table has the primary key, the initial partition field stored in the metadata table is obtained.
[0077] The primary key is a combination of one or more fields in the database table, which can uniquely identify each row of data records in the metadata table. When the metadata table has the primary key, it means that the data records in the incremental data are included, and the data records are uniquely guaranteed by the constraints of the primary key.
[0078] S23, when the target partition field and the initial partition field meet a preset condition, obtaining an initial partition;
[0079] The target partition field is the field of the target partition.
[0080] The initial partition field is the field of the initial partition.
[0081] The initial partition refers to the partition where data records are initially stored in the data lake.
[0082] The target partition refers to the partition in the data lake where data records are ultimately stored.
[0083] Among them, the target partition is relative to the initial partition, and the target partition is usually determined during data migration, data integration, data loading or data query.
[0084] When the data lake is ready to save data records, it determines which partition the data records should be stored in based on certain rules or requirements. This partition is the target partition. By clarifying the target partition, the data lake can manage data more effectively, increase the speed of data access, and ensure that the organization of data meets your business needs.
[0085] S24, obtaining the data records stored in the initial partition;
[0086] The step of obtaining the data records stored in the initial partition includes:
[0087] Access the initial partition through an access interface provided by the data lake;
[0088] A read operation is performed on the data block of the initial partition to obtain the data record stored in the initial partition.
[0089] Among them, by accessing the initial partition through the access interface provided by the data lake, the initial partition can be quickly located, avoiding a comprehensive scan of the entire data lake, thereby greatly shortening the positioning time.
[0090] The data block of the initial partition is read to obtain the data record stored in the initial partition. The data block is the basic unit of data storage. By directly reading the data block, the system's parsing and processing time for the data record can be reduced, thereby improving the reading speed of the data record.
[0091] S25, updating the data record based on the target partition field and a preset update method to obtain the updated data record;
[0092] Exemplarily, based on the target partition field and a preset update method, the data record is updated to obtain the updated data record, including:
[0093] The target partition field and the data record are concatenated to obtain the updated data record.
[0094] Among them, the target partition field enhances the manageability and scalability of the updated data record. The target partition field and the data record are spliced to obtain the updated data record. Since the partition strategy can be easily adjusted through the target partition field, even if there is an increase in data volume and changes in business needs, the updated data record can adapt to new data storage and management needs.
[0095] S26, obtaining a target partition corresponding to the target partition field, and migrating the updated data record from the initial partition to the target partition.
[0096] Among them, the initial partition and the target partition are different partitions.
[0097] For the sake of illustration, an example is given below:
[0098] For example, the data lake has multiple partitions, namely partition 1, partition 2, partition 3, partition 4, and partition 5. When the initial partition is partition 1, the target partition is any one of partition 2, partition 3, partition 4, and partition 5.
[0099] For example, the data lake has multiple partitions, namely partition 1, partition 2, partition 3, partition 4, and partition 5. When the initial partition is partition 2, the target partition is any one of partition 1, partition 3, partition 4, and partition 5.
[0100] For example, the data lake has multiple partitions, namely partition 1, partition 2, partition 3, partition 4, partition 5, partition 6, and partition 7. When the initial partition is partition 1, the target partition is any one of partition 2, partition 3, partition 4, partition 5, partition 6, and partition 7.
[0101] For example, the data lake has multiple partitions, namely partition 1, partition 2, partition 3, partition 4, partition 5, partition 6, and partition 7. When the initial partition is partition 7, the target partition is any one of partition 1, partition 3, partition 4, partition 5, partition 6, and partition 6.
[0102] Among them, by clarifying the target partition where the data should be stored, the data source can accurately locate the migration path of the updated data records, reduce unnecessary intermediate links and data redundancy, and thus significantly improve the migration efficiency of the updated data records.
[0103] Wherein, after obtaining the target partition corresponding to the target partition field and migrating the updated data record from the initial partition to the target partition, the partition migration method includes:
[0104] A deletion instruction is generated, and the deletion instruction is executed to delete the data records stored in the initial partition.
[0105] Deleting the data records stored in the initial partition can cause data duplication between the initial partition and the target partition.
[0106] The embodiments of the present invention can be applied to data lakes, and can improve the data lake's ability to support frequently changing scenarios, ensure that the data lake can quickly adapt to rapid changes in business needs, maintain the timeliness and accuracy of incremental data processing, and provide timely and reliable information support for decision-making.
[0107] In the embodiment of the present invention, the beneficial effects lie in two aspects. On the one hand, the target partition corresponding to the target partition field is obtained, and the updated data records are migrated from the initial partition to the target partition. Since no manual migration is required, the migration time of the data records stored in the initial partition is reduced, which is beneficial to improving the migration efficiency of the data records stored in the initial partition. On the other hand, the automatic migration of the data records stored in the initial partition is realized, which will not be affected by manual intervention, so it is beneficial to enhance the stability of the updated data records during the migration process.
[0108] See also Figure 3 , Figure 3 yes Figure 1 A specific implementation flow diagram of step S23 is described in detail as follows:
[0109] S31, obtaining a field value in the target partition field, and obtaining a field value in the initial partition field;
[0110] S32: When the field value in the target partition field is inconsistent with the field value in the initial partition field, obtain the initial partition corresponding to the field value in the initial partition field.
[0111] In an embodiment of the present invention, when the field value in the target partition field is inconsistent with the field value in the initial partition field, the initial partition corresponding to the field value in the initial partition field is obtained, so that the initial partition can be located and access to the initial partition is facilitated.
[0112] See also Figure 4 , Figure 4 yes Figure 1 A specific implementation flow diagram of step S25 is described in detail as follows:
[0113] S41, obtaining an update instruction, executing the update instruction, and updating the initial partition field in the data record to the target partition field;
[0114] S42: Select the data record using the target partition field as the updated data record.
[0115] In an embodiment of the present invention, the data record using the target partition field is selected as the updated data record. The target partition field is often more in line with current data access characteristics and business needs, and can more effectively reduce query time, improve data retrieval flexibility, and optimize resource utilization.
[0116] See also Figure 5 , Figure 5 yes Figure 1 A specific implementation flow diagram of step S26 is described in detail as follows:
[0117] S51, obtaining a target partition corresponding to the target partition field, and accessing the target partition corresponding to the target partition field;
[0118] S52, generating a migration instruction, executing the migration instruction, and migrating the updated data record from the initial partition to the target partition.
[0119] Among them, executing the migration instruction to migrate the updated data record from the initial partition to the target partition can allow the updated data record to be transferred between the initial partition and the target partition, thereby reducing the risk of human operation errors and improving the accuracy and reliability of the updated data record.
[0120] The updated data records are migrated from the initial partition to the target partition, and the target partition can organize the updated data records more accurately, so that the query operation can locate the required data range more quickly, significantly improving the query performance.
[0121] In an embodiment of the present invention, a migration instruction is generated and executed to migrate the updated data records from the initial partition to the target partition, thereby realizing automatic migration of the data records stored in the initial partition without being affected by human intervention, and thus helping to enhance the stability of the updated data records during the migration process.
[0122] See also Figure 6 , Figure 6 is a schematic diagram of a partition migration device in one embodiment of the present invention. Figure 6 As shown, the partition migration device includes a first acquisition module 101, a second acquisition module 102, a third acquisition module 103, a fourth acquisition module 104, an update module 105, and a migration module 106. The functional modules are described in detail as follows:
[0123] The first acquisition module 101 is used to connect to a data source and acquire incremental data from the data source;
[0124] The second acquisition module 102 is used to acquire the target partition field in the incremental data, acquire the metadata table in the data lake, and acquire the initial partition field stored in the metadata table;
[0125] A third acquisition module 103, configured to acquire an initial partition when the target partition field and the initial partition field meet a preset condition;
[0126] A fourth acquisition module 104, used to acquire the data records stored in the initial partition;
[0127] An updating module 105, configured to update the data record based on the target partition field and a preset updating method to obtain the updated data record;
[0128] The migration module 106 is used to obtain the target partition corresponding to the target partition field, and migrate the updated data record from the initial partition to the target partition.
[0129] In one embodiment, the second acquisition module 102 includes:
[0130] A first acquisition subunit is used to acquire the primary key and target partition field in the incremental data, and acquire the metadata table in the data lake through the query interface provided by the data lake;
[0131] A query subunit, used for obtaining a query request for the primary key, and querying whether the primary key exists in the metadata table according to the query request;
[0132] The second acquisition subunit is used to acquire the initial partition field stored in the metadata table when the metadata table has the primary key.
[0133] In one embodiment, the third acquisition module 103 includes:
[0134] A third acquisition subunit is used to acquire the field value in the target partition field and the field value in the initial partition field;
[0135] The comparison subunit is used to obtain the initial partition corresponding to the field value in the initial partition field when the field value in the target partition field is inconsistent with the field value in the initial partition field.
[0136] In one embodiment, the fourth acquisition module 104 includes:
[0137] A first access subunit, configured to access the initial partition through an access interface provided by the data lake;
[0138] The fourth acquisition subunit is used to perform a read operation on the data block of the initial partition to obtain the data record stored in the initial partition.
[0139] In one embodiment, the updating module 105 includes:
[0140] An update subunit, configured to obtain an update instruction, execute the update instruction, and update the initial partition field in the data record to the target partition field;
[0141] The selection subunit is used to select the data record using the target partition field as the updated data record.
[0142] In one embodiment, the migration module 106 includes:
[0143] An access subunit, used to obtain a target partition corresponding to the target partition field, and access the target partition corresponding to the target partition field;
[0144] The migration subunit is used to generate a migration instruction, execute the migration instruction, and migrate the updated data record from the initial partition to the target partition.
[0145] In one embodiment, the partition migration device further includes:
[0146] The deletion module is used to generate a deletion instruction, execute the deletion instruction, and delete the data records stored in the initial partition.
[0147] In the embodiment of the present invention, the beneficial effects lie in two aspects. On the one hand, the target partition corresponding to the target partition field is obtained, and the updated data records are migrated from the initial partition to the target partition. Since no manual migration is required, the migration time of the data records stored in the initial partition is reduced, which is beneficial to improving the migration efficiency of the data records stored in the initial partition. On the other hand, the automatic migration of the data records stored in the initial partition is realized, which will not be affected by manual intervention, so it is beneficial to enhance the stability of the updated data records during the migration process.
[0148] For the specific definition of the partition migration device, please refer to the definition of the partition migration method above, which will not be repeated here.
[0149] Each module in the partition migration device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0150] See also Figure 7 , Figure 7 is a schematic diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a server device, and its internal structure diagram may be as shown in FIG. Figure 7 The computer device includes a processor, a memory, a network interface and a database connected through a system bus.
[0151] The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the computer program is executed by the processor, a function or step of a partition migration method can be implemented on the server device.
[0152] See also Figure 8 , Figure 8 is another structural diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device may be a client device, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. When the computer program is executed by the processor, a function or step of a partition migration method can be implemented on the client device.
[0153] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor.
[0154] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description in the aforementioned method embodiment. In order to avoid repetition, they will not be described one by one here.
[0155] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.
[0156] The above description and accompanying drawings fully illustrate the embodiments of the present disclosure so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. Parts and subsamples of some embodiments may be included in or replace parts and subsamples of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of a stated sub-sample, whole, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other sub-samples, wholes, steps, operations, elements, components and / or groups of these. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between the various embodiments may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.
[0157] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.
[0158] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some sub-samples can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.
[0159] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.
Claims
1. A partition migration method, characterized in that: include: Connecting to a data source, and obtaining incremental data from the data source; Obtain a target partition field in the incremental data, obtain a metadata table in the data lake, and obtain an initial partition field stored in the metadata table; When the target partition field and the initial partition field meet a preset condition, obtaining an initial partition; Acquire the data records stored in the initial partition; Based on the target partition field and a preset update method, the data record is updated to obtain the updated data record; A target partition corresponding to the target partition field is obtained, and the updated data record is migrated from the initial partition to the target partition.
2. The partition migration method according to claim 1, characterized in that: The obtaining of the target partition field in the incremental data, obtaining the metadata table in the data lake, and obtaining the initial partition field stored in the metadata table includes: Obtain the primary key and target partition field in the incremental data, and obtain the metadata table in the data lake through the query interface provided by the data lake; A query request for the primary key is obtained, and whether the metadata table contains the primary key is queried according to the query request; when the metadata table contains the primary key, an initial partition field stored in the metadata table is obtained.
3. The partition migration method according to claim 1, characterized in that: When the target partition field and the initial partition field meet a preset condition, obtaining the initial partition includes: Obtaining the field value in the target partition field, and obtaining the field value in the initial partition field; When the field value in the target partition field is inconsistent with the field value in the initial partition field, the initial partition corresponding to the field value in the initial partition field is obtained.
4. The partition migration method according to claim 1, characterized in that: The obtaining of the data records stored in the initial partition includes: Access the initial partition through an access interface provided by the data lake; A read operation is performed on the data block of the initial partition to obtain the data record stored in the initial partition.
5. The partition migration method according to claim 1, characterized in that: The updating of the data record based on the target partition field and the preset updating mode to obtain the updated data record includes: Obtain an update instruction, execute the update instruction, and update the initial partition field in the data record to the target partition field; The data record using the target partition field is selected as the updated data record.
6. The partition migration method according to claim 1, characterized in that: The obtaining the target partition corresponding to the target partition field and migrating the updated data record from the initial partition to the target partition includes: Obtain the target partition corresponding to the target partition field, and access the target partition corresponding to the target partition field; A migration instruction is generated, and the migration instruction is executed to migrate the updated data record from the initial partition to the target partition.
7. The partition migration method according to any one of claims 1 to 6, characterized in that: After obtaining the target partition corresponding to the target partition field and migrating the updated data record from the initial partition to the target partition, the partition migration method includes: A deletion instruction is generated, and the deletion instruction is executed to delete the data records stored in the initial partition.
8. A partition migration device, characterized in that: include: A first acquisition module is used to connect to a data source and acquire incremental data from the data source; A second acquisition module is used to acquire a target partition field in the incremental data, acquire a metadata table in the data lake, and acquire an initial partition field stored in the metadata table; A third acquisition module, configured to acquire an initial partition when the target partition field and the initial partition field meet a preset condition; A fourth acquisition module, used to acquire the data records stored in the initial partition; An updating module, used for updating the data record based on the target partition field and a preset updating method to obtain the updated data record; The migration module is used to obtain the target partition corresponding to the target partition field and migrate the updated data record from the initial partition to the target partition.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the partition migration method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the partition migration method according to any one of claims 1 to 7 are implemented.