A method, device, equipment and medium for automatically updating data in a data warehouse partition

Abstract

The present invention relates to a method, apparatus, device, and medium for automatically updating partitioned data in a data warehouse. The method comprises: obtaining at least one type of change data corresponding to a target partition table in a data warehouse, and querying historical data corresponding to various types of change data in the target partition table; for each type of change data and corresponding historical data, determining target data for the change data based on the change data and the corresponding historical data, wherein the target data satisfies preset update requirements; for each type of change data and corresponding historical data, updating data in the historical data that is in the same partition as the target data based on the target data; and repartitioning the target partition table based on the updated data of various types. The present application implements data updates and merging of newly added data in a target partition table in a Hive data warehouse, effectively solving the problems of historical data updates and data merging in a Hive data warehouse.

Description

Technical Field

[0001] The present invention relates to the technical field of data updating, and in particular to a method, device, equipment and medium for automatically updating partitioned data in a data warehouse. Background Art

[0002] In the era of rapid development of big data, the technology of building a Hadoop file storage system to store massive data and using Hive to analyze and mine massive structured data has been widely used, but Hive is not good at updating data.

[0003] In real-world business systems, data modifications (e.g., order data) or data warehousing delays can occur, significantly impacting subsequent operations such as statistical analysis. Hive is a data warehouse analysis tool based on HDFS (Hadoop Distributed File System), which does not support modifications to existing data files. Currently, Hive's support for operations such as update and delete operations on HDFS is poor. To ensure accurate data modification and subsequent statistical analysis, a solution for updating partitioned data is urgently needed. Summary of the Invention

[0004] In order to update historical partition data, the present application provides a method, device, equipment and medium for automatically updating data in a data warehouse partition.

[0005] In a first aspect, the present application provides a method for automatically updating partitioned data in a data warehouse, comprising:

[0006] Acquire at least one type of change data corresponding to a target partition table in a data warehouse, and query the target partition table for historical data corresponding to various types of the change data;

[0007] For each type of the changed data and the corresponding historical data, determining target data for the changed data according to the changed data and the corresponding historical data, wherein the target data meets a preset update requirement;

[0008] For each type of the changed data and the corresponding historical data, updating the data in the same partition as the target data in the historical data according to the target data, the historical data including data of at least one partition, each of the partitions corresponding to at least one type of data;

[0009] Based on the updated data of various types, the target partition table is repartitioned.

[0010] The beneficial effects of the present invention are as follows: the queried target data is overwritten with the corresponding historical data according to the update requirements, and the various types of updated data in the target partition table are repartitioned, thereby realizing the data update of the target partition table in the Hive data warehouse and the merging of the newly added data, thereby effectively solving the problems of historical data update and data merging in the Hive data warehouse, ensuring the accuracy and reliability of the data in the data warehouse, and providing effective protection for the analysis and mining of subsequent data.

[0011] Furthermore, the target partition table includes data of different partitions, each partition corresponds to a creation date, and obtaining at least one type of change data corresponding to the target partition table in the data warehouse and querying historical data corresponding to each type of the change data in the target partition table include:

[0012] Parsing the change data log to obtain a parsing result, wherein the parsing result includes the change data, the type of the change data, and the creation date of the change data, and the change data log includes information indicating the data type and the creation date of the change data;

[0013] For each type of the changed data, based on a creation date of the changed data, determining an original partition corresponding to the changed data from a target partition table in the data warehouse, wherein the original partition includes at least one partition, and each partition in the original partition includes data of the same type as the changed data;

[0014] For each type of the changed data, the data corresponding to the original partition is determined as the historical data corresponding to the type of changed data.

[0015] The beneficial effect of adopting the above further solution is: by creating the original partition related to the date query, the corresponding historical data is obtained, which improves the query performance and facilitates the search for historical data.

[0016] Furthermore, for each type of the change data and the corresponding historical data, determining target data of the change data according to the change data and the corresponding historical data includes:

[0017] For each type of the change data and the corresponding historical data, merging the change data and the historical data corresponding to the type of change data to obtain merged data;

[0018] For each type of the changed data and the corresponding historical data, the merged data is screened according to the update requirements corresponding to the changed data of that type to obtain the target data.

[0019] The beneficial effect of adopting the above further solution is: since the data on the HDFS file system is stored in text format, once the data is generated, it is impossible to directly update a single piece of data. Therefore, by merging the data in all partitions corresponding to the date list and the changed data of each type, it is convenient to update each type of data.

[0020] Furthermore, for each group of data corresponding to each type of the changed data, updating the data in the same partition as the target data in the historical data according to the target data includes:

[0021] For the merged data corresponding to each type of the changed data, other data except the target data in the merged data is overwritten according to the target data corresponding to the changed data of this type, so as to complete the update of the target data.

[0022] The beneficial effect of adopting the above further solution is that, among the merged data of the same type, the target data that meets the update requirements is retained, thereby realizing the data update of the target partition table and the merging of the newly added data.

[0023] Furthermore, the method further comprises:

[0024] Generate a temporary partition according to the parsing result of the change data log, wherein the temporary partition includes various types of change data;

[0025] The parsing result also includes the creation date of each of the changed data. For each type of the changed data, determining the original partition corresponding to the changed data from the target partition table in the data warehouse includes:

[0026] For each type of changed data, a partition in the target partition table corresponding to the creation date of the changed data is searched, and the partition is used as the original partition corresponding to the changed data.

[0027] The beneficial effect of adopting the above further solution is that the corresponding original partition is searched according to the creation date of the changed data, thereby improving the convenience of searching the original partition.

[0028] In a second aspect, the present application provides a data warehouse partition data automatic update device, comprising:

[0029] An acquisition query module is used to obtain at least one type of change data corresponding to a target partition table in a data warehouse, and query the historical data corresponding to various types of the change data in the target partition table;

[0030] a determination module, configured to determine, for each type of the change data and the corresponding historical data, target data of the change data according to the change data and the corresponding historical data, wherein the target data satisfies a preset update requirement;

[0031] an updating module, configured to update, for each type of the changed data and the corresponding historical data, data in the historical data in the same partition as the target data according to the target data, wherein the historical data includes data of at least one partition, and each partition corresponds to at least one type of data;

[0032] The repartitioning module is used to repartition the target partition table based on the updated data of various types.

[0033] In a third aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the processor is coupled to the memory;

[0034] The processor is configured to execute a computer program stored in the memory, so that the electronic device executes the method according to any one of the first aspects.

[0035] In a fourth aspect, the present application provides a computer-readable storage medium comprising a computer program or instructions, which, when executed on a computer, causes the computer to execute the method as described in any one of the first aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a flow chart of a method for automatically updating partitioned data in a data warehouse according to an embodiment of the present application;

[0037] Figure 2 This is a structural block diagram of a device for automatically updating data in a data warehouse partition according to an embodiment of the present application;

[0038] Figure 3 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0039] The present application is further described in detail below with reference to the accompanying drawings.

[0040] Embodiments of the present application provide a method for automatically updating data in a data warehouse partition. The method can be executed by a device, which can be a server or a terminal device. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be, but is not limited to, a smartphone, a tablet computer, or a desktop computer.

[0041] like Figure 1As shown, a method for automatically updating partitioned data in a data warehouse is described, with an electronic device as the execution subject. The main process of the method is described as follows (steps S101 to S104):

[0042] Step S101: obtaining at least one type of change data corresponding to a target partition table in a data warehouse, and querying the target partition table for historical data corresponding to various types of the change data.

[0043] In this embodiment, a data warehouse refers to a centralized repository that can store, manage, and query various structured data. A data warehouse can be established by using Hive, a data warehouse tool built on the Hadoop distributed system. Users can create data warehouses by using Hive.

[0044] In a data warehouse, a target partitioned table refers to a specific, partitioned data table used to store and manage business data within the data warehouse. This table can be designed based on business needs and data characteristics, and data can be distributed across multiple physical storage locations (i.e., partitions) according to a partitioning strategy (such as time range, geographic location, creation date, etc.). It's easy to understand that a target partitioned table contains multiple partitions.

[0045] In this embodiment, change data refers to data records that have been modified or updated in the data warehouse. Change data can be generated by business operations, transaction processing, or other data update activities. Historical data refers to data records that already exist in the data warehouse. Historical data is the original data stored in the data warehouse to support analysis, reporting, and historical queries.

[0046] For example, in an order data warehouse, change data may include order records of newly placed orders or status updates of existing orders, and historical data may include order records of all previous orders, which may have changed over a period of time or remained unchanged.

[0047] By using the type of the changed data, historical data of the same type is obtained, which improves the accuracy of obtaining historical data and facilitates subsequent data analysis and mining. In this embodiment, the type can be order number, that is, the data with the same order number as the order number corresponding to the current changed data in the target partition table is selected as the historical data.

[0048] Step S102: For each type of the changed data and the corresponding historical data, determine target data of the changed data according to the changed data and the corresponding historical data, where the target data meets a preset update requirement.

[0049] In this embodiment, the update requirement can be to select the data with the latest update time, so the data with the latest update time among the change data and the corresponding historical data is used as the target data. It is easy to understand that the update requirement can be adjusted according to actual conditions, and different update requirements will result in different target data.

[0050] It is easy to understand that the Creation Date refers to the date and time when the data was created. For each piece of data, there will be an associated Creation Date field to identify when the data was created. The Creation Date can be used to determine the newness of the data.

[0051] When comparing the creation date of the change data with the historical data of the same type, the data with the latest update time is used as the target data, so that the target data can represent the latest state of the change data and the corresponding historical data. In this embodiment, the target data can be data in the change data.

[0052] Step S103: For each type of the changed data and the corresponding historical data, update the data in the same partition as the target data in the historical data according to the target data, where the historical data includes data of at least one partition, and each partition corresponds to at least one type of data.

[0053] In this embodiment, when the target data is data in the changed data, the partition corresponding to the target data refers to the partition in which the target data is stored according to the partition key.

[0054] Step S104: repartitioning the target partition table based on the updated data of various types.

[0055] In this embodiment, the queried target data is overwritten with the corresponding historical data according to the update requirements such as the actual creation date of the data, and the various types of updated data in the target partition table are repartitioned in a dynamic partitioning manner, thereby realizing the data update of the target partition table in the Hive data warehouse and the merging of the new data, thereby effectively solving the problem of historical data update and data merging in the Hive data warehouse, ensuring the accuracy and reliability of the data in the data warehouse, and providing effective protection for the subsequent data analysis and mining.

[0056] In this embodiment, the target partition table includes data of different partitions, and each partition corresponds to a creation date. Step S101 specifically includes the following processing: parsing the change data log to obtain a parsing result, wherein the parsing result includes the change data, the type of the change data and the creation date of the change data, and the change data log includes information indicating the data type and the creation date of the change data; for each type of the change data, based on the creation date of the change data, determining the original partition corresponding to the change data from the target partition table in the data warehouse, wherein the original partition includes at least one partition, and each partition in the original partition includes data of the same type as the change data; for each type of the change data, determining the data corresponding to the original partition as the historical data corresponding to the change data of this type.

[0057] In this embodiment, it is necessary to perform regular daily analysis of the changed data. This analysis can be performed by extracting the changed data from a data source (e.g., a log file, a data stream, etc.). In a data warehouse, partitions in a target partition table are typically divided based on a partition key. The partition key may include, for example, a creation date, a time, a geographic location, etc. In this embodiment, the partition key is the creation date, so that each partition stores data corresponding to a specific creation date.

[0058] In this embodiment, to reduce the direct impact on the data warehouse and improve overall performance and stability, a temporary partition is generated under the target partition table based on the parsing results of the change data log. The change data is temporarily stored in the temporary partition under the target partition table. The temporary partition can include multiple types of change data. The temporary partition is a subset of the target partition table and serves as a temporary storage area for change data. By initially storing change data in the temporary partition, the load on the data warehouse and the need for locking can be reduced, thereby improving overall performance.

[0059] For each type of the changed data, based on the creation date of the changed data, the original partition corresponding to the changed data is determined from the target partition table in the data warehouse, which specifically includes the following processing: for each type of changed data, the partition in the target partition table corresponding to the creation date of the changed data is searched, and the partition is used as the original partition corresponding to the changed data.

[0060] In this embodiment, searching for the partition in the target partition table corresponding to the creation date of the changed data specifically includes: querying the actual creation date corresponding to each of the changed data, generating a date list based on each of the actual creation dates; and, based on the date list, obtaining the partition in the target partition table corresponding to each date in the date list. For example, the creation date may be expressed in the format of xx / xx / xx / xx, while the data update time may be expressed in the format of xx / xx / xx / xx. For a partition containing historical data corresponding to a certain type of changed data, the corresponding partition date may be the same as the actual creation date of the changed data, while the update time of the changed data and the corresponding historical data may be different.

[0061] By querying related partitions through the date list, the corresponding historical data can be obtained without scanning the entire target partition table, reducing the amount of data that needs to be processed and thus improving query performance.

[0062] In this embodiment, step S102 specifically includes the following processing: for each type of the change data and the corresponding historical data, the change data and historical data corresponding to the change data of this type are merged to obtain merged data; for each type of the change data and the corresponding historical data, according to the update requirements corresponding to the change data of this type, the merged data is filtered to obtain the target data.

[0063] By merging the data in all partitions corresponding to the date list and the change data of each type, a new data set is generated. Then, based on the type corresponding to the data, the data in the new data set can be grouped and divided to obtain at least one group of merged data containing the same type of change data and historical data. The number of groups of merged data is the same as the number of data types.

[0064] Since data on the HDFS file system is stored in text format, a single piece of data cannot be directly updated once it is generated. Therefore, by merging the data in all partitions corresponding to the date list and the changed data of each type, it is easier to update each type of data.

[0065] In this embodiment, step S103 specifically includes the following processing: for each type of merged data corresponding to the changed data, overwriting other data in the merged data except the target data according to the target data corresponding to the changed data of that type to complete the update of the target data.

[0066] In this embodiment, the merged data includes one or more data corresponding to the same order number; when there are multiple data, the latest one is taken as the target data according to the update time, so as to achieve the purpose of updating the data; when there is only one data, that is, the corresponding order number has no new changes, when taking the latest data according to the update time, the historical data is used as the target data.

[0067] Based on the same technical concept, the present application also provides a data warehouse partition data automatic update device, such as Figure 2 As shown, the data warehouse partition data automatic update device 200 mainly includes:

[0068] An acquisition and query module 201 is configured to acquire at least one type of change data corresponding to a target partition table in a data warehouse, and query the target partition table for historical data corresponding to the various types of change data;

[0069] A determination module 202 is configured to determine, for each type of the change data and the corresponding historical data, target data of the change data according to the change data and the corresponding historical data, wherein the target data satisfies a preset update requirement;

[0070] An updating module 203 is configured to update, for each type of the changed data and the corresponding historical data, data in the historical data that is in the same partition as the target data according to the target data, wherein the historical data includes data of at least one partition, and each partition corresponds to at least one type of data;

[0071] The repartitioning module 204 is configured to repartition the target partition table based on the updated data of various types.

[0072] Optionally, the target partition table includes data of different partitions, each partition corresponds to a creation date, and the acquisition query module 201 includes:

[0073] a parsing submodule configured to parse the change data log to obtain a parsing result, wherein the parsing result includes the change data, the type of the change data, and the creation date of the change data, and the change data log includes information indicating the type of data and the creation date of the change data;

[0074] a first determining submodule, configured to determine, for each type of the changed data, an original partition corresponding to the changed data from a target partition table in the data warehouse based on a creation date of the changed data, wherein the original partition includes at least one partition, and each partition in the original partition includes data of the same type as the changed data;

[0075] The second determining submodule is configured to determine, for each type of the changed data, the data corresponding to the original partition as the historical data corresponding to the type of changed data.

[0076] Optionally, the determining module 202 includes:

[0077] a merging submodule, configured to merge the change data and the historical data corresponding to each type of the change data to obtain merged data;

[0078] The screening submodule is used to screen the merged data for each type of the changed data and the corresponding historical data according to the update requirements corresponding to the type of changed data to obtain the target data.

[0079] Optionally, the updating module 203 includes:

[0080] The overwriting and updating submodule is used to overwrite the other data except the target data in the merged data corresponding to each type of the changed data according to the target data corresponding to the changed data of this type, so as to complete the update of the target data.

[0081] Optionally, a temporary partition is generated according to the parsing result of the change data log, and the temporary partition includes various types of change data; the first determination submodule includes:

[0082] The search submodule is used to search for the partition in the target partition table corresponding to the creation date of each type of changed data, and use the partition as the original partition corresponding to the changed data.

[0083] In one example, the module in any of the above devices can be one or more integrated circuits configured to implement the above methods, such as: one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms.

[0084] For another example, when the modules in the device can be implemented in the form of a processing element scheduling program, the processing element can be a general-purpose processor, such as a central processing unit (CPU) or other processor capable of calling programs. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0085] Various objects such as messages / information / equipment / network elements / systems / devices / actions / operations / processes / concepts that may appear in this application are named. It is understandable that these specific names do not constitute a limitation on the relevant objects. The names assigned may change with factors such as scenarios, contexts or usage habits. The understanding of the technical meaning of the technical terms in this application should be mainly determined from the functions and technical effects embodied / executed in the technical solutions.

[0086] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0087] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0088] Based on the same technical concept, the present application also provides an electronic device, such as Figure 3 As shown, the electronic device 300 includes a processor 301 and a memory 302 , and may further include an information input / information output (I / O) interface 303 , one or more communication components 304 , and a communication bus 305 .

[0089] The processor 301 is used to control the overall operation of the electronic device 300 to complete all or part of the steps in the above-mentioned method for automatically updating data in a data warehouse partition. The memory 302 is used to store various types of data to support the operation of the electronic device 300. Such data may include, for example, instructions for any application or method operating on the electronic device 300, as well as application-related data. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as one or more of static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0090] The I / O interface 303 provides an interface between the processor 301 and other interface modules, which may be a keyboard, a mouse, buttons, etc. These buttons may be virtual buttons or physical buttons. The communication component 304 is used to test wired or wireless communication between the electronic device 300 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more thereof, may include: a Wi-Fi component, a Bluetooth component, and an NFC component.

[0091] Communication bus 305 may include a path for transmitting information between the aforementioned components. Communication bus 305 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, for example. Communication bus 305 may be divided into an address bus, a data bus, a control bus, and the like.

[0092] The electronic device 300 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the data warehouse partition data automatic update method given in the above embodiment.

[0093] The electronic device 300 may include but is not limited to mobile terminals such as digital broadcast receivers, PDAs (personal digital assistants), PMPs (portable multimedia players), and fixed terminals such as digital TVs and desktop computers, and may also be servers.

[0094] Based on the same technical concept, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned method for automatically updating data in a data warehouse partition are implemented.

[0095] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc., which can store program codes.

[0096] The terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0097] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0098] In the description of this specification, the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" mean that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0099] Although the embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and are not to be construed as limitations on the present invention. A person skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

Claims

1. A method for automatically updating partition data in a data warehouse, characterized in that: include: Acquire at least one type of change data corresponding to a target partition table in a data warehouse, and query the target partition table for historical data corresponding to various types of the change data; For each type of the change data and the corresponding historical data, determining target data for the change data according to the change data and the corresponding historical data, wherein the target data meets a preset update requirement, and the update requirement is to select the data with the latest update time; For each type of the changed data and the corresponding historical data, updating the data in the same partition as the target data in the historical data according to the target data, the historical data including data of at least one partition, each of the partitions corresponding to at least one type of data; Repartitioning the target partition table based on the updated data of various types; The target partition table includes data of different partitions, each partition corresponds to a creation date, and obtaining at least one type of change data corresponding to the target partition table in the data warehouse and querying historical data corresponding to various types of the change data in the target partition table include: Parsing the change data log to obtain a parsing result, wherein the parsing result includes the change data, the type of the change data, and the creation date of the change data, and the change data log includes information indicating the data type and the creation date of the change data; For each type of the changed data, based on a creation date of the changed data, determining an original partition corresponding to the changed data from a target partition table in the data warehouse, wherein the original partition includes at least one partition, and each partition in the original partition includes data of the same type as the changed data; For each type of the changed data, determining the data corresponding to the original partition as the historical data corresponding to the type of changed data; For each type of the change data and the corresponding historical data, determining target data for the change data according to the change data and the corresponding historical data includes: For each type of the change data and the corresponding historical data, merging the change data and the historical data corresponding to the type of change data to obtain merged data; For each type of the changed data and the corresponding historical data, the merged data is screened according to the update requirements corresponding to the changed data of that type to obtain the target data.

2. A method for automatically updating data in a data warehouse partition according to claim 1, characterized in that: For each group of data corresponding to each type of changed data, updating the data in the same partition as the target data in the historical data according to the target data includes: For the merged data corresponding to each type of the changed data, other data except the target data in the merged data is overwritten according to the target data corresponding to the changed data of this type, so as to complete the update of the target data.

3. A method for automatically updating partition data in a data warehouse according to claim 1 or 2, characterized in that: The method further comprises: Generate a temporary partition according to the parsing result of the change data log, wherein the temporary partition includes various types of change data; The parsing result also includes the creation date of each of the changed data. For each type of the changed data, determining the original partition corresponding to the changed data from the target partition table in the data warehouse includes: For each type of changed data, a partition in the target partition table corresponding to the creation date of the changed data is searched, and the partition is used as the original partition corresponding to the changed data.

4. A data warehouse partition data automatic update device, characterized in that: include: An acquisition query module is used to obtain at least one type of change data corresponding to a target partition table in a data warehouse, and query the historical data corresponding to various types of the change data in the target partition table; a determination module configured to determine, for each type of the change data and the corresponding historical data, target data of the change data according to the change data and the corresponding historical data, wherein the target data satisfies a preset update requirement, wherein the update requirement is to select the data with the latest update time; an updating module, configured to update, for each type of the changed data and the corresponding historical data, data in the historical data in the same partition as the target data according to the target data, wherein the historical data includes data of at least one partition, and each partition corresponds to at least one type of data; A repartitioning module, configured to repartition the target partition table based on the updated data of various types; The target partition table includes data of different partitions, each partition corresponds to a creation date, and the acquisition query module includes: a parsing submodule configured to parse the change data log to obtain a parsing result, wherein the parsing result includes the change data, the type of the change data, and the creation date of the change data, and the change data log includes information indicating the type of data and the creation date of the change data; a first determining submodule, configured to determine, for each type of the changed data, an original partition corresponding to the changed data from a target partition table in the data warehouse based on a creation date of the changed data, wherein the original partition includes at least one partition, and each partition in the original partition includes data of the same type as the changed data; A second determining submodule is configured to determine, for each type of the changed data, the data corresponding to the original partition as the historical data corresponding to the changed data of that type; The modules to be determined include: a merging submodule, configured to merge the change data and the historical data corresponding to each type of the change data to obtain merged data; The screening submodule is used to screen the merged data for each type of the changed data and the corresponding historical data according to the update requirements corresponding to the type of changed data to obtain the target data.

5. An electronic device, characterized in that: comprising a processor and a memory, wherein the processor is coupled to the memory; The processor is configured to execute the computer program stored in the memory, so that the electronic device performs the method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that The method comprises a computer program or an instruction, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 3.

Images