A data processing method, device, apparatus, and storage medium

Abstract

Embodiments of the present application provide a data processing method, device and equipment and a storage medium to solve the problem of inconsistency between a data warehouse and a data platform when a part of data is deleted from the data platform. The method comprises: obtaining a deletion table corresponding to a data source from a data operation layer; the deletion table is used to record target source data in source data obtained by the data operation layer from the data source, which has been executed with a deletion operation by the data operation layer; obtaining a source-pasting table corresponding to the data source from the data operation layer of the data platform; the source-pasting table is used to record incremental data generated due to source data increment change of the data source; deleting corresponding data stored in the data warehouse according to the target source data recorded in the deletion table; and updating corresponding data in the data warehouse according to the incremental data recorded in the source-pasting table.

Description

Technical Field

[0001] This application relates to the field of big data technology, and in particular to a data processing method, apparatus, device and storage medium. Background Technology

[0002] A data warehouse (DW), also known as an enterprise data warehouse, is a subject-oriented, integrated, relatively stable, and historically reflective data collection storage system. It aggregates structured data from different sources for comparison and analysis in the field of business intelligence. Generally, a data warehouse is a repository containing various types of data and is highly modeled.

[0003] A data platform, built upon big data, is a data infrastructure platform that integrates structured and unstructured data. It primarily provides services to businesses by directly offering datasets. The data platform emerged to address the limitations of data warehouses in handling unstructured data and the long development cycles of reports. Therefore, it initially sets aside business needs and extracts all of the enterprise's data into a large dataset, including both structured and unstructured data. When business users have specific requirements, they can then extract the necessary smaller datasets from the data platform and provide them to data applications as separate datasets.

[0004] The classic data flow between a data warehouse and a data platform is as follows: data flows from the data platform's data operations layer (also known as the source layer or operational data store, or ODS layer), sequentially to the data warehouse's data warehouse details (DWD) layer, data warehouse middle (DWM) layer, data warehouse service (DWS) layer, and finally to the application data service (ADS) layer. The initial stage of this data flow is the data warehouse's data details layer retrieving data from the data platform's data operations layer.

[0005] It's important to note that the aforementioned data operations layer (also known as the source layer or operational data layer) is the layer closest to the source data. Data from the data source (i.e., source data, or raw data) is extracted, cleaned, and transmitted before being loaded into this layer. Generally, data doesn't require extensive data cleaning; the source data can be passed directly to the data operations layer. Noise reduction, deduplication, and outlier handling can be handled by the data warehouse after the data arrives. The functions of the data operations layer include: 1) serving as the preparation area for the data warehouse; 2) providing source data for the data detail layer; and 3) minimizing the impact on business systems.

[0006] As can be seen from the above data flow, if a portion of data is added to the data platform, that portion of data can enter the data warehouse according to this data flow, thereby increasing the amount of data in the data warehouse accordingly.

[0007] However, the existing technologies mentioned above do not provide a solution for deleting the corresponding data from the data warehouse when a portion of the data is deleted from the data platform. This can lead to data inconsistencies between the data warehouse and the data platform, a problem that the existing technologies urgently need to address. Summary of the Invention

[0008] To address the problem in existing technologies where the corresponding data in the data warehouse cannot be deleted when a portion of data is deleted from the data platform, leading to data inconsistency between the data warehouse and the data platform, embodiments of this application provide a data processing method, apparatus, device, and storage medium.

[0009] One embodiment of this application provides a data processing method applied to a data warehouse. The method includes: obtaining a deletion table corresponding to a data source from a data operations layer; the deletion table is used to record target source data from the source data obtained by the data operations layer from the data source, for which a deletion operation has been performed by the data operations layer; obtaining a source-attached table corresponding to the data source from the data operations layer of the data platform; the source-attached table is used to record incremental data generated by incremental changes to the source data of the data source; deleting corresponding data stored in the data warehouse according to the target source data recorded in the deletion table; and updating corresponding data in the data warehouse according to the incremental data recorded in the source-attached table.

[0010] One embodiment of this application provides a data processing method applied to a data platform. The method includes: obtaining target source data from source data obtained by the data operation layer of the data platform from a data source, wherein the data operation layer has performed a deletion operation on the target source data; recording the target source data in a deletion table; obtaining incremental data generated by incremental changes to the source data of the data source; recording the incremental data in a pasting source table; and providing the deletion table and the pasting source table to a data warehouse so that the data warehouse can perform: deleting the corresponding data stored in the data warehouse according to the target source data recorded in the deletion table; and updating the corresponding data in the data warehouse according to the incremental data recorded in the pasting source table.

[0011] One embodiment of this application provides a data processing apparatus applied to a data warehouse. The apparatus includes: a deletion table acquisition module, used to acquire a deletion table corresponding to a data source from a data operation layer; the deletion table is used to record target source data from the source data acquired by the data operation layer from the data source, for which a deletion operation has been performed by the data operation layer; a pasting source table acquisition module, used to acquire a pasting source table corresponding to the data source from the data operation layer of the data platform; the pasting source table is used to record incremental data generated by incremental changes to the source data of the data source; a data deletion module, used to delete corresponding data stored in the data warehouse according to the target source data recorded in the deletion table; and a data update module, used to update corresponding data in the data warehouse according to the incremental data recorded in the pasting source table.

[0012] One embodiment of this application provides a data processing apparatus applied to a data platform. The apparatus includes: a target source data acquisition module, used to acquire target source data from source data acquired by the data operation layer of the data platform from a data source, wherein the data operation layer has performed a deletion operation on the target source data; a deletion table recording module, used to record the target source data into a deletion table; an incremental data acquisition module, used to acquire incremental data generated by incremental changes in the source data of the data source; a pasting source table recording module, used to record the incremental data into a pasting source table; and a providing module, used to provide the deletion table and the pasting source table to a data warehouse, so that the data warehouse can perform: deleting the corresponding data stored in the data warehouse according to the target source data recorded in the deletion table; and updating the corresponding data in the data warehouse according to the incremental data recorded in the pasting source table.

[0013] One embodiment of this application provides an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program stored in the memory to perform a data processing method applied to a data warehouse.

[0014] One embodiment of this application provides an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program stored in the memory to perform a data processing method applied to a data platform.

[0015] One embodiment of this application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a data processing method.

[0016] The above-described technical solutions adopted in the embodiments of this application can achieve at least the following beneficial effects:

[0017] Deleting data in the data operations layer is recorded in a deletion table. Based on the deletion table, the data deleted in the data operations layer can be identified, and the corresponding data stored in the data warehouse can be deleted accordingly. Incremental data is recorded in a source table. Based on the source table, the updated data in the data operations layer can be identified, and the corresponding data stored in the data warehouse can be updated accordingly. This ensures the consistency between the data in the data warehouse and the data in the data operations layer. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 These are schematic diagrams illustrating application scenarios of the data processing apparatus according to some embodiments of this application;

[0020] Figure 2 This is an exemplary flowchart of a data processing method according to some embodiments of this application;

[0021] Figure 3 This is an exemplary flowchart of a data processing method according to other embodiments of this application. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] As will be known to those skilled in the art, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0024] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0025] To address the problem in existing technologies where, when a portion of data is deleted from the data platform, the corresponding data in the data warehouse cannot be deleted, leading to data inconsistency between the data warehouse and the data platform, this application provides a data processing solution.

[0026] The data processing solutions include data processing methods and devices applied to data platforms, as well as data processing methods and devices applied to data warehouses.

[0027] Figure 1 These are schematic diagrams illustrating application scenarios of a data processing apparatus according to some embodiments of this application. For example... Figure 1 As shown, application scenario 100 may include network 110, data processing device 120 and data processing device 130.

[0028] Network 110 can connect the various components of application scenario 100 and / or connect application scenario 100 with external resources. Network 110 enables communication between the components and with other parts outside application scenario 100, facilitating the exchange of data and / or information. Network 110 can be a local area network, a wide area network, the Internet, or a combination of various network structures.

[0029] Data processing device 120 and data processing device 130 can be applied to a data platform and a data warehouse, respectively. For example, data processing device 120 is applied to a data platform to perform the functions that the data platform can perform, and data processing device 130 is applied to a data warehouse to perform the functions that the data warehouse can perform.

[0030] A data platform, built upon big data, is a data infrastructure platform that integrates structured and unstructured data. It primarily provides services to businesses by directly offering datasets. The data platform emerged to address the limitations of data warehouses in handling unstructured data and the long development cycles of reports. Therefore, it initially sets aside business needs and extracts all of the enterprise's data into a large dataset, including both structured and unstructured data. When business users have specific requirements, they can then extract the necessary smaller datasets from the data platform and provide them to data applications as separate datasets.

[0031] A data warehouse can store, clean, and process various types of data, including measurement data, business data, third-party data, user information, behavior logs, performance logs, and more. Ultimately, it provides the data needed by customers, data analysts, product managers, operations teams, and decision-makers through reports, APIs, FTP, databases, and other means. This provides data support for process improvement, product and performance enhancement, cost reduction, quality monitoring, and decision-making.

[0032] Data processing device 120 and data processing device 130 can communicate via network 110. For example, data processing device 120 can extract and clean data, and send the cleaned data to data processing device 130 via network 110 for storage.

[0033] This scene diagram only shows two data processing devices, but in a real scene, there may be more data processing devices.

[0034] Different data processing devices can be located on different hardware devices, or they can be partially or entirely located on the same hardware device.

[0035] It should be noted that application scenario 100 is provided for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can make various modifications or variations based on the description in this application. For example, the application scenario may also include a database. Furthermore, the application scenario may be implemented on other devices to achieve similar or different functions. However, these variations and modifications will not depart from the scope of this application.

[0036] This application provides a data processing apparatus for use in a data warehouse, the apparatus comprising:

[0037] The deletion table acquisition module is used to obtain the deletion table corresponding to the data source from the data operation layer; the deletion table is used to record the target source data that has been deleted by the data operation layer from the source data obtained by the data operation layer from the data source.

[0038] The source table acquisition module is used to obtain the source table corresponding to the data source from the data operation layer of the data platform; the source table is used to record incremental data generated when the source data of the data source undergoes incremental changes;

[0039] The data deletion module is used to delete the corresponding data stored in the data warehouse based on the target source data recorded in the deletion table;

[0040] The data update module is used to update the corresponding data in the data warehouse based on the incremental data recorded in the source table.

[0041] For more details regarding the deletion table retrieval module, source table retrieval module, data deletion module, and data update module, please refer to [link / reference]. Figure 2 And related explanations.

[0042] In one alternative implementation, the deletion table retrieval module can specifically be used for:

[0043] The deletion table is retrieved from the data operation layer according to the predetermined data acquisition cycle.

[0044] In one alternative implementation, the source table acquisition module can specifically be used for:

[0045] According to the predetermined data acquisition cycle, the post source table is obtained from the data operation layer.

[0046] In one alternative implementation, the data deletion module can specifically be used for:

[0047] Delete the corresponding data stored in the data detail layer and data service layer of the data warehouse, respectively.

[0048] In one alternative implementation, the data update module can specifically be used for:

[0049] Update the corresponding data stored in the data detail layer and the data service layer respectively.

[0050] In an optional implementation, the data processing apparatus further includes a loading module, configured to rerun the batch process based on the table-level dependency relationship between the data service layer and the data detail layer, the table-level dependency relationship between the data detail layer and the data operation layer, and the time of the target source data recorded in the deleted table, and load the data of the data service layer corresponding to the time.

[0051] It should be noted that the above description of the system and its modules is for convenience only and should not be construed as limiting this application to the scope of the embodiments described. It is understood that those skilled in the art, after understanding the principles of this system, may arbitrarily combine the various modules or construct subsystems connected to other modules without departing from these principles. For example, the deletion table acquisition module and the pasting source table acquisition module can be integrated into one module. Alternatively, the modules can share a storage device, or each module can have its own separate storage device. Such variations are all within the scope of protection of this application.

[0052] Figure 2 This is an exemplary flowchart of a data processing method according to some embodiments of this application. In some embodiments, process 200 may be executed by data processing device 130. Figure 2 As shown, process 200 may include the following steps:

[0053] Step 210: Obtain the deletion table corresponding to the data source from the data operations layer.

[0054] In some embodiments, step 210 may be performed by the delete table retrieval module.

[0055] The data operation layer (also known as the source layer or operational data layer) is part of the data platform and is the layer closest to the source data. Data from the data source (i.e., source data, or raw data) is extracted, cleaned, and transmitted before being loaded into this layer.

[0056] A data source can refer to the origin of the source data. Examples include databases, message queues, and logs.

[0057] A deletion table can refer to a table used to store deleted data. The deletion table can be pre-defined. In some embodiments, the deletion table records the target source data from the source data obtained by the data operations layer from the data source, which has been deleted by the data operations layer.

[0058] Target source data refers to the data within the source data that has been deleted by the data operations layer. For example, if the source data includes A, B, and C, and C has been deleted by the data operations layer, then C is called target source data, and C is recorded in the deletion table.

[0059] In some embodiments, the data processing device may communicate with the data operation layer to obtain the deletion table corresponding to the data source.

[0060] In some embodiments, the data processing device may periodically or in real-time obtain the deletion table corresponding to the data source from the data operation layer.

[0061] In some embodiments, the data processing device may retrieve the deletion table from the data operation layer according to a predetermined data retrieval period. For example, if the predetermined data retrieval period is T, the data processing device may retrieve the deletion table from the data operation layer every T time intervals.

[0062] Step 220: Obtain the post source table corresponding to the data source from the data operation layer of the data platform.

[0063] In some embodiments, step 220 may be performed by the source table acquisition module.

[0064] A source table can refer to a table that carries data. In some embodiments, a source table is used to record incremental data generated when the source data of a data source undergoes an incremental change. For example, if the source data includes A and B, and an incremental change occurs, changing the source data to A, B, and D, then D is called incremental data, and D is recorded in the source table.

[0065] In some embodiments, the data processing device may communicate with the data operation layer to obtain the post source table corresponding to the data source.

[0066] In some embodiments, the data processing device may periodically or in real-time obtain the post source table corresponding to the data source from the data operation layer of the data platform.

[0067] In some embodiments, the data processing device may obtain the sticker source table from the data operation layer according to a predetermined data acquisition cycle.

[0068] Step 230: Delete the corresponding data stored in the data warehouse based on the target source data recorded in the deletion table.

[0069] In some embodiments, step 230 may be performed by the data deletion module.

[0070] In some embodiments, the data processing device can delete corresponding data stored in the data warehouse based on the target source data recorded in the deletion table. Continuing with the example above, if the target source data recorded in the deletion table is C, then the data processing device should delete data C stored in the data warehouse.

[0071] In some embodiments, the data processing apparatus may delete the corresponding data stored in the data detail layer and the data service layer of the data warehouse, respectively. Continuing with the example above, the data processing apparatus may delete the stored data C from both the data detail layer and the data service layer of the data warehouse.

[0072] Step 240: Update the corresponding data in the data warehouse based on the incremental data recorded in the source table.

[0073] In some embodiments, step 240 may be performed by the data update module.

[0074] In some embodiments, the data processing apparatus can update the corresponding data in the data warehouse based on the incremental data recorded in the source table. Continuing with the example above, if the incremental data recorded in the source table is D, then the data processing apparatus will add data D to the data warehouse.

[0075] In some embodiments, the data processing apparatus can update the corresponding data stored in the data detail layer and the data service layer, respectively. Continuing with the example above, the data processing apparatus can add new data D from the data detail layer and the data service layer of the data warehouse, respectively.

[0076] In a data warehouse, there is a data service layer above the data detail layer. The data service layer is used to further integrate, summarize and process the data from the data detail layer. It can be seen that ensuring data consistency requires attention not only to the data detail layer and the data operation layer, but also to the data service layer and the data detail layer.

[0077] In some embodiments, process 200 may further include: rerunning the batch process based on the table-level dependency of the data service layer on the data detail layer, the table-level dependency of the data detail layer on the data operation layer, and the time of deleting the target source data recorded in the table, and loading the data of the corresponding time in the data service layer.

[0078] Table-level dependency refers to a relationship formed based on the existing relationship between tables. For example, Table 1 has attributes including name, student ID, and grade; Table 2 has attributes including student ID, grade, and ranking. Both Table 1 and Table 2 share the attribute student ID, which is unique to each student and is also called the primary key. Table 1 and Table 2 are linked through this primary key, so they can be said to have a table-level dependency. That is, based on the student ID and grade in Table 2, querying the student ID in Table 1 will retrieve the student's name; similarly, based on the name and student ID in Table 1, querying the student ID in Table 2 will retrieve the student's grade.

[0079] In some embodiments, the data processing apparatus may include a loading module. In some embodiments, the loading module may rerun the batch process based on the table-level dependencies of the data service layer to the data detail layer, the table-level dependencies of the data detail layer to the data operation layer, and the time when the target source data recorded in the table was deleted, to load the data from the data service layer corresponding to that time. For example, if the time when the target source data recorded in the table was deleted was 13:16:20, the data processing apparatus may load the data from the data service layer at 13:16:20 by performing an ETL (extract, transform, load) operation based on the table-level dependencies of the data service layer to the data detail layer, the table-level dependencies of the data detail layer to the data operation layer, and the time of 13:16:20.

[0080] It not only focuses on the data detail layer and the data operation layer, but also on the data service layer and the data detail layer. Through the methods in the above embodiments, data consistency among the data operation layer, the data detail layer, and the data service layer can be achieved.

[0081] The methods described in the above embodiments can be used to record deleted data in the data operation layer in a deletion table, which allows the data to be known from the deletion table, and to delete the corresponding data stored in the data warehouse based on the deletion table; incremental data can be recorded in a source table, which allows the data to be known from the source table, and to update the corresponding data stored in the data warehouse based on the source table; thus ensuring the consistency between the data in the data warehouse and the data in the data operation layer.

[0082] This application also provides a data processing apparatus applied to a data platform. The apparatus includes: a target source data acquisition module, used to acquire target source data from source data acquired by the data operation layer of the data platform from a data source, wherein the data operation layer has performed a deletion operation on the target source data; a deletion table recording module, used to record the target source data in a deletion table; an incremental data acquisition module, used to acquire incremental data generated by incremental changes in the source data of the data source; a pasting source table recording module, used to record the incremental data in a pasting source table; and a providing module, used to provide the deletion table and the pasting source table to a data warehouse, so that the data warehouse can perform: deleting the corresponding data stored in the data warehouse according to the target source data recorded in the deletion table; and updating the corresponding data in the data warehouse according to the incremental data recorded in the pasting source table.

[0083] For more details regarding the target source data acquisition module, table record deletion module, incremental data acquisition module, source table record pasting module, and provision module, please refer to [link to relevant documentation]. Figure 3 And related explanations.

[0084] It should be noted that the above description of the system and its modules is for convenience only and should not be construed as limiting this application to the scope of the embodiments described. It is understood that those skilled in the art, after understanding the principles of the system, may arbitrarily combine the various modules or construct subsystems connected to other modules without departing from these principles. For example, the target source data acquisition module and the table record deletion module can be integrated into one module. Alternatively, the modules can share a storage device, or each module can have its own separate storage device. Such variations are all within the scope of protection of this application.

[0085] Figure 3 This is an exemplary flowchart of a data processing method according to other embodiments of this application. In some embodiments, process 300 may be executed by data processing device 120. Figure 3 As shown, process 300 may include the following steps:

[0086] Step 310: Obtain the target source data from the source data obtained by the data operation layer of the data platform from the data source, which has been deleted by the data operation layer.

[0087] In some embodiments, step 310 may be performed by the target source data acquisition module.

[0088] In some embodiments, the data processing device can acquire target source data from the source data obtained by the data operation layer of the data platform from the data source, specifically the source data that has been deleted by the data operation layer. For example, this can be achieved through tools like CDC (Change Data Capture), manual input, or reading logs.

[0089] Step 320: Record the target source data in the deletion table.

[0090] In some embodiments, step 320 may be performed by the delete table record module.

[0091] In some embodiments, the data processing device may record target source data into a deletion table.

[0092] Step 330: Obtain incremental data generated due to incremental changes in the source data of the data source.

[0093] In some embodiments, step 330 may be performed by the incremental data acquisition module.

[0094] In some embodiments, the data processing device can acquire incremental data resulting from incremental changes in the source data of the data source through various feasible methods. For example, through CDC tools, manual input, reading logs, etc.

[0095] Step 340: Record the incremental data in the post source table.

[0096] In some embodiments, step 340 may be performed by the source table record module.

[0097] In some embodiments, the data processing device may record incremental data into the patch source table.

[0098] Step 350: Provide the deleted table and the pasted source table to the data warehouse.

[0099] In some embodiments, step 350 may be performed by the providing module.

[0100] In some embodiments, the data processing apparatus may provide a delete table and a paste source table to the data warehouse so that the data warehouse can perform the following actions: delete the corresponding data stored in the data warehouse based on the target source data recorded in the delete table; and update the corresponding data in the data warehouse based on the incremental data recorded in the paste source table. Further explanation regarding deletion and updating can be found in [link to relevant documentation]. Figure 2 And related explanations.

[0101] The methods described in the above embodiments can be used to obtain a deletion table that records target source data and a pasting table that records incremental data, providing a foundation for the data warehouse to execute data processing methods and ensure data consistency between the data platform and the data warehouse.

[0102] This application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program stored in the memory to perform a data processing method applied to a data warehouse.

[0103] This application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program; the processor is coupled to the memory and is used to execute the computer program stored in the memory to perform a data processing method applied to a data platform.

[0104] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement a data processing method.

[0105] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0106] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0107] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method applied to a data warehouse, characterized in that, The method includes: The deletion table corresponding to the data source is obtained from the data operation layer; the deletion table is used to record the target source data that has been deleted by the data operation layer from the source data obtained by the data operation layer from the data source. The data operation layer of the data platform obtains the source table corresponding to the data source; the source table is used to record incremental data generated when the source data of the data source undergoes incremental changes. Based on the target source data recorded in the deletion table, delete the corresponding data stored in the data warehouse; Update the corresponding data in the data warehouse based on the incremental data recorded in the source table; Based on the table-level dependencies of the data service layer to the data detail layer, the table-level dependencies of the data detail layer to the data operation layer, and the time of deleting the target source data recorded in the table, the batch is rerun and the data corresponding to the time in the data service layer is loaded. The table-level dependencies refer to the dependent relationships formed based on the relationships that exist between tables.

2. The method according to claim 1, characterized in that: Obtaining the deletion table corresponding to the data source from the data operation layer includes: obtaining the deletion table from the data operation layer according to a predetermined data acquisition cycle; Obtaining the post source table corresponding to the data source from the data operation layer of the data platform includes: obtaining the post source table from the data operation layer according to the predetermined data acquisition cycle.

3. The method according to claim 1, characterized in that: Deleting the corresponding data stored in the data warehouse includes: deleting the corresponding data stored in the data detail layer and the data service layer of the data warehouse respectively; Updating the corresponding data in the data warehouse includes updating the corresponding data stored in the data detail layer and the data service layer, respectively.

4. A data processing method applied to a data platform, characterized in that, The method The law includes: The data operation layer of the data platform obtains the target source data from the source data source, which has been deleted by the data operation layer. Record the target source data in the deletion table; Obtain incremental data generated due to incremental changes to the source data in the data source; Record the incremental data into the post source table; The deletion table and the source table are provided to the data warehouse so that the data warehouse can perform the following actions: delete the corresponding data stored in the data warehouse based on the target source data recorded in the deletion table; and update the corresponding data in the data warehouse based on the incremental data recorded in the source table. The method further includes: The time of the target source data is recorded in the deletion table so that the data warehouse can perform the following: based on the table-level dependency relationship between the data service layer and the data detail layer, the table-level dependency relationship between the data detail layer and the data operation layer, and the time of the target source data recorded in the deletion table, the batch process is rerun to load the data corresponding to the time in the data service layer. The table-level dependency relationship refers to the dependent relationship formed based on the relationship between tables.

5. A data processing apparatus, applied to a data warehouse, characterized in that, The device includes: The deletion table acquisition module is used to obtain the deletion table corresponding to the data source from the data operation layer; the deletion table is used to record the target source data that has been deleted by the data operation layer from the source data obtained by the data operation layer from the data source. The source table acquisition module is used to obtain the source table corresponding to the data source from the data operation layer of the data platform; the source table is used to record incremental data generated when the source data of the data source undergoes incremental changes; The data deletion module is used to delete the corresponding data stored in the data warehouse based on the target source data recorded in the deletion table; The data update module is used to update the corresponding data in the data warehouse based on the incremental data recorded in the source table; The loading module is used to rerun the batch process based on the table-level dependencies of the data service layer to the data detail layer, the table-level dependencies of the data detail layer to the data operation layer, and the time of the target source data recorded in the deleted table, and load the data of the data service layer corresponding to the time.

6. A data processing apparatus, applied to a data platform, characterized in that, The device includes: The target source data acquisition module is used to acquire the target source data that has been deleted by the data operation layer from the source data obtained by the data operation layer of the data platform from the data source. The delete table record module is used to record the target source data into the delete table; The incremental data acquisition module is used to acquire incremental data generated when the source data of the data source undergoes incremental changes. The source table recording module is used to record the incremental data into the source table; A module is provided to provide the deletion table and the source table to the data warehouse so that the data warehouse can perform the following actions: delete the corresponding data stored in the data warehouse according to the target source data recorded in the deletion table; and update the corresponding data in the data warehouse according to the incremental data recorded in the source table. The deletion table record module is also used to record the time of the target source data in the deletion table so that the data warehouse can perform the following: based on the table-level dependency relationship between the data service layer and the data detail layer, the table-level dependency relationship between the data detail layer and the data operation layer, and the time of the target source data recorded in the deletion table, rerun the batch and load the data corresponding to the time in the data service layer. The table-level dependency relationship refers to the dependent relationship formed based on the relationship between tables.

7. A computing device, characterized in that, include: Memory and processor, among which, The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program stored in the memory for performing the method according to any one of claims 1-3.

8. A computing device, characterized in that, include: Memory and processor, among which, The memory is used to store computer programs; The processor, coupled to the memory, is configured to execute the computer program stored in the memory for performing the method of claim 4.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-3 or claim 4.

Images