Streaming and batch integrated data updating method, device and equipment, storage medium and product
By recording the write time of real-time data and the generation time of offline data in the real-time table, the problem of the separation between real-time data and offline data is solved, and the timeliness and accuracy of data are achieved.
Patent Information
- Application Number
- CN202511355952.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-22
- Publication Date
- 2026-01-23
AI Technical Summary
In existing technologies, the storage methods for real-time data and offline data are separate, resulting in an inherent and irreversible fragmentation of data that cannot be effectively combined.
By adding a first time attribute field to the real-time table to record the write time of real-time data, and adding a second time attribute field to record the generation time after extracting, transforming, cleaning and loading offline data, the system determines whether to update the data in the real-time table based on the relationship between the two time attributes.
It enables integrated streaming and batch updates of real-time and offline data, ensuring data timeliness and accuracy, and solving the problem of data fragmentation.
Smart Images
Figure CN121387907A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of big data, and in particular to a stream-batch integrated data updating method and device, computer equipment, computer readable storage medium and computer program product. BACKGROUND
[0002] With the rapid development of the Internet of Things and the trend of digital transformation of traditional enterprises, the volume of data is growing at an unprecedented rate.
[0003] In the traditional technology, the storage problem of different types of data is generally solved by a stream-batch separation scheme, that is, only real-time data is stored, and offline data is not used for compensation. However, this way will cause the natural separation of real-time data and offline data, and it is irreversible. For example, real-time data is stored using components such as Kafka, Iceberg, Hudi, and offline data is stored using schemes such as Hive, ParquetFile, and OrcFile. Although the two can be merged into the same view at the structured query language (SQL) processing level using splicing and other methods, they are still heterogeneous data in actual underlying storage. SUMMARY
[0004] Therefore, it is necessary to provide a stream-batch integrated data updating method, device, computer equipment, computer readable storage medium and computer program product capable of compensating offline data into real-time tables, to realize stream-batch integrated data updating of offline data and real-time data.
[0005] In a first aspect, the present application provides a stream-batch integrated data updating method, which comprises:
[0006] obtaining real-time data and writing the real-time data into a real-time table, wherein the real-time table comprises a first time attribute field for recording the writing time of the real-time data;
[0007] obtaining offline data from a data source and performing extraction, transformation, cleaning and loading processing on the offline data to obtain offline data in a target format;
[0008] adding a second time attribute field to the offline data in the target format, wherein the second time attribute field is used to record the generation time of the offline data;
[0009] determining whether to update the data in the real-time table according to the writing time of the real-time data and the generation time of the offline data.
[0010] In one of the embodiments, the acquiring real-time data and writing the real-time data into a real-time table comprises:
[0011] A Mixed-Hive table is created by an Amoro component, and a first time attribute column is added to the Mixed-Hive table, the first time attribute column being used to store a first time attribute field corresponding to real-time data;
[0012] Real-time data is acquired and written into the Mixed-Hive table;
[0013] The first time attribute column is automatically filled according to a writing time of the real-time data.
[0014] In one of the embodiments, the acquiring offline data from a data source and performing extraction, conversion, cleaning and loading processing on the offline data to obtain offline data in a target format comprises:
[0015] Offline data formed is extracted from a data source, the data source comprising at least one of a database, a file system and a log file;
[0016] The extracted offline data is subjected to conversion processing, the conversion processing comprising at least one of filtering, sorting, grouping and aggregation;
[0017] The offline data subjected to the conversion processing is subjected to cleaning processing, the cleaning processing comprising at least one of deduplication, missing value filling and data verification;
[0018] The offline data subjected to the cleaning processing is loaded into a target system to obtain offline data in a target format, the target system comprising at least one of a target database, a target data warehouse and a target data lake.
[0019] In one of the embodiments, the adding a second time attribute field to the offline data in the target format comprises:
[0020] A date of the day is determined, and a date obtained by subtracting 1 from the date of the day is taken as a generation time of the offline data;
[0021] The offline data in the target format is stored in a Mixed-Hive table form, and a second time attribute column is added to the table, the second time attribute column being used to store the second time attribute field.
[0022] In one of the embodiments, before determining whether to update data in the real-time table according to the writing time of the real-time data and the generation time of the offline data, the method further comprises:
[0023] It is judged whether the real-time table contains a primary key.
[0024] In the case that the real-time table does not contain the primary key, offline data in the target format is extracted and completely covers the real-time data in the real-time table.
[0025] In one of the embodiments, the determining whether to update the data in the real-time table according to the write time of the real-time data and the generation time of the offline data comprises:
[0026] In the case that the real-time table contains the primary key, it is determined whether the write time of the real-time data is less than the generation time of the offline data.
[0027] If the write time of the real-time data is not less than the generation time of the offline data, the real-time data in the real-time table is not updated.
[0028] If the write time of the real-time data is less than the generation time of the offline data, offline data in the target format is extracted and covers the real-time data corresponding to the primary key in the real-time table.
[0029] In a second aspect, the present application further provides a stream-batch integrated data updating device, which comprises:
[0030] A real-time data writing module is configured to acquire real-time data and write the real-time data into a real-time table, wherein the real-time table contains a first time attribute field, and the first time attribute field is configured to record the write time of the real-time data.
[0031] An offline data acquiring module is configured to acquire offline data from a data source and perform extraction, conversion and loading processing on the offline data to obtain offline data in a target format.
[0032] An offline data processing module is configured to add a second time attribute field to the offline data in the target format, and the second time attribute field is configured to record the generation time of the offline data.
[0033] A data updating module is configured to determine whether to update the data in the real-time table according to the write time of the real-time data and the generation time of the offline data.
[0034] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program:
[0035] Acquire real-time data and write the real-time data into a real-time table, wherein the real-time table contains a first time attribute field, and the first time attribute field is configured to record the write time of the real-time data.
[0036] Obtaining offline data from a data source, and performing extraction, transformation and loading processing on the offline data to obtain offline data in a target format;
[0037] Adding a second time attribute field to the offline data in the target format, the second time attribute field being used to record generation time of the offline data;
[0038] Determining whether to update data in the real-time table according to the write time of the real-time data and the generation time of the offline data.
[0039] In a fourth aspect, the present application further provides a computer readable storage medium, having a computer program stored thereon, the computer program being executed by a processor to implement the following steps:
[0040] Obtaining real-time data, and writing the real-time data into a real-time table, the real-time table including a first time attribute field, the first time attribute field being used to record write time of the real-time data;
[0041] Obtaining offline data from a data source, and performing extraction, transformation and loading processing on the offline data to obtain offline data in a target format;
[0042] Adding a second time attribute field to the offline data in the target format, the second time attribute field being used to record generation time of the offline data;
[0043] Determining whether to update data in the real-time table according to the write time of the real-time data and the generation time of the offline data.
[0044] In a fifth aspect, the present application further provides a computer program product, including a computer program, the computer program being executed by a processor to implement the following steps:
[0045] Obtaining real-time data, and writing the real-time data into a real-time table, the real-time table including a first time attribute field, the first time attribute field being used to record write time of the real-time data;
[0046] Obtaining offline data from a data source, and performing extraction, transformation and loading processing on the offline data to obtain offline data in a target format;
[0047] Adding a second time attribute field to the offline data in the target format, the second time attribute field being used to record generation time of the offline data;
[0048] Determining whether to update data in the real-time table according to the write time of the real-time data and the generation time of the offline data.
[0049] The aforementioned integrated batch and stream data update method, apparatus, computer equipment, computer-readable storage medium, and computer program product acquire real-time data and write it into a real-time table. The real-time table includes a first time attribute field, which records the write time of the real-time data. This accurately marks the update time of all written real-time data, facilitating subsequent determination of whether to overwrite the written real-time data. Offline data is acquired from a data source and processed by extraction, transformation, cleaning, and loading to obtain offline data in the target format. A second time attribute field is added to the target-format offline data, recording its generation time. This marks the generation time of the offline data, using it as a basis for determining whether to overwrite the real-time data. Based on the write time of the real-time data and the generation time of the offline data, it is determined whether to update the data in the real-time table. Thus, by using the corresponding times of the real-time and offline data, it is clear whether the data in the real-time table needs to be updated, ensuring the timeliness and accuracy of the data in the real-time table and achieving integrated batch and stream data processing for both offline and real-time data. Attached Figure Description
[0050] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0051] Figure 1 This is an application environment diagram of the batch and stream integrated data update method in one embodiment;
[0052] Figure 2 This is a flowchart illustrating a batch and stream integrated data update method in one embodiment.
[0053] Figure 3 This is a flowchart illustrating the batch and stream integrated data update method in another embodiment;
[0054] Figure 4 This is a schematic diagram illustrating the principle of a batch and stream integrated data update method.
[0055] Figure 5 This is a structural block diagram of a batch processing integrated data update device in one embodiment;
[0056] Figure 6 This is a structural block diagram of the batch processing integrated data update device in another embodiment;
[0057] Figure 7An internal structure diagram of a computer device in one embodiment. DETAILED DESCRIPTION
[0058] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and not to limit the present application.
[0059] It should be noted that the terms "first", "second", etc. used in the present application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "include" and "have" and any variations thereof used in the present application are intended to cover non-exclusive inclusion. The term "multiple" used in the present application refers to two and more than two. The term "and / or" used in the present application refers to one of the options or any combination of multiple options.
[0060] In order to facilitate understanding of the technical solutions in various embodiments of the present application, first, a brief description of the technical terms that may appear in the embodiments of the present application is made:
[0061] 1) Apache Amoro (referred to as Amoro) is a Lakehouse management system based on open data lake format, which aims to provide a lake-native architecture out-of-the-box experience for data platforms through plug-in and self-management capabilities. In the Amoro component, the table format using the Mixed-Hive structure is enhanced for the Hive table, supporting real-time writing, but it cannot support the splicing of offline data to real-time data.
[0062] 2) Amoro Mixed Hive is a lake-warehouse integrated solution based on Apache Amoro, which aims to solve the pain points of traditional Hive in real-time data processing, storage cost and data consistency. Its core features include Hive read-write compatibility, data self-optimization and stream-batch integration support, which can seamlessly integrate with the existing Hive ecosystem and improve data processing efficiency. Among them, Mixed Hive adopts a hierarchical storage architecture, including two key components BaseStore and ChangeStore. BaseStore is based on Iceberg table implementation, used to store basic data and support atomicity, consistency, isolation and persistence, as well as primary key update and time travel (Time Travel) function; ChangeStore is an independent Iceberg table used to record real-time change data, which is merged with BaseStore through the Merge-on-Read mechanism, thereby realizing minute-level data freshness.
[0063] The stream-batch integrated data updating method provided by the embodiments of the present application can be applied to an application environment as shown in Figure 1 The terminal 102 communicates with the server 104 through a network. The data storage system can store data required to be processed by the server 104. The data storage system can be integrated on the server 104, or placed on a cloud or other network server. The terminal 102 can generate real-time data. The server 104 acquires the real-time data and writes the real-time data into a real-time table. The real-time table includes a first time attribute field, which is used to record the write time of the real-time data. Offline data is acquired from a data source, and the offline data is subjected to extraction, conversion, cleaning and loading processing to obtain offline data in a target format. A second time attribute field is added to the offline data in the target format, and the second time attribute field is used to record the generation time of the offline data. Whether to update the data in the real-time table is determined according to the write time of the real-time data and the generation time of the offline data. The terminal 102 can be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, unmanned aerial vehicles, low-altitude flying vehicles, Internet of Things devices, and the like. The Internet of Things devices can be smart speakers, smart televisions, smart air conditioners, smart vehicle-mounted devices, projection devices, and the like. The server 104 can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0064] In an exemplary embodiment, as shown in Figure 2 A stream-batch integrated data updating method is provided. The method is described by taking a server in Figure 1 as an example, and includes the following steps 201 to 204. Wherein:
[0065] Step 201, acquiring real-time data and writing the real-time data into a real-time table. The real-time table includes a first time attribute field, which is used to record the write time of the real-time data.
[0066] In this embodiment, the real-time table generally refers to a table with real-time data collection function. Real-time data can be written into the real-time table through real-time stream. It should be understood that although the real-time table can be used to collect real-time data, it does not exclude storing offline data in the real-time table.
[0067] For example, a Mixed-Hive table is created through an Amoro component, and a first time attribute column is added to the Mixed-Hive table. The first time attribute column is used to store the first time attribute field corresponding to the real-time data. Real-time data is acquired and written into the Mixed-Hive table. The first time attribute column is automatically filled according to the write time of the real-time data.
[0068] Step 202, obtaining offline data from a data source, and performing extraction, transformation, cleaning and loading processing on the offline data to obtain offline data in a target format.
[0069] In this embodiment, the offline data can be processed by ETL first. The purpose of ETL processing is to extract data from various sources, clean, transform and integrate, and then load into the target system or data warehouse to support data analysis and decision making. It ensures the accuracy, reliability and consistency of data. Through the ETL process, data scattered in different systems and formats can be integrated together to form a unified data view, thereby better supporting business processes and decision making. In addition, ETL can also help clean and verify data, eliminate duplicate, incomplete or inaccurate data, and improve data quality and reliability.
[0070] For example, the formed offline data is extracted from the data source, and the data source includes at least one of a database, a file system, and a log file; the extracted offline data is processed by conversion, and the conversion processing includes at least one of filtering, sorting, grouping and aggregation; the offline data after conversion processing is cleaned, and the cleaning processing includes at least one of de-duplication, missing value filling, and data verification; the offline data after cleaning processing is loaded into a target system to obtain offline data in a target format; the target system includes at least one of a target database, a target data warehouse, and a target data lake.
[0071] Step 203, adding a second time attribute field to the offline data in the target format, the second time attribute field being used to record the generation time of the offline data.
[0072] In this embodiment, the offline data after ETL processing can be added with a time attribute, such as a second time attribute field, so that the generation time of each offline data can be clearly determined.
[0073] For example, the date of the current day is determined, and the date after subtracting 1 from the current day is used as the generation time of the offline data; the offline data in the target format is stored in the form of a Mixed-Hive table, and a second time attribute column is added to the table, which is used to store the second time attribute field.
[0074] It should be understood that the embodiment does not limit the update time of the offline data. In addition to setting the daily cut (by day) to update the offline data, a more accurate update time can also be flexibly set. For example, according to the business requirements, the offline data can be updated every 1 hour, and the generation time corresponding to the offline data is the current time minus 1 hour.
[0075] Step 204, according to the write time of real-time data and the generation time of offline data, determine whether to update the data in the real-time table.
[0076] In this embodiment, according to the write time of real-time data and the generation time of offline data, it can be determined whether the data in the real-time table needs to be updated. For example, if the generation time of offline data is later than the write time of real-time data, offline data needs to be offset into the real-time table. If the generation time of offline data is earlier than the write time of real-time data, offline data does not need to be offset into the real-time table. Thus, both the timeliness and accuracy of the data in the real-time table can be ensured.
[0077] In the above-mentioned stream batch integrated data updating method, real-time data is obtained and written into a real-time table, the real-time table includes a first time attribute field, and the first time attribute field is used to record the write time of the real-time data. Thus, the update time of all written real-time data can be accurately marked, which facilitates subsequent judgment of whether to cover part of the written real-time data. Offline data is obtained from a data source, and the offline data is extracted, converted, cleaned and loaded to obtain offline data in a target format. A second time attribute field is added to the offline data in the target format, and the second time attribute field is used to record the generation time of the offline data. Thus, the generation time of the offline data can be marked, and the generation time is used as a basis for judgment to determine whether the real-time data needs to be covered. According to the write time of real-time data and the generation time of offline data, it is determined whether to update the data in the real-time table. Thus, whether the data in the real-time table needs to be updated can be determined according to the corresponding time of the real-time data and the offline data, the timeliness and accuracy of the data in the real-time table can be ensured, and stream batch integration of offline data and real-time data is realized.
[0078] In another exemplary embodiment, as shown in Figure 3 , a stream batch integrated data updating method is provided, which is applied to a server in Figure 1 as an example for illustration, including the following steps 301 to 308. Among them:
[0079] Step 301, obtaining real-time data and writing the real-time data into a real-time table, the real-time table including a first time attribute field, and the first time attribute field being used to record the write time of the real-time data.
[0080] Step 302, obtaining offline data from a data source, and extracting, converting, cleaning and loading the offline data to obtain offline data in a target format.
[0081] Step 303, adding a second time attribute field to the offline data in the target format, and the second time attribute field being used to record the generation time of the offline data.
[0082] The specific implementation process and technical effects of steps 301-303 in the embodiment of the application are described in the foregoing Figure 2 The related descriptions of steps 201-203 in the method embodiment are described above, and will not be repeated here.
[0083] In step 304, it is determined whether the real-time table contains the primary key. If not, step 305 is performed, and if yes, step 306 is performed.
[0084] In step 305, the offline data in the target format is extracted, and the real-time data in the real-time table is completely overwritten.
[0085] In step 306, it is determined whether the write time of the real-time data is less than the generation time of the offline data. If yes, step 307 is performed, and if not, step 308 is performed.
[0086] In step 307, the offline data in the target format is extracted, and the real-time data corresponding to the primary key in the real-time table is overwritten.
[0087] In step 308, the real-time data in the real-time table is not updated.
[0088] In the embodiment, it is first determined whether the real-time table contains the primary key. In the case that the real-time table does not contain the primary key, the offline data in the target format is extracted, and the real-time data in the real-time table is completely overwritten. In the case that the real-time table contains the primary key, it is determined whether the write time of the real-time data is less than the generation time of the offline data. If the write time of the real-time data is not less than the generation time of the offline data, the real-time data in the real-time table is not updated. If the write time of the real-time data is less than the generation time of the offline data, the offline data in the target format is extracted, and the real-time data corresponding to the primary key in the real-time table is overwritten.
[0089] An exemplary Figure 4 A schematic diagram of the principle of a stream-batch integrated data updating method is shown in FIG. 1. Figure 4 As shown, the process of data processing can be divided into three main stages, a real-time data processing stage, an ETL processing stage (which can include data extraction, conversion, cleaning and loading), and an offline data processing stage. For the real-time data processing stage, real-time data is first sucked in, and then crt_time (i.e., the write time of the real-time data) is generated according to the data write time. For the offline data stage, daily cutting can be performed, i.e., the offline data is updated once a day. During the updating process, the offline data needs to be processed by ETL, and a crt_time field is also added. Since the offline data is more accurate than the real-time data, the crt_time of the offline part is set to (t-1) day 23:59:59, so as to ensure that the offline data can accurately overwrite the real-time data.
[0090] Further, in the ETL processing stage, firstly, it is judged whether the data in the real-time table contains a primary key. In the case that the real-time table does not contain a primary key, offline data in a target format is extracted and is used to completely overwrite the real-time data in the real-time table. In the case that the real-time table contains a primary key, it is judged whether the crt_time field in the real-time table has been updated on the current day. If the crt_time in the real-time table > the crt_time in the offline table, it means that the data has been updated on the t day, and thus the data cannot be overwritten. If the crt_time in the real-time table < the crt_time in the offline table, the update is performed.
[0091] It should be understood that the embodiments of the present application do not limit the value of t. In addition to the unit of day, the unit of hour can also be set. For example, every 6 hours, it is judged whether the data in the real-time table needs to be updated.
[0092] It should be understood that although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, at least part of the steps in the flowcharts involved in the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be alternately executed with at least part of other steps or steps or stages in other steps. It can be understood that the steps in different embodiments can be freely combined as needed, and various non-contradictory schemes formed by the combination are within the scope of protection of the present application.
[0093] Based on the same inventive concept, the embodiments of the present application also provide a stream-batch integrated data updating apparatus for implementing the above-mentioned stream-batch integrated data updating method. The problem-solving implementation scheme provided by the apparatus is similar to the implementation scheme described in the above method, and thus the specific limitations in one or more stream-batch integrated data updating apparatus embodiments provided below can refer to the limitations of the stream-batch integrated data updating method in the above text, which will not be described here again.
[0094] In an exemplary embodiment, as shown in Figure 5 a stream-batch integrated data updating apparatus is provided, comprising: a real-time data writing module 501, an offline data acquisition module 502, an offline data processing module 503, and a data updating module 504, wherein:
[0095] The real-time data writing module 501 is configured to obtain real-time data and write the real-time data into a real-time table, wherein the real-time table comprises a first time attribute field, and the first time attribute field is configured to record a writing time of the real-time data.
[0096] The offline data obtaining module 502 is configured to obtain offline data from a data source and perform extraction, conversion, cleaning and loading processing on the offline data to obtain offline data in a target format.
[0097] The offline data processing module 503 is configured to add a second time attribute field to the offline data in the target format, and the second time attribute field is configured to record a generation time of the offline data.
[0098] The data updating module 504 is configured to determine whether to update data in the real-time table according to the writing time of the real-time data and the generation time of the offline data.
[0099] For example, the real-time data writing module 501 is configured to create a Mixed-Hive table through an Amoro component, add a first time attribute column to the Mixed-Hive table, store the first time attribute field corresponding to the real-time data in the first time attribute column, obtain the real-time data, write the real-time data into the Mixed-Hive table, and automatically fill the first time attribute column according to the writing time of the real-time data.
[0100] For example, the offline data obtaining module 502 is configured to extract offline data formed from a data source, the data source comprises at least one of a database, a file system and a log file, perform conversion processing on the extracted offline data, the conversion processing comprises at least one of filtering, sorting, grouping and aggregation, perform cleaning processing on the offline data after the conversion processing, the cleaning processing comprises at least one of de-duplication, missing value filling and data verification, load the offline data after the cleaning processing into a target system to obtain offline data in a target format, and the target system comprises at least one of a target database, a target data warehouse and a target data lake.
[0101] For example, the offline data processing module 503 is configured to determine a date of the day, take a date obtained by subtracting 1 from the date of the day as the generation time of the offline data, store the offline data in the target format in the form of a Mixed-Hive table, and add a second time attribute column to the table, and the second time attribute column is configured to store the second time attribute field.
[0102] In another example embodiment, as shown in Figure 6 a flow batch integrated data updating apparatus is provided, and in Figure 5On the basis of the illustrated device, the judging module 505 can be further included, and the judging module 505 is configured to judge whether the primary key is included in the real-time table.
[0103] The data updating module 504 is specifically configured to: in the case that the primary key is not included in the real-time table, extract the offline data in the target format, and completely cover the real-time data in the real-time table.
[0104] The data updating module 504 is specifically configured to: in the case that the primary key is included in the real-time table, determine whether the write time of the real-time data is less than the generation time of the offline data; if the write time of the real-time data is not less than the generation time of the offline data, the real-time data in the real-time table is not updated; if the write time of the real-time data is less than the generation time of the offline data, the offline data in the target format is extracted, and the real-time data corresponding to the primary key in the real-time table is covered.
[0105] The above-mentioned various modules in the stream-batch integrated data updating device can be all or partially realized by software, hardware and a combination thereof. The above-mentioned various modules can be embedded in or independent of the processor in the computer device in the hardware form, or can be stored in the memory in the computer device in the software form, so as to be called and executed by the processor to perform the operations corresponding to the above-mentioned various modules.
[0106] In an exemplary embodiment, a computer device is provided, which can be a terminal, and an internal structure diagram of the computer device can be as shown in FIG. 6. Figure 7As shown in the figure. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. Among them, the processor, the memory and the input / output interface are connected through the system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The input / output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used to communicate with the external terminal in a wired or wireless manner. The wireless manner can be realized through WIFI, mobile cellular network, near field communication (Near Field Communication, NFC) or other technologies. The computer program is executed by the processor to realize a flow batch integrated data updating method. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.
[0107] Those skilled in the art can understand that, Figure 7 The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. A specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different component arrangement.
[0108] In one exemplary embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the following steps:
[0109] Obtaining real-time data and writing the real-time data into a real-time table, the real-time table including a first time attribute field for recording the write time of the real-time data; obtaining offline data from a data source and performing extraction, transformation, cleaning and loading processing on the offline data to obtain offline data in a target format; adding a second time attribute field to the offline data in the target format, the second time attribute field being used to record the generation time of the offline data; and determining whether to update the data in the real-time table according to the write time of the real-time data and the generation time of the offline data.
[0110] In one embodiment, the processor executing the computer program further implements the following steps:
[0111] Create a Mixed-Hive table through the Amoro component, and add a first time attribute column to the Mixed-Hive table, the first time attribute column being used to store a first time attribute field corresponding to real-time data; obtain the real-time data and write the real-time data into the Mixed-Hive table; and automatically fill the first time attribute column according to a write time of the real-time data.
[0112] In one embodiment, the processor, when executing the computer program, also implements the following steps:
[0113] Extract the formed offline data from a data source, the data source including at least one of a database, a file system, and a log file; perform conversion processing on the extracted offline data, the conversion processing including at least one of filtering, sorting, grouping, and aggregation; perform cleaning processing on the offline data after the conversion processing, the cleaning processing including at least one of deduplication, filling of missing values, and data verification; load the offline data after the cleaning processing into a target system to obtain offline data in a target format; and the target system including at least one of a target database, a target data warehouse, and a target data lake.
[0114] In one embodiment, the processor, when executing the computer program, also implements the following steps:
[0115] Determine a date of the day, and use a date obtained by subtracting 1 from the date of the day as a generation time of offline data; store the offline data in the target format in the form of a Mixed-Hive table, and add a second time attribute column to the table, the second time attribute column being used to store a second time attribute field.
[0116] In one embodiment, the processor, when executing the computer program, also implements the following steps:
[0117] Before determining whether to update data in a real-time table according to a write time of the real-time data and a generation time of the offline data, determine whether the real-time table contains a primary key; in a case where the real-time table does not contain the primary key, extract the offline data in the target format and completely overwrite the real-time data in the real-time table.
[0118] In one embodiment, the processor, when executing the computer program, also implements the following steps:
[0119] In a case where the real-time table contains the primary key, determine whether the write time of the real-time data is less than the generation time of the offline data; if the write time of the real-time data is not less than the generation time of the offline data, do not update the real-time data in the real-time table; and if the write time of the real-time data is less than the generation time of the offline data, extract the offline data in the target format and overwrite the real-time data corresponding to the primary key in the real-time table.
[0120] In one embodiment, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the method steps in each of the above embodiments.
[0121] In one embodiment, a computer program product is provided, and the computer program product comprises a computer program, and the computer program is executed by a processor to implement the method steps in each of the above embodiments.
[0122] It should be noted that the user information (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of the relevant data need to comply with the relevant provisions.
[0123] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, the processes of the above-mentioned embodiment methods can be included. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., and is not limited thereto. The processor involved in the embodiments provided in the present application can be a general processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., and is not limited thereto.
[0124] The technical features of the above embodiments can be combined arbitrarily. In order to make the description simple, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.
[0125] The above embodiments only express several implementation ways of the present application, and the description is specific and detailed, but it should not be understood as a limitation to the patent scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, several modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.
Claims
1. A stream-batch integrated data updating method, characterized in that, The method comprises: acquiring real-time data and writing the real-time data into a real-time table, the real-time table comprising a first time attribute field for recording a writing time of the real-time data; acquiring offline data from a data source and performing extraction, conversion, cleaning and loading processing on the offline data to obtain offline data in a target format; adding a second time attribute field to the offline data in the target format, the second time attribute field being used to record a generation time of the offline data; determining whether to update data in the real-time table according to the writing time of the real-time data and the generation time of the offline data.
2. The method of claim 1, wherein, The acquiring real-time data and writing the real-time data into a real-time table comprises: creating a Mixed-Hive table through an Amoro component and adding a first time attribute column to the Mixed-Hive table, the first time attribute column being used to store a first time attribute field corresponding to real-time data; acquiring real-time data and writing the real-time data into the Mixed-Hive table; automatically filling the first time attribute column according to a writing time of the real-time data.
3. The method of claim 1, wherein, The acquiring offline data from a data source and performing extraction, conversion, cleaning and loading processing on the offline data to obtain offline data in a target format comprises: extracting offline data already formed from a data source, the data source comprising at least one of a database, a file system and a log file; performing conversion processing on the extracted offline data, the conversion processing comprising at least one of filtering, sorting, grouping and aggregation; performing cleaning processing on the offline data after conversion, the cleaning processing comprising at least one of de-duplication, filling of missing values and data verification; loading the offline data after cleaning into a target system to obtain offline data in a target format, the target system comprising at least one of a target database, a target data warehouse and a target data lake.
4. The method of claim 1, wherein, The adding a second time attribute field to the offline data in the target format comprises: determining a date of the day and taking a date obtained by subtracting 1 from the date of the day as a generation time of the offline data; storing the offline data in the target format in the form of a Mixed-Hive table and adding a second time attribute column to the table, the second time attribute column being used to store the second time attribute field.
5. The method according to any one of claims 1 to 4, characterized in that, Before the determining whether to update data in the real-time table according to the writing time of the real-time data and the generation time of the offline data, the method further comprises: determining whether the real-time table comprises a primary key; in the case where the real-time table does not comprise a primary key, extracting the offline data in the target format and completely covering real-time data in the real-time table.
6. The method of claim 5, wherein, The determining whether to update data in the real-time table according to the writing time of the real-time data and the generation time of the offline data comprises: in the case where the real-time table comprises a primary key, determining whether the writing time of the real-time data is less than the generation time of the offline data; If the write time of the real-time data is not less than the generation time of the offline data, the real-time data in the real-time table is not updated; If the write time of the real-time data is less than the generation time of the offline data, the offline data in the target format is extracted, and the real-time data corresponding to the primary key in the real-time table is overwritten.
7. A stream-batch integrated data updating apparatus characterized by comprising: The device comprises: a real-time data writing module, configured to acquire real-time data and write the real-time data into a real-time table, wherein the real-time table comprises a first time attribute field, and the first time attribute field is used to record the write time of the real-time data; an offline data acquisition module, configured to acquire offline data from a data source and perform extraction, conversion and loading processing on the offline data to obtain offline data in a target format; an offline data processing module, configured to add a second time attribute field to the offline data in the target format, and the second time attribute field is used to record the generation time of the offline data; a data updating module, configured to determine whether to update the data in the real-time table according to the write time of the real-time data and the generation time of the offline data.
8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 6.
9. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.
10. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 6.