Partition view generation method and apparatus

Abstract

The application provides a partition view generation method, which comprises the following steps: extracting Instant information of the current day from a preset time axis when the progress of data writing into a HUDI table reaches a target watermark; detecting whether cross-day data is included in each Instant information corresponding file segment, and adding data filtering logic to the current Instant information when it is detected that the cross-day data is included in the file segment corresponding to the current Instant information, wherein the data filtering logic is used to delete the cross-day data included in the file segment corresponding to the current Instant information; performing data snapshot processing on all the Instant information after detection processing to obtain Instant information of a partition view; and performing file merging processing on all file segments of the current day according to the Instant information of the partition view to generate a partition view. The technical scheme of the application can accurately distinguish the actual data boundary.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a partition view generation method and device, computer equipment and computer readable storage medium. BACKGROUND

[0002] In the process of writing data to the HUDI table, the data can be partitioned according to the processing time or event time of the data. However, since the data is written in real time, the partition data written to the partition is also changing all the time. When triggering a task to obtain data of the day downstream, the data obtained may be incomplete or contain too much cross-day data. In addition, since some delayed data of the day will also be available successively, this partition data is uncertain for the downstream, and the partition boundary is also unclear. SUMMARY

[0003] The purpose of the present application is to provide a partition view generation method, device, computer equipment and computer readable storage medium to solve the above problems.

[0004] One aspect of an embodiment of the present application provides a partition view generation method, which comprises:

[0005] When the progress of writing data into the HUDI table reaches the target watermark, extracting Instant information of the day from a preset time axis;

[0006] Detecting whether cross-day data is included in the file shard corresponding to each extracted Instant information, and adding data filtering logic to the current Instant information when it is detected that the cross-day data is included in the file shard corresponding to the current Instant information, wherein the data filtering logic is used to delete the cross-day data included in the file shard corresponding to the current Instant information;

[0007] Performing data snapshot processing on all Instant information after detection and processing to obtain Instant information of the partition view;

[0008] Performing file merging processing on all file shards of the day according to the Instant information of the partition view to generate a partition view.

[0009] Optionally, the Instant information includes an event time interval of all data included in the file shard, and the detection of whether cross-day data is included in the file shard corresponding to each extracted Instant information comprises:

[0010] Detecting whether cross-day time is included in the event time interval in each extracted Instant information;

[0011] When it is detected that the event time interval in the current Instant information contains cross-day time, it is determined that the file segment corresponding to the current Instant information includes cross-day data.

[0012] Optionally, the data snapshot processing on all the Instant information after the detection processing to obtain the partition view Instant information comprises:

[0013] The first snapshot operation is performed on the Instant information containing data filtering logic in all the Instant information after the detection processing to obtain the first target Instant information of the partition view;

[0014] The second snapshot operation is performed on the Instant information not containing data filtering logic in all the Instant information after the detection processing to obtain the second target Instant information of the partition view, wherein the partition view Instant information comprises the first target Instant information and the second target Instant information.

[0015] Optionally, the file merging processing on all the file segments of the current day according to the partition view Instant information to generate the partition view comprises:

[0016] The data filtering logic operation is performed on the file segment containing cross-day data according to the partition view Instant information to obtain the target file segment with cross-day data deleted;

[0017] All the file segments of the current day are obtained according to the partition view Instant information;

[0018] The file merging processing is performed on all the obtained file segments of the current day and the target file segment to generate the partition view.

[0019] Optionally, the method further comprises:

[0020] When it is detected that the sum of the event time of the data written into the HUDI table and the delay time set by the user exceeds the target water mark, it is determined that the progress of writing data into the HUDI table reaches the target water mark.

[0021] Optionally, the method further comprises:

[0022] The partition view Instant information and the partition view are written into the preset partition.

[0023] Optionally, the method further comprises:

[0024] Set the Instant information of the partition view as a node on the time axis for query.

[0025] Yet another aspect of the embodiments of the present application provides a partition view generation apparatus, which comprises:

[0026] an extraction module configured to extract the Instant information of the day from a preset time axis when the progress of data writing into the HUDI table reaches a target watermark;

[0027] a detection module configured to detect whether the file slice corresponding to each Instant information includes cross-day data, and add data filtering logic to the current Instant information when it is detected that the file slice corresponding to the current Instant information contains cross-day data, wherein the data filtering logic is configured to delete the cross-day data contained in the file slice corresponding to the current Instant information;

[0028] a snapshot module configured to perform data snapshot processing on all the Instant information after detection processing, to obtain the Instant information of the partition view;

[0029] a generation module configured to perform file merging processing on all the file slices of the day according to the Instant information of the partition view, to generate the partition view.

[0030] Yet another aspect of the embodiments of the present application provides a computer device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor is configured to implement the steps of the partition view generation method as described above when executing the computer program.

[0031] Yet another aspect of the embodiments of the present application provides a computer readable storage medium, which stores a computer program executable by at least one processor, so as to enable the at least one processor to execute the steps of the partition view generation method as described above.

[0032] The partition view generation method and device, computer device and computer readable storage medium provided by the embodiments of the present application extract the Instant information of the day from the preset time axis when the progress of data written into the HUDI table reaches the target watermark; detect whether the cross-day data is included in the file fragments corresponding to each Instant information extracted, and add data filtering logic to the current Instant information when it is detected that the cross-day data is included in the file fragments corresponding to the current Instant information, wherein the data filtering logic is used to delete the cross-day data included in the file fragments corresponding to the current Instant information; perform data snapshot processing on all Instant information after detection processing to obtain the Instant information of the partition view; perform file merging processing on all file fragments of the day according to the Instant information of the partition view to generate the partition view. The partition view in the present application is obtained by dividing and merging all file fragments written into the HUDI table on the target watermark, and therefore the partition view generated in the present application only includes the data of the day and no longer includes the cross-day data, so that the actual boundary of the data obtained by the downstream is accurately distinguished. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 An environment diagram according to an embodiment of the present application is schematically shown;

[0034] Figure 2 A flowchart of a partition view generation method according to the first embodiment of the present application is schematically shown;

[0035] Figure 3 A step refinement flowchart of detecting whether the cross-day data is included in the file fragments corresponding to each Instant information extracted according to the first embodiment of the present application is schematically shown;

[0036] Figure 4 A step refinement flowchart of performing data snapshot processing on all Instant information after detection processing to obtain the Instant information of the partition view according to the first embodiment of the present application is schematically shown;

[0037] Figure 5 A step flowchart of performing file merging processing on all file fragments of the day according to the Instant information of the partition view to generate the partition view according to the first embodiment of the present application is schematically shown;

[0038] Figure 6 A schematic diagram of a time axis is schematically shown;

[0039] Figure 7Fig. 3 shows a block diagram of a partition view generation apparatus according to an embodiment of the present application;

[0040] Figure 8 Fig. 4 shows a hardware architecture diagram of a computer device suitable for implementing the partition view generation method according to an embodiment of the present application. DETAILED DESCRIPTION

[0041] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0042] It should be noted that the terms "first", "second" and the like in the embodiments of the present application are only used for description purposes and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features with "first" and "second" can explicitly or implicitly include at least one of the features. In addition, the technical solutions of each embodiment can be combined with each other, but it must be based on the fact that the technical solutions can be realized by those of ordinary skill in the art. When the combination of technical solutions appears contradictory or unachievable, it should be considered that the combination of technical solutions does not exist and is not within the scope of protection claimed by the present application.

[0043] In the description of the present application, it should be understood that the numerical reference numbers before the steps do not indicate the order of execution of the steps before and after, but are only used to facilitate the description of the present application and to distinguish each step, and therefore should not be understood as limiting the present application.

[0044] The following are explanations of some terms related to the present application:

[0045] Flink belongs to a distributed real-time computing system and is used for stateful computation on unbounded and bounded data streams. Flink is designed to run in all common cluster environments to perform computation at memory speed and any scale. The state generally refers to the state of a specific task / operator.

[0046] HDFS (Hadoop Distributed File System) is the Hadoop distributed file system.

[0047] Stream processing is used for processing (such as querying) stream data. Stream data is usually online data and has no boundary.

[0048] SQL(Structured Query Language) is a database language with data manipulation and data definition functions. This language has interactive features that provide great convenience for users. Database management systems should make full use of SQL language to improve the quality and efficiency of computer application systems. SQL language can not only be applied independently in terminals, but also can be used as a sub-language to provide effective assistance for other program design. In this program application, SQL can be used together with other program languages to optimize program functions, and further provide more comprehensive information for users.

[0049] HUDI(Hadoop Updates and Incrementals, Hadoop updates and increments), which stores and manages large analysis data sets through DFS (HDFS or cloud storage), supports update operations in the current data table. Hudi organizes tables into directory structures under a specified directory (basepath) on HDFS. Tables are divided into multiple partitions, and partitions exist in the form of directories. Under each directory, there will be multiple file shards belonging to the partition. Similar to Hive tables, each Hudi table partition is uniquely identified by a partition path (Partition Path).

[0050] Data lake, a storage repository or system that stores data in raw format.

[0051] Watermark, a mechanism proposed by Flink for processing EventTime window calculations, is essentially a timestamp.

[0052] Commit, atomic writing of records to a dataset.

[0053] Savepoint, a method to implement "sub-transaction" (subtransaction) in database transaction processing, also known as nested transaction. The transaction can be rolled back to the savepoint without affecting the changes before the savepoint is created, without abandoning the entire transaction.

[0054] Event time, the generation time of data.

[0055] In order to facilitate the understanding of the technical solutions provided by the embodiments of the present application by those skilled in the art, the related art is described as follows:

[0056] Data can be partitioned according to the processing time or event time of the data during the process of writing into the HUDI table. However, since the data is written in real time, the partition data written into the partition is also constantly changing. When triggering a task to acquire the data of the day downstream, the data obtained can be incomplete or contain too much data across days, and in addition, some delayed data of the day will also be available successively, so that the partition data is uncertain for the downstream, and the partition boundary is also unclear. When the downstream needs to perform statistical analysis on the data of the day, the actual boundary of the data cannot be distinguished.

[0057] In view of the problem that the actual boundary of the data cannot be distinguished in the prior art, the present application provides a partition view generation scheme. In these schemes, the accurate division of the actual boundary of the data can be achieved, and all Instant information on a preset timeline can be snapped. In addition, when the data of the day is needed to be acquired, it is not necessary to export all the data of the day to the hive table, but the data can be directly acquired from the generated partition view.

[0058] The exemplary application environment of the present application is provided below. Figure 1 The application environment of the present application is schematically shown.

[0059] The service platform 2 can provide services such as partition view generation, which can be a single server, a server cluster or a cloud computing service center. The service platform 2 is built-in with a partition view generation device 10 for accelerating query.

[0060] The terminal 4 can be configured to send a query request, etc. The terminal can be any type of computing device, such as a smartphone, a tablet device, a laptop computer, a smart television, a vehicle terminal, etc. The terminal and the service platform 2 can be connected through a network. The network can include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls and / or proxy devices, etc. The network can include physical links, such as coaxial cable links, twisted pair cable links, optical fiber links and combinations thereof and / or the like. The network can include wireless links, such as cellular links, satellite links, Wi-Fi links and / or the like.

[0061] It should be noted that, Figure 1 The number of terminals and servers in the above-mentioned application environment is only illustrative and does not limit the patent protection scope of the present application.

[0062] Below, several embodiments will be provided under the above-mentioned exemplary application environment. For ease of understanding, the partition view generation device 10 in the above-mentioned application environment will be taken as the execution subject to explain the partition view generation scheme. Figure 1 In the above-mentioned application environment, the partition view generation device 10 can be configured to perform the following steps.

[0063] Embodiment One

[0064] Figure 2 An embodiment one of the application is schematically shown in the flow chart of the partition view generation method, including steps S20-S23, which are described as follows:

[0065] Step S20, when the data writing into the HUDI table reaches the target watermark, the Instant information of the day is extracted from the preset timeline.

[0066] Specifically, the data can be real-time data generated by mobile terminals collected through burying points, or other real-time streaming data; it can also be Binlog logs based on MySQL, or incremental data describing database data changes. The data can be written into the HUDI table by Fink. The HUDI table is organized into a directory structure under a specified directory (basepath) on HDFS, and the table is divided into multiple partitions. The partition is in the form of a directory, and under each directory there will be multiple file shards belonging to the partition. Similar to Hive tables, each Hudi table partition is uniquely identified by a partition path (Partition Path).

[0067] The watermark is a key time node for collecting the overall processing progress of the job data stream. The target watermark is a time node for generating a partition view preset by the user.

[0068] The HUDI provides a timeline. The timeline includes all Instant operations performed on the HUDI table at different instant times (Instant time), thereby providing an instant view of the HUDI table and supporting data retrieval in the order of arrival. That is, the timeline has multiple Instant information. The HUDI guarantees the atomicity of the operations performed on the timeline and the consistency of the instant time-based timeline. The timeline corresponds to a group of files under the.hoodie metadata folder under the table base path. Specifically, the data obtained by the new Instant operation is saved as a single file shard, and the data obtained by the old Instant operation is archived to the timeline archive folder to limit the number of files listed by writers and queries.

[0069] The Instant information can include the following contents:

[0070] (1) Instant action: The type of operation performed on the HUDI table. Instant actions include, but are not limited to: COMMIT, which atomically writes a set of records to the HUDI table; CLEAN, a background activity that deletes old file versions that are no longer needed from the HUDI table; DELTA_COMMIT, which atomically writes a batch of records to a MergeOnRead storage type dataset, some or all of which may be written only to the incremental log; COMPACTION, a background activity that coordinates the differential data structures in the HUDI table.

[0071] (2) Instant time: Usually a timestamp that monotonically increases in the order of operation start time.

[0072] (3) Instant state: The current state of an Instant. Each Instant has metadata information in Avro or JSON format, which describes in detail the state of the operation and the state of the Instant at this instant. Any given instant will be in one of the following states: REQUESTED, indicating that it has been scheduled but not yet initialized; INFLIGHT, indicating that the operation is currently being executed; COMPLETED, indicating that the operation has been completed on the timeline.

[0073] (4) The event time interval of all data contained in the file slice. For example, if the generation time interval of all data contained in the current file slice is 15:20:00 on October 10, 2022 to 15:40:00 on October 10, 2022, then the event time interval is 15:20:00 on October 10, 2022 to 15:40:00 on October 10, 2022.

[0074] In this embodiment, during the continuous writing of data to the HUDI table, the event time of the data being written to the HUDI table is monitored to determine whether the current data writing progress has reached the target watermark. When the current data writing progress is detected to have reached the target watermark, the operation of generating a partitioned view is triggered. When the current data writing progress is detected to have not reached the target watermark, data continues to be written to the HUDI table, and the operation of generating a partitioned view is triggered when the current data writing progress is detected to have reached the target watermark.

[0075] In an optional embodiment, when the sum of the event time of the data written into the HUDI table and the delay time preset by the user exceeds the target watermark, it is determined that the progress of data writing into the HUDI table reaches the target watermark.

[0076] Specifically, the delay time can be set and modified according to actual conditions.

[0077] For example, the delay time is 30 minutes, the target watermark is 00:30:00 of the next day of the current day, when it is detected that the event time of the data written into the HUDI table is later than 24:00:00 of the current day, it can be determined that the progress of data writing into the HUDI table reaches the target watermark; when it is detected that the event time of the data written into the HUDI table is not later than 24:00:00 of the current day, it can be determined that the progress of data writing into the HUDI table has not reached the target watermark.

[0078] In this embodiment, whether the progress of data writing into the HUDI table reaches the target watermark is determined by the sum of the event time of the data and the delay time preset by the user, which can make the distinction of the actual boundary of the data more accurate.

[0079] In another optional embodiment, whether the progress of data writing into the HUDI table reaches the target watermark can also be determined by comparing whether the current watermark matches the target watermark in real time. If the current watermark is the same as the target watermark, it can be determined that the progress of data writing into the HUDI table reaches the target watermark.

[0080] In step S21, it is detected whether the file fragment corresponding to each Instant information extracted includes cross-day data, and when it is detected that the file fragment corresponding to the current Instant information contains cross-day data, data filtering logic is added to the current Instant information, wherein the data filtering logic is used to delete the cross-day data contained in the file fragment corresponding to the current Instant information.

[0081] Specifically, the cross-day data refers to data that is not generated on the current day. For example, if the current data is generated on October 16, 2022, the current data is cross-day data.

[0082] In the embodiment, after each Instant information is extracted, the data contained in the file fragment corresponding to each Instant information is detected to determine whether the file fragment contains cross-day data. When it is determined that the file fragment contains cross-day data, data filtering logic is added to the Instant information, so that subsequent cross-day data deletion operations can be performed on the data in the file fragment according to the data filtering logic. When it is determined that the file fragment does not contain cross-day data, no processing is performed on the Instant information.

[0083] In an optional embodiment, the Instant information includes an event time interval of all data contained in the file fragment, refer to Figure 3 The detection of whether the extracted file fragment corresponding to each Instant information includes cross-day data includes: step S30, detecting whether the event time interval in the extracted Instant information contains cross-day time; step S31, when it is detected that the event time interval in the current Instant information contains cross-day time, it is determined that the file fragment corresponding to the current Instant information includes cross-day data.

[0084] Specifically, the cross-day time refers to a time that does not belong to the current day, for example, when the current day is October 15, 2022, 00:05:00 on October 16, 2022 belongs to the cross-day time.

[0085] In the embodiment, by detecting the event time interval in each Instant information, without detecting the event time of each data in the file fragment corresponding to each Instant information, it can be determined that the file fragment corresponding to the current Instant information includes cross-day data, which improves the detection efficiency and reduces resource consumption.

[0086] Step S22, data snapshot processing is performed on all Instant information after detection processing to obtain Instant information of the partition view.

[0087] Specifically, savepoint method can be used to perform data snapshot processing on all Instant information after detection processing to obtain Instant information of the partition view.

[0088] In an optional embodiment, in order to save data storage space, refer to Figure 4 The data snapshot processing on all Instant information after detection processing to obtain Instant information of the partition view includes:

[0089] Step S40, performing a first snapshot operation on the Instant information containing data filtering logic in all the Instant information after the detection processing, to obtain the first target Instant information of the partition view.

[0090] Specifically, the first snapshot operation can be an operation of copying the Instant information containing data filtering logic, and accordingly, the obtained first target Instant information is the Instant information added with data filtering logic.

[0091] Step S41, performing a second snapshot operation on the Instant information not containing data filtering logic in all the Instant information after the detection processing, to obtain the second target Instant information of the partition view, wherein the Instant information of the partition view comprises the first target Instant information and the second target Instant information.

[0092] Specifically, the second target Instant information can be used to indicate which Instant information has not changed (without adding data filtering logic) relative to the original Instant information. The second target Instant information can comprise an identifier indicating that the original Instant information has not changed, such as "1". The second target Instant information also contains information indicating to which original Instant information the current Instant information specifically belongs, such as "the second".

[0093] Step S23, performing file merging processing on all file shards of the day according to the Instant information of the partition view, to generate the partition view.

[0094] Specifically, by performing file merging processing on all file shards of the day according to the Instant information of the partition view, the partition view is generated, so that the data in the generated partition view will only contain data of the day, and will not contain cross-day data. In this way, when subsequent data query analysis is performed, data with very obvious actual boundaries can be queried.

[0095] In optional embodiments, refer to Figure 5The file merging processing of all file segments of the day according to the Instant information of the partition view includes the following steps. In step S50, data filtering logic operation is performed on the file segment containing cross-day data to obtain a target file segment in which the cross-day data is deleted. In step S51, all file segments of the day are obtained according to the Instant information of the partition view. In step S52, file merging processing is performed on the obtained all file segments of the day and the target file segment to generate the partition view.

[0096] In this embodiment, the data filtering logic operation is first performed on the file segment containing cross-day data, so that the obtained target file segment is clean and does not contain files with cross-day data. Then, the target file segment can be replaced by the corresponding file segment, and then the file merging processing is performed together with other file segments to generate the partition view.

[0097] In an embodiment, when generating the partition view, the latest version of the merged file can also be taken out from the data file of the day according to the Instant information of the partition view, and then the latest version of the merged file and other small file segments and target file segments that do not belong to the latest version of the merged file are used to generate the partition view.

[0098] It should be noted that the latest version of the merged file refers to the file segment generated by performing file merging operation at the latest time on the preset time axis.

[0099] The partition view generation method provided in the embodiments of the present application extracts Instant information of the day from a preset time axis when the progress of data written into the HUDI table reaches the target watermark. It is detected whether cross-day data is included in the file segment corresponding to each extracted Instant information, and when it is detected that the file segment corresponding to the current Instant information contains cross-day data, data filtering logic is added to the current Instant information, wherein the data filtering logic is used to delete the cross-day data included in the file segment corresponding to the current Instant information. Data snapshot processing is performed on all Instant information after detection processing to obtain Instant information of the partition view. File merging processing is performed on all file segments of the day according to the Instant information of the partition view to generate the partition view. The partition view in the present application is obtained by cutting and merging all file segments written into the HUDI table of the day based on the target watermark, so that the partition view generated in the present application only contains data of the day and no longer contains cross-day data, so that the actual boundary of the data is accurately distinguished.

[0100] In an exemplary embodiment, the method further comprises:

[0101] writing the Instant information of the partition view and the partition view into a preset partition.

[0102] Specifically, the preset partition is a partition designated by the user in advance, for example, the partition name (partitionName) designated in advance is logdate1. In this way, after the Instant information of the partition view and the partition view are generated, they are written into logdate1.

[0103] In an embodiment, in order to distinguish the partition view from normal instant partition data, the preset partition and the normal instant partition data are stored in different directories.

[0104] In an exemplary embodiment, the method further comprises:

[0105] setting the Instant information of the partition view as a node on the time axis for querying.

[0106] Specifically, in order to facilitate the downstream engine to query the partition view, the Instant information of the partition view can be set as a node on the time axis, thereby opening a channel from the partition view to the time axis. In this way, when the downstream engine queries the partition view, the Instant information on the time axis can be directly queried.

[0107] It should be noted that when the downstream engine queries the partition view, it also needs to determine whether the partition view is set in the HUDI table. When the partition view is set in the HUDI table, the query request of the user can be routed to the Instant information of the partition view.

[0108] For ease of understanding, the following will be described in combination with Figure 6 An exemplary application is provided.

[0109] The data in the multiple Instant information (DeltacommitT1, DeltacommitT2 and DeltacommitT3) belonging to November 1, 2020 on the Timeline is analyzed, and it is found that the cross-day data appears in DeltacommitT3. Therefore, after this, the partition view generation operation is performed, and the Instant information (Fork T4) corresponding to the partition view generation operation is set as a node on the Timeline. Similarly, the data in the multiple Instant information (DeltacommitT5, DeltacommitT6 and CompactT7) belonging to November 2, 2020 on the Timeline is analyzed, and it is found that the cross-day data appears in DeltacommitT7. Therefore, after this, the partition view generation operation is performed, and the Instant information (Fork T8) corresponding to the partition view generation operation is set as a node on the Timeline.

[0110] Embodiment Two

[0111] Figure 7 A block diagram of a partition view generation apparatus according to Embodiment Two of the present application is shown, which can be divided into one or more program modules, the one or more program modules are stored in a storage medium and executed by one or more processors to complete the embodiments of the present application. The program modules referred to in the embodiments of the present application refer to a series of computer program instruction segments capable of completing a specific function, and the functions of the program modules in the embodiments will be specifically described below.

[0112] As shown in Figure 7 The partition view generation apparatus 70 can include an extraction module 71, a detection module 72, a snapshot module 73 and a generation module 74, wherein:

[0113] The extraction module 71 is configured to extract the Instant information of the day from the preset timeline when the progress of data writing into the HUDI table reaches the target watermark;

[0114] The detection module 72 is configured to detect whether the cross-day data is included in the file slice corresponding to each Instant information, and add data filtering logic in the current Instant information when it is detected that the cross-day data is included in the file slice corresponding to the current Instant information, wherein the data filtering logic is used to delete the cross-day data included in the file slice corresponding to the current Instant information;

[0115] The snapshot module 73 is configured to perform data snapshot processing on all the Instant information after detection processing, to obtain Instant information of a partition view.

[0116] The generation module 74 is configured to perform file merging processing on all the file segments of the day according to the Instant information of the partition view, to generate a partition view.

[0117] In an optional embodiment, the Instant information includes an event time interval of all data contained in a file segment, and the detection module 72 is further configured to detect whether the event time interval in each extracted Instant information contains cross-day time, and determine that the file segment corresponding to the current Instant information includes cross-day data when it is detected that the event time interval in the current Instant information contains cross-day time.

[0118] In an optional embodiment, the snapshot module 73 is further configured to perform a first snapshot operation on the Instant information containing data filtering logic in all the Instant information after detection processing, to obtain first target Instant information of a partition view, and perform a second snapshot operation on the Instant information not containing data filtering logic in all the Instant information after detection processing, to obtain second target Instant information of a partition view, wherein the Instant information of the partition view includes the first target Instant information and the second target Instant information.

[0119] In an optional embodiment, the generation module 74 is further configured to perform a data filtering logic operation on the file segment containing cross-day data according to the Instant information of the partition view, to obtain a target file segment in which cross-day data is deleted, obtain all the file segments of the day according to the Instant information of the partition view, and perform file merging processing on the obtained all the file segments of the day and the target file segment, to generate a partition view.

[0120] In an optional embodiment, the partition view generation apparatus 70 further includes a determination module.

[0121] The determination module is configured to determine that the progress of data writing into the HUDI table reaches the target watermark when it is detected that the sum of the event time of the data written into the HUDI table and the delay time set by the user exceeds the target watermark.

[0122] In an optional embodiment, the partition view generation apparatus 70 further includes a writing module.

[0123] The writing module is further configured to write the Instant information of the partition view and the partition view into a preset partition.

[0124] In an optional embodiment, the partition view generation apparatus 70 further comprises a setting module.

[0125] The setting module is configured to set the Instant information of the partition view as a node on the timeline for querying.

[0126] Embodiment Three

[0127] Figure 8 A hardware architecture schematic diagram of a computer device suitable for implementing the partition view generation method according to Embodiment Three of the present application is schematically shown. The computer device 10000 can be the service platform 2 or a part thereof. The computer device 10000 is a device capable of automatically performing numerical calculation and / or information processing according to pre-set or stored instructions. For example, it can be a rack server, a blade server, a tower server or a cabinet server (including a standalone server or a server cluster composed of multiple servers) and the like. As shown in the figure, the computer device 10000 at least includes but is not limited to a memory 10010, a processor 10020, a network interface 10030 which can be communicatively linked through a system bus. Among them: Figure 8

[0128] ​The memory 10010 includes at least one type of computer-readable storage media, such as a flash memory, a hard disk, a multimedia card micro (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, and the like. In some embodiments, the memory 10010 can be an internal memory module of the computer device 10000, such as a hard disk or a memory of the computer device 10000. In other embodiments, the memory 10010 can also be an external memory device of the computer device 10000, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Of course, the memory 10010 can include both an internal memory module and an external memory device of the computer device 10000. In this embodiment, the memory 10010 is generally used to store an operating system and various application programs installed in the computer device 10000, such as program codes of the partition view generation method, and the like. In addition, the memory 10010 can also be used to temporarily store various data that have been output or will be output.

[0129] The processor 10020 can be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 10020 is generally used to control the overall operation of the computer device 10000, such as performing control and processing related to data interaction or communication of the computer device 10000, and the like. In this embodiment, the processor 10020 is used to run program codes or process data stored in the memory 10010.

[0130] The network interface 10030 can include a wireless network interface or a wired network interface, and is generally used to establish a communication connection between the computer device 10000 and other computer devices. For example, the network interface 10030 is used to connect the computer device 10000 with an external terminal through a network, establish a data transmission channel and a communication connection between the computer device 10000 and the external terminal, and the like. The network can be an Intranet, the Internet, a Global System of Mobile communication (GSM), a Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, Wi-Fi, and the like wireless or wired network.

[0131] It should be noted that, Figure 8 Only the computer device with the components 10010-10030 is shown, but it should be understood that all the shown components are not required to be implemented, and more or fewer components can be alternatively implemented.

[0132] In this embodiment, the partition view generation method stored in the memory 10010 can also be divided into one or more program modules, and executed by one or more processors (in this embodiment, the processor 10020) to complete the present application.

[0133] Embodiment Four

[0134] The present embodiment also provides a computer readable storage medium, the computer readable storage medium has a computer program stored thereon, and the computer program is executed by a processor to implement the steps of the partition view generation method in the embodiments.

[0135] In this embodiment, the computer readable storage medium includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the computer readable storage medium can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. In other embodiments, the computer readable storage medium can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the computer device. Of course, the computer readable storage medium can also include both the internal storage unit and the external storage device of the computer device. In this embodiment, the computer readable storage medium is usually used to store an operating system and various application software installed on the computer device, such as program codes of the partition view generation method in the embodiments, etc. In addition, the computer readable storage medium can also be used to temporarily store various data that have been output or will be output.

[0136] Obviously, those skilled in the art should understand that each module or each step of the above-mentioned embodiments of the present application can be realized by a general computing device, which can be concentrated on a single computing device or distributed on a network composed of multiple computing devices, and optionally, each module or each step can be realized by program codes executable by a computing device, so that each module or each step can be stored in a storage device and executed by a computing device, and in some cases, the steps shown or described can be executed in different order, or each module or each step can be manufactured into an individual integrated circuit module, or multiple modules or steps can be manufactured into a single integrated circuit module. Therefore, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0137] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings of the present application, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for generating a partitioned view, characterized in that, The method for generating the partitioned view includes: When the progress of writing data to the HUDI table reaches the target watermark, extract the Instant information for that day from the preset timeline; The system detects whether the file segments corresponding to each extracted Instant information include cross-day data. When it detects that the file segments corresponding to the current Instant information contain cross-day data, it adds data filtering logic to the current Instant information. The data filtering logic is used to delete the cross-day data contained in the file segments corresponding to the current Instant information. A data snapshot is taken of all the detected and processed Instant information to obtain the Instant information of the partitioned view; Based on the Instant information of the partition view, all file fragments for the day are merged to generate a partition view; The step of performing data snapshot processing on all detected Instant information to obtain Instant information for the partitioned view includes: Perform the first snapshot operation on all Instant information containing data filtering logic after detection and processing to obtain the first target Instant information of the partition view; A second snapshot operation is performed on all Instant information that does not contain data filtering logic after detection and processing to obtain the second target Instant information of the partition view, wherein the Instant information of the partition view includes the first target Instant information and the second target Instant information.

2. The partitioned view generation method according to claim 1, characterized in that, The Instant information includes the event time interval of all data contained in the file fragment, and the detection of whether the file fragment corresponding to each extracted Instant information includes cross-day data includes: Detect whether the event time intervals in each extracted Instant message contain time spanning multiple days; When it is detected that the event time interval in the current Instant information contains cross-day times, it is determined that the file fragment corresponding to the current Instant information includes cross-day data.

3. The method for generating a partitioned view according to claim 1, characterized in that, The step of merging all file fragments for the day based on the Instant information of the partition view to generate the partition view includes: Based on the Instant information of the partition view, perform data filtering logic operations on the file fragments containing cross-day data to obtain the target file fragments for which cross-day data is deleted; Retrieve all file fragments for the day based on the Instant information of the partition view; The file fragments obtained for the day and the target file fragment are merged to generate a partition view.

4. The method for generating a partitioned view according to any one of claims 1 to 3, characterized in that, The method further includes: When the sum of the event time of data written to the HUDI table and the user-preset delay time exceeds the target watermark, it is determined that the progress of data writing to the HUDI table has reached the target watermark.

5. The method for generating a partitioned view according to any one of claims 1 to 3, characterized in that, The method further includes: Write the Instant information of the partition view and the partition view to the preset partition.

6. The method for generating a partitioned view according to any one of claims 1 to 3, characterized in that, The method further includes: Set the Instant information of the partitioned view as a node on the timeline for querying.

7. A partitioned view generation device, characterized in that, The device includes: The extraction module is used to extract the Instant information of the day from the preset timeline when the progress of writing data to the HUDI table reaches the target watermark. The detection module is used to detect whether the file segments corresponding to each extracted Instant information include cross-day data, and when it is detected that the file segments corresponding to the current Instant information contain cross-day data, data filtering logic is added to the current Instant information, wherein the data filtering logic is used to delete the cross-day data contained in the file segments corresponding to the current Instant information. The snapshot module is used to perform data snapshot processing on all detected and processed Instant information to obtain the Instant information of the partition view; The generation module is used to perform file merging processing on all file fragments of the day based on the Instant information of the partition view, and generate a partition view; The step of performing data snapshot processing on all detected Instant information to obtain Instant information for the partitioned view includes: Perform the first snapshot operation on all Instant information containing data filtering logic after detection and processing to obtain the first target Instant information of the partition view; A second snapshot operation is performed on all Instant information that does not contain data filtering logic after detection and processing to obtain the second target Instant information of the partition view, wherein the Instant information of the partition view includes the first target Instant information and the second target Instant information.

8. A computer device, the computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it is used to implement the steps of the partition view generation method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program thereon, characterized in that, The computer program may be executed by at least one processor to cause the at least one processor to perform the steps of the partition view generation method according to any one of claims 1 to 6.

10. A computer program product, the computer program product comprising a computer program, characterized in that, The computer program may be executed by at least one processor to cause the at least one processor to perform the steps of the partition view generation method according to any one of claims 1 to 6.

Images