Data writing method and device of Hudi data, equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence and information security, and relates to a data writing method for Hud i data, which comprises the following steps: when a data writing request is received, acquiring to-be-written data in the data writing request, and constructing a cache table in a cache space based on the to-be-written data; writing to-be-written data into the cache table, and generating a data writing strategy which comprises a data writing trigger condition and a writing mode; and writing the data in the cache table into the target address based on the data writing strategy. The invention further provides a data writing device of the Hudi data, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the to-be-written data can be stored in a block chain. According to the method and the device, the to-be-written data is written into the cache table and then is written according to the data writing strategy, so that system resources occupied by writing each time are saved, and the data writing efficiency of the Hudi data is improved.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and in particular to a method, device, equipment and storage medium for writing Hudi data. Background Art

[0002] Hudi is an open source data processing framework for storing and managing large analytical data sets. It is designed to integrate seamlessly with existing big data ecosystems such as Apache Spark, Apache Hive, Presto, etc. In Hudi, the Copy-on-Write (CoW) mechanism is used to optimize the performance of write operations, especially when processing large-scale data sets. Through the CoW mechanism, Hudi can avoid unnecessary data copying when writing new data, thereby improving write efficiency and reducing storage space usage.

[0003] In the process of implementing the present invention, the inventors found that although the existing CoW mechanism has greatly improved the performance compared with traditional database data writing, its writing efficiency is gradually decreasing as the data increases. How to ensure the writing efficiency in the case of large data volume has become a difficult problem that needs to be solved urgently. Summary of the invention

[0004] The purpose of the embodiments of the present application is to propose a data writing method, device, computer equipment and storage medium for Hudi data, the main purpose of which is to improve the data writing efficiency of Hudi data.

[0005] In order to solve the above technical problems, the embodiment of the present application provides a method for writing data to Hudi data, which adopts the following technical solution:

[0006] When receiving a data write request, obtaining the data to be written in the data write request, and constructing a cache table in the cache space based on the data to be written;

[0007] Write the data to be written into the cache table, and generate a Hudi data writing strategy, where the Hudi data writing strategy includes a data writing trigger condition and a writing method;

[0008] Based on the Hudi data writing strategy, the data in the cache table is written to the target address.

[0009] Furthermore, before constructing a cache table in the cache space based on the data to be written, the method for writing Hudi data further includes the following steps:

[0010] When using the Write reservation function of the Hudi data processing framework, reserve the first space according to the preset configuration parameters;

[0011] The cache space is generated based on the first space.

[0012] Furthermore, the data write trigger condition is that the number of data in the cache table reaches a preset number, and the step of writing the data in the cache table to the target address based on the Hudi data write strategy specifically includes:

[0013] Monitoring the amount of data in the cache table;

[0014] If the number of data in the cache table reaches the preset number, the preset data writing method is used to write the data in the cache table to the target address. The preset data writing method is any one or more combinations of incremental update, full overwrite, merge write or transactional write.

[0015] Furthermore, the writing mode is batch append writing, and the data in the cache table is written to the target address based on the Hudi data writing strategy, including the following steps:

[0016] When the data writing trigger condition is met, the data in the cache table is appended to the target address in a batch append manner.

[0017] Further, constructing a cache table in the cache space based on the data to be written includes:

[0018] Create a node class, and define a linked list node based on the node class, wherein the node class includes a data value, a key value, and a pointer to a previous node and a next node;

[0019] Create a cache class that contains the head node and tail node of the linked list, as well as a dictionary to store the mapping between keys and corresponding nodes;

[0020] A cache instance is created according to the linked list nodes, the cache class and the data to be written to obtain the cache table.

[0021] Furthermore, writing the data to be written into the cache table and generating a Hudi data writing strategy includes:

[0022] Determine the priority of data writing, where the priority is classified based on the importance, timeliness or business needs of the data;

[0023] According to the priority, different data writing trigger conditions are defined for different categories of data, and the trigger conditions may be the amount of data, time interval or occurrence of a specific event;

[0024] Configure a corresponding writing mode for each data writing trigger condition, wherein the writing mode includes incremental update, full overwrite, merge write or transactional write;

[0025] Set the failed retry mechanism for data writing, including the number of retries, retry interval, and retry strategy;

[0026] Configure performance optimization parameters for data writing, including parallelism, buffer size, and compression algorithm;

[0027] Define security requirements for data writing, including data encryption, access control, and audit logs;

[0028] The Hudi data writing strategy is generated according to the trigger condition, the writing method, the failure retry mechanism, the performance optimization parameters and the security requirements.

[0029] In order to solve the above technical problems, the embodiment of the present application also provides a data writing device for Hudi data, which adopts the following technical solution:

[0030] A request parsing module, configured to, upon receiving a data write request, obtain the data to be written in the data write request, and construct a cache table in the cache space based on the data to be written;

[0031] A strategy generation module, used to write the data to be written into the cache table and generate a Hudi data writing strategy, wherein the Hudi data writing strategy includes a data writing trigger condition and a writing method;

[0032] A data writing module is used to write the data in the cache table to the target address based on the Hudi data writing strategy.

[0033] Furthermore, the data writing device of the Hudi data also includes:

[0034] The space reservation module is used to reserve the first space according to the preset configuration parameters when using the Write reservation function of the Hudi data processing framework;

[0035] A space generation module is used to generate the cache space based on the first space.

[0036] Furthermore, the data writing trigger condition is that the number of data in the cache table reaches a preset number, and the data writing module includes:

[0037] A data monitoring unit, used to monitor the amount of data in the cache table;

[0038] The first writing unit is used to write the data in the cache table to the target address by adopting a preset data writing method if the number of data in the cache table reaches the preset number, and the preset data writing method is any one or more combinations of incremental update, full overwrite, merge write or transactional write.

[0039] Furthermore, the writing mode is batch append writing, and the data writing module further includes:

[0040] The append writing unit is used to append the data in the cache table to the target address in a batch append manner when the data writing trigger condition is met.

[0041] Furthermore, the data writing device of the Hudi data also includes:

[0042] The space cleaning module is used to clean up and release the cache space.

[0043] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0044] When receiving a data write request, obtaining the data to be written in the data write request, and constructing a cache table in the cache space based on the data to be written;

[0045] Write the data to be written into the cache table, and generate a Hudi data writing strategy, where the Hudi data writing strategy includes a data writing trigger condition and a writing method;

[0046] Based on the Hudi data writing strategy, the data in the cache table is written to the target address.

[0047] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0048] When receiving a data write request, obtaining the data to be written in the data write request, and constructing a cache table in the cache space based on the data to be written;

[0049] Write the data to be written into the cache table, and generate a Hudi data writing strategy, where the Hudi data writing strategy includes a data writing trigger condition and a writing method;

[0050] Based on the Hudi data writing strategy, the data in the cache table is written to the target address.

[0051] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0052] By building a cache table in the cache space, writing the data to be written into the cache table, and then writing it according to the Hudi data writing strategy, the system resources occupied by each write are saved and the data writing efficiency of Hudi data is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0054] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0055] Figure 2 A flowchart of an embodiment of a method for writing data to Hudi data according to the present application;

[0056] Figure 3 It is a structural schematic diagram of an embodiment of a data writing device for Hudi data according to the present application;

[0057] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0059] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0060] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0061] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0062] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0063] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0064] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0065] It should be noted that the data writing method of Hudi data provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the data writing device of Hudi data is generally arranged in the server / terminal device.

[0066] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0067] Continue to refer Figure 2 , shows a flow chart of an embodiment of a method for writing data to Hudi data according to the present application. The method for writing data to Hudi data comprises the following steps:

[0068] Step S201, when a data write request is received, the data to be written in the data write request is obtained, and a cache table is constructed in the cache space based on the data to be written.

[0069] Specifically, when the client needs to write the data to be written into the database, it sends a data write request to the server. The data write request includes but is not limited to: request identification, client information, data content (data to be written), data structure (data structure of the data to be written), target address, timestamp, transaction information, security credentials, etc. After receiving the data write request, the server parses the data write request to obtain the data to be written, and then builds a cache table in the cache space according to the data structure and quantity of the data to be written.

[0070] Among them, the cache table is a technology used in data processing and storage systems to improve data access speed and query performance.

[0071] Cache space refers to an area of ​​memory used to store temporary data that is preloaded to speed up subsequent access.

[0072] It should be noted that the cache table is built based on the Hudi table structure, which is conducive to quickly writing the data in the cache table to the target address where the Hudi data is stored, thereby improving the data writing efficiency.

[0073] In this embodiment, the data writing method of Hudi data runs on the electronic device (for example Figure 1 The server / terminal device shown in the figure can receive the data to be written sent by the client through a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0074] Taking the field of financial risk control as an example, the data to be written includes user information, transaction records, etc.

[0075] Step S202, write the data to be written into the cache table, and generate a Hudi data write strategy, which includes a data write trigger condition and a write method.

[0076] Specifically, after the cache table is generated, the data to be written is written into the cache table, and a write strategy is generated according to the pre-set parameter information and the data situation in the cache table. The write strategy includes the triggering conditions and the writing method of data writing.

[0077] The writing method includes but is not limited to direct writing, batch writing, append writing, etc. The triggering condition of data writing can be set according to actual needs, such as triggering by time, or triggering by the data in the cache table reaching a fixed amount of data, or other methods, which are not specifically limited here.

[0078] In a specific implementation, constructing a cache table in the cache space based on the data to be written includes:

[0079] Create a node class and define a linked list node based on the node class. The node class contains data values, key values, and pointers to the previous node and the next node.

[0080] Create a cache class that contains the head node and tail node of the linked list, as well as a dictionary to store the mapping between keys and corresponding nodes;

[0081] A cache instance is created according to the linked list nodes, the cache class and the data to be written to obtain the cache table.

[0082] In this embodiment, pre-generating the cache table is helpful to improve the efficiency in the data writing stage.

[0083] In a specific implementation, writing the data to be written into the cache table and generating the Hudi data writing strategy includes:

[0084] Determine the priority of data writing, where the priority is classified based on the importance, timeliness or business needs of the data;

[0085] According to the priority, different data writing trigger conditions are defined for different categories of data, and the trigger conditions may be the amount of data, time interval or occurrence of a specific event;

[0086] Configure the corresponding writing mode for each data writing trigger condition, including incremental update, full overwrite, merge write or transactional write;

[0087] Set the failed retry mechanism for data writing, including the number of retries, retry interval, and retry strategy;

[0088] Configure performance optimization parameters for data writing, such as parallelism, buffer size, and compression algorithm, to improve data writing efficiency and reduce system resource consumption;

[0089] Define security requirements for data writing, including data encryption, access control, and audit logs to protect data security and compliance;

[0090] Generate a Hudi data writing strategy based on trigger conditions, writing methods, failure retry mechanism, performance optimization parameters, and security requirements.

[0091] In this embodiment, by analyzing the data to be written and automatically generating a corresponding writing strategy, it is beneficial to improve the efficiency of data writing.

[0092] Step S203, based on the Hudi data writing strategy, write the data in the cache table to the target address.

[0093] It should be noted that the target address in this embodiment is the address corresponding to the database or data space storing Hudi data.

[0094] The main difference between Hudi data and traditional data is its optimization of the data lake environment and some specific functions it provides. The following are some key features of Hudi data compared to ordinary data:

[0095] Transactional Writes: Hudi supports transactional writes, which means it ensures the consistency and reliability of data writes, similar to transaction processing in traditional database systems.

[0096] Incremental updates and deletes: Hudi allows incremental updates and deletes to data instead of overwriting the entire dataset. This is very useful for processing rapidly changing data, such as streaming data or real-time events.

[0097] Multiple data format support: Hudi can be integrated with multiple data formats, including Parquet, Avro, etc., which makes it more flexible and can adapt to different data storage needs.

[0098] Query performance optimization: Hudi provides a data indexing mechanism for real-time and batch queries to optimize query performance, allowing users to perform various query operations more efficiently without sacrificing performance.

[0099] Architectural flexibility: Hudi's architectural design is flexible and can be integrated with existing data lake architectures and tools. It supports deployment on the cloud or locally and can be used with different storage backends (such as HDFS, S3, etc.).

[0100] Timeline management: Hudi maintains a timeline that records all operations on the data set, thereby providing a view of the data set from different time points.

[0101] File Organization: Hudi organizes datasets into a directory structure. The dataset is divided into multiple partitions, which are folders containing the data files for that partition, similar to Hive tables.

[0102] Storage types and views: Hudi defines how data is indexed and laid out on a distributed file system, and how primitives and timeline activities are implemented on top of this organization. It supports different storage types and views, such as copy-on-write and merge-on-read, as well as near-real-time, incremental, and read-optimized views.

[0103] Data Lake Management: Hudi’s focus on large-scale data lake management makes it a powerful tool for handling complex data operations in large-scale data lakes.

[0104] In this embodiment, by constructing a cache table in the cache space, writing the data to be written into the cache table, and then writing according to the Hudi data writing strategy, the system resources occupied by each write are saved and the data writing efficiency of Hudi data is improved.

[0105] In an optional implementation manner of the present embodiment, before step S201, that is, before building a cache table in the cache space based on the data to be written, the data writing method of Hudi data further includes the following steps:

[0106] When using the Write reservation function of the Hudi data processing framework, reserve the first space according to the preset configuration parameters;

[0107] Based on the first space, a cache space is generated.

[0108] Among them, the preset configuration parameters can be adjusted according to actual application needs and hardware resources, and are not specifically limited here.

[0109] The cache space is generated based on the first space, specifically, all or part of the space is allocated from the first space as the area of ​​the cache space, and then generated by dividing the data space.

[0110] In this embodiment, by generating cache space in advance, the time for subsequent storage of data to be written is reduced, thereby improving the efficiency of the data writing process.

[0111] In an optional implementation manner of the first embodiment of the present invention, in step S203, the data writing trigger condition is that the number of data in the cache table reaches a preset number, and based on the Hudi data writing strategy, the step of writing the data in the cache table to the target address specifically includes:

[0112] Monitor the amount of data in the cache table;

[0113] If the number of data in the cache table reaches a preset number, the data in the cache table is written to the target address using a preset data writing method, where the preset data writing method is any one or more combinations of incremental update, full overwrite, merge write or transactional write.

[0114] The preset number of entries may be a directly preset number, or may be dynamically generated according to current system resources and the amount of data in the cache table, which is not specifically limited here.

[0115] Among them, static setting is to directly set a fixed number of data in the cache table as the preset number. For example, it can be set that when the data entries in the cache table reach 1,000, the data write operation is triggered. This method is simple and direct, and is suitable for scenarios with low real-time requirements.

[0116] Dynamic generation is to dynamically adjust the preset number of entries based on the current system resource usage and the amount of data in the cache table. This can be achieved through the following steps:

[0117] Monitor system resources: Real-time monitoring of key indicators such as system memory usage, CPU load, disk I / O, etc.

[0118] Analyze data access patterns: Analyze the access frequency and patterns of data in the cache table to identify hot data and access peaks.

[0119] Predict data growth trends: Based on historical data and current trends, predict the amount of data growth in the future.

[0120] Intelligent adjustment strategy: According to the above monitoring and analysis results, the preset number of entries is intelligently adjusted. For example, when system resources are sufficient and data access frequency is high, the preset number of entries can be appropriately increased to reduce the frequency of write operations; conversely, when resources are tight or data access frequency is low, the preset number of entries can be reduced to avoid excessive occupation of system resources.

[0121] Implement a feedback mechanism: Implement a feedback mechanism to evaluate the effectiveness of the data writing strategy and further adjust the preset number of entries based on the actual results.

[0122] The method of dynamically generating a preset number of entries can more flexibly adapt to different system loads and business needs, improve resource utilization and data writing efficiency. This method requires more complex logic and algorithms to implement, but it can better balance performance and resource consumption, and is suitable for scenarios with high requirements for real-time and system performance.

[0123] In this embodiment, the amount of data in the cache table is used as a monitoring indicator. When the amount of data in the cache table reaches a preset number, writing is triggered to avoid resource waste caused by multiple operations, which is conducive to improving resource utilization and data writing efficiency.

[0124] In an optional implementation manner of the first embodiment of the present invention, in step S203, the writing mode is batch append writing, and based on the Hudi data writing strategy, the data in the cache table is written to the target address, including the following steps:

[0125] When the data write trigger condition is met, the data in the cache table is appended to the target address using a batch append method.

[0126] Preferably, the trigger condition adopted in this embodiment is to perform batch appending when a preset number of entries is reached, such as 1000 entries, so as to avoid server congestion caused by excessive data and to avoid wasting system resources due to frequent calls.

[0127] Among them, Bulk Insert is a data processing technology, especially in databases and big data processing systems, for efficiently inserting a large number of data records into data storage. This technology is usually used to reduce the overhead of write operations and improve the efficiency of data insertion. In this embodiment, the data is generated sequentially, and the use of bulk append does not require frequent updates compared to full coverage, merge writes or transactional writes, which can reduce the number of database writes and improve efficiency.

[0128] In this embodiment, a large amount of data is appended and written at the same time based on a batch append method, which saves system resources and is conducive to improving the efficiency of data writing.

[0129] In an optional implementation manner of the first embodiment of the present invention, after step S203, that is, after the step of writing the data in the cache table to the target address based on the Hudi data writing strategy, the following steps are further included:

[0130] Clean up and release the cache space.

[0131] Among them, in the traditional data writing environment, if the cache space is not built, a large amount of redundant data will be generated. In this embodiment, the cleaning and release of the Hudi data cache space is an important step, which helps to improve resource utilization and data writing efficiency. Cleaning and release mainly include the following aspects:

[0132] Delete old versions of files: Hudi's cleaning service (Cleaner) regularly deletes old versions of data files that are no longer needed because they have been merged or overwritten. This is achieved by configuring hoodie.cleaner.policy, and you can choose different strategies, such as keeping the latest N file versions, keeping the files in the last N hours, or keeping the latest N submissions.

[0133] Log file merging: In Merge on Read (MoR) mode, Hudi merges incremental updates in log files into base data files. This process is called compaction, which can reduce the number of log files, optimize read performance, and free up storage space.

[0134] Merging of small files: Hudi provides a small file processing function that can merge small files into larger files to reduce the number of files and improve the efficiency of the file system. This can be achieved by configuring hoodie.cleaner.fileversions.retained, which controls the number of retained file versions.

[0135] Configuration of cleanup strategy: Users can configure cleanup strategies based on data access patterns and business requirements to ensure that ongoing queries are not affected while cleaning up old versions of files. For example, you can set a certain number of file versions to be retained, or retain file versions based on time.

[0136] Metadata table maintenance: Hudi's metadata table records the metadata information of the file, including the version and status of the file. The cleanup service will update the metadata table to ensure that the correct file version can be accessed when querying.

[0137] Space Reclamation: Hudi's cleanup service reclaims space that is no longer needed to control storage costs. This includes deleting old versions of data files and cleaning up log files.

[0138] By timely cleaning and releasing the cache space, resource utilization is saved, providing better protection for data processing, which is conducive to improving the efficiency of subsequent data writing.

[0139] It should be emphasized that in order to further ensure the privacy and security of the data to be written, the above-mentioned abnormality identification results can also be stored in a node of a blockchain.

[0140] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0141] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0142] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0143] This application can be applied in the field of smart finance, thereby promoting the construction of smart cities.

[0144] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0145] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0146] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a device for writing data to Hudi data. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0147] like Figure 3 As shown, the data writing device 300 of Hudi data described in this embodiment includes: a request parsing module 301, a strategy generating module 302 and a data writing module 303.

[0148] in:

[0149] The request parsing module 301 is used to obtain the data to be written in the data writing request when receiving the data writing request, and to build a cache table in the cache space based on the data to be written;

[0150] A strategy generation module 302 is used to write the data to be written into the cache table and generate a Hudi data writing strategy, which includes a data writing trigger condition and a writing method;

[0151] The data writing module 303 is used to write the data in the cache table to the target address based on the Hudi data writing strategy.

[0152] In this embodiment, by constructing a cache table in the cache space, writing the data to be written into the cache table, and then writing according to the Hudi data writing strategy, the system resources occupied by each write are saved and the data writing efficiency of Hudi data is improved.

[0153] Furthermore, the data writing device of Hudi data also includes:

[0154] The space reservation module is used to reserve the first space according to the preset configuration parameters when using the Write reservation function of the Hudi data processing framework;

[0155] The space generation module is used to generate a cache space based on the first space.

[0156] In this embodiment, by generating cache space in advance, the time for subsequent storage of data to be written is reduced, thereby improving the efficiency of the data writing process.

[0157] Furthermore, the request parsing module 301 includes:

[0158] A first creation unit, used for creating a node class, defining a linked list node based on the node class, wherein the node class includes a data value, a key value, and a pointer to a previous node and a next node;

[0159] The second creation unit is used to create a cache class, which includes the head node and tail node of the linked list, and a dictionary to store the mapping between the key and the corresponding node;

[0160] The third creation unit is used to create a cache instance according to the linked list nodes, the cache class and the data to be written, so as to obtain the cache table.

[0161] In this embodiment, by constructing a cache table in the cache space, writing the data to be written into the cache table, and then writing according to the Hudi data writing strategy, the system resources occupied by each write are saved and the data writing efficiency of Hudi data is improved.

[0162] Furthermore, the strategy generation module 302 includes:

[0163] A priority determination unit, used to determine the priority of data writing, wherein the priority is classified based on the importance, timeliness or business requirements of the data;

[0164] A first configuration unit, configured to define different data write trigger conditions for different categories of data according to the priority, wherein the trigger condition may be the amount of data, a time interval, or the occurrence of a specific event;

[0165] A second configuration unit is used to configure a corresponding writing mode for each data writing trigger condition, wherein the writing mode includes incremental update, full coverage, merge writing or transactional writing;

[0166] The third configuration unit is used to set a failed retry mechanism for data writing, including the number of retries, the retry interval, and the retry strategy;

[0167] A fourth configuration unit, used to configure performance optimization parameters for data writing, such as parallelism, buffer size, and compression algorithm, to improve the efficiency of data writing and reduce the consumption of system resources;

[0168] The fifth configuration unit is used to define the security requirements for data writing, including data encryption, access control, and audit logs to protect data security and compliance;

[0169] A strategy generation unit is used to generate the Hudi data writing strategy according to the trigger condition, the writing mode, the failure retry mechanism, the performance optimization parameter and the security requirement.

[0170] In this embodiment, by analyzing the data to be written and automatically generating a corresponding writing strategy, it is beneficial to improve the efficiency of data writing.

[0171] Furthermore, the data writing trigger condition is that the number of data in the cache table reaches a preset number, and the data writing module 303 includes:

[0172] A data monitoring unit, used to monitor the amount of data in the cache table;

[0173] The first writing unit is used to write the data in the cache table to the target address by using a preset data writing method if the number of data in the cache table reaches a preset number. The preset data writing method is any one or more combinations of incremental update, full overwrite, merge write or transactional write.

[0174] In this embodiment, the amount of data in the cache table is used as a monitoring indicator. When the amount of data in the cache table reaches a preset number, writing is triggered to avoid resource waste caused by multiple operations, which is conducive to improving resource utilization and data writing efficiency.

[0175] Furthermore, the writing mode is batch append writing, and the data writing module 303 also includes:

[0176] The append writing unit is used to append the data in the cache table to the target address in a batch append manner when the data writing trigger condition is met.

[0177] In this embodiment, a large amount of data is appended and written at the same time based on a batch append method, which saves system resources and is conducive to improving the efficiency of data writing.

[0178] Furthermore, the data writing device of Hudi data also includes:

[0179] The space cleaning module is used to clean up and release the cache space.

[0180] In this embodiment, by timely clearing and releasing the cache space, resource utilization is saved, better protection is provided for data processing, and the efficiency of subsequent data writing is improved.

[0181] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0182] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0183] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0184] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the data writing method of Hudi data, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0185] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions stored in the memory 41 or process data, such as computer-readable instructions for running the data writing method of the Hudi data.

[0186] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0187] This embodiment builds a cache table in the cache space, writes the data to be written into the cache table, and then writes it according to the Hudi data writing strategy, thereby saving system resources occupied by each write and improving the data writing efficiency of Hudi data.

[0188] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the data writing method of Hudi data as described above.

[0189] This embodiment builds a cache table in the cache space, writes the data to be written into the cache table, and then writes it according to the Hudi data writing strategy, thereby saving system resources occupied by each write and improving the data writing efficiency of Hudi data.

[0190] The non-Company software tools or components appearing in the embodiments of this application are merely examples and do not represent actual use.

[0191] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0192] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A method for writing Hud i data, characterized in that: The steps include: When receiving a data write request, obtaining the data to be written in the data write request, and constructing a cache table in the cache space based on the data to be written; Write the data to be written into the cache table, and generate a Hudi data writing strategy, where the Hudi data writing strategy includes a data writing trigger condition and a writing method; Based on the Hudi data writing strategy, the data in the cache table is written to the target address.

2. The method for writing Hudi data according to claim 1, characterized in that: Before building a cache table in the cache space based on the data to be written, the method for writing Hudi data further includes the following steps: When using the Writer reservation function of the Hudi data processing framework, the first space is reserved according to the preset configuration parameters; The cache space is generated based on the first space.

3. The method for writing Hudi data according to claim 1, characterized in that: The data write trigger condition is that the number of data in the cache table reaches a preset number, and the step of writing the data in the cache table to the target address based on the Hudi data write strategy specifically includes: Monitoring the amount of data in the cache table; If the number of data in the cache table reaches the preset number, the preset data writing method is used to write the data in the cache table to the target address. The preset data writing method is any one or more combinations of incremental update, full overwrite, merge write or transactional write.

4. The method for writing Hudi data according to claim 3, characterized in that: The step of writing the data in the cache table to the target address based on the Hudi data writing strategy includes the following steps: When the data writing trigger condition is met, the data in the cache table is appended to the target address in a batch append manner.

5. The method for writing Hudi data according to claim 1, characterized in that: The constructing a cache table in the cache space based on the data to be written comprises: Create a node class, and define a linked list node based on the node class, wherein the node class includes a data value, a key value, and a pointer to a previous node and a next node; Create a cache class that contains the head node and tail node of the linked list, as well as a dictionary to store the mapping between keys and corresponding nodes; A cache instance is created according to the linked list nodes, the cache class and the data to be written to obtain the cache table.

6. The method for writing Hudi data according to claim 1, characterized in that: Writing the data to be written into the cache table and generating a Hudi data writing strategy includes: Determine the priority of data writing, where the priority is classified based on the importance, timeliness or business needs of the data; According to the priority, different data writing trigger conditions are defined for different categories of data, and the trigger conditions may be the amount of data, the time interval or the occurrence of a specific event; Configure a corresponding writing mode for each data writing trigger condition, wherein the writing mode includes incremental update, full overwrite, merge write or transactional write; Set the failed retry mechanism for data writing, including the number of retries, retry interval, and retry strategy; Configure performance optimization parameters for data writing, including parallelism, buffer size, and compression algorithm; Define security requirements for data writing, including data encryption, access control, and audit logs; The Hudi data writing strategy is generated according to the trigger condition, the writing method, the failure retry mechanism, the performance optimization parameters and the security requirements.

7. A data writing device for Hudi data, characterized in that: include: A request parsing module, configured to, upon receiving a data write request, obtain the data to be written in the data write request, and construct a cache table in the cache space based on the data to be written; A strategy generation module, used to write the data to be written into the cache table and generate a Hudi data writing strategy, wherein the Hudi data writing strategy includes a data writing trigger condition and a writing method; A data writing module is used to write the data in the cache table to the target address based on the Hudi data writing strategy.

8. The data writing device of Hudi data according to claim 7 is characterized in that: The data writing trigger condition is that the number of data in the cache table reaches a preset number, and the data writing module includes: A data monitoring unit, used to monitor the amount of data in the cache table; The first writing unit is used to write the data in the cache table to the target address by adopting a preset data writing method if the number of data in the cache table reaches the preset number, and the preset data writing method is any one or more combinations of incremental update, full overwrite, merge write or transactional write.

9. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the method for writing Hudi data as described in any one of claims 1 to 6 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the method for writing Hudi data according to any one of claims 1 to 6 are implemented.

Images