Multi-target table data writing method and device, electronic equipment and storage medium

By generating target resource allocation identifiers and write objects in the Flink engine and dynamically matching target table structure metadata, the problem of writing data to multiple target tables is solved, achieving efficient multi-table path data writing and improving computing resource utilization and task scalability.

CN118939419APending Publication Date: 2024-11-12CHINA TELECOM ARTIFICIAL INTELLIGENCE TECHNOLOGY (BEIJING) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410978473.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

When implementing synchronous writing of data across multiple tables in a database, the existing Flink engine's native components and tools are insufficient to meet the requirements and cannot efficiently handle data writing to multiple target tables.

Method used

By obtaining the element characteristics of the data to be written, a target resource allocation identifier is generated. The resource allocation mapping table and the write object mapping table are traversed and matched to dynamically obtain the structural metadata of the target table, generate the target write object and store it in the corresponding mapping table, and finally write the data to the target table based on the target resource allocation object.

Benefits of technology

This enables a single Flink task to process multiple data sources and write to multiple target table storage paths, improving the utilization of computing resources and the scalability of the task, and avoiding path writing logic that is locked by fixed mapping relationships.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939419A_ABST
    Figure CN118939419A_ABST
Patent Text Reader

Abstract

The invention discloses a multi-target table data writing method and device, electronic equipment and a storage medium. The method comprises the steps of obtaining to-be-written data; calling the first function class to generate a target resource allocation identifier of the to-be-written data according to the element characteristics of the to-be-written data; performing traversal matching in a preset resource allocation mapping table based on the target resource allocation identifier to obtain a matching result; when the matching result is null, obtaining structural metadata of the target table according to the target table name; generating a target write-in object of the target table in response to the creation instruction based on the structural metadata; storing the target write-in object in a preset write-in object mapping table; calling a second function class according to the target write-in object to generate a target resource allocation object; storing the target resource allocation object in a resource allocation mapping table; and calling the target writing object based on the target resource allocation object to write the to-be-written data into the target table. The method can realize multi-target table data writing, and can be widely applied to the technical field of data processing.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, electronic device and storage medium for writing data into a multi-target table. Background Art

[0002] In the field of real-time data collection, conversion and persistence, the Flink computing engine is a common and efficient implementation tool and open source framework. Based on Flink, real-time and offline data can be processed in a distributed, low-latency, exactly-once semantic, and highly reliable manner. For example, by configuring the data source, target table information, and processing logic, a Flink task can be started to write data from the data source to the target table in real time. However, in some complex application scenarios, such as synchronizing data in an entire database and writing it to multiple target tables, native components and tool classes are difficult to meet the requirements. Therefore, how to achieve synchronous writing of multiple tables in an entire database based on the powerful performance of the Flink engine is an urgent problem to be solved. Summary of the invention

[0003] The present invention provides a method, device, electronic device and storage medium for writing data into a multi-target table, aiming to solve one of the technical problems in the related art at least to a certain extent.

[0004] On the one hand, an embodiment of the present invention provides a method for writing data into a multi-target table, comprising:

[0005] Get the data to be written;

[0006] The first function class is called according to the element characteristics of the data to be written to generate a target resource allocation identifier for the data to be written; the target resource allocation identifier carries target library table information, and the target library table information carries a target table name of the target table;

[0007] Based on the target resource allocation identifier, a traversal match is performed in a preset resource allocation mapping table to obtain a matching result;

[0008] When the matching result is empty, obtain the structural metadata of the target table according to the target table name;

[0009] Based on the structural metadata, generating a target write object of the target table in response to a create instruction; storing the target write object in a preset write object mapping table;

[0010] Calling the second function class according to the target write object generates a target resource allocation object; storing the target resource allocation object in the resource allocation mapping table;

[0011] Based on the target resource allocation object, the target writing object is called to write the data to be written into the target table.

[0012] Optionally, the resource allocation mapping table is used to store a first key-value pair, the key element and the value element of the first key-value pair correspond to the associated resource allocation identifier and the resource allocation object in sequence; performing traversal matching in a preset resource allocation mapping table based on the target resource allocation identifier to obtain a matching result includes the following steps:

[0013] Based on the target resource allocation identifier, traverse and match the key elements of all first key-value pairs in the resource allocation mapping table;

[0014] When a key element identical to the target resource allocation identifier is matched, the resource allocation object corresponding to the value element associated with the first key-value pair where the key element is located is taken as the matching result;

[0015] When no key element with the same key as the target resource allocation identifier is matched, the output matching result is empty.

[0016] Optionally, when the matching result is not empty, the matching result contains a resource allocation object; the method further comprises the following steps:

[0017] The resource allocation object in the matching result is used as the target resource allocation object, and the execution jumps to the step of calling the target writing object based on the target resource allocation object to write the to-be-written data into the target table.

[0018] Optionally, obtaining the structure metadata of the target table according to the target table name includes the following steps:

[0019] According to the target table name, obtain the structure metadata of the target table through HTTP request query.

[0020] Optionally, storing the target write object in a preset write object mapping table includes the following steps:

[0021] The target library table information is used as the key element of the second key-value pair, and the target write object is used as the value element of the second key-value pair to construct a second key-value pair;

[0022] The second key-value pair is stored in the write object mapping table.

[0023] Optionally, the write object mapping table is used to store the second key-value pair, and the key element and the value element of the second key-value pair correspond to the associated library table information and the write object in sequence; the method further includes the following steps:

[0024] Based on the target library table information, traverse and match the key elements of all second key-value pairs written into the object mapping table;

[0025] When a key element identical to the target library table information is matched, the write object corresponding to the value element associated with the second key-value pair where the key element is located is used as the target write object, and the execution is jumped to the step of calling the second function class according to the target write object to generate the target resource allocation object.

[0026] Optionally, storing the target resource allocation object in a resource allocation mapping table includes the following steps:

[0027] Using the target resource allocation identifier as the key element of the third key-value pair, and using the target resource allocation object as the value element of the third key-value pair, to construct a third key-value pair;

[0028] The third key-value pair is stored in the resource allocation mapping table.

[0029] On the other hand, an embodiment of the present invention provides a device for writing data into a multi-target table, comprising:

[0030] The first module is used to obtain data to be written;

[0031] The second module is used to call the first function class to generate a target resource allocation identifier of the data to be written according to the element characteristics of the data to be written; the target resource allocation identifier carries target library table information, and the target library table information carries the target table name of the target table;

[0032] The third module is used to perform traversal matching in a preset resource allocation mapping table based on the target resource allocation identifier to obtain a matching result;

[0033] The fourth module is used to obtain the structural metadata of the target table according to the target table name when the matching result is empty;

[0034] A fifth module is used to generate a target write object of a target table in response to a create instruction based on the structural metadata; and store the target write object in a preset write object mapping table;

[0035] The sixth module is used to call the second function class according to the target write object to generate a target resource allocation object; and store the target resource allocation object in a resource allocation mapping table;

[0036] The seventh module is used to call the target writing object based on the target resource allocation object to write the data to be written into the target table.

[0037] Optionally, when the matching result is not empty, the matching result contains a resource allocation object; the device further includes:

[0038] The eighth module is used to take the resource allocation object in the matching result as the target resource allocation object and jump to execute the seventh module.

[0039] Optionally, the write object mapping table is used to store the second key-value pair, and the key element and the value element of the second key-value pair correspond to the associated library table information and the write object in sequence; the device also includes:

[0040] A ninth module is used to traverse and match the key elements of all second key-value pairs written into the object mapping table based on the target library table information;

[0041] The tenth module is used to, when a key element identical to the target library table information is matched, take the write object corresponding to the value element associated with the second key-value pair where the key element is located as the target write object, and jump to execute the sixth module.

[0042] On the other hand, an embodiment of the present invention provides an electronic device, including: a processor and a memory; the memory is used to store programs; the processor executes the program to implement the above-mentioned multi-target table data writing method.

[0043] On the other hand, an embodiment of the present invention provides a computer storage medium, in which a program executable by a processor is stored. When the program executable by the processor is executed by the processor, it is used to implement the above-mentioned method for writing data into multiple target tables.

[0044] The embodiment of the present invention obtains the data to be written; calls the first function class according to the element characteristics of the data to be written to generate the target resource allocation identifier of the data to be written; the target resource allocation identifier carries the target library table information, and the target library table information carries the target table name of the target table; performs traversal matching in the preset resource allocation mapping table based on the target resource allocation identifier to obtain a matching result; when the matching result is empty, obtains the structural metadata of the target table according to the target table name; based on the structural metadata, generates the target write object of the target table in response to the creation instruction; stores the target write object in the preset write object mapping table; calls the second function class according to the target write object to generate the target resource allocation object; stores the target resource allocation object in the resource allocation mapping table; calls the target write object based on the target resource allocation object to write the data to be written into the target table. The embodiment of the present invention can carry the library table information and the target table name included in the target resource allocation identifier, and can establish the corresponding write object and specify the corresponding resource allocation object in combination with the subsequent steps to avoid the path writing logic locked by the fixed mapping relationship. The embodiment of the present invention can solve the problem that the native FileSink can only write a single path and it is difficult to write multiple paths at the same time, and realize the goal of processing multiple table data sources and writing to multiple target table storage paths by a single Flink task. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0046] Figure 1 A schematic diagram of an implementation environment for writing data into multiple target tables provided by an embodiment of the present invention;

[0047] Figure 2 A schematic flow chart of a method for writing data into a multi-target table provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the architecture of Flink and its core components provided by an embodiment of the present invention;

[0049] Figure 4 A schematic diagram of an example of target table writing of a result of writing data into multiple target tables provided by an embodiment of the present invention;

[0050] Figure 5 A schematic diagram of another target table writing example of a multi-target table data writing result provided by an embodiment of the present invention;

[0051] Figure 6 A schematic diagram of the process of writing single-table data of the native FileSink provided in an embodiment of the present invention;

[0052] Figure 7 A schematic diagram of an architecture for creating multiple sinks based on native FileSink to write multiple target table paths provided in an embodiment of the present invention;

[0053] Figure 8 A schematic diagram of the overall process principle of the method for writing data into a multi-target table provided by an embodiment of the present invention;

[0054] Fig. 9 A schematic diagram of an architecture for writing multiple target table paths based on the method of the present invention provided in an embodiment of the present invention;

[0055] Fig.10 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0057] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100", "second / S200", etc. in the specification, claims and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0058] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0059] It is understandable that the multi-target table data writing method provided in the embodiment of the present invention can be applied to any computer device with data processing and computing capabilities, and this computer device can be various terminals or servers. When the computer device in the embodiment is a server, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smart phone, a tablet computer, a laptop computer, a desktop computer, etc., but is not limited to this.

[0060] To facilitate the understanding of the technical solution of the present invention, the technical feature terms that may appear in the embodiments of the present invention are first explained:

[0061] Flink (Apache Flink): An open source stream processing and batch processing framework designed to process large-scale data with high throughput and low latency requirements. It provides distributed data stream processing and data batch processing capabilities with high scalability, fault tolerance and flexibility.

[0062] FileSink: A component in Flink responsible for writing data.

[0063] HDFS / hdfs (Hadoop Distributed File System): Part of the Apache Hadoop ecosystem, it is a distributed file system used to store and manage large-scale data sets.

[0064] Schema: The structural definition of a table in a database, including table name, column name, data type, constraints and other information. It is used to describe the logical and physical structure of a table and defines the fields and their properties in the table. Schema is an abstract collection of metadata, including a set of schema components: mainly declarations of elements and attributes, and definitions of complex and simple data types. These schema components are usually created when processing a batch of schema documents. The schema document contains the source language definition of the schema component.

[0065] Source: data source;

[0066] Transform: data conversion;

[0067] Sink: data receiving;

[0068] FileSink: As the specific implementation class of Sink, it is responsible for initializing the various components and environments for data writing, and serves as the entry and startup item for the data writing process;

[0069] BucketAssigner / bucketAssigner: responsible for generating bucketId and assigning data elements to different buckets. You can set the assignment rules yourself.

[0070] Bucket: A logical container for storing data. Flink allocates data to different buckets, which allocate data to different physical storage locations or storage units according to rules and policies.

[0071] BulkWriter / bulkWriter: Responsible for writing large amounts of data in batches to external storage systems, and can convert elements in the Flink data stream into the format required by the external storage system

[0072] BucketMap / bucketMap: responsible for caching Buckets. You can quickly get the corresponding Bucket through bucketId.

[0073] BucketWriterMap / bucketWriterMap: responsible for caching BulkWriter. The corresponding BulkWriter can be quickly obtained through the library table information.

[0074] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0075] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0076] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0077] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present invention.

[0078] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a method for writing data to a multi-target table. The following is an example of the multi-target table data writing method being applied in the terminal 102. It can be understood that the multi-target table data writing method can also be applied in the server 101.

[0079] Reference Figure 2 , Figure 2 A flowchart of a method for writing data to a multi-target table applied to a terminal (for example, the terminal may be a core component of Flink) provided in an embodiment of the present invention. The execution subject of the method for writing data to a multi-target table may be any of the aforementioned computer devices (including a server or a terminal). Figure 2 , the method comprises the following steps:

[0080] S100, obtaining data to be written;

[0081] First of all, it should be noted that Figure 3 As shown in the figure, in Flink, data processing tasks are usually divided into three core components: source (data source), transform (data conversion) and sink (data reception). The sink stage is responsible for writing the processed data to the target table hdfs path.

[0082] For example, in some specific implementations, data is acquired through the source component, and pre-processing of data conversion can be implemented through the transform component, and then subsequent related processing steps are executed through the sink.

[0083] S200, calling a first function class according to element characteristics of the data to be written to generate a target resource allocation identifier for the data to be written;

[0084] The target resource allocation identifier carries target library table information, and the target library table information carries the target table name of the target table;

[0085] Exemplarily, in some specific implementations, when FileSink receives a data element (i.e., data to be written), BucketAssigner (i.e., the first function class) generates a bucketId (i.e., a target resource allocation identifier) ​​with library table information. For example, the path dbA / tableA / dt=20240315 / hm=0100 includes the library name dbA, table name tableA, date 20240315, and time 0100, so that data from different tables will fall under different paths.

[0086] Some specific application scenario examples are as follows: the result directory generated after running the test task locally is as follows Figure 4 and Figure 5 As shown, the data is written into tableA and tableB paths respectively according to the upstream data, and partitioned by date and time. Specifically, the upstream data generally comes from the binlog data of the data source, which contains the data source library table information and time and date information. The library table time information (i.e., element characteristics) is retained during the process of being transmitted to FileSink through the message queue. BucketAssigner parses the library table and date and hour information in element and splices them into bucketId, for example, dbA / tableA / dt=20240315 / hm=0100.

[0087] S300, performing traversal matching in a preset resource allocation mapping table based on the target resource allocation identifier to obtain a matching result;

[0088] It should be noted that the resource allocation mapping table is used to store the first key-value pair, and the key element and value element of the first key-value pair correspond to the associated resource allocation identifier and resource allocation object in sequence; in some embodiments, step S300 may include the following steps: based on the target resource allocation identifier, traversing and matching the key elements of all first key-value pairs in the resource allocation mapping table; when the key element identical to the target resource allocation identifier is matched, the resource allocation object corresponding to the value element associated with the first key-value pair where the key element is located is taken as the matching result; when the key element identical to the target resource allocation identifier is not matched, the output matching result is empty.

[0089] Among them, in some embodiments, when the matching result is not empty, the matching result contains a resource allocation object; the method may also include the following steps: using the resource allocation object in the matching result as the target resource allocation object, and jumping to execute the step of calling the target write object based on the target resource allocation object to write the data to be written into the target table.

[0090] Exemplarily, in some specific implementations, bucketId will be first attempted to be retrieved from bucketMap. Specifically, based on the already generated bucketId (i.e., target resource allocation), it is determined whether the bucketId (i.e., target resource allocation) already exists in the bucketMap (i.e., resource allocation mapping table) with bucketId (i.e., resource allocation identifier) ​​as the key (i.e., key element) and Bucket object (i.e., resource allocation object) as the value (i.e., value element). If so, the corresponding bucket is retrieved.

[0091] S400, when the matching result is empty, obtaining the structural metadata of the target table according to the target table name;

[0092] It should be noted that, in some embodiments, obtaining the structural metadata of the target table according to the target table name may include the following steps: obtaining the structural metadata of the target table through an http request query according to the target table name.

[0093] Exemplarily, in some specific implementations, if the matching result is empty, the schema information (ie, structural metadata) of the target table is queried via an http request.

[0094] S500, generating a target write object of a target table in response to a create instruction based on the structural metadata; storing the target write object in a preset write object mapping table;

[0095] It should be noted that, in some embodiments, storing the target write object in a preset write object mapping table may include the following steps: using the target library table information as the key element of a second key-value pair, and, using the target write object as the value element of the second key-value pair, to construct a second key-value pair; storing the second key-value pair in the write object mapping table.

[0096] Among them, in some embodiments, the write object mapping table is used to store the second key-value pair, and the key element and value element of the second key-value pair correspond to the associated library table information and the write object in sequence; the method may also include the following steps: based on the target library table information, traversing and matching the key elements of all second key-value pairs in the write object mapping table; when a key element identical to the target library table information is matched, the write object corresponding to the value element associated with the second key-value pair where the key element is located is used as the target write object, and the execution is jumped to the step of calling the second function class according to the target write object to generate a target resource allocation object.

[0097] Exemplarily, in some specific implementations, the schema information (i.e., structural metadata) of the target table obtained through http request query can be used to construct a BulkWriter (i.e., the target write object) and put it into bucketWriterMap (i.e., the write object mapping table). In some preferred implementations, it is also possible to directly determine from the custom bucketWriterMap whether it contains the corresponding BulkWriter of the library table where the data element is located; then the BulkWriter can be used to generate a Bucket (i.e., the target resource allocation object) using BucketFactory (i.e., the second function class).

[0098] S600, calling the second function class according to the target write object to generate a target resource allocation object; storing the target resource allocation object in a resource allocation mapping table;

[0099] It should be noted that, in some embodiments, storing the target resource allocation object in the resource allocation mapping table may include the following steps: using the target resource allocation identifier as the key element of a third key-value pair, and using the target resource allocation object as the value element of the third key-value pair to construct a third key-value pair; storing the third key-value pair in the resource allocation mapping table.

[0100] For example, in some specific implementations, the BucketFactory (i.e., the second function class) can be used to generate a Bucket (i.e., the target resource allocation object) using the BulkWriter (i.e., the target writing object). At the same time, the created Bucket can also be associated with the bucketId generated in the previous step and put into bucketMap.

[0101] S700: Call a target writing object based on a target resource allocation object to write the data to be written into a target table.

[0102] For example, in some specific implementations, the BulkWriter of the target table is called by the bucket to complete the writing of data that meets the corresponding table field type.

[0103] In order to explain the principle of the technical solution of the present invention in detail, the overall process of the present invention is described below in combination with some specific embodiments. It is easy to understand that the following is an explanation of the technical principle of the present invention and cannot be regarded as a limitation of the present invention.

[0104] First of all, it should be noted that Figure 3 As shown in the figure, in Flink, data processing tasks are usually divided into three core components: source (data source), transform (data conversion) and sink (data reception). The sink stage is responsible for writing the processed data to the target table hdfs path.

[0105] Among them, Flink's native FileSink can write data to a single table, such as Figure 6 As shown, the processing flow is as follows:

[0106] During the construction of FileSink, the file writing base path of the target table, the BulkWriterFactory class containing the schema information of the target table, and the BucketAssigner class used to generate bucketId are passed in. These classes jointly determine the data format and data writing path of the target table. For example, the base path and bucketId are spliced ​​into a complete path for data storage, and the BulkWriter generated by BulkWriterFactory determines the schema information of the written data. When FileSink receives data such as element, BucketAssigner will calculate the bucketId to which it belongs based on the element, and then determine whether the bucketId already exists in the bucketMap with bucketId as the key and Bucket object as the value. If it exists, the bucket is taken out; if not, a Bucket is generated by BucketFactory and put into bucketMap. The generated or taken bucket calls the BulkWriter generated by BulkWriterFactory to actually execute the writing of element data, thus completing the process of writing element data to the single table storage path.

[0107] If you do not rewrite the source code to construct a custom FileSink, if you want to write to multiple table paths at the same time, you need to create multiple sinks, divert the data, and write to multiple target table paths according to different target tables, such as Figure 7 As shown, multiple sinks need to be built to divert the data. This has multiple problems. First, building multiple sinks will waste a lot of computing resources. Second, the scalability is low. If there are too many target tables, resources will eventually be exhausted. Therefore, the best strategy is to reconstruct the sink so that a single sink has the ability to write to multiple HDFS paths.

[0108] In order to achieve the goal of writing multi-table data into multi-table storage paths, the embodiment of the present invention reconstructs the internal process of FileSink by rewriting FileSink, BucketAssigner, BulkWriter and other classes, such as Figure 8 As shown, the new processing flow is as follows:

[0109] During the new construction of FileSink, the target table's file writing base path and BucketAssigner will be passed in, and BulkWriterFactory will no longer be passed in. When FileSink receives a data element, the new BucketAssigner will generate a bucketId with library table information, for example, the path dbA / tableA / dt=20240315 / hm=0100 contains the library name dbA, table name tableA, date 20240315, and time 0100, so that data from different tables will fall under different paths. Then it will also try to get the bucketId from the bucketMap. If there is no corresponding Bucket, it will determine whether the corresponding BulkWriter of the library table where the data element is located is contained in the custom bucketWriterMap. If not, it will query the schema information of the table through http requests and build a BulkWriter and put it into the bucketWriterMap. Then, the BulkWriter bound to the target table schema will be used to generate a Bucket, so that the bucket can call the BulkWriter of the target table to complete the data writing that meets the corresponding table field type.

[0110] After refactoring the FileSink related components and processes, Flink's processing flow becomes as follows: Fig. 9 As shown, you only need to build a sink to divert data inside the sink and write it to different HDFS paths, thus achieving maximum resource utilization and high scalability.

[0111] The result directory generated after running the test task locally is as follows Figure 4 and Figure 5 As shown, data is written into tableA and tableB paths respectively according to the upstream data, and partitioned by date and time.

[0112] Compared with the prior art, the present invention has at least the following beneficial effects:

[0113] 1. A multi-table path data writing method based on Flink is proposed. The existing technology can only write to a single table path with a single sink. To write to multiple table paths, multiple sinks or multiple tasks are required, which consumes a lot of computing resources. This solution realizes a single sink writing to any number of table paths, thus maximizing the utilization of computing resources.

[0114] 2. The existing technology needs to determine the target table write path when initializing the task. This solution reconstructs the FileSink component and process to dynamically generate the target table write path according to the upstream data content, thereby greatly improving the scalability of the task. The new target table path does not require code modification or task restart. The task can automatically adapt to the new target table write path address.

[0115] On the other hand, an embodiment of the present invention provides a device for writing data into a multi-target table, which may include:

[0116] The first module is used to obtain data to be written;

[0117] The second module is used to call the first function class to generate a target resource allocation identifier of the data to be written according to the element characteristics of the data to be written; the target resource allocation identifier carries target library table information, and the target library table information carries the target table name of the target table;

[0118] The third module is used to perform traversal matching in a preset resource allocation mapping table based on the target resource allocation identifier to obtain a matching result;

[0119] The fourth module is used to obtain the structural metadata of the target table according to the target table name when the matching result is empty;

[0120] A fifth module is used to generate a target write object of a target table in response to a create instruction based on the structural metadata; and store the target write object in a preset write object mapping table;

[0121] The sixth module is used to call the second function class according to the target write object to generate a target resource allocation object; and store the target resource allocation object in a resource allocation mapping table;

[0122] The seventh module is used to call the target writing object based on the target resource allocation object to write the data to be written into the target table.

[0123] In some embodiments, when the matching result is not empty, the matching result contains a resource allocation object; the apparatus may further include:

[0124] The eighth module is used to take the resource allocation object in the matching result as the target resource allocation object and jump to execute the seventh module.

[0125] In some embodiments, the write object mapping table is used to store the second key-value pair, and the key element and the value element of the second key-value pair correspond to the associated library table information and the write object in sequence; the device may also include:

[0126] A ninth module is used to traverse and match the key elements of all second key-value pairs written into the object mapping table based on the target library table information;

[0127] The tenth module is used to, when a key element identical to the target library table information is matched, take the write object corresponding to the value element associated with the second key-value pair where the key element is located as the target write object, and jump to execute the sixth module.

[0128] The contents of the method embodiments of the present invention are all applicable to the device embodiments. The functions specifically implemented by the device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0129] On the other hand, an embodiment of the present invention further provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above sensitive information method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0130] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0131] like Fig.10 As shown, Fig.10 A specific example of the hardware structure of an electronic device according to an embodiment is shown, and the electronic device includes:

[0132] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0133] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the network node population optimization method of the embodiment of the present invention;

[0134] Input / output interface 1003, used to implement information input and output;

[0135] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0136] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0137] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0138] The electronic device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0139] The contents of the method embodiments of the present invention are all applicable to the electronic device embodiments. The functions specifically implemented by the electronic device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] Another aspect of an embodiment of the present invention further provides a computer-readable storage medium, wherein the storage medium stores a program, and the program is executed by a processor to implement the above method.

[0141] It should be noted that the computer-readable medium shown in the embodiment of the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read to Only Memory, CD to ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0142] The contents of the method embodiments of the present invention are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0143] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0144] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] It should be noted that, although several modules of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to an embodiment of the present invention, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0146] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present invention can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD to ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation of the present invention.

[0147] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0148] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0149] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0150] The logic and / or steps represented in the flowchart or otherwise described herein, for example, may be considered as an ordered list of executable instructions for implementing logical functions, and may be embodied in any computer-readable medium for use by an instruction execution device, apparatus or device (such as a computer-based device, a device including a processor, or other device that can fetch instructions from an instruction execution device, apparatus or device and execute the instructions), or in conjunction with such instruction execution device, apparatus or device. For purposes of this specification, "computer-readable medium" may be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution device, apparatus or device, or in conjunction with such instruction execution device, apparatus or device.

[0151] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0152] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution device. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0153] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0154] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0155] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for writing data into a multi-target table, characterized in that: The following steps are involved: Get the data to be written; Calling a first function class according to the element characteristics of the data to be written to generate a target resource allocation identifier for the data to be written; The target resource allocation identifier carries target library table information, and the target library table information carries the target table name of the target table; Performing traversal matching in a preset resource allocation mapping table based on the target resource allocation identifier to obtain a matching result; When the matching result is empty, obtaining the structural metadata of the target table according to the target table name; generating a target write object of the target table in response to a create instruction based on the structural metadata; Storing the target write object in a preset write object mapping table; Calling a second function class according to the target write object generates a target resource allocation object; Storing the target resource allocation object in the resource allocation mapping table; The target writing object is called based on the target resource allocation object to write the to-be-written data into the target table.

2. The method for writing data into a multi-target table according to claim 1, characterized in that: The resource allocation mapping table is used to store a first key-value pair, wherein the key element and the value element of the first key-value pair correspond to the associated resource allocation identifier and the resource allocation object in sequence; performing traversal matching in the preset resource allocation mapping table based on the target resource allocation identifier to obtain a matching result includes the following steps: Based on the target resource allocation identifier, traverse and match the key elements of all the first key-value pairs in the resource allocation mapping table; When the key element identical to the target resource allocation identifier is matched, the resource allocation object corresponding to the value element associated with the first key-value pair where the key element is located is taken as the matching result; When the key element identical to the target resource allocation identifier is not matched, the matching result is output as empty.

3. The method for writing data into a multi-target table according to claim 1, characterized in that: When the matching result is not empty, the matching result contains a resource allocation object; the method further comprises the following steps: The resource allocation object in the matching result is used as the target resource allocation object, and the step of calling the target write object based on the target resource allocation object to write the to-be-written data into the target table is executed.

4. The method for writing data into a multi-target table according to claim 1, characterized in that: The step of obtaining the structural metadata of the target table according to the target table name comprises the following steps: According to the target table name, the structure metadata of the target table is obtained through an http request query.

5. The method for writing data into a multi-target table according to claim 1, characterized in that: The step of storing the target write object in a preset write object mapping table comprises the following steps: The target library table information is used as the key element of a second key-value pair, and the target write object is used as the value element of the second key-value pair to construct the second key-value pair; The second key-value pair is stored in the write object mapping table.

6. The method for writing data into a multi-target table according to claim 1, characterized in that: The write object mapping table is used to store the second key-value pair, and the key element and the value element of the second key-value pair correspond to the associated library table information and the write object in sequence; the method further includes the following steps: Based on the target library table information, traverse and match the key elements of all the second key-value pairs in the write object mapping table; When the key element that is identical to the target library table information is matched, the write object corresponding to the value element associated with the second key-value pair where the key element is located is used as the target write object, and the step of calling the second function class according to the target write object to generate the target resource allocation object is jumped to execution.

7. The method for writing data into a multi-target table according to claim 1, characterized in that: The step of storing the target resource allocation object in the resource allocation mapping table comprises the following steps: The target resource allocation identifier is used as a key element of a third key-value pair, and the target resource allocation object is used as a value element of the third key-value pair to construct the third key-value pair; The third key-value pair is stored in the resource allocation mapping table.

8. A device for writing data into a multi-target table, characterized in that: include: The first module is used to obtain data to be written; The second module is used to call the first function class to generate a target resource allocation identifier of the data to be written according to the element characteristics of the data to be written; The target resource allocation identifier carries target library table information, and the target library table information carries the target table name of the target table; A third module is used to perform traversal matching in a preset resource allocation mapping table based on the target resource allocation identifier to obtain a matching result; The fourth module is used for obtaining the structural metadata of the target table according to the target table name when the matching result is empty; A fifth module, configured to generate a target write object of the target table in response to a create instruction based on the structure metadata; Storing the target write object in a preset write object mapping table; A sixth module is used to call the second function class to generate a target resource allocation object according to the target write object; Storing the target resource allocation object in the resource allocation mapping table; The seventh module is used to call the target writing object based on the target resource allocation object to write the data to be written into the target table.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.