Data processing method and device, equipment and storage medium
By acquiring configuration files and database mapping rules, the system automatically queries and creates target data tables, solving the real-time and reliability issues in multi-source heterogeneous database data consumption, and achieving efficient and stable data synchronization and system scalability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-12
- Publication Date
- 2026-03-10
AI Technical Summary
Existing data processing solutions suffer from poor real-time performance and low data reliability when consuming data from multi-source heterogeneous databases. In particular, changes in the source table structure can easily lead to data inconsistency, and the synchronization strategy cannot be dynamically adjusted to balance processing efficiency and flexibility.
By obtaining the theme information, target database information, and database mapping rules from the configuration file, the system automatically queries whether the target data table exists and creates the target data table based on the table structure information if it does not exist. This achieves seamless connection of the data synchronization process, decouples the data source and destination, and improves the system's adaptability and scalability.
It achieves efficient synchronization and reliability in the data consumption process, reduces system operation and maintenance complexity and manpower costs, and ensures the stability and flexibility of data synchronization.
Smart Images
Figure CN121636595A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a data processing method, apparatus, device, and storage medium. Background Technology
[0002] With the increasing demand for big data and real-time data processing, enterprises typically need to access data in real time from multiple heterogeneous data sources (such as MySQL, OceanBase, MongoDB, etc.) and ultimately write the data to the target database. In this process, enterprises often use the distributed messaging system Kafka as an intermediate layer for data access to process data from multiple heterogeneous data sources. Currently, mainstream data stream processing solutions mainly include native Kafka Consumer solutions, KafkaConnect standard connectors, stream processing framework relay solutions, and custom script batch processing solutions.
[0003] However, existing technologies still have significant shortcomings when consuming data from multi-source heterogeneous databases. For example, native Kafka Consumer solutions require pre-defined data serialization formats, making it difficult to automatically identify mixed format scenarios. Furthermore, writing to different databases requires developing separate client logic, resulting in high maintenance costs. The Kafka Connect standard connector is complex to configure in heterogeneous database scenarios, has limited conversion capabilities, and lacks dynamic scalability. Stream processing framework relay solutions introduce additional processing latency and increase system complexity. Custom script batch processing solutions suffer from poor real-time performance and low reliability, failing to meet high throughput requirements.
[0004] In particular, existing data stream processing solutions lack dynamic perception and synchronization mechanisms for changes in table structure, which can easily lead to data inconsistency when the source table structure changes; and they cannot dynamically adjust the synchronization strategy based on the integrity of message fields, making it difficult to balance processing efficiency and flexibility.
[0005] Therefore, how to quickly, flexibly, and stably consume various heterogeneous data into the target database has become an urgent problem for relevant technical personnel in this field. Summary of the Invention
[0006] This application provides a data processing method to solve the problems of poor real-time performance and low data reliability in existing data processing schemes when consuming data from multi-source heterogeneous databases.
[0007] This application also provides a data processing apparatus to solve the problems of poor real-time performance and low data reliability in existing data processing schemes when consuming data from multi-source heterogeneous databases.
[0008] This application also provides a data processing device to solve the problems of poor real-time performance and low data reliability in existing data processing solutions when consuming data from multi-source heterogeneous databases.
[0009] This application also provides a computer-readable storage medium to address the problems of poor real-time performance and low data reliability in existing data processing schemes when consuming data from multi-source heterogeneous databases.
[0010] The embodiments of this application adopt the following technical solutions: A data processing method includes: obtaining a configuration file, the configuration file including at least topic information of the Kafka to be consumed, target database information, table creation configuration data, and database mapping rules; obtaining consumption messages from the Kafka corresponding to the topic information through a data consumption program based on the topic information; determining the target database and target data table according to the consumption messages and the database mapping rules; querying the target database to see if the target data table exists; if the result is negative, determining the table structure information of the target data table, creating the target data table in the target database based on the table structure information, and synchronizing the data to be consumed to the target data table; if the result is positive, synchronizing the data to be consumed to the target data table.
[0011] A data processing apparatus includes: a configuration file acquisition unit for acquiring a configuration file, the configuration file including at least topic information of the Kafka to be consumed, target database information, table creation configuration data, and database mapping rules; a message acquisition unit for acquiring consumption messages from the Kafka corresponding to the topic information through a data consumption program based on the topic information; a target database determination unit for determining the target database and target data table based on the consumption messages and the database mapping rules; a query unit for querying whether the target data table exists in the target database; a table creation unit for determining the table structure information of the target data table when the query unit determines the result is negative, creating the target data table in the target database based on the table structure information, and synchronizing the data to be consumed to the target data table; and a synchronization unit for synchronizing the data to be consumed to the target data table when the query unit determines the result is positive.
[0012] A data processing device, comprising: The processor; and a memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the following operations: obtain a configuration file, the configuration file including at least topic information of the Kafka to be consumed, target database information, table creation configuration data, and database mapping rules; obtain consumption messages from the Kafka corresponding to the topic information through a data consumption program based on the topic information; determine the target database and target data table according to the consumption messages and the database mapping rules; query the target database to see if the target data table exists; if the result is negative, determine the table structure information of the target data table, create the target data table in the target database based on the table structure information, and synchronize the data to be consumed to the target data table; if the result is positive, synchronize the data to be consumed to the target data table.
[0013] A computer-readable storage medium stores one or more programs that, when executed by an electronic device including multiple applications, cause the electronic device to perform the following operations: obtain a configuration file, the configuration file including at least topic information of the Kafka to be consumed, target database information, table creation configuration data, and database mapping rules; obtain consumption messages from the Kafka corresponding to the topic information through a data consumption program based on the topic information; determine the target database and target data table according to the consumption messages and the database mapping rules; query the target database to see if the target data table exists; if the result is negative, determine the table structure information of the target data table, create the target data table in the target database based on the table structure information, and synchronize the data to be consumed to the target data table; if the result is positive, synchronize the data to be consumed to the target data table.
[0014] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects: Using the data processing method provided in this application embodiment, when data processing from multiple heterogeneous data sources is required, a configuration file can first be obtained. This configuration file includes at least the topic information of the Kafka to be consumed, target database information, table creation configuration data, and database mapping rules. Then, based on the topic information in the configuration file, a data consumption program can obtain consumption messages from the Kafka corresponding to the topic information. Based on the obtained consumption messages, the target database and target data table are determined according to the database mapping rules. The existence of the target data table is checked in the target database. If the result is negative, the table structure information of the target data table is determined, the target data table is created in the target database according to the table structure information, and the data to be consumed is synchronized to the target data table. If the result is positive, the data to be consumed is synchronized to the target data table. The method provided in this application embodiment achieves two advantages. First, by using pre-configured database mapping rules in the configuration file, a flexible mapping between the logical database and the physical target database in the consumption message can be realized, decoupling the data source and data destination. Thus, during data consumption, the data consumption program can route data from different sources to the specified target database, greatly enhancing the adaptability and scalability of the system architecture. Second, by automatically querying and determining whether the target data table exists, and automatically creating the table based on the obtained table structure information when the table does not exist, a "seamless connection" of the data synchronization process is achieved, avoiding data synchronization failure due to missing target tables, reducing manual table creation procedures, and effectively reducing the system's operational complexity and labor costs. Attached Figure Description
[0015] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic flowchart illustrating a data processing method provided in an embodiment of this application. Figure 2 This is a schematic diagram of the specific structure of a data processing device provided in an embodiment of this application; Figure 3 This is a schematic diagram of the specific structure of a data processing device provided in an embodiment of this application. Detailed Implementation
[0016] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0017] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.
[0018] This application provides a data processing method to address the problems of poor real-time performance and low data reliability in existing data processing solutions when consuming data from multi-source heterogeneous databases.
[0019] The execution subject of the data processing method provided in this application embodiment may be, but is not limited to, at least one of a data consumption program, a data processing middleware, a data processing system, and a data consumption platform.
[0020] For ease of description, the following description uses a data processing system and a unified data consumption program running on that system as examples to illustrate the implementation of this method. It should be understood that the use of a data processing system and a unified data consumption program as the executing entities is merely illustrative and should not be construed as a limitation of the method.
[0021] The schematic diagram of the specific implementation flow of the data processing method provided in this application is shown below. Figure 1 As shown, the main steps include the following: Step 11, obtain the configuration file; The obtained configuration file includes at least the topic information of the Kafka to be consumed, the target database information, the table creation configuration data, and the database mapping rules.
[0022] When the data processing system starts up, it can first load and parse a pre-configured configuration file, which may contain, but is not limited to, the following types of key information: 1. Kafka cluster information: including the names of one or more Kafka topics to be consumed, the address of the Kafka cluster, the consumer group ID, etc.
[0023] 2. Target database information: This includes the target database's Java Database Connectivity (JDBC) connection address (Uniform Resource Locator, URL), username, password, and other authentication information.
[0024] 3. Table creation and synchronization configuration: This controls whether to enable the automatic table creation function and the source of table structure information (such as a remote metadata service interface).
[0025] 4. Database mapping rules: Define the mapping relationship between the source database name and the target database name. For example, map the source database name db_source to the target database name db_target.
[0026] 5. Filtering rules: In this embodiment of the application, the filtering rules can be blacklists and whitelists. For example, a blacklist can be configured to add certain databases or data tables that do not want to be processed to the blacklist. Then, when processing data, the unified data consumption program can filter the received consumption messages according to the blacklist, and the messages that hit the blacklist will be directly filtered out.
[0027] Step 12: Based on the topic information in the configuration file obtained by executing Step 11, the data consumption program retrieves consumption messages from the Kafka corresponding to the topic information. In this embodiment of the application, the unified data consumer program can connect to the Kafka cluster as a data consumer based on the Kafka topic information in the configuration file, and start continuously pulling messages under the specified Kafka topic, that is, obtaining the corresponding consumption messages.
[0028] It should be noted that, in this embodiment of the application, the unified data consumption program is configured to consume message data from multiple different databases simultaneously, and based on the parsed database name and data table name, to input the data into the corresponding target database in parallel, thereby greatly improving the timeliness of data synchronization.
[0029] Step 13: Based on the consumption message obtained by executing step 12, determine the target database and target data table according to the database mapping rules; It's important to note that once the unified data consumption program receives the raw consumption packets, it can filter them according to the filtering rules (e.g., blacklists and whitelists) in the configuration file obtained in step 11. For example, the unified data consumption program can parse the source database name and table name carried in the raw consumption packet. If the parsed source database name and table name are in the blacklist, the raw consumption packet will be discarded and no further processing will be performed. This filters out irrelevant or unnecessary data, reducing system load.
[0030] In this embodiment of the application, the unified data consumption program can filter consumption messages in the following manner: filtering the consumption messages according to the filtering rules in the configuration file to obtain consumption messages to be processed; and determining the target database and target data table according to the database mapping rules based on the consumption messages to be processed.
[0031] For filtered consumer messages, the unified data consumer program can determine the target database corresponding to the consumed data by performing the following sub-steps: Sub-step 1301: Obtain the source database name and source table name based on the consumption message to be processed; Specifically, the unified data consumption program can parse the data record to be processed from specific fields or metadata of the data to be processed, and extract the original source database name and source table name.
[0032] Sub-step 1302: Based on the database mapping rules, determine the target database and target table corresponding to the source database name; Specifically, the unified data consumption program can map the parsed source database name to the final target database name according to the preset database mapping rules in the configuration file. For example, if the configuration rule is "test01" -> "prod_warehouse", then all packets with the source database "test01" will have their target database determined to be "prod_warehouse". The target table name is the same as the source table name by default, but it can also be mapped using similar rules.
[0033] Step 14: Query the target database to see if the target data table exists; In this embodiment of the application, the unified data consumption program can connect to the target database determined by executing step 13, and check whether the target data table exists in the target database by executing an SQL query statement. If the result is negative, step 15 is executed; if the result is positive, step 16 is executed.
[0034] Step 15: When it is determined through step 14 that the target data table does not exist in the target database, determine the table structure information of the target data table, create the target data table in the target database according to the table structure information, and synchronize the data to be consumed to the target data table. In this embodiment, the unified data consumption program can send a request to a pre-configured remote metadata service interface, which is typically provided by the source data governance platform. The request parameters are the source database name and the source table name. The interface returns a JSON-formatted table structure information describing the complete structure of the source table (including field names, data types, primary keys, indexes, etc.).
[0035] The unified data consumption program can dynamically generate the corresponding SQLCREATE TABLE statement by parsing this JSON-formatted table structure information. Subsequently, the unified data consumption program executes this statement in the target database, thereby creating a target data table with the same structure as the source table, and synchronizing the data to be consumed to this newly created target data table. For details on the specific data synchronization method, please refer to the relevant description in step 16, which will not be repeated here.
[0036] Step 16: When it is determined through step 14 that the target data table exists in the target database, the data to be consumed is synchronized to the target data table.
[0037] When the target data table is confirmed to exist in the target database, to ensure the data synchronization process continues stably even if the source table structure changes, and to avoid data write failures due to table structure mismatches, in this embodiment, the unified data consumer program can also obtain the latest structure information of the current source table from the remote interface and compare it with the original target data table in the target database. If a difference is found (e.g., a new field has been added to the source table), the unified data consumer program can incrementally synchronize the table structure of the original target data table in the target database to ensure structural consistency between the two ends.
[0038] After unifying the data table structure, the unified data consumption program can perform data synchronization according to the following sub-steps: Sub-step 1601: Determine whether the consumer message to be processed contains full field information; Specifically, the unified data consumption program can check the consumption message to be processed. If it contains the values of all fields in the target data table, it is determined that the consumption message to be processed contains full field information; if it only contains some of the changed field information (for example, in a database update operation, usually only the modified field and its primary key are transmitted), it is determined that the consumption message to be processed only contains changed field information.
[0039] Sub-step 1602: When it is determined that the consumption message to be processed contains full field information, the data to be consumed is synchronized to the target data table using transaction and batch processing methods. For pending consumption messages containing full-field information, the unified data consumption program can use a "transaction + batch" approach for data synchronization.
[0040] Specifically, in this embodiment, the unified data consumption program can record the data to be consumed corresponding to multiple pending consumption messages in memory, accumulating them into a batch. Then, within a database transaction, the entire batch is committed to the target database at once. This approach significantly reduces the number of network interactions with the database, fully utilizes the database's batch processing capabilities, and thus achieves extremely high data synchronization throughput. Sub-step 1603: When it is determined that the consumption message to be processed only contains changed field information, the changed fields are synchronized to the target data table by dynamically constructing a structured query SQL statement.
[0041] For pending consumption messages that only contain change field information, the unified data consumer program can process them by dynamically constructing SQL using JDBC.
[0042] Specifically, in this embodiment, the unified data consumption program parses the field names and corresponding values in the consumption message to be processed, and then dynamically generates an "UPDATE" statement. In this way, only the fields that have changed are updated, avoiding full field overwrite, making the operation more accurate, reducing unnecessary I / O pressure on the database, and ensuring the correctness of the data when processing update operations.
[0043] Using the data processing method provided in this application embodiment, when data processing from multiple heterogeneous data sources is required, a configuration file can first be obtained. This configuration file includes at least the topic information of the Kafka to be consumed, target database information, table creation configuration data, and database mapping rules. Then, based on the topic information in the configuration file, a data consumption program can obtain consumption messages from the Kafka corresponding to the topic information. Based on the obtained consumption messages, the target database and target data table are determined according to the database mapping rules. The existence of the target data table is checked in the target database. If the result is negative, the table structure information of the target data table is determined, the target data table is created in the target database according to the table structure information, and the data to be consumed is synchronized to the target data table. If the result is positive, the data to be consumed is synchronized to the target data table. The method provided in this application embodiment achieves two advantages. First, by using pre-configured database mapping rules in the configuration file, a flexible mapping between the logical database and the physical target database in the consumption message can be realized, decoupling the data source and data destination. Thus, during data consumption, the data consumption program can route data from different sources to the specified target database, greatly enhancing the adaptability and scalability of the system architecture. Second, by automatically querying and determining whether the target data table exists, and automatically creating the table based on the obtained table structure information when the table does not exist, a "seamless connection" of the data synchronization process is achieved, avoiding data synchronization failure due to missing target tables, reducing manual table creation procedures, and effectively reducing the system's operational complexity and labor costs.
[0044] In one embodiment, this application also provides a data processing apparatus to address the problems of poor real-time performance and low data reliability in existing data processing solutions when consuming data from multi-source heterogeneous databases. A schematic diagram of the specific structure of this data processing apparatus is shown below. Figure 2 As shown, it includes: configuration file acquisition unit 21, message acquisition unit 22, target database determination unit 23, query unit 24, table creation unit 25, and synchronization unit 26.
[0045] The configuration file acquisition unit 21 is used to acquire the configuration file, which includes at least the topic information of the Kafka to be consumed, the target database information, the table creation configuration data, and the database mapping rules. The message acquisition unit 22 is used to acquire consumption messages from the Kafka corresponding to the topic information through a data consumption program based on the topic information. The target database determination unit 23 is used to determine the target database and the target data table according to the consumption message and the database mapping rules. Query unit 24 is used to query the target database to determine if the target data table exists. The table creation unit 25 is used to determine the table structure information of the target data table when the query unit 24 determines that the result is negative, create the target data table in the target database according to the table structure information, and synchronize the data to be consumed to the target data table. Synchronization unit 26 is used to synchronize the data to be consumed to the target data table when the query unit 24 determines that the result is yes.
[0046] In one implementation, the target database determination unit 23 is specifically used for: filtering the consumption messages according to the filtering rules in the configuration file to obtain consumption messages to be processed; and determining the target database and target data table according to the consumption messages to be processed and the database mapping rules.
[0047] In one implementation, the target database determination unit 23 is specifically used for: obtaining the source database name and source table name according to the consumer message to be processed; and determining the target database and target table corresponding to the source database name according to the database mapping rules.
[0048] In one implementation, the table creation unit 25 is specifically used to: determine the table structure information corresponding to the source table name as the table structure information of the target data table; and create a target data table in the target database according to the table structure information.
[0049] In one implementation, the synchronization unit 26 is specifically used to: determine whether the consumption message to be processed contains full field information; if it is determined that the consumption message to be processed contains full field information, then use transaction and batch processing to synchronize the data to be consumed to the target data table; if it is determined that the consumption message to be processed only contains changed field information, then use dynamic construction of structured query SQL statements to synchronize the changed fields to the target data table.
[0050] In one implementation, the data consumption program is configured to consume message data from multiple different databases simultaneously, and to input the data into the corresponding target database in parallel based on the parsed database name and data table name.
[0051] Using the data processing apparatus provided in this application embodiment, when data processing from multiple heterogeneous data sources is required, a configuration file can first be obtained. This configuration file includes at least the topic information of the Kafka to be consumed, target database information, table creation configuration data, and database mapping rules. Then, based on the topic information in the configuration file, a data consumption program can obtain consumption messages from the Kafka corresponding to the topic information. Based on the obtained consumption messages, the target database and target data table are determined according to the database mapping rules. The existence of the target data table is checked in the target database. If the determination result is negative, the table structure information of the target data table is determined, the target data table is created in the target database according to the table structure information, and the data to be consumed is synchronized to the target data table. If the determination result is positive, the data to be consumed is synchronized to the target data table. The method provided in this application embodiment achieves two advantages. First, by using pre-configured database mapping rules in the configuration file, a flexible mapping between the logical database and the physical target database in the consumption message can be realized, decoupling the data source and data destination. Thus, during data consumption, the data consumption program can route data from different sources to the specified target database, greatly enhancing the adaptability and scalability of the system architecture. Second, by automatically querying and determining whether the target data table exists, and automatically creating the table based on the obtained table structure information when the table does not exist, a "seamless connection" of the data synchronization process is achieved, avoiding data synchronization failure due to missing target tables, reducing manual table creation procedures, and effectively reducing the system's operational complexity and labor costs.
[0052] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.
[0053] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.
[0054] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.
[0055] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a data processing device at the logical level. The processor executes the program stored in memory and specifically performs the following operations: Obtain the configuration file, which includes at least the topic information of the Kafka to be consumed, the target database information, table creation configuration data, and database mapping rules; based on the topic information, obtain consumption messages from the Kafka corresponding to the topic information through the data consumption program; based on the consumption messages and according to the database mapping rules, determine the target database and the target data table; query the target database to see if the target data table exists; if the result is negative, determine the table structure information of the target data table, create the target data table in the target database according to the table structure information, and synchronize the data to be consumed to the target data table; if the result is positive, synchronize the data to be consumed to the target data table.
[0056] The above is as stated in this application. Figure 3The methods executed by the data processing electronic device disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.
[0057] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.
[0058] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figure 1 The method of the illustrated embodiment is specifically used to perform the following operations: Obtain the configuration file, which includes at least the topic information of the Kafka to be consumed, the target database information, table creation configuration data, and database mapping rules; based on the topic information, obtain consumption messages from the Kafka corresponding to the topic information through the data consumption program; based on the consumption messages and according to the database mapping rules, determine the target database and the target data table; query the target database to see if the target data table exists; if the result is negative, determine the table structure information of the target data table, create the target data table in the target database according to the table structure information, and synchronize the data to be consumed to the target data table; if the result is positive, synchronize the data to be consumed to the target data table.
[0059] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0060] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0061] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0062] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0063] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0064] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0065] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0066] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0067] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0068] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A data processing method, characterized by, The method comprises the following steps: obtaining a configuration file, wherein the configuration file comprises at least topic information of a Kafka to be consumed, target database information, table creation configuration data and database mapping rules; acquiring consumption messages from the Kafka corresponding to the topic information through a data consumption program according to the topic information in the configuration file; determining a target database and a target data table according to the database mapping rules and the consumption messages; querying whether the target data table exists in the target database; when the result of the query is negative, determining table structure information of the target data table, creating the target data table in the target database according to the table structure information, and synchronizing the data to be consumed to the target data table; when the result of the query is positive, synchronizing the data to be consumed to the target data table.
2. The method of claim 1, wherein, The method further comprises the following steps: filtering the consumption messages according to filtering rules in the configuration file to obtain consumption messages to be processed; determining the target database and the target data table according to the database mapping rules and the consumption messages to be processed.
3. The method of claim 2, wherein, The method further comprises the following steps: acquiring a source database name and a source table name according to the consumption messages to be processed; determining the target database and the target table corresponding to the source database name according to the database mapping rules.
4. The method of claim 3, wherein, The method further comprises the following steps: determining the table structure information of the target data table as the table structure information of the target data table according to the source table name; creating the target data table in the target database according to the table structure information.
5. The method of claim 4, wherein, The method further comprises the following steps: determining whether the consumption messages to be processed contain full field information; when the consumption messages to be processed contain full field information, synchronizing the data to be consumed to the target data table by using a transaction and batch processing mode; when the consumption messages to be processed contain only change field information, synchronizing the change field to the target data table by dynamically constructing a structured query language (SQL) statement.
6. The method of claim 1, wherein, The data consumption program is configured to simultaneously consume message data of multiple different databases, and parallelly store data into corresponding target databases based on parsed database names and data table names.
7. A data processing apparatus, characterized by, The method comprises the following steps: a configuration file obtaining unit is configured to obtain a configuration file, wherein the configuration file comprises at least topic information of a Kafka to be consumed, target database information, table creation configuration data and database mapping rules; a message acquiring unit is configured to acquire consumption messages from the Kafka corresponding to the topic information through a data consumption program according to the topic information in the configuration file; a target database determining unit is configured to determine a target database and a target data table according to the database mapping rules and the consumption messages; The query unit is configured to query whether the target data table exists in the target database. The table creation unit is configured to, when the query unit determines that the target data table does not exist, determine table structure information of the target data table, create a target data table in the target database according to the table structure information, and synchronize the to-be-consumed data to the target data table. The synchronization unit is configured to, when the query unit determines that the target data table exists, synchronize the to-be-consumed data to the target data table.
8. The apparatus of claim 7, wherein, The target database determination unit is specifically configured to: filter the consumption message according to a filtering rule in the configuration file to obtain a to-be-processed consumption message; determine a target database and a target data table according to the to-be-processed consumption message and according to the database mapping rule.
9. A data processing device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the method of any one of claims 1 to 6.
10. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device can perform the method of any one of claims 1 to 6.