Data processing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to the technical field of computers, and discloses a data processing method and device, computer equipment, a storage medium and a program product. The method comprises the steps of obtaining target event information corresponding to a target database; wherein the target event information comprises metadata configuration information and metadata coordinate information; determining target metadata matched with the metadata coordinate information from a target database; acquiring incremental metadata information of the target metadata from a target database according to the metadata configuration information; and synchronizing the incremental metadata information to a business database.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing method, apparatus, computer equipment, storage medium and program product. Background Art

[0002] In metadata management systems, full data collection tasks are usually performed periodically or on demand to update the information in the metadata warehouse to ensure that it is consistent with the actual status of the database. That is, all objects in the entire database or a specified range are traversed to fully collect their definitions and status information, and then the database is updated based on the collected definitions and status information. However, the full data collection process requires access to a large number of database objects, which makes the collection time-consuming and affects system performance. Summary of the invention

[0003] In view of this, the present invention provides a data processing method, apparatus, computer equipment, storage medium and program product.

[0004] In a first aspect, the present invention provides a data processing method, comprising: obtaining target event information corresponding to a target database; wherein the target event information comprises: metadata configuration information and metadata coordinate information; determining target metadata matching the metadata coordinate information from the target database; according to the metadata configuration information, collecting incremental metadata information of the target metadata from the target database; and synchronizing the incremental metadata information to a business database.

[0005] The data processing method provided in this embodiment, when acquiring the target event information corresponding to the target database, accurately locates the target metadata that needs to be collected through the metadata coordinate information, and then only collects the incremental metadata information of the target metadata through the metadata configuration information, thereby avoiding a large number of unnecessary database accesses during the full collection process, thereby improving the collection efficiency.

[0006] In one possible implementation, synchronizing the incremental metadata information to a business database includes: detecting, by database name and table name, whether target metadata corresponding to the incremental metadata information exists in the business database corresponding to the target database; wherein the quantity and type of initial metadata of the business database and the initial metadata of the target database are the same; if the target metadata corresponding to the incremental metadata information exists in the business database, updating the target metadata in the business database according to the incremental metadata information; if the target metadata corresponding to the incremental metadata information does not exist in the business database, adding the incremental metadata information to the business database.

[0007] The data processing method provided in this embodiment detects whether the target metadata corresponding to the incremental metadata information exists in the business database corresponding to the target database through the database name and table name, and performs corresponding operations (updates or adds metadata) according to the detection results, thereby avoiding a comprehensive scan of the entire database or large-scale data migration, and improving the efficiency of data processing.

[0008] In a possible implementation, obtaining target event information corresponding to a target database includes: detecting whether the number of idle connections in a database connection pool of the target database satisfies the number of idle connections; if the number of idle connections in the database connection pool of the target database satisfies the number of idle connections, executing the step of obtaining target event information corresponding to the target database.

[0009] The data processing method provided in this embodiment, if the number of idle connections in the database connection pool of the target database meets the number of idle connections, the load of the target database is low, that is, metadata is collected when the load of the target database is low, thereby reducing the impact on the target database.

[0010] In a possible implementation, the method further includes: displaying the updated target metadata in the business database on a server page.

[0011] The data processing method provided in this embodiment increases the transparency of the system by displaying the target metadata on the page, so that the user can clearly understand the status and content of the business database, thereby enhancing the credibility of the system and the trust of the user.

[0012] In a possible implementation, synchronizing the incremental metadata information to the business database includes: synchronizing the incremental metadata information to the business database through a message queue.

[0013] The data processing method provided in this embodiment uses the message queue as an intermediate layer to achieve decoupling between data producers and consumers. Data producers only need to put incremental metadata information into the queue without worrying about when consumers process it. This asynchronous processing method improves the flexibility and response speed of the system.

[0014] In one possible implementation, when there are multiple pieces of incremental metadata information, the incremental metadata information is synchronized to the business database through a message queue, including: putting the incremental metadata information into the message queue in a preset order; when a scheduled task is triggered, synchronizing the incremental metadata information in the message queue to the business database in sequence in the preset order.

[0015] The data processing method provided in this embodiment can ensure that the data remains orderly when synchronized to the business database by putting the incremental metadata information into the message queue in a preset order, avoiding potential problems caused by data disorder. The scheduled task trigger synchronization mechanism can adjust the synchronization frequency according to actual needs, avoiding resource consumption caused by unnecessary real-time synchronization. At the same time, the message queue can buffer a large amount of data, balance the speed of data production and consumption, and optimize resource utilization.

[0016] In a second aspect, the present invention provides a data processing device, which includes: a monitoring module, used to obtain target event information corresponding to a target database; wherein the target event information includes: metadata configuration information and metadata coordinate information; a parsing module, used to determine target metadata matching the metadata coordinate information from the target database; a collection module, used to collect incremental metadata information of the target metadata from the target database according to the metadata configuration information; and a synchronization module, used to synchronize the incremental metadata information to a business database.

[0017] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the data processing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0018] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the data processing method of the first aspect or any corresponding embodiment thereof.

[0019] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the data processing method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 is a structural block diagram of a data processing system according to an embodiment of the present invention;

[0022] Figure 2 is a flow chart of a data processing method according to an embodiment of the present invention;

[0023] Figure 3 is a schematic diagram of a data processing method according to an embodiment of the present invention;

[0024] Figure 4 is a structural block diagram of a data processing device according to an embodiment of the present invention;

[0025] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0027] Please refer to Figure 1 , Figure 1 is a structural block diagram of a data processing system according to an embodiment of the present invention.

[0028] Combination Figure 1 As shown, the data processing system includes: a metadata collection task module and a metadata update module.

[0029] The metadata collection task module includes: database connection pool management module, incremental automatic collection strategy module, and transmission module.

[0030] The metadata update module includes: an asynchronous update processing module, an incremental synchronization to the metadata business library module, and a metadata change real-time query module.

[0031] The database connection pool management module is used to maintain the connection with the target database for a period of time after the metadata collection task is initiated, and to prepare for monitoring Data Definition Language (DDL) events.

[0032] The incremental automatic collection strategy module establishes a communication connection with the database connection pool management module. It is used to capture the target event information of DDL event changes of objects at all levels in the target database when monitoring the DDL event trigger of the target database, and parse the target event information. It describes the changed metadata configuration information and metadata coordinate information in the target database through a unified data message format. This is the changed message content, and extracts the incremental metadata information of the target metadata from the target database based on the changed metadata configuration information and metadata coordinate information.

[0033] The transmission module establishes a communication connection with the incremental automatic collection strategy module to put the incremental metadata information of the target metadata into the message queue, use the unique identifier of the instance as the TOPIC identifier of the message queue, publish the parsed message content to the specified TOPIC, and complete the production and push of the change message.

[0034] The asynchronous update processing module is used to subscribe to metadata change messages in the specified queue according to TOPIC. When a new metadata change message enters the message queue, it consumes the message asynchronously and extracts the metadata change information from it.

[0035] The incremental synchronization to the metadata service library module communicates with the asynchronous update processing module to connect to the metadata service library based on the extracted metadata change information, query based on the database name and table name, and determine whether the metadata already exists. If it exists, the metadata is updated; if not, new metadata is inserted.

[0036] The metadata change real-time query module is used to provide a query interface, display the updated metadata information on the server page, and can also display the historical records of the queried metadata changes on the server page according to the time range.

[0037] According to an embodiment of the present invention, a data processing method embodiment is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0038] In this embodiment, a data processing method is provided, which can be used in the above-mentioned data processing system. Figure 2 is a flow chart of a data processing method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0039] Step S201, obtaining target event information corresponding to a target database; wherein the target event information includes: metadata configuration information and metadata coordinate information.

[0040] The metadata configuration information may indicate the rules and parameters for processing and collecting metadata in the target database, such as which metadata to collect, the frequency of collection, etc. The metadata coordinate information is used to uniquely identify specific metadata in the target database, such as the name of the table, the name of the field, etc.

[0041] The target database may represent the source database that needs metadata operation or data synchronization, and the target database contains a large amount of business data and related metadata. The target event information may represent information about a specific event that occurs in the target database, which may include metadata configuration information and metadata coordinate information.

[0042] Deploy a DDL event listener in the target database. When a DDL event triggers an operation in the target database, the listener is triggered to capture the event and obtain the target event information corresponding to the target database.

[0043] As an example, a MySQL database can choose to monitor the Binlog file and filter the MySQL Binlog log information based on the DDL statement keywords.

[0044] Step S202: determining target metadata matching the metadata coordinate information from a target database.

[0045] When a trigger operation of a DDL event occurs in the target database, the target event information corresponding to the DDL event is determined, and the target event information includes metadata configuration information and metadata coordinate information. Then, the corresponding target metadata is searched from the target database.

[0046] As an example, SQL query or other database operation languages ​​are used to locate target metadata according to coordinate information (such as table name, field name, etc.).

[0047] As an example, a new database mydb is created in the current instance (the executed SQL statement is: createdatabase mydb;). After listening to this DDL event, it is parsed and a metadata change message, that is, incremental metadata information, is generated.

[0048] The message format example is as follows:

[0049] {

[0050] "level": "database",

[0051] "name": "mydb",

[0052] "operation": "CREATE"

[0053] }

[0054] Among them, level can represent the level of metadata, name can represent the name of the metadata that has changed, and operation represents the operation type of the change.

[0055] Step S203: collecting incremental metadata information of target metadata from the target database according to the metadata configuration information.

[0056] The incremental metadata that needs to be collected can be determined according to the metadata configuration information, and then the incremental metadata information of the incremental metadata is extracted from the target database.

[0057] As an example, it is possible to determine which metadata is newly added or changed by comparing the timestamp, version number, or hash value of the metadata, and then collect incremental metadata information.

[0058] As an example, based on metadata configuration information (such as only collecting newly added and modified field information), compare the differences between current metadata and historical metadata, determine which fields are newly added or modified, and extract incremental metadata information of these fields.

[0059] As an example, the target database has stored metadata of table A, and the target event information represents metadata of table B newly added to the target database. Then, the metadata of the newly added table B needs to be extracted from the target database.

[0060] Step S204: synchronizing the incremental metadata information to the business database.

[0061] The business database can represent a database outside the target database that is used to store or process business logic, which may involve further analysis, display or application of data. After the incremental metadata information is extracted from the target database, the incremental metadata information is synchronized to the business database.

[0062] As an example, after the listener completes parsing of the DDL event, the metadata change message obtained by parsing is published to a message queue (such as RocketMQ), and the message type is an ordered message.

[0063] As an example, you can specify the current TOPIC as MySQL_Number1, and put the corresponding message content into the TOPIC in the order of DDL changes. Subscribe to the incremental metadata information in the specified queue according to TOPIC. When incremental metadata information enters the message queue, consume the incremental metadata information asynchronously and extract the incremental metadata information from it. According to the extracted metadata change information, connect to the metadata business library, query according to the database name and table name, and determine whether the metadata already exists. If it exists, update the metadata corresponding to the incremental metadata information in the business database according to the incremental metadata information; if it does not exist, insert the incremental metadata information.

[0064] The data processing method provided in this embodiment, when acquiring the target event information corresponding to the target database, accurately locates the target metadata that needs to be collected through the metadata coordinate information, and then only collects the incremental metadata information of the target metadata through the metadata configuration information, thereby avoiding a large number of unnecessary database accesses during the full collection process, thereby improving the collection efficiency.

[0065] In a possible implementation, the above step S204 includes:

[0066] Step S2041, using the database name and table name, detect whether the target metadata corresponding to the incremental metadata information exists in the business database corresponding to the target database; wherein the initial metadata of the business database and the initial metadata of the target database are the same in quantity and type.

[0067] The incremental metadata information may represent operations such as modification, change, and addition of a database, or operations such as modification, change, and addition of a table. Specifically, the provided database name and table name are used as identifiers to query whether there is target metadata corresponding to the incremental metadata information in the business database.

[0068] It should be noted that if the user is not logging into the instance for the first time, that is, before the user logs into the instance, the historical metadata of the target database already exists in the business database, that is, the initial metadata of the business database and the initial metadata of the target database are the same in quantity and type. Therefore, step S201 to step S204 are executed.

[0069] Step S2042: If the target metadata corresponding to the incremental metadata information exists in the business database, the target metadata in the business database is updated according to the incremental metadata information.

[0070] Step S2043: If the target metadata corresponding to the incremental metadata information does not exist in the business database, the incremental metadata information is added to the business database.

[0071] After confirming that the target metadata exists in the business database, the incremental metadata information is used to update the corresponding record in the business database. After confirming that the target metadata does not exist in the business database, the incremental metadata information is inserted into the business database as a new record.

[0072] As an example, according to the specific content of the incremental metadata information (such as the changed value of the field, the newly added field, etc.), write the corresponding SQL update statement or call the update API provided by the database to update the metadata in the business database. According to the structure of the incremental metadata information, write the corresponding SQL insert statement or call the insert API provided by the database to add the new metadata record to the business database.

[0073] As an example, the business database has already saved the metadata of table A, but what is pulled back from the message queue is the incremental metadata information of table B. In this case, the business database needs to add and save the incremental metadata information of table B. If the incremental metadata information of table A is pulled from the message queue, it can be updated in the original table A data.

[0074] The data processing method provided in this embodiment detects whether the target metadata corresponding to the incremental metadata information exists in the business database corresponding to the target database through the database name and table name, and performs corresponding operations (updates or adds metadata) according to the detection results, thereby avoiding a comprehensive scan of the entire database or large-scale data migration, and improving the efficiency of data processing.

[0075] In a possible implementation, the above step S201 includes:

[0076] Step S2011, detecting whether the number of idle connections in the database connection pool of the target database meets the number of idle connections.

[0077] The database connection pool can be used to improve the reuse rate of database connections and reduce the overhead of establishing and releasing connections. Among them, the database connection pool will maintain a certain number of database connections for the application to obtain when needed. The number of idle connections can represent the number of connections that are currently not in use (i.e., idle) in the database connection pool. This value reflects the available resources of the connection pool. Specifically, check whether the number of idle connections in the database connection pool of the target database reaches or exceeds the preset idle connection number threshold. Among them, the preset idle connection number threshold can represent the number of idle connections corresponding to the server corresponding to the target database when the load is low. Among them, the number of idle connections can be 30, 40, etc., which is not specifically limited here.

[0078] As an example, for metadata collection, you can use the HikariCP database connection pool, configure the maximum number of connections to 50, the minimum number of idle connections to 10, and set the connection timeout to 30 seconds to ensure efficient acquisition and release of connections. After connecting to the target database, retain a connection and identify the connection with an instance ID.

[0079] It should be noted that the purpose of identification through instance ID is to reuse database connections. This instance ID is the client's unique identifier for the business database instance.

[0080] Step S2012: If the number of idle connections in the database connection pool of the target database satisfies the number of idle connections, a step of obtaining target event information corresponding to the target database is performed.

[0081] If step S2011 detects that the number of idle connections meets the requirement (i.e. reaches or exceeds the preset threshold), the next step is performed, i.e., obtaining target event information related to a specific event or operation in the target database. This usually involves executing a database query or calling a related API to obtain the required data.

[0082] The data processing method provided in this embodiment, if the number of idle connections in the database connection pool of the target database meets the number of idle connections, the load of the target database is low, that is, metadata is collected when the load of the target database is low, thereby reducing the impact on the target database.

[0083] In a possible implementation, the method further includes: displaying the updated target metadata in the business database on a server page.

[0084] The target metadata in the business database can represent specific or interesting data fields or records in the business database, which contain information about the data itself, such as data name, type, structure, etc. The server page can represent the web page running on the server, which is used to display data or provide interactive functions to the client (such as a browser).

[0085] As an example, write and execute SQL queries directly in database management tools (such as MySQL Workbench, SQL Server Management Studio). Embed SQL queries in server-side code and execute queries through database connection libraries (such as Python's pymysql, Node.js's mysql or mongoose).

[0086] As an example, through the query interface, users are supported to obtain updated metadata information, and can also query the history of metadata changes based on a time range.

[0087] The data processing method provided in this embodiment increases the transparency of the system by displaying the target metadata on the page, so that the user can clearly understand the status and content of the business database, thereby enhancing the credibility of the system and the trust of the user.

[0088] In a possible implementation, when there are multiple pieces of incremental metadata information, synchronizing the incremental metadata information to the business database through the message queue includes:

[0089] Step a1: put the incremental metadata information into the message queue according to a preset order.

[0090] Capture data change events and extract incremental metadata information. Sort the incremental metadata information according to the preset order. Put the sorted incremental metadata information into the message queue.

[0091] Step a2: When the scheduled task is triggered, the incremental metadata information in the message queue is synchronized to the business database in sequence according to the preset order.

[0092] Scheduled tasks can represent tasks that are automatically executed at a predetermined time interval or at a specific time point. In data synchronization scenarios, scheduled tasks are usually used to trigger the synchronization process of data from a message queue to a business database. Specifically, a scheduled task is triggered at a predetermined time. Incremental metadata information is read from the target message queue in a preset order. The read incremental metadata information is synchronized to the business database.

[0093] As an example, one incremental metadata is read from the message queue each time and then inserted into the business database.

[0094] In a possible implementation, multiple incremental metadata may be read from a message queue at one time and then inserted into a business database in batches.

[0095] The data processing method provided in this embodiment can ensure that the data remains orderly when synchronized to the business database by putting the incremental metadata information into the message queue in a preset order, avoiding potential problems caused by data disorder. The scheduled task trigger synchronization mechanism can adjust the synchronization frequency according to actual needs, avoiding resource consumption caused by unnecessary real-time synchronization. At the same time, the message queue can buffer a large amount of data, balance the speed of data production and consumption, and optimize resource utilization.

[0096] Please refer to Figure 3 , Figure 3 It is a schematic diagram of a data processing method provided according to an embodiment of the present invention.

[0097] Combination Figure 3As shown in the figure, if the user logs in to the instance for the first time, the historical metadata of the target database does not exist in the business database. Then the DMS service (Data Management Service) needs to pull all the metadata from the target database at one time. When the target database triggers a DDL event, the listener is triggered to capture the event and obtain the target event information corresponding to the target database. After the listener completes the parsing of the DDL event, the metadata change message obtained by the parsing is published to the message queue (Message Queue, MQ), and the message type adopts an ordered message. When incremental metadata information enters the message queue, the incremental metadata information is consumed asynchronously and extracted from it. According to the extracted metadata change information, the metadata business library is connected, and the query is performed according to the database name and table name to determine whether the metadata already exists. If it exists, the metadata corresponding to the incremental metadata information in the business database is updated according to the incremental metadata information; if it does not exist, the incremental metadata information is inserted. Then, through the query interface, users are supported to obtain updated metadata information, and the historical records of metadata changes can also be queried according to the time range.

[0098] In the present embodiment, a data processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0099] This embodiment provides a data processing device, such as Figure 4 As shown, it includes: a monitoring module 401, which is used to obtain target event information corresponding to a target database; wherein the target event information includes: metadata configuration information and metadata coordinate information; a parsing module 402, which is used to determine target metadata that matches the metadata coordinate information from the target database; a collection module 403, which is used to collect incremental metadata information of the target metadata from the target database according to the metadata configuration information; and a synchronization module 404, which is used to synchronize the incremental metadata information to the business database.

[0100] In one possible implementation, the synchronization module 404 includes: a first detection unit, used to detect whether the target metadata corresponding to the incremental metadata information exists in the business database corresponding to the target database through the database name and table name; wherein the quantity and type of the initial metadata of the business database and the initial metadata of the target database are the same; an update unit, used to update the target metadata in the business database according to the incremental metadata information if the target metadata corresponding to the incremental metadata information exists in the business database; and an addition unit, used to add the incremental metadata information to the business database if the target metadata corresponding to the incremental metadata information does not exist in the business database.

[0101] In one possible implementation, the monitoring module 401 includes: a second detection unit, used to detect whether the number of idle connections in the database connection pool of the target database meets the idle connection number; and an execution unit, used to execute the step of obtaining target event information corresponding to the target database if the number of idle connections in the database connection pool of the target database meets the idle connection number.

[0102] In a possible implementation, the apparatus further includes: a display module, configured to display the updated target metadata in the service database on a server page.

[0103] In a possible implementation, the display module includes: placing the incremental metadata information into a message queue in a preset order, and when a scheduled task is triggered, synchronizing the incremental metadata information in the message queue to a business database in sequence in the preset order.

[0104] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0105] The data processing in this embodiment is presented in the form of a functional unit, where the functional unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0106] The embodiment of the present invention also provides a computer device having the above Figure 4 The data processing shown.

[0107] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0108] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0109] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0110] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0111] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0112] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0113] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0114] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0115] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A data processing method, characterized in that: The method comprises: Acquire target event information corresponding to the target database; wherein the target event information includes: metadata configuration information and metadata coordinate information; Determining target metadata matching the metadata coordinate information from the target database; According to the metadata configuration information, collecting incremental metadata information of the target metadata from the target database; The incremental metadata information is synchronized to the business database.

2. The data processing method according to claim 1, characterized in that: The step of synchronizing the incremental metadata information to a business database includes: By using the database name and the table name, detecting whether the target metadata corresponding to the incremental metadata information exists in the business database corresponding to the target database; wherein the initial metadata of the business database and the initial metadata of the target database are the same in quantity and type; If the target metadata corresponding to the incremental metadata information exists in the business database, updating the target metadata in the business database according to the incremental metadata information; If the target metadata corresponding to the incremental metadata information does not exist in the business database, the incremental metadata information is added to the business database.

3. The data processing method according to claim 1, characterized in that: The step of obtaining target event information corresponding to the target database includes: Detecting whether the number of idle connections in the database connection pool of the target database meets the number of idle connections; If the number of idle connections in the database connection pool of the target database satisfies the number of idle connections, a step of acquiring target event information corresponding to the target database is performed.

4. The data processing method according to claim 2, characterized in that: The method further comprises: The updated target metadata in the business database is displayed on the server page.

5. The data processing method according to claim 1, characterized in that: Synchronizing the incremental metadata information to the business database includes: The incremental metadata information is synchronized to the business database through the message queue.

6. The data processing method according to claim 5, characterized in that: When there are multiple pieces of incremental metadata information, synchronizing the incremental metadata information to the business database through a message queue includes: Putting the incremental metadata information into the message queue according to a preset order; When a scheduled task is triggered, the incremental metadata information in the message queue is synchronized to the business database in sequence according to a preset order.

7. A data processing device, characterized in that: The device comprises: A monitoring module is used to obtain target event information corresponding to a target database; wherein the target event information includes: metadata configuration information and metadata coordinate information; A parsing module, used for determining target metadata matching the metadata coordinate information from the target database; A collection module, used for collecting incremental metadata information of the target metadata from the target database according to the metadata configuration information; The synchronization module is used to synchronize the incremental metadata information to the business database.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the data processing method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the data processing method according to any one of claims 1 to 6.

10. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the data processing method according to any one of claims 1 to 6.

Images