Database import task management method and system, terminal equipment and storage medium

Abstract

The invention provides an import task management method and system of a database, terminal equipment and a storage medium, the method is applied to an importer manager, and the method comprises the following steps: defining an importer type and an operation mode of an import task through a configuration file of a target database; according to the type of the importer, loading a corresponding importer plug-in; when the operation mode is a global mode, starting an instance of the importer plug-in on all nodes of a cluster of the target database so as to import target data; and when the operation mode is a distributed mode, starting an instance of the importer plug-in according to a specified node of a channel distributor so as to import target data. According to the method, the importer manager is configured in the database kernel in advance, the boundary between the database and the external import tool is eliminated, the import task is uniformly defined through the configuration file, and the problems of inconsistent states, asynchronous configuration and the like caused by system separation in the prior art are solved.

Description

Technical Field

[0001] This invention application relates to the field of database management technology, and in particular to a database import task management method, system, terminal device and storage medium. Background Technology

[0002] Currently, enterprise data architectures are often multi-source and heterogeneous, meaning they have multiple external data sources with diverse and complex data types and structures. External data sources can include message queues (Kafka, RabbitMQ), real-time log files, CSV / JSON files on FTP servers, and other relational or NoSQL databases. To unify and aggregate this data into a core, purely in-memory database for analysis and processing, enterprises typically deploy, run, and maintain a complex data integration platform.

[0003] The aforementioned data integration platform integrates multiple tools. For example, enterprises need to select and deploy different data synchronization tools or write dedicated synchronization scripts for each data source, such as Kafka Connect, Flume, Logstash, or custom scripts. These tools are actually separated from the database. The management operations of data imported into the database, such as configuration, monitoring, start-up, and upgrades, are scattered across different systems, greatly increasing the complexity and cost of operation and maintenance, and resulting in poor reliability. Summary of the Invention

[0004] This invention application provides a database import task management method, system, terminal device, and storage medium to solve the technical problem of how to improve the reliability of data import status management in memory-based databases.

[0005] To address the aforementioned technical problems, this invention provides a database import task status management method, applied to an importer manager, wherein the importer manager is pre-configured in the kernel of the target database; The imported data status management method includes: The importer type and running mode of the import task are defined through the configuration file of the target database; When the target database starts, the corresponding importer plugin is loaded according to the importer type; According to the operating mode, when the operating mode is global mode, instances of the importer plugin are started on all nodes of the target database cluster to import the target data; when the operating mode is distributed mode, instances of the importer plugin are started on nodes specified by the channel allocator to import the target data.

[0006] As a preferred embodiment, before loading the corresponding importer plugin, the following is included: The importer plugin is obtained by reading the pre-configured importer section of the configuration file, parsing and mapping the importer to a set of standard operating states, and building the association between the importer and the predefined rule set.

[0007] As a preferred embodiment, before parsing and mapping the importer to a set of standard operating states, the method further includes: Extract the task parameters of the imported task and generate a status object based on the task parameters; wherein, the status object is used to record the current status, historical changes and association rules; Bind the operation rule to each state object and verify whether the state object matches the target database; Upon successful verification, the standard operating state is configured based on the state object.

[0008] As a preferred embodiment, before activating an instance of the importer plugin, the following is also included: Calculate resource requirements based on the task type and execution mode of the imported task; Based on the resource requirements, allocate a thread pool to the importer plugin; A thread executor is created for each of the importer plugins, and the initialization of the importer plugins is implemented under the control of the SQL interface.

[0009] As a preferred embodiment, the import task management method further includes: Real-time monitoring of the instance's performance metrics; wherein the performance metrics include at least one of processing rate, error rate, and data latency; The thread pool is scaled based on the performance metrics and the current load of the target database.

[0010] As a preferred embodiment, the import task management method further includes: Monitor the instance's runtime logs and performance metrics in real time; Based on the operation logs and performance metrics, determine whether the instance has failed; When it is determined that a failure has occurred in the instance, the type of failure is identified; Execute the corresponding recovery action according to the fault type and preset strategy.

[0011] As a preferred embodiment, the import task management method further includes: Obtain the stop command through the management interface of the target database; According to the stop command, a stop signal is sent to the importer plugin so that the importer plugin exits when it has completed the import of the current batch of target data.

[0012] Accordingly, this application also provides a database import task management system, applied to an importer manager, wherein the importer manager is pre-configured in the kernel of the target database; The imported data status management system includes a definition module, a loading module, and an import module; wherein... The definition module is used to define the importer type and running mode of the import task through the configuration file of the target database; The loading module is used to load the corresponding importer plugin according to the importer type when the target database starts. The import module is configured to, depending on the operating mode, launch instances of the importer plugin on all nodes of the target database cluster when the operating mode is global mode, thereby importing the target data; and launch instances of the importer plugin on nodes specified by the channel allocator when the operating mode is distributed mode, thereby importing the target data.

[0013] As a preferred embodiment, the import task management system further includes an importer plugin building module, which is used before loading the corresponding importer plugin: The importer plugin is obtained by reading the pre-configured importer section of the configuration file, parsing and mapping the importer to a set of standard operating states, and building the association between the importer and the predefined rule set.

[0014] As a preferred embodiment, the import task management system further includes a runtime configuration module, which is used before parsing and mapping the importer to a set of standard runtime states: Extract the task parameters of the imported task and generate a status object based on the task parameters; wherein, the status object is used to record the current status, historical changes and association rules; Bind the operation rule to each state object and verify whether the state object matches the target database; Upon successful verification, the standard operating state is configured based on the state object.

[0015] As a preferred embodiment, the import task management system further includes an initialization module, which is used before the instance of the importer plugin is launched: Calculate resource requirements based on the task type and execution mode of the imported task; Based on the resource requirements, allocate a thread pool to the importer plugin; A thread executor is created for each of the importer plugins, and the initialization of the importer plugins is implemented under the control of the SQL interface.

[0016] As a preferred embodiment, the import task management system further includes a thread pool adjustment module, which is used for: Real-time monitoring of the instance's performance metrics; wherein the performance metrics include at least one of processing rate, error rate, and data latency; The thread pool is scaled based on the performance metrics and the current load of the target database.

[0017] As a preferred embodiment, the import task management system further includes a recovery module, which is used for: Monitor the instance's runtime logs and performance metrics in real time; Based on the operation logs and performance metrics, determine whether the instance has failed; When it is determined that a failure has occurred in the instance, the type of failure is identified; Execute the corresponding recovery action according to the fault type and preset strategy.

[0018] As a preferred embodiment, the import task management system further includes an importer plugin exit module, which is used for: Obtain the stop command through the management interface of the target database; According to the stop command, a stop signal is sent to the importer plugin so that the importer plugin exits when it has completed the import of the current batch of target data.

[0019] Accordingly, this application also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the database import task management method.

[0020] Accordingly, this application also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the database import task management method.

[0021] Compared with the prior art, this invention application has the following beneficial effects: This invention provides a database import task management method, system, terminal device, and storage medium. The import task status management method is applied to an importer manager, which is pre-configured in the kernel of the target database. The import data status management method includes: defining the importer type and running mode of the import task through the configuration file of the target database; loading the corresponding importer plugin according to the importer type when the target database starts; and, according to the running mode, starting instances of the importer plugin on all nodes of the target database cluster when the running mode is global mode, thereby realizing the import of target data; and starting instances of the importer plugin on nodes specified by the channel allocator when the running mode is distributed mode, thereby realizing the import of target data. This invention pre-configures the importer manager within the database kernel, eliminating the boundary between the database and external import tools. It also unifies import tasks through configuration files, resolving issues such as inconsistent states and asynchronous configurations caused by system separation in existing technologies. Direct kernel-level management significantly reduces system failures caused by inter-component communication failures or version incompatibility, fundamentally improving the reliability of data import tasks and the stability of the entire system. Furthermore, by configuring the importer manager within the target database kernel and unifying import tasks through configuration files, it completely changes the traditional approach of configuring and managing multiple distributed external systems separately. This eliminates the need for users to learn and maintain multiple heterogeneous tools, greatly simplifying the operation and maintenance process, reducing management and learning costs, and achieving a shift from distributed to centralized operation and maintenance. Moreover, dynamically loading corresponding plugins based on the importer type provides high scalability and flexibility. When accessing new data sources, only the corresponding importer plugin needs to be developed and configured, without modifying the database kernel or restarting existing services. This allows the database to easily and quickly adapt to the evolving data ecosystem of enterprises. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating an embodiment of the database import task management method provided in this invention application.

[0023] Figure 2 This is a flowchart illustrating a preferred embodiment of the database import task management method provided in this invention application.

[0024] Figure 3 This is a schematic diagram of an embodiment of the database import task management system provided in this application. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] Example 1 Please refer to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the database import task management method provided in this invention application.

[0027] In this embodiment, the import task management method is applied to the importer manager, which is pre-configured in the kernel of the target database and can be automatically initialized when the target database starts.

[0028] Figure 1 The imported task management method shown includes steps S101 to S103; each step is described in detail below: Step S101: Define the importer type and running mode of the import task through the configuration file of the target database.

[0029] In this embodiment, a dedicated importer configuration section can be provided in the main configuration file of the target database. Users or administrators can define multiple different import tasks declaratively. Specifically, the task's unique identifier, importer type (such as Kafka, CSV, or JDBC), data source connection information, target stream table / regular table, associated stored procedures, and running mode (global mode or distributed mode) can be defined.

[0030] Prior to step S102, the importer manager can verify the integrity and correctness of each import task configuration, such as checking whether the specified stored procedure exists and whether the network address is reachable.

[0031] Preferably, before step S102, the importer section pre-configured in the configuration file can be read, parsed and mapped to a set of standard running states (such as initialization, running, paused, error and termination), and the association between the importer and the predefined rule set can be constructed to obtain the importer plugin.

[0032] In this preferred embodiment, for example, the rule set may include resource thresholds (such as triggering an alarm when CPU utilization is >80%) or fault recovery strategies (such as automatically restarting after 3 consecutive failures), and these rules may be based on conditional judgment (such as comparing the current metric with the threshold).

[0033] Before mapping importers to a set of standard runtime states, each standard runtime state must first be defined and configured. For example, such as... Figure 2 As shown, Figure 2 This is a flowchart illustrating a preferred embodiment of the database import task management method provided in this invention application. Figure 2 The illustrated implementation includes steps S201 to S203; each step is described in detail below: Step S201: Extract the task parameters of the import task and generate a status object based on the task parameters; wherein, the status object is used to record the current status, historical changes and association rules of the importer.

[0034] Step S202: Bind the operation rule to each state object and verify whether the state object matches the target database.

[0035] Step S203: Upon successful verification, the standard operating state is configured based on the state object.

[0036] In this preferred embodiment, the task parameters include the task type and the corresponding data source information. For the above-mentioned operation rules, in some examples, it could be: during operation, if data latency is detected to exceed a threshold, automatic rate adjustment is triggered. The rules can be stored in a system table in the database for easy dynamic modification.

[0037] Step S202, verifying whether the state object matches the target database, can also be verifying whether the import task configuration matches the topology of the target database (e.g., in distributed mode, confirming whether the nodes are correctly allocated).

[0038] In some examples, status information, including task ID, current status, duration, and key metrics (such as processing rate), can be queried in real time using monitoring views within the target database (such as the IMPORTER_STATUS view). Detection is achieved by periodically scanning (e.g., performing an internal check every 5 seconds) in conjunction with logging to capture status change events.

[0039] Furthermore, users or administrators can dynamically update the rule set using SQL commands, enabling flexible state management. Commands can specify new rules; for example, if the error rate exceeds 5%, the import task can be paused, and the importer manager will apply these rules immediately without restarting the database.

[0040] Step S102: When the target database is started, the corresponding importer plugin is loaded according to the importer type.

[0041] This embodiment adopts a plug-in architecture to support different types of importers. The importer manager can dynamically load and instantiate the corresponding importer implementation (such as KafkaImporter, CsvFileImporter) based on the importer type (such as the type field).

[0042] For each successfully initialized importer instance, a separate, controlled thread pool executor can be created. This ensures resource isolation between different import tasks, and an exception in one task will not block or exhaust the resources of other tasks.

[0043] In some preferred embodiments, before starting an instance of the importer plugin in step S103, the following can be done: calculate resource requirements based on the task type and running mode of the import task; allocate a thread pool for the importer plugin based on the resource requirements; create a thread executor for each importer plugin and initialize the importer plugin under the control of the SQL interface.

[0044] For example, the above resource requirements could be, for instance, estimating I / O load and allocating a minimum thread pool for a CSV file import task; or, for a Kafka task, calculating concurrency based on the number of partitions.

[0045] Resource allocation follows a predefined formula (e.g., number of threads = number of cores × load factor), and simulation tests are performed before startup (e.g., running a small batch of data).

[0046] Before starting an instance, you can also perform a health check loop, such as testing data source connectivity (ping operation) and target table availability, to ensure that there are no blockages before activating the instance.

[0047] In addition, some implementations can use internal monitoring metrics (such as startup latency and resource utilization) to detect the health of the initialization process. Detection is achieved through the database's system logs and real-time views (such as IMPORTER_INIT_METRICS), recording key events (such as loading time and error codes), and setting threshold alarms (such as triggering when initialization timeout > 10 seconds).

[0048] Step S103: According to the operating mode, when the operating mode is global mode, the instance of the importer plugin is started on all nodes of the target database cluster to realize the import of target data; when the operating mode is distributed mode, the instance of the importer plugin is started on the node specified by the channel allocator to realize the import of target data.

[0049] Generally, the importer manager manages the state of an importer instance throughout its entire lifecycle, including multiple phases such as startup, running, stopping, and reloading. The running modes primarily include global mode and distributed mode.

[0050] Among them, Start mainly refers to determining the startup method based on the running mode configured for each task.

[0051] Global Mode: Suitable for tasks such as file scanning. ImporterManager starts an instance of the importer on every node in the cluster.

[0052] Distributed Mode: Suitable for partitionable data sources such as Kafka. The importer manager registers importers with the channel allocator, which then determines which partitions to process on which target database nodes.

[0053] Preferably, when in running mode, the performance metrics of the instance are monitored in real time; wherein the performance metrics include at least one of processing rate, error rate, and data latency; The thread pool is scaled based on the performance metrics and the current load of the target database.

[0054] Specifically, a monitoring scan can be performed at fixed intervals (e.g., every 10 seconds) to collect metrics (processing rate, error rate, data latency). Based on these metrics, predefined rules are applied for automatic adjustments; for example, if data latency increases, the number of threads is dynamically increased; if the error rate rises, the batch size is reduced.

[0055] In addition, the importer manager can maintain a resource pool that automatically scales the thread pool based on the current system load (such as CPU / memory usage). For example, it can use simple scaling calculations (e.g., new thread count = old thread count × (1 + load increase)) to ensure resource isolation. The scaling process is performed in batches to avoid abrupt changes affecting stability.

[0056] If the running mode of an import task needs to be changed (such as switching from global mode to distributed mode), the importer manager can use a transition process to pause the current instance and then reinitialize the new mode to ensure a seamless switch.

[0057] Administrators can use the SQL query interface to obtain the current status, cumulative processing records, and abnormal events.

[0058] In some preferred embodiments, for the aforementioned Stop state, the import task management method further includes: obtaining a stop command through the management interface of the target database; and sending a stop signal to the importer plugin according to the stop command, so that the importer plugin exits when it completes the import of the current batch of target data.

[0059] Specifically, through the database management interface (such as the SQL command `ALTER IMPORTER my_kafka_importSTOP`), administrators can dynamically and safely stop a running import task. The importer manager will send a stop signal to the corresponding importer (plugin), and the importer will exit after completing the processing of the current batch.

[0060] For reload: When the configuration file is modified, a reload can be triggered via command. The importer manager will compare the old and new configurations, stop deleted import tasks, start newly added tasks, and restart tasks whose configurations have been modified. The entire process does not require restarting the database.

[0061] In addition, the importer manager can continuously monitor the running status of all importer instances it manages. If an importer thread exits abnormally, the importer manager can recover according to preset strategies (such as automatic restart), improving system robustness.

[0062] Specifically, the importer manager can monitor the instance's runtime logs and performance metrics in real time; based on the runtime logs and performance metrics, it can determine whether the instance has failed; when it is determined that the instance has failed, it can identify the failure type (such as network disconnection or data format error, temporary failure or persistent failure, etc.); and execute corresponding recovery actions according to the failure type and preset policies, for example, automatically retrying (up to 3 times) for short-term failures and pausing the task for persistent failures.

[0063] Some implementations may use rule-based recovery procedures, such as automatically restarting a specified thread pool if thread blocking is detected, or switching to an alternate source if the data source is unavailable. The self-healing process includes a waiting period (e.g., a 5-second retry interval) to ensure that recovery does not interfere with normal operation.

[0064] All failure events can be logged in the database audit log, and a notification mechanism (such as sending email alerts) can be configured.

[0065] Detection methods can include indicator threshold monitoring, log analysis, health scoring, and control methods. Specifically: Metric threshold monitoring: Detection is triggered when the data processing rate drops by more than 50%.

[0066] Log analysis: Scan error logs, identify keywords (such as connection failed), and quantify the frequency of failures.

[0067] Health score: Calculate a comprehensive score (such as a weighted average based on error rate and latency), and activate self-healing when it falls below a threshold.

[0068] Control method: Enable or disable the self-healing function via SQL commands. The control method includes a feedback loop: detect the fault → apply recovery rules → verify successful recovery; if it fails, escalate to manual intervention.

[0069] For the aforementioned "Stop" state, resources can be released one by one (e.g., closing the thread pool, disconnecting the data source connection), and a cleanup loop can be executed to ensure no residual state remains. After the imported task terminates, the task status and historical data are stored in a database audit table for subsequent analysis.

[0070] Accordingly, such as Figure 3 As shown, this application also provides a database import task management system 300, applied to an importer manager, wherein the importer manager is pre-configured in the kernel of the target database; The imported data status management system includes a definition module 301, a loading module 302, and an import module 303; wherein... The definition module 301 is used to define the importer type and running mode of the import task through the configuration file of the target database; The loading module 302 is used to load the corresponding importer plugin according to the importer type when the target database is started. The import module 303 is configured to, according to the operating mode, start instances of the importer plugin on all nodes of the target database cluster when the operating mode is global mode, thereby realizing the import of target data; and when the operating mode is distributed mode, start instances of the importer plugin on nodes specified by the channel allocator, thereby realizing the import of target data.

[0071] As a preferred embodiment, the import task management system 300 further includes an importer plugin construction module, which is used before loading the corresponding importer plugin: The importer plugin is obtained by reading the pre-configured importer section of the configuration file, parsing and mapping the importer to a set of standard operating states, and building the association between the importer and the predefined rule set.

[0072] As a preferred embodiment, the import task management system 300 further includes a runtime configuration module, which is used before parsing and mapping the importer to a set of standard runtime states: Extract the task parameters of the imported task and generate a status object based on the task parameters; wherein, the status object is used to record the current status, historical changes and association rules; Bind the operation rule to each state object and verify whether the state object matches the target database; Upon successful verification, the standard operating state is configured based on the state object.

[0073] As a preferred embodiment, the import task management system 300 further includes an initialization module, which is used before the instance of the importer plugin is launched: Calculate resource requirements based on the task type and execution mode of the imported task; Based on the resource requirements, allocate a thread pool to the importer plugin; A thread executor is created for each of the importer plugins, and the initialization of the importer plugins is implemented under the control of the SQL interface.

[0074] As a preferred embodiment, the import task management system 300 further includes a thread pool adjustment module, which is used for: Real-time monitoring of the instance's performance metrics; wherein the performance metrics include at least one of processing rate, error rate, and data latency; The thread pool is scaled based on the performance metrics and the current load of the target database.

[0075] As a preferred embodiment, the import task management system 300 further includes a recovery module, which is used for: Monitor the instance's runtime logs and performance metrics in real time; Based on the operation logs and performance metrics, determine whether the instance has failed; When it is determined that a failure has occurred in the instance, the type of failure is identified; Execute the corresponding recovery action according to the fault type and preset strategy.

[0076] As a preferred embodiment, the import task management system 300 further includes an importer plugin exit module, which is used for: Obtain the stop command through the management interface of the target database; According to the stop command, a stop signal is sent to the importer plugin so that the importer plugin exits when it has completed the import of the current batch of target data.

[0077] Accordingly, this application also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the database import task management method.

[0078] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal, connecting various parts of the terminal via various interfaces and lines.

[0079] The memory can be used to store the computer program. The processor implements various functions of the terminal by running or executing the computer program stored in the memory and calling the data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0080] Accordingly, this application also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the database import task management method.

[0081] Wherein, if the modules of the device / terminal equipment / system integration are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by a processor, it can implement the steps of the various method embodiments described above. Wherein, the computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc.

[0082] Compared with the prior art, this invention application has the following beneficial effects: This invention provides a database import task management method, system, terminal device, and storage medium. The import task status management method is applied to an importer manager, which is pre-configured in the kernel of the target database. The import data status management method includes: defining the importer type and running mode of the import task through the configuration file of the target database; loading the corresponding importer plugin according to the importer type when the target database starts; and, according to the running mode, starting instances of the importer plugin on all nodes of the target database cluster when the running mode is global mode, thereby realizing the import of target data; and starting instances of the importer plugin on nodes specified by the channel allocator when the running mode is distributed mode, thereby realizing the import of target data. This invention pre-configures the importer manager within the database kernel, eliminating the boundary between the database and external import tools. It also unifies import tasks through configuration files, resolving issues such as inconsistent states and asynchronous configurations caused by system separation in existing technologies. Direct kernel-level management significantly reduces system failures caused by inter-component communication failures or version incompatibility, fundamentally improving the reliability of data import tasks and the stability of the entire system. Furthermore, by configuring the importer manager within the target database kernel and unifying import tasks through configuration files, it completely changes the traditional approach of configuring and managing multiple distributed external systems separately. This eliminates the need for users to learn and maintain multiple heterogeneous tools, greatly simplifying the operation and maintenance process, reducing management and learning costs, and achieving a shift from distributed to centralized operation and maintenance. Moreover, dynamically loading corresponding plugins based on the importer type provides high scalability and flexibility. When accessing new data sources, only the corresponding importer plugin needs to be developed and configured, without modifying the database kernel or restarting existing services. This allows the database to easily and quickly adapt to the evolving data ecosystem of enterprises.

[0083] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention for those skilled in the art.

Claims

1. A method for managing database import tasks, characterized in that, Applied to an importer manager, which is pre-configured in the kernel of the target database; The imported data status management method includes: The importer type and running mode of the import task are defined through the configuration file of the target database; When the target database starts, the corresponding importer plugin is loaded according to the importer type; According to the operating mode, when the operating mode is global mode, instances of the importer plugin are started on all nodes of the target database cluster to import the target data; when the operating mode is distributed mode, instances of the importer plugin are started on nodes specified by the channel allocator to import the target data.

2. The database import task management method as described in claim 1, characterized in that, Before loading the corresponding importer plugin, the following is included: The importer plugin is obtained by reading the pre-configured importer section of the configuration file, parsing and mapping the importer to a set of standard operating states, and building the association between the importer and the predefined rule set.

3. The database import task management method as described in claim 2, characterized in that, Before parsing and mapping the importer to a set of standard runtime states, the following is also included: Extract the task parameters of the imported task and generate a status object based on the task parameters; wherein, the status object is used to record the current status, historical changes and association rules; Bind the operation rule to each state object and verify whether the state object matches the target database; Upon successful verification, the standard operating state is configured based on the state object.

4. The database import task management method as described in claim 2, characterized in that, Before launching an instance of the importer plugin, the following is also included: Calculate resource requirements based on the task type and execution mode of the imported task; Based on the resource requirements, allocate a thread pool to the importer plugin; A thread executor is created for each of the importer plugins, and the initialization of the importer plugins is implemented under the control of the SQL interface.

5. The database import task management method as described in claim 1, characterized in that, The import task management method also includes: Real-time monitoring of the instance's performance metrics; wherein the performance metrics include at least one of processing rate, error rate, and data latency; The thread pool is scaled based on the performance metrics and the current load of the target database.

6. The database import task management method as described in claim 1, characterized in that, The import task management method also includes: Monitor the instance's runtime logs and performance metrics in real time; Based on the operation logs and performance metrics, determine whether the instance has failed; When it is determined that a failure has occurred in the instance, the type of failure is identified; Execute the corresponding recovery action according to the fault type and preset strategy.

7. A database import task management method as described in any one of claims 1 to 6, characterized in that, The import task management method also includes: Obtain the stop command through the management interface of the target database; According to the stop command, a stop signal is sent to the importer plugin so that the importer plugin exits when it has completed the import of the current batch of target data.

8. A database import task management system, characterized in that, Applied to an importer manager, which is pre-configured in the kernel of the target database; The imported data status management system includes a definition module, a loading module, and an import module; wherein... The definition module is used to define the importer type and running mode of the import task through the configuration file of the target database; The loading module is used to load the corresponding importer plugin according to the importer type when the target database starts. The import module is configured to, depending on the operating mode, launch instances of the importer plugin on all nodes of the target database cluster when the operating mode is global mode, thereby importing the target data; and launch instances of the importer plugin on nodes specified by the channel allocator when the operating mode is distributed mode, thereby importing the target data.

9. A terminal device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the database import task management method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the database import task management method as described in any one of claims 1 to 7.

Images