Method, device, equipment and computer storage medium for importing data into a database

Abstract

This application discloses a method, apparatus, device, and computer storage medium for importing data into a database, relating to the field of database technology. The method involves segmenting the target data file to be transmitted into chunks, treating each chunk as a transmission task to improve the parallelism of data transmission. During the execution of a transmission task, the method locates the data reading position in the target chunk file based on the task offset, and generates a batch of database import instructions based on the data read at each position. These instructions are then sent to the target database to import the data. Simultaneously, the task offset is updated to identify which data has been transmitted, allowing for the location of subsequent data reading positions. This enables data retransmission from interrupted points, avoiding the need to delete data and re-execute transmission when a task fails, thus improving the efficiency of data import.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to the field of database technology, providing a method, apparatus, device, and computer storage medium for importing data into a database. Background Technology

[0002] With the popularization of big data technology, more and more enterprises are introducing big data technology stacks and using big data technology to process and mine massive amounts of data, and then transmitting the calculation results to downstream systems for use.

[0003] In related technologies, computation results are provided to downstream systems primarily through two methods. One method involves unloading the computation results to the local machine after obtaining them and distributing them to downstream systems via file transfer tools. The downstream systems then load these results into their databases using data loading tools. However, this method is lengthy, error-prone, and has high latency. The other method involves directly writing the results to the downstream database using big data technology stacks. However, if the process fails during the writing process, all data must be deleted and the import process restarted to ensure data integrity. This wastes a significant amount of time, resulting in extremely low import efficiency. Summary of the Invention

[0004] This application provides a method, apparatus, device, and computer storage medium for importing data into a database, which enables breakpoint retransmission during data import and improves the efficiency of data import.

[0005] On the one hand, a method for importing data into a database is provided, the method including:

[0006] Based on the preset source data storage path, read the target data file to be imported;

[0007] The target data file is segmented, and the transmission task list is updated based on the obtained multiple segmented files; wherein, in the transmission task list, each transmission task corresponds to one segmented file;

[0008] The transmission task list is iterated over until it is empty; during each iteration, one transmission task is read and the target fragment files corresponding to the transmission task are imported into the target database in batches. The import process for each batch includes:

[0009] Based on the corresponding task offset, the data reading position in the target shard file is located, and based on the data read at the data reading position, a database import instruction for this batch is generated; wherein, the task offset is used to indicate the data that has been imported into the target shard file;

[0010] The generated database import command is sent to the target database, and the task offset is updated.

[0011] On the one hand, an apparatus for importing data into a database is provided, the apparatus comprising:

[0012] The data reading unit is used to read the target data file to be imported based on the preset source data storage path;

[0013] The fragmentation processing unit is used to fragment the target data file and update the transmission task list based on the obtained fragmented files; wherein, in the transmission task list, each transmission task corresponds to one fragmented file;

[0014] A parallel transmission unit is used to iterate through the transmission task list until the list is empty; wherein, during each iteration, one transmission task is read and the target fragment files corresponding to the transmission task are imported into the target database in batches, and the import process of each batch includes:

[0015] Based on the corresponding task offset, the data reading position in the target shard file is located, and based on the data read at the data reading position, a database import instruction for this batch is generated; wherein, the task offset is used to indicate the data that has been imported into the target shard file;

[0016] The generated database import command is sent to the target database, and the task offset is updated.

[0017] In one possible implementation, the data reading unit is further configured to:

[0018] In response to the database import operation, the task execution parameters required for this database import task are obtained. The task execution parameters include data source information and source data operation information.

[0019] Based on the data source information, determine the data source type of the data source to be imported;

[0020] If the data source type is an internal data source, and it is determined based on the source data operation information that no database operation is required for the source data, then the storage path of the internal data source is updated to the source data storage path.

[0021] If the data source type is an external data source, and it is determined based on the source data operation information that no database operation is required for the source data, then the source data is read from the external data source and stored in the source data storage path.

[0022] In one possible implementation, the data reading unit is further configured to:

[0023] If it is an internal data source, and it is determined based on the source data operation information that a database operation needs to be performed on the source data, then after reading the source data from the internal data source and performing the database operation, the source data after the operation is stored in the internal data source, and the storage path of the internal data source is updated to the source data storage path.

[0024] If the data source type is an external data source, and it is determined based on the source data operation information that no database operation is required for the source data, then after reading the source data from the external data source and performing the database operation, the source data after the operation is stored in the source data storage path.

[0025] In one possible implementation, the source data storage path indicates a storage path in a distributed file system; then the data reading unit is specifically used for:

[0026] The big data computing engine reads source data from the external data source and stores it in the distributed file system.

[0027] In one possible implementation, the fragmentation processing unit is specifically used for:

[0028] If a reference field required for sharding is specified, then the sharding reference value is determined based on the value of the reference field in the target data file.

[0029] If no baseline field is specified for sharding, the sharding reference value is determined based on the value of the primary key in the target data file.

[0030] Based on the fragmentation reference value and the specified number of fragments, determine the fragmentation point when the target data file is fragmented;

[0031] Based on the partitioning points, the target data file is partitioned to obtain multiple partitioned files.

[0032] In one possible implementation, the parallel transmission unit is specifically used for:

[0033] During each iteration, a transfer task is read and added to the task execution pool;

[0034] The idle target processes in multiple parallel transmission threads read the transmission tasks to be executed from the task execution pool, and import the target fragment files corresponding to the transmission tasks into the target database in batches.

[0035] In one possible implementation, the parallel transmission unit is specifically used for:

[0036] Based on the corresponding task offset, locate the data reading position in the target fragment file;

[0037] Determine whether the data reading position is the end of the target fragment file;

[0038] If it is not the end, then data is read sequentially based on the data reading position, and each time data is read, a database import instruction for this batch is generated based on the read data;

[0039] The generated database import command will then be sent to the target database, including:

[0040] If the number of generated database import commands reaches the maximum number for this batch, the generated database import commands will be sent to the target database.

[0041] In one possible implementation, the parallel transmission unit is specifically used for:

[0042] If it is the end, determine whether there are any unfinished transmission tasks in the updated transmission task list;

[0043] If it does not exist, update the task status of this database import task to a success status;

[0044] If it exists, continue reading new transmission tasks and add them to the task execution pool.

[0045] In one possible implementation, the parallel transmission unit is further configured to:

[0046] If this database import task fails, a task failure indication will be output.

[0047] In response to a retransmission instruction initiated in response to the task failure indication, the transmission task is read again from the transmission task list;

[0048] Based on the task offset corresponding to the transmission task, the data reading position in the target fragment file is located, and the transmission task continues to be executed based on the data reading position.

[0049] On one hand, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above methods.

[0050] On the one hand, a computer storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of any of the above methods.

[0051] On one hand, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium, and executes the computer program, causing the computer device to perform the steps of any of the methods described above.

[0052] In this embodiment, the target data file to be transmitted is segmented, with each segment treated as a transmission task. During the execution of each transmission task, the task offset is used to locate the data reading position within the target segment. Based on the data read at that position, a batch of database import instructions is generated and sent to the target database to import the data. Simultaneously, the task offset is updated to identify which data has been transmitted and thus locate subsequent data reading positions. By recording the task offset, if a transmission task fails or the entire database import task fails, the data reading position for retransmission can be located based on the task offset, enabling data retransmission from a breakpoint. This avoids the need to delete data and re-execute transmission when a task fails, improving data import efficiency. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0054] Figure 1 This is a schematic diagram illustrating an application scenario provided in the embodiments of this application;

[0055] Figure 2 A system architecture diagram for importing data into a database is provided for embodiments of this application;

[0056] Figure 3 A flowchart illustrating a method for importing data into a database, as provided in an embodiment of this application;

[0057] Figure 4 A flowchart illustrating the data reading process provided in this application embodiment;

[0058] Figure 5 A flowchart illustrating the fragmentation process provided in an embodiment of this application;

[0059] Figure 6 A flowchart illustrating the data transmission process provided in an embodiment of this application;

[0060] Figure 7 A schematic diagram of a device for importing data into a database, provided in an embodiment of this application;

[0061] Figure 8 This is a schematic diagram of the composition structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0063] To facilitate understanding of the technical solutions provided in the embodiments of this application, some key terms used in the embodiments of this application will be explained below:

[0064] Derivative: refers to the process of importing data into a database, and is often simply called derivative.

[0065] A Distributed File System (DFS) is a file system that manages physical storage resources that are not necessarily directly connected to local nodes, but rather connected to nodes through a computer network; or it can be a complete hierarchical file system formed by combining several different logical disk partitions or volume labels. DFS provides a logical tree-like file system structure for resources distributed anywhere on the network, making it easier for users to access shared files distributed across the network. A single DFS shared folder serves as an access point relative to other shared folders on the network. A typical DFS is HDFS (Hadoop Distributed File System).

[0066] Spark is a general-purpose in-memory parallel computing framework developed by the AMP (Algorithms Machines People) lab at the University of California, Berkeley.

[0067] Structured Query Language (SQL) is a special-purpose programming language, a database query and programming language used to access, query, update, and manage relational database systems.

[0068] Hive is a data warehouse tool based on Hadoop, used for data extraction, transformation, and loading. It is a SQL component that can store, query, and analyze large-scale data stored in Hadoop.

[0069] Sqoop is an open-source tool primarily used for transferring data between Hadoop and traditional databases.

[0070] Parquet files are a columnar storage file format commonly used in big data components such as Pig, Spark, and Hive. Their file extension is .parquet.

[0071] A hash function, also known as a hash algorithm, transforms an input of arbitrary length into a fixed-length output, called a hash value. This transformation is a compression mapping; that is, the space of hash values ​​is usually much smaller than the space of inputs. Different inputs may hash to the same output, so it is impossible to determine a unique input value from the hash value. Simply put, it's a function that compresses a message of arbitrary length into a message digest of a fixed length. This calculation function can map strings or other types of data that are difficult to compare into an integer.

[0072] Resumable download: This refers to dividing the download or upload task (a file or a compressed package) into several parts during the download or upload process. Each part is uploaded or downloaded using a separate thread. If a network failure occurs, the download or upload can resume from the part that has already been uploaded or downloaded, instead of having to start the download and upload process from the beginning.

[0073] JavaScript Object Notation (JSON) is a lightweight data interchange format.

[0074] The following is a brief description of the technical solutions of the embodiments of this application.

[0075] Currently, when data results computed using big data technology stacks such as Hive or Spark are supplied to downstream systems, the derived methods used do not support breakpoint retransmission. Therefore, in order to ensure data integrity, all data must be deleted before importing from the beginning, which wastes a lot of time and makes the import efficiency extremely low.

[0076] Based on this, this application provides a method for importing data into a database. In this method, the target data file to be transmitted is segmented, with each segment treated as a transmission task. When executing a transmission task, the task offset is used to locate the data reading position within the target segment file. Based on the data read at that position, a batch of database import instructions is generated and sent to the target database to import the data. Simultaneously, the task offset is updated to identify which data has been transmitted and thus locate subsequent data reading positions. By recording the task offset, if a transmission task fails or the entire database import task fails, the data reading position for retransmission can be located based on the task offset, enabling data retransmission from a breakpoint. This avoids the need to delete data and re-execute transmission when a task fails, improving the efficiency of data import.

[0077] Furthermore, the derivative process in this application embodiment is developed based on the Spark big data technology stack, and can also realize parallel and rapid import of data into the target database according to the user's configuration, which further improves the efficiency of data import and reduces the time required for data import.

[0078] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application are applicable. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0079] The solution provided in this application's embodiments can be applied to derivative scenarios. For example... Figure 1 The diagram shown is an application scenario provided by an embodiment of this application. In this scenario, a data source device 101, a derivative device 102, and a target database 103 may be included.

[0080] Data source device 101 is a device that provides data sources. It can be a database that uses any storage method to store the data results calculated by big data technology stacks such as Hive or Spark. For example, it can be a local storage database of a big data computing cluster, or it can be a dedicated database for storing data results.

[0081] The derivative device 102 is a device for implementing the derivative process in the embodiments of this application. It can be a terminal device capable of implementing the derivative process, or a server device. For example, it can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, i.e., content delivery network (CDN), and big data and artificial intelligence platforms, but it is not limited to these.

[0082] In one possible implementation, the data source device 101 and the derivative device 102 can be completely different devices, in which case the data source device 101 is an external data source of the derivative device 102.

[0083] In one possible implementation, the data source device 101 and the derivative device 102 can be implemented by the same device. That is, the derivative device 102 itself is a distributed cluster that can realize distributed big data computing. The computing results are provided by itself. In this case, the derivative device itself is a data source, and the data source device 101 can be considered an internal data source of the derivative device 102.

[0084] The target database 103 is the target database for data import. It can be a database with any possible data structure, used to store the calculation results of various big data components and supply the calculation results to downstream systems for use.

[0085] The data derivation device 102 may include one or more processors, memory, and I / O interfaces for interaction. Furthermore, the data derivation device 102 may be configured with a database to store task execution parameters and task status data required during the derivation process. The memory of the data derivation device 102 may also store the program instructions required for execution in the data import method provided in this embodiment of the application. These program instructions, when executed by the processor, can be used to implement the data import process provided in this embodiment of the application.

[0086] In practical applications, if a database import command is initiated, the data import device 102 can obtain the target data file to be transmitted from the data source device 101. It then segments the target data file into chunks, treating each chunk as a transmission task. During the execution of each transmission task, the task offset is used to locate the data reading position within the target chunk. Based on the data read at that position, a batch of database import commands is generated and sent to the target database to import the data. Simultaneously, the task offset is updated to identify which data has been transmitted and locate subsequent data reading positions. By recording the task offset, if a transmission task fails or the entire database import task fails, the task offset can be used to locate the data reading position for retransmission, enabling breakpoint retransmission. This avoids the need to delete data and re-execute the transmission when a task fails, improving the efficiency of data import.

[0087] In this embodiment, the data source device 101, the data derivative device 102, and the target database 103 can be directly or indirectly connected via one or more networks. This network can be a wired network or a wireless network; for example, a wireless network can be a mobile cellular network or a Wireless-Fidelity (WIFI) network, or any other possible network. This embodiment does not limit the types of networks used.

[0088] like Figure 2 The diagram shown is a system architecture diagram for importing data into a database according to an embodiment of this application. In this architecture, a data source, a data import tool, and a target database may be included. The data import tool mainly consists of a task data module, a data source reading module, a data sharding module, and a data parallel transmission module.

[0089] (1) Task Data Module

[0090] The task data module is used to store the task execution parameters and task status data during the database import task. The task execution parameters may include data source information, target database information, parallelism, sharding algorithm, and data sharding directory, etc. The task status data may include the processing status of each stage and the task offset of each shard file, etc.

[0091] The task data module runs through the entire lifecycle of the database import task. It records the stages of the database import task and the status and progress of each stage. In particular, during the parallel data transmission stage, it records the task offset of each file segment. This is the basis for implementing breakpoint resume. When a task fails and is retried, execution can be resumed based on the relevant task data of each transmission task.

[0092] (2) Data source reading module

[0093] The data source reading module uses big data technologies such as Spark to read data from data sources and save it to HDFS. It supports both internal data sources (i.e., data sources located in the big data cluster where the data transmission task resides) and external data sources. Internal data sources include Hive tables and HDFS files in Parquet file format with schemas. External data sources include various relational database tables. The data source reading module also supports executing custom SQL operations for related data processing.

[0094] The data source reading module is upstream of the data sharding module; it passes the read data to the data sharding module.

[0095] (3) Data sharding module

[0096] The data sharding module receives data from the data source reading module, divides it into shard files according to the sharding algorithm, and supplies them to the data transmission module for parallel transmission.

[0097] Specifically, taking the source data storage path as ds_HDFS_path as an example, this module reads data from ds_HDFS_path and stores the data on different shards through a sharder, dividing the data into a specified number of files through the above process.

[0098] (4) Data Parallel Transmission Module

[0099] The parallel data transmission module is used to read the fragmented data files according to parameters such as parallelism and file format, and then write the data to the target database. During the writing process, it records the offset or completion status of each fragment file.

[0100] Specifically, the parallel data transmission module transmits the already fragmented data to the specified target database table. It generates a fragment file transmission task pool based on the specified parallelism, which can simultaneously transmit multiple fragment files in batches (batch_size) to the target database table. Generally, the parallelism cannot exceed the number of fragment files. After each batch is successfully submitted, the task offset of the fragment file in the task data module must be updated first so that if the task fails, the breakpoint execution can continue from the task offset. After the fragment file transmission is completed, the fragment file transmission status is updated to success. After all fragment files are successfully transmitted, the task status is updated to success.

[0101] The following describes a method for importing data into a database provided by an exemplary embodiment of this application, in conjunction with the application scenarios and system architecture described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0102] See Figure 3 The diagram shown is a flowchart illustrating a method for importing data into a database according to an embodiment of this application. The specific implementation flow of this method is as follows:

[0103] Step 301: Based on the preset source data storage path, read the target data file to be imported.

[0104] In this embodiment of the application, the process of step 301 can be achieved through... Figure 2 The data source reading module shown is used for execution. Figure 2 The task data module shown can store source data storage path information. When importing data into the database, the source data storage path information can be read from the task data module to obtain the source data storage path. Then, the various data files to be imported can be read from the storage space corresponding to the source data storage path.

[0105] The data import process can be performed after big data computation is completed. After computation, the results can be stored in a pre-defined source data storage path, allowing data to be retrieved from that path for import into the database. Alternatively, it can be performed simultaneously with big data computation. In this case, the source data storage path can be used as the storage path for the computation results, which will then be saved there. Alternatively, during big data computation, the results can be migrated to the source data storage path concurrently.

[0106] In one possible implementation, the data reading process can be carried out through, for example... Figure 4 The process shown can be executed before or simultaneously with step 301; this embodiment does not impose any limitations on this. See also Figure 4 The diagram shown is a flowchart illustrating the data reading process provided in an embodiment of this application.

[0107] Step 401: In response to the database import operation, obtain the task execution parameters required for this database import task. The task execution parameters include data source information and source data operation information.

[0108] In one possible implementation, the database import operation can be performed after the big data calculation is completed, by notifying the user that the big data calculation process is complete. Based on this completion indication, the user can confirm that the data needs to be imported into the target database and initiate a database import operation for these calculation results. At the same time as performing this operation, relevant task running parameters can also be configured.

[0109] In one possible implementation, the database import operation can also be performed before big data computation. That is, when configuring big data computation, the configuration is made to import the computation results into the target database after the computation is completed. In this case, the data import operation can also be regarded as the operation of initiating big data computation.

[0110] Of course, database import operations can also be initiated in other situations, and this application embodiment does not limit this.

[0111] In this embodiment, the task execution parameters are configured specifically for this database import task and are used to indicate the relevant task parameters for this database import task, including but not limited to the following parameters:

[0112] (1) Task information, such as task name, task status, etc.

[0113] (2) Data source information, which indicates information related to the data source, such as the data source type, source data type, and source data storage path. For example, the ds_HDFS_path field indicates the source data storage path.

[0114] (3) Target database information, which indicates information related to the target database to be imported, such as database type, data type, database storage path, database import command, data source type, etc. Different data source types have different data source attribute information.

[0115] (4) Parallelism, which indicates the degree of concurrency of the transmission task when the database import task is executed. For example, when the parallelism is set to 10, the transmission task can be executed by 10 data transmission threads respectively, thereby improving the execution efficiency of the task.

[0116] (5) Sharding algorithm, used to indicate the sharding algorithm used in sharding processing.

[0117] (6) Data fragment directory, which indicates the storage directory of the fragment files obtained from the fragmentation process.

[0118] In the embodiments of this application, the task data can be described using any possible data structure, such as JSON structure.

[0119] Step 402: Based on the data source information, determine the data source type of the data source to be imported.

[0120] Step 403: If the data source type is an internal data source, then load the internal data source information.

[0121] In this embodiment, when the data source is of different types, the execution logic can be different, and the parameter configuration can also be different accordingly. Therefore, in practical applications, in order to avoid loading too many parameters, after determining the data source type, the corresponding data source information can be selected to be loaded according to the data source type.

[0122] Step 404: Based on the source data operation information, determine whether database operations are required on the source data.

[0123] In this embodiment of the application, during the data import process, database operations on the data are also supported. For example, a predefined SQL statement can be used, and during data import, this SQL statement can be executed on the data to perform the corresponding data processing. Therefore, before placing the data into the specified source data storage path, it is necessary to determine whether an SQL statement has been specified.

[0124] Step 405: If the result of step 404 is negative, that is, it is determined that no database operation is required for the source data, then update the storage path of the internal data source to the storage path of the source data.

[0125] Specifically, if no SQL is specified, no SQL needs to be executed. Since the data source is internal, data can be read directly. To reduce the data migration process, the storage path of the internal data source can be updated to the source data storage path, allowing data to be read directly from the internal data source. For example, the `ds_HDFS_path` field can be updated to the storage path of the internal data source.

[0126] Step 406: If the determination result of step 404 is yes, that is, it is determined that database operations need to be performed on the source data, then after reading the source data from the internal data source and performing the database operations, the source data after the operations is stored in the internal data source, and the storage path of the internal data source is updated to the source data storage path.

[0127] Specifically, if a specific SQL statement is specified, the SQL statement needs to be executed before the result is saved to the internal data source, and then the storage path of the internal data source is updated to the source data storage path. Alternatively, a new storage path can be created in the internal storage to store the data after the SQL statement is executed; this storage path then serves as the source data storage path. For example, the data after the SQL statement is executed can be saved to the path indicated by ds_HDFS_path.

[0128] Step 407: If the data source type is an external data source, then load the external data source information.

[0129] Step 408: Based on the source data operation information, determine whether database operations are required on the source data.

[0130] Step 409: If the result of step 408 is negative, that is, it is determined that no database operation is required for the source data, then the source data is read from the external data source and stored in the source data storage path.

[0131] Similarly, if no specific SQL operation needs to be executed, the read data can be directly saved to the path indicated by ds_HDFS_path.

[0132] Step 410: If the determination result of step 408 is yes, that is, it is determined that no database operation is required for the source data, then after reading the source data from the external data source and performing the database operation, the source data after the operation is stored in the source data storage path.

[0133] Similarly, if a specific SQL operation needs to be executed, the data will be saved to the path indicated by ds_HDFS_path after the SQL is executed.

[0134] In this embodiment, when storing source data, a distributed file system such as HDFS can be used. That is, the source data storage path can be a storage path within HDFS. Then, when reading data, a big data computing engine can read source data from an external data source and store it in HDFS. For example, tools such as Spark or Sqoop can be used to read data files from a data source and store them in HDFS, for example, in the ds_HDFS_path of HDFS. During the database import process, the various data files to be imported can be read from ds_HDFS_path.

[0135] Before performing the import task, you need to configure the relevant parameters for the data source reading module. These configuration parameters can include one or more of the following combinations:

[0136] (1) Data source type: This can be used to determine the data source type. Different data source types have different data source attribute information.

[0137] (2) The instruction information indicating whether to execute the specified SQL can be used to determine whether to specify SQL.

[0138] (3) Source data storage path, which is used to store the data files that need to be imported.

[0139] Please continue reading below. Figure 3 .

[0140] Step 302: Perform fragmentation on the target data file and update the transmission task list based on the obtained fragmented files; wherein, in the transmission task list, each transmission task corresponds to one fragmented file.

[0141] In this embodiment of the application, step 302 may be... Figure 2 The data sharding module shown is used for execution.

[0142] To enhance the parallelism of data import during the database import process, this embodiment divides a single data file into multiple fragments. These fragments can then be transmitted simultaneously, improving data import efficiency. Specifically, after reading the target data file from the source data storage path, a data splitter can store the data contained in the target data file across different fragments, thus dividing the target data file into a specified number of fragments for parallel transmission by subsequent data transmission modules.

[0143] Specifically, as mentioned above, the task execution parameters can include the sharding algorithm required for sharding processing. Additionally, a base field for sharding processing can be specified, meaning that sharding will be based on this field during the sharding process. See [link to relevant documentation]. Figure 5 The diagram shown is a flowchart illustrating the fragmentation process provided in an embodiment of this application.

[0144] Step 501: Read the target data file from the source data storage path.

[0145] Step 502: Determine whether to specify the baseline fields required for fragmentation processing.

[0146] Before performing the import task, you need to configure the relevant parameters for the data sharding module. These parameters can all be stored in the task data module. The configuration parameters can include one or more of the following parameters in combination:

[0147] (1) The fields on which the sharding depends determine whether a base field is specified. If it is specified, the specified base field can be read from the field. If it is not configured, the default value can be used.

[0148] (2) Number of fragments: indicates that a data file needs to be split into multiple fragment data files.

[0149] (3) Directory for storing fragmented data files, used to indicate the directory where the fragmented data files are stored.

[0150] Step 503: If the result of step 502 is yes, that is, the reference field required for sharding is specified, then the sharding reference value is determined based on the value of the reference field in the target data file.

[0151] In one possible implementation, the sharding reference value can be a hash value. In this case, a hash calculation is performed on the value of the base field in the target data file to obtain the hash value of the base field, which is then used as the sharding reference value.

[0152] Of course, other digest algorithms can also be used to obtain the fragmentation reference value, and this application embodiment does not limit this.

[0153] Step 504: If the result of step 502 is no, that is, no baseline field required for sharding is specified, then the sharding reference value is determined based on the value of the primary key in the target data file.

[0154] For example, if the first column is the primary key, then the first column field is used as the sharding field. The values ​​of the first column field are hashed, and the resulting hash value is used as the sharding reference value.

[0155] Step 505: Based on the fragmentation reference value and the specified number of fragments, determine the fragmentation point when the target data file is fragmented.

[0156] Specifically, since the length of the sharding reference value is fixed, the sharding reference value can be divided into multiple segments according to the specified number of shards. The dividing point between two segments is the sharding division point. Each segment can correspond to a part of the target data file, thereby achieving the purpose of sharding.

[0157] In one possible implementation, the fragmentation process can be performed by taking a modulo, that is, taking the modulo between the fragmentation reference value and the specified number of fragments to obtain the fragment to which each segment of data in the target data file belongs.

[0158] Step 506: Based on the partitioning points, the target data file is partitioned to obtain multiple partitioned files.

[0159] Specifically, based on the partitioning points, it can be determined which data belong to the same partition. These data are then saved to the same partition file, thus obtaining the partition files corresponding to the target data file. Based on each partition file, corresponding transmission tasks are generated and updated in the transmission task list.

[0160] In practical applications, a fragment file directory and a transmission task list can be configured. The fragment files obtained after fragmentation are stored in the fragment file directory. Correspondingly, when transmission is performed, the fragment file directory is accessed to generate corresponding transmission tasks for each fragment file and add them to the transmission task list.

[0161] Please continue reading below. Figure 3 .

[0162] Step 303: Iterate through the transfer task list until it is empty; during each iteration, read one transfer task and import the target fragment files corresponding to the transfer task into the target database in batches. The import process for each batch includes the following steps:

[0163] S3031: Based on the corresponding task offset, locate the data reading position in the target fragment file, and generate the database import instruction for this batch based on the data read from the data reading position; wherein, the task offset is used to indicate the data that has been imported into the target fragment file.

[0164] S3032: Send the generated database import command to the target database and update the task offset.

[0165] Specifically, to improve data import efficiency, fragmented files can be transferred in parallel. The transfer tasks to be executed can be stored in a task execution pool. When a corresponding transfer thread is idle, it can retrieve a transfer task from the task execution pool and execute it. Therefore, for each transfer task in the transfer task list, one transfer task is read during each iteration and added to the task execution pool. Then, idle target processes from multiple parallel transfer threads can read the transfer tasks to be executed from the task execution pool and import the target fragmented files corresponding to the transfer tasks into the target database in batches.

[0166] Before performing the import task, it is necessary to configure the relevant parameters of the data parallel transmission module. These parameters can also be stored uniformly in the task data module. The configuration parameters can include one or more of the following parameters in combination:

[0167] (1) Parallelism, used to indicate the degree of parallelism during data transmission. For example, if the parallelism value is configured to 3, data can be imported simultaneously through 3 transmission threads.

[0168] (2) Batch size (batch_size) indicates the number of instruction records transmitted in each batch. For example, setting batch_size to 1000 means that a database transaction will be committed once the number of instruction records in each batch reaches 1000.

[0169] Each transmission task stores its own status data. These parameters can also be stored uniformly in the task data module. As shown below, the status data can take the form of a list of fragmented file transmission tasks. This list can include the fragmented data file path of each fragmented data file, the task offset of the fragmented data file to indicate the offset that the fragmented data file has been transmitted, and the transmission status of the fragmented data file.

[0170] Each transmission task added to the transmission task list will be configured with the aforementioned status data. The status data of each transmission task indicates the file path, task offset, and transmission status of the segment file targeted by the transmission task. When no transmission is initially performed, the task offset is zero and is updated according to the actual transmission progress. In this way, even if the import task encounters an error and needs to be transmitted again, it is not necessary to start the transmission from the beginning. Instead, the transmission can be continued by locating the already transmitted data position based on the task offset. When all the data of the segment file is successfully imported, the transmission status is set to success.

[0171] In this embodiment, the target data file to be transmitted is segmented, with each segment treated as a transmission task. During the execution of each transmission task, the task offset is used to locate the data reading position within the target segment. Based on the data read at that position, a batch of database import instructions is generated and sent to the target database to import the data. Simultaneously, the task offset is updated to identify which data has been transmitted and thus locate subsequent data reading positions. By recording the task offset, if a transmission task fails or the entire database import task fails, the data reading position for retransmission can be located based on the task offset, enabling breakpoint retransmission. This avoids the need to delete data and re-execute transmission when a task fails, improving data import efficiency.

[0172] The process in step 303 can be as follows: Figure 2 The parallel transmission module shown is used to perform this.

[0173] The following section provides a detailed description of the data parallel transmission module's transmission process. (See also...) Figure 6 The diagram shown is a flowchart illustrating the data transmission process provided in an embodiment of this application.

[0174] Step 601: Load task data information.

[0175] Step 602: Confirm whether this database import task is being run for the first time.

[0176] Specifically, the distinction between first run and non-first run is whether this database import task is being executed for the first time. If the database import task fails and is then attempted again, it is considered a non-first run.

[0177] Step 603: If the result of step 602 is yes, that is, this database import task is being run for the first time, then traverse the sharded file directory, generate a transfer task list, and update the task status data.

[0178] Specifically, when this database import task is being run for the first time, a corresponding transfer task list needs to be created. That is, based on the fragment file directories corresponding to each fragment file obtained after the above fragment processing, the corresponding p_file_tks are generated and the task status data is updated.

[0179] Step 604: If the result of step 602 is negative, or if the transmission task list has been generated, then incomplete transmission tasks can be read from the transmission task list.

[0180] In this embodiment, during the execution of the database import task, task failure may occur. If a task failure is detected, a retransmission attempt can be automatically initiated. Alternatively, a task failure indication can be output to the user, who can then choose whether to retransmit based on the indication. If retransmission is required, a retransmission operation is triggered. Correspondingly, the device can respond to the retransmission indication initiated in response to the task failure indication. In this case, since the task has not been executed for the first time, the device can continue to read the transmission task from the transmission task list, locate the data reading position in the target fragment file based on the task offset corresponding to the transmission task, and continue to execute the transmission task based on the data reading position.

[0181] Step 605: Place the read transfer task into the task execution pool so that it can be executed by each transfer thread to import the corresponding file segments.

[0182] Step 606: When a transmission thread executes a transmission task, it will locate the data reading position in the corresponding target fragment file according to the task offset of the transmission task.

[0183] Step 607: Determine if the data reading position is at the end of the target fragment file.

[0184] Step 608: If the result of step 607 is negative, that is, the data reading position is not the end, then read the data sequentially based on the data reading position, and generate a database import instruction for this batch based on the data read each time.

[0185] Specifically, the process of importing a database is essentially inserting data into the target database. In this case, the database import command can insert statements into the database tables of the target database.

[0186] Step 609: Determine whether the number of generated database import commands has reached the maximum number for this batch.

[0187] Step 610: If the result of step 609 is yes, then send the generated database import command to the target database and update the task offset.

[0188] Specifically, based on the generated database import instructions, a database transaction is packaged and generated, and then the values ​​are committed to the target database.

[0189] If the result of step 609 is negative, then proceed to step 608 to continue generating database import commands until the number of generated database import commands reaches the maximum limit for this batch.

[0190] Step 611: If the result of step 607 is yes, that is, when it is the end, determine whether there are any unfinished transmission tasks in the updated transmission task list.

[0191] If the result of step 611 is yes, that is, there are incomplete transmission tasks, then proceed to step 604 to continue reading new transmission tasks and adding them to the task execution pool.

[0192] Step 612: If the result of step 611 is negative, that is, all transfer tasks have been completed, then update the task status of this database import task to a successful status, and the database import task ends.

[0193] In summary, this application provides a data import scheme for large-scale data import into relational database tables based on the Spark big data technology stack. It can realize parallel and rapid data import into the target database according to the user's configuration, and supports resuming interrupted downloads after failure. It greatly improves the efficiency of retrying when importing large amounts of data into the database. Through parameterized configuration, the simplified data import process shields the complex big data and distributed technologies, and can quickly respond to business needs.

[0194] Please see Figure 7 Based on the same inventive concept, embodiments of this application also provide an apparatus 70 for importing data into a database, the apparatus comprising:

[0195] The data reading unit 701 is used to read the target data file to be imported based on the preset source data storage path;

[0196] The fragmentation processing unit 702 is used to fragment the target data file and update the transmission task list based on the obtained multiple fragmented files; wherein, in the transmission task list, each transmission task corresponds to one fragmented file;

[0197] The parallel transmission unit 703 is used to iterate through the transmission task list until the list is empty. During each iteration, it reads one transmission task and imports the corresponding target fragment files into the target database in batches. Each batch import process includes:

[0198] Based on the corresponding task offset, the data reading position in the target shard file is located, and the database import instructions for this batch are generated based on the data read from the data reading position; wherein, the task offset is used to indicate the data that has been imported into the target shard file;

[0199] Send the generated database import command to the target database and update the task offset.

[0200] In one possible implementation, the data reading unit 701 is further configured to:

[0201] In response to the database import operation, obtain the task execution parameters required for this database import task. The task execution parameters include data source information and source data operation information.

[0202] Based on the data source information, determine the data source type of the data source to be imported;

[0203] If the data source type is an internal data source, and it is determined based on the source data operation information that no database operation is required for the source data, then the storage path of the internal data source is updated to the source data storage path.

[0204] If the data source type is an external data source, and it is determined based on the source data operation information that no database operation is required on the source data, then the source data is read from the external data source and stored in the source data storage path.

[0205] In one possible implementation, the data reading unit 701 is further configured to:

[0206] If it is an internal data source, and it is determined based on the source data operation information that database operations need to be performed on the source data, then after reading the source data from the internal data source and performing the database operations, the source data after the operations is stored in the internal data source, and the storage path of the internal data source is updated to the source data storage path.

[0207] If the data source type is an external data source, and it is determined based on the source data operation information that no database operation is required on the source data, then after reading the source data from the external data source and performing the database operation, the source data after the operation is stored in the source data storage path.

[0208] In one possible implementation, the source data storage path indicates the storage path of the distributed file system; then the data reading unit 701 is specifically used for:

[0209] The big data computing engine reads source data from external data sources and stores it in a distributed file system.

[0210] In one possible implementation, the fragmentation processing unit 702 is specifically used for:

[0211] If a base field required for sharding is specified, the sharding reference value is determined based on the value of the base field in the target data file.

[0212] If no base field is specified for sharding, the sharding reference value is determined based on the value of the primary key in the target data file.

[0213] Based on the fragmentation reference value and the specified number of fragments, determine the fragmentation point when the target data file is fragmented;

[0214] Based on the partitioning points, the target data file is partitioned into multiple partitioned files.

[0215] In one possible implementation, the parallel transmission unit 703 is specifically used for:

[0216] During each iteration, a transfer task is read and added to the task execution pool;

[0217] The idle target processes in multiple parallel transmission threads read the transmission tasks to be executed from the task execution pool and import the target fragment files corresponding to the transmission tasks into the target database in batches.

[0218] In one possible implementation, the parallel transmission unit 703 is specifically used for:

[0219] Based on the corresponding task offset, locate the data reading position in the target fragment file;

[0220] Determine if the data read position is at the end of the target fragment file;

[0221] If it is not the end, then the data is read sequentially based on the data reading position, and each time the data is read, a database import instruction for this batch is generated based on the read data;

[0222] The generated database import command will then be sent to the target database, including:

[0223] If the number of generated database import commands reaches the maximum limit for this batch, the generated database import commands will be sent to the target database.

[0224] In one possible implementation, the parallel transmission unit 703 is specifically used for:

[0225] If it is the end, check if there are any unfinished transmission tasks in the updated transmission task list;

[0226] If it does not exist, update the task status of this database import task to a success status;

[0227] If it exists, continue reading new transfer tasks and add them to the task execution pool.

[0228] In one possible implementation, the parallel transmission unit 703 is further configured to:

[0229] If this database import task fails, a task failure indication will be output.

[0230] In response to a retransmission instruction initiated in response to a task failure indication, the transmission task is read from the transmission task list.

[0231] Based on the task offset corresponding to the transmission task, locate the data reading position in the target fragment file, and continue to execute the transmission task based on the data reading position.

[0232] With the aforementioned device and by recording the task offset, when a transmission task fails or the entire database import task fails, the data reading position at the time of retransmission can be located based on the task offset, thereby realizing the data breakpoint retransmission function. This avoids the need to delete data and re-execute the transmission when the task fails, thus improving the efficiency of data import.

[0233] This device can be used to execute the methods shown in the various embodiments of this application. Therefore, the functions that each functional module of this device can achieve can be referred to the description of the foregoing embodiments, and will not be repeated here.

[0234] Please see Figure 8 Based on the same technical concept, embodiments of this application also provide a computer device. In one embodiment, the computer device can be... Figure 1 The server shown or Figure 2 The cloud-related device shown is a computer device such as... Figure 8 As shown, it includes a memory 801, a communication module 803, and one or more processors 802.

[0235] The memory 801 is used to store computer programs executed by the processor 802. The memory 801 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0236] Memory 801 may be volatile memory, such as random-access memory (RAM); memory 801 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 801 may be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 801 may be a combination of the above-described memories.

[0237] Processor 802 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 802 is used to implement the aforementioned method of importing data into the database when it calls a computer program stored in memory 801.

[0238] The communication module 803 is used to communicate with terminal devices and other servers.

[0239] This application embodiment does not limit the specific connection medium between the memory 801, communication module 803, and processor 802 described above. This application embodiment... Figure 8 The memory 801 and the processor 802 are connected via a bus 804, and the bus 804 is in Figure 8 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 804 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 8 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0240] The memory 801 stores a computer storage medium, which stores computer-executable instructions. The computer-executable instructions are used to implement the method of importing data into a database according to the embodiments of this application. The processor 802 is used to execute the method of importing data into a database according to the above embodiments.

[0241] Based on the same inventive concept, embodiments of this application also provide a storage medium storing a computer program that, when run on a computer, causes the computer to perform the steps of the method for importing data into a database according to various exemplary embodiments of this application described above.

[0242] In some possible implementations, various aspects of the method for importing data into a database provided in this application can also be implemented in the form of a computer program product, which includes a computer program that, when run on a computer device, causes the computer device to perform the steps in the method for importing data into a database according to various exemplary embodiments of this application as described above. For example, the computer device can perform the steps of the various embodiments.

[0243] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0244] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on a computer device. However, the program product of this application is not limited thereto. In this application, the readable storage medium may be any tangible medium that contains or stores a program, and the computer program included therein may be used by or in conjunction with a command execution system, apparatus, or device.

[0245] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0246] Computer programs contained on readable media may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0247] Computer programs for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages.

[0248] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0249] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0250] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0251] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0252] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for importing data into a database, characterized in that, The method includes: Based on the preset source data storage path, read the target data file to be imported; If a reference field required for sharding is specified, then the sharding reference value is determined based on the value of the reference field in the target data file. If no baseline field is specified for sharding, the sharding reference value is determined based on the value of the primary key in the target data file. Based on the fragmentation reference value and the specified number of fragments, determine the fragmentation point when the target data file is fragmented; Based on the partitioning points, the target data file is partitioned to obtain multiple partitioned files. Based on the obtained multiple fragment files, the transmission task list is updated; wherein, in the transmission task list, each transmission task corresponds to one fragment file; The transmission task list is iterated over until it is empty; during each iteration, one transmission task is read and the target fragment files corresponding to the transmission task are imported into the target database in batches. The import process for each batch includes: Based on the corresponding task offset, the data reading position in the target shard file is located, and based on the data read at the data reading position, a database import instruction for this batch is generated; wherein, the task offset is used to indicate the data that has been imported into the target shard file; The generated database import command is sent to the target database, and the task offset is updated.

2. The method as described in claim 1, characterized in that, Before reading the target data file to be imported based on the preset source data storage path, the method further includes: In response to the database import operation, the task execution parameters required for this database import task are obtained. The task execution parameters include data source information and source data operation information. Based on the data source information, determine the data source type of the data source to be imported; If the data source type is an internal data source, and it is determined based on the source data operation information that no database operation is required for the source data, then the storage path of the internal data source is updated to the source data storage path. If the data source type is an external data source, and it is determined based on the source data operation information that no database operation is required for the source data, then the source data is read from the external data source and stored in the source data storage path.

3. The method as described in claim 2, characterized in that, After determining the data source type of the data source to be imported based on the data source information, the method further includes: If it is an internal data source, and it is determined based on the source data operation information that a database operation needs to be performed on the source data, then after reading the source data from the internal data source and performing the database operation, the source data after the operation is stored in the internal data source, and the storage path of the internal data source is updated to the source data storage path. If the data source type is an external data source, and it is determined based on the source data operation information that no database operation is required for the source data, then after reading the source data from the external data source and performing the database operation, the source data after the operation is stored in the source data storage path.

4. The method as described in claim 2, characterized in that, The source data storage path indicates the storage path of the distributed file system; The step of reading source data from an external data source and storing it in the source data storage path includes: The big data computing engine reads source data from the external data source and stores it in the distributed file system.

5. The method according to any one of claims 1 to 4, characterized in that, During each iteration, a transfer task is read, and the target fragment files corresponding to the transfer task are imported into the target database in batches, including: During each iteration, a transfer task is read and added to the task execution pool; The idle target processes in multiple parallel transmission threads read the transmission tasks to be executed from the task execution pool, and import the target fragment files corresponding to the transmission tasks into the target database in batches.

6. The method according to any one of claims 1 to 4, characterized in that, Based on the corresponding task offset, the data read position in the target fragment file is located, and based on the data read from the data read position, a database import instruction for this batch is generated, including: Based on the corresponding task offset, locate the data reading position in the target fragment file; Determine whether the data reading position is the end of the target fragment file; If it is not the end, then data is read sequentially based on the data reading position, and each time data is read, a database import instruction for this batch is generated based on the read data; The generated database import command will then be sent to the target database, including: If the number of generated database import commands reaches the maximum number for this batch, the generated database import commands will be sent to the target database.

7. The method as described in claim 6, characterized in that, After determining whether the data read position is the end of the target fragment file, the method further includes: If it is the end, determine whether there are any unfinished transmission tasks in the updated transmission task list; If it does not exist, update the task status of this database import task to a success status; If it exists, continue reading new transmission tasks and add them to the task execution pool.

8. The method according to any one of claims 1 to 5, characterized in that, After sending the generated database import command to the target database and updating the task offset, the method further includes: If this database import task fails, a task failure indication will be output. In response to a retransmission instruction initiated in response to the task failure indication, the transmission task is read again from the transmission task list; Based on the task offset corresponding to the transmission task, the data reading position in the target fragment file is located, and the transmission task continues to be executed based on the data reading position.

9. An apparatus for importing data into a database, characterized in that, The device includes: The data reading unit is used to read the target data file to be imported based on the preset source data storage path; The fragmentation processing unit is configured to: if a base field required for fragmentation processing is specified, determine a fragmentation reference value based on the value of the base field in the target data file; if no base field required for fragmentation processing is specified, determine the fragmentation reference value based on the value of the primary key in the target data file; determine the fragmentation partitioning point for fragmentation processing of the target data file based on the fragmentation reference value and the specified number of fragments; perform fragmentation processing on the target data file based on the fragmentation partitioning point to obtain multiple fragmented files; and update the transmission task list based on the obtained multiple fragmented files; wherein, in the transmission task list, each transmission task corresponds to one fragmented file. A parallel transmission unit is used to iterate through the transmission task list until the list is empty; wherein, during each iteration, one transmission task is read and the target fragment files corresponding to the transmission task are imported into the target database in batches, and the import process of each batch includes: Based on the corresponding task offset, the data reading position in the target shard file is located, and based on the data read at the data reading position, a database import instruction for this batch is generated; wherein, the task offset is used to indicate the data that has been imported into the target shard file; The generated database import command is sent to the target database, and the task offset is updated.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

11. A computer storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the steps of the method according to any one of claims 1 to 8.

Images