Fault-tolerant method and system for data import based on distributed database

Abstract

The application discloses a fault-tolerant method and system for realizing data import based on a distributed database, belongs to the technical field of database data migration, and aims to solve the technical problem of how to realize the support for setting the fault-tolerant row number during the data import of a database, save the resource consumption of a user, and improve the migration efficiency of the user. The technical scheme is as follows: when a user executes a data import command in a distributed database, the user sets the number of error data rows in a data file allowed in this import; the database starts a thread for reading the data file, a thread for processing error data, and a channel for transmitting error data; the data file is read, and whether the data in the data file meets the requirements is judged during data conversion; when the thread for processing error data receives error data sent by the thread for reading the data file, whether the current counter count exceeds the fault-tolerant row number set by the user is judged, and the error data is recorded into a fixed file.

Description

Technical Field

[0001] This invention relates to the field of database migration technology, specifically a fault-tolerant method and system for data import based on a distributed database. Background Technology

[0002] In distributed databases, the ability to support fault tolerance during data import is a crucial criterion for evaluating the efficiency and ease of use during database migration. Current data import technologies suffer from the following drawbacks:

[0003] 1) Fault tolerance for data files is not supported during data import:

[0004] The system does not handle potential errors in the data file during import. If an error exists in the data file, the entire import will fail. If only one line of data in the user's file has a problem, the entire import job will fail, and the already imported data will need to be rolled back. This undoubtedly consumes a significant amount of manpower, machine resources, and time. It is clearly undesirable for users to see hundreds of millions of rows of data fail to import due to a single line of data.

[0005] 2) No data checks are performed during import:

[0006] During data import, the data files were not checked beforehand. Although all the data can be imported, some or all of it may become unusable during use. While this won't consume significant resources for data rollback, it creates a potential vulnerability for the database's supported business operations, potentially leading to online service unavailability.

[0007] Data import typically involves writing massive amounts of data, and the entire import process can take a long time. If the entire import fails and is rolled back due to minor corruption of user data, the manpower and time costs are enormous. Users would prefer that the correct data be imported correctly and that they can see how many rows of data are corrupted and what the specific errors are, so that they can correct the data file and proceed with the next import.

[0008] Therefore, how to support setting the number of fault-tolerant rows during database data import, so as to save users' resource consumption and improve users' migration efficiency, is a technical problem that urgently needs to be solved. Summary of the Invention

[0009] The technical objective of this invention is to provide a fault-tolerant method and system for data import based on a distributed database, in order to solve the problem of how to support setting the number of fault-tolerant rows during data import from a database, thereby saving user resources and improving user migration efficiency.

[0010] The technical objective of this invention is achieved as follows: a fault-tolerant method for data import based on a distributed database, the specific method of which is as follows:

[0011] When a user executes a data import command in a distributed database, they can set the number of erroneous rows in the data file that are allowed to be imported, i.e., set the number of rows to be allowed to be correct.

[0012] The database starts several threads for reading data files and one thread for handling error data, and opens a channel for transmitting error data;

[0013] Read the data file and, during data transformation, determine whether the data in the file meets the requirements:

[0014] If the condition is not met, the erroneous data is transmitted into the channel, and the next line of data is read.

[0015] When the thread that handles error data receives error data from the thread that reads the data file, it checks whether the current counter count exceeds the number of error-tolerant lines set by the user, and then records the error data to a fixed file.

[0016] As a preferred option, when executing a data import command in a distributed database, the user can set the number of data rows in the data file that are allowed to contain errors during the import, as follows:

[0017] The database import command is parsed, and it is determined whether the user has provided a number of rows for error tolerance.

[0018] If so, it means that the fault tolerance function is enabled;

[0019] The database checks the validity of the number of rows entered by the user for error tolerance: only natural numbers greater than zero are allowed; otherwise, an error is reported and the user is prompted that the number of rows entered for error tolerance is invalid.

[0020] Preferably, the database starts several threads for reading data files and one thread for handling error data, and opens a channel for transmitting error data, as follows:

[0021] The database starts several independent threads to read the data file input by the user;

[0022] The database starts a thread to log erroneous data in the data files;

[0023] A many-to-one channel is opened. Each thread reading the data file sends a message containing the current error data to the channel when it encounters erroneous data.

[0024] As a preferred method, when reading the data file and performing data conversion, it is determined whether the data in the data file meets the requirements, specifically as follows:

[0025] Read file data line by line, and compare the number of columns in the database table with the number of fields in the user data based on the table's metadata.

[0026] If they are not equal, the data in that row is considered to be erroneous, and the erroneous data is sent to the channel to continue processing the next row of data.

[0027] If the data passes the field count validation, then each field in the read row of data is converted sequentially according to the column type based on the metadata of the table in the database.

[0028] If any field fails to be parsed or converted, the data in that row is considered erroneous, and the erroneous data is sent to the channel to continue processing the next row.

[0029] More preferably, when the thread handling erroneous data receives erroneous data from the thread reading the data file, it checks whether the current counter count exceeds the user-defined number of error-tolerant lines, and then records the erroneous data to a fixed file, as follows:

[0030] When the thread that handles error data receives error data from the file reading thread, it first updates its own counter.

[0031] Compare the value in the counter with the value passed in by the user:

[0032] If the value in the counter is less than the value passed by the user, it indicates that the fault tolerance process continues to accept error data, writes the error data in the current message to a fixed file, and continues to process the next line of error data;

[0033] If the value in the counter is greater than the value passed by the user, it indicates that the current number of fault-tolerant rows has exceeded the user's preset maximum number of rows. An error should be reported immediately, and all threads should be notified to exit, roll back the data, end the import task, and prompt the user that the current number of fault-tolerant rows has exceeded the preset fault tolerance limit.

[0034] A fault-tolerant system for data import based on a distributed database, the system comprising:

[0035] The configuration module is used to set the number of erroneous data rows allowed in the data file during the data import command when executing a data import command in a distributed database; that is, to set the number of fault-tolerant rows.

[0036] The startup module is used to start several threads in the database for reading data files and one thread for handling error data, and to open a channel for transmitting error data.

[0037] The reading module is used to read data files and, during data conversion, determines whether the data in the files meets the requirements.

[0038] If the condition is not met, the erroneous data is transmitted into the channel, and the next line of data is read.

[0039] The judgment module is used to determine whether the current counter count exceeds the user-defined number of faulty lines when the thread processing error data receives error data from the thread reading the data file, and then record the error data to a fixed file.

[0040] Preferably, the operation of the setting module is as follows:

[0041] (1) Parse the database import command and determine whether the user has provided the number of rows to be allowed to be faulty:

[0042] If so, it means that the fault tolerance function is enabled;

[0043] (2) Database checks the validity of the number of rows with tolerance entered by the user: Only natural numbers greater than zero are allowed to be entered; otherwise, an error is reported and the user is prompted that the number of rows with tolerance entered is invalid.

[0044] The specific working process of the activation module is as follows:

[0045] (1) The database starts several independent threads to read the data file input by the user;

[0046] (2) The database starts a thread to record erroneous data in the data file;

[0047] (3) Open a many-to-one channel. Each thread reading the data file sends a message to the channel when it encounters erroneous data. The message contains the current erroneous data.

[0048] Preferably, the reading module reads the file data line by line, and compares the number of columns in the database table with the number of fields in the user data based on the metadata of the table in the database.

[0049] If they are not equal, the data in that row is considered to be erroneous, and the erroneous data is sent to the channel to continue processing the next row of data.

[0050] If the data passes the field count validation, then each field in the read row of data is converted sequentially according to the column type based on the metadata of the table in the database.

[0051] If any field fails to be parsed or converted, the data in that row is considered erroneous, and the erroneous data is sent to the channel to continue processing the next row of data.

[0052] The specific working process of the judgment module is as follows:

[0053] (1) When the thread that handles error data receives error data from the file reading thread, it first updates its own counter;

[0054] (2) Compare the value in the counter with the value passed in by the user:

[0055] If the value in the counter is less than the value passed by the user, it indicates that the fault tolerance process continues to accept error data, writes the error data in the current message to a fixed file, and continues to process the next line of error data;

[0056] If the value in the counter is greater than the value passed by the user, it indicates that the current number of fault-tolerant rows has exceeded the user's preset maximum number of rows. An error should be reported immediately, and all threads should be notified to exit, roll back the data, end the import task, and prompt the user that the current number of fault-tolerant rows has exceeded the preset fault tolerance limit.

[0057] An electronic device includes: a memory and at least one processor;

[0058] The memory contains computer programs;

[0059] The at least one processor executes the computer program stored in the memory, causing the at least one processor to execute the fault-tolerant method for data import based on the distributed database as described above.

[0060] A computer-readable storage medium storing a computer program that can be executed by a processor to implement the fault-tolerant method for data import based on a distributed database as described above.

[0061] The fault-tolerant method and system for data import based on a distributed database of the present invention have the following advantages:

[0062] (i) This invention provides users with an option during import, allowing users to specify the number of data rows that may have errors in this import operation, and records all erroneous data to a file;

[0063] (ii) After completing the first import, users only need to modify the erroneous data in the file and then re-import the erroneous data, which greatly saves users' resource consumption and improves the efficiency of user migration. Attached Figure Description

[0064] The invention will be further described below with reference to the accompanying drawings.

[0065] Appendix Figure 1 This is a flowchart illustrating a fault-tolerant method for data import based on a distributed database. Detailed Implementation

[0066] The following detailed description of the fault-tolerant method and system for data import based on a distributed database, with reference to the accompanying drawings and specific embodiments, is provided in the specification.

[0067] Example 1:

[0068] As attached Figure 1 As shown in the figure, this embodiment provides a fault-tolerant method for data import based on a distributed database. The method is as follows:

[0069] S1. When a user executes a data import command in a distributed database, they set the number of data rows in the data file that are allowed to have errors during the import, i.e., they set the number of rows to be allowed to have errors.

[0070] S2. The database starts several threads for reading data files and one thread for handling error data, and opens a channel for transmitting error data;

[0071] S3. Read the data file and, during data conversion, determine whether the data in the file meets the requirements:

[0072] If the condition is not met, the erroneous data is transmitted into the channel, and the next line of data is read.

[0073] S4. When the thread that handles error data receives error data from the thread that reads the data file, it determines whether the current counter count exceeds the number of error-tolerant lines set by the user, and then records the error data to a fixed file.

[0074] In step S1 of this embodiment, when the user executes the data import command in the distributed database, the specific setting of the number of data rows in the data file allowed to contain errors during this import is as follows:

[0075] S101. Parse the database import command and determine if the user has provided the number of rows to be allowed to be faulty:

[0076] If so, it means that the fault tolerance function is enabled;

[0077] S102. Database checks the validity of the number of rows entered by the user for error tolerance: Only natural numbers greater than zero are allowed to be entered; otherwise, an error is reported and the user is prompted that the number of rows entered for error tolerance is invalid.

[0078] In step S2 of this embodiment, the database starts several threads for reading data files and one thread for handling error data, and opens a channel for transmitting error data, as detailed below:

[0079] S201. The database starts several independent threads and begins reading the data file input by the user;

[0080] S202. The database starts a thread to record erroneous data in the data file;

[0081] S203. Open a many-to-one channel. Each thread reading the data file sends a message containing the current error data to the channel when it encounters erroneous data.

[0082] In step S3 of this embodiment, when reading the data file and performing data conversion, the specific steps for determining whether the data in the data file meets the requirements are as follows:

[0083] Read file data line by line, and compare the number of columns in the database table with the number of fields in the user data based on the table's metadata.

[0084] If they are not equal, the data in that row is considered to be erroneous, and the erroneous data is sent to the channel to continue processing the next row of data.

[0085] If the data passes the field count validation, then each field in the read row of data is converted sequentially according to the column type based on the metadata of the table in the database.

[0086] If any field fails to be parsed or converted, the data in that row is considered erroneous, and the erroneous data is sent to the channel to continue processing the next row.

[0087] In step S4 of this embodiment, when the thread for processing error data receives error data from the thread reading the data file, it determines whether the current counter count exceeds the user-defined number of error-tolerant lines, and then records the error data to a fixed file as follows:

[0088] S401: When the thread that handles error data receives error data from the file reading thread, it first updates its own counter.

[0089] S402. Compare the value in the counter with the value passed in by the user:

[0090] ① If the value in the counter is less than the value passed by the user, it indicates that the fault tolerance process continues to accept error data, writes the error data in the current message into a fixed file, and continues to process the next line of error data;

[0091] ② If the value in the counter is greater than the value passed by the user, it indicates that the current number of fault-tolerant rows has exceeded the user's preset maximum number of rows. An error should be reported immediately, and all threads should be notified to exit, roll back the data, end the import task, and prompt the user that the current number of fault-tolerant rows has exceeded the preset fault tolerance limit.

[0092] Example 2:

[0093] This embodiment provides a fault-tolerant system for data import based on a distributed database. The system includes:

[0094] The configuration module is used to set the number of erroneous data rows allowed in the data file during the data import command when executing a data import command in a distributed database; that is, to set the number of fault-tolerant rows.

[0095] The startup module is used to start several threads in the database for reading data files and one thread for handling error data, and to open a channel for transmitting error data.

[0096] The reading module is used to read data files and, during data conversion, determines whether the data in the files meets the requirements.

[0097] If the condition is not met, the erroneous data is transmitted into the channel, and the next line of data is read.

[0098] The judgment module is used to determine whether the current counter count exceeds the user-defined number of faulty lines when the thread processing error data receives error data from the thread reading the data file, and then record the error data to a fixed file.

[0099] The specific working process of the setting module in this embodiment is as follows:

[0100] (1) Parse the database import command and determine whether the user has provided the number of rows to be allowed to be faulty:

[0101] If so, it means that the fault tolerance function is enabled;

[0102] (2) Database checks the validity of the number of rows with tolerance entered by the user: Only natural numbers greater than zero are allowed to be entered; otherwise, an error is reported and the user is prompted that the number of rows with tolerance entered is invalid.

[0103] The specific working process of the activation module in this embodiment is as follows:

[0104] (1) The database starts several independent threads to read the data file input by the user;

[0105] (2) The database starts a thread to record erroneous data in the data file;

[0106] (3) Open a many-to-one channel. Each thread reading the data file sends a message to the channel when it encounters erroneous data. The message contains the current erroneous data.

[0107] In this embodiment, the reading module reads file data line by line and compares the number of columns in the database table with the number of fields in the user data based on the metadata of the table.

[0108] If they are not equal, the data in that row is considered to be erroneous, and the erroneous data is sent to the channel to continue processing the next row of data.

[0109] If the data passes the field count validation, then each field in the read row of data is converted sequentially according to the column type based on the metadata of the table in the database.

[0110] If any field fails to be parsed or converted, the data in that row is considered erroneous, and the erroneous data is sent to the channel to continue processing the next row.

[0111] The specific working process of the judgment module in this embodiment is as follows:

[0112] (1) When the thread that handles error data receives error data from the file reading thread, it first updates its own counter;

[0113] (2) Compare the value in the counter with the value passed in by the user:

[0114] If the value in the counter is less than the value passed by the user, it indicates that the fault tolerance process continues to accept error data, writes the error data in the current message to a fixed file, and continues to process the next line of error data;

[0115] If the value in the counter is greater than the value passed by the user, it indicates that the current number of fault-tolerant rows has exceeded the user's preset maximum number of rows. An error should be reported immediately, and all threads should be notified to exit, roll back the data, end the import task, and prompt the user that the current number of fault-tolerant rows has exceeded the preset fault tolerance limit.

[0116] Example 3:

[0117] This embodiment also provides an electronic device, including: a memory and a processor;

[0118] The memory stores the instructions executed by the computer.

[0119] The processor executes computer execution instructions stored in the memory, causing the processor to execute the fault-tolerant method for data import based on a distributed database in any embodiment of the present invention.

[0120] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), off-the-shelf programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0121] Memory is used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and by accessing data stored in the memory. Memory can mainly include a program storage area and a data storage area. The program storage area can store the operating system, at least one application program required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, memory can also include high-speed random access memory, and can also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart memory cards (SMC), secure digital cards (SD cards), flash memory cards, at least one disk storage device, flash memory devices, or other volatile solid-state storage devices.

[0122] Example 4:

[0123] This invention also provides a computer-readable storage medium storing multiple instructions, which are loaded by a processor to cause the processor to execute the fault-tolerant data import method based on a distributed database according to any embodiment of this invention. Specifically, a system or apparatus equipped with a storage medium may be provided, on which software program code implementing the functions of any of the above embodiments is stored, and the computer (or CPU or MPU) of the system or apparatus may read and execute the program code stored in the storage medium.

[0124] In this case, the program code read from the storage medium can itself implement the function of any of the above embodiments, and therefore the program code and the storage medium storing the program code constitute part of the present invention.

[0125] Storage media embodiments for providing program code include floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RYM, DVD-RW, DVD+RW), magnetic tapes, non-volatile memory cards, and ROMs. Alternatively, program code can be downloaded from a server computer via a communication network.

[0126] Furthermore, it should be clear that not only can the program code read by the computer be executed, but also the operating system or other components operating on the computer can be instructed based on the program code to perform some or all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0127] Furthermore, it is understood that the program code read from the storage medium is written to the memory set in the expansion board inserted into the computer or to the memory set in the expansion unit connected to the computer. Then, based on the instructions of the program code, the CPU or other components installed on the expansion board or expansion unit execute some and all of the actual operations, thereby realizing the function of any of the embodiments described above.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fault-tolerant method for data import based on a distributed database, characterized in that, The method is as follows: When a user executes a data import command in a distributed database, they can set the number of erroneous rows in the data file that are allowed to be imported, i.e., set the number of rows to be allowed to be correct. The database starts several threads for reading data files and one thread for handling error data, and opens a channel for transmitting error data; Read the data file and, during data transformation, determine whether the data in the file meets the requirements: If the condition is not met, the erroneous data is transmitted into the channel, and the next line of data is read. When the thread that handles error data receives error data from the thread that reads the data file, it checks whether the current counter count exceeds the number of error-tolerant lines set by the user, and then records the error data to a fixed file. Specifically, when a user executes a data import command in a distributed database, the number of data rows allowed to contain errors in the data file during the import is set as follows: The database import command is parsed, and it is determined whether the user has provided a number of rows for error tolerance. If so, it means that the fault tolerance function is enabled; The database checks the validity of the number of rows entered by the user for error tolerance: only natural numbers greater than zero are allowed; otherwise, an error is reported and the user is prompted that the number of rows entered for error tolerance is invalid. The database starts several threads for reading data files and one thread for handling error data, and opens a channel for transmitting error data, as detailed below: The database starts several independent threads to read the data file input by the user; The database starts a thread to log erroneous data in the data files; A many-to-one channel is opened. Each thread reading the data file sends a message containing the current error data to the channel when it encounters erroneous data. When reading a data file and performing data transformation, the process of determining whether the data in the file meets the requirements is as follows: Read file data line by line, and compare the number of columns in the database table with the number of fields in the user data based on the table's metadata. If they are not equal, the data in that row is considered to be erroneous, and the erroneous data is sent to the channel to continue processing the next row of data. If the data passes the field count validation, then each field in the read row of data is converted sequentially according to the column type based on the metadata of the table in the database. If any field fails to be parsed or converted, the data in that row is considered erroneous, and the erroneous data is sent to the channel to continue processing the next row of data. When the thread handling error data receives error data from the thread reading the data file, it checks whether the current counter count exceeds the user-defined number of error-tolerant lines, and then records the error data to a fixed file, as follows: When the thread that handles error data receives error data from the file reading thread, it first updates its own counter. Compare the value in the counter with the value passed in by the user: If the value in the counter is less than the value passed by the user, it indicates that the fault tolerance process continues to accept error data, writes the error data in the current message to a fixed file, and continues to process the next line of error data; If the value in the counter is greater than the value passed by the user, it indicates that the current number of fault-tolerant rows has exceeded the user's preset maximum number of rows. An error should be reported immediately, and all threads should be notified to exit, roll back the data, end the import task, and prompt the user that the current number of fault-tolerant rows has exceeded the preset fault tolerance limit.

2. A fault-tolerant system for data import based on a distributed database, characterized in that, The system includes: The configuration module is used to set the number of erroneous data rows allowed in the data file during the data import command when executing a data import command in a distributed database; that is, to set the number of fault-tolerant rows. The startup module is used to start several threads in the database for reading data files and one thread for handling error data, and to open a channel for transmitting error data. The reading module is used to read data files and, during data conversion, determines whether the data in the files meets the requirements. If the condition is not met, the erroneous data is transmitted into the channel, and the next line of data is read. The judgment module is used to determine whether the current counter count exceeds the user-defined number of fault-tolerant lines when the thread processing error data receives error data from the thread reading the data file, and then record the error data to a fixed file; The specific working process of the settings module is as follows: (1) Parse the database import command and determine whether the user has provided the number of rows to be allowed for error correction: If so, it means that the fault tolerance function is enabled; (2) Database checks the validity of the number of rows with tolerance entered by the user: only natural numbers greater than zero are allowed to be entered, otherwise an error is reported and the user is prompted that the number of rows with tolerance entered is invalid; The specific process of enabling the module is as follows: (1) The database starts several independent threads to read the data file input by the user; (2) The database starts a thread to record erroneous data in the data file; (3) Open a many-to-one channel. Each thread reading the data file sends a message to the channel when it encounters erroneous data. The message contains the current erroneous data. The reading module reads file data line by line and compares the number of columns in the database table with the number of fields in the user data based on the database table's metadata. If they are not equal, the data in that row is considered to be erroneous, and the erroneous data is sent to the channel to continue processing the next row of data. If the data passes the field count validation, then each field in the read row of data is converted sequentially according to the column type based on the metadata of the table in the database. If any field fails to be parsed or converted, the data in that row is considered erroneous, and the erroneous data is sent to the channel to continue processing the next row of data. The specific working process of the judgment module is as follows: (1) When the thread that handles error data receives error data from the file reading thread, it first updates its own counter; (2) Compare the value in the counter with the value passed in by the user: If the value in the counter is less than the value passed by the user, it indicates that the fault tolerance process continues to accept error data, writes the error data in the current message to a fixed file, and continues to process the next line of error data; If the value in the counter is greater than the value passed by the user, it indicates that the current number of fault-tolerant rows has exceeded the user's preset maximum number of rows. An error should be reported immediately, and all threads should be notified to exit, roll back the data, end the import task, and prompt the user that the current number of fault-tolerant rows has exceeded the preset fault tolerance limit.

3. An electronic device, comprising: include: Memory and at least one processor; The memory contains computer programs; The at least one processor executes the computer program stored in the memory, causing the at least one processor to perform the fault-tolerant method for data import based on a distributed database as described in claim 1.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be executed by a processor to implement the fault-tolerant method for data import based on a distributed database as described in claim 1.

Images