Method, device and medium for migrating data files to a target database

Abstract

Embodiments of the present disclosure relate to methods, devices, and media for migrating data files to a target database. According to the method, multiple records of a data file from a source database are read in sequence, each record including multiple fields separated by a delimiter; conflict character detection is performed on each read record so as to process records with conflict characters; an associated output format conversion expression is obtained, and based on the output format conversion expression, each record after conflict character detection is converted from a first output format associated with the source database to a second output format associated with the target database; and the resulting data file is imported into the target database. In this way, data files of various output formats, especially large data files, can be normally migrated to the target database in a unified manner without interruption of migration due to the presence of conflict characters.

Description

Technical Field

[0001] Embodiments of the present disclosure generally relate to the field of data migration, and more particularly to a method, apparatus, and medium for migrating data files to a target database. Background Art

[0002] In the financial sector, migrating data files from one database to another is often necessary for data security reasons. During data file migration, if any record in the data file contains a character that conflicts with the encoding of a delimiter, this character may be misidentified, leading to an interruption in the migration process and preventing the data file from being properly migrated to the target database. Currently, the ostrich strategy is commonly used to avoid this problem. Specifically, characters with low usage probability or special characters such as the hidden character (e.g., ASCII 001) are often used as delimiters to separate fields in a record. However, such characters are often difficult to input and more cumbersome to process. Furthermore, since the data files that need to be migrated are typically large data files in the order of GB, TB, or even higher, even if such characters are used as delimiters, there is a high probability that characters that conflict with the encoding of the delimiter will be found in the data file to be migrated, causing the migration process to fail. Summary of the Invention

[0003] In response to the above problems, the present disclosure provides a method, device and medium for migrating data files to a target database, so that data files of various output formats, especially large data files, can be normally migrated to the target database in a unified manner without interruption of migration due to the presence of conflicting characters.

[0004] According to a first aspect of the present disclosure, a method for migrating a data file to a target database is provided, comprising: sequentially reading in a plurality of records from a data file of a source database, each record comprising a plurality of fields separated by a delimiter; performing conflict character detection on each read record so as to process records having conflict characters, the conflict characters indicating characters that conflict with the delimiter; obtaining an associated output format conversion expression, and converting each record after conflict character detection from a first output format associated with the source database to a second output format associated with the target database based on the output format conversion expression; and importing the resulting result data file into the target database.

[0005] According to a second aspect of the present disclosure, a computing device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor so as to enable the at least one processor to perform the method of the first aspect of the present disclosure.

[0006] In a third aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method of the first aspect of the present disclosure.

[0007] In some embodiments, the character encoding format currently used by the data file is a first character encoding format, and sequentially reading in multiple records of the data file from the source database includes: sequentially reading in the multiple records in a second character encoding format in the form of a byte stream, where the second character encoding format is the character encoding format used by the operating system used to perform the migration of the data file.

[0008] In some embodiments, conflict character detection is performed on each record read in, including: character-by-character analysis of each field included in the corresponding record read in the second character encoding format to determine the first byte encoding of each first character in the plurality of first characters encoded in the second character encoding format in the corresponding field; determining whether the first byte encoding is the same as the second byte encoding of the separator encoded in the second character encoding format; in response to determining that the first byte encoding is the same as the second byte encoding, determining to which second character encoded in the first character encoding format of the corresponding record the first byte encoding belongs; if the determined second character is not the separator, determining one or more first characters of the plurality of first characters associated with the determined second character as conflict characters.

[0009] In some embodiments, the first character encoding method is the same as or different from the second character encoding method, and the first character encoding format is GBK encoding format, GB2312 encoding format, UTF-8 encoding format, UTF-16 encoding format or ISO-8859-1 encoding format.

[0010] In some embodiments, processing a record containing conflicting characters includes: deleting the detected conflicting characters from the corresponding record, or marking all fields in the corresponding record with designated characters supported by the target database.

[0011] In some embodiments, based on the output format conversion expression, converting each record after conflict character detection from a first output format associated with the source database to a second output format associated with the target database includes: building a lexical library for the output format conversion expression, the lexical library including multiple recognition rules for different types of operands and operators included in the output format conversion expression and priorities for different types of operators included in the output format conversion expression; performing lexical analysis on the output format conversion expression according to the multiple recognition rules to generate a corresponding word list for the output format conversion expression; performing grammatical analysis on the generated word list to build a syntax tree for the output format conversion expression; and generating an output result having the second output format for the corresponding record according to the syntax tree.

[0012] In some embodiments, performing lexical analysis on the output format conversion expression according to the multiple recognition rules to generate a corresponding word list for the output format conversion expression includes: creating a blank word list; recognizing a first word included in the output format conversion expression according to the multiple recognition rules to insert the recognized first word into the word list; and recognizing subsequent words included in the output format conversion expression one by one according to the multiple recognition rules, and inserting the recognized subsequent words into the word list.

[0013] In some embodiments, according to the multiple recognition rules, identifying the first word included in the output format conversion expression includes: reading a first character in the output format conversion expression; determining which recognition rule of the multiple recognition rules the first character satisfies; reading a second character immediately following the first character in the output format conversion expression; if the second character does not satisfy the determined recognition rule, recognizing the first character as the first word.

[0014] In some embodiments, according to the multiple recognition rules, identifying the first word included in the output format conversion expression also includes: if the second character meets the determined recognition rule, reading the subsequent characters immediately after the second character until the read characters do not meet the determined recognition rule; identifying the word composed in sequence of all the read characters that meet the determined recognition rule as the first word, and inserting the first word into the word list.

[0015] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0017] Figure 1 A schematic diagram of a system 100 for implementing a method for migrating a data file to a target database according to an embodiment of the present invention is shown.

[0018] Figure 2 A flowchart of a method 200 for migrating a data file to a target database according to an embodiment of the present disclosure is shown.

[0019] Figure 3 FIG. 3 is a flowchart of a method 300 for detecting conflicting characters for each read record according to an embodiment of the present disclosure.

[0020] Figure 4 A flow chart of a method 400 for converting records from a first output format to a second output format according to an embodiment of the present disclosure is shown.

[0021] Figure 5 A block diagram of an electronic device 500 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0022] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] As used herein, the term "including" and its variations represent open inclusion, i.e., "including but not limited to." Unless otherwise stated, the term "or" means "and / or." The term "based on" means "based at least in part on." The terms "an example embodiment" and "an embodiment" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "first," "second," etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0024] As described above, during data file migration, if any record in the data file includes a character that conflicts with the encoding of a delimiter, this character will be misidentified, leading to an interruption in the migration process and preventing the data file from being properly migrated to the target database. Currently, an ostrich strategy is commonly used to avoid this problem. Specifically, coded characters with a low probability of use or special characters such as hidden characters are usually used as delimiters to separate fields (also known as domains) in a record. However, on the one hand, such characters are generally difficult to input and more cumbersome to process; on the other hand, since the data files that need to be migrated are generally large data files of GB, TB, or even higher magnitude, during the migration process, even if such characters are used as delimiters, there is still a high probability that characters that conflict with the encoding of such delimiters will be found in the data file to be migrated, leading to the failure of the migration process.

[0025] Furthermore, because the data files to be migrated to a target database can come from a variety of data sources, with different output formats and sizes, each data file from each source must be processed and re-formatted before it can be transferred to the target database. Currently, there is no unified method for migrating data files of varying output formats into a target database, requiring separate programs to be written for each data file from each data source.

[0026] In order to at least partially solve the above-mentioned problems and one or more of other potential problems, an exemplary embodiment of the present disclosure proposes a method for migrating a data file to a target database, comprising: sequentially reading in a plurality of records from a data file of a source database, each record comprising a plurality of fields separated by a delimiter; performing conflict character detection on each read record so as to process records having conflict characters, wherein the conflict characters indicate characters that conflict with the delimiter; obtaining an associated output format conversion expression to convert each record after conflict character detection from a first output format associated with the source database to a second output format associated with the target database based on the output format conversion expression; and importing the resulting result data file into the target database. In this way, data files of various output formats, especially large data files, can be normally migrated to the target database in a unified manner without interruption of migration due to the presence of conflict characters.

[0027] Figure 1 FIG. 1 is a schematic diagram of a system 100 for implementing a method for migrating a data file to a target database according to an embodiment of the present invention. Figure 1As shown, the system 100 may include, for example, a computing device 110, a plurality of user terminals 120-1, 120-M to 120-N, and a network 130. The computing device 110 may exchange data with the plurality of user terminals 120-1, 120-M to 120-N via the network 130. The user terminals 120-1 to 120-N may include, for example, but are not limited to, a user's mobile terminal, tablet device, personal computer, laptop computer, desktop computer, etc., which may be used to issue an operation request to the computing device 110, including transferring a data file from a source database 140 to a target database 150. In the present disclosure, the user terminal may be part of the computing device 110 or may be independent of the computing device 110. The source database 140 and the target database 150 may be coupled to the computing device 110 via a network, or directly connected to the computing device 110, or may also be part of the computing device 110 (e.g., different databases stored on one or more storage devices of the computing device 110).

[0028] After receiving the operation request, the computing device 110 may be used to implement migration of the data file from the source database 140 to the target database 150. Figure 1 In the figure, the system 100 is shown to include only one source database 140, but the system 100 may include multiple different source databases 140 to migrate data files from these different source databases 140 to the target database 150. Of course, it should be understood that the system 100 may also include multiple target databases. In actual use, the source database and the target database are relative terms, which mainly depends on which database the data files are to be migrated to. The source of the data files is called the source database, and the destination of the data files is called the target database. In some embodiments, the computing device 110 may include at least one processor and at least one memory coupled to the at least one processor, the memory storing instructions executable by the at least one processor, which, when executed by the at least one processor, performs the method 200 for migrating data files to the target database described below.

[0029] Figure 2 FIG. 2 shows a flow chart of a method 200 for migrating a data file to a target database according to an embodiment of the present disclosure. Figure 1 The computing device 110 shown is executed, and may also be executed on Figure 5 The method 200 is executed at the electronic device 500. It should be understood that the method 200 may further include additional blocks not shown and / or may omit the blocks shown, and the scope of the present disclosure is not limited in this respect.

[0030] In step 202, a plurality of records from a data file of a source database are sequentially read in, each record including a plurality of fields separated by a delimiter.

[0031] The source database can be, for example, Figure 1 The source database 140 mentioned in .

[0032] In this disclosure, a data file refers to a file containing multiple lines of data records, each of which is referred to as a record. In such a data file, each record includes multiple fields separated by delimiters, and each field is associated with an attribute, such as name, ID number, bank card number, most recent withdrawal date, withdrawal amount, etc. The specific attribute depends on the content of the data file.

[0033] Because conflicting characters will be detected and processed later, in the present disclosure, any character can be used as the delimiter for separating fields in a record. This does not necessarily require the use of coded characters with a low probability of use or special characters such as hidden characters. In some embodiments of the present disclosure, for ease of input and processing, some commonly used characters can be used as delimiters, such as, but not limited to, space characters or the symbol "|".

[0034] In the present disclosure, the character encoding format currently used by the data file to be migrated can be a first character encoding format, and the character encoding format used by the operating system for executing the migration of the data file can be a second character encoding format, and the first character encoding method can be the same as or different from the second character encoding method. Since the data file needs to be read in by the operating system for corresponding processing, the multiple records of the data file from the source database read in sequence actually refer to the multiple records of the data file read in sequence in the second character encoding format. In some embodiments, the multiple records of the data file can be read in sequence in the second character encoding format in the form of a byte stream (for example, using apache common io stream technology). By reading these records in the form of a byte stream, the speed of reading and writing data files can be effectively improved, and the memory occupancy rate is low, thereby effectively avoiding the problems of being unable to open large data files and memory jamming and easy overflow.

[0035] As mentioned above, the first character encoding method may be the same as or different from the second character encoding method, so the second character encoding method may be, for example, the ISO-8859-1 encoding format. In this case, the first character encoding method may also be the ISO-8859-1 encoding format, or may be other encoding formats, such as the GBK encoding format, the GB2312 encoding format, the UTF-8 encoding format, or the UTF-16 encoding format.

[0036] Suppose in an example, the first character encoding format is GBK and the second character encoding format is ISO-8859-1. It should be understood that GBK uses a double-byte encoding method and needs to use 2-byte encoding to encode a character, while the ISO-8859-1 encoding format uses a single-byte encoding method and uses 1-byte encoding to encode a character. Therefore, for example, for the character "倈" included in a data file, the GBK encoding format will encode it into a character "82, 7C", while the ISO-8859-1 encoding format will encode it into two characters, which are "82" and "7C" respectively. Additionally, if the delimiter is "|", the GBK encoding format will encode this delimiter into a character "007C", while the ISO-8859-1 encoding format will encode this delimiter into a character "7C".

[0037] In step 204, conflict character detection is performed on each read record so as to process the records with conflict characters.

[0038] In the present disclosure, a conflict character indicates a character that conflicts with a delimiter, and this conflict refers to a conflict in encoding between the two.

[0039] For example, according to the previous example, it can be seen that there is a conflict between the two characters "82" and "7C" obtained by encoding "倈" in the ISO-8859-1 encoding format and the character obtained by encoding the delimiter in the ISO-8859-1 encoding format, because one of the two characters "82" and "7C" obtained by encoding "倈" in the ISO-8859-1 encoding format, which is "7C", is the same as the character obtained by encoding the delimiter in the ISO-8859-1 encoding format, which is "7C". Therefore, if this conflict is not processed, it will cause the operating system to misidentify "倈", resulting in a processing error when inputting the corresponding record into the target database, and further leading to the inability to achieve the normal warehousing of the corresponding data file. However, in the present disclosure, by performing conflict character detection on each record and processing the records with conflict characters, the problem of the data file being unable to be warehoused normally can be effectively avoided.

[0040] In the present disclosure, processing the records with conflict characters may include but is not limited to deleting the detected conflict characters from the corresponding records, or identifying all fields in the corresponding records where the conflict characters are detected with a specified character supported by the target database (for example, a bracketing character such as a quotation mark), and the specific processing can be selected according to actual usage needs.

[0041] The following will be combined with Figure 3 Make a further and more detailed description of the conflict character detection for each record.

[0042] In step 206 , an associated output format conversion expression is obtained, so as to convert each record after conflicting character detection from a first output format associated with the source database to a second output format associated with the target database based on the output format conversion expression.

[0043] In the present disclosure, an output format conversion expression is used to indicate how to obtain a second output format associated with a target database based on a first output format associated with a source database.

[0044] In some embodiments, to uniformly migrate data files of various output formats to a target database, multiple output format conversion expressions can be pre-built to convert the output format of a source database record to the output format of a corresponding target database record. Each output format conversion expression is associated with the identifier of the source database and the identifier of the target database and stored in a table. This allows conversion to be performed by querying the table to retrieve the associated output format conversion expression when data file migration is required.

[0045] For clarity, the following is an example of an output format conversion expression: $0||'|'||substr($1,2,4)||'|'||length($3)||'|'||-(($5+100) / 2)||'|'. In this expression, $0 represents all fields in the current record of the data file to be migrated, || represents a connector (that is, the contents on both sides of the symbol are connected), | represents a separator, and $n represents the nth field in the current record, where n is an arbitrary integer.

[0046] Of course, according to actual needs, before performing the output format conversion in step 206, other processing may be performed on each record, such as performing some encryption operations on the data in sensitive fields.

[0047] The following will be combined Figure 4 A method 400 for converting records from a first output format associated with a source database to a second output format associated with a target database based on an associated output format conversion expression is further described in greater detail.

[0048] In step 208, the final result data file is imported into the target database.

[0049] After completing the format conversion of all records in the data file, the final result data file can be imported into the target database for storage.

[0050] Figure 3FIG. 0 shows a flowchart of a method 300 for detecting conflicting characters for each read record according to an embodiment of the present disclosure. The method 300 may be executed by a computing device 110 as shown in Figure 1 or may be executed at an electronic device 500 as shown in Figure 5 . It should be understood that the method 300 may further include additional blocks not shown and / or may omit the blocks shown. The scope of the present disclosure is not limited in this regard.

[0051] In step 302, each field included in the corresponding record read in the second character encoding format is analyzed character by character to determine the first byte encoding of each first character among the multiple first characters encoded in the second character encoding format of the corresponding field.

[0052] In step 304, it is determined whether the first byte encoding (i.e., any one of the first byte encodings determined in step 302) is the same as the second byte encoding of the delimiter encoded in the second character encoding format.

[0053] Assume that the currently used second character encoding format is ISO-8859-1, the first character encoding format used by the data file is GBK, and the delimiter used in the data file is "|". Therefore, when analyzing the first character "7C" among the two first characters "82" and "7C" obtained by encoding the character "倈" in the data file in the second encoding format, it can be determined that the first byte encoding "7C" of this first character is the same as the second byte encoding "7C" of the delimiter "|" encoded in this second encoding format.

[0054] In step 306, in response to determining that the first byte encoding (i.e., the first byte encoding in step 304) is the same as the second byte encoding (the second byte encoding of the delimiter encoded in the second character encoding format), it is determined which second character among the multiple second characters encoded in the first character encoding format of the corresponding record the first byte encoding belongs to.

[0055] In the present disclosure, for the sake of distinction, each character encoded in the second character encoding format is referred to as a first character, and each character obtained in the first character encoding format is referred to as a second character.

[0056] Continuing with the previous example, since the first byte encoding of the first character "7C" obtained by encoding in the second encoding format in "Lai" is the same as the second byte encoding "7C" obtained by encoding the delimiter "|" in this second encoding format, it is necessary to determine which second character obtained by encoding in the first character encoding format the first character "7C" that has the same encoding as the delimiter belongs to in the corresponding record. In this example, "7C" belongs to the second character "82,7C" obtained by encoding in the first character encoding format of this record, that is, it belongs to the character "Lai", rather than belonging to the delimiter itself.

[0057] In step 308, if the determined second character (i.e., the character encoded in the first character encoding format determined in step 306) is not a delimiter, then one or more first characters associated with the determined second character among the multiple first characters (i.e., the multiple first characters of the corresponding fields included in the corresponding record read in the second character encoding format in step 302) are determined as conflicting characters.

[0058] Continuing with the previous example, since the second character "82,7C" obtained by encoding "Lai" in the first character encoding format is associated with the two consecutive first characters "82" and "7C" read in the second character encoding format in step 302, these two consecutive first characters "82" and "7C" can be determined as conflicting characters.

[0059] By adopting the above means, the present disclosure can detect conflicting characters for each record in the data file during the migration process of the data file, so as to be able to process the records with conflicting characters in a timely manner, and thus can effectively prevent the interruption of the data file migration process due to the existence of conflicting characters. Moreover, by adopting this means, in the present disclosure, any character can be used as the delimiter character, without the need to use special characters such as hidden characters as the delimiter character, thereby making the input and processing of each record in the data file simpler and more convenient.

[0060] Figure 4 The flowchart of a method 400 for converting a record from a first output format to a second output format according to an embodiment of the present disclosure is shown. Method 400 can be executed by a computing device 110 as Figure 1 shown, or can also be executed at an electronic device 500 as Figure 5 shown. It should be understood that method 400 may further include additional boxes not shown and / or boxes shown may be omitted, and the scope of the present disclosure is not limited in this regard.

[0061] In the present disclosure, method 400 is performed after the corresponding record has undergone conflicting character detection and processing, so as to ensure that there are no problems with conflicting characters in this format conversion.

[0062] In step 402, a lexical library is constructed for the output format conversion expression.

[0063] In the present disclosure, the constructed lexical library should include multiple recognition rules for different types of operands and operators included in the output format conversion expression and the priorities of different types of operators included in the output format conversion expression, so as to facilitate subsequent lexical analysis and syntax analysis.

[0064] For example, Table 1 below represents an example lexical library built for the example output format conversion expression mentioned above.

[0065] Table 1

[0066]

[0067] In step 404 , the output format conversion expression is lexically analyzed according to a plurality of recognition rules (ie, the plurality of recognition rules in the lexical library constructed in step 402 ) to generate a corresponding word list for the output format conversion expression.

[0068] In the present disclosure, in order to generate a corresponding word list, a blank word list needs to be created first. Then, the first word included in the output format conversion expression can be identified according to the multiple recognition rules in the lexical library constructed in step 402, so as to insert the identified first word into the word list.

[0069] In the present disclosure, the first word included in the output format conversion expression can be identified in the following manner. First, the first character in the output format conversion expression is read, and it is determined which recognition rule of the multiple recognition rules the first character satisfies. Then, the second character immediately following the first character in the output format conversion expression is read. If the second character does not meet the determined recognition rule (i.e., the recognition rule satisfied by the first character), the first character is identified as the first word. However, if the second character meets the determined recognition rule (i.e., the recognition rule satisfied by the first character), the subsequent character immediately following the second character is read, and when the subsequent character meets the determined recognition rule, the characters are continued to be read until the read characters no longer meet the determined recognition rule. Then, the word composed in order of all the characters read that meet the determined recognition rule is determined to be the first word, and the first word is inserted into the word list.

[0070] After the first word of the output format conversion expression is identified, subsequent words included in the output format conversion expression are identified one by one according to the plurality of recognition rules in the lexical library constructed in step 402, and the identified subsequent words are inserted into the corresponding word list. The method for identifying each subsequent word of the output format conversion expression is substantially the same as the method for identifying the first word of the output format conversion expression, except that the starting character of the subsequent word is the character immediately following the previously identified word in the output format conversion expression, rather than the first character included in the output format conversion expression.

[0071] In some embodiments, when each recognized word is inserted into the generated word list, a semicolon may be inserted before the word to distinguish it from other words.

[0072] For example, for the output format conversion expression mentioned above, the word list generated for it can be expressed as follows:

[0073] $;0;||||||substr;($1,2,4)|||||length($3)||||--(($5+100) / 2)|||

[0074] In step 406 , the generated word list is parsed to construct a syntax tree for the output format conversion expression.

[0075] For example, the word list generated in step 404 can be read in sequence, and then a stack is used to temporarily store the most recently read word. When an operand and "(" are encountered, they are directly pushed onto the stack; when an operator with a priority higher than the priority of the operator read in before is encountered, the operator is popped out of the stack, and a tree node association of the popped operator and the operand is established, and then the operator is changed into the operand node and pushed back onto the stack, and the priority before the update is the priority of the operator closest to the top of the stack, and the cycle is cycled until the current priority is higher than the previous priority, and then the current operator is pushed onto the stack; when ")" or "," are encountered, the stack is also popped out for calculation, wherein ")" needs to be calculated until the first "(" in the stack, and "," is calculated until the first "(" or ",". The entire algorithm continues until the word list is finished reading, and the only element in the stack is the root node of the syntax tree. If an error occurs during the calculation process, the output custom expression parsing exception is output.

[0076] In step 408 , an output result having a second output format is generated for the corresponding record according to the syntax tree constructed in step 406 .

[0077] For example, based on the corresponding record, the value of each leaf node of the syntax tree constructed in step 406 can be recursively calculated starting from the root node of the syntax tree. Then, based on the value of each leaf node, the value of the root node of the syntax tree can be calculated backtrackingly. Finally, the value of the root node is determined as the output result of the corresponding record in the second output format.

[0078] Figure 5 1 shows a schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure. Figure 1 The computing device 110 shown can be implemented by an electronic device 500. As shown, the electronic device 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 502 or computer program instructions loaded from a storage unit 508 into a random access memory (RAM) 503. In the random access memory 503, various programs and data required for the operation of the electronic device 500 can also be stored. The central processing unit 501, the read-only memory 502, and the random access memory 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0079] Multiple components in the electronic device 500 are connected to the input / output interface 505, including: an input unit 506, such as a keyboard, a mouse, a microphone, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a magnetic disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0080] The various processes and procedures described above, such as methods 200-400, may be performed by the central processing unit 501. For example, in some embodiments, the methods 200-400 may be implemented as a computer software program that is tangibly embodied in a machine-readable medium, such as the storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed onto the device 500 via the read-only memory 502 and / or the communication unit 509. When the computer program is loaded into the random access memory 503 and executed by the central processing unit 501, one or more actions of the methods 200-400 described above may be performed.

[0081] The present disclosure relates to methods, apparatuses, systems, electronic devices, computer-readable storage media, and / or computer program products. The computer program products may include computer-readable program instructions for executing various aspects of the present disclosure.

[0082] A computer-readable storage medium can be a tangible device that can hold and store instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination thereof. As used herein, a computer-readable storage medium is not to be construed as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through an electrical wire.

[0083] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge computing devices. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0084] The computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., utilizing an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions. The electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0085] Various aspects of the present disclosure are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0086] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0087] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0088] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and the part of the module, program segment or instruction contains one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the prescribed function or action, or can be implemented by a combination of dedicated hardware and computer instructions.

[0089] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for migrating a data file to a target database, comprising: Reading a plurality of records from a data file of a source database in sequence, each record including a plurality of fields separated by a delimiter; Performing conflict character detection on each read record so as to process records with conflict characters, wherein the conflict characters indicate characters that conflict with the separator; Obtaining an associated output format conversion expression to convert each record after conflict character detection from a first output format associated with the source database to a second output format associated with the target database based on the output format conversion expression, including: constructing a lexical library for the output format conversion expression, the lexical library including a plurality of recognition rules regarding different types of operands and operators included in the output format conversion expression and priorities of different types of operators included in the output format conversion expression, so as to perform lexical analysis and grammatical analysis, so as to construct a syntax tree for the output format conversion expression; determining the value of a root node of the constructed syntax tree as the output result of the corresponding record in the second output format; and The resulting data file is input into the target database.

2. The method according to claim 1 , wherein the character encoding format currently used by the data file is a first character encoding format, the character encoding format used by the operating system for performing the migration of the data file is a second character encoding format, and sequentially reading multiple records from the data file of the source database comprises: The plurality of records are sequentially read in a byte stream manner using a second character encoding format.

3. The method according to claim 2, wherein performing conflict character detection on each read record comprises: Performing character-by-character analysis on each field included in the corresponding record read in the second character encoding format to determine a first byte code of each first character of a plurality of first characters encoded in the second character encoding format in the corresponding field; determining whether the first byte code is the same as a second byte code of the delimiter encoded in the second character encoding format; In response to determining that the first byte code is the same as the second byte code, determining to which second character of a plurality of second characters encoded in a first character encoding format of the corresponding record the first byte code belongs; If the determined second character is not the separator, one or more first characters associated with the determined second character among the plurality of first characters are determined as conflicting characters.

4. The method according to claim 2, wherein the first character encoding format is the same as or different from the second character encoding format, and the first character encoding format is GBK encoding format, GB2312 encoding format, UTF-8 encoding format, UTF-16 encoding format or ISO-8859-1 encoding format.

5. The method according to claim 1, wherein processing the record having conflicting characters comprises: The detected conflicting characters are deleted from the corresponding record, or all fields in the corresponding record are marked with designated characters supported by the target database.

6. The method according to claim 1, wherein, based on the output format conversion expression, converting each record after conflict character detection from a first output format associated with the source database to a second output format associated with the target database comprises: performing lexical analysis on the output format conversion expression according to the plurality of recognition rules, so as to generate a corresponding word list for the output format conversion expression; Performing grammatical analysis on the generated word list to construct a syntax tree for the output format conversion expression; An output result having the second output format is generated for the corresponding record according to the syntax tree.

7. The method according to claim 6, wherein performing lexical analysis on the output format conversion expression according to the plurality of recognition rules to generate a corresponding word list for the output format conversion expression comprises: Create a blank word list; identifying a first word included in the output format conversion expression according to the plurality of recognition rules, so as to insert the identified first word into the word list; According to the plurality of recognition rules, subsequent words included in the output format conversion expression are recognized one by one, and the recognized subsequent words are inserted into the word list.

8. The method according to claim 7, wherein identifying the first word included in the output format conversion expression according to the plurality of recognition rules comprises: Read the first character in the output format conversion expression; determining which recognition rule of the plurality of recognition rules the first character satisfies; Reading a second character immediately following the first character in the output format conversion expression; If the second character does not meet the determined recognition rule, the first character is recognized as the first word.

9. The method according to claim 8, wherein identifying the first word included in the output format conversion expression according to the plurality of recognition rules further comprises: If the second character meets the determined recognition rule, reading subsequent characters immediately following the second character until the read characters do not meet the determined recognition rule; A word composed of all the read characters that meet the determined recognition rule in sequence is identified as the first word, and the first word is inserted into the word list.

10. A computing device comprising: at least one processor; as well as a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 9.

Images