A data file loading method, device, computer device and storage medium

Abstract

An embodiment of the present invention discloses a method, device, computer device and storage medium for loading data files. The data file loading method includes: obtaining the content of the data file to be extracted loaded into the memory, and determining the delimiter of the content of the data file to be extracted; parsing the content of the data file to be extracted line by line according to the delimiter, and generating the target data file content corresponding to the List data structure; performing verification on each column of data in the target data file content line by line based on a regular expression, and counting the data length of each column of data; generating an SQL statement according to the verified target data file content and the counted data length of each column of data, and creating a table structure corresponding to the content of the data file to be extracted. The technical solution of the embodiment of the present invention can achieve automatic parsing of the content of the data file, adapt to general structured data files, and reduce the development workload.

Description

Technical Field

[0001] Embodiments of the present invention relate to the technical field of data analysis, and in particular, to a data file loading method, device, computer device, and storage medium. Background Art

[0002] In the daily application development within a bank, when there is a need to import data, the data is generally provided by an external system. The data file and its content format are usually text type files. The file content represents structured data in rows. Each piece of data is concatenated with data items according to a pre-agreed special symbol. For example, taking the delimiter as |!, the content is as follows:

[0003] 001|!Zhang San|!13500000001|!20|!Fengtai District, Beijing

[0004] 002|!Li Si|!13500000002|!21|!Dongcheng District, Beijing ..........

[0006] Generally, for the business requirement of persisting the above file into a database, the organization unit providing the file will provide a table structure description document for this business file before providing the file data. Developers will create a data table structure in the database according to this table structure description document, and then develop a data import program to finally complete the business requirement.

[0007] Although the above table structure creation method can meet the business implementation needs, it will waste too much storage space and resources, and the processing method is not flexible. In addition, in some cases, such as when there is no organization unit responsible for maintaining historical file data, or when real-time dynamic business data provided during the system operation cannot be manually created with a table structure in advance, etc., there are no conditions to first create a table structure according to the description document and then insert data. Summary of the Invention

[0008] Embodiments of the present invention provide a data file loading method, device, computer device, and storage medium to automatically complete the parsing of the data file content, adapt to general structured data files, and reduce the development workload.

[0009] In a first aspect, embodiments of the present invention provide a data file loading method, which includes:

[0010] Obtain the content of the data file to be extracted loaded into the memory, and determine the delimiter of the content of the data file to be extracted;

[0011] Parse the content of the data file to be extracted line by line according to the splitting character, and generate the content of the target data file corresponding to the List data structure;

[0012] Check each column of data in the content of the target data file line by line based on regular expressions, and count the data length of each column;

[0013] Generate an SQL statement and create a table structure corresponding to the content of the data file to be extracted according to the content of the target data file after verification and the counted data length of each column.

[0014] Further, determining the splitting character of the content of the data file to be extracted includes:

[0015] Obtain a general splitting character, and determine the splitting character of the content of the data file to be extracted according to the number of data columns in each row of the content of the data file to be extracted after splitting by the general splitting character.

[0016] Further, generating the content of the target data file corresponding to the List data structure includes:

[0017] Take each row of data in the parsed content of the data file to be extracted as a String array;

[0018] Store the String array corresponding to each row of data in the List data structure in order, and generate the content of the target data file corresponding to the List data structure.

[0019] Further, before checking each column of data in the content of the target data file line by line based on regular expressions, it also includes:

[0020] Obtain the field type of the content of the target data file, and generate a regular expression corresponding to each field type;

[0021] Checking each column of data in the content of the target data file line by line based on regular expressions includes:

[0022] Traverse the content of the target data file corresponding to the List data structure, and check the field type of each column of data in the content of the target data file line by line based on regular expressions.

[0023] Further, after checking each column of data in the content of the target data file line by line based on regular expressions, it also includes:

[0024] If the field type of the current column of data in the content of the target data file does not conform to the preset set of field type ranges, remove the field type corresponding to the current column of data;

[0025] If the field type of the current column data in the content of the target data file conforms to the preset field type range set, the field type corresponding to the current column data is retained.

[0026] Further, after counting the data lengths of each column of data, it further includes:

[0027] Taking the maximum data length among the data values corresponding to each column of data as the field definition length;

[0028] According to the content of the target data file after verification and the data lengths of each column of data counted, generate an SQL statement and create the table structure corresponding to the content of the data file to be extracted, including:

[0029] According to the content of the target data file after verification and the field definition length, generate an SQL statement and create the table structure corresponding to the content of the data file to be extracted.

[0030] Further, the data file loading method further includes:

[0031] Perform the data insertion operation corresponding to the content of the data file to be extracted according to the table structure.

[0032] In a second aspect, an embodiment of the present invention further provides a data file loading device, which includes:

[0033] A file content extraction module, configured to obtain the content of the data file to be extracted loaded into the memory and determine the delimiter of the content of the data file to be extracted;

[0034] A target data file content generation module, configured to parse the content of the data file to be extracted line by line according to the delimiter and generate the target data file content corresponding to the List data structure;

[0035] A table structure analysis module, configured to verify each column of data in the content of the target data file line by line based on a regular expression and count the data lengths of each column of data;

[0036] A table structure generation module, configured to generate an SQL statement and create the table structure corresponding to the content of the data file to be extracted according to the content of the target data file after verification and the data lengths of each column of data counted.

[0037] In a third aspect, an embodiment of the present invention further provides a computer device, which includes:

[0038] One or more processors;

[0039] A storage device, configured to store multiple programs,

[0040] When at least one of the multiple programs is executed by the one or more processors, the one or more processors are caused to implement a data file loading method provided in the embodiment of the first aspect of the present invention.

[0041] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements a data file loading method provided in the embodiment of the first aspect of the present invention.

[0042] The technical solution of the embodiment of the present invention includes: obtaining the content of the data file to be extracted loaded into the memory, and determining the delimiter of the content of the data file to be extracted; parsing the content of the data file to be extracted line by line according to the delimiter, and generating the target data file content corresponding to the List data structure; validating each column of data in the target data file content line by line based on a regular expression, and counting the data length of each column of data; generating an SQL statement according to the validated target data file content and the counted data length of each column of data, and creating a table structure corresponding to the content of the data file to be extracted. This solves the problems that the current table structure creation method wastes storage space and resources, the processing method is not flexible, and neither of them has the processing condition of first creating a table structure according to the specification document and then inserting data, so as to realize automatic parsing of the data file content, adapt to general structured data files, and reduce the development workload. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a flowchart of a data file loading method provided in Embodiment 1 of the present invention;

[0044] Figure 2 is a flowchart of a data file loading method provided in Embodiment 2 of the present invention;

[0045] Figure 3 is a stage flowchart of a data file loading method provided in an embodiment of the present invention;

[0046] Figure 4 is a structural diagram of a data file loading device provided in Embodiment 3 of the present invention;

[0047] Figure 5 is a schematic hardware structure diagram of a computer device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] In order to make the objectives, technical solutions, and advantages of the present invention clearer, the following further describes the specific embodiments of the present invention in detail with reference to the accompanying drawings. It can be understood that the specific embodiments described herein are only for explaining the present invention, rather than limiting the present invention.

[0049] It should also be noted that, for ease of description, only the parts related to the present invention rather than all the content are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be implemented in parallel, concurrently, or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operations are completed, but it can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, and so on.

[0050] Embodiment 1

[0051] Figure 1 The figure is a flowchart of a data file loading method provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation where a table structure is created in real time based on the content of each column in a data file. The data file loading method can be executed by a data file loading device, and the data file loading device can be implemented in the form of software and / or hardware. The data file loading method specifically includes the following steps:

[0052] S110. Obtain the content of the data file to be extracted loaded into the memory, and determine the delimiter of the content of the data file to be extracted.

[0053] Among them, the content of the data file to be extracted is a data file provided by an external system, and the content of the data file to be extracted can be a text type file.

[0054] Specifically, through the file IO reading method, the content of the data file to be extracted is loaded into the memory, and then the content of the data file to be extracted loaded into the memory can be obtained.

[0055] It should be noted that the file IO reading method can read the file by bytes, and reading the file by bytes is mostly used to read text type files.

[0056] Based on the above embodiment, determining the delimiter of the content of the data file to be extracted includes: obtaining a general delimiter, and determining the delimiter of the content of the data file to be extracted according to the number of data columns in each row of the content of the data file to be extracted after being segmented by the general delimiter.

[0057] Among them, the general delimiter can be an existing general delimiter symbol. For example, the general delimiter can be symbols such as |!, @ @, etc.

[0058] Specifically, in the case where the delimiter is not clear, obtain a general delimiter character, and poll and judge the delimiter in the content of the data file to be extracted by the general delimiter character, and take the delimiter that is closest to the number of data columns in each row of the content of the data file to be extracted after splitting as the delimiter of the current content of the data file to be extracted.

[0059] S120. Parse the content of the data file to be extracted line by line according to the delimiter character, and generate the content of the target data file corresponding to the List data structure.

[0060] Among them, the content of the target data file is the data file content of the List data structure generated after judging the delimiter character for the content of the data file to be extracted and parsing it line by line according to the determined delimiter.

[0061] Specifically, according to the determined delimiter character of the content of the data file to be extracted, parse the content of the data file to be extracted line by line, take each row of data in the content of the data file to be extracted after parsing as a String array, store the String array corresponding to each row of data in the List data structure in order, and generate the content of the target data file corresponding to the List data structure.

[0062] S130. Check each column of data in the content of the target data file line by line based on a regular expression, and count the data length of each column of data.

[0063] Among them, a regular expression is also called a rule expression, and regular expressions are usually used to retrieve and replace text that conforms to a certain pattern or rule.

[0064] Further, before checking each column of data in the content of the target data file line by line based on a regular expression, it also includes: obtaining the field type of the content of the target data file, and generating a corresponding regular expression for each field type.

[0065] Among them, the field type can be a field type supported by the preset data file content. For example, the field type can be one or a combination of a string type varchar, an integer value type int, a decimal value type decimal, a date type datetime, etc. The specific field types supported by the data file content can be selected and set according to the actual business expansion.

[0066] It can be understood that each field type can correspond to a regular expression respectively.

[0067] Specifically, each column of data in the target data file content is verified line by line based on a regular expression, including: traversing the content of the target data file corresponding to the List data structure, and verifying the field type of each column of data in the target data file content line by line based on a regular expression.

[0068] Furthermore, if the field type of the current column of data in the target data file content does not conform to the preset field type range set, the field type corresponding to the current column of data is removed; if the field type of the current column of data in the target data file content conforms to the preset field type range set, the field type corresponding to the current column of data is retained.

[0069] In this embodiment, a preset field type range set can be predefined for the field type of each column.

[0070] When the verification of the field type of the current column of data in the target data file content based on a regular expression fails, that is, when the field type of the current column of data in the target data file content does not conform to the preset field type range set, the field type corresponding to the current column of data is removed, that is, the next line does not need to perform a repeated judgment on this field type for the corresponding column of data in the current column.

[0071] When the verification of the field type of the current column of data in the target data file content based on a regular expression passes, that is, when the field type of the current column of data in the target data file content conforms to the preset field type range set, the field type corresponding to the current column of data is retained.

[0072] It can be understood that after verifying each column of data in the target data file content line by line based on a regular expression, the finally retained field type is the field type that meets the current data column format.

[0073] In addition, it should be noted that if the finally retained field type is empty, the field type of the target data file content can use the default varchar type.

[0074] S140. Generate an SQL statement and create a table structure corresponding to the content of the data file to be extracted according to the verified content of the target data file and the data length of each column of data counted.

[0075] Based on the above embodiment, after counting the data length of each column of data, it further includes: taking the maximum data length among the data values corresponding to each column of data as the field definition length to meet the length requirements of all data in the column.

[0076] Specifically, according to the content of the target data file after verification and the defined field lengths, generate SQL statements and create the table structure corresponding to the content of the data file to be extracted.

[0077] Further, based on the above embodiments, perform data insertion operations corresponding to the content of the data file to be extracted according to the table structure.

[0078] Specifically, according to the data volume size of the content of the data file to be extracted, perform data insertion operations in a single or batch manner.

[0079] It should be noted that the row data with problems in the data insertion operation can be recorded and summarized separately, and reflected and explained in the generated data import result file, which is the content of the data file to be extracted including the table structure.

[0080] The technical solution of the embodiment of the present invention obtains the content of the data file to be extracted loaded into the memory, and determines the delimiter characters of the content of the data file to be extracted; parses the content of the data file to be extracted line by line according to the delimiter characters, and generates the content of the target data file corresponding to the List data structure; verifies each column of data in the content of the target data file line by line based on regular expressions, and counts the data lengths of each column of data; according to the content of the target data file after verification and the counted data lengths of each column of data, generate SQL statements and create the table structure corresponding to the content of the data file to be extracted. It solves the problems that the current table structure creation method wastes storage space and resources, the processing method is not flexible, and neither has the processing conditions of creating the table structure according to the specification document first and then inserting data, so as to realize the automatic parsing of the data file content, adapt to general structured data files, and reduce the development workload.

[0081] Embodiment Two

[0082] Figure 2 It is a flowchart of a data file loading method provided by the second embodiment of the present invention. Figure 3 It is a stage flowchart of a data file loading method provided by the embodiment of the present invention. This embodiment is optimized based on the above embodiments.

[0083] Correspondingly, the method of this embodiment specifically includes:

[0084] S210. Obtain the content of the data file to be extracted loaded into the memory by means of file I / O reading.

[0085] Exemplarily, refer to Figure 3, Step S210, as the file loading phase of the data file loading method in this embodiment, loads the content of the data file to be extracted into the memory through the file I / O reading method, and thus the content of the data file to be extracted loaded into the memory can be obtained.

[0086] S211. Obtain the general delimiter character, and determine the delimiter character of the content of the data file to be extracted according to the number of data columns in each row of the content of the data file to be extracted after being delimited by the general delimiter character.

[0087] S212. Parse the content of the data file to be extracted row by row according to the delimiter character.

[0088] S213. Take each row of data in the content of the data file to be extracted after parsing as a String array.

[0089] S214. Store the String array corresponding to each row of data in the List data structure in order and generate the target data file content corresponding to the List data structure.

[0090] Exemplarily, see Figure 3 , Steps S211 to S214, as the file content extraction phase of the data file loading method in this embodiment, first determine the delimiter symbol of the content of the data file to be extracted, and further parse row by row according to the determined delimiter symbol, extract the data structure of the content of the data file to be extracted, and generate the target data file content corresponding to the List data structure.

[0091] S215. Obtain the field type of the target data file content and generate a regular expression corresponding to each field type.

[0092] S216. Traverse the target data file content corresponding to the List data structure, verify the field type of each column of data in the target data file content row by row based on the regular expression, and count the data length of each column of data.

[0093] Further, if the field type of the current column of data in the target data file content does not conform to the preset field type range set, then remove the field type corresponding to the current column of data; if the field type of the current column of data in the target data file content conforms to the preset field type range set, then retain the field type corresponding to the current column of data.

[0094] S217. Take the maximum data length among the data values corresponding to each column of data as the field definition length.

[0095] S218. Generate an SQL statement according to the verified target data file content and the field definition length, and create the table structure corresponding to the content of the data file to be extracted.

[0096] Exemplarily, refer to Figure 3 , steps S215 to S218 are the table structure analysis and creation phase of the data file loading method in this embodiment. First, the field type analysis is performed, and then the field length analysis is performed. According to the information analyzed above, SQL statements are generated and the table structure corresponding to the content of the data file to be extracted is created.

[0097] S219. Perform a data insertion operation corresponding to the content of the data file to be extracted according to the table structure.

[0098] Exemplarily, refer to Figure 3 , steps S215 to S218 are the data persistence phase of the data file loading method in this embodiment. According to the table structure, a data insertion operation corresponding to the content of the data file to be extracted is performed.

[0099] The technical solution of the embodiment of the present invention automatically analyzes and determines the field types of column data in the content of the data file to be extracted, allocates appropriate field types and data lengths, does not waste database space, determines and analyzes the appropriate field types based on the content of each column in the content of the data file to be extracted, and creates the table structure in real time. In addition, it automatically completes the parsing and storage of the content of the data file to be extracted, adapts to general structured files, and reduces the development workload.

[0100] Embodiment III

[0101] Figure 4 The structure diagram of a data file loading device provided in Embodiment III of the present invention is applicable to the situation where the table structure is created in real time based on the content of each column in the data file.

[0102] As Figure 4 shown, the data file loading device includes: a file content extraction module 410, a target data file content generation module 420, a table structure analysis module 430, and a table structure generation module 440, where:

[0103] The file content extraction module 410 is configured to obtain the content of the data file to be extracted loaded into the memory and determine the delimiter of the content of the data file to be extracted;

[0104] The target data file content generation module 420 is configured to parse the content of the data file to be extracted line by line according to the delimiter and generate the target data file content corresponding to the List data structure;

[0105] The table structure analysis module 430 is configured to perform a check on each column of data in the target data file content line by line based on a regular expression and count the data length of each column of data;

[0106] The table structure generation module 440 is configured to generate an SQL statement and create a table structure corresponding to the content of the data file to be extracted according to the verified content of the target data file and the data length of each column of data counted.

[0107] The data file loading device of this embodiment obtains the content of the data file to be extracted loaded into the memory and determines the delimiter of the content of the data file to be extracted; parses the content of the data file to be extracted row by row according to the delimiter and generates the target data file content corresponding to the List data structure; verifies each column of data in the target data file content row by row based on a regular expression, and counts the data length of each column of data; generates an SQL statement and creates a table structure corresponding to the content of the data file to be extracted according to the verified content of the target data file and the data length of each column of data counted. It solves the problems that the current table structure creation method wastes storage space and resources, the processing method is not flexible, and neither has the processing condition of first creating a table structure according to the specification document and then inserting data, so as to realize automatically parsing the content of the data file, adapting to general structured data files, and reducing the development workload.

[0108] Based on the above embodiments, determining the delimiter of the content of the data file to be extracted includes:

[0109] Obtain a general delimiter, and determine the delimiter of the content of the data file to be extracted according to the number of data columns in each row of the content of the data file to be extracted after being segmented by the general delimiter.

[0110] Based on the above embodiments, generating the target data file content corresponding to the List data structure includes:

[0111] Take each row of data in the parsed content of the data file to be extracted as a String array.

[0112] Store the String array corresponding to each row of data in the List data structure in order, and generate the target data file content corresponding to the List data structure.

[0113] Based on the above embodiments, before verifying each column of data in the target data file content row by row based on a regular expression, it further includes:

[0114] Obtain the field type of the target data file content, and generate a regular expression corresponding to each field type.

[0115] Verifying each column of data in the target data file content row by row based on a regular expression includes:

[0116] Traverse the content of the target data file corresponding to the List data structure, and check the field type of each column of data in the content of the target data file line by line based on a regular expression.

[0117] Based on the above embodiments, after checking each column of data in the content of the target data file line by line based on a regular expression, it further includes:

[0118] If the field type of the current column of data in the content of the target data file does not conform to the preset set of field type ranges, then remove the field type corresponding to the current column of data;

[0119] If the field type of the current column of data in the content of the target data file conforms to the preset set of field type ranges, then retain the field type corresponding to the current column of data.

[0120] Based on the above embodiments, after counting the data lengths of each column of data, it further includes:

[0121] Take the maximum data length among the data values corresponding to each column of data as the field definition length;

[0122] Generate an SQL statement and create the table structure corresponding to the content of the data file to be extracted according to the content of the target data file after verification and the counted data lengths of each column of data, including:

[0123] Generate an SQL statement and create the table structure corresponding to the content of the data file to be extracted according to the content of the target data file after verification and the field definition length.

[0124] Based on the above embodiments, the data file loading device further includes:

[0125] Execute the data insertion operation corresponding to the content of the data file to be extracted according to the table structure.

[0126] The data file loading device provided in the above embodiments can execute the data file loading method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the data file loading method.

[0127] Embodiment 4

[0128] Figure 5 It is a schematic structural diagram of a computer device provided in Embodiment 4 of the present invention. As Figure 5 shown, the computer device includes a processor 510, a memory 520, an input device 530, and an output device 540; the number of processors 510 in the computer device can be one or more. Figure 5Taking a processor 510 as an example; the processor 510, memory 520, input device 530, and output device 540 in the computer device can be connected through a bus or other means. Figure 5 Taking the connection through the bus as an example.

[0129] The memory 520, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the data file loading method in the embodiments of the present invention (for example, the file content extraction module 410, target data file content generation module 420, table structure analysis module 430, and table structure generation module 440 in the data file loading device). The processor 510 executes various functional applications and data processing of the computer device by running the software programs, instructions, and modules stored in the memory 520, that is, implementing the above-mentioned data file loading method.

[0130] The memory 520 may mainly include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory 520 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other non-volatile solid-state storage devices. In some instances, the memory 520 may further include a memory remotely set relative to the processor 510, and these remote memories can be connected to the computer device through a network. Examples of the above network include but are not limited to the Internet, enterprise intranet, local area network, mobile communication network, and their combinations.

[0131] The input device 530 can be used to receive input digital or character information, and generate key signal inputs related to the user settings and function controls of the computer device. The output device 540 may include a display device such as a display screen.

[0132] Embodiment Five

[0133] Embodiment Five of the present invention further provides a storage medium containing computer-executable instructions, and the computer-executable instructions are used to execute a data file loading method when executed by a computer processor. The data file loading method includes:

[0134] Obtain the data file content to be extracted loaded into the memory, and determine the delimiter characters of the data file content to be extracted;

[0135] Parse the data file content to be extracted line by line according to the delimiter characters, and generate the target data file content corresponding to the List data structure;

[0136] Check each column of data in the content of the target data file line by line based on regular expressions, and count the data length of each column of data.

[0137] Generate an SQL statement and create the table structure corresponding to the content of the data file to be extracted according to the content of the target data file after verification and the data length of each column of data counted.

[0138] Of course, for a storage medium containing computer-executable instructions provided by an embodiment of the present invention, the computer-executable instructions are not limited to the method operations described above, and can also execute related operations in the data file loading method provided by any embodiment of the present invention.

[0139] From the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software and necessary general-purpose hardware. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk, or optical disc of a computer, and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments of the present invention.

[0140] It should be noted that in the embodiments of the above data file loading device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of mutual distinction and do not limit the protection scope of the present invention.

[0141] Note that the above is only a preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and more other equivalent embodiments can be included without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A data file loading method, characterized in that, including: Obtain the content of the data file to be extracted loaded into the memory, and determine the delimiter of the content of the data file to be extracted; Parse the content of the data file to be extracted line by line according to the delimiter, and generate the content of the target data file corresponding to the List data structure; Check each column of data in the content of the target data file line by line based on a regular expression, and count the data length of each column of data; wherein, the content of the check is the field type; Generate an SQL statement and create a table structure corresponding to the content of the data file to be extracted according to the content of the target data file after verification and the data length of each column of data counted; Determine the delimiter of the content of the data file to be extracted, including: Obtain a general delimiter, and determine the delimiter of the content of the data file to be extracted according to the number of data columns in each row of the content of the data file to be extracted after being delimited by the general delimiter; The step of determining the delimiter of the content of the data file to be extracted according to the number of data columns in each row of the content of the data file to be extracted after being delimited by the general delimiter includes: polling and judging the delimiter in the content of the data file to be extracted through the general delimiter, and taking the delimiter closest to the number of data columns in each row of the content of the data file to be extracted after being delimited as the delimiter of the current content of the data file to be extracted.

2. The data file loading method according to claim 1, wherein Generate the content of the target data file corresponding to the List data structure, including: Take each row of data in the parsed content of the data file to be extracted as a String array; Store the String array corresponding to each row of data in the List data structure in order, and generate the content of the target data file corresponding to the List data structure.

3. The data file loading method according to claim 1, characterized in that Before checking each column of data in the content of the target data file line by line based on a regular expression, it further includes: Obtain the field type of the content of the target data file, and generate a regular expression corresponding to each field type; Checking each column of data in the content of the target data file line by line based on a regular expression includes: Traverse the content of the target data file corresponding to the List data structure, and check the field type of each column of data in the content of the target data file line by line based on a regular expression.

4. The data file loading method according to claim 3, wherein After checking each column of data in the content of the target data file line by line based on a regular expression, it further includes: If the field type of the current column of data in the content of the target data file does not conform to the preset field type range set, remove the field type corresponding to the current column of data; If the field type of the current column of data in the content of the target data file conforms to the preset field type range set, retain the field type corresponding to the current column of data.

5. The data file loading method according to claim 1, characterized in that After counting the data length of each column of data, it further includes: Take the maximum data length among the data values corresponding to each column of data as the field definition length; Generate an SQL statement and create a table structure corresponding to the content of the data file to be extracted according to the content of the target data file after verification and the data length of each column of data counted, including: Generate an SQL statement and create a table structure corresponding to the content of the data file to be extracted according to the verified content of the target data file and the defined length of the fields.

6. The data file loading method according to claim 1, characterized in that The data file loading method further includes: Perform a data insertion operation corresponding to the content of the data file to be extracted according to the table structure.

7. A data file loading device, characterized in that, It includes: A file content extraction module, configured to obtain the content of the data file to be extracted loaded into the memory and determine the delimiter of the content of the data file to be extracted; A target data file content generation module, configured to parse the content of the data file to be extracted line by line according to the delimiter and generate the target data file content corresponding to the List data structure; A table structure analysis module, configured to verify each column of data in the target data file content line by line based on a regular expression and count the data length of each column of data; wherein, the content of the verification is the field type; A table structure generation module, configured to generate an SQL statement and create a table structure corresponding to the content of the data file to be extracted according to the verified content of the target data file and the counted data length of each column of data; Determining the delimiter of the content of the data file to be extracted includes: Obtain a general delimiter, and determine the delimiter of the content of the data file to be extracted according to the number of data columns in each row of the content of the data file to be extracted after being split by the general delimiter; the determining the delimiter of the content of the data file to be extracted according to the number of data columns in each row of the content of the data file to be extracted after being split by the general delimiter includes: polling and judging the delimiter in the content of the data file to be extracted through the general delimiter, and taking the delimiter closest to the number of data columns in each row of the content of the data file to be extracted after being split as the delimiter of the current content of the data file to be extracted.

8. A computer device, characterized in that, The computer device includes: One or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the data file loading method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the data file loading method according to any one of claims 1-6.

Images