Data import method, device, computer equipment and storage medium

Abstract

The present application relates to a data import method, device, computer equipment and storage medium. The method comprises: in response to an uploaded file list, according to different types of files in the file list, reading and displaying the characteristic information of each file in the file list by type; in response to generating an import task instruction, generating a corresponding file import task for a target file, the target file is determined based on the characteristic information annotation of each file; executing the file import task, reading the data column attribute information in the target file corresponding to the file import task, and according to the correspondence between the preset data column attribute information and the data table attribute field, importing the data in each data column of the target file into the corresponding attribute field of the database data table. The use of this method can make files of multiple different formats uniformly imported into the database.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data import method, device, computer equipment and storage medium. Background Art

[0002] With the development of information technology, there are more and more scenarios for data exchange in the form of files. When exchanging data between different systems, the file data needs to be formatted and uniformly imported into the database, so that it is convenient to query and obtain standardized data in the database for statistical analysis. In order to ensure the smooth transmission and entry of files between the two systems, the two parties need to pre-agree on the file format type, encoding rules, field content, etc. before exchanging data. The receiving party receives and parses the file data of the specified format, encoding, and field order according to the agreed entry rules.

[0003] However, due to the diversity of file formats and file sizes, databases with pre-set entry rules cannot support multiple different types of file data, and cannot uniformly process multiple file formats. If there is dirty data or abnormal data in the file (for example, too many or too few columns of data), all data import will fail. Summary of the invention

[0004] Based on this, it is necessary to provide a data import method, device, computer equipment and storage medium to address the above technical problems.

[0005] A data import method, the method comprising:

[0006] In response to the uploaded file list, according to different types of files in the file list, characteristic information of each file in the file list is read and displayed by type;

[0007] In response to the instruction to generate an import task, generating a corresponding file import task for a target file, wherein the target file is determined based on the feature information annotation of each file;

[0008] Execute the file import task, read the data column attribute information in the target file corresponding to the file import task, and import the data in each data column of the target file into the attribute field corresponding to the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0009] In one embodiment, in response to the uploaded file list, according to different types of files in the file list, characteristic information of each file in the file list is read and displayed by type, including:

[0010] In response to a file list uploaded by a user, reading a file format of each file in the file list;

[0011] If there are files in the file list in a compressed file format, decompress the files in the compressed file format to obtain a file list in which all files are in a decompressed file format;

[0012] The characteristic information of each file in the file list of the decompressed file format is read and displayed by type.

[0013] In one embodiment, after generating a corresponding file import task for the target file in response to the generating import task instruction, the method further includes:

[0014] Generate data table attribute labels according to the acquired data table attribute requirement information;

[0015] Matching the data table attribute tag with the file import task to obtain the file import task carrying the data table attribute tag as the file import task to be processed;

[0016] Generate and display the list of file import tasks to be processed according to the order of task identifiers carried by the file import tasks to be processed;

[0017] The executing the file import task, reading the data column attribute information in the target file corresponding to the file import task, and importing the data in each data column of the target file into the attribute field corresponding to the database data table according to the preset correspondence between the data column attribute information and the data table attribute field, includes:

[0018] In the list of pending file import tasks, the pending file import tasks are executed in sequence according to the order of the task identifications;

[0019] For each of the file import tasks to be processed, the data column attribute information in the target file corresponding to the file import task to be processed is read, and according to the correspondence between the preset data column attribute information and the data table attribute field, the data in each data column of the target file is imported into the corresponding attribute field of the database data table.

[0020] In one embodiment, for each of the to-be-processed file import tasks, reading the data column attribute information in the target file corresponding to the to-be-processed file import task, and importing the data in each data column of the target file into the corresponding attribute field of the database data table according to the preset correspondence between the data column attribute information and the data table attribute field, comprises:

[0021] For each of the to-be-processed file import tasks, performing a label configuration check on the to-be-processed file import task;

[0022] If the data table attribute tag matched by the file import task to be processed satisfies the preset tag configuration rule, the data column attribute information in the target file corresponding to the file import task to be processed is read, and according to the correspondence between the preset data column attribute information and the data table attribute field, the data in each data column of the target file is imported into the corresponding attribute field of the database data table.

[0023] In one embodiment, the executing the file import task, reading the data column attribute information in the target file corresponding to the file import task, and importing the data in each data column of the target file into the attribute field corresponding to the database data table according to the preset correspondence between the data column attribute information and the data table attribute field, includes:

[0024] Execute the file import task, pre-process the data contained in each data column in the target file, and obtain pre-processed standard data;

[0025] Reading attribute information of each data column in the target file, and retrieving the correspondence between the attribute information of each data column and the attribute field of the data table pre-stored in the database according to the attribute information of each data column;

[0026] According to the corresponding relationship, the standard data corresponding to the attribute information of each data column is imported into the corresponding data table attribute field of the database.

[0027] In one embodiment, the executing the file import task and preprocessing the data contained in each data column in the target file to obtain preprocessed standard data includes:

[0028] In response to the determined target correction type and the corresponding target correction rule, data correction is performed on the data in the target file to obtain first target file data after data correction;

[0029] In response to the determined data cleaning rule, the first target file data is cleaned to obtain second target file data, and the second target file data is used as preprocessed standard data.

[0030] In one embodiment, the reading of each data column attribute information in the target file and, according to each data column attribute information, retrieving the correspondence between the data column attribute information and the data table attribute field pre-stored in the database includes:

[0031] Read and display the attribute information of each data column in the target file;

[0032] In response to the determined data import rule and the target data table, configuring the correspondence between the data column attribute information of the target file and the data table attribute field of the target data table according to the data import rule;

[0033] The corresponding relationship is checked, and if the corresponding relationship meets the preset verification rule, the corresponding relationship after the verification is retrieved and imported as the standard corresponding relationship.

[0034] In one embodiment, the method further comprises:

[0035] If the corresponding relationship does not satisfy the preset verification rule, in response to the received user configuration modification instruction, the corresponding relationship is modified so that the corresponding relationship satisfies the preset verification rule.

[0036] A data import device, comprising:

[0037] A reading and displaying module, configured to respond to an uploaded file list and read and display characteristic information of each file in the file list by type according to different types of files in the file list;

[0038] A task generation module, configured to generate a corresponding file import task for a target file in response to a generate import task instruction, wherein the target file is determined based on the feature information annotation of each file;

[0039] The execution import module is used to execute the file import task, read the data column attribute information in the target file corresponding to the file import task, and import the data in each data column of the target file into the corresponding attribute field of the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0040] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0041] In response to the uploaded file list, according to different types of files in the file list, characteristic information of each file in the file list is read and displayed by type;

[0042] In response to the instruction to generate an import task, generating a corresponding file import task for a target file, wherein the target file is determined based on the feature information annotation of each file;

[0043] Execute the file import task, read the data column attribute information in the target file corresponding to the file import task, and import the data in each data column of the target file into the attribute field corresponding to the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0044] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0045] In response to the uploaded file list, according to different types of files in the file list, characteristic information of each file in the file list is read and displayed by type;

[0046] In response to the instruction to generate an import task, generating a corresponding file import task for a target file, wherein the target file is determined based on the feature information annotation of each file;

[0047] Execute the file import task, read the data column attribute information in the target file corresponding to the file import task, and import the data in each data column of the target file into the attribute field corresponding to the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0048] The above-mentioned data import method, device, computer equipment and storage medium respond to the uploaded file list, read and display the characteristic information of each file in the file list according to the different types of files in the file list; respond to the generation of the import task instruction, generate a corresponding file import task for the target file, and the target file is determined based on the characteristic information annotation of each file; execute the file import task, read the data column attribute information in the target file corresponding to the file import task, and import the data in each data column of the target file into the corresponding attribute field of the database data table according to the preset correspondence between the data column attribute information and the data table attribute field. The use of this method can make file data of different types and formats uniformly imported into the database. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 An application environment diagram of a data import method in an embodiment;

[0050] Figure 2 A schematic diagram of a flow chart of a data import method in one embodiment;

[0051] Figure 3 This is a diagram showing an interface for displaying a list of uploaded files in one embodiment;

[0052] Figure 4 A schematic flow chart of a step of decompressing a compressed file in a file list in one embodiment;

[0053] Figure 5 A schematic diagram of a process for generating a file import task list in one embodiment;

[0054] Figure 6A diagram showing a display interface of a file import task list in an embodiment;

[0055] Figure 7 A schematic diagram of a process for sequentially processing file import tasks in one embodiment;

[0056] Figure 8 A flowchart of a step of checking the attribute label configuration of a file import task in one embodiment;

[0057] Fig. 9 A flowchart of a method for executing a file import task in one embodiment;

[0058] Fig.10 A diagram showing a display interface of a corresponding relationship between data column attribute information and data table attribute fields in an embodiment;

[0059] Fig.11 A schematic diagram of a flow chart of steps for preprocessing different types of data in one embodiment;

[0060] Fig.12 A diagram showing an import rule configuration display interface for a file import task in an embodiment;

[0061] Fig.13 A flowchart of steps for configuring and verifying the corresponding relationship of file import tasks in one embodiment;

[0062] Fig.14 is a structural block diagram of a data import device in an embodiment;

[0063] Fig.15 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0065] The data import method provided in this application can be applied to Figure 1In the application environment shown. The terminal 102 communicates with the server 104 through the network. The server 104 responds to the file list uploaded by the user through the terminal 102, and reads and displays the characteristic information of each file in the file list according to the different types of files in the file list; then, in response to the generation of the import task instruction triggered by the user, the target file generates a corresponding file import task, wherein the target file is determined based on the characteristic information annotation of each file; the file import task is executed, the data column attribute information in the target file corresponding to the file import task is read, and according to the preset correspondence between the data column attribute information and the data table attribute field, the data in each data column of the target file is imported into the corresponding attribute field of the database data table.

[0066] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, and tablet computers, and the server 104 may be implemented as an independent server or a server cluster consisting of multiple servers. Optionally, the database server may be a server independent of the server 104, so after the server 104 completes the target file reading and mapping configuration operation (obtaining the corresponding relationship between the data column attribute information and the data table attribute field), the server 104 may execute the file import task to import the target file into the database server.

[0067] In one embodiment, Figure 2 As shown, a data import method is provided, which is applied to Figure 1 The server 104 in the example is used as an example to illustrate, and the following steps are included:

[0068] Step 201 , in response to an uploaded file list, characteristic information of each file in the file list is read and displayed according to different types of files in the file list.

[0069] In implementation, the user accesses the web server through the client device and uploads the candidate import files to the web server. The web server (referred to as the computer device) responds to the file list uploaded by the user (including the set of all candidate import files), reads the candidate import files by type according to the different types of files in the file list, and displays the characteristic information of each file in the file list. The specific import file list is as follows: Figure 3 As shown, in the import file list, the user can check the target file, where different types of files can be but are not limited to Excel type, txt type, xml type, etc. The characteristic information of each file in the file list includes: file name, file size, file path, file type, etc.

[0070] Step 202, in response to the generation of the import task instruction, generates a corresponding file import task for the target file, where the target file is determined based on the feature information annotation of each file.

[0071] In implementation, the target file is determined based on the characteristic information annotation of each file, that is, the user checks the target file according to the characteristic information of each file in the file list displayed on the client display interface, determines the target file, and clicks the generate import file task button according to the determined target file. The computer device background responds to the instruction to generate the import task and generates a corresponding file import task for the target file.

[0072] Step 203, execute the file import task, read the data column attribute information in the target file corresponding to the file import task, and import the data in each data column of the target file into the corresponding attribute field of the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0073] During implementation, the computer device executes the file import task, reads the data column attribute information in the target file corresponding to the file import task, and imports the data in each data column of the target file into the corresponding attribute field of the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0074] In the above data import method, in response to the uploaded file list, according to the different types of files in the file list, the characteristic information of each file in the file list is read and displayed by type; in response to the generation of the import task instruction, the corresponding file import task is generated for the target file, and the target file is determined based on the characteristic information annotation of each file; the file import task is executed, the data column attribute information in the target file corresponding to the file import task is read, and according to the correspondence between the preset data column attribute information and the data table attribute field, the data in each data column of the target file is imported into the corresponding attribute field of the database data table. By adopting this method, file data of different types and formats can be uniformly imported into the database.

[0075] In one embodiment, Figure 4 As shown, the specific processing process of step 201 is as follows:

[0076] Step 2011, in response to the file list uploaded by the user, read the file format of each file in the file list.

[0077] In implementation, after the user uploads the candidate import file through the client device, the computer device responds to the file list uploaded by the user and reads the file format information of each file in the file list, that is, determines the file format (or file type) of the file based on the file name field suffix.

[0078] Step 212: If there are files in the file list in a compressed file format, the files in the compressed file format are decompressed to obtain a file list in a decompressed file format.

[0079] In implementation, if there are files in the file list in a compressed file format, the computer device first decompresses the files in the compressed file format to obtain a file list in a decompressed file format.

[0080] Step 2013, read and display the characteristic information of each file in the file list of the decompressed file format by type.

[0081] During implementation, the computer device reads all the decompressed files in the file list by type (for example, reading files in txt format as one category, and reading files in Excel format as one category), reads the characteristic information of each file, and then arranges and displays them by category on the display interface.

[0082] In this embodiment, the format type of the file uploaded by the user is preliminarily determined to determine whether there is a compressed format file in the file list, and then the compressed format file is decompressed so that all files in the file list can be previewed and the characteristic information of each file can be displayed on the display interface.

[0083] In one embodiment, Figure 5 As shown, after step 202, the method further includes:

[0084] Step 202 (a), generating a data table attribute tag according to the acquired data table attribute requirement information.

[0085] In implementation, the computer device generates a data table attribute label according to the acquired data table attribute requirement information. Specifically, different data tables are stored in the database, and each data table is configured according to specific business requirements. When the attribute information in the data table needs to be processed and supplemented for this import task, the data table attribute requirement information input by the user is acquired, and the corresponding data table attribute label is generated according to the data table attribute requirement information.

[0086] Specifically, in an import task, in addition to importing the data content in the target file into the data table of the database, the data table also needs to have other attribute data in addition to the target file data (for example, bill attributes, contract attributes, customer name attributes, etc.) in accordance with the company's business needs. These attributes will be generated into corresponding data table attribute tags.

[0087] Step 202 (b) matches the data table attribute tag with the file import task to obtain the file import task carrying the data table attribute tag as the file import task to be processed.

[0088] In implementation, the computer device matches the generated data table attribute tag with the file import task (ie, tags the corresponding file import task with the attribute tag), and obtains the file import task carrying the data table attribute tag as the file import task to be processed.

[0089] Step 202 (c), generating and displaying a list of file import tasks to be processed according to the order of task identifiers carried by the file import tasks to be processed.

[0090] In implementation, the computer device sorts the task identifiers of each pending file import task, generates a file import task list, and displays the file import task list on the display interface. Specifically, the file import task list, such as Figure 6 As shown, in the file import task list, the task identifier (also referred to as the task ID) can be a string data randomly matched by the computer device. In addition, the computer device can also perform process monitoring on the file import task. Therefore, the task type, task start time, execution duration, task status, current node and other information are also displayed in the file import task list. The information displayed in the specific file import task list is not limited in the embodiments of the present application.

[0091] On this basis, if Figure 7 As shown, the processing process of step 203 is:

[0092] Step 701: In the list of pending file import tasks, the pending file import tasks are executed in sequence according to the order of task identification.

[0093] In implementation, the computer device obtains a list of pending file import tasks, and executes each pending file import task in sequence according to the order of task identifiers in the list.

[0094] Step 702, for each file import task to be processed, read the data column attribute information in the target file corresponding to the file import task to be processed, and import the data in each data column of the target file into the corresponding attribute field of the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0095] During implementation, for each file import task to be processed, the data column attribute information in the target file corresponding to the file import task to be processed is read, and according to the correspondence between the preset data column attribute information and the data table attribute field, the data in each data column of the target file is imported into the corresponding attribute field of the database data table.

[0096] In one embodiment, Figure 8 As shown, the specific processing process of step 2032 is as follows:

[0097] Step 801 : for each file import task to be processed, a tag configuration check is performed on the file import task to be processed.

[0098] During implementation, the computer device performs a label configuration check on each file import task to be processed. Specifically, the computer device performs a label configuration check on the file import task to be processed based on the configuration rules of the data table attribute labels that need to be matched by each preset file import task.

[0099] Step 802, if the data table attribute tag matched by the file import task to be processed meets the preset tag configuration rule, then read the data column attribute information in the target file corresponding to the file import task to be processed, and import the data in each data column of the target file into the corresponding attribute field of the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0100] During implementation, if the data table attribute tag matched by the file import task to be processed meets the preset tag configuration rules, the computer device imports the data in each data column of the target file into the corresponding attribute field of the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0101] Specifically, after detecting that the file import task meets the preset label configuration rules, the computer device imports the target file corresponding to the file import task into the database data table according to the file import process described in step 702, which will not be repeated in the embodiments of the present application.

[0102] In one embodiment, Fig. 9 As shown, the specific processing process of step 203 is as follows:

[0103] Step 2031, executing the file import task, preprocessing the data contained in each data column in the target file to obtain preprocessed standard data.

[0104] In practice, when the computer device executes a file import task, it is necessary to preprocess the data of each data column of the target file in the file import task to ensure that file data of different types and formats have uniform specifications, thereby obtaining preprocessed standard data.

[0105] Step 2032, reading the attribute information of each data column in the target file, and according to the attribute information of each data column, retrieving the correspondence between the attribute information of the data column and the attribute field of the data table pre-stored in the database.

[0106] In implementation, the computer device reads the attribute information of each data column in the target file, and retrieves the correspondence between the attribute information of each data column and the attribute field of the data table pre-stored in the database according to the attribute information of each data column. Fig.10 As shown, according to the data column attribute "insurance policy number" in the target file, the corresponding relationship between the data column attribute information and the data table field is queried as follows: insurance policy number——PolicyNumber1.

[0107] Step 2033: According to the corresponding relationship, the standard data corresponding to the attribute information of each data column is imported into the corresponding data table attribute field of the database.

[0108] In implementation, the computer device imports the pre-processed standard data corresponding to the attribute information of each data column in the target file into the corresponding data table attribute field of the database according to the corresponding relationship.

[0109] In one embodiment, Fig.11 As shown, the specific processing process of step 2031 is as follows:

[0110] Step 1101 , in response to the determined target correction type and the corresponding target correction rule, data in the target file is corrected to obtain first target file data after the data correction.

[0111] In implementation, the user triggers the command by clicking on the client device, determines the target correction type, and the specific data correction rule configuration interface, such as Fig.12 As shown, the configuration entry and the corresponding check items and configuration types are displayed in the data correction rule configuration interface. For the target correction type marked by the user, the computer device responds to the determined target correction type and the target correction rule corresponding to the target correction type, and corrects the data in the target file to obtain the first target file data after the data correction. Among them, the target correction type includes: general correction type (for example, data font type, font size, font color, etc.), date-specific correction type (for example, display format of date data) and data-specific correction type (for example, data value range correction).

[0112] Step 1102 , in response to the determined data cleaning rule, perform data cleaning on the first target file data to obtain second target file data, and use the second target file data as pre-processed standard data.

[0113] In implementation, the user triggers entry into a data cleaning interface by clicking (or checking) an instruction, determines (configures) data cleaning rules in the cleaning interface, and performs data cleaning on the first target file data in response to the determined data cleaning rules to remove missing data or erroneous data therein to obtain second target file data, and uses the second target file data as preprocessed standard data.

[0114] Optionally, if there are still non-obvious abnormal data (i.e., abnormal data that cannot be corrected and cleaned) in the second target file data after preprocessing, when the second target file data corresponding to the attribute information of each data column is imported into the data table attribute field corresponding to the database, this type of non-obvious abnormal data is discarded (i.e., marked and skipped first) to import the remaining normal data, and then, the discarded non-obvious abnormal data is stored separately in a certain error data file according to the mark, so as to facilitate the later analysis of the error data.

[0115] In one embodiment, Fig.13 As shown, the specific processing process of step 2032 is as follows:

[0116] Step 1301, read and display the attribute information of each data column in the target file.

[0117] In implementation, the computer device reads and displays the attribute information of each data column in the target file.

[0118] Step 1302 , in response to the determined data import rule and the target data table, configure the correspondence between the data column attribute information of the target file and the data table attribute fields of the target data table according to the data import rule.

[0119] In implementation, the user determines the data import rules by checking a box, and the computer device configures the correspondence between the data column attribute information of the target file and the data table attribute fields of the target data table according to the determined data import rules and the target data table.

[0120] Step 1303, the corresponding relationship is checked. If the corresponding relationship meets the preset verification rules, the corresponding relationship after verification is retrieved and imported as the standard corresponding relationship.

[0121] In the implementation, the computer device verifies the correspondence between the obtained data column attribute information and the data table attribute field. If the correspondence meets the preset verification rules, the correspondence after verification is retrieved as the standard import correspondence for importing the data in the target file. At the same time, when data is imported according to the standard import correspondence, in the correspondence display interface, if a certain data in the target file is successfully imported according to the correspondence, the data record is displayed with a green background. If a certain data in the target file does not completely correspond to the data table attribute field (or there is no completely matching correspondence), the data record is displayed with a yellow background in the correspondence display interface. If a certain data in the target file does not completely correspond to the data table attribute field, the data record is displayed with a red background.

[0122] Optionally, if the corresponding relationship does not satisfy a preset verification rule, in response to a received user configuration modification instruction, the corresponding relationship is modified so that the corresponding relationship satisfies the preset verification rule.

[0123] Specifically, if the correspondence does not meet the preset verification rules, the user can modify the correspondence between the data column attribute information and the data table attribute fields in the data import rule configuration interface by dragging the table and other triggering operations (i.e., the user modifies the configuration instructions). At the same time, the user clicks the "Configuration Completed" button to trigger the computer device to recognize and end the configuration modification task.

[0124] It should be understood that although Figure 2 , 4 The steps in the flowcharts of -5, 7-9, 11, and 13 are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2 , 4 -5, 7-9, 11, at least part of the steps in 13 may include multiple steps or multiple stages. These steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0125] In one embodiment, Fig.14 As shown, a data import device 1400 is provided, comprising: a reading and displaying module 1410, a task generating module 1420 and an import executing module 1430, wherein:

[0126] The reading and displaying module 1410 is used to respond to the uploaded file list and read and display the characteristic information of each file in the file list according to the different types of files in the file list;

[0127] A task generation module 1420 is used to generate a corresponding file import task for a target file in response to a generate import task instruction, where the target file is determined based on the feature information annotation of each file;

[0128] The execution import module 1430 is used to execute the file import task, read the data column attribute information in the target file corresponding to the file import task, and import the data in each data column of the target file into the corresponding attribute field of the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0129] In one embodiment, the reading and displaying module 1410 is specifically used to respond to the file list uploaded by the user and read the file format of each file in the file list;

[0130] If there are files in the file list that are in compressed file format, decompress the files in compressed file format to obtain a file list that is all in decompressed file format;

[0131] Read and display the characteristic information of each file in the file list of the decompressed file format by type.

[0132] In one embodiment, the apparatus 1400 further includes:

[0133] Generate data table attribute labels according to the acquired data table attribute requirement information;

[0134] Match the data table attribute label with the file import task to obtain the file import task carrying the data table attribute label as the file import task to be processed;

[0135] Generate and display a list of pending file import tasks according to the order of task identifiers carried by the pending file import tasks;

[0136] The execution import module 1430 is specifically used to execute the file import tasks to be processed in sequence according to the task identification order in the list of file import tasks to be processed;

[0137] For each file import task to be processed, read the data column attribute information in the target file corresponding to the file import task to be processed, and import the data in each data column of the target file into the corresponding attribute field of the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0138] In one embodiment, the execution import module 1430 is specifically used to perform a tag configuration check on each file import task to be processed;

[0139] If the data table attribute label matched by the file import task to be processed meets the preset label configuration rules, the data column attribute information in the target file corresponding to the file import task to be processed is read, and according to the correspondence between the preset data column attribute information and the data table attribute field, the data in each data column of the target file is imported into the corresponding attribute field of the database data table.

[0140] In one embodiment, the execution import module 1430 is further specifically used to execute the file import task, preprocess the data contained in each data column in the target file, and obtain the preprocessed standard data;

[0141] Read the attribute information of each data column in the target file, and retrieve the correspondence between the attribute information of each data column and the attribute field of the data table pre-stored in the database according to the attribute information of each data column;

[0142] According to the corresponding relationship, the standard data corresponding to the attribute information of each data column is imported into the corresponding data table attribute field of the database.

[0143] In one embodiment, the execution import module 1430 is further specifically used to perform data correction on the data in the target file in response to the determined target correction type and the corresponding target correction rule, to obtain the first target file data after the data correction;

[0144] In response to the determined data cleaning rule, the first target file data is cleaned to obtain second target file data, and the second target file data is used as preprocessed standard data.

[0145] In one embodiment, the execution import module 1430 is further specifically used to read and display attribute information of each data column in the target file;

[0146] In response to the determined data import rule and the target data table, configuring the correspondence between the data column attribute information of the target file and the data table attribute fields of the target data table according to the data import rule;

[0147] The corresponding relationship is checked. If the corresponding relationship meets the preset verification rules, the corresponding relationship after verification is retrieved and imported as the standard corresponding relationship.

[0148] For the specific definition of the data import device, please refer to the definition of the data import method above, which will not be repeated here. Each module in the above data import device can be implemented in whole or in part by software, hardware and a combination thereof. The above modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0149] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.15 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store file data to be imported. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data import method is implemented.

[0150] Those skilled in the art will understand that Fig.15 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0151] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0152] In response to the uploaded file list, according to different types of files in the file list, characteristic information of each file in the file list is read and displayed by type;

[0153] In response to the instruction to generate an import task, a corresponding file import task is generated for the target file, where the target file is determined based on the feature information annotation of each file;

[0154] Execute the file import task, read the data column attribute information in the target file corresponding to the file import task, and import the data in each data column of the target file into the corresponding attribute field of the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0155] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0156] In response to a file list uploaded by a user, reading a file format of each file in the file list;

[0157] If there are files in the file list that are in compressed file format, decompress the files in compressed file format to obtain a file list that is all in decompressed file format;

[0158] Read and display the characteristic information of each file in the file list of the decompressed file format by type.

[0159] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0160] Generate data table attribute labels according to the acquired data table attribute requirement information;

[0161] Match the data table attribute label with the file import task to obtain the file import task carrying the data table attribute label as the file import task to be processed;

[0162] A list of file import tasks to be processed is generated and displayed according to the order of task identifiers carried by the file import tasks to be processed.

[0163] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0164] In the list of pending file import tasks, the pending file import tasks are executed in sequence according to the order of task identification;

[0165] For each file import task to be processed, read the data column attribute information in the target file corresponding to the file import task to be processed, and import the data in each data column of the target file into the corresponding attribute field of the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0166] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0167] For each file import task to be processed, a label configuration check is performed on the file import task to be processed;

[0168] If the data table attribute label matched by the file import task to be processed meets the preset label configuration rules, the data column attribute information in the target file corresponding to the file import task to be processed is read, and according to the correspondence between the preset data column attribute information and the data table attribute field, the data in each data column of the target file is imported into the corresponding attribute field of the database data table.

[0169] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0170] Execute the file import task, pre-process the data contained in each data column in the target file, and obtain the pre-processed standard data;

[0171] Read the attribute information of each data column in the target file, and retrieve the correspondence between the attribute information of each data column and the attribute field of the data table pre-stored in the database according to the attribute information of each data column;

[0172] According to the corresponding relationship, the standard data corresponding to the attribute information of each data column is imported into the corresponding data table attribute field of the database.

[0173] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0174] In response to the determined target correction type and the corresponding target correction rule, data correction is performed on the data in the target file to obtain first target file data after data correction;

[0175] In response to the determined data cleaning rule, the first target file data is cleaned to obtain second target file data, and the second target file data is used as preprocessed standard data.

[0176] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0177] Read and display the attribute information of each data column in the target file;

[0178] In response to the determined data import rule and the target data table, configuring the correspondence between the data column attribute information of the target file and the data table attribute fields of the target data table according to the data import rule;

[0179] The corresponding relationship is checked. If the corresponding relationship meets the preset verification rules, the corresponding relationship after verification is retrieved and imported as the standard corresponding relationship.

[0180] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0181] In response to the uploaded file list, according to different types of files in the file list, characteristic information of each file in the file list is read and displayed by type;

[0182] In response to the instruction to generate an import task, a corresponding file import task is generated for the target file, where the target file is determined based on the feature information annotation of each file;

[0183] Execute the file import task, read the data column attribute information in the target file corresponding to the file import task, and import the data in each data column of the target file into the corresponding attribute field of the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0184] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0185] In response to a file list uploaded by a user, reading a file format of each file in the file list;

[0186] If there are files in the file list that are in compressed file format, decompress the files in compressed file format to obtain a file list that is all in decompressed file format;

[0187] Read and display the characteristic information of each file in the file list of the decompressed file format by type.

[0188] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0189] Generate data table attribute labels according to the acquired data table attribute requirement information;

[0190] Match the data table attribute label with the file import task to obtain the file import task carrying the data table attribute label as the file import task to be processed;

[0191] A list of file import tasks to be processed is generated and displayed according to the order of task identifiers carried by the file import tasks to be processed.

[0192] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0193] In the list of pending file import tasks, the pending file import tasks are executed in sequence according to the order of task identification;

[0194] For each file import task to be processed, read the data column attribute information in the target file corresponding to the file import task to be processed, and import the data in each data column of the target file into the corresponding attribute field of the database data table according to the correspondence between the preset data column attribute information and the data table attribute field.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0196] For each file import task to be processed, a label configuration check is performed on the file import task to be processed;

[0197] If the data table attribute label matched by the file import task to be processed meets the preset label configuration rules, the data column attribute information in the target file corresponding to the file import task to be processed is read, and according to the correspondence between the preset data column attribute information and the data table attribute field, the data in each data column of the target file is imported into the corresponding attribute field of the database data table.

[0198] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0199] Execute the file import task, pre-process the data contained in each data column in the target file, and obtain the pre-processed standard data;

[0200] Read the attribute information of each data column in the target file, and retrieve the correspondence between the attribute information of each data column and the attribute field of the data table pre-stored in the database according to the attribute information of each data column;

[0201] According to the corresponding relationship, the standard data corresponding to the attribute information of each data column is imported into the corresponding data table attribute field of the database.

[0202] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0203] In response to the determined target correction type and the corresponding target correction rule, data correction is performed on the data in the target file to obtain first target file data after data correction;

[0204] In response to the determined data cleaning rule, the first target file data is cleaned to obtain second target file data, and the second target file data is used as preprocessed standard data.

[0205] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0206] Read and display the attribute information of each data column in the target file;

[0207] In response to the determined data import rule and the target data table, configuring the correspondence between the data column attribute information of the target file and the data table attribute fields of the target data table according to the data import rule;

[0208] The corresponding relationship is checked. If the corresponding relationship meets the preset verification rules, the corresponding relationship after verification is retrieved and imported as the standard corresponding relationship.

[0209] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0210] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0211] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data import method, characterized in that: The method comprises: In response to the uploaded file list, according to different types of files in the file list, characteristic information of each file in the file list is read and displayed by type; In response to the generation of the import task instruction, a corresponding file import task is generated for the target file, wherein the target file is determined based on the characteristic information annotation of each file, wherein the user checks the target file according to the characteristic information of each file in the file list displayed on the client display interface to determine the target file; Execute the file import task, read the data column attribute information in the target file corresponding to the file import task, and import the data in each data column of the target file into the corresponding attribute field of the database data table according to the preset correspondence between the data column attribute information and the data table attribute field; The executing the file import task, reading the data column attribute information in the target file corresponding to the file import task, and importing the data in each data column of the target file into the attribute field corresponding to the database data table according to the preset correspondence between the data column attribute information and the data table attribute field, includes: Executing the file import task, and in response to the determined target correction type and the corresponding target correction rule, performing data correction on the data in the target file to obtain first target file data after data correction; In response to the determined data cleaning rule, the first target file data is cleansed to obtain second target file data, and the second target file data is used as preprocessed standard data; Reading attribute information of each data column in the target file, and retrieving the correspondence between the attribute information of each data column and the attribute field of the data table pre-stored in the database according to the attribute information of each data column; According to the corresponding relationship, the standard data corresponding to the attribute information of each data column is imported into the attribute field of the data table corresponding to the database; if there is still abnormal data that cannot be corrected and cleaned in the preprocessed second target file data, when the second target file data corresponding to the attribute information of each data column is imported into the attribute field of the data table corresponding to the database, the abnormal data mark of this type is skipped and the remaining normal data is imported, and the abnormal data is stored separately in the error data file according to the mark; The reading of each data column attribute information in the target file and, according to each data column attribute information, retrieving the correspondence between the data column attribute information and the data table attribute field pre-stored in the database includes: Read and display the attribute information of each data column in the target file; In response to the determined data import rule and the target data table, configuring the correspondence between the data column attribute information of the target file and the data table attribute field of the target data table according to the data import rule; The corresponding relationship is checked, and if the corresponding relationship meets the preset verification rule, the corresponding relationship after the verification is retrieved and imported as the standard corresponding relationship.

2. The method according to claim 1, characterized in that In response to the uploaded file list, according to different types of files in the file list, characteristic information of each file in the file list is read and displayed by type, including: In response to a file list uploaded by a user, reading a file format of each file in the file list; If there are files in the file list in a compressed file format, decompress the files in the compressed file format to obtain a file list in which all files are in a decompressed file format; The characteristic information of each file in the file list of the decompressed file format is read and displayed by type.

3. The method according to claim 1, characterized in that After generating a corresponding file import task for the target file in response to the generating import task instruction, the method further includes: Generate data table attribute labels according to the acquired data table attribute requirement information; Matching the data table attribute tag with the file import task to obtain the file import task carrying the data table attribute tag as the file import task to be processed; Generate and display the list of file import tasks to be processed according to the order of task identifiers carried by the file import tasks to be processed; The executing the file import task, reading the data column attribute information in the target file corresponding to the file import task, and importing the data in each data column of the target file into the attribute field corresponding to the database data table according to the preset correspondence between the data column attribute information and the data table attribute field, includes: In the list of pending file import tasks, the pending file import tasks are executed in sequence according to the order of the task identifications; For each of the file import tasks to be processed, the data column attribute information in the target file corresponding to the file import task to be processed is read, and according to the correspondence between the preset data column attribute information and the data table attribute field, the data in each data column of the target file is imported into the corresponding attribute field of the database data table.

4. The method according to claim 3, characterized in that For each of the to-be-processed file import tasks, the data column attribute information in the target file corresponding to the to-be-processed file import task is read, and according to the preset correspondence between the data column attribute information and the data table attribute field, the data in each data column of the target file is imported into the attribute field corresponding to the database data table, including: For each of the to-be-processed file import tasks, performing a label configuration check on the to-be-processed file import task; If the data table attribute tag matched by the file import task to be processed satisfies the preset tag configuration rule, the data column attribute information in the target file corresponding to the file import task to be processed is read, and according to the correspondence between the preset data column attribute information and the data table attribute field, the data in each data column of the target file is imported into the corresponding attribute field of the database data table.

5. The method according to claim 1, characterized in that The method further comprises: If the corresponding relationship does not satisfy the preset verification rule, in response to the received user configuration modification instruction, the corresponding relationship is modified so that the corresponding relationship satisfies the preset verification rule.

6. A data import device, characterized in that: The device comprises: A reading and displaying module, configured to respond to an uploaded file list and read and display characteristic information of each file in the file list by type according to different types of files in the file list; A task generation module is used to generate a corresponding file import task for a target file in response to a generate import task instruction, wherein the target file is determined based on the characteristic information annotation of each file, wherein the user selects the target file according to the characteristic information of each file in the file list displayed on the client display interface to determine the target file; An execution import module is used to execute the file import task, read the data column attribute information in the target file corresponding to the file import task, and import the data in each data column of the target file into the corresponding attribute field of the database data table according to the preset correspondence between the data column attribute information and the data table attribute field; The execution import module is also used to execute the file import task, in response to the determined target correction type and the corresponding target correction rule, perform data correction on the data in the target file to obtain the first target file data after the data correction; in response to the determined data cleaning rule, perform data cleaning on the first target file data to obtain the second target file data, and use the second target file data as the pre-processed standard data; read the attribute information of each data column in the target file, and according to the attribute information of each data column, retrieve the correspondence between the data column attribute information pre-stored in the database and the attribute field of the data table; According to the corresponding relationship, the standard data corresponding to the attribute information of each data column is imported into the attribute field of the data table corresponding to the database; if there is still abnormal data that cannot be corrected and cleaned in the preprocessed second target file data, when the second target file data corresponding to the attribute information of each data column is imported into the attribute field of the data table corresponding to the database, the abnormal data mark of this type is skipped and the remaining normal data is imported, and the abnormal data is stored separately in the error data file according to the mark; The execution import module is also used to read and display the attribute information of each data column in the target file; In response to the determined data import rule and the target data table, configuring the correspondence between the data column attribute information of the target file and the data table attribute field of the target data table according to the data import rule; The corresponding relationship is checked, and if the corresponding relationship meets the preset verification rule, the corresponding relationship after the verification is retrieved and imported as the standard corresponding relationship.

7. The device according to claim 6, characterized in that The reading and display module is specifically used for: In response to a file list uploaded by a user, reading a file format of each file in the file list; If there are files in the file list that are in compressed file format, decompress the files in compressed file format to obtain a file list that is all in decompressed file format; Read and display the characteristic information of each file in the file list of the decompressed file format by type.

8. The device according to claim 6, characterized in that The import module is also used to: For each file import task to be processed, a label configuration check is performed on the file import task to be processed; If the data table attribute label matched by the file import task to be processed meets the preset label configuration rules, the data column attribute information in the target file corresponding to the file import task to be processed is read, and according to the correspondence between the preset data column attribute information and the data table attribute field, the data in each data column of the target file is imported into the corresponding attribute field of the database data table.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images