Data import method and device, equipment, medium and program product

Abstract

The invention provides a data importing method and device, equipment, a medium and a program product, which can be applied to the technical fields of big data, databases and financial science and technology. The method comprises the following steps: processing a file name of a to-be-imported file by utilizing a character string segmentation method to obtain an extended name of the file name; a file header of the to-be-imported file is read through a file reading method corresponding to the extension name, and the file header comprises a plurality of field names; determining a plurality of data conversion rules of field values respectively corresponding to a plurality of field names of the to-be-imported file based on the extension name and the file header; based on a plurality of data conversion rules, converting the field values corresponding to the plurality of field names respectively to obtain converted field values; the multiple conversion field values are imported into a data table to be filled according to the corresponding relation between the field values and the field names, a target data table is obtained, and the data table to be filled is created in a database according to the file header.

Description

Technical Field

[0001] The present disclosure relates to the fields of big data, database, and financial technology, and specifically to a data import method, device, equipment, medium, and program product. Background Art

[0002] As business and scientific research become more and more dependent on data, the diversity and scale of data continue to grow, bringing new challenges to data storage and data processing. Storing and managing data through a database can simplify operations, so data files are usually imported into a database before processing to improve processing efficiency.

[0003] In the process of implementing the present disclosure, the inventors found that the prior art has at least the following problems: different data files usually come from different systems and have different file formats. The above-mentioned format diversity increases the complexity of the data import process. Therefore, it is necessary to manually configure data mapping and conversion rules, which has high labor costs, low processing efficiency, and difficulty in ensuring accuracy. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a data import method, apparatus, device, medium and program product.

[0005] According to a first aspect of the present disclosure, a data import method is provided, including: processing the file name of a file to be imported by using a string segmentation method to obtain an extension of the file name; reading a file header of the file to be imported by using a file reading method corresponding to the extension, wherein the file header includes multiple field names; based on the extension and the file header, determining multiple data conversion rules for field values ​​corresponding to each of the multiple field names of the file to be imported; based on the multiple data conversion rules, converting the field values ​​corresponding to each of the multiple field names to obtain converted field values; and importing the multiple converted field values ​​into a data table to be filled according to the correspondence between the field values ​​and the field names to obtain a target data table, wherein the data table to be filled is created in a database based on the file header.

[0006] According to an embodiment of the present disclosure, a file header of a file to be imported is read using a file reading method corresponding to an extension name, including: determining the file type of the file to be imported based on the extension name; determining the file reading method required to read the file type based on the file type; reading the file to be imported using the file reading method to determine the file content of the file to be imported, wherein the file content includes a file header and multiple field values; and determining multiple field names included in the file header based on the file content.

[0007] According to an embodiment of the present disclosure, based on the extension name and the file header, multiple data conversion rules for field values ​​corresponding to multiple field names of the file to be imported are determined, including: based on each field name, determining the initial data type of the field name in the file to be imported; based on the extension name, determining a historical file with the same file type and the same file header as the file to be imported; based on the historical file, determining the target data type of the field name in the database; and based on the initial data type and the target data type, determining the data conversion rule for the field value corresponding to the field name.

[0008] According to an embodiment of the present disclosure, data conversion rules include data format conversion rules and data cleaning rules; based on multiple data conversion rules, field values ​​corresponding to multiple field names are converted to obtain converted field values, including: for each field value, using the data format conversion rule to convert the field value into a format to obtain a target format field value; based on the data cleaning rules, determining the validity of the target format field value; and when the target format field value is valid, determining the converted field value based on the target format field value.

[0009] According to an embodiment of the present disclosure, multiple conversion field values ​​are imported into a data table to be filled according to the correspondence between field values ​​and field names to obtain a target data table, including: for each conversion field value, determining the field name corresponding to the conversion field value; importing the conversion field value into the column corresponding to the field name in the data table to be filled; and in the case that the file to be imported does not include the conversion field value that has not been imported into the data table to be filled, determining the data table to be filled as the target data table.

[0010] According to an embodiment of the present disclosure, a file name of a file to be imported is processed using a string segmentation method to obtain an extension of the file name, including: determining the file name of the file to be imported based on the file to be imported; processing the file name using a string segmentation method to determine the segmentation characters in the file name; and segmenting the file name using the segmentation characters to obtain a file description of the file to be imported and an extension of the file to be imported.

[0011] According to an embodiment of the present disclosure, the data import method also includes: obtaining the number of imported field values ​​in the data table to be filled; using a file reading method corresponding to the extension name to read the file to be imported and obtain the number of field values ​​in the file to be imported; and determining the import progress of the target data table based on the number of imported field values ​​and the number of field values ​​in the file to be imported.

[0012] A second aspect of the present disclosure provides a data import device, comprising:

[0013] The extension determination module is used to process the file name of the file to be imported using a string segmentation method to obtain the extension of the file name;

[0014] A file header determination module, used to read the file header of the file to be imported using a file reading method corresponding to the extension name, wherein the file header includes a plurality of field names;

[0015] A rule determination module, used for determining multiple data conversion rules of field values ​​corresponding to multiple field names of the file to be imported based on the extension name and the file header;

[0016] A field conversion module, configured to convert field values ​​corresponding to the plurality of field names respectively based on a plurality of data conversion rules to obtain converted field values; and

[0017] The data import module is used to import multiple conversion field values ​​into the data table to be filled according to the correspondence between the field values ​​and the field names to obtain the target data table, wherein the data table to be filled is created in the database according to the file header.

[0018] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0019] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0020] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor.

[0021] According to an embodiment of the present disclosure, the file name of the file to be imported is processed, the extension is determined, and the file reading method corresponding to the extension is used to read the file to be imported, and multiple field names in the file header are determined. Accurate and automatic reading of the file to be imported can be achieved according to the different extensions of different files to be imported, thereby improving the efficiency of file reading. Based on the extension and the file header, the data conversion rules of the multiple field values ​​in the file to be imported are determined, and the multiple field values ​​are converted. Based on different field names and extensions, the data conversion rules of the multiple field values ​​can be accurately and efficiently determined, thereby improving the efficiency and accuracy of data import. The converted field values ​​obtained by data conversion are imported into the filling data table according to the corresponding relationship with the field name, and the target data table is determined after the import is completed. The automation and accuracy of data import are further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0023] Figure 1 The application scenario diagram of the data import method, apparatus, device, medium and program product according to the embodiment of the present disclosure is schematically shown;

[0024] Figure 2 A flowchart of a data import method according to an embodiment of the present disclosure is schematically shown;

[0025] Figure 3 A flowchart of a data import method according to another embodiment of the present disclosure is schematically shown;

[0026] Figure 4 A structural block diagram of a data import device according to an embodiment of the present disclosure is schematically shown; and

[0027] Figure 5 A block diagram of an electronic device suitable for implementing a data import method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0028] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0029] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0030] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0031] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0032] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0033] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0034] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0035] In the process of processing heterogeneous data files, data integration tools can be used to import data files of different formats into the database, but data integration tools also require a lot of manual configuration, which saves limited labor costs. In addition, when the data format changes, data integration tools have poor flexibility and low accuracy.

[0036] An embodiment of the present disclosure provides a data import method, including: using a string segmentation method to process the file name of a file to be imported to obtain an extension of the file name; using a file reading method corresponding to the extension to read a file header of the file to be imported, wherein the file header includes multiple field names; based on the extension and the file header, determining multiple data conversion rules for field values ​​corresponding to each of the multiple field names of the file to be imported; based on the multiple data conversion rules, converting the field values ​​corresponding to each of the multiple field names to obtain converted field values; and importing the multiple converted field values ​​into a data table to be filled according to the correspondence between the field values ​​and the field names to obtain a target data table, wherein the data table to be filled is created in a database based on the file header.

[0037] Figure 1 The application scenario diagram of the data import method, apparatus, device, medium and program product according to the embodiments of the present disclosure is schematically shown.

[0038] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0039] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0040] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0041] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0042] It should be noted that the data import method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the data import device provided in the embodiment of the present disclosure can generally be set in the server 105. The data import method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the data import device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0044] The following will be based on Figure 1 The scene described by Figure 2~Figure 3 The data import method of the disclosed embodiment is described in detail.

[0045] Figure 2 The flowchart of the data importing method according to the embodiment of the present disclosure is schematically shown.

[0046] like Figure 2 As shown, the data import method of this embodiment includes operations S210 to S250.

[0047] In operation S210, the file name of the file to be imported is processed using a string segmentation method to obtain an extension of the file name.

[0048] According to an embodiment of the present disclosure, the file name of the file to be imported may be in the format of "file description.extension". The file name of the file to be imported is obtained, and the file name is segmented using a string segmentation method to determine the extension of the file to be imported.

[0049] In operation S220, a file header of the file to be imported is read using a file reading method corresponding to the extension name, wherein the file header includes a plurality of field names.

[0050] According to an embodiment of the present disclosure, a file reading method for reading the file to be imported is determined according to an extension name, wherein the extension name may include xlsx, csv, xml, json, etc.

[0051] According to an embodiment of the present disclosure, a file reading method is used to read the file content of a file to be imported, and a file header is determined from the file content, wherein the file header may be a column title of a data table. For files to be imported with different extensions, the method for determining the file header is also different, for example, the file header may be determined based on the title row of an xlsx or csv file, the file header may be determined based on the key in a key-value pair of a json file, or the file header may be determined based on information such as a column name of an xml file.

[0052] In operation S230 , a plurality of data conversion rules for field values ​​corresponding to a plurality of field names of the file to be imported are determined based on the extension name and the file header.

[0053] According to an embodiment of the present disclosure, a file header includes multiple field names, and conversion methods for each field name are determined respectively, wherein the conversion methods include data type conversion, data encoding method conversion, etc. The data type conversion includes, for example, conversion from int type data to char type data, from float type data to int type data, etc. The data encoding method conversion includes conversion from UTF-8 encoding method to ASCII encoding method, etc.

[0054] According to an embodiment of the present disclosure, the data conversion rules of the field value include rules from reading the field value in the file to be imported to completing the conversion of the field value. According to the file reading method determined based on the extension name and the conversion method of the field name, the data conversion rules of the field value corresponding to each field name can be determined.

[0055] In operation S240 , field values ​​corresponding to the plurality of field names are converted based on the plurality of data conversion rules to obtain converted field values.

[0056] According to an embodiment of the present disclosure, based on multiple data conversion rules, field values ​​corresponding to multiple field names are read, and data type conversion and data encoding method conversion are performed according to the conversion method to obtain the converted field value after the conversion is completed.

[0057] In operation S250, multiple converted field values ​​are imported into the data table to be filled according to the correspondence between the field values ​​and the field names to obtain a target data table, wherein the data table to be filled is created in the database according to the file header.

[0058] According to an embodiment of the present disclosure, a data table to be filled is created in a database according to multiple field names in a file header, wherein the data table to be filled includes multiple field names, and no field values ​​are filled under the field names.

[0059] According to an embodiment of the present disclosure, the correspondence between the conversion field value and the field name is determined, and multiple conversion field values ​​are imported into the columns to which the corresponding field names belong in the data table to be filled. After all conversion field values ​​are imported into the data table to be filled, a target data table is obtained.

[0060] According to an embodiment of the present disclosure, the file name of the file to be imported is processed, the extension is determined, and the file reading method corresponding to the extension is used to read the file to be imported, and multiple field names in the file header are determined. Accurate and automatic reading of the file to be imported can be achieved according to the different extensions of different files to be imported, thereby improving the efficiency of file reading. Based on the extension and the file header, the data conversion rules of the multiple field values ​​in the file to be imported are determined, and the multiple field values ​​are converted. Based on different field names and extensions, the data conversion rules of the multiple field values ​​can be accurately and efficiently determined, thereby improving the efficiency and accuracy of data import. The converted field values ​​obtained by data conversion are imported into the filling data table according to the corresponding relationship with the field name, and the target data table is determined after the import is completed. The automation and accuracy of data import are further improved.

[0061] According to an embodiment of the present disclosure, a file header of a file to be imported is read using a file reading method corresponding to an extension name, including: determining the file type of the file to be imported based on the extension name; determining the file reading method required to read the file type based on the file type; reading the file to be imported using the file reading method to determine the file content of the file to be imported, wherein the file content includes a file header and multiple field values; and determining multiple field names included in the file header based on the file content.

[0062] According to an embodiment of the present disclosure, the file reading methods for reading files to be imported of different file types are different. The file type of the file to be imported can be determined based on the extension name, and the file reading method for reading the file to be imported can be determined according to the file type.

[0063] According to an embodiment of the present disclosure, a file to be imported is read using a file reading method, and a file header of the file to be imported includes multiple field names and multiple field values, as well as correspondences between the multiple field names and the multiple field values.

[0064] According to an embodiment of the present disclosure, the file type of the file to be imported is determined based on the extension name, the file reading method is further determined, and the file to be imported is read using the file reading method, thereby determining multiple field names included in the file header. The file reading method corresponding to the file to be imported can be selected quickly and accurately, thereby improving the processing speed of data import.

[0065] According to an embodiment of the present disclosure, based on the extension name and the file header, multiple data conversion rules for field values ​​corresponding to multiple field names of the file to be imported are determined, including: based on each field name, determining the initial data type of the field name in the file to be imported; based on the extension name, determining a historical file with the same file type and the same file header as the file to be imported; based on the historical file, determining the target data type of the field name in the database; and based on the initial data type and the target data type, determining the data conversion rule for the field value corresponding to the field name.

[0066] According to an embodiment of the present disclosure, based on each field name, the initial data type of the field name in the file to be imported is determined, wherein the field value corresponding to the field name can be read and parsed to determine the initial data type of the field name in the file to be imported, and the initial data type can also be determined in the historical data conversion record based on the field name, the historical data conversion record includes the initial data type and target data type of the historical data, and the historical data is the data imported using the data import method.

[0067] According to an embodiment of the present disclosure, based on the extension name, a historical file having the same file type and the same file header as the file to be imported is determined, and based on the historical file, a target data type of the field name in the database is determined.

[0068] According to an embodiment of the present disclosure, the target data type may also be determined based on historical data conversion records.

[0069] According to an embodiment of the present disclosure, a data conversion rule for a field value corresponding to a field name is determined based on an initial data type and a target data type, wherein the data conversion rule is used to convert a field value of the initial data type into a field value of the target data type.

[0070] According to an embodiment of the present disclosure, the initial data type of the field name is determined based on the field name, and the target data type of the field name is determined using a historical file, which can ensure the correctness of the initial data type and the target data type, thereby ensuring the correctness of the data conversion rules determined based on the initial data type and the target data type.

[0071] According to an embodiment of the present disclosure, data conversion rules include data format conversion rules and data cleaning rules; based on multiple data conversion rules, field values ​​corresponding to multiple field names are converted to obtain converted field values, including: for each field value, using the data format conversion rule to convert the field value into a format to obtain a target format field value; based on the data cleaning rules, determining the validity of the target format field value; and when the target format field value is valid, determining the converted field value based on the target format field value.

[0072] According to an embodiment of the present disclosure, for each field value, the field value is format-converted using a data format conversion rule to obtain a target format field value, wherein the format conversion includes converting the data type and data encoding method of the field value.

[0073] According to an embodiment of the present disclosure, the validity of the target format field value is determined based on the data cleaning rule, wherein the data cleaning rule may include whether the target format field value allows null values, the character length of the target format field value, the value range of the target format field value, etc. For example, if the field name corresponding to the target format field value is "age", the target format field value data type may be int, the data cleaning rule is non-null, and the value range is greater than 0 and less than a certain upper limit value, if a certain target format field value is "-2", then according to the data cleaning rule, it can be determined that the target format field value is invalid.

[0074] According to an embodiment of the present disclosure, data cleaning is performed on invalid target format field values, valid target format field values ​​are retained, and based on all valid target format field values, conversion field values ​​are determined.

[0075] According to the embodiments of the present disclosure, for each field value, the field value is first formatted to obtain a target field value, and then the data cleaning rules are used to determine whether the target field value needs to be cleaned. This ensures that the format conversion of the target field value is correct, and cleans the data that does not meet the requirements of the data value range, character length, etc., further ensuring the correctness of the data in the converted field value, thereby improving the accuracy of data import.

[0076] According to an embodiment of the present disclosure, multiple conversion field values ​​are imported into a data table to be filled according to the correspondence between field values ​​and field names to obtain a target data table, including: for each conversion field value, determining the field name corresponding to the conversion field value; importing the conversion field value into the column corresponding to the field name in the data table to be filled; and in the case that the file to be imported does not include the conversion field value that has not been imported into the data table to be filled, determining the data table to be filled as the target data table.

[0077] According to an embodiment of the present disclosure, the correspondence between the converted field value and the field name is determined based on the correspondence between the field value and the converted field value and the correspondence between the field value and the field name. The column corresponding to the field name in the data table to be filled is determined, and the converted field value is imported into the column corresponding to the field name in the data table to be filled, thereby completing the import of the converted field value.

[0078] According to an embodiment of the present disclosure, all conversion field values ​​are imported into the data table to be filled according to the above method. After the conversion field values ​​corresponding to all field values ​​in the file to be imported are imported into the data table to be filled, the data table to be filled is determined as the target data table.

[0079] According to an embodiment of the present disclosure, for each conversion field value, the field name corresponding to it is determined, and the conversion field value is imported into the corresponding column in the data table to be filled. After all the conversion field values ​​are imported, the target data table is determined. Through the correspondence between the conversion field value and the field value, field name and column in the data table to be filled, the conversion field value can be accurately filled into the corresponding column, thereby improving the accuracy of data import.

[0080] According to an embodiment of the present disclosure, a file name of a file to be imported is processed using a string segmentation method to obtain an extension of the file name, including: determining the file name of the file to be imported based on the file to be imported; processing the file name using a string segmentation method to determine the segmentation characters in the file name; and segmenting the file name using the segmentation characters to obtain a file description of the file to be imported and an extension of the file to be imported.

[0081] According to an embodiment of the present disclosure, a file name is determined based on the file to be imported, wherein the file path of the file to be imported can be obtained first, and the basename method or pathlib library in Python can be used to determine the file name of the file to be imported from the file path.

[0082] According to an embodiment of the present disclosure, a string segmentation method is used to parse the file name to determine the segmentation character in the file name. For example, when the file name format is "file description.extension", the segmentation character can be determined to be ".".

[0083] According to an embodiment of the present disclosure, a file name is segmented based on a segmentation character using a string segmentation method to obtain a file description of the file to be imported and an extension of the file to be imported, wherein the string segmentation method may be a split method, etc., wherein the segmentation character is passed as a parameter to the split method, and the split method is called using the file name to complete the segmentation of the file name.

[0084] According to an embodiment of the present disclosure, the file name of the file to be imported and the separator characters in the file name are determined to segment the file name and determine the extension of the file name. The file name can be accurately segmented by characters according to the file names of different system environments to determine the extension, so that the file to be imported can be read and subsequently operated based on the extension, thereby improving the efficiency of data import.

[0085] According to an embodiment of the present disclosure, the data import method also includes: obtaining the number of imported field values ​​in the data table to be filled; using a file reading method corresponding to the extension name to read the file to be imported and obtain the number of field values ​​in the file to be imported; and determining the import progress of the target data table based on the number of imported field values ​​and the number of field values ​​in the file to be imported.

[0086] According to an embodiment of the present disclosure, a data table to be filled is read, a field value counting method is used, and the number of imported field values ​​in the data table to be filled is obtained. A file reading method corresponding to an extension is used to read a file to be imported, and a field value counting method is used to obtain the number of field values ​​in the file to be imported. And based on the ratio between the number of imported field values ​​and the number of field values ​​in the file to be imported, the import progress of the target data table is determined.

[0087] According to an embodiment of the present disclosure, since the field values ​​in the file to be imported may be judged as invalid fields and thus cleaned during the data cleaning process, the import progress of the target data table can also be determined based on the ratio between the number of imported field values ​​and the number of converted field values.

[0088] According to an embodiment of the present disclosure, the import progress of the target data table can be intuitively determined by the number of imported field values ​​in the data table to be filled and the number of field values ​​in the file to be imported or the number of converted field values, so that the user can query or adjust the resources used for data import according to the import progress.

[0089] According to an embodiment of the present disclosure, the data import method also includes: receiving the file to be imported submitted by the user through the file reading sub-interface of the data import interface; and receiving the data conversion rules for the field names of the file to be imported submitted by the user through the configuration sub-interface of the data import interface.

[0090] According to an embodiment of the present disclosure, the data import interface may include multiple sub-interfaces, wherein the user may submit the file to be imported by dragging the file to the file reading sub-interface, after parsing the file to be imported using the data import method, the user may select one or more from multiple field names through the configuration sub-interface, and customize the data conversion rules for the field names, and the user may also query the import progress of the target data table through the progress display sub-interface. This improves the flexibility of data import and improves the user experience.

[0091] According to an embodiment of the present disclosure, the data import method further includes: in response to an extension plug-in submitted by a user, determining a newly added extension corresponding to the extension plug-in and a file reading method corresponding to the newly added extension; when determining that the extension of the file to be imported is the newly added extension, using the file reading method corresponding to the newly added extension to read the file to be imported. The data import method can be incremented through the extension plug-in submitted by the user to increase the file types to which the data import method can be applied.

[0092] Figure 3 A flow chart of a data import method according to another embodiment of the present disclosure is schematically shown.

[0093] like Figure 3 As shown, the data import method of this embodiment includes operations S301 to S311.

[0094] In operation S301, obtain the file to be imported, and determine the file name of the file to be imported. In operation S302, determine the extension of the file to be imported based on the file name of the file to be imported. In operation S303, based on the extension of the file to be imported, use the file reading method corresponding to the extension to read the file content of the file to be imported, and determine the multiple field names included in the file header. In operation S304, determine the data conversion rules for each of the multiple field values ​​in the imported file. In operation S305, determine whether the user has submitted the data conversion rules in the configuration sub-interface. If yes, perform operation S306; if not, perform operation S307. In operation S306, convert the field value based on the data conversion rule submitted by the user, and determine the converted field value. In operation S307, convert the field value based on the data conversion rule generated by the data import method, and determine the converted field value. In operation S308, import the converted field value into the data table to be filled, and determine the target data table.

[0095] Before operation S301, the file to be imported may be determined through operation S309. In operation S309, the user submits the file to be imported on the file reading subpage.

[0096] Before operation S303, the range of extensions that can be processed by the data import method can be expanded through operations S310 to S311. In operation S310, the user submits an extension plug-in. In operation S311, the extension plug-in is parsed to determine the newly added extension and the corresponding file reading method.

[0097] Based on the above data import method, the present disclosure also provides a data import device. Figure 4 The device is described in detail.

[0098] Figure 4 The structural block diagram of the data importing device according to the embodiment of the present disclosure is schematically shown.

[0099] like Figure 4 As shown, the data importing device 400 of this embodiment includes an extension determination module 410 , a file header determination module 420 , a rule determination module 430 , a field conversion module 440 and a data importing module 450 .

[0100] The extension determination module 410 is used to process the file name of the file to be imported using a string segmentation method to obtain the extension of the file name. In one embodiment, the extension determination module 410 can be used to perform the operation S210 described above, which will not be repeated here.

[0101] The file header determination module 420 is used to read the file header of the file to be imported using a file reading method corresponding to the extension, wherein the file header includes multiple field names. In one embodiment, the file header determination module 420 can be used to perform the operation S220 described above, which will not be repeated here.

[0102] The rule determination module 430 is used to determine multiple data conversion rules for field values ​​corresponding to multiple field names of the file to be imported based on the extension and the file header. In one embodiment, the rule determination module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0103] The field conversion module 440 is used to convert the field values ​​corresponding to the multiple field names respectively based on multiple data conversion rules to obtain converted field values. In one embodiment, the field conversion module 440 can be used to perform the operation S240 described above, which will not be described in detail here.

[0104] The data import module 450 is used to import multiple converted field values ​​into the to-be-filled data table according to the correspondence between the field values ​​and the field names, to obtain a target data table, wherein the to-be-filled data table is created in the database according to the file header. In one embodiment, the data import module 450 can be used to perform the operation S250 described above, which will not be described in detail here.

[0105] According to an embodiment of the present disclosure, the file header determination module 420 includes a file type determination submodule, a method determination submodule, a content determination submodule and a field name determination submodule.

[0106] The file type determination submodule is used to determine the file type of the file to be imported based on the extension.

[0107] The method determination submodule is used to determine the file reading method required to read the file type based on the file type.

[0108] The content determination submodule is used to read the file to be imported using a file reading method to determine the file content of the file to be imported, wherein the file content includes a file header and multiple field values.

[0109] The field name determination submodule is used to determine multiple field names included in the file header based on the file content.

[0110] According to an embodiment of the present disclosure, the rule determination module 430 includes a data type determination submodule, a history file determination submodule, a target type determination submodule, and a rule determination submodule.

[0111] The data type determination submodule is used to determine the initial data type of the field name in the file to be imported based on each field name.

[0112] The historical file determination submodule is used to determine the historical file having the same file type and the same file header as the file to be imported based on the extension name.

[0113] The target type determination submodule is used to determine the target data type of the field name in the database based on the historical file.

[0114] The rule determination submodule is used to determine the data conversion rule of the field value corresponding to the field name based on the initial data type and the target data type.

[0115] According to an embodiment of the present disclosure, the field conversion module 440 includes a format conversion submodule, a validity determination submodule, and a conversion result determination submodule.

[0116] The format conversion submodule is used to convert the format of each field value using the data format conversion rules to obtain the target format field value.

[0117] The validity determination submodule is used to determine the validity of the target format field value based on the data cleaning rules.

[0118] The conversion result determination submodule is used to determine the conversion field value based on the target format field value when the target format field value is valid.

[0119] According to an embodiment of the present disclosure, the data import module 450 includes a field corresponding submodule, a data column determining submodule and a data filling submodule.

[0120] The field corresponding submodule is used to determine the field name corresponding to the conversion field value for each conversion field value.

[0121] The data column determination submodule is used to import the converted field value into the column corresponding to the field name in the data table to be filled.

[0122] The data filling submodule is used to determine the data table to be filled as the target data table when the file to be imported does not include the conversion field value that has not been imported into the data table to be filled.

[0123] According to an embodiment of the present disclosure, the extension determination module 410 includes a file name determination submodule, a character determination submodule, and a file name segmentation submodule.

[0124] The file name determination submodule is used to determine the file name of the file to be imported based on the file to be imported.

[0125] The character determination submodule is used to process the file name using a string segmentation method and determine the segmentation characters in the file name.

[0126] The file name segmentation submodule is used to segment the file name using segmentation characters to obtain the file description of the file to be imported and the extension of the file to be imported.

[0127] According to an embodiment of the present disclosure, the data importing device 400 further includes a first quantity determination module, a second quantity determination module and a progress determination module.

[0128] The first quantity determination module is used to obtain the quantity of the imported field values ​​in the data table to be filled.

[0129] The second quantity determination module is used to read the file to be imported using the file reading method corresponding to the extension name, and obtain the quantity of field values ​​in the file to be imported.

[0130] The progress determination module is used to determine the import progress of the target data table based on the number of imported field values ​​and the number of field values ​​in the file to be imported.

[0131] According to an embodiment of the present disclosure, any multiple modules of the extension determination module 410, the file header determination module 420, the rule determination module 430, the field conversion module 440 and the data import module 450 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the extension determination module 410, the file header determination module 420, the rule determination module 430, the field conversion module 440 and the data import module 450 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of the extension determination module 410, the file header determination module 420, the rule determination module 430, the field conversion module 440 and the data import module 450 can be at least partially implemented as a computer program module, which can perform corresponding functions when executed.

[0132] Figure 5 A block diagram of an electronic device suitable for implementing a data import method according to an embodiment of the present disclosure is schematically shown.

[0133] like Figure 5As shown, the electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 to a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0134] In RAM 503, various programs and data required for the operation of electronic device 500 are stored. Processor 501, ROM 502 and RAM 503 are connected to each other via bus 504. Processor 501 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 502 and / or RAM 503. It should be noted that the program can also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in the one or more memories.

[0135] According to an embodiment of the present disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the input / output (I / O) interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 508 including a hard disk, etc.; and a communication portion 509 including a network interface card such as a LAN card, a modem, etc. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage portion 508 as needed.

[0136] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0137] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0138] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program contains program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data import method provided by the embodiment of the present disclosure.

[0139] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 501. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0140] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 509, and / or installed from the removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0141] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the processor 501, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0142] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0143] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0145] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A data import method, characterized in that: The method comprises: Process the file name of the file to be imported using a string segmentation method to obtain the extension of the file name; Using a file reading method corresponding to the extension name, reading a file header of the file to be imported, wherein the file header includes a plurality of field names; Based on the extension name and the file header, determine multiple data conversion rules for field values ​​corresponding to the multiple field names of the file to be imported; Based on the plurality of data conversion rules, converting the field values ​​corresponding to the plurality of field names respectively to obtain converted field values; and Importing the plurality of converted field values ​​into the data table to be filled according to the correspondence between the field values ​​and the field names to obtain a target data table, wherein the data table to be filled is created in a database according to the file header.

2. The method according to claim 1, characterized in that: The method of reading the file header of the file to be imported by using the file reading method corresponding to the extension name includes: Based on the extension, determine the file type of the file to be imported; Based on the file type, determining the file reading method required to read the file type; Using the file reading method to read the file to be imported, and determining the file content of the file to be imported, wherein the file content includes the file header and the plurality of field values; and Based on the file content, the multiple field names included in the file header are determined.

3. The method according to claim 2, characterized in that The step of determining, based on the extension name and the file header, a plurality of data conversion rules for field values ​​corresponding to the plurality of field names of the file to be imported, comprises: Based on each of the field names, determining the initial data type of the field name in the file to be imported; Based on the extension, determine a historical file having the same file type and the same file header as the file to be imported; Determining, based on the history file, a target data type of the field name in the database; and Based on the initial data type and the target data type, a data conversion rule for the field value corresponding to the field name is determined.

4. The method according to claim 3, characterized in that The data conversion rules include data format conversion rules and data cleaning rules; The converting the field values ​​corresponding to the plurality of field names respectively based on the plurality of data conversion rules to obtain converted field values ​​includes: For each of the field values, convert the field value into a format using the data format conversion rule to obtain a target format field value; Determining the validity of the target format field value based on the data cleansing rule; and In case the target format field value is valid, the conversion field value is determined based on the target format field value.

5. The method according to claim 1, characterized in that: The step of importing the plurality of converted field values ​​into the data table to be filled according to the correspondence between the field values ​​and the field names to obtain the target data table includes: For each of the conversion field values, determining the field name corresponding to the conversion field value; Importing the converted field value into the column corresponding to the field name in the to-be-filled data table; and In a case where the file to be imported does not include the conversion field value that has not been imported into the data table to be filled, the data table to be filled is determined as the target data table.

6. The method according to claim 1, characterized in that The method of using a string segmentation method to process the file name of the file to be imported to obtain the extension of the file name includes: Based on the file to be imported, determine the file name of the file to be imported; Processing the file name using a string segmentation method to determine a segmentation character in the file name; and The file name is segmented using the segmentation character to obtain a file description of the file to be imported and an extension of the file to be imported.

7. The method according to claim 1, characterized in that The method further comprises: Obtain the number of imported field values ​​in the data table to be filled; Using a file reading method corresponding to the extension name, reading the file to be imported, and obtaining the number of field values ​​in the file to be imported; and Based on the number of the imported field values ​​and the number of the field values ​​in the file to be imported, the import progress of the target data table is determined.

8. A data import device, characterized in that: The device comprises: The extension determination module is used to process the file name of the file to be imported by using a string segmentation method to obtain the extension of the file name; A file header determination module, used to read the file header of the to-be-imported file using a file reading method corresponding to the extension name, wherein the file header includes a plurality of field names; A rule determination module, used for determining a plurality of data conversion rules of field values ​​corresponding to the plurality of field names of the to-be-imported file, based on the extension name and the file header; A field conversion module, configured to convert the field values ​​corresponding to the plurality of field names respectively based on the plurality of data conversion rules to obtain converted field values; and The data import module is used to import the multiple conversion field values ​​into the data table to be filled according to the correspondence between the field values ​​and the field names to obtain the target data table, wherein the data table to be filled is created in the database according to the file header.

9. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

11. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images