Data import configuration method and device, storage medium and computer equipment

Abstract

According to the data import configuration method and device, the storage medium and the computer equipment provided by the invention, when the data configuration is imported, the source data file is uploaded firstly, the target table configuration corresponding to the file is determined, and the import rule configuration is generated based on the template configuration and the target table configuration, so that automatic configuration of the file data import rule can be realized; because the import rule configuration comprises the corresponding relationship between the template column name and the plurality of field names, the field reading can be performed on the source data file according to the import rule configuration, and the data is imported according to the reading result to generate the temporary table; and finally, after a verification function of each column of data in the temporary table is determined based on the import rule configuration, data verification is carried out on the temporary table, so that the column data passing verification is migrated into the target table, and the verification function can visualize business internal logic into a rule and then build the rule in the import rule configuration. Automatic verification of the data in the temporary table is achieved, and the development efficiency of the data import function is further improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data import configuration method, device, storage medium and computer equipment. Background Art

[0002] With the continuous advancement of information technology, low-code development platforms are rapidly becoming a key force in promoting software development, providing strong support for enterprises to build core competitiveness. In the software development model, developers need to manually migrate business data such as Excel to database tables, and programmers need to write code based on specific business logic to achieve data import and subsequent table verification.

[0003] The development of data import function usually takes 1 to 3 days, especially when faced with complex business needs, and places extremely high demands on the professional skills of developers. However, different developers have different technical levels and programming habits, especially when faced with complex business needs, which leads to low efficiency in data import function development and uneven quality of development results, making it difficult to ensure consistency and maintainability, which in turn makes operation and maintenance difficult and subsequent modifications more complex. Summary of the invention

[0004] The purpose of this application is to solve at least one of the above-mentioned technical defects, especially the technical defects that the data import function development efficiency in the prior art is low, and the quality of development results is uneven, it is difficult to ensure consistency and maintainability, which leads to high difficulty in operation and maintenance and complexity of subsequent modifications.

[0005] The present application provides a data import configuration method, the method comprising:

[0006] Upload a source data file, determine a target table configuration corresponding to the source data file, and generate an import rule configuration for the source data file based on a preset template configuration and the target table configuration; the import rule configuration includes a correspondence between a template column name and a plurality of field names;

[0007] Read the fields of the source data file according to the import rule configuration, import the data according to the reading result, and generate a temporary table with the template column name as the table header;

[0008] The verification function of each column of data in the temporary table is determined based on the import rule configuration, and the data of the temporary table is verified by using each verification function, so as to migrate the column data that passes the verification to the target table.

[0009] Optionally, determining the target table configuration corresponding to the source data file includes:

[0010] Determine the target table to which the source data file is to be imported, extract multiple field names in the target table, and determine the field attribute of each field name;

[0011] A target table configuration corresponding to the source data file is generated according to each field name and the field attribute corresponding to each field name.

[0012] Optionally, the generating the import rule configuration of the source data file based on the preset template configuration and the target table configuration includes:

[0013] Filter out the field names corresponding to the field names in the target table configuration from the template fields in the preset template configuration, and combine the filter results with the default fields in the template configuration to form template column names;

[0014] The import rule corresponding to each field name in the template column name is determined according to the template configuration, and the import rule configuration of the source data file is generated according to the template column name and each import rule.

[0015] Optionally, the reading fields of the source data file according to the import rule configuration includes:

[0016] Determine the file format of the source data file and the field mapping rule corresponding to the file format in the import rule configuration;

[0017] The field mapping rule is used to perform field mapping in the source data file, and data is read according to the mapping result.

[0018] Optionally, the template list includes template fields and default fields;

[0019] The step of importing data according to the read result and generating a temporary table with the template column name as a table header includes:

[0020] Determine the field type of each field name in the template field, and perform data type conversion on the read result according to each field type to obtain corresponding conversion data under the template field;

[0021] Generate corresponding default data under the default field based on the current data import time;

[0022] A temporary table is generated according to the conversion data and the default data, with the template column name as a table header.

[0023] Optionally, the determining a verification function for each column of data in the temporary table based on the import rule configuration includes:

[0024] The field name corresponding to each column of data in the temporary table is determined, and the import rule corresponding to each field name is determined based on the import rule configuration, and each import rule includes a verification function.

[0025] Optionally, the using various verification functions to perform data verification on the temporary table includes:

[0026] For each check function, determine the column data corresponding to the check function in the temporary table, and use the check function to perform a logic check on the column data to obtain a check result;

[0027] If the verification result is failed, the information is recorded in the error information column corresponding to the column data;

[0028] If the verification result is passed, the error information column corresponding to the column data is not updated.

[0029] The present application also provides a data import configuration device, comprising:

[0030] A rule configuration module, used to upload a source data file, determine a target table configuration corresponding to the source data file, and generate an import rule configuration for the source data file based on a preset template configuration and the target table configuration; the import rule configuration includes a correspondence between a template column name and a plurality of field names;

[0031] A data import module, configured to read fields of the source data file according to the import rule configuration, and to import data according to the reading result, and to generate a temporary table with the template column name as a table header;

[0032] The data verification module is used to determine the verification function of each column of data in the temporary table based on the import rule configuration, and use each verification function to perform data verification on the temporary table to migrate the column data that passes the verification to the target table.

[0033] The present application also provides a storage medium, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the data import configuration method as described in any of the above embodiments.

[0034] The present application also provides a computer device, comprising: one or more processors, and a memory;

[0035] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the data import configuration method described in any one of the above embodiments are performed.

[0036] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0037] The data import configuration method, device, storage medium and computer equipment provided by the present application can first upload the source data file and determine the target table configuration corresponding to the source data file when importing the data configuration, and generate the import rule configuration of the source data file based on the preset template configuration and the target table configuration, so that the automatic configuration of the rules for data import of the source data file can be realized through the template configuration and the target table configuration, thereby improving the programming uniformity and development efficiency; since the import rule configuration includes the correspondence between the template column name and the multiple field names, the source data file can be read according to the import rule configuration, and the data can be imported according to the reading result, and a temporary table with the template column name as the table header is generated, thereby realizing the integration capability and maintainability of the data import process; finally, the verification function of each column of data in the temporary table can be determined based on the import rule configuration, and each verification function can be used to verify the data of the temporary table, so as to migrate the column data that passes the verification to the target table, where the verification function can visualize the internal logic of the business as a rule and then be built into the import rule configuration, so as to realize the automatic verification of the data in the temporary table, ensure the consistency and maintainability of the data, and further improve the development efficiency of the data import function. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0039] Figure 1 A flowchart of a data import configuration method provided in an embodiment of the present application;

[0040] Figure 2 A schematic diagram of a process for generating a temporary table provided in an embodiment of the present application;

[0041] Figure 3 A schematic diagram of the structure of a data import configuration device provided in an embodiment of the present application;

[0042] Figure 4 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] The development of data import function usually takes 1 to 3 days, especially when faced with complex business needs, and places extremely high demands on the professional skills of developers. However, different developers have different technical levels and programming habits, especially when faced with complex business needs, which leads to low efficiency in data import function development and uneven quality of development results, making it difficult to ensure consistency and maintainability, which in turn makes operation and maintenance difficult and subsequent modifications more complex.

[0045] In one embodiment, Figure 1 As shown, Figure 1 A flowchart of a data import configuration method provided in an embodiment of the present application; the present application provides a data import configuration method, which specifically includes the following:

[0046] S110: Upload the source data file, determine the target table configuration corresponding to the source data file, and generate the import rule configuration of the source data file based on the preset template configuration and the target table configuration; the import rule configuration includes the correspondence between the template column name and multiple field names.

[0047] In this step, when the user needs to import and migrate business data, he can first upload the source data file containing the business data to the computer device, and then determine the target table configuration corresponding to the source data file through the computer device, so that the computer device can generate the import rule configuration of the source data file based on the preset template configuration and target table configuration, thereby realizing the automatic configuration of data import rules and improving programming uniformity and development efficiency.

[0048] It is understandable that the source data file of this application refers to a file containing the business data to be imported, which may include multiple formats, such as CSV, Excel, JSON, XML, etc., depending on the source and format of the data; and the target table configuration refers to the structure and properties of the target table to which the business data will be imported, including the table name, column name, data type, length, etc. of the target table. In addition, the template configuration refers to a predefined configuration in a computer device, which is used to guide the parsing and mapping of data, such as field mapping rules, data type conversion, data verification rules, etc.

[0049] Therefore, when determining the target table configuration corresponding to the source data file, the computer device can automatically generate the import rule configuration of the source data file by combining the field mapping and data conversion rules in the template configuration and the table structure and field attributes in the target table configuration, including the correspondence between the template column name and multiple field names. To elaborate, the import rule configuration here can be composed of a template column name composed of multiple field names in the target table to be imported and its data import rules, which are used to guide the import of business data of the source data file into the target table, thereby improving the efficiency and accuracy of data import, while reducing human errors and programming workload.

[0050] In summary, the source data file is the source of business data, the target table configuration defines the storage structure of business data, and the template configuration provides rules for parsing and mapping business data. Therefore, based on these configurations, computer equipment can automatically generate import rule configurations to guide the automatic import of business data into the target table, thereby improving programming uniformity and development efficiency.

[0051] S120: Read the fields of the source data file according to the import rule configuration, import the data according to the reading result, and generate a temporary table with the template column name as the table header.

[0052] In this step, after the import rule configuration of the source data file is automatically generated through step S110, the computer device can read the fields of the source data file according to the import rule configuration to obtain the reading results, and then import the data according to the reading results to generate a temporary table with the template column name as the table header, thereby realizing the integration capability and maintainability of the data import process.

[0053] Specifically, after generating the import rule configuration, the computer device can load the import rule configuration to parse the source data file according to the loading result. For source data files of different formats, the computer device can adopt different parsing and field reading methods; for example, for CSV files, fields may be separated according to delimiters such as commas and tabs, and for Excel files, data may be obtained by reading cells, without limitation here.

[0054] Furthermore, after the computer device obtains the reading result, it can construct a blank temporary table, which can use the template column name as the table header, and it has the same column name as the target table. Therefore, after the reading result is imported into the temporary table, the computer device can further perform verification processing and directly migrate it to the target table, further improving the integration capability and maintainability of the data import process. However, it should be noted that after the reading result is imported into the temporary table, the computer device can also add a primary key according to the attributes of the data in the temporary table of the reading result, further improving the maintainability of the data.

[0055] S130: Determine a verification function for each column of data in the temporary table based on the import rule configuration, and use each verification function to verify the data in the temporary table, so as to migrate the column data that passes the verification to the target table.

[0056] In this step, after the temporary table is generated through step S120, the computer device can determine the verification function of each column of data in the temporary table based on the import rule configuration, and use each verification function to verify the data of the temporary table to migrate the column data that passes the verification to the target table. Here, the verification function can visualize the internal logic of the business as a rule and then be built into the import rule configuration to realize automatic verification of the data in the temporary table, ensure the consistency and maintainability of the data, and further improve the efficiency of data import function development.

[0057] It can be understood that since the import rule configuration is generated based on the preset template configuration and target table configuration, and the template configuration is provided with a verification function for each field name, that is, the corresponding column data in the temporary table, through the import rule configuration, the computer device can quickly identify and process the data records in the temporary table that do not comply with the rules, thereby reducing manual intervention and error rate.

[0058] Specifically, after the temporary table is generated, the computer device can use the built-in verification function to verify the data in the table one by one. Here, multiple different verification functions can be executed simultaneously to ensure that the data meets all preset constraints. For each execution result, the computer device can determine whether it passes the verification based on the return result of the verification function. If the execution result fails the verification, the device can record an error message, mark the record as invalid, or take other corresponding processing measures, which are not limited here. Only when the execution result passes all verifications will the computer device migrate it from the temporary table to the target table to ensure that only accurate and qualified data will be finally stored, thereby improving the accuracy and reliability of data import.

[0059] Furthermore, through the automated data import configuration of the present application, the computer device can ensure that the data in the temporary table maintains consistency and integrity before and after migration to the target table, thereby improving the reliability of the data. Among them, the built-in verification function and the visual configuration interface can make the data import verification process flexible. During this process, the template device in the computer device can be adjusted according to actual business needs, thereby eliminating the need for a large amount of code modification and maintenance, thereby reducing the development cost and maintenance difficulty of the data import function.

[0060] Furthermore, the target table generated by the present application may include a static table, which is named with a table name without a date suffix; the target table may also include a dynamic table, which is named with a table name with an import date suffix. The import date suffix in the table name of the dynamic table is automatically generated by the computer device according to the system time.

[0061] In the above embodiment, when importing data configuration, the source data file can be uploaded first, and the target table configuration corresponding to the source data file can be determined, and the import rule configuration of the source data file can be generated based on the preset template configuration and the target table configuration, so that the automatic configuration of the rules for importing the source data file data can be realized through the template configuration and the target table configuration, thereby improving programming uniformity and development efficiency; since the import rule configuration includes the correspondence between the template column name and the multiple field names, the source data file can be read according to the import rule configuration, and data can be imported according to the reading result, and a temporary table with the template column name as the table header is generated, thereby realizing the integration capability and maintainability of the data import process; finally, the verification function of each column of data in the temporary table can be determined based on the import rule configuration, and each verification function can be used to verify the data of the temporary table, so as to migrate the column data that has passed the verification to the target table, where the verification function can visualize the internal logic of the business as a rule and then be built into the import rule configuration, so as to realize the automatic verification of the data in the temporary table, ensure the consistency and maintainability of the data, and further improve the development efficiency of the data import function.

[0062] In one embodiment, the process of determining the target table configuration corresponding to the source data file in step S110 may include:

[0063] S111: Determine the target table to be imported into the source data file, extract multiple field names in the target table, and determine the field attributes of each field name.

[0064] S112: Generate a target table configuration corresponding to the source data file according to each field name and the field attribute corresponding to each field name.

[0065] In this embodiment, when determining the target table configuration corresponding to the source data file, the computer device can first determine the target table to which the source data file is to be imported, then extract multiple field names in the target table, and determine the field attributes of each field name. Then, the computer device can generate the target table configuration corresponding to the source data file according to the various field names and the field attributes corresponding to each field name, which is used to define the storage structure of the business data in the source data file.

[0066] Among them, field attributes refer to the characteristics or regulations of each field name in the target table. These characteristics or regulations define the storage method, constraints and role of the data under the field name in the database. In this application, they can include field types, such as numeric type, character type or date / time type, etc., and can also include key attributes, such as primary key or foreign key, etc.

[0067] Specifically, the computer device can determine the target table to be imported into the source data through user-specified, configuration file-read or predefined rules, and then extract all field names from the definition of the target table. The field name here is the identifier of the column in the table, which defines the data items that can be stored in the target table. For each extracted field name, the computer device can determine its corresponding field attribute, and then generate a target table configuration for defining the storage structure of the business data in the source data file, including a list of field names, attribute descriptions of each field name, and the relationship between them, to ensure the accuracy, consistency and availability of the data during the import process.

[0068] In one embodiment, the process of generating the import rule configuration of the source data file based on the preset template configuration and the target table configuration in step S110 may include:

[0069] S113: Filter out field names corresponding to the field names in the target table configuration from the template fields in the preset template configuration, and combine the filter results with the default fields in the template configuration to form template column names.

[0070] S114: Determine the import rule corresponding to each field name in the template column name according to the template configuration, and generate the import rule configuration of the source data file according to the template column name and each import rule.

[0071] In this embodiment, the template configuration pre-set in the computer device is composed of two parts: template fields and default fields. The template fields refer to data fields configured in the target table that refer to business data types, such as user ID, user account, age, etc., and the default field values ​​are data fields configured in the target table for real-time recording of data import information, such as importer, import time, import ID, and information error column.

[0072] Specifically, when the computer device generates the import rule configuration of the source data file, since the field names in the template field cover the field names in multiple target tables, the computer device can first filter out the field names corresponding to the field names in the target table configuration from the template field, and then combine the filtered results with the fixed default fields in a preset order to form the template column name. Then, the computer device can determine the import rule corresponding to each field name in the template column name according to the template configuration, and generate the import rule configuration of the source data file according to the template column name and each import rule.

[0073] It is understandable that each field name in the template configuration has a corresponding import rule set in advance. For each field name in the template field, its import rule can be to directly extract and import data from the corresponding source data file through the field mapping rule; for each field name in the default field, its import rule can be to obtain it through the built-in variable rule.

[0074] For example, when using built-in variable rules, the computer device can specify a corresponding function or a custom function during configuration to obtain the corresponding value according to the function during import. For example, to import time, the computer device can import through the built-in variable "system time"; to import ID, the computer device can import through the built-in variable "current user login ID".

[0075] In addition, the computer device can also import data by table association. For example, when importing "user ID", if the source data file only has "user name", the computer device can associate the "user name" of the source data file with the system user table to obtain the "user ID", and then import the data of the "user ID".

[0076] In one embodiment, the step S120 of reading fields of the source data file according to the import rule configuration may include:

[0077] S121: Determine the file format of the source data file and the field mapping rule corresponding to the file format in the import rule configuration.

[0078] S122: Perform field mapping in the source data file using the field mapping rule, and read data according to the mapping result.

[0079] In this embodiment, when reading data, the computer device can first determine the file format of the source data file and the field mapping rules corresponding to the file format in the import rule configuration, and then use the field mapping rules to perform field mapping in the source data file and read the data according to the mapping results.

[0080] It is understandable that the data parsing methods of source data files of different file formats are different. For example, for CSV files, fields can be separated according to delimiters such as commas and tabs; for Excel files, data can be parsed by reading cells. Therefore, when the computer device determines the file format of the source data file, it can perform field mapping in the source data file according to the field mapping rules corresponding to the file format in the import rule configuration, extract the corresponding data, and perform preliminary data cleaning, such as removing blank characters, processing missing values, etc., to obtain the read data.

[0081] In one embodiment, Figure 2 As shown, Figure 2 A schematic diagram of a process for generating a temporary table provided in an embodiment of the present application; Figure 2 In step S120, the template list may include template fields and default fields; wherein the process of importing data according to the read result and generating a temporary table with the template column name as the table header may include:

[0082] S123: Determine the field type of each field name in the template field, and perform data type conversion on the read result according to each field type to obtain corresponding conversion data under the template field.

[0083] S124: Generate corresponding default data under the default field based on the current data import time.

[0084] S125: Generate a temporary table with the template column name as the table header according to the conversion data and the default data.

[0085] In this embodiment, when importing data, the computer device can first determine the field type of each field name in the template field, and convert the data type of the read result according to each field type to obtain the corresponding conversion data under the template field. The computer device can then generate the corresponding default data under the default field based on the current data import time, and then generate a temporary table with the template column name as the table header based on the conversion data and the default data.

[0086] Specifically, the computer device can first match each data in the read result with each field name in the template field in the template column name one by one, and then convert the data under each field name one by one according to the field type of each field name in the template field to ensure the uniformity of data format and type.

[0087] For example, for text fields, computer devices can remove invalid characters, standardize encoding formats, and ensure unified storage standards; for numeric fields, computer devices can handle the conversion between integers and floating-point numbers, and verify whether the values ​​are within the predefined range of values; for date fields, computer devices can parse their formats and standardize them into a unified time representation; for Boolean fields, computer devices can convert their values ​​into logical expressions, such as converting "yes" to True.

[0088] Furthermore, after completing the field type conversion of the template field in the template column name, the computer device can also generate corresponding default data for the default field in the template column name to record the real-time import information. Subsequently, the computer device can integrate the conversion data and the default data, and generate a temporary table with the template column name as the table header. The temporary table is a structured data storage form, and its structure strictly follows the field definition of the template column name. It not only contains the valid data in the source data file, but also contains the operation records automatically generated by the computer device for subsequent import record tracking.

[0089] In one embodiment, the process of determining the verification function of each column of data in the temporary table based on the import rule configuration in step S130 may include:

[0090] S131: Determine the field name corresponding to each column of data in the temporary table, and determine the import rule corresponding to each field name based on the import rule configuration, each import rule includes a verification function.

[0091] In this embodiment, when verifying the data in the temporary table, the computer device can determine the field name corresponding to each column of data in the temporary table, and determine the import rule corresponding to each field name based on the import rule configuration. Since each import rule includes a verification function, the computer device can use the verification function in the import rule to verify the data under the corresponding field name.

[0092] It is understandable that the import rule configuration, as a set of import rules for each field name, defines the validation and conversion requirements that need to be followed during the data import process. Each field name has clear validation rules in the import rule configuration. These rules usually include the format requirements, integrity requirements, uniqueness requirements of the field value, and logical requirements between data. In addition, the validation rules corresponding to each field name in the import rule configuration also contain specific validation functions. These validation functions are programmed logic that can automatically verify whether the field value meets the corresponding rules.

[0093] In one embodiment, the process of using various verification functions to verify data in the temporary table in step S130 may include:

[0094] S132: For each verification function, determine the column data corresponding to the verification function in the temporary table, and use the verification function to perform a logic check on the column data to obtain a verification result.

[0095] S133: If the verification result is failure, information is recorded in the error information column corresponding to the column data.

[0096] S134: If the verification result is passed, then the error information column corresponding to the column data is not updated.

[0097] In this embodiment, when performing data verification on the temporary table, for each verification data, the computer device can determine the column data corresponding to the verification function in the temporary table, and use the verification function to perform a logical verification on the column data to obtain a verification result. If the verification result is not passed, the information is recorded in the error information column corresponding to the column data; if the verification result is passed, the information in the error information column corresponding to the column data is not updated.

[0098] Specifically, during the verification process, the computer device can process each column of data in the temporary table one by one. For each column of data, the computer device can first identify the verification function corresponding to its field name for logical verification. For example, for the "age" field name, the verification function can check whether all values ​​are within a reasonable range, such as between 1 and 40, and verify whether they are integers; for the "order date" field name, the verification function may parse the date format and ensure that all date values ​​are valid and logically meet the requirements. When the verification result fails, the computer device can update the information in the corresponding error information column; when the verification result passes, the computer device can directly skip the verification of the next data.

[0099] The data import configuration device provided in an embodiment of the present application is described below. The data import configuration device described below and the data import configuration method described above can be referenced to each other.

[0100] In one embodiment, Figure 3 As shown, Figure 3 A schematic diagram of the structure of a data import device provided in an embodiment of the present application; the present application also provides a data import configuration device, including a rule configuration module 210, a data import module 220 and a data verification module 230, specifically including the following:

[0101] The rule configuration module 210 is used to upload the source data file, determine the target table configuration corresponding to the source data file, and generate the import rule configuration of the source data file based on the preset template configuration and the target table configuration; the import rule configuration includes the correspondence between the template column name and multiple field names.

[0102] The data import module 220 is used to read the fields of the source data file according to the import rule configuration, and import the data according to the reading result to generate a temporary table with the template column name as the table header.

[0103] The data verification module 230 is used to determine the verification function of each column of data in the temporary table based on the import rule configuration, and use each verification function to perform data verification on the temporary table to migrate the column data that passes the verification to the target table.

[0104] In the above embodiment, when importing data configuration, the source data file can be uploaded first, and the target table configuration corresponding to the source data file can be determined, and the import rule configuration of the source data file can be generated based on the preset template configuration and the target table configuration, so that the automatic configuration of the rules for importing the source data file data can be realized through the template configuration and the target table configuration, thereby improving programming uniformity and development efficiency; since the import rule configuration includes the correspondence between the template column name and the multiple field names, the source data file can be read according to the import rule configuration, and data can be imported according to the reading result, and a temporary table with the template column name as the table header is generated, thereby realizing the integration capability and maintainability of the data import process; finally, the verification function of each column of data in the temporary table can be determined based on the import rule configuration, and each verification function can be used to verify the data of the temporary table, so as to migrate the column data that has passed the verification to the target table, where the verification function can visualize the internal logic of the business as a rule and then be built into the import rule configuration, so as to realize the automatic verification of the data in the temporary table, ensure the consistency and maintainability of the data, and further improve the development efficiency of the data import function.

[0105] In one embodiment, the rule configuration module 210 may include:

[0106] The field name determination submodule is used to determine the target table to be imported into the source data file, extract multiple field names in the target table, and determine the field attributes of each field name.

[0107] The configuration generation submodule is used to generate the target table configuration corresponding to the source data file according to each field name and the field attributes corresponding to each field name.

[0108] In one embodiment, the rule configuration module 210 may further include:

[0109] The column name formation submodule is used to filter out the field names corresponding to the field names in the target table configuration from the template fields in the preset template configuration, and combine the filtering results with the default fields in the template configuration to form the template column names.

[0110] The rule determination submodule is used to determine the import rule corresponding to each field name in the template column name according to the template configuration, and generate the import rule configuration of the source data file according to the template column name and each import rule.

[0111] In one embodiment, the data import module 220 includes:

[0112] The format determination submodule is used to determine the file format of the source data file and the field mapping rules corresponding to the file format in the import rule configuration.

[0113] The data reading submodule is used to map fields in the source data file using field mapping rules and read data according to the mapping results.

[0114] In one embodiment, the data import module 220 may include:

[0115] The data conversion submodule is used to determine the field type of each field name in the template field, and convert the data type of the read result according to each field type to obtain the corresponding conversion data under the template field.

[0116] The data generation submodule is used to generate the corresponding default data under the default field based on the current data import time.

[0117] The data combination submodule is used to generate a temporary table with the template column name as the table header based on the conversion data and default data.

[0118] In one embodiment, the data verification module 230 may include:

[0119] The function determination submodule is used to determine the field name corresponding to each column of data in the temporary table, and determine the import rule corresponding to each field name based on the import rule configuration. Each import rule contains a verification function.

[0120] In one embodiment, the data verification module 230 may further include:

[0121] The logic check submodule is used to determine the column data corresponding to each check function in the temporary table, and use the check function to perform logic check on the column data to obtain a check result.

[0122] The first confirmation submodule is used to record information in an error information column corresponding to the column data if the verification result is failure.

[0123] The second confirmation submodule is used for not updating the error information column corresponding to the column data if the verification result is passed.

[0124] In one embodiment, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the data import configuration method as described in any of the above embodiments.

[0125] In one embodiment, the present application also provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the data import configuration method as described in any one of the above embodiments.

[0126] Indicatively, Figure 4 As shown, Figure 4 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 may be provided as a server. Figure 4 , the computer device 300 includes a processing component 302, which further includes one or more processors, and a memory resource represented by a memory 301, for storing instructions executable by the processing component 302, such as an application. The application stored in the memory 301 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute instructions to perform the data import configuration method of any of the above embodiments.

[0127] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0128] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0129] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0130] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0131] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data import configuration method, characterized in that: The method comprises: Upload a source data file, determine a target table configuration corresponding to the source data file, and generate an import rule configuration for the source data file based on a preset template configuration and the target table configuration; the import rule configuration includes a correspondence between a template column name and a plurality of field names; Read the fields of the source data file according to the import rule configuration, import the data according to the reading result, and generate a temporary table with the template column name as the table header; The verification function of each column of data in the temporary table is determined based on the import rule configuration, and the data of the temporary table is verified by using each verification function, so as to migrate the column data that passes the verification to the target table.

2. The data import configuration method according to claim 1, characterized in that: The determining of the target table configuration corresponding to the source data file includes: Determine the target table to which the source data file is to be imported, extract multiple field names in the target table, and determine the field attribute of each field name; A target table configuration corresponding to the source data file is generated according to each field name and the field attribute corresponding to each field name.

3. The data import configuration method according to claim 1, characterized in that: The generating the import rule configuration of the source data file based on the preset template configuration and the target table configuration includes: Filter out the field names corresponding to the field names in the target table configuration from the template fields in the preset template configuration, and combine the filter results with the default fields in the template configuration to form template column names; The import rule corresponding to each field name in the template column name is determined according to the template configuration, and the import rule configuration of the source data file is generated according to the template column name and each import rule.

4. The data import configuration method according to claim 1, characterized in that: The reading of fields from the source data file according to the import rule configuration includes: Determine the file format of the source data file and the field mapping rule corresponding to the file format in the import rule configuration; The field mapping rule is used to perform field mapping in the source data file, and data is read according to the mapping result.

5. The data import configuration method according to claim 1, characterized in that: The template list includes template fields and default fields; The step of importing data according to the read result and generating a temporary table with the template column name as a table header includes: Determine the field type of each field name in the template field, and perform data type conversion on the read result according to each field type to obtain corresponding conversion data under the template field; Generate corresponding default data under the default field based on the current data import time; A temporary table is generated according to the conversion data and the default data, with the template column name as a table header.

6. The data import configuration method according to claim 1, characterized in that: The step of determining the verification function for each column of data in the temporary table based on the import rule configuration includes: The field name corresponding to each column of data in the temporary table is determined, and the import rule corresponding to each field name is determined based on the import rule configuration, and each import rule includes a verification function.

7. The data import configuration method according to claim 1, characterized in that: The using various verification functions to perform data verification on the temporary table includes: For each check function, determine the column data corresponding to the check function in the temporary table, and use the check function to perform a logic check on the column data to obtain a check result; If the verification result is failed, the information is recorded in the error information column corresponding to the column data; If the verification result is passed, the error information column corresponding to the column data is not updated.

8. A data import configuration device, characterized in that: include: A rule configuration module, used to upload a source data file, determine a target table configuration corresponding to the source data file, and generate an import rule configuration for the source data file based on a preset template configuration and the target table configuration; The import rule configuration includes a correspondence between a template column name and a plurality of field names; A data import module, configured to read fields of the source data file according to the import rule configuration, and to import data according to the reading result, and to generate a temporary table with the template column name as the table header; The data verification module is used to determine the verification function of each column of data in the temporary table based on the import rule configuration, and use each verification function to perform data verification on the temporary table to migrate the column data that passes the verification to the target table.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the data import configuration method as described in any one of claims 1 to 7.

10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the data import configuration method according to any one of claims 1 to 7 are performed.

Images