Data platform heterogeneous CSV file analysis method and terminal

Abstract

The invention discloses a data platform heterogeneous CSV file analysis method and a terminal, and the method comprises the steps: recognizing a CSV file uploaded by a user through a language recognition model, obtaining a source country, reading the CSV file through a codec corresponding to the source country, obtaining a report type of the CSV file, obtaining a corresponding analysis template according to the report type, and carrying out the analysis of the CSV file according to the analysis template. According to the method, the CSV file is analyzed according to the analysis template, the CSV file is preprocessed based on the analysis template to obtain the CSV file conforming to the specification, the CSV file conforming to the specification is analyzed according to the analysis template, one analysis template does not need to be configured for each language of each report type, more analysis templates do not need to be maintained, and the analysis efficiency is higher, so that the analysis efficiency is improved, and the maintenance cost is reduced; and before analysis, the CSV file is preprocessed based on the analysis template to obtain the CSV file conforming to the specification, so that the analysis is more facilitated, the analysis efficiency is improved, and the data in the CSV file can be better applied after the analysis.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data platform heterogeneous CSV file parsing method and terminal. Background Art

[0002] CSV is a popular open data format, which is widely used in various fields due to its simplicity and effectiveness in storing and disseminating data. However, data published in this format usually does not conform to strict specifications, which makes it a difficult task to automatically extract data from CSV files.

[0003] When CSV files belong to the same type of report but involve different source countries, there will be a problem: for example, in other countries such as Japan / the United States / Mexico / Italy / Germany / France, the column names are in different languages, but they need to be uniformly identified as the same column during data analysis. If manual processing is used, the workload is huge and errors may occur.

[0004] Existing data platform heterogeneous CSV file parsing methods often rely on users to manually configure a large number of parsing templates. For example, for CSV files of the same report type but in different languages, parsing templates for each language are configured. When parsing, it is necessary to first find the parsing template of the corresponding type and language, and then perform the parsing process. This method results in high maintenance costs and low parsing efficiency. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a data platform heterogeneous CSV file parsing method and terminal to improve parsing efficiency and reduce maintenance costs.

[0006] In order to solve the above technical problems, a technical solution adopted by the present invention is: A data platform heterogeneous CSV file parsing method, comprising the steps of: Receive a CSV file uploaded by a user, and use a language recognition model to identify the CSV file to obtain the country of origin; Read the CSV file using a codec corresponding to the source country and obtain a report type of the CSV file; Acquire a corresponding parsing template according to the report type, and preprocess the CSV file based on the parsing template to obtain a CSV file that complies with the specification, wherein the parsing template includes CSV file column names in different languages ​​and database column names corresponding to the CSV file column names in different languages; The CSV file that complies with the specification is parsed according to the CSV file column names in different languages ​​in the parsing template and the database column names corresponding to the CSV file column names in different languages ​​to obtain a parsed data file.

[0007] In order to solve the above technical problems, another technical solution adopted by the present invention is: A data platform heterogeneous CSV file parsing terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: Receive a CSV file uploaded by a user, and use a language recognition model to identify the CSV file to obtain the country of origin; Read the CSV file using a codec corresponding to the source country and obtain a report type of the CSV file; Acquire a corresponding parsing template according to the report type, and preprocess the CSV file based on the parsing template to obtain a CSV file that complies with the specification, wherein the parsing template includes CSV file column names in different languages ​​and database column names corresponding to the CSV file column names in different languages; The CSV file that complies with the specification is parsed according to the CSV file column names in different languages ​​in the parsing template and the database column names corresponding to the CSV file column names in different languages ​​to obtain a parsed data file.

[0008] The beneficial effects of the present invention are: using a language recognition model to identify a CSV file uploaded by a user, obtaining the country of origin, using a codec corresponding to the country of origin to read the CSV file, and obtaining the report type of the CSV file, obtaining a corresponding parsing template according to the report type, and preprocessing the CSV file based on the parsing template to obtain a CSV file that meets the specifications, and parsing the CSV file that meets the specifications according to the parsing template. The report type of the present invention corresponds to the parsing template one by one, one report type corresponds to one parsing template, and the parsing template includes CSV file column names in different languages ​​and database column names corresponding to the CSV file column names in different languages, that is, there is no need to configure a parsing template for each language of each report type. After reading the CSV file, as long as the report type of the CSV file is known, the corresponding parsing template can be used for parsing, without maintaining more parsing templates, and the parsing efficiency is also higher, thereby improving the parsing efficiency and reducing the maintenance cost. Before parsing, the CSV file is preprocessed based on the parsing template to obtain a CSV file that meets the specifications, which is more conducive to parsing, improves the parsing efficiency, and also facilitates better application of the data in the CSV file after parsing. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A flowchart of a method for parsing heterogeneous CSV files on a data platform according to an embodiment of the present invention; Figure 2 This is a structural schematic diagram of a data platform heterogeneous CSV file parsing terminal according to an embodiment of the present invention; Figure 3 It is a parsing flow chart of a data platform heterogeneous CSV file parsing method in an embodiment of the present invention; Figure 4 This is a parsing diagram of a heterogeneous CSV file parsing method for a data platform according to an embodiment of the present invention. DETAILED DESCRIPTION

[0010] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0011] Please refer to Figure 1 , a data platform heterogeneous CSV file parsing method, comprising the steps of: Receive a CSV file uploaded by a user, and use a language recognition model to identify the CSV file to obtain the country of origin; Read the CSV file using a codec corresponding to the source country and obtain a report type of the CSV file; Acquire a corresponding parsing template according to the report type, and preprocess the CSV file based on the parsing template to obtain a CSV file that complies with the specification, wherein the parsing template includes CSV file column names in different languages ​​and database column names corresponding to the CSV file column names in different languages; The CSV file that complies with the specification is parsed according to the CSV file column names in different languages ​​in the parsing template and the database column names corresponding to the CSV file column names in different languages ​​to obtain a parsed data file.

[0012] From the above description, it can be seen that the beneficial effects of the present invention are: using a language recognition model to identify the CSV file uploaded by the user, obtaining the source country, using the codec corresponding to the source country to read the CSV file, and obtaining the report type of the CSV file, obtaining the corresponding parsing template according to the report type, and preprocessing the CSV file based on the parsing template to obtain a CSV file that meets the specifications, and parsing the CSV file that meets the specifications according to the parsing template. The report type of the present invention corresponds to the parsing template one by one, one report type corresponds to a parsing template, and the parsing template includes CSV file column names in different languages ​​and database column names corresponding to CSV file column names in different languages, that is, there is no need to configure a parsing template for each language of each report type. After reading the CSV file, as long as the report type of the CSV file is known, the corresponding parsing template can be used for parsing. There is no need to maintain more parsing templates, and the parsing efficiency is also higher, thereby improving the parsing efficiency and reducing the maintenance cost. Before parsing, the CSV file is preprocessed based on the parsing template to obtain a CSV file that meets the specifications, which is more conducive to parsing, improves parsing efficiency, and also facilitates better application of data in the CSV file after parsing.

[0013] Furthermore, the language recognition model is used to identify the CSV file, and the country of origin obtained includes: Using the trained ‌LangID model to identify the CSV file, and obtaining the language type of the CSV file; The source country is determined based on the language of the CSV file.

[0014] From the above description, we can see that the ‌LangID model can efficiently process large text data, is suitable for large-scale text processing needs, and has more accurate language recognition. Using the trained ‌LangID model to identify CSV files and obtain the language type of the CSV files, we can accurately and quickly determine the country of origin of the CSV files, avoiding encoding errors when reading CSV files later and affecting the parsing results.

[0015] Furthermore, after reading the CSV file using a codec corresponding to the source country and obtaining the report type of the CSV file, the method further includes: Check whether the CSV file is in a valid CSV format and obtain a check result.

[0016] The preprocessing of the CSV file based on the parsing template to obtain a CSV file that complies with the specification includes: The CSV file is preprocessed based on the parsing template according to the inspection result to obtain a CSV file that complies with the specification.

[0017] From the above description, it can be seen that the CSV file is first checked for CSV format, and then pre-processed based on the parsing template according to the check result, so that the processed CSV file can better meet the unified business specification requirements.

[0018] Further, the checking whether the CSV file is in a valid CSV format and obtaining the checking result include: The column names of the CSV file are checked according to the configuration information corresponding to the report type of the CSV file to obtain a check result.

[0019] From the above description, we can see that after knowing the report type of the CSV file, we can check the column names of the CSV file according to the configuration information corresponding to the report type of the CSV file. The configuration information contains the column names that should exist in a certain report type. By checking the column names of the CSV file, we can directly and quickly determine whether the column names of the file meet the requirements.

[0020] Further, the inspection result includes the presence of missing or redundant column names; The preprocessing of the CSV file based on the parsing template according to the inspection result to obtain a CSV file that complies with the specification includes: If the check result is that there are missing column names, determine whether the missing column names in the CSV file are required based on the database column names in the parsing template; if not, ignore the missing column names; if required, save the missing column names as error information in the parsing template; If the checking result is that there are redundant column names, the redundant column names are saved as error information in the parsing template; The data format and numerical precision of the columns in the CSV file are converted according to the data format and numerical precision corresponding to the database column names in the parsing template to obtain a CSV file that complies with the specifications.

[0021] From the above description, we can know that if the check result shows that there are missing column names or redundant column names, they will be saved as error messages in the parsing template so that users can confirm them in time to ensure the final parsing effect. The data requirements of CSV files from different source countries are different. According to the data format and numerical precision corresponding to the database column name in the parsing template, the data format and numerical precision of the columns in the CSV file are converted. The data format and numerical precision can be converted into the form required by the user to better use the data.

[0022] Furthermore, the parsing template also includes field types and field type attributes corresponding to CSV file column names in different languages; The CSV file that meets the specification is parsed according to the CSV file column names in different languages ​​in the parsing template and the database column names corresponding to the CSV file column names in different languages, and the parsed data file includes: Determine the column names of the CSV file in the target language from the parsing template according to the column names in the CSV file that complies with the specification; Parse the column names in the CSV file that conforms to the specifications into database column names corresponding to the CSV file column names in the target language, add field types and field type attributes corresponding to the CSV file column names in the target language to the database column names in the CSV file that conforms to the specifications, and obtain a parsed data file.

[0023] From the above description, it can be seen that the parsing template includes field types and field type attributes corresponding to the CSV file column names in different languages. According to the column names in the CSV file that conforms to the specifications, the CSV file column names in the target language can be found from the parsing template, and then the parsing can be completed according to the corresponding database column names, field types and field type attributes to obtain the parsed data file, which is more efficient and convenient.

[0024] Furthermore, it also includes: If the column names of the CSV file in the target language cannot be determined from the parsing template based on the column names in the CSV file that complies with the specifications, the column names in the CSV file that cannot be determined are saved in the parsing template, and a notification message is sent to the user.

[0025] From the above description, it can be seen that if the column name of the CSV file in the target language cannot be matched from the parsing template, the column name in the CSV file that cannot be determined to meet the specification can be saved in the parsing template, and a notification message can be sent to the user, so that the user can optimize the parsing template in time and improve maintenance efficiency.

[0026] Furthermore, it also includes: Performing data cleaning on the parsed data file to obtain a cleaned data file; Performing data verification on the cleaned data file to obtain a verification result; If the verification result is passed, the cleaned data file that passes the verification is stored in the database.

[0027] From the above description, it can be seen that after parsing, the data file is also cleaned and verified, and then stored in the database, so as to ensure the data quality of the CSV file.

[0028] Furthermore, it also includes: Obtain training data based on business needs; The training data is used to train the ‌LangID model to obtain a trained ‌LangID model.

[0029] From the above description, we can see that by obtaining training data according to business needs and using the training data to train the LangID model, we can train the model based on our own data for the current business scenario, which increases the scope of application and accuracy of the LangID model and improves the language recognition effect.

[0030] Please refer to Figure 2 Another embodiment of the present invention provides a data platform heterogeneous CSV file parsing terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in the above-mentioned data platform heterogeneous CSV file parsing method is implemented.

[0031] The above-mentioned data platform heterogeneous CSV file parsing method and terminal of the present invention can be applied to the scene where the CSV file needs to be parsed, and the following is explained through specific implementation methods: Please refer to Figure 1 , Figure 3 and Figure 4 , Embodiment 1 of the present invention is: A data platform heterogeneous CSV file parsing method, comprising the steps of: S1. Receive a CSV file uploaded by a user, and use a language recognition model to identify the CSV file to obtain the country of origin, such as Figure 3 As shown, specifically including S11-S13: S11. Receive the CSV file uploaded by the user.

[0032] S12. Use the trained ‌LangID model to identify the CSV file to obtain the language type of the CSV file.

[0033] S13. Determine the country of origin according to the language type of the CSV file.

[0034] For example, if the trained ‌LangID‌ model is used to identify a CSV file and the language is identified as Japanese, then it can be determined that the source country of the CSV file is Japan.

[0035] In an optional implementation manner, before S1, the method further includes: Acquire training data according to business needs; use the training data to train the ‌LangID model to obtain a trained ‌LangID model.

[0036] Among them, the training process of the LangID model is optimized through parallel processing to make full use of multi-core processors, and uses sharding technology similar to MapReduce to allow parallelization when running in specified memory. The train.py tool can be used to perform complete training. The specific training process is the existing technology. The present invention improves the training data, that is, the data required for business needs is added to the original data. For example, for a cross-border e-commerce company, it can know in advance which countries' languages ​​will be processed, and a large amount of relevant report data can be provided to the original training data for the languages ​​used. In this way, the trained model can meet the daily usage scenarios. If there are new languages ​​in the future, the training data can be updated and the model can be retrained.

[0037] S2. Read the CSV file using a codec corresponding to the source country, and obtain a report type of the CSV file.

[0038] In an optional implementation, the report type is a commodity report, an order report, a purchase report, or the like.

[0039] Reading the CSV file using the codec that corresponds to the country of origin ensures that no encoding errors occur when reading the file, which could lead to incorrect data being obtained after subsequent parsing.

[0040] For cross-border e-commerce companies, the file formats of different countries for the same report are different, as shown in Table 1. The first row of Table 1 is the abbreviations of different countries, and each subsequent row is the column name of the same column in different countries. It can be seen that some column names are available in the reports of some countries, but not in the reports of some countries. Therefore, it is necessary to check the format of the CSV file and pre-process it. See the subsequent steps for details.

[0041] Table 1 Listing information of different countries in the same report

[0042] S3. Check whether the CSV file is in a valid CSV format and obtain a check result, such as Figure 3 shown.

[0043] Specifically, the column names of the CSV file are checked according to the configuration information corresponding to the report type of the CSV file to obtain a check result, where the check result includes whether there are no missing or redundant column names or whether there are missing or redundant column names.

[0044] In an optional implementation, the configuration information may include the required column name, the data format of the required column name, the numerical precision, whether it is required, etc.

[0045] For example, if the report type of the CSV file is a product report, then check whether the column names of the CSV file are consistent with the required column names in the configuration information according to the configuration information corresponding to the product report, so as to determine whether there are missing or redundant column names in the CSV file or whether there are no missing or redundant column names.

[0046] In an optional implementation, the column names of the CSV file may be checked according to the data type, precision, and whether they are required to determine whether they meet the format requirements, thereby ensuring that the file data subsequently parsed is usable.

[0047] S4. Obtain a corresponding parsing template according to the report type, and pre-process the CSV file based on the parsing template to obtain a CSV file that meets the specifications, such as Figure 3 As shown, specifically including S41-S42: S41. Obtain a corresponding parsing template according to the report type, wherein the parsing template includes CSV file column names in different languages, database column names corresponding to CSV file column names in different languages, field types, and field type attributes. In an optional implementation, the field types include numeric types (such as int, tinyint, float, decimal), date and time types (such as DATE, TIME, YEAR, DATETIME, TIMESTAMP), string types (such as CHAR, VARCHAR, TINYBLOB, TINYTEXT, BLOB, TEXT, MEDIUMBLOB, MEDIUMTEXT, LONGBLOB, LONGTEXT), enumeration types, and spatial data types; the field type attributes include whether it can be empty, whether there is a default value, precision settings, etc.

[0048] One report type corresponds to one parsing template. For example, assuming that the CSV file belongs to a personnel information report, which includes a name column and a gender column, the CSV file column names in different languages ​​in the parsing template corresponding to the personnel information report and the database column names corresponding to the CSV file column names in different languages ​​can be in the form of: {name, name, Nom et prénom, vollständiger Name, nome completo}[CSV file column name]<=>{name}[database column name]; {gender, sex, Sexe, Geschlecht, Sesso}[CSV file column name]<=>{sex}[database column name], such as Figure 4 shown.

[0049] S42, pre-processing the CSV file based on the parsing template according to the inspection result to obtain a CSV file that complies with the specification, specifically including S421-S423: S421. If the check result shows that there are missing column names, determine whether the missing column names in the CSV file are required based on the database column names in the parsing template. If not, ignore the missing column names. If required, save the missing column names as error information in the parsing template.

[0050] When a required column name is missing in a CSV file, it is saved in the parsing template as an error message so that after parsing is completed, the user can confirm the CSV file based on the error message recorded in the parsing template.

[0051] S422. If the checking result is that there are redundant column names, the redundant column names are saved as error information in the parsing template.

[0052] When there are additional column names in the CSV file that are not in the parsing template, they are saved in the parsing template as error information so that after the parsing is completed, the user can modify the parsing template and re-parse it, which improves work efficiency and avoids data loss.

[0053] S423. Convert the data format and numerical precision of the columns in the CSV file according to the data format and numerical precision corresponding to the database column names in the parsing template to obtain a CSV file that complies with the specifications.

[0054] In an optional embodiment, the data format and numerical precision corresponding to the database column name in the parsing template are determined from the configuration information, and the data format and numerical precision of the columns in the CSV file are converted according to the data format and numerical precision corresponding to the database column name in the parsing template to obtain a CSV file that complies with the specifications.

[0055] For example, the format of the date column in the CSV file is month / day / year, while the format of the database date column required by the user is year / month / day; the numerical precision of the price column in the CSV file is 4 digits, while the precision of the database price column required by the user is 2 digits. In this case, the format of the date column in the CSV file is converted accordingly to obtain a CSV file that meets the specifications.

[0056] S5, parsing the CSV file that complies with the specification according to the CSV file column names in different languages ​​in the parsing template and the database column names corresponding to the CSV file column names in different languages ​​to obtain a parsed data file, specifically including S51-S53: S51. Determine the column names of the CSV file in the target language from the parsing template according to the column names in the CSV file that complies with the specification.

[0057] For example, a CSV file with France as its source country belongs to a personnel information report, including name and gender columns, specifically Nom et prénom and Sexe. Then, the column names of the CSV file in the target language can be found from the above parsing template, such as Figure 4 shown.

[0058] S52: Parse the column names in the CSV file that conforms to the specifications into database column names that correspond to the column names in the CSV file of the target language, and add field types and field type attributes that correspond to the column names in the CSV file of the target language to the database column names in the CSV file that conforms to the specifications, to obtain a parsed data file.

[0059] For example, after determining the column names of the target language CSV file, we can find that the database column names corresponding to Nom et prénom and Sexe are name and sex respectively. Then, Nom et prénom and Sexe in the CSV file that meets the specifications can be parsed into name and sex, such as Figure 4 As shown, add the corresponding field types and field type attributes.

[0060] By adding corresponding field types and field type attributes during parsing, it is convenient to use the data directly later and the data is more compliant.

[0061] S53. If the column names of the CSV file in the target language cannot be determined from the parsing template according to the column names in the CSV file that complies with the specifications, the column names in the CSV file that cannot be determined are saved in the parsing template, and a notification message is sent to the user.

[0062] In an optional implementation, the column names in the CSV file that cannot be determined to comply with the specification are saved as error messages in the parsing template, and notification information is sent to the user, which is more intelligent and reduces maintenance costs.

[0063] S6. Cleaning the parsed data file to obtain a cleaned data file.

[0064] S7. Perform data verification on the cleaned data file to obtain a verification result to ensure data integrity and accuracy.

[0065] S8. If the verification result is passed, the cleaned data file that has passed the verification is stored in the database.

[0066] S9. If the verification result is failed, obtain data of the unexpected column.

[0067] For example, the name of the CSV column expected to be obtained in the cleaned data file is: user_name, but product_user_name appears in the file. This is unexpected data, indicating that the other party's report has been optimized and upgraded, but this part of the data does not want to be discarded, so the database feature can be used to store JSON fields (if there are multiple columns, they can be stored in a dictionary) to obtain this part of the data and store it as {"product_user_name": value}. Later, a column can be added to the database, and then SQL statements can be used to complete the data in this row. In this way, there is no need to repeat the analysis to consume server resources. In addition, the parsing template can be supplemented according to this part of the data, and the parsing of the product_user_name column can be added to the parsing template.

[0068] The above steps can be implemented based on Python. Python can process data very conveniently and can flexibly handle various exceptions. The characteristics of Python language can be used to more flexibly configure the relationship between CSV files in different countries and languages ​​and the fields required for data analysis. By using the schematics library to parse files in different countries and languages, relevant content can be accurately parsed and output to the database.

[0069] The above-mentioned parsing method of the present invention enables the parsing of CSV files in different national languages ​​to be unified, and can ensure timely writing into the database, thereby shortening the parsing time; one report type corresponds to one parsing template, which is convenient for later maintenance, and can run automatically throughout the process without manual intervention. Abnormal columns can be discovered and processed in a timely manner, which can reduce the impact of upstream file data on analysis and improve data quality; in addition, automated processing minimizes the risk of human error and provides accurate results in related reports.

[0070] Please refer to Figure 2 , Embodiment 2 of the present invention is: A data platform heterogeneous CSV file parsing terminal includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in the above-mentioned data platform heterogeneous CSV file parsing method is implemented.

[0071] In summary, the present invention provides a data platform heterogeneous CSV file parsing method and terminal, which uses a language recognition model to identify the CSV file uploaded by the user, obtains the source country, uses the codec corresponding to the source country to read the CSV file, and obtains the report type of the CSV file, obtains the corresponding parsing template according to the report type, and pre-processes the CSV file based on the parsing template to obtain a CSV file that meets the specifications, and parses the CSV file that meets the specifications according to the parsing template. The report type of the present invention corresponds to the parsing template one by one, and one report type corresponds to one parsing template. The parsing template includes CSV file column names in different languages ​​and database column names corresponding to the CSV file column names in different languages, that is, there is no need to configure a parsing template for each language of each report type, and the CSV is read. After creating a CSV file, as long as you know the report type of the CSV file, you can use the corresponding parsing template to parse it. There is no need to maintain a large number of parsing templates, and the parsing efficiency is higher, thereby improving parsing efficiency and reducing maintenance costs. Before parsing, the CSV file is preprocessed based on the parsing template to obtain a CSV file that meets the specifications, which is more conducive to parsing, improves parsing efficiency, and also facilitates better application of the data in the CSV file after parsing. In addition, the LangID model can efficiently process large text data, which is suitable for large-scale text processing needs, and the language recognition is more accurate. The trained LangID model is used to identify the CSV file and obtain the language type of the CSV file. It can accurately and quickly determine the country of origin of the CSV file, avoiding encoding errors when reading the CSV file later and affecting the parsing results.

[0072] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data platform heterogeneous CSV file parsing method, characterized in that: Includes steps: Receive a CSV file uploaded by a user, and use a language recognition model to identify the CSV file to obtain the country of origin; Read the CSV file using a codec corresponding to the source country and obtain a report type of the CSV file; Acquire a corresponding parsing template according to the report type, and preprocess the CSV file based on the parsing template to obtain a CSV file that complies with the specification, wherein the parsing template includes CSV file column names in different languages ​​and database column names corresponding to the CSV file column names in different languages; The CSV file that complies with the specification is parsed according to the CSV file column names in different languages ​​in the parsing template and the database column names corresponding to the CSV file column names in different languages ​​to obtain a parsed data file.

2. A data platform heterogeneous CSV file parsing method according to claim 1, characterized in that: The language recognition model is used to identify the CSV file, and the country of origin obtained includes: Using the trained ‌LangID model to identify the CSV file, and obtaining the language type of the CSV file; The source country is determined based on the language of the CSV file.

3. A data platform heterogeneous CSV file parsing method according to claim 1, characterized in that: After reading the CSV file using a codec corresponding to the source country and obtaining the report type of the CSV file, the method further includes: Check whether the CSV file is in a valid CSV format, and obtain a check result; The preprocessing of the CSV file based on the parsing template to obtain a CSV file that complies with the specification includes: The CSV file is preprocessed based on the parsing template according to the inspection result to obtain a CSV file that complies with the specification.

4. A data platform heterogeneous CSV file parsing method according to claim 3, characterized in that: The checking whether the CSV file is in a valid CSV format and obtaining the checking result include: The column names of the CSV file are checked according to the configuration information corresponding to the report type of the CSV file to obtain a check result.

5. A data platform heterogeneous CSV file parsing method according to claim 3, characterized in that: The inspection results include the presence of missing or redundant column names; The preprocessing of the CSV file based on the parsing template according to the inspection result to obtain a CSV file that complies with the specification includes: If the check result is that there are missing column names, determine whether the missing column names in the CSV file are required based on the database column names in the parsing template; if not, ignore the missing column names; if required, save the missing column names as error information in the parsing template; If the checking result is that there are redundant column names, the redundant column names are saved as error information in the parsing template; The data format and numerical precision of the columns in the CSV file are converted according to the data format and numerical precision corresponding to the database column names in the parsing template to obtain a CSV file that complies with the specifications.

6. A data platform heterogeneous CSV file parsing method according to claim 1, characterized in that: The parsing template also includes field types and field type attributes corresponding to CSV file column names in different languages; The CSV file that meets the specification is parsed according to the CSV file column names in different languages ​​in the parsing template and the database column names corresponding to the CSV file column names in different languages, and the parsed data file includes: Determine the column names of the CSV file in the target language from the parsing template according to the column names in the CSV file that complies with the specification; Parse the column names in the CSV file that conforms to the specifications into database column names corresponding to the CSV file column names in the target language, add field types and field type attributes corresponding to the CSV file column names in the target language to the database column names in the CSV file that conforms to the specifications, and obtain a parsed data file.

7. A data platform heterogeneous CSV file parsing method according to claim 6, characterized in that: Also includes: If the column names of the CSV file in the target language cannot be determined from the parsing template based on the column names in the CSV file that complies with the specifications, the column names in the CSV file that cannot be determined are saved in the parsing template, and a notification message is sent to the user.

8. A data platform heterogeneous CSV file parsing method according to claim 1, characterized in that: Also includes: Performing data cleaning on the parsed data file to obtain a cleaned data file; Performing data verification on the cleaned data file to obtain a verification result; If the verification result is passed, the cleaned data file that passes the verification is stored in the database.

9. A data platform heterogeneous CSV file parsing method according to claim 1, characterized in that: Also includes: Obtain training data based on business needs; The training data is used to train the ‌LangID model to obtain a trained ‌LangID model.

10. A data platform heterogeneous CSV file parsing terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the processor implements each step of a method for parsing heterogeneous CSV files on a data platform as described in any one of claims 1 to 9.

Images