Data processing method and device

Abstract

The invention relates to the technical field of artificial intelligence and data processing, in particular to a data format conversion and import method and device based on artificial intelligence. The method comprises the following steps: acquiring a source data file containing a plurality of fields, wherein the names, formats or units of the fields are not uniform; analyzing semantics of the source data through a large language model (LLM) or other artificial intelligence models, and determining a mapping relation with a target data structure; deducing or complementing missing or non-direct corresponding fields, including unit or format conversion, character string analysis, numerical calculation, time conversion and deduction based on external knowledge or business rules; and generating target data meeting the requirements of the target system, and importing the target data into a database or outputting the target data as a file in a specified format. The method can be suitable for various scenes, such as reckoning the year of birth according to the age, deriving the city according to the postcode, analyzing the packaging information to calculate the weight of a single piece, converting the exchange rate, reckoning the natural language time and the like. According to the method and the device, the intelligent import of the open data file can be realized without a fixed template, the manual operation cost is remarkably reduced, the data processing efficiency and accuracy are improved, and the method and the device have good universality and expansibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and data processing, and particularly relates to a method and device for intelligently converting source data into target data format using a large language model (LLM) and performing data transmission and import. BACKGROUND

[0002] In the process of traditional enterprise informatization, data import is a common requirement. Existing solutions usually require users to download fixed templates (such as Excel templates) and fill in according to preset fields such as "name", "gender", "position", etc., otherwise it cannot be imported.

[0003] This way has the following problems.

[0004] 1. Strong template constraints, high user learning cost.

[0005] 2. Field matching needs manual adjustment, low efficiency.

[0006] 3. The target system needs to be re-made when it is upgraded or changed, and the maintenance cost is high.

[0007] With the development of large language models, it has the ability of semantic understanding and format mapping. With the help of LLM, non-standardized or open format data can be intelligently analyzed and mapped to realize automatic data sorting and import. SUMMARY

[0008] TECHNICAL PROBLEM

[0009] The present application aims to solve the problems of complicated data processing process, easy to make mistakes and inflexible in the prior art, and provides an automatic data conversion and import method based on artificial intelligence large model.

[0010] TECHNICAL SCHEME

[0011] The present application provides a data processing method, comprising the following steps:

[0012] S101: Obtain source data.

[0013] The original data provided by the user can be Excel, CSV, text file, etc., and the fields can be non-standardized.

[0014] S102: Large model data arrangement.

[0015] Use artificial intelligence large model to analyze and arrange the source data. The large model can recognize and understand each field and its meaning in the source data, extract key information, and perform necessary data cleaning and preprocessing.

[0016] S103: Convert to target data format.

[0017] The processed data is converted into data that conforms to the format of the target data source according to the structure of the target data source (such as name, gender, position, birth year). This process includes field mapping, data type conversion, data verification, and other operations, which are handled by the large language model (LLM) through prompt words.

[0018] S104: Data transmission and import.

[0019] The converted data is transmitted to the target data source through the network or other means.

[0020] S105: Target data source.

[0021] The final data is successfully imported into the target database table, completing the entire data processing flow. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is the overall architecture diagram provided by the embodiments of the present application.

[0023] Figure 2 is the flow diagram provided by the embodiments of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical scheme and advantages of the present application clearer, the present application will be described in further detail below with reference to the accompanying drawings. The described embodiments should not be considered as limiting the present application, and all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0025] Suppose the user needs to import an Excel or CSV file containing product information into a target database table, and the source data and target database table fields are as follows.

[0026] Source data (Excel or CSV):

[0027] Product ID.

[0028] Product name.

[0029] Packaging (such as "500g x 12 boxes" or "box").

[0030] Price (including currency symbol, such as "$10.5" or "¥75").

[0031] Expiration date (mixed format of month / day / year).

[0032] Target database table fields:

[0033] Product code.

[0034] Product name.

[0035] Single piece weight (g).

[0036] Monetary uniform price (RMB).

[0037] Validity period (standard YYYY-MM-DD).

[0038] Step details

[0039] S101: Select Excel.

[0040] The user selects an Excel or CSV file containing product information through the system interface.

[0041] S102: Read Excel content

[0042] The system reads the content of the Excel or CSV file uploaded by the user, extracting field information such as product ID, product name, packaging, price, and validity period.

[0043] S103: Large model data processing.

[0044] Use artificial intelligence large model to analyze and process the read data, such as through the following prompt word request large model: "You are a professional data processing expert, the user will send the data to be processed, and the format of the target data source, please fully understand, analyze the format of the two data sources, process the data to meet the format of the target data source."

[0045] So the large model will process the data as follows:

[0046] 1 Analyze complex strings.

[0047] For complex strings in the "packaging" field (such as "500g×12box"), the large model can analyze and disassemble the single piece weight. For example, "500g×12box" will be parsed as a single piece weight of 500 grams.

[0048] 2 Unit conversion.

[0049] If the packaging unit is not grams (g), the large model will perform corresponding unit conversion. For example, if the packaging unit is kilograms (kg), it will be converted to grams (g).

[0050] 3 Format standardization.

[0051] For the "validity period" field, the large model will standardize the mixed format of month / day / year (YYYY-MM-DD). For example, "12 / 31 / 2023" will be converted to "2023-12-31".

[0052] 4 Exchange rate conversion.

[0053] For the "price" field, the large model will identify the currency symbol and convert prices other than RMB to RMB. For example, "$10.5" will be converted to the corresponding RMB amount.

[0054] S104: Convert to target data format.

[0055] According to the structure of the target database table, the processed data is converted into the target data format. Specifically, it includes:

[0056] Map "product ID" to "product code".

[0057] Map "product name" to "product name".

[0058] Store the parsed and converted single piece weight in the "single piece weight" field.

[0059] Store the exchange rate converted price in the "currency unified price" field.

[0060] Store the standardized validity period in the "validity period" field.

[0061] S105: Data transmission and import

[0062] The converted data is transmitted to the target database through the network or other means, and the import operation is performed. Ensure that the data is complete and accurate in the target database table.

[0063] S106: Target database table.

[0064] The final data is successfully imported into the target database table, completing the entire data processing process.

[0065] Example Suppose the user uploads an Excel file containing the following product information:

[0066] Item ID Item Name Packaging Price Expiration Date 1 Apple 500g x 12 boxes 10.5 12 / 31 / 2023 2 Banana Case 75 01 / 15 / 2024

[0067] Processing process

[0068] S101-S102: The user selects and uploads an Excel file, and the system reads the content.

[0069] S103: For the record with product ID 001:

[0070] Parse "500g x 12 boxes" to get single piece weight of 500 grams.

[0071] Convert "$10.5" to RMB (assuming the exchange rate is 1 USD = 7 RMB), get 73.5 RMB.

[0072] Convert "12 / 31 / 2023" to "2023-12-31".

[0073] For records with commodity ID 002:

[0074] "Case" cannot directly parse the weight of a single piece, which may require manual intervention or default value processing.

[0075] The price is already in Chinese yuan, no conversion is needed.

[0076] Convert "01 / 15 / 2024" to "2024-01-15".

[0077] S104: Convert the sorted data to the target data format.

[0078] S105-S106: Transmit and import the data into the target database table.

[0079] Conclusion

[0080] The present application provides an automatic data conversion and import method and system based on artificial intelligence large model, which significantly improves the efficiency and accuracy of data processing, reduces manual intervention, and is suitable for various data processing scenarios.

[0081] The above is only an embodiment of the present application, and is not intended to limit the protection scope of the present application. Any modifications, equivalent replacements and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An artificial intelligence-based data format conversion and import method, characterized by, Comprising the following steps: Source data acquisition: Acquire at least one non-standardized source data file, which includes one or more fields with inconsistent names, formats or units; Semantic analysis and mapping: Analyze the meaning of the source data fields through a large language model (LLM) or other artificial intelligence model, and determine the corresponding relationship between the fields and the target data structure based on semantic understanding; Field derivation and completion: For missing, ambiguous or non-directly corresponding fields in the source data, use artificial intelligence models for derivation or completion, including but not limited to: Unit or format conversion; String parsing and disassembly; Numerical calculation or time conversion; External knowledge or business rule inference; Target data generation: Convert the source data into a data format that meets the requirements of the target system, including database tables, interface data or files, according to the mapping and derivation results; Import or output: Import the target data into the predetermined system or output as a specified format file.

Images