Intelligent Field Mapping Method and System Based on Large Language Model

Abstract

This invention discloses an intelligent field mapping method and system based on a large language model, including a task management module, a data acquisition and parsing module, an intelligent mapping and transformation module, and a result generation and delivery module. The task management module receives data processing task requests submitted by users and creates and initializes data processing task instances based on these requests. The data acquisition and parsing module, connected to the task management module, schedules distributed acquisition nodes to perform data capture operations based on the data source identifier and source platform type in the task instance. The intelligent mapping and transformation module is connected to the data acquisition and parsing module. This invention offers advantages such as end-to-end automation, significantly improved efficiency, enhanced data quality and accuracy, good adaptability and scalability, traceability and interpretability, and continuous self-optimization.

Description

Technical Field

[0001] This invention belongs to the field of data integration and automated processing technology, specifically relating to an intelligent field mapping method and system based on a large language model. Background Technology

[0002] In the process of enterprise digital transformation, the diversified development of e-commerce businesses necessitates the integration of product data from different heterogeneous e-commerce platforms to support core business functions such as internal inventory management, marketing analysis, and customer profiling. Current data integration processes primarily rely on manual processing or traditional ETL (Extract-Transform-Load) tools, which suffer from the following significant technical shortcomings:

[0003] 1. Fragmented processes and inefficiency

[0004] Current data processing requires a phased approach: first, data collection rules are manually configured; then, data is captured; next, structured data is manually organized; then, the mapping relationship between source and target fields is manually established; then, transformation scripts are written; next, the data is validated; and finally, the data is imported into the target system. Each phase requires the involvement of personnel with different roles, resulting in a fragmented process and a long processing cycle for product data, which fails to meet the company's need for timely data delivery.

[0005] 2. Manual Mapping and Common Mistakes

[0006] Traditional ETL tools require manual coding of field mapping rules. Due to the significant differences in field naming rules among different e-commerce platforms, manual judgment is prone to semantic misjudgment, resulting in a high mapping error rate. At the same time, when the target system template is updated or the field structure of the source platform is adjusted, the mapping rules need to be manually modified again, resulting in extremely high maintenance costs.

[0007] 3. Poor adaptability and scalability

[0008] The existing tools' data collection and mapping logic is highly coupled to specific platforms. Adding a new e-commerce platform requires redeveloping the data collection strategy, parsing rules, and mapping templates, resulting in a long development cycle. In addition, traditional tools only support one-to-one field mapping and cannot handle complex scenarios such as "merging multiple source fields into one target field" or "splitting one source field into multiple target fields," resulting in serious lack of adaptability.

[0009] 4. Technical barriers and lack of traceability

[0010] Traditional ETL tools require users to master SQL, Python scripting, or specific tool configuration languages. Non-technical personnel cannot independently complete data import tasks. At the same time, existing tools do not record the basis for generating mapping rules. When data errors occur, it is impossible to trace whether the error is due to mapping logic errors, transformation script errors, or abnormal source data. This results in low troubleshooting efficiency and further extends the data processing cycle.

[0011] With the powerful natural language understanding and semantic reasoning capabilities of large language models (LLMs) such as GPT-4 and Wenxin Yiyan, which can accurately identify the semantic relationships between different fields, a certain degree of support has been provided for breaking through the technical bottleneck of traditional data integration. In order to solve the above problems, there is an urgent need in this field for an intelligent field mapping solution based on large language models that can achieve full-process automation, support heterogeneous platform adaptation, reduce technical threshold and have traceability. Summary of the Invention

[0012] To address the shortcomings of existing technologies, this invention provides an intelligent field mapping method and system based on a large language model that enables automation, standardization, and intelligence of the entire process of e-commerce product data collection and warehousing. This method solves problems such as process fragmentation, low efficiency of manual processing, high error rate, and poor adaptability in existing technologies.

[0013] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0014] An intelligent field mapping system based on a large language model includes:

[0015] The task management module is used to receive data processing task requests submitted by users. The request includes a data source identifier, a source platform type, and a target data template identifier. The task management module creates and initializes a data processing task instance based on the request.

[0016] The data acquisition and parsing module, connected to the task management module, is used to schedule distributed acquisition nodes to perform data capture operations based on the data source identifier and source platform type in the task instance, and parse the captured unstructured or semi-structured data into a structured source data set containing multiple source fields and corresponding values.

[0017] The intelligent mapping and conversion module, connected to the data acquisition and parsing module, is used to receive the structured source data set and the target data template, and realize the automated semantic mapping and data conversion between the source fields and the target fields through the whole process of semantic parsing, intelligent reasoning, logical generation and verification optimization.

[0018] The system also includes a result generation and delivery module, which is connected to the intelligent mapping and transformation module. This module generates a target data file from the final data after mapping, transformation, and verification, according to the field order and format specified in the target data template. The target data file is then stored in an object storage service or pushed to the downstream target business system via an API.

[0019] Preferably, the intelligent mapping and transformation module includes a semantic modeling unit, a mapping generation and evaluation unit, a transformation logic generation unit, a consistency verification and conflict resolution unit, and an incremental learning unit;

[0020] The output of the semantic modeling unit is connected to the input of the mapping generation and evaluation unit. The semantic modeling unit is used to load metadata associated with the target data template, construct a semantic intent description for each target field that includes field name, business meaning, data type, constraints and data examples, and normalize and semantically represent each source field in the structured source data set.

[0021] The output of the mapping generation and evaluation unit is connected to the input of the transformation logic generation unit. The mapping generation and evaluation unit is used to input the semantic representation of the source field and the semantic intent description of the target field into the pre-trained large language model, and perform semantic similarity matching through the large language model to generate at least one candidate mapping relationship.

[0022] The output of the conversion logic generation unit is connected to the input of the consistency verification and conflict resolution unit. The conversion logic generation unit is used to automatically select or combine predefined atomic conversion functions to form an executable data conversion chain based on the constraints and data format requirements of the target field for the established mapping relationship.

[0023] The output of the consistency verification and conflict resolution unit is connected to the input of the incremental learning unit. The consistency verification and conflict resolution unit is used to verify the transformation result according to the constraints in the semantic intent description of the target field after data transformation. In case of verification failure, mapping relationship confidence lower than the preset threshold, or conflict with multiple high confidence candidates, the unit will trigger a preset rollback strategy or submit the data to the human-machine collaboration interface for user decision.

[0024] Incremental learning units are used to capture and store mapping relationships and transformation logic that have been confirmed or corrected by the user, forming a knowledge base. This knowledge base is used to fine-tune the large language model or as a high-priority rule applied to subsequent similar tasks.

[0025] Preferably, the candidate mapping relationship is accompanied by a confidence score and reasoning basis, and the candidate mapping relationship includes one-to-one, one-to-many, or many-to-one forms.

[0026] Preferably, the data source is identified as a URL or store ID; the unstructured or semi-structured data includes HTML page data or API response data; the atomic transformation functions include extracting JSON paths, truncating strings, formatting dates, normalizing price units, or extracting the first element of an array.

[0027] Preferably, the constraints include required attributes, enumeration value range, data length limit, or data precision requirement; the preset rollback strategy is to call previously confirmed mapping rules or conversion logic.

[0028] This invention also discloses an intelligent field mapping method based on a large language model, implemented based on the aforementioned intelligent field mapping system based on a large language model, comprising the following steps:

[0029] S101: Create Task: Receives a data processing task request containing the data source identifier, source platform type and target data template identifier, and creates and initializes a data processing task instance;

[0030] S102: Data Acquisition and Parsing: Based on the data source identifier and source platform type in the task instance, dispatch distributed acquisition nodes to perform data capture, and parse the captured unstructured or semi-structured data into a structured source data set containing multiple source fields and corresponding values.

[0031] S103: Intelligent Mapping and Transformation:

[0032] S103a: Semantic Modeling: Load the metadata associated with the target data template, construct a semantic intent description for each target field, and normalize and semantically represent the source fields in the structured source data set.

[0033] S103b: Generate mapping: Input the semantic representation of the source field and the semantic intent description of the target field into the pre-trained large language model, and generate candidate mapping relationships with confidence scores and reasoning basis through the large language model;

[0034] S103c: Generation Transformation Logic: For established candidate mapping relationships, based on the constraints and data format requirements of the target field, predefined atomic transformation functions are automatically combined to form a data transformation chain;

[0035] S103d: Verification and Resolution: Perform consistency verification on the conversion result based on the constraints of the target field. If the verification fails or there is a mapping conflict, trigger a rollback strategy or submit it to the human-computer collaboration interface for user decision.

[0036] S104: Result delivery: According to the format requirements of the target data template, generate the target data file from the verified final data, and store the target data file in the object storage service or push it to the downstream target business system;

[0037] S105: Learning and Iteration: Store the mapping relationships and transformation logic confirmed or corrected by the user into the knowledge base, and use the knowledge base to fine-tune the large language model or optimize the processing rules for subsequent similar tasks.

[0038] Preferably, in step S102, a load balancing strategy is adopted when scheduling distributed collection nodes. The load balancing strategy includes allocating capture tasks according to the current task volume of the collection node, or allocating collection nodes according to the nearest region of the data source. When parsing unstructured data, key tags or key fields in the data are extracted first, and then a structured source data set is generated through regular expressions or structured templates.

[0039] Preferably, in step S103a, when performing semantic representation on the source field, word segmentation, part-of-speech tagging, and keyword extraction are used to convert the text content of the source field into a vector or text description containing business semantics; when constructing the semantic intent description of the target field, the historical mapping records and business description documents of the target field are read from the metadata database to supplement the completeness of the semantic intent description.

[0040] Preferably, in step S103d, the consistency check includes:

[0041] Data type validation: Checks whether the converted data matches the preset data type of the target field;

[0042] Constraint validation: Check whether required fields are not empty, whether enumerated field values ​​are within the preset range, and whether data length meets the limit;

[0043] Logical consistency check: Check whether the multi-field association is reasonable. The multi-field association includes the numerical logic of the price field and the discount field or the order logic of the date field.

[0044] By adopting the above technical solution, the present invention has the following beneficial effects:

[0045] (1) Traditional manual processing of e-commerce product data takes a long time. This invention automates the entire process from collection to parsing to mapping to conversion to verification and finally to delivery, which shortens the processing time and greatly improves the processing efficiency. At the same time, through distributed collection nodes and rule priority application mechanism, it supports single task batch processing of product data to meet the needs of large-scale data import of enterprises.

[0046] (2) This invention significantly reduces data errors through LLM semantic-level mapping, multi-dimensional consistency verification and conflict resolution mechanisms;

[0047] (3) The system of the present invention supports the adaptation of heterogeneous e-commerce platforms through “collection strategy library + semantic modeling + incremental learning”. When adding a new platform, only the collection strategy needs to be configured, and the core mapping logic does not need to be modified. The development cost is lower than that of traditional methods. At the same time, it supports complex mapping scenarios such as one-to-one, many-to-one, and one-to-many, and adapts to the diverse data integration needs of enterprises.

[0048] (4) The system of the present invention provides a visual front-end interface. Users do not need to write scripts or master complex tools. They only need to input the data source identifier, select the platform type and target template to complete the task initiation. The human-computer collaboration interface provides conflict solution suggestions and historical case references. Non-technical personnel can independently complete the data import, reducing the dependence on developers.

[0049] (5) The present invention has traceability and self-optimization capabilities. The system records the entire process log of the task, and data problems can be accurately located to specific links. Through knowledge base accumulation and LLM fine-tuning, the system automation rate gradually increases from the initial stage, and the mapping accuracy also gradually increases. As the usage time increases, the system performance continues to be optimized without manual intervention.

[0050] In summary, this invention has the advantages of enabling end-to-end automation, significantly improving efficiency, enhancing data quality and accuracy, having good adaptability and scalability, providing traceability and interpretability, and possessing continuous self-optimization capabilities. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the present invention. Detailed Implementation

[0052] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0053] The components of the embodiments of the invention described and shown in the accompanying drawings can typically be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention.

[0054] Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0055] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0056] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0057] Example 1

[0058] In this embodiment, an intelligent field mapping method and system based on a large language model is proposed. This aims to solve the technical problems of "inefficient process fragmentation, error-prone manual mapping, poor adaptability, and high technical barriers" in existing data integration processes. It achieves end-to-end automated processing of e-commerce product data from heterogeneous data sources to the target business system, improving data processing efficiency and quality, lowering the usage threshold for non-technical personnel, and possessing continuous self-optimization capabilities. This invention achieves full-process automation through modular design. Its core innovation lies in introducing a large language model to automatically generate semantic-level field mapping and transformation logic, and combining this with an incremental learning mechanism to continuously optimize system performance.

[0059] In one embodiment of the present invention, the intelligent field mapping system based on a large language model includes:

[0060] The task management module is used to receive data processing task requests submitted by users. The request includes a data source identifier, a source platform type, and a target data template identifier. The task management module creates and initializes a data processing task instance based on the request. Specifically, the data source identifier is a URL or a store ID.

[0061] The task management module serves as the user's interaction point with the system, responsible for task creation, status tracking, and log recording. Its core logic involves users submitting task requests through the front-end interface, requiring them to fill in information such as the data source identifier, source platform type, and target data template identifier. The data source identifier can be the URL of an e-commerce platform store's homepage, store ID, or product list API address. The source platform type can be selected from a dropdown list (multiple different platform types are possible). The system matches preset collection and analysis strategies based on the platform type. The target data template identifier can be a template corresponding to an internal business system (i.e., the internal business system used in this invention, such as Customer Experience Management (CEM), Customer Data Platform (CDP), Marketing Automation, and other SaaS (Software as a Service) platforms). The template predefines the metadata of the target fields (field name, type, constraints, etc.). The module creates a unique task ID and stores the task information (task ID, user ID, data source identifier, platform identifier, template identifier, creation time, task status) in the task database, allowing users to view task progress in real time (e.g., pending collection, parsing, mapping, completed, failed, etc.).

[0062] The data acquisition and parsing module, connected to the task management module, is used to schedule distributed acquisition nodes to perform data capture operations based on the data source identifier and source platform type in the task instance, and parse the captured unstructured or semi-structured data into a structured source data set containing multiple source fields and corresponding values. Specifically, the unstructured or semi-structured data includes HTML page data or API response data.

[0063] Understandably, the data acquisition and parsing module is positioned to automatically capture and structure the data from heterogeneous data sources, providing standardized input for subsequent mapping. Its core logic is acquisition strategy adaptation, distributed acquisition scheduling, and data parsing.

[0064] Specifically, collection strategy adaptation refers to the module receiving a task instance and then calling the appropriate collection rules from the strategy library based on the source platform type. Different platforms have different appropriate collection rules. Distributed collection scheduling refers to using load balancing algorithms (such as round-robin or least connections) to distribute collection tasks to multiple distributed nodes. Each node is configured with an independent IP proxy and user agent to prevent a single node from being blocked by the platform. At the same time, the nodes report the task progress to the scheduler in real time. If a node does not respond within a timeout period (e.g., more than 30 seconds), the scheduler automatically reassigns the task to other nodes to ensure collection reliability. Data parsing refers to the collection nodes transmitting the captured raw data (HTML / API response) to the parsing engine, which then calls preset parsing rules based on the platform type.

[0065] The intelligent mapping and conversion module, connected to the data acquisition and parsing module, is used to receive the structured source data set and the target data template, and realize the automated semantic mapping and data conversion between the source fields and the target fields through the whole process of semantic parsing, intelligent reasoning, logical generation and verification optimization.

[0066] The intelligent mapping and conversion module of this invention is the core of the intelligent field mapping system based on a large language model. It achieves intelligent field processing through semantic modeling, LLM mapping generation, conversion logic construction, verification and resolution, and incremental learning. Specifically, it includes five units: a semantic modeling unit, a mapping generation and evaluation unit, a conversion logic generation unit, a consistency verification and conflict resolution unit, and an incremental learning unit.

[0067] And a result generation and delivery module, connected to the intelligent mapping and transformation module, is used to generate a target data file from the final data after mapping, transformation and verification according to the field order and format specified by the target data template, and store the target data file in the object storage service or push it to the downstream target business system through API;

[0068] The function of the result generation and delivery module of this invention is to output the processed qualified data according to the target template format and deliver it to the downstream system. Its core logic includes target data file generation and data delivery method. Target data file generation refers to generating a standard file from the verified data according to the field order and format requirements of the target template. It supports multiple formats (such as Excel, CSV, etc.). The data delivery method is to upload the file to the object storage service, generate a time-sensitive download link, notify the user via email or SMS, or call the API interface of the downstream business system and adopt a batch push strategy. If the push fails, it will automatically retry (e.g., retry three times with an interval of 5 seconds). The retry failure is logged and the administrator is notified. The final data and task logs (collection records, mapping records, verification records) are stored in the archive database and retained for a certain period of time (e.g., 1 year) for traceability.

[0069] It is understood that the intelligent field mapping system and method based on the large language model of the present invention are based on the large language model, which is the common large language model on the market (such as GPT-4, Wenxin Yiyan, etc.).

[0070] The specific connection feedback relationships and functions of the five units of the intelligent mapping and conversion module of this invention are as follows:

[0071] The output of the semantic modeling unit is connected to the input of the mapping generation and evaluation unit. The semantic modeling unit is used to load metadata associated with the target data template, construct a semantic intent description for each target field that includes the field name, business meaning, data type, constraints, and data examples, and normalize and semantically represent each source field in the structured source data set. Specifically, the constraints include required attributes, enumerated value ranges, data length limits, or data precision requirements. The preset rollback strategy is to call previously confirmed mapping rules or transformation logic.

[0072] The semantic modeling unit transforms source and target fields into descriptions with unified semantic dimensions, providing clear input for LLM mapping. Its target field semantic intent construction refers to loading metadata of the target data template from the metadata database and generating a semantic intent description containing multi-dimensional information for each target field. The source field semantic representation refers to normalizing each source field in the structured source data, eliminating ambiguity and extracting semantic information, specifically including text cleaning, semantic extraction, and format unification.

[0073] The output of the mapping generation and evaluation unit is connected to the input of the transformation logic generation unit. The mapping generation and evaluation unit is used to input the semantic representation of the source field and the semantic intent description of the target field into the pre-trained large language model. The large language model performs semantic similarity matching to generate at least one candidate mapping relationship. Specifically, the candidate mapping relationship is accompanied by a confidence score and reasoning basis. The candidate mapping relationship includes one-to-one, one-to-many, or many-to-one forms.

[0074] Understandably, the mapping generation and evaluation unit utilizes the semantic reasoning capabilities of LLM to automatically generate mapping relationships between source and target fields and evaluate the reliability of the mapping. Specific steps include LLM selection and Prompt design, as well as mapping generation and evaluation. Specifically, a pre-trained large language model (such as GPT-4Turbo or Wenxin Yiyan 4.0) can be selected, supporting API calls. The Prompt adopts a structured format of "problem description + input data + output requirements". After receiving the Prompt, the LLM generates mapping conclusions through semantic similarity analysis. Moreover, the mapping generation and evaluation unit supports multiple types of mapping relationships, such as one-to-one mapping, many-to-one mapping, and one-to-many mapping.

[0075] The output of the conversion logic generation unit is connected to the input of the consistency verification and conflict resolution unit. The conversion logic generation unit is used to automatically select or combine predefined atomic conversion functions to form an executable data conversion chain based on the constraints and data format requirements of the target field for the established mapping relationship. Specifically, the atomic conversion functions include extracting JSON paths, truncating strings, formatting dates, normalizing price units, or extracting the first element of an array.

[0076] It is understandable that the conversion logic generation unit function is to automatically generate an executable data conversion chain for the established mapping relationship, ensuring that the source field value meets the target field requirements. In this invention, a variety of atomic conversion functions can be predefined to cover common data processing scenarios, such as data processing, data type conversion, string processing, array processing, date conversion, unit conversion, etc. The conversion chain generation logic filters candidate atomic functions according to the constraints and data format requirements of the target field, and then combines the functions in the order of "data extraction → format conversion → logic processing" to form a conversion chain. The conversion chain is stored in the form of an executable script.

[0077] The output of the consistency verification and conflict resolution unit is connected to the input of the incremental learning unit. The consistency verification and conflict resolution unit is used to verify the transformation result according to the constraints in the semantic intent description of the target field after data transformation. In case of verification failure, mapping relationship confidence lower than the preset threshold, or conflict with multiple high confidence candidates, the unit will trigger a preset rollback strategy or submit the data to the human-machine collaboration interface for user decision.

[0078] Understandably, the consistency verification and conflict resolution unit functions to verify the correctness of the transformation results, handle mapping conflicts, and ensure that the data meets the requirements of the target system. The consistency verification process includes data type verification, constraint verification, and logical consistency verification. Data type verification checks whether the transformed data matches the target field type. Constraint verification includes mandatory field verification, enumeration value verification, and length and precision verification. Logical consistency verification checks the logical relationship between multiple fields. The conflict resolution strategy of this invention includes automatic resolution and human-machine collaborative resolution. Automatic resolution is for low-risk conflicts, where the system automatically executes a preset strategy. Human-machine collaborative resolution is for high-risk conflicts (such as empty mandatory fields or logical conflicts). The system pushes the conflict item to the human-machine collaborative interface, which displays information such as the conflict field, conflict cause, candidate solutions, and historical cases. After the user selects a solution through the interface, the system records the user's operation and performs corresponding processing, while storing the solution in the knowledge base.

[0079] The incremental learning unit captures and stores user-confirmed or corrected mapping relationships and transformation logics to form a knowledge base. This knowledge base is used to fine-tune the large language model or as high-priority rules applied to subsequent similar tasks. The incremental learning unit forms a bidirectional connection with the consistency verification and conflict resolution unit and the mapping generation and evaluation unit. On the one hand, the "historical confirmed mapping rules" and "high-priority transformation logic" stored in the incremental learning unit are fed back to the mapping generation and evaluation unit as reference data, assisting it in prioritizing the use of reliable historical rules when generating new mapping relationships. On the other hand, the "historical conflict resolution cases" in the incremental learning unit are pushed to the consistency verification and conflict resolution unit, providing it with solution references for handling new conflicts and improving the efficiency of automated conflict resolution.

[0080] The incremental learning unit function of this invention is to continuously optimize LLM performance and system automation rate by accumulating user-confirmed mapping rules and transformation logic in a knowledge base. The knowledge base is designed to use a MySQL database for storage. The LLM optimization mechanism includes rule-priority application, model fine-tuning, and effect verification steps. Rule-priority application means that when processing subsequent similar tasks (same source platform + same target template), the system prioritizes searching for rules in the knowledge base. If a matching rule exists (source field and target field are consistent), the transformation chain is directly applied without calling the LLM, thereby improving the automation rate. Model fine-tuning means that every quarter, a certain number (1000+) of user-confirmed high-confidence rules (confidence ≥ 0.9) are selected from the knowledge base, and a fine-tuning dataset is constructed in the "Prompt-Response" format for incremental training of the LLM. Effect verification means that after fine-tuning, the model performance is verified through a test set (a certain number, such as 200 rules that were not involved in fine-tuning). If the mapping accuracy improves by more than a certain amount (such as an improvement of 5%, specifically from 88% to 94%), the online LLM version is updated; otherwise, it is rolled back to the original version.

[0081] This invention also discloses an intelligent field mapping method based on a large language model, such as automatically importing product data from an e-commerce platform X into a target business system A. Specifically, the intelligent field mapping method based on a large language model of this invention is implemented based on the aforementioned intelligent field mapping system based on a large language model, and includes the following steps:

[0082] S101: Create Task: Receives a data processing task request containing a data source identifier, source platform type, and target data template identifier. Creates and initializes a data processing task instance. Specifically, the user initiates a data processing task on the front-end interface, inputs the data source identifier, selects the source platform type, and the target template. After receiving the request, the task management module creates a task ID, initializes the task status, and stores the task information in the task database. More specifically, the user initiates a data import task on the front-end interface of business system A. He inputs the URL of a store homepage of e-commerce platform X as the data source identifier, selects "e-commerce platform X" in the platform type drop-down box, and selects "business system A product standard template v3.0" from the template list as the target template. The task management module in the system backend receives this information, creates a unique task ID, and records all associated information.

[0083] S102: Data Acquisition and Parsing: Based on the data source identifier and source platform type in the task instance, the distributed acquisition nodes are scheduled to perform data crawling. The crawled unstructured or semi-structured data is parsed into a structured source data set containing multiple source fields and corresponding values. Specifically, in step S102, a load balancing strategy is adopted when scheduling the distributed acquisition nodes. The load balancing strategy includes allocating crawling tasks according to the current task volume of the acquisition node, or allocating acquisition nodes according to the nearest region of the data source. When parsing unstructured data, key tags or key fields in the data are extracted first, and then a structured source data set is generated through regular expressions or structured templates. More specifically, when the data acquisition and parsing module receives the task, the scheduler selects an acquisition strategy that is compatible with the platform type "e-commerce platform X" and distributes the task to one or more distributed acquisition nodes. The acquisition nodes simulate user browsing behavior, traverse the product list pages and all product detail pages in the store, crawl the complete HTML content or call its hidden API. Subsequently, the parsing program extracts key information from the returned data to form a structured source data set.

[0084] S103: Intelligent Mapping and Transformation:

[0085] S103a: Semantic Modeling: Load the metadata associated with the target data template, construct the semantic intent description of each target field, and normalize and semantically represent the source fields in the structured source data set. Specifically, in step S103a, when performing semantic representation on the source fields, word segmentation, part-of-speech tagging, and keyword extraction are used to convert the text content of the source fields into vectors or text descriptions containing business semantics. When constructing the semantic intent description of the target field, the historical mapping records and business description documents of the target field are read from the metadata database to supplement the completeness of the semantic intent description. More specifically, the semantic modeling unit first loads the metadata of "Business System A Product Standard Template v3.0" from the database, which is the semantic intent description of the target field.

[0086] S103b: Generate mapping: Input the semantic representation of the source field and the semantic intent description of the target field into the pre-trained large language model, and generate candidate mapping relationships with confidence scores and reasoning basis through the large language model;

[0087] S103c: Generation of Transformation Logic: For the established candidate mapping relationship, based on the constraints and data format requirements of the target field, predefined atomic transformation functions are automatically combined to form a data transformation chain. More specifically, the mapping generation and evaluation unit and the transformation logic generation unit submit the source field (title, main_pictures, price_info, detail_attributes, item_url) and the intent description of the target field as input Prompt to the large language model. The LLM performs inference and generates the following mapping and transformation plan (expressed in internal DSL or JSON format).

[0088] S103d: Validation and Resolution: The conversion result is validated for consistency based on the constraints of the target field. If validation fails or mapping conflicts occur, a rollback strategy is triggered, or the result is submitted to the human-machine collaboration interface for user decision. Specifically, in step S103d, the consistency validation includes:

[0089] Data type validation: Checks whether the converted data matches the preset data type of the target field;

[0090] Constraint validation: Check whether required fields are not empty, whether enumerated field values ​​are within the preset range, and whether data length meets the limit;

[0091] Logical consistency check: Check whether the multi-field association is reasonable, including the numerical logic of the price field and the discount field or the order logic of the date field;

[0092] This invention executes a transformation chain to obtain transformed data. Subsequently, a consistency verification and conflict resolution unit checks whether each field of each data item meets the template requirements.

[0093] S104: Result Delivery: According to the format requirements of the target data template, the final data that has passed verification is used to generate a target data file. The target data file is then stored in the object storage service or pushed to the downstream target business system. More specifically, after all data rows have been successfully converted and verified, the result generation and delivery module will create an Excel file whose column headers are completely consistent with "Business System A Product Standard Template v3.0" (product_name, primary_image_url,...). This file is then uploaded to the company's object storage (such as AWS S3), and a time-sensitive secure download link is generated and returned to the user. Alternatively, the API interface of Business System A can be directly called to push the data into the database in batches. The task is then complete.

[0094] S105: Learning and Iteration: The user-confirmed or corrected mapping relationships and transformation logic are stored in the knowledge base. The knowledge base is used to fine-tune the large language model or optimize the processing rules for subsequent similar tasks. In step S105, when fine-tuning the large language model, an incremental training method is adopted. The mapping data confirmed by the user in the past 3 months in the knowledge base is divided into training set and validation set in an 8:2 ratio. Model fine-tuning is performed once every quarter. After fine-tuning, the model mapping accuracy is tested through the validation set. If the accuracy improves by more than 5%, the online model is updated. More specifically, if the user confirms on the interface that current_price should always be mapped to promo_price, the incremental learning unit stores this "promo_price->current_price" rule and its context (platform X, template v3.0) in the knowledge base. In the future, if there is an identical task, the system will prioritize using this verified rule without having to call LLM for inference again, realizing "one-time confirmation, permanent automatic".

[0095] This embodiment does not impose any limitation on the shape, material, structure, etc. of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention shall fall within the protection scope of the technical solution of the present invention.

Claims

1. An intelligent field mapping system based on a large language model, characterized in that, include: The task management module is used to receive data processing task requests submitted by users. The request includes a data source identifier, a source platform type, and a target data template identifier. The task management module creates and initializes a data processing task instance based on the request. The data acquisition and parsing module, connected to the task management module, is used to schedule distributed acquisition nodes to perform data capture operations based on the data source identifier and source platform type in the task instance, and parse the captured unstructured or semi-structured data into a structured source data set containing multiple source fields and corresponding values. The intelligent mapping and conversion module, connected to the data acquisition and parsing module, is used to receive the structured source data set and the target data template, and realize the automated semantic mapping and data conversion between the source fields and the target fields through the whole process of semantic parsing, intelligent reasoning, logical generation and verification optimization. And a result generation and delivery module, connected to the intelligent mapping and transformation module, is used to generate a target data file from the final data after mapping, transformation and verification according to the field order and format specified by the target data template, and store the target data file in the object storage service or push it to the downstream target business system through API; The intelligent mapping and transformation module includes a semantic modeling unit, a mapping generation and evaluation unit, a transformation logic generation unit, a consistency verification and conflict resolution unit, and an incremental learning unit. The semantic modeling unit is used to load metadata associated with the target data template, construct a semantic intent description for each target field including field name, business meaning, data type, constraints and data examples, and normalize and semantically represent each source field in the structured source data set. The mapping generation and evaluation unit is used to input the semantic representation of the source field and the semantic intent description of the target field into a pre-trained large language model, and perform semantic similarity matching through the large language model to generate at least one candidate mapping relationship. The transformation logic generation unit is used to automatically select or combine predefined atomic transformation functions to form an executable data transformation chain based on the constraints and data format requirements of the target field for the established mapping relationship. The consistency verification and conflict resolution unit is used to verify the transformation result according to the constraints in the semantic intent description of the target field after data transformation. In case of verification failure, mapping relationship confidence lower than the preset threshold, or conflict with multiple high confidence candidates, a preset rollback strategy is triggered or the data is submitted to the human-machine collaboration interface for user decision. Incremental learning units are used to capture and store mapping relationships and transformation logic that have been confirmed or corrected by the user, forming a knowledge base. This knowledge base is used to fine-tune the large language model or as a high-priority rule applied to subsequent similar tasks.

2. The intelligent field mapping system based on a large language model according to claim 1, characterized in that: The output of the semantic modeling unit is connected to the input of the mapping generation and evaluation unit, the output of the mapping generation and evaluation unit is connected to the input of the transformation logic generation unit, the output of the transformation logic generation unit is connected to the input of the consistency verification and conflict resolution unit, and the output of the consistency verification and conflict resolution unit is connected to the input of the incremental learning unit.

3. The intelligent field mapping system based on a large language model according to claim 2, characterized in that: The candidate mapping relationship is accompanied by a confidence score and reasoning basis, and the candidate mapping relationship includes one-to-one, one-to-many or many-to-one forms.

4. The intelligent field mapping system based on a large language model according to claim 2, characterized in that: The data source is identified as a URL or store ID; the unstructured or semi-structured data includes HTML page data or API response data; the atomic transformation functions include extracting JSON paths, truncating strings, formatting dates, normalizing price units, or extracting the first element of an array.

5. The intelligent field mapping system based on a large language model according to claim 2, characterized in that: The constraints include required attributes, enumeration value range, data length limit, or data precision requirement; the preset rollback strategy is to call previously confirmed mapping rules or conversion logic.

6. A method for intelligent field mapping based on a large language model, implemented based on the intelligent field mapping system based on a large language model as described in any one of claims 1-5, characterized in that, Includes the following steps: S101: Create Task: Receives a data processing task request containing the data source identifier, source platform type and target data template identifier, and creates and initializes a data processing task instance; S102: Data Acquisition and Parsing: Based on the data source identifier and source platform type in the task instance, dispatch distributed acquisition nodes to perform data capture, and parse the captured unstructured or semi-structured data into a structured source data set containing multiple source fields and corresponding values. S103: Intelligent Mapping and Transformation: S103a: Semantic Modeling: Load the metadata associated with the target data template, construct a semantic intent description for each target field, and normalize and semantically represent the source fields in the structured source data set. S103b: Generate mapping: Input the semantic representation of the source field and the semantic intent description of the target field into the pre-trained large language model, and generate candidate mapping relationships with confidence scores and reasoning basis through the large language model; S103c: Generation Transformation Logic: For established candidate mapping relationships, based on the constraints and data format requirements of the target field, predefined atomic transformation functions are automatically combined to form a data transformation chain; S103d: Verification and Resolution: Perform consistency verification on the conversion result based on the constraints of the target field. If the verification fails or there is a mapping conflict, trigger a rollback strategy or submit it to the human-computer collaboration interface for user decision. S104: Result delivery: According to the format requirements of the target data template, generate the target data file from the verified final data, and store the target data file in the object storage service or push it to the downstream target business system; S105: Learning and Iteration: Store the mapping relationships and transformation logic confirmed or corrected by the user into the knowledge base, and use the knowledge base to fine-tune the large language model or optimize the processing rules for subsequent similar tasks.

7. The intelligent field mapping method based on a large language model according to claim 6, characterized in that: In step S102, a load balancing strategy is adopted when scheduling distributed collection nodes. The load balancing strategy includes allocating capture tasks according to the current task volume of the collection node, or allocating collection nodes according to the nearest region of the data source. When parsing unstructured data, key tags or key fields are extracted from the data first, and then a structured source data set is generated through regular expressions or structured templates.

8. The intelligent field mapping method based on a large language model according to claim 6, characterized in that: In step S103a, when performing semantic representation on the source field, word segmentation, part-of-speech tagging, and keyword extraction are used to convert the text content of the source field into a vector or text description containing business semantics; when constructing the semantic intent description of the target field, the historical mapping records and business description documents of the target field are read from the metadata database to supplement the completeness of the semantic intent description.

9. The intelligent field mapping method based on a large language model according to claim 6, characterized in that: In step S103d, the consistency check includes: Data type validation: Checks whether the converted data matches the preset data type of the target field; Constraint validation: Check whether required fields are not empty, whether enumerated field values ​​are within the preset range, and whether data length meets the limit; Logical consistency check: Check whether the multi-field association is reasonable. The multi-field association includes the numerical logic of the price field and the discount field or the order logic of the date field.

Images