A data processing method, an electronic device, and a storage medium

Abstract

The application discloses a data processing method, an electronic device and a storage medium, and relates to the technical field of data processing, and comprises the following steps: determining a working mode according to user input data; in a zero modeling mode / knowledge transfer mode, splitting a to-be-processed field into a semantic unit sequence, matching the semantic unit with a standard word library to generate a candidate standard code, constructing a candidate mapping relationship and calculating a mapping confidence to determine a target mapping relationship, then determining an inter-table association relationship of a target table, generating a data definition statement and a query language script, and generating a target table instance after successful verification; in a language optimization mode, matching an optimization rule library, constructing script fragments before and after optimization, and performing equivalence verification, and applying an optimization result after user confirmation. The scheme realizes automation and knowledge reuse of insurance data warehouse modeling, improves the accuracy and industry adaptability of field semantic understanding, and guarantees the correctness, auditability and production environment safety of script generation and optimization.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, electronic device and storage medium. Background Technology

[0002] When implementing data warehouse projects, enterprises need to process the raw data tables of their business systems into target models that meet regulatory reporting and other requirements. This process heavily relies on the manual experience of data architects and engineers, resulting in low efficiency and poor reusability. Among existing auxiliary tools, templates based on fixed rules can only recognize standardized named fields, making it difficult to handle commonly used abbreviations in the industry and incurring high maintenance costs. In recent years, solutions using large language models combined with retrieval enhancement generation technology have been introduced, but because they retrieve fragmented text snippets, they lack structured semantic relationships between fields and tables, leading to untraceable reasoning processes, poor interpretability, insufficient understanding of industry terminology, and a tendency to produce erroneous results. At the same time, general data governance tools do not have built-in industry standard thesaurus and regulatory dictionaries, failing to directly meet industry requirements. Furthermore, existing automatic optimization solutions for structured query languages ​​lack mandatory protection for critical business logic areas, posing a risk of altering business results in the production environment. Summary of the Invention

[0003] This application provides a data processing method, electronic device, and storage medium to at least address the following problems in the related technologies: data processing relies on human experience, resulting in low efficiency and non-reusability of cross-organizational experience; rule templates lack semantic understanding capabilities; retrieval enhancement generation architecture reasoning is untraceable and has weak understanding of industry terminology; general governance tools lack industry depth; and automatic optimization of structured query language lacks security protection mechanisms.

[0004] This application provides a data processing method, which includes: receiving user input data and determining a corresponding working mode based on the user input data; in response to the working mode being a zero-based modeling mode or a knowledge transfer mode, obtaining the field to be processed from the user input data, and semantically decomposing the field to be processed to obtain a semantic unit sequence, wherein the semantic unit sequence includes multiple semantic units and their arrangement order; matching the semantic units with a preset standard lexicon to obtain standard semantics, and generating candidate standard codes based on the standard semantics and the arrangement order of the multiple semantic units; and constructing a candidate mapping relationship between the field to be processed and the candidate standard codes. The system performs a query in a pre-defined mapping knowledge base to obtain a first query result; based on the first query result, the semantic matching degree between the candidate standard codes and the fields to be processed, it calculates the mapping confidence of the candidate mapping relationships; based on the mapping confidence, it determines the target mapping relationship from the candidate mapping relationships; based on the user input data, it determines the target table, and based on the target mapping relationship, it determines the inter-table relationships corresponding to the target table; based on the inter-table relationships, it generates the data definition statement and query language script corresponding to the target table, and performs an equivalence check on the query language script to obtain a first check result; in response to a successful first check result, it executes the data definition statement and query language script to generate a target table instance.

[0005] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above data processing methods.

[0006] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data processing methods.

[0007] This application determines the working mode based on user input data: In the zero-based modeling / knowledge transfer mode, the fields to be processed are split into semantic unit sequences, the semantic units are matched with a standard lexicon to generate candidate standard codes, candidate mapping relationships are constructed and mapping confidence is calculated to determine the target mapping relationship, thereby determining the inter-table relationships of the target table, generating data definition statements and query language scripts, and generating target table instances after successful verification; in the language optimization mode, the matching of the optimization rule base is optimized, script fragments before and after optimization are constructed and equivalence verification is performed, and the optimization results are applied after user confirmation. This solution automates insurance data warehouse modeling and knowledge reuse, improves the accuracy of field semantic understanding and industry adaptability, and ensures the correctness, auditability, and production environment security of script generation and optimization. Attached Figure Description

[0008] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application; Figure 2 A detailed flowchart of a data processing method provided in an embodiment of this application; Figure 3 This application provides a schematic diagram of field mapping multi-hop inference for a data processing method according to an embodiment of the present application. Figure 4 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application; Figure 5 A three-layer structure diagram of a knowledge base enhancement generation architecture provided in this application embodiment; Figure 6 This is an internal structural diagram of an electronic device for implementing a data processing method, provided as an embodiment of this application. Detailed Implementation

[0010] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0011] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0012] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0013] Embodiments of this application provide a data processing method, such as... Figure 1As shown, the data processing method includes: receiving user input data and determining the corresponding working mode based on the user input data; responding to whether the working mode is a zero-based modeling mode or a knowledge transfer mode, obtaining the fields to be processed from the user input data, and semantically decomposing the fields to be processed to obtain a semantic unit sequence, wherein the semantic unit sequence includes multiple semantic units and their arrangement order; matching the semantic units with a preset standard lexicon to obtain standard semantics, and generating candidate standard codes based on the standard semantics and the arrangement order of the multiple semantic units; constructing a candidate mapping relationship between the fields to be processed and the candidate standard codes, and then mapping the fields to be processed to a preset standard lexicon. The system performs a query in the knowledge base to obtain the first query result; based on the first query result, the semantic matching degree between the candidate standard codes and the fields to be processed, it calculates the mapping confidence of the candidate mapping relationship; based on the mapping confidence, it determines the target mapping relationship from the candidate mapping relationships; based on the user input data, it determines the target table, and based on the target mapping relationship, it determines the inter-table relationships corresponding to the target table; based on the inter-table relationships, it generates the data definition statement and query language script corresponding to the target table, and performs equivalence verification on the query language script to obtain the first verification result; in response to the first verification result being successful, it executes the data definition statement and query language script to generate the target table instance.

[0014] Specifically, the above method automatically transforms unstructured user input into standardized data models and executable scripts, reducing the cost of manual modeling, improving the automation and accuracy of data table construction, and ensuring the correctness of script execution through a verification mechanism.

[0015] In a specific implementation, in response to the first verification result being a failure, a verification failure message is generated, and the reason for the failure and the corresponding script fragment location are displayed to the user.

[0016] Furthermore, such as Figure 2 As shown, the data processing method further includes: in response to the working mode being language optimization mode, obtaining the language script to be optimized from the user input data, and matching the language script to be optimized with a preset optimization rule base to obtain the target optimization rule; performing the corresponding script optimization operation on the language script to be optimized according to the target optimization rule, and constructing multiple sets of pre-optimization script fragments and post-optimization script fragments; performing equivalence verification on the post-optimization script fragments based on the pre-optimization script fragments to obtain a second verification result; in response to the second verification result being successful, generating application confirmation information for the post-optimization script fragments, and applying the post-optimization script fragments to the language script to be optimized based on the user's confirmation operation on the application confirmation information to obtain the post-optimization language script; in response to the second verification result being unsuccessful, stopping the script optimization operation and recording the reason for the failure.

[0017] Specifically, the above method achieves automated optimization of language scripts, while ensuring that the optimization does not change the script semantics through equivalence checks, thereby improving script quality and execution efficiency and reducing optimization risks.

[0018] In a specific implementation, after applying the optimized script fragment to the language script to be optimized to obtain the optimized language script / stop the script optimization operation and record the reason for failure, the data processing method further includes: recording the optimized script fragment and the corresponding unoptimized script fragment to the optimization rule base.

[0019] In a specific implementation, S1. Working mode determination: The working mode is automatically determined based on the type of data input by the user. If only the source layer table structure and target table scope are input, then Mode 1: Processing from scratch is entered; if an existing data model and a new source table are input, then Mode 2: Knowledge transfer is entered; if an existing query language processing script is input, then Mode 3: Language optimization is entered. S2. If... Figure 3 As shown, field semantic processing (knowledge base enhanced generation core): Based on the knowledge base enhanced generation architecture, it retrieves field semantic entities from the standard thesaurus and historical mapping records from the mapping knowledge base, forming a structured relationship network of field—semantic unit—standard code—target field; multi-hop reasoning is performed along this relationship network to output candidate mapping relationships and confidence scores; fields with confidence scores below a preset threshold are marked for manual review. S3. Relationship inference: Based on the insurance industry subject domain rule base, such as the policy subject domain which must have product, channel, and insured dimensions, combined with primary key co-occurrence, dimension table pattern recognition, and reference model association instances, it infers the relationships between data tables required for the target table, and outputs the inference basis and confidence score for each relationship. S4. Rule Constraint Generation: Generate data definition statements and processing query language statements for the target table under preset constraints. Among these, P0 level 0 rules are optimization forbidden zones and absolutely cannot be changed; P1 level 1 rules are mandatory rules for performance bottlenecks; P2 level 2 rules are structural optimization suggestion rules; P3 level 3 rules are general rules for syntax optimization; and P4 level 4 rules are optional rules for configuration optimization. After generation, perform equivalence checks on the query language statements to verify that there is no expansion of join cardinality, null value semantics are consistent, aggregation results are consistent, and the correspondence between columns in joint queries remains unchanged. S5. Knowledge Accumulation: Write back the manually confirmed field mappings, relationships, and custom optimization rules to the knowledge base in a structured format, including a standard thesaurus, a mapping knowledge base, and an optimization rule base. This knowledge base is used for knowledge base enhancement and retrieval enhancement in the next processing task, forming a knowledge flywheel.

[0020] In a specific implementation, after users manually confirm (adopt / adjust / reject) field mapping relationships, inter-table relationships, and optimization rules, the confirmation results are written back to the knowledge base in the following ways: field mapping results are written back to the mapping knowledge base, and the historical hit rate of the mapping is updated synchronously; user-confirmed or newly added field semantic units are written back to the standard thesaurus; custom rules confirmed by users in language optimization mode are automatically upgraded to P3 rules and entered into the optimization rule base after passing equivalence verification; all write-back operations are accompanied by version numbers and operator identifiers, supporting version rollback and change traceability.

[0021] Furthermore, the language script to be optimized is matched with a preset optimization rule base to obtain target optimization rules, including: parsing the language script to be optimized to obtain multiple script fragments in the language script to be optimized; matching the script fragments with optimization rules in the optimization rule base; and in response to a successful match between a script fragment and any optimization rule in the optimization rule base, determining the successfully matched optimization rule as the target optimization rule.

[0022] Further, an equivalence check is performed on the optimized script fragment based on the unoptimized script fragment to obtain a second check result, including: executing the unoptimized script fragment and the corresponding optimized script fragment to obtain a first execution result and a second execution result, respectively; performing a join cardinality check on the first script execution result and the second script execution result to obtain a cardinality check result, wherein the join cardinality check is used to check whether the number of rows in the result table of the first script execution result is equal to the number of rows in the result table of the second script execution result; performing a null value check on the first script execution result and the second script execution result to obtain a null value check result, wherein the null value check is used to check whether the first script execution result and the second script execution result are equal to the number of rows in the result table of the second script execution result. The system checks whether the value of a field in the execution result of a first script is empty, and whether the value of the same field in the execution result of a second script is empty. It then performs an aggregate check on the execution results of the first and second scripts to obtain an aggregate check result, whereby the aggregate check verifies whether the aggregate value in the execution result of the first script is equal to the corresponding aggregate value in the execution result of the second script. If the cardinality check result, the null value check result, and the aggregate check result are all passed, the second check result is determined to be successful. If any one of the cardinality check result, the null value check result, or the aggregate check result is failed, the second check result is determined to be unsuccessful.

[0023] Furthermore, the corresponding working mode is determined based on the user input data, including: in response to the user input data including target table definition information, the working mode is determined to be the zero-based modeling mode; in response to the user input data including the target table data model, the working mode is determined to be the knowledge transfer mode; in response to the user input data including the language script to be optimized, the working mode is determined to be the language optimization mode.

[0024] Specifically, the above method enables the system to automatically recognize user intent without requiring users to manually select a mode, thereby improving interaction efficiency and ease of use.

[0025] In a specific implementation, this application adopts a dual-input enhancement mode: an industry reference model is superimposed on the basic input as an additional retrieval source, which can significantly improve the confidence level; the industry reference model is not a necessary input, and the system can still work normally with only the basic input. In the mapping confidence and final confidence calculation nodes, the target table data model in the user input data can also be used as an industry reference model for enhancement.

[0026] Furthermore, semantic segmentation of the field to be processed is performed to obtain a sequence of semantic units, including: segmenting the field to be processed according to a preset field naming rule to obtain multiple segmentation units; abbreviating and restoring the multiple segmentation units according to the abbreviation rules in the standard thesaurus to obtain multiple semantic units; and combining the multiple semantic units into a sequence of semantic units according to the order of their arrangement in the field to be processed.

[0027] Furthermore, the semantic units are matched with a preset standard lexicon to obtain standard semantics, and candidate standard codes are generated according to the standard semantics and the arrangement order among multiple semantic units. This includes: matching the semantic units with the root words in the standard lexicon to obtain the corresponding standard semantics; combining the standard semantics according to the arrangement order among multiple semantic units to obtain combined semantics; querying the standard codes corresponding to the combined semantics in the standard lexicon, and determining the queried standard codes as candidate standard codes.

[0028] Further, based on the first query result, the semantic matching degree between the candidate standard code and the field to be processed, the mapping confidence of the candidate mapping relationship is calculated, including: obtaining the semantic matching degree between the candidate standard code and the field to be processed, and multiplying the semantic matching degree by a first preset weight to obtain a first product; obtaining the historical hit rate of the candidate standard code from the first query result, and multiplying the historical hit rate by a second preset weight to obtain a second product; in response to the absence of a preset peer reference model, adding the first product and the second product to obtain the mapping confidence; in response to the existence of a peer reference model, querying the candidate mapping relationship in the peer reference model to obtain a second query result, and determining whether to use the preset added value as the reference contribution value of the peer reference model based on the second query result; multiplying the reference contribution value by a third preset weight to obtain a third product, and adding the first product, the second product and the third product to obtain the mapping confidence.

[0029] Specifically, the above methods improve the comprehensiveness and accuracy of mapping confidence assessment, and enhance the reliability of mapping decisions by leveraging historical data and peer experience.

[0030] In a specific implementation, obtaining the semantic matching degree between the candidate standard code and the field to be processed includes: obtaining multiple semantic units corresponding to the field to be processed, and the standard semantics corresponding to the semantic units in the standard lexicon; inputting the semantic units and the corresponding standard semantics into a preset word vector model to obtain a first vector representation corresponding to the semantic unit and a second vector representation corresponding to the standard semantics; calculating the cosine similarity between the first vector representation and the second vector representation; determining the cosine similarity as the semantic matching degree between the semantic unit and the corresponding standard semantics; and determining the semantic matching degree between the candidate standard code and the field to be processed based on the semantic matching degrees corresponding to the multiple semantic units.

[0031] Further, determining whether to use the preset bonus value as the reference contribution value of the peer reference model based on the second query result includes: in response to the second query result indicating that there is a historical mapping record in the peer reference model corresponding to the candidate standard code, using the preset bonus value as the reference contribution value of the peer reference model; in response to the second query result indicating that there is no historical mapping record in the peer reference model corresponding to the candidate standard code, determining the reference contribution value to be zero.

[0032] Further, determining the target mapping relationship from the candidate mapping relationships based on the mapping confidence score includes: in response to a mapping confidence score greater than or equal to a first preset threshold, determining the candidate standard code as the target standard code, and forming a target mapping relationship between the field to be processed and the target standard code, wherein the target mapping relationship is marked as automatically adopted; in response to a mapping confidence score less than the first preset threshold but greater than or equal to a second preset threshold, determining the candidate standard code as the target standard code, and forming a target mapping relationship between the field to be processed and the target standard code, wherein the target mapping relationship is marked as awaiting manual review; in response to a mapping confidence score less than the second preset threshold, maintaining the candidate mapping relationship, and marking the candidate mapping relationship as a candidate for display.

[0033] Specifically, the above method achieves hierarchical processing of mapping results, which ensures the automatic adoption of high-confidence mappings while retaining space for manual intervention and display of low-confidence results, thus balancing efficiency and accuracy.

[0034] In a specific implementation, the first preset threshold can be set to 0.85, the second preset threshold can be set to 0.60, and can also be adjusted according to the actual situation.

[0035] In a specific implementation, the mapping confidence score is calculated using a weighted formula: Confidence = w1 × SemanticScore + w2 × HistoryHitRate + w3 × ReferenceBoost. Here, w1, w2, and w3 are weights; SemanticScore is the semantic matching score; HistoryHitRate is the historical hit rate in the query results; and ReferenceBoost is an additional boost when the mapping exists in a peer reference model, typically with a value of 0.05. The three-level threshold rule is as follows: when Confidence ≥ 0.85, it is automatically adopted; when 0.60 ≤ Confidence < 0.85, it is marked as requiring manual review; when Confidence < 0.60, it is not adopted and is only displayed as a candidate.

[0036] Furthermore, determining the target table based on user input data includes: in response to the working mode being a zero-based modeling mode, determining multiple target fields required for the target table based on the target table definition information, and determining the target table based on the multiple target fields; in response to the working mode being a knowledge transfer mode, determining multiple target fields required for the target table based on the target table data model, and determining the target table based on the multiple target fields.

[0037] Further, the table relationships corresponding to the target table are determined based on the target mapping relationship, including: determining multiple source tables corresponding to the target table based on the target mapping relationship; in response to the existence of the same primary key field among multiple source tables, using the same primary key field as the first association basis and determining the first association confidence level corresponding to the first association basis; in response to the existence of a field in the source table that conforms to the preset primary key naming rule, using the field that conforms to the primary key naming rule as the second association basis and determining the second association confidence level corresponding to the second association basis; determining the association dimension corresponding to the subject domain to which the target table belongs from the preset subject domain rules, and determining the source tables to be associated with the target table based on the association dimension. The target table is used as the source table to be associated with as the third association basis, and the third association confidence level corresponding to the third association basis is determined. In response to the existence of a peer reference model, the association instance in the peer reference model that matches the source table is used as the fourth association basis, and the fourth association confidence level corresponding to the fourth association basis is determined. The final association confidence level is calculated based on at least one of the first association confidence level, the second association confidence level, the third association confidence level, and the fourth association confidence level. The inter-table association relationship is determined based on at least one of the first association basis, the second association basis, the third association basis, and the fourth association basis, as well as the final association confidence level.

[0038] Specifically, the above method comprehensively judges the relationships between tables from multiple dimensions, improves the accuracy and robustness of the relationship identification, and adapts to complex data environments.

[0039] In specific implementations, each association includes an inference chain containing the association basis type and association confidence level: The first association basis is primary key co-occurrence. When source table A and source table B have primary key fields with the same field name and data type, it is determined that the two can establish an association through this primary key, and the association confidence level is set to a high level, generally within the range of [0.6, 0.8]. The second association basis is dimension table pattern recognition. When the field names in the source table conform to the naming pattern of the dimension table primary key, an association is established, and the association confidence level is set to a high level, generally within the range of [0.8, 1]. The third association basis is subject domain rules. Based on the business rule that insurance industry policy subject domains are required to have three dimensions—product dimension, channel dimension, and insured dimension—it is inferred that the corresponding dimension table needs to be associated, and the association confidence level is set to a good level, generally within the range of [0.5, 0.6]. The fourth association basis is reference model enhancement. When the user provides a peer reference model from other data sources and the same association instance exists in the model, a preset bonus value, such as 0.05, is added to the original association confidence level, which can also be adjusted according to the actual situation.

[0040] Furthermore, based on the inter-table relationships, data definition statements and query language scripts corresponding to the target table are generated, including: determining the field mapping paths between source fields in multiple source tables and target fields in the target table based on the target mapping relationships; determining the connection methods between multiple source tables corresponding to the target table and the connection methods between them, wherein the connection methods include connection types and connection fields; generating the query language script content based on the field mapping paths and connection methods, wherein the query language script is used to extract source fields from multiple source tables according to the field mapping paths and load the extracted source fields into the target table according to the connection methods; generating data definition statements for the target table based on the target table definition information in response to the working mode being a zero-based modeling mode; and generating data definition statements for the target table based on the target table data model in response to the working mode being a knowledge transfer mode; wherein the data definition statements include field names and data types.

[0041] Specifically, the above method realizes the automated generation of data loading scripts from the source table to the target table, clarifies the field mapping and connection logic, and improves the efficiency of modeling and data integration.

[0042] In its specific implementation, the hierarchical query language optimization rule engine is divided into five levels based on business risk level: P0 is an optimization no-go zone, including business calculation logic, outer grouping operations, column mapping, and aggregate function semantics, which must not be changed; P1 is a performance bottleneck, including missing partition pruning and Cartesian products, which must be optimized; P2 is a structure optimization zone, including filtering before joining and rewriting subqueries into common table expressions, which is recommended for optimization; P3 is a syntax optimization zone, including changing all-field queries to explicit field lists and rewriting range judgment subqueries into join queries, which are routine optimization items; P4 is a configuration optimization zone, including adjusting compression format, adjusting the number of partitions, and adjusting parallelism, which are optional optimization items. Each optimization change records the script fragment before optimization, the script fragment after optimization, the equivalence check result, and the corresponding rule identifier, supporting item-by-item review and rollback.

[0043] Further, an equivalence check is performed on the query language script to obtain a first check result, including: executing the query language script to obtain the script execution result; performing a join cardinality check on the result table in the script execution result to obtain a cardinality check result, wherein the join cardinality check is used to check whether the number of rows in the result table is equal to the number of rows in the source table used as the join base table, and the number of rows in the join base table remains unchanged in the joins between multiple source tables; performing null value checks on the field values ​​in the script execution result to obtain a null value check result, wherein the null value check is used to check whether the value of the field when the value is null is consistent with the null value of the corresponding target table; and performing an equivalence check on the aggregate values ​​in the script execution result. Perform an aggregation check to obtain the aggregation check result, whereby the aggregation check is used to verify whether the aggregation value is consistent with the preset aggregation result; perform a joint query check on the query language script to obtain the query check result, whereby the joint query check is used to verify whether the fields located in the same column position in the multiple query branches of the joint query in the query language script correspond to the same target field; if the cardinality check result, null value check result, aggregation check result, and query check result are all passed, the first check result is determined to be successful; if any one of the cardinality check result, null value check result, aggregation check result, and query check result is failed, the first check result is determined to be unsuccessful.

[0044] Specifically, the above methods comprehensively verify the correctness of the script execution results, promptly identify problems such as join bloat, inconsistent handling of null values, aggregation errors, and misalignment of columns in joint queries, and ensure the reliability of the generated scripts.

[0045] In a specific implementation, Insurance Company A needs to process its source layer policy data into a regulatory reporting broad table (policy_circ_report). The target table has approximately 500 fields and needs to be associated with 7 source tables. Step S100: The user uploads the source tables (t_policy_main, t_insured, t_product, t_channel, etc., a total of 7 tables) and the target table specifications (standard field list). The system pattern determination module identifies it as Pattern 1: Modeling from scratch. Step S200: The field semantic inferencer triggers knowledge base enhancement generation retrieval for the target field policyno: 1st hop splitting → [POLICY, NO], 2nd hop root matching → policy + number, 3rd hop combination matching → candidate standard code {t_policy_main.PLCY_NO, t_policy_main.POLICY_ID}, 4th hop mapping knowledge base check → PLCY_NO is mapped to policyno in 3 historical insurance companies, 5th hop confidence calculation → 0.96, automatically adopted. Similarly, the knowledge base enhancement retrieval is triggered for insured_amount: the third hop yields candidate INSSRD_AMT, the fourth hop has a historical hit rate of 0.78, and the fifth hop has a confidence level of 0.84, triggering the manual review flag. Step S300: The association inferrer identifies that the target field risk_code_lv2 needs to be supplemented from the product dimension table t_product. Based on the primary key co-occurrence rule: t_policy_main.PROD_CD and t_product.PROD_CD have the same name and type, with a confidence level of 0.77; at the same time, the subject domain rule requires that the policy subject must have a product dimension, with a confidence level of 0.55; the combined output table association relationship is: t_policy_main LEFTJOIN t_product ON PROD_CD, based on the primary key co-occurrence rule and the subject domain rule, with a final confidence level of 0.66. Step S400: The data definition statement and query language generator generates a complete CREATE TABLE policy_circ_report statement and INSERT INTO policy_circ_report SELECT ... FROM t_policy_main LEFT JOIN t_product ... LEFT JOIN t_insured ... processing script under the P0-P4 constraints.The equivalence checker executes automatically: ① Join cardinality check: checks whether the left outer join operation ensures that the number of rows in the left table does not increase or decrease, and ensures that the main table records do not experience row expansion or loss due to the join operation; ② Null value check: checks whether the null value of the insured person's related fields conforms to the business definition of the field being null; ③ Aggregation check: checks whether there are grouping aggregation operations in the script. If not, this check is skipped; ④ Union query check: checks whether there are column mapping operations in the script. If not, this check is skipped. All four checks pass. Step S500: The user manually reviews and confirms the acceptance of insured_amount. The system increments the historical hit count of this mapping by 1 and updates the mapping knowledge base. At the same time, the user adds an optimization rule to the query language script to partition and prune t_policy_main by statistical month. After the equivalence check passes, this rule is upgraded to a P1 performance bottleneck rule and added to the optimization rule base.

[0046] In the specific implementation, Insurance Company A has completed the modeling of its regulatory reporting data. New customer Insurance Company B needs to build the same wide table for regulatory reporting data, but B's naming style for the source layer fields differs from A's (B uses pinyin abbreviations). Step S100: The user uploads Insurance Company A's existing regulatory reporting data model and Insurance Company B's source table. The system identifies this as Mode Two: Knowledge Transfer. Step S200: The retrieval is enhanced using Insurance Company A's model as a synchronous reference model. A retrieval is triggered on Insurance Company B's source field BAOXIAN_JINER: 1st hop splitting → [BAOXIAN, JINER], 2nd hop root matching → insurance + amount (Note: the standard thesaurus has already precipitated pinyin mapping rules), 3rd hop candidate → insured_amount / premium_amount, 4th hop query Insurance Company A's reference model → Insurance Company A uses insured_amount in this context (the main policy table), reference model enhancement +0.05, 5th hop confidence → 0.89, automatic adoption. Steps S300-S500: Following the same process as above, in the final B insurance company's 500 fields, 420 fields (84%) were automatically matched, 60 fields (12%) triggered manual review, and 20 fields (4%) required manual addition of new word roots.

[0047] In a specific implementation, a user submits an existing inefficient query language script, hoping to optimize it. Step S100: The system recognizes the input as a query language script and enters Mode 3: Language Optimization. Step S400: The system parses the query language script and scans the rule bases P0-P4. Three areas for optimization are identified: ① Missing partition pruning (P1); ② Changing the field query to an explicit field list (P3); ③ Rewriting the range judgment subquery into a join query (P3). Simultaneously, it is identified that the outer grouping operation cannot be removed (P0 forbidden zone, marked as not to be rewritten). The equivalence checker constructs comparison query statements before and after optimization for each optimization, verifying item by item: the number of data rows output before and after optimization remains consistent; the calculation results of aggregate functions within each group remain consistent; and the handling and output results of null fields remain consistent. After all verifications pass, the optimized query language script is output, along with a change details table (original fragment → new fragment → rule ID → equivalence check result). Step S500: The user confirms the application of all optimizations. The system records the query language optimization case in the case set of the optimization rule base for quick matching and reuse of similar query language scripts in the future.

[0048] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0049] Embodiments of this application also provide a data processing system, such as... Figure 4As shown, the data processing system includes: an input parsing module, used to parse the source table structure, target table definition information or target table data model, and optional peer reference models in the user input data; a pattern determination module, used to determine the corresponding working mode based on the user input data: in response to the user input data including target table definition information, the working mode is determined to be a zero-based modeling mode; in response to the user input data including the target table data model, the working mode is determined to be a knowledge transfer mode; in response to the user input data including the language script to be optimized, the working mode is determined to be a language optimization mode; and a knowledge base enhanced retrieval engine, used to semantically decompose the fields to be processed to obtain a semantic unit sequence, and to combine multiple semantic units in the semantic unit sequence... The system matches the meta-lexicon with a pre-defined standard lexicon to obtain standard semantics, and generates candidate standard codes based on the standard semantics and the order of multiple semantic units. It also performs queries in a pre-defined mapping knowledge base to obtain query results and retrieves the historical hit rate of candidate standard codes from the query results. A field mapping inferencer constructs candidate mapping relationships between the fields to be processed and candidate standard codes, calculates the mapping confidence of the candidate mapping relationships based on the query results, the semantic matching degree between the candidate standard codes and the fields to be processed, and determines the target mapping relationship from the candidate mapping relationships based on the mapping confidence. An inter-table association inferencer determines the source field corresponding to the target field in the target table based on the target mapping relationship, and then... The source fields determine multiple source tables corresponding to the target table; in response to the existence of the same primary key field among multiple source tables, the same primary key field is used as the first join basis, and the first join confidence level corresponding to the first join basis is determined; in response to the existence of fields in multiple source tables whose field names conform to the preset dimension table primary key naming rules, the fields that conform to the dimension table primary key naming rules are used as the second join basis, and the second join confidence level corresponding to the second join basis is determined; the join dimension corresponding to the subject domain to which the target table belongs is determined from the preset subject domain rules, and the source tables to be joined by the target table are determined according to the join dimension, the source tables to be joined by the target table are used as the third join basis, and the third join confidence level corresponding to the third join basis is determined; in response to the existence of A pre-defined peer reference model uses the association instances that match multiple source tables in the peer reference model as the fourth association basis and determines the fourth association confidence level corresponding to the fourth association basis. The final association confidence level is calculated based on the first, second, third, and fourth association confidence levels. The inter-table association relationships between multiple source tables are then determined based on the first, second, third, and fourth association basis and the final association confidence level, serving as the inter-table association relationships corresponding to the target table. A data definition statement and query language script generator is used to determine the field mapping path between source fields in multiple source tables and target fields in the target table based on the target mapping relationship.Based on the inter-table relationships, determine the multiple source tables corresponding to the target table and the connection methods between them, including the connection type and connection fields. Based on the field mapping paths and connection methods, generate the query language script content. In response to the zero-based modeling mode, generate the target table's data definition statement based on the target table's definition information. In response to the knowledge transfer mode, generate the target table's data definition statement based on the target table's data model; the data definition statement includes field names and data types. An equivalence validator is used to execute the query language script and obtain the script execution results. A join cardinality check is performed on the result table in the script execution results to obtain... The process includes: cardinality verification results, where the join cardinality verification checks whether the number of rows in the result table is equal to the number of rows in the source table used as the join reference table (a source table whose row count remains constant in joins between multiple source tables); null value verification is performed on the field values ​​in the script execution results to obtain null value verification results, where null value verification checks whether the value obtained when a field is null is consistent with the corresponding null value obtained in the target table; aggregation verification is performed on the aggregate values ​​in the script execution results to obtain aggregation verification results, where aggregation verification checks whether the aggregate values ​​are consistent with the preset aggregation results; and joint query verification is performed on the query language script to obtain query verification results, where joint query verification... This is used to verify whether fields in the same column of a joint query in a query language script correspond to the same target field. If the cardinality check, null value check, aggregation check, and query check all pass, the first check result is determined to be successful. If any one of these checks fails, the first check result is determined to be unsuccessful. A manual review interaction module is used to display the mapping confidence and the inference chain of the target mapping relationship, and to receive manual feedback. Specifically, if the mapping confidence is greater than or equal to a first preset threshold, the target mapping relationship is marked as automatically adopted. If the mapping confidence level is less than a first preset threshold but greater than or equal to a second preset threshold, the target mapping relationship is marked as awaiting manual review; if the mapping confidence level is less than the second preset threshold, the candidate mapping relationship is maintained and marked as a candidate for display; the knowledge accumulation module is used to obtain the user's confirmation operation on the target mapping relationship and write the target mapping relationship back to the mapping knowledge base based on the confirmation operation; it is also used to write back the user-added field roots to the standard thesaurus and write back the user-confirmed custom optimization rules to the optimization rule base; the version management module is used to record the version number and the person making the change in the standard thesaurus, mapping knowledge base, and optimization rule base, and supports version rollback.

[0050] like Figure 5As shown, the knowledge base enhancement generation architecture in this application consists of the following three layers: (1) Structured knowledge layer: ① Standard lexicon, which stores the semantic entities of insurance industry fields, including field roots such as INSRD=insured, AMT=amount, PLCY=policy, standard codes, regulatory dictionary fields, abbreviation rules, and subject domain affiliation; ② Mapping knowledge base, which stores the field mapping history and confidence level across data sources. Each record includes the source field, target field, source identifier, number of hits, and historical confidence level; ③ Optimization rule base, which stores hierarchical query language optimization rules. Each rule includes the rule identifier, priority, triggering condition, rewrite action, equivalence verification script, and whether it is an optimization forbidden zone. (2) Structured retrieval layer: Taking the source field INSSRD_AMT as an example, the five-hop reasoning process is as follows: The first hop is semantically split into two semantic units [INSSRD, AMT]; the second hop is semantic unit matching to determine the standard semantics of INSSRD = insured person and AMT = amount; the third hop is combined matching to obtain the candidate standard code INSURED_AMOUNT; the fourth hop is knowledge base enhancement, querying the historical mapping records of this field in other data sources in the mapping knowledge base; the fifth hop is confidence calculation, combining semantic matching degree, historical hit rate, and reference model enhancement to output the mapping confidence, and finally output the standard code, mapping confidence and complete reasoning link. The complete reasoning link includes the inter-table relationship. (3) Intelligent generation layer: Generate data structure definition statements and query language processing scripts based on the structured retrieval results; after generation, automatically execute four equivalence checks, including association cardinality expansion detection, null value semantic consistency check, aggregation caliber consistency check, and set operation column position mapping invariance check.

[0051] For a description of the features in the embodiments corresponding to the data processing system, please refer to the relevant descriptions in the embodiments corresponding to the data processing method, which will not be repeated here.

[0052] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data processing method embodiments.

[0053] When the program instructions are read and executed by one or more processors, they can also perform operations corresponding to the steps in the above method embodiments, as described above, and will not be repeated here. Reference Figure 6This exemplifies the architecture of an electronic device, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620 can communicate with each other via a communication bus 630.

[0054] The processor 610 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to achieve the technical solution provided in this application.

[0055] The memory 620 can be implemented as a read-only memory (ROM), random access memory (RAM), static storage device, dynamic storage device, etc. The memory 620 can store an operating system 621 for controlling the operation of the electronic device 600, and a basic input / output system (BIOS) 622 for controlling the low-level operations of the electronic device 600. Additionally, it can store a web browser 623, data storage management 624, and an icon / font processing system 625. The icon / font processing system 625 can be the application program that specifically implements the aforementioned steps in this embodiment. In summary, when implementing the technical solution provided in this application through software or firmware, the relevant program code is stored in the memory 620 and executed by the processor 610.

[0056] Input / output interface 613 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0057] Network interface 614 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0058] Bus 630 includes a pathway for transmitting information between various components of the device, such as processor 610, video display adapter 611, disk drive 612, input / output interface 613, network interface 614, and memory 620.

[0059] In addition, the electronic device 600 can also obtain information on specific claim conditions from a virtual resource object claim condition information database (not shown in the figure) for use in condition judgment.

[0060] It should be noted that although the above-described electronic device 600 only shows a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, a memory 620, and a bus 630, in specific implementations, the electronic device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the solution of this application, and does not necessarily include all the components shown in the figures.

[0061] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause an electronic device (which may be a personal computer, cloud server, or network device, etc.) to execute the methods of various embodiments or some parts of the embodiments of this application.

[0062] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above data processing method embodiments when it is run.

[0063] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0064] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above data processing method embodiments.

[0065] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above data processing method embodiments.

[0066] Any of the components, modules, units, parts, methods, and operations described herein can be implemented using software, firmware, hardware (e.g., fixed logic circuitry), manual processing, or any combination thereof. Alternatively or additionally, any functionality described herein can be performed at least in part by one or more hardware logic components, such as, but not limited to, a central processing unit (CPU), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), an application-specific standard product (ASSP), a system-on-a-chip (SoC), a complex programmable logic device (CPLD), a microprocessor (MCU), etc. The terms "system," "computing device," or "apparatus" as used herein encompass various means, devices, and machines for processing data, including, for example, one or more programmable processors, computers, SoCs, or combinations thereof. The apparatus may also include code that creates an execution environment for the computer program in question, such as code constituting processor firmware, a protocol stack, a database management system, an operating system, a cross-platform runtime environment, a virtual machine, or one or more combinations thereof. The aforementioned computer program (also known as a program, software, software application, app, script, or code) can be written in any form of programming language, including compiled or interpreted languages, declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, object, or other unit suitable for a computing environment.

[0067] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0068] The data processing method, electronic device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A data processing method, characterized in that, The method includes: Receive user input data and determine the corresponding working mode based on the user input data; In response to the working mode being a zero-based modeling mode or a knowledge transfer mode, the fields to be processed in the user input data are obtained, and the fields to be processed are semantically decomposed to obtain a semantic unit sequence, wherein the semantic unit sequence includes multiple semantic units and the arrangement order of the multiple semantic units; The semantic units are matched with a preset standard lexicon to obtain standard semantics, and candidate standard codes are generated according to the standard semantics and the arrangement order among the multiple semantic units. The fields to be processed are constructed into candidate mapping relationships with the candidate standard codes, and a first query result is obtained by querying in a preset mapping knowledge base; Based on the first query result, the semantic matching degree between the candidate standard code and the field to be processed, the mapping confidence of the candidate mapping relationship is calculated; The target mapping relationship is determined from the candidate mapping relationships based on the mapping confidence. The target table is determined based on the user input data, and the inter-table relationships corresponding to the target table are determined based on the target mapping relationship. Based on the relationships between the tables, generate the data definition statement and query language script corresponding to the target table, and perform equivalence verification on the query language script to obtain the first verification result; In response to the first verification result being successful, the data definition statement and the query language script are executed to generate a target table instance.

2. The data processing method according to claim 1, characterized in that, The method further includes: In response to the working mode being language optimization mode, the language script to be optimized is obtained from the user input data, and the language script to be optimized is matched with a preset optimization rule library to obtain the target optimization rule; According to the target optimization rules, the corresponding script optimization operation is performed on the language script to be optimized, and multiple sets of script fragments before and after optimization are constructed. Based on the script fragment before optimization, the script fragment after optimization is subjected to equivalence verification to obtain a second verification result; In response to the second verification result being successful, application confirmation information for the optimized script fragment is generated, and based on the user's confirmation operation of the application confirmation information, the optimized script fragment is applied to the language script to be optimized to obtain the optimized language script; In response to the second verification result being a failure, the script optimization operation is stopped and the reason for the failure is recorded.

3. The data processing method according to claim 1, characterized in that, Determining the corresponding working mode based on the user input data includes: In response to the user input data including target table definition information, the working mode is determined to be the zero-based modeling mode; In response to the user input data including the target table data model, the working mode is determined to be the knowledge transfer mode; In response to the user input data including the language script to be optimized, the working mode is determined to be language optimization mode.

4. The data processing method according to any one of claims 1-3, characterized in that, The step of calculating the mapping confidence of the candidate mapping relationship based on the semantic matching degree between the first query result, the candidate standard code, and the field to be processed includes: Obtain the semantic matching degree between the candidate standard code and the field to be processed, and multiply the semantic matching degree by a first preset weight to obtain a first product; Obtain the historical hit rate of the candidate standard code from the first query result, and multiply the historical hit rate by the second preset weight to obtain the second product; In response to the absence of a pre-defined peer reference model, the first product and the second product are added together to obtain the mapping confidence. In response to the existence of the peer reference model, the candidate mapping relationship is queried in the peer reference model to obtain a second query result, and it is determined whether to use the preset bonus value as the reference contribution value of the peer reference model based on the second query result. The reference contribution value is multiplied by a third preset weight to obtain a third product, and the first product, the second product, and the third product are added together to obtain the mapping confidence.

5. The data processing method according to claim 4, characterized in that, Determining the target mapping relationship from the candidate mapping relationships based on the mapping confidence includes: In response to the mapping confidence being greater than or equal to a first preset threshold, the candidate standard code is determined as the target standard code, and the field to be processed and the target standard code constitute the target mapping relationship, wherein the target mapping relationship is marked as automatically adopted; In response to the mapping confidence being less than the first preset threshold and greater than or equal to the second preset threshold, the candidate standard code is determined as the target standard code, and the field to be processed and the target standard code constitute the target mapping relationship, wherein the target mapping relationship is marked as pending manual review; In response to the mapping confidence being less than the second preset threshold, the candidate mapping relationship is maintained, and the candidate mapping relationship is marked as a candidate for display.

6. The data processing method according to claim 5, characterized in that, Determining the inter-table relationships corresponding to the target table based on the target mapping relationship includes: Based on the target mapping relationship, determine multiple source tables corresponding to the target table; In response to the existence of the same primary key field among the multiple source tables, the same primary key field is used as the first association basis, and the first association confidence level corresponding to the first association basis is determined; In response to the existence of a field in the source table that conforms to a preset primary key naming rule, the field that conforms to the primary key naming rule is used as the second association basis, and the second association confidence level corresponding to the second association basis is determined; The association dimension corresponding to the topic domain to which the target table belongs is determined from the preset topic domain rules, and the source table to be associated with the target table is determined according to the association dimension; The source table that the target table needs to be associated with is used as the third association basis, and the third association confidence level corresponding to the third association basis is determined. In response to the existence of the peer reference model, the association instance in the peer reference model that matches the source table is used as the fourth association basis, and the fourth association confidence level corresponding to the fourth association basis is determined; The final association confidence is calculated based on at least one of the first association confidence, the second association confidence, the third association confidence, and the fourth association confidence. The inter-table association relationship is then determined based on at least one of the first association basis, the second association basis, the third association basis, and the fourth association basis, as well as the final association confidence.

7. The data processing method according to claim 6, characterized in that, The step of generating the data definition statement and query language script corresponding to the target table based on the inter-table relationships includes: Based on the target mapping relationship, determine the field mapping path between the source fields in the multiple source tables and the target fields in the target table; Based on the relationships between the tables, determine the connection methods between the multiple source tables corresponding to the target table and the multiple source tables, wherein the connection methods include connection type and connection fields; Based on the field mapping path and the connection method, the script content of the query language script is generated, wherein the query language script is used to extract source fields from the multiple source tables according to the field mapping path, and load the extracted source fields into the target table according to the connection method; In response to the working mode being the zero-based modeling mode, the data definition statement for the target table is generated based on the target table definition information; In response to the working mode being the knowledge transfer mode, a data definition statement for the target table is generated based on the target table data model; The data definition statement includes field names and data types.

8. The data processing method according to claim 7, characterized in that, The step of performing an equivalence check on the query language script to obtain a first check result includes: Execute the query language script and obtain the script execution result; A connection cardinality check is performed on the result table in the script execution result to obtain the cardinality check result. The connection cardinality check is used to check whether the number of rows in the result table is equal to the number of rows in the source table that serves as the connection benchmark table. The number of rows in the connection benchmark table remains unchanged in the connections between the multiple source tables. Perform null value validation on the field values ​​in the script execution result to obtain null value validation results. The null value validation is used to verify whether the value obtained when the field value is empty is consistent with the null value obtained in the target table. The aggregated values ​​in the script execution result are subjected to aggregate verification to obtain an aggregate verification result, wherein the aggregate verification is used to verify whether the aggregated values ​​are consistent with the preset aggregated result; Perform a joint query validation on the query language script to obtain a query validation result. The joint query validation is used to verify whether the fields located in the same column position in the multiple query branches of the joint query in the query language script correspond to the same target field. If the cardinality check result, the null value check result, the aggregation check result, and the query check result are all passed, the first check result is determined to be successful. If any one of the cardinality check result, the null value check result, the aggregation check result, and the query check result fails, the first check result is determined to be a failure.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the data processing method as described in any one of claims 1-8 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1-8.

Images