Intelligent data demand transfer design method and system based on large model and knowledge graph

Abstract

The invention provides a data demand intelligent conversion design method and system based on a large model and a knowledge graph, and the method comprises the steps: constructing a hierarchical knowledge storage system, and dynamically distributing a data model to a gold layer, a silver layer and a bronze layer for storage according to a comprehensive score; generating a data demand sorting list; performing knowledge recall based on a storage scheduling optimization algorithm; performing four-layer semantic quality inspection on the recall result; standardized design is generated, model knowledge passing quality inspection is packaged into constraint rules, and a large model is called through hierarchical Prompt engineering to generate a data design document. According to the method, the aperture matching accuracy is greatly improved, the constraint conflict rate is remarkably reduced, and high quality and consistency of output are ensured through a four-layer semantic matching quality inspection mechanism. The scheme effectively solves the problems of easy missing of key constraint conditions, non-uniform design standards, high error rate and the like in manual processing.

Description

Technical Field

[0001] This invention relates to the field of intelligent data management technology, specifically to a method and system for intelligent transformation of data requirements based on large models and knowledge graphs. Background Technology

[0002] In enterprise data processing workflows, the transformation from data requirements to data design relies heavily on manual operations. Data designers need to interpret requirements documents line by line, identify business entities and data attributes, and sort out logical relationships. This labor-intensive work mode is inefficient and struggles to cope with rapidly growing business demands.

[0003] Existing technologies mainly include rule template matching, knowledge retrieval, visual configuration tools, and general large model generation. Rule template matching extracts requirement keywords through predefined rules, but it lacks flexibility and struggles to handle diverse semantics. Knowledge retrieval systems employ single-path sequential scanning, requiring a full traversal of the knowledge base when handling multi-element requirements, resulting in low recall efficiency. Visual configuration tools are essentially still manual auxiliary tools and cannot achieve full-process automation. While general large models can generate SQL or data models, they lack enterprise domain knowledge guidance, often resulting in issues such as non-standard field naming and missing constraints.

[0004] The core flaw of these methods lies in their inability to effectively combine natural language understanding capabilities with structured enterprise knowledge. On the one hand, they struggle to accurately identify implicit business rules and cross-entity relationships scattered across different locations in requirements documents; on the other hand, the design experience and business knowledge accumulated by enterprises over a long period are fragmented and lack effective structured management and intelligent reuse mechanisms. This results in a significant amount of manual intervention required for each design, which is not only inefficient but also leads to inconsistent design standards due to differences in understanding among different designers, increasing the complexity of subsequent development and maintenance.

[0005] Therefore, there is an urgent need for an intelligent method that can deeply integrate the semantic understanding capabilities of large models with enterprise knowledge graphs to achieve efficient, accurate, and standardized transformation of data requirements into design solutions. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this invention proposes a data requirement intelligent transformation design method and system based on large models and knowledge graphs, solving the core pain points in the traditional data requirement transformation design process. In terms of efficiency improvement, it significantly shortens the retrieval path length, greatly reduces system response time, improves the hit rate of hot data layers, and significantly improves the overall processing speed.

[0007] To achieve the above objectives, this invention proposes a data requirement intelligent transformation design method based on large models and knowledge graphs, comprising the following steps: S1: Construct a hierarchical knowledge repository system. Divide the data model design document into three parts: heading chain, table content, and body summary using document segmentation technology. Use large-scale model semantic understanding technology combined with a domain synonym dynamic recognition model to design tables. Perform triple enhancement processing: in-document semantic fusion, out-of-document semantic fusion, and data lineage fusion based on AST syntax tree. Dynamically allocate the data model to the three layers of knowledge repository: gold, silver, and bronze, based on the comprehensive score. S2: Generate a data requirement sorting list, extract elements from the input data requirement document, identify requirement priorities, obtain constraints, and output a structured sorting list containing a list of requirement elements, priority levels, and constraints. S3: Based on the warehouse scheduling optimization algorithm, knowledge retrieval is performed. Different retrieval strategies are adopted according to the number of elements in the sorting order. For single-element demand, the corresponding warehouse area is directly located for retrieval. For multi-element demand, the real-time optimal decision algorithm is used to dynamically plan the retrieval path. S4: Perform four-layer semantic quality checks on the recall results, calculate the matching degree between the recall model and the demand from the entity layer, attribute layer, relationship layer and structure layer, and remove unqualified models based on the comprehensive matching degree. S5: Generate standardized designs by encapsulating the model knowledge that has passed quality inspection into constraint rules, and generating data design documents by calling the large model through the layered Prompt project.

[0008] Furthermore, the formula for calculating the comprehensive score in S1 is as follows:

[0009] in, This is the overall score; The constraint strength is defined as [0,1], and its value range is calculated based on the ratio of the number of strong constraint fields to the total number of fields. The access popularity is calculated based on the number of visits within 30 days, with a value range of [0,1]. The reusability is calculated based on the number of downstream references, with a value range of [0,1]. , , Let be the weighting coefficient, satisfying + + = 1; when Score ≥ 0.8, it is assigned to the Gold layer; when 0.6 ≤ Score < 0.8, it is assigned to the Silver layer; when Score < 0.6, it is assigned to the Bronze layer.

[0010] Furthermore, the intra-document semantic fusion in S1 specifically includes: Count the number of occurrences of each entity element. cnt When the same entity attribute conflicts in both the table and the text, select "entity". cnt Larger attributes are used as the final attributes; selected entity attributes are used as context to generate local fusion model design elements through large model prompts.

[0011] Furthermore, the data lineage fusion based on the AST syntax tree in S1 includes: The abstract syntax tree is extracted using SQL parsing tools to identify projection, aggregation, and filtering operators; the scope of the operators is analyzed, and lineage triples are constructed. ,in For the source field, For operator type, For the target field; if and only if When using a projection operator, perform field attribute expansion.

[0012] Furthermore, the instantaneous optimal decision-making algorithm in S3 includes: Define the set of semantic elements of the requirements Where n is the total number of elements; construct the warehouse area distance matrix. ,in: ; For the hierarchical distance, when and At the same node = 1, when at different nodes on the same level = 2, when crossing layers ,in , These are the levels where the elements are located; For the correlation distance, in The strength of the association between elements in a knowledge graph; , For distance weighting coefficients, satisfying + = 1.

[0013] Furthermore, the path planning process of the instantaneous optimal decision algorithm is as follows: Initialize the set of visited features ,in The highest priority feature; for each feature in the remaining feature set E\S Calculate candidate distances: ; in The element at the end of the current path. as elements Priority weight, higher priority = 2, medium priority = 1.5, low priority = 1; Select to make The smallest element is selected as the next search target; this process is repeated until S = E.

[0014] Furthermore, the formula for calculating the overall matching degree in S4 is as follows:

[0015] in, For overall matching degree; The entity layer matching degree is calculated using the cosine similarity of the BERT embedding vectors, with a value range of [0,1]. The attribute matching degree is calculated based on field completeness and type consistency, and its value range is [0,1]. The relational matching degree is calculated based on the degree to which foreign key constraints are satisfied, and its value ranges from [0,1]. The structural layer matching degree is calculated based on the similarity of the model architecture, and its value ranges from [0,1]. = 0.3、 = 0.2、 = 0.3、 = 0.2 is the hierarchical weight coefficient; when M < 0.6, it is judged as an unqualified model and is removed.

[0016] Furthermore, the layered Prompt project in S5 includes: In the instruction layer, the role is defined as "Senior Data Architect," and the original requirement description (context) and format constraints are injected. constraint Domain adaptation layer, injecting domain constraints. constraint and term mapping table term mapping The constraint strategy layer guides the large model to perform inference in the order of "identifying primary and foreign keys → mapping data types → generating constraints" through chain-thinking prompts.

[0017] Furthermore, it also includes a dynamic cargo location adjustment mechanism: Upgrade conditions are defined as follows: When H > 0.7 and R > 0.5 are met for 3 consecutive months, the model is upgraded from the current layer to the next layer; when the C value in a single month increases by more than 50% compared to the previous month, the model is upgraded to the Silver layer. Demotion conditions are defined as follows: When H < 0.2 is met for 6 consecutive months, the model is demoted to the next layer. Circuit breaker mechanism is defined as follows: When the capacity utilization rate of the Bronze layer exceeds 80%, models are sorted in ascending order of their comprehensive score, and the bottom 10% of models are eliminated.

[0018] A data requirement intelligent transformation design system based on large models and knowledge graphs, applicable to any of the methods described above, comprising: The hierarchical knowledge repository construction module is used to execute step S1; The demand sorting list generation module is used to execute step S2; The warehouse dispatch recall module is used to execute step S3; The semantic quality inspection module is used to perform the S4 step; A standardized design generation module is used to execute the S5 steps; The output of the standardized design generation module can be fed back as new knowledge to the hierarchical knowledge repository construction module, forming a knowledge optimization closed loop.

[0019] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention provides a data requirement intelligent transformation design method and system based on large models and knowledge graphs, which significantly improves the accuracy of caliber matching, significantly reduces the constraint conflict rate, and ensures high quality and consistency of output through a four-layer semantic matching quality inspection mechanism. The solution effectively solves problems such as the easy omission of key constraints, inconsistent design standards, and high error rate in manual processing.

[0020] 2. This invention provides a data requirement intelligent transformation design method and system based on large models and knowledge graphs. It achieves automatic parsing and element extraction of requirement documents, supports multimodal fusion enhancement processing, and can identify implicit business rules and cross-entity relationships. A dynamic hierarchical knowledge repository system is established, supporting intelligent classification, storage, and scheduling of knowledge, and realizing the structured accumulation and efficient reuse of historical design experience.

[0021] 3. This invention provides a method and system for intelligently transforming data requirements into design based on large models and knowledge graphs. It unifies design standards such as data model naming, field definition, and relational constraints, automatically generates design documents that conform to enterprise specifications, and reduces manual correction work. It also supports differentiated processing of requirements with different priorities, achieving a fully automated closed loop throughout the entire process.

[0022] 4. This invention provides a data requirement intelligent transformation design method and system based on large models and knowledge graphs, breaking through the bottlenecks of traditional RAG in table processing and semantic conflict resolution; solving the full-scale scanning problem in multi-factor knowledge retrieval, and realizing cross-layer shortest path planning through the WSOA algorithm; establishing refined data lineage analysis to distinguish between direct references and processing logical relationships, avoiding error propagation; and realizing a knowledge construction paradigm based on dynamic cognitive architecture, supporting refined extraction of domain knowledge and multimodal fusion enhancement. Attached Figure Description

[0023] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is a schematic diagram of the system architecture of the present invention; Figure 2 This is a schematic diagram of the construction of a hierarchical data model knowledge warehouse; Figure 3 This is a schematic diagram of a data demand sorting order; Figure 4 This is a schematic diagram illustrating knowledge retrieval based on dynamic planning and scheduling in warehouses. Figure 5 This is a quality inspection diagram of the recall results; Figure 6 Schematic diagrams are generated for standardized design. Detailed Implementation

[0025] The technical solution of the present invention will be more clearly and completely explained below with reference to the accompanying drawings and through the description of preferred embodiments of the present invention.

[0026] like Figure 1 As shown, the present invention is: S1: Construct a hierarchical knowledge repository system. Divide the data model design document into three parts: heading chain, table content, and body summary using document segmentation technology. Use large-scale model semantic understanding technology combined with a domain synonym dynamic recognition model to design tables. Perform triple enhancement processing: in-document semantic fusion, out-of-document semantic fusion, and data lineage fusion based on AST syntax tree. Dynamically allocate the data model to the three layers of knowledge repository: gold, silver, and bronze, based on the comprehensive score. S2: Generate a data requirement sorting list, extract elements from the input data requirement document, identify requirement priorities, obtain constraints, and output a structured sorting list containing a list of requirement elements, priority levels, and constraints. S3: Knowledge retrieval is performed based on the warehouse scheduling optimization algorithm. Different retrieval strategies are adopted according to the number of elements in the structured sorting order. For single-element demand, the corresponding warehouse area is directly located for retrieval. For multi-element demand, the real-time optimal decision algorithm is used to dynamically plan the retrieval path. S4: Perform four-layer semantic quality checks on the recall results, calculate the matching degree between the recall model and the demand from the entity layer, attribute layer, relationship layer and structure layer, and remove unqualified models based on the comprehensive matching degree. S5: Generate standardized designs by encapsulating the model knowledge that has passed quality inspection into constraint rules, and generating data design documents by calling the large model through the layered Prompt project.

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.

[0028] This example uses a data requirement transformation scenario of a telecommunications operator as an example. The operator has more than 1,200 historical data design documents, covering eight business areas including user domain, product domain, and accounting domain.

[0029] When constructing a hierarchical knowledge repository system, each document is first segmented into three parts: heading_chain (e.g., "Chapter 3 Data Model Design" - "3.1 User Subject Domain"), content (a table containing field codes, types, and descriptions), and summary (the main text description). To address the limitation of traditional rule matching, which can only recognize fixed table headers, this embodiment constructs a domain thesaurus. For example, "field name," "field code," and "code" are grouped together. By understanding these synonym relationships through a large model, the table recognition rate is successfully improved from 68% to 94%.

[0030] During the multimodal fusion enhancement process, when the "cert_no" field appears 4 times in the table (described as "ID number") and 2 times in the text (described as "ID card number"), the system adopts the "ID number" definition based on the high-frequency priority principle. Simultaneously, vector retrieval reveals that the document "Customer Master Data Specification v2.0," with a similarity score above 0.85, provides a more detailed definition of "user_id" as "globally unique user identifier, generated using the snowflake algorithm," which is then incorporated into the current knowledge. Regarding data lineage, the system uses AST parsing to identify that "user_info.cert_no" is directly mapped to "id_number" via a projection operator; therefore, the attribute description of "cert_no" is extended to the "id_number" field.

[0031] According to the formula for calculating the comprehensive score The system scores each data model. For example, the user_info table has a constraint strength C=0.85 (85% of fields have strong constraints), access frequency H=0.92 (920 visits in 30 days), and reusability R=0.88 (referenced by 880 downstream models), resulting in a score of 0.885, and is therefore assigned to the Gold tier. The test_log table, however, has lower metrics (C=0.45, H=0.20, R=0.15), with a score of only 0.255, and is assigned to the Bronze tier.

[0032] When the system received the request to "urgently analyze the 5G package upgrade status of users in various provinces," it extracted four key elements through text analysis: "5G package upgrade," "province-level statistics," "monthly statistics," and "exclusion of test users." Because the request contained the keyword "urgent" and required updates "before 8 AM daily," the system classified it as a high-priority request. It also identified the following constraints: time constraint ("complete before 8 AM daily"), data range constraint ("5G package users from January 2024 to present"), quality constraint ("exclusion of test users with test_flag=1"), and compliance constraint ("sensitive user information must be anonymized").

[0033] During the knowledge retrieval phase, since the demand comprises four elements, the system employs an instant optimal decision-making algorithm for path planning. Taking "5G package upgrade" as the starting point (located in the golden layer - product domain), the system calculates the comprehensive distance to other elements. The shortest distance is to "monthly statistics" (hierarchical distance is 2, correlation distance is 2.0, and comprehensive distance is...). Therefore, this element is accessed first. Through gradual optimization, the final search path was determined to be: 5G package upgrade → monthly statistics → province dimension → test users, with a total path length of 6.5, which is 29% shorter than the average length of 9.2 for random paths.

[0034] After recalling relevant models from each layer, the system performs a four-layer semantic quality check on the main model, dwd_5g_package_upgrade_d. The entity layer achieved a similarity of 0.89 using BERT vector calculations; the attribute layer scored 0.975 because the model included all required attributes and had good type matching; the relationship layer's user-package mapping matched the requirements, scoring 0.90; and the structure layer's monthly partitioning design fully met the requirements, scoring 0.925. According to the formula... It far exceeds the threshold of 0.6 and passes quality inspection.

[0035] In the standardized design generation stage, such as Figure 6 As shown, the system constructs a three-layer Prompt: the instruction layer defines the role of "senior data architect" and requires compliance with the enterprise data standard specification v3.0; the domain adaptation layer injects specific constraints of the telecommunications industry, such as the requirement to include month_id partition, the requirement to mark sensitive labels for cert_no, and the mapping of provinces to province_code(VARCHAR(4)); the constraint strategy layer guides the model to identify user_id+month_id as a composite primary key through chain thinking, uses BIGINT type for the identifier class field, and generates a check constraint of package_type IN ('4G','5G').

[0036] The final generated dws_5g_upgrade_monthly_stats table fully meets the business requirements, including core fields such as province_code and month_id, setting a stat_month partition, and recording key information such as update_frequency='daily' and data_filter='test_flag != 1' in the table attributes.

[0037] Through three months of practical application with the operator, the average design time decreased from 4.5 hours to 0.5 hours, the field naming standardization rate increased from 68% to 96%, and the knowledge reuse rate increased from 12% to 78%, fully verifying the effectiveness of the method of this invention. In particular, the warehouse scheduling optimization algorithm reduced the average retrieval path from 3.2 hops to 1.8 hops, providing strong support for rapid response to complex and multi-factor requirements.

[0038] As a specific implementation method, hierarchical data model knowledge warehouse construction, such as Figure 2 As shown; Data model corpus collection: First, data model corpus collection is conducted, covering data design documents and other materials in a specific domain to ensure that the collected content comprehensively covers the knowledge of that domain. Then, document segmentation techniques are applied to divide the original documents into three parts: heading_chain (recording chapter titles), content (recording tables within chapters), and summary (recording the main text of each chapter).

[0039] This invention differs from traditional unified storage methods by proposing a dynamic hierarchical knowledge construction approach based on logistics and warehousing concepts. It achieves joint processing of tables and text, multi-level fusion enhancement, and hierarchical storage to a knowledge base through the following four steps: The first step involves applying text preprocessing techniques: denoising, word segmentation, and part-of-speech tagging are performed on the collected corpus to clearly identify the text structure. Based on this, data model entities are identified in the text, entity attributes and attribute constraint rules are extracted, and the semantic elements of the preliminary data model design are formed.

[0040] The second step is to perform refined identification of tables within the chapters: Traditional RAG processing mostly just divides the document into pieces and then directly vectorizes and stores the entire table, or uses a fixed table header template to extract the model design table. This results in cases of storing too much or too little data, and it cannot adapt to diverse situations. For example, model design can have the following multiple situations.

[0041] Scenario 1 is shown in Table 1:

[0042] Scenario 2 is shown in Table 2:

[0043] Scenario 3 is shown in Table 3:

[0044] This invention proposes leveraging the semantic understanding capabilities of large models, combined with domain-specific synonyms as context, such as: [field name, Chinese description of the field, field description]. Hint words are written to extract tables related to model design, ultimately refining the model design tables for different scenarios.

[0045] The third step is multimodal fusion enhancement: multimodal fusion enhancement is performed on the semantic elements of the preliminary model design and the model design table formed in the above two steps.

[0046] In-document semantic fusion enhancement: First, collect the occurrence count of each entity element (entity_cnt). Iterate through the semantic elements of the model design in the first step and the model design table in the second step. Set conflict handling rules: When an entity attribute appears in both the first and second steps, select the one with the higher occurrence count. Combine these as context and write large model prompt words for semantic fusion enhancement to form locally fused model design elements.

[0047] Semantic fusion enhancement outside the document: Iteratively obtain the fused model design entities and attributes within the document, use vector matching to obtain model design elements whose similarity to the knowledge of the constructed document exceeds the threshold, and also write prompt words to perform cross-document semantic fusion enhancement to form a globally fused model design element.

[0048] Data lineage fusion enhancement: Traditional methods typically use table-level or field-level lineage to propagate upstream and downstream relationships, filling in missing Chinese field names. This approach fails to precisely express whether fields have direct reference relationships or processing logic relationships. For fields undergoing secondary processing, there is a possibility of erroneous propagation. This invention introduces field operator caliber identification and operator scope analysis to precisely identify the processing logic between fields, improving fusion accuracy. First, an SQL parsing tool is used to parse the SQL and extract the AST syntax tree structure, including operators such as projection, aggregation, and filtering. Then, scope analysis is performed to ultimately form an operator lineage relationship of [field, operator operation, field]. For example, direct reference relationship: field 1 -> projection -> field 2; processing relationship: field 3 -> expression -> field 4, field 5 -> aggregation -> field 6. Finally, only fields with direct reference relationships are extended and enhanced. As shown in Table 4: Table 4

[0049] Step 4, Layered Knowledge Model: The model is divided into Gold, Silver, and Bronze knowledge layers based on the scores calculated by entity attributes, including constraint rules and access frequency, as shown in Table 5. The core dimensions of the layering are as follows: Overall score = ω1⋅C + ω2⋅H + ω3⋅R (Dimension C: Constraint rule strength, 30D access popularity H, downstream reuse R) Table 5

[0050] Dynamic model storage location adjustment mechanism: Storage areas are reassigned monthly based on model access, referencing, and changes in constraint rules. For example: Upgrade criteria: Gold Tier: H > 0.7 and R > 0.5 for three consecutive months. Silver Tier: C increases by 50% in a single month. Downgrade rules: Bronze tier: H < 0.2 for 6 consecutive months Circuit breaker mechanism: When the bronze layer capacity is >80%, the last 10% of models are automatically eliminated. Data demand sorting list, such as Figure 3 As shown; Demand sorting list extraction: Using the same document segmentation and text analysis techniques as above, extract the list of elements, priorities, and constraints of the requirements to generate a demand sorting list.

[0051] Step 1: List of data requirements elements: Based on the submitted data requirements, text preprocessing techniques (including noise reduction, word segmentation, and part-of-speech tagging) are used to achieve a standardized representation of the text structure. Subsequently, information extraction methods are used to identify the core entities in the data model, extract their attribute features and semantic constraints to form a list of data requirement elements.

[0052] The second step is to define the priority hierarchy of requirements: The priority categories are shown in Table 6: Table 6

[0053] The methods for prioritizing requirements are shown in Table 7: Table 7

[0054] Step 3: Obtaining Requirements and Constraints Requirements constraints are the technical, business, or resource limitations that must be met to implement the requirements; they must be precisely extracted from the requirements document. The following are the specific implementation methods: The constraint classification and extraction logic are shown in Table 8. Table 8

[0055] Constraint Extraction Method Keyword matching, business rule engine, and large-scale model semantic understanding methods were used to extract constraint information from the requirements documents, as shown in Table 9. Table 9

[0056] Step 4: Output structured sorting list The final demand sorting list generated from the first three steps contains three parts of structured data: data demand elements, demand priority, and data demand constraints.

[0057] Example: { "Demand Factors": ["Real-time Inventory Monitoring", "Order Category Analysis"], Priority: High "Constraints": { Time constraint: Update every 5 minutes. Data range: "Data from the current day only", "Compliance Terms": "Sensitive Field De-identification" } } Knowledge retrieval based on warehouse dynamic programming scheduling, such as... Figure 4 As shown; Traditional knowledge retrieval only generates semantic vectors of demand and lacks a priority hierarchical mechanism, resulting in problems such as "inefficient retrieval paths" and "unreasonable resource allocation." To improve efficiency, this invention proposes a recall path planning method based on the Warehouse Scheduling Optimization Algorithm (WSOA), which achieves efficient multi-element retrieval through hierarchical scheduling and path optimization. The algorithm is divided into two main scenarios: Single-element demand scenarios When the demand contains only a single key semantic element, the system directly locates and retrieves the corresponding warehouse area.

[0058] Example: In the request "urgent query for interest rate data in the financial industry", "financial industry" is the core element. The system directly retrieves models related to interest rate data in the "Gold Zone - Financial Classification".

[0059] Multi-factor demand scenarios The instant optimal decision-making algorithm selects the shortest sorting path and prioritizes accessing adjacent storage areas (e.g., "User Segmentation" and "Conversion Rate" both belong to "Silver Zone - Operations Category," so they are merged for retrieval). The application of the instant optimal decision-making algorithm is particularly crucial for complex requirements involving multiple semantic elements. The core idea of ​​the instant optimal decision-making algorithm is to make the best choice at each step under the current state, aiming to lead to a globally optimal solution.

[0060] In this scenario, the optimal choice is to select the next warehouse area closest to the currently visited warehouse area for retrieval, in order to reduce the total length of the retrieval path and improve retrieval efficiency.

[0061] For example, for the requirement of "analyzing the impact of e-commerce user segmentation on conversion rate," the system extracts two key semantic elements: "user segmentation" and "conversion rate." Both elements belong to the "Baiyin Zone - Operations Category." The system first calculates the distance from the current location (assumed to be the starting search point) to the storage locations of models related to "user segmentation" within the "Baiyin Zone - Operations Category," and also to the storage locations of models related to "conversion rate." This distance calculation is based on the knowledge node location within the storage area or on pre-defined hierarchical distances. If the distance to the model storage location related to "user segmentation" is closer, the system first accesses that location to perform a model retrieval and obtain the relevant model set. Then, using that location as the new current location, the system recalculates the distance to the model storage location related to "conversion rate," accesses that location to perform a retrieval, and merges the two retrieval results. Through this real-time optimal decision-making strategy, the system can quickly plan a relatively optimal retrieval path under complex multi-factor requirements, reducing invalid retrieval operations.

[0062] The entire process consists of the following four steps: Step 1: Problem Definition: Let the set of semantic elements after demand parsing be E = {e1,e2,…,en}, and let A(ei) be the storage area in the knowledge base corresponding to each element ei (e.g., “Gold Zone - Financial Classification”).

[0063] The goal is to determine the element retrieval order π = [eπ1,eπ2,…,eπn] using an instantaneous optimal decision algorithm, so as to minimize the total retrieval path length L(π).

[0064] Step 2: Define the core parameters, as shown in Table 10: Table 10

[0065] Step 3: Algorithm Flow: From initial elements Starting from the current position, each time select the remaining elements with the shortest distance to the current position as the next search target, until all elements have been searched. Formal description: Initialization: Select the highest priority element

[0066] 2. Iteration:

[0067] right Calculate candidate distances:

[0068] in: This refers to the element at the end of the current path. This is a priority weight adjustment term (the higher the priority, the smaller the distance penalty).

[0069] Choose to The smallest element As the next search target:

[0070] 3. Update:

[0071] 4. Repeat until S=E Step 4: Multi-dimensional distance measurement: By combining semantic hierarchy with knowledge graphs, a distance calculation model that is more suitable for business scenarios can be constructed to improve the rationality of path planning.

[0072] 1. Hierarchical distance: distance between nodes in the same level = 1, distance between different nodes in the same level = 2, distance across levels = level difference × 2 elements (Field = Finance, Category = Risk Control) and The distance between (domain = finance, category = risk control) is 1; and The distance between (domain = finance, category = reports) is 2.

[0073] 2. Association distance:

[0074] in, Representation elements and The strength of the association.

[0075] Example: In the requirement "Analyze the impact of e-commerce user segmentation on conversion rate", "user segmentation" and "conversion rate" both belong to "Baiyin District - Operations Category". The system prioritizes searching for elements with closer proximity (e.g., "user segmentation" first), then searches for "conversion rate" in a new location, and merges the results.

[0076] Recall results and quality inspection Figure 5 As shown; In the knowledge retrieval process, the system will perform a four-layer semantic matching quality check on the retrieved models, comprehensively evaluating them from four dimensions: entity layer, attribute layer, relationship layer, and structure layer. Each layer uses semantic analysis algorithms to calculate its matching degree with the target requirement, ensuring a high degree of semantic consistency of the retrieved models.

[0077] Hierarchical matching Perform a four-layer semantic matching quality check on the recall model (entity → attribute → relation → structure) and remove models with low matching degree (such as matching degree < 60%).

[0078] Entity matching: Verify whether the core entities in the recall model are consistent with the entities defined in the requirements, including naming, type and business meaning.

[0079] Attribute matching: Check whether the attribute fields contained in the entity are complete and accurate, and evaluate the degree of matching of field names, data types and value ranges.

[0080] Relationship matching: Confirm whether the relationships between entities meet expectations, including logical consistency such as foreign key constraints and master-slave relationships.

[0081] Structure matching: Starting from the overall model architecture, evaluate the similarity and compatibility of model hierarchy, table structure, index design, etc.

[0082] Finally, the system will perform a weighted comprehensive score on the four-layer matching results. If the comprehensive matching degree is lower than the set threshold (e.g., 60%), it will be judged as an unqualified model and automatically removed; otherwise, it will enter the next stage of the standardized design and generation process.

[0083] Comprehensive matching degree calculation method The following is a comprehensive matching degree calculation model based on a four-layer semantic matching quality inspection mechanism: Matching degree calculation ; Decision criteria: if M < θ then discard the model, else proceed to the standardized design process.

[0084] Hierarchical independence The matching degree (Q / A / P / T) of each layer needs to be calculated independently to avoid error propagation. For example, the entity layer (Q) does not depend on the attribute layer (A), but the structure layer (P) must be based on the relationship layer (T). Dynamic weighting mechanism Weighting coefficients ( , , , Configure according to business scenarios: Scenarios with strong entity dependence (such as financial models): ↑, ↑ (Entity and relationship weights increase) Highly scalable scenarios (such as IoT architecture): ↑ (Increase in weight of structural layer) S5: Standardized Design Generation Knowledge Encapsulation and Constraint Injection: Based on the knowledge format retrieved in the previous step, qualified model design knowledge is "encapsulated" into constraint rules, and these constraints are injected through the dynamic large model Prompt to drive the generation of standardized design content. The specific implementation includes the following layers: Command layer: Defines model roles and core tasks.

[0085] Prompt = "You are a senior data architect who needs to generate a standardized SQL design based on the following requirements: {context}. Output requirements: {format_constraint}" in: 1. Role Definition: The job skill tags in the enterprise knowledge graph are used.

[0086] 2{context}: Original requirement description.

[0087] 3{format_constraint}: The design knowledge context to be recalled (used to constrain the output format).

[0088] Domain adaptation layer: Injects domain-specific constraints and terminology mappings.

[0089] Prompt = " 1. Area 1: Injecting ACID transaction constraints -> finance_constraint = 'Must contain a version number field (version BIGINT) and a partition field (acct VARCHAR(32))'" 2. Domain 2: Application Partition Standard Terminology Mapping -> buss_mapping = {'Device Number': 'prod_inst_id', 'Account Partition': 'month_id'} 3...." Constraint Strategy Layer: Guides the model to perform chain-of-thought (COT) reasoning by applying fine-grained constraints.

[0090] cot_prompt = " Please think about this step by step: 1. Identify primary key and foreign key relationships -> Check if {field} has a foreign key relationship in the knowledge base.

[0091] 2. Mapping data types -> Convert {raw_type} to {std_type} according to enterprise standards.

[0092] 3. Generate constraints -> Extract CHECK rules from the requirements description.

[0093] The final output should conform to: {format}.

[0094] Implementation Results Forecast: This solution is expected to significantly improve the quality and efficiency of design generation. The specific expected results are shown in Table 11. Table 11

[0095] In the current enterprise data requirements-to-design process, manually interpreting requirements documents and designing data models segment by segment is time-consuming and prone to misinterpretation, leading to design results that do not match business needs. This patent significantly improves the efficiency and standardization of data design through the deep integration of large-scale models and knowledge graphs. As a specific example: A business department of a certain operator put forward the following data requirements: Monthly data trends based on the development volume of new regional united front work Cloud PCs and dedicated network lines are used as statistical dimensions. 3. Data source: By using product coding cloud computers and dedicated network lines, we can expand the scope of new product development. Filter out users whose projects are not yet completed, and use data from the data platform as the base data;

[0096] The understanding and execution process for the above data requirements is as follows: Step 1: Extract the following requirement elements by analyzing the table and paragraphs of the requirements: (Development volume of new strategic businesses) --[defined as]-> (Statistical dimension='monthly') (Development volume of new strategic businesses) --[defined as]-> (Statistical dimension='by region') (Development volume of new business) --[defined as]-> (Completion status='Completed') (New Business Development Volume) --[defined as]-> (Product code in ('Cloud PC', 'Dedicated Network Line')) (Development volume of new strategic businesses) --[Data source]-> (dim_area) (Development of new business) --[Data Source]-> (dwa_prod_inst_m) Priority: Medium The second step involves enhancing the requirements using historical design knowledge. This includes supplementing table naming standards, partitioning specifications, and region hierarchies. Non-standardized `lan_name` fields are converted to the standard `area_name` field encoding. Source table relationships and implicit business constraints are also addressed. For example, new business requirements necessitate filtering out postpaid users. The enhanced and supplemented requirement knowledge triplet is as follows: (dim_area) --[area level]-> (area_level = 4) (dwa_prod_inst_m) --[association]-> (dim_area), associated attribute dwa_prod_inst_m.area_code = dim_area.area_code (New business development volume) --[defined as]-> (Payment type !='Postpaid') (net_name) --[standard definition]-> (grid_name) (Summary Layer) --[Naming Definition]-> (dws_${Business Type}_${Payment Period}) (Partitioning Specification) --[Partition Field]-> (Payment Period as Partition) The third step involves dynamically generating prompt words using a large model, as follows: You are a senior data architect who needs to generate standardized data model designs based on the following requirements. Monthly data trends based on the development volume of new regional united front work The statistical dimensions for the development volume of strategic emerging businesses are 'by month' and 'by region'. The regional hierarchy has four levels, including province, city, county, and grid, with the finest granularity being the grid, derived from dim_area. The growth of strategic new business comes from the association between dim_area and dwa_prod_inst_m, which is linked through area_name. The region field is standardized with the constraint area_name, varchar(60). The statistical scope for new business development volume is based on paid types! = 'Postpaid' and the product code is in ('Cloud PC', 'Dedicated Network Line'). The naming convention for summary layer tables is: dws_${business type}_${payment period} Step 4: The following is automatically generated through the design of large model data: CREATE TABLE IF NOT EXISTS dws_zxye_develop_m ( province_name VARCHAR(60) COMMENT 'Provincial Region Name', city_name VARCHAR(60) COMMENT 'City-level region name', county_name VARCHAR(60) COMMENT 'County-level region name', grid_name VARCHAR(60) COMMENT 'Grid-level region name', month STRING COMMENT 'Statistical Month (yyyyMM)', new_business_cnt INT COMMENT 'New Business Development Volume:' The development criteria are as follows: 1. Statistical criteria: Payment type = 'Postpaid' and product code in ('Cloud PC', 'Dedicated Network Line') 2. Statistical dimensions: grid (grid_name), month (payment period) 3. The region dim_area and the user instance dwa_prod_inst_m are associated through area_name. ' ) COMMENT 'Monthly Summary Table of New United Front Business Developments by Region' PARTITIONED BY (stat_month STRING COMMENT 'Statistical Month Partition (yyyyMM)'); The final step involves manual adjustments to complete the entire data requirement design.

[0097] Through comparative analysis, the combination of large models and knowledge graphs can effectively, accurately and quickly extract entities and relationships from requirements documents, and transform them into standardized data requirements designs based on historical experience. This meets the data design needs of many rapidly changing businesses and effectively improves the intelligent efficiency of data design.

[0098] The above-described specific embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Various modifications, substitutions, and improvements made by those skilled in the art to the technical solutions of the present invention based on the provided textual description and drawings, without departing from the design concept and spirit of the present invention, should all fall within the scope of protection of the present invention. The scope of protection of the present invention is determined by the claims.

Claims

1. A data requirement intelligent transformation design method based on large models and knowledge graphs, characterized in that: Includes the following steps: S1: Construct a hierarchical knowledge repository system. Divide the data model design document into three parts: heading chain, table content, and body summary using document segmentation technology. Use large-scale model semantic understanding technology combined with a domain synonym dynamic recognition model to design tables. Perform triple enhancement processing: in-document semantic fusion, out-of-document semantic fusion, and data lineage fusion based on AST syntax tree. Dynamically allocate the data model to the three layers of knowledge repository: gold, silver, and bronze, based on the comprehensive score. S2: Generate a data requirement sorting list, extract elements from the input data requirement document, identify requirement priorities, obtain constraints, and output a structured sorting list containing a list of requirement elements, priority levels, and constraints. S3: Knowledge retrieval is performed based on the warehouse scheduling optimization algorithm. Different retrieval strategies are adopted according to the number of elements in the structured sorting order. For single-element demand, the corresponding warehouse area is directly located for retrieval. For multi-element demand, the real-time optimal decision algorithm is used to dynamically plan the retrieval path. S4: Perform four-layer semantic quality checks on the recall results, calculate the matching degree between the recall model and the demand from the entity layer, attribute layer, relationship layer and structure layer, and remove unqualified models based on the comprehensive matching degree. S5: Generate standardized designs by encapsulating the model knowledge that has passed quality inspection into constraint rules, and generating data design documents by calling the large model through the layered Prompt project.

2. The intelligent data requirement transformation design method based on large models and knowledge graphs according to claim 1, characterized in that, The formula for calculating the comprehensive score in S1 is as follows: ； in, This is the overall score; The constraint strength is defined as [0,1], and its value range is calculated based on the ratio of the number of strong constraint fields to the total number of fields. The access popularity is calculated based on the number of visits within 30 days, with a value range of [0,1]. The reuse factor is [0,1], calculated based on the number of downstream references. , , Let be the weighting coefficient, satisfying + + = 1; when A score ≥ 0.8 is assigned to the Gold layer; a score ≤ 0.6 is assigned to the Silver layer; and a score < 0.6 is assigned to the Bronze layer.

3. The intelligent data requirement transformation design method based on large models and knowledge graphs according to claim 1, characterized in that, The intra-document semantic fusion in S1 specifically includes: Count the number of occurrences of each entity element. cnt When the same entity attribute conflicts in both the table and the text, select "entity". cnt Larger attributes are used as the final attributes; selected entity attributes are used as context to generate local fusion model design elements through large model prompts.

4. The intelligent data requirement transformation design method based on large models and knowledge graphs according to claim 1, characterized in that, The data lineage fusion based on the AST syntax tree in S1 includes: The abstract syntax tree is extracted using SQL parsing tools to identify projection, aggregation, and filtering operators; the scope of the operators is analyzed, and lineage triples are constructed. ,in For the source field, For operator type, For the target field; if and only if When using a projection operator, perform field attribute expansion.

5. The intelligent data requirement transformation design method based on large models and knowledge graphs according to claim 1, characterized in that, The instantaneous optimal decision-making algorithm in S3 includes: Define the set of semantic elements of the requirements Where n is the total number of elements; construct the warehouse area distance matrix. ,in: ； For the hierarchical distance, when and At the same node = 1, when at different nodes on the same level = 2, when crossing layers ,in , These are the levels where the elements are located; For the correlation distance, ,in The strength of the association between elements in a knowledge graph; , For distance weighting coefficients, satisfying + = 1.

6. The intelligent data requirement transformation design method based on large models and knowledge graphs according to claim 5, characterized in that, The path planning process of the instantaneous optimal decision-making algorithm is as follows: Initialize the set of visited features ,in The highest priority feature; for each feature in the remaining feature set E\S Calculate candidate distances: ； in The element at the end of the current path. as elements Priority weight, higher priority = 2, medium priority = 1.5, low priority = 1; Select to make The smallest element is selected as the next search target; this process is repeated until S = E.

7. The intelligent data demand transformation design method based on large models and knowledge graphs according to claim 1, characterized in that, The formula for calculating the overall matching degree in S4 is as follows: ； in, For overall matching degree; The entity layer matching degree is calculated using the cosine similarity of the BERT embedding vectors, with a value range of [0,1]. The attribute matching degree is calculated based on field completeness and type consistency, and its value range is [0,1]. The relational matching degree is calculated based on the degree to which foreign key constraints are satisfied, and its value ranges from [0,1]. The structural layer matching degree is calculated based on the similarity of the model architecture, and its value ranges from [0,1]. = 0.3、 = 0.2、 = 0.3、 = 0.2 is the hierarchical weight coefficient; when M < 0.6, it is judged as an unqualified model and is removed.

8. The intelligent data requirement transformation design method based on large models and knowledge graphs according to claim 1, characterized in that, The layered Prompt project in S5 includes: In the instruction layer, the role is defined as "Senior Data Architect," and the original requirement description (context) and format constraints are injected. constraint Domain adaptation layer, injecting domain constraints. constraint and term mapping table term mapping The constraint strategy layer guides the large model to perform inference in the order of "identifying primary and foreign keys → mapping data types → generating constraints" through chain-thinking prompts.

9. The intelligent data requirement transformation design method based on large models and knowledge graphs according to claim 2, characterized in that, It also includes a dynamic cargo location adjustment mechanism: Upgrade conditions are defined as follows: When H > 0.7 and R > 0.5 are met for 3 consecutive months, the model is promoted from the current layer to the next layer; when the C value in a single month increases by more than 50% compared to the previous month, the model is promoted to the Silver layer. Demotion conditions are defined as follows: When H < 0.2 is met for 6 consecutive months, the model is demoted to the next layer. Circuit breaker mechanism is defined as follows: When the capacity utilization rate of the Bronze layer exceeds 80%, models are sorted in ascending order of their comprehensive score, and the bottom 10% of models are eliminated.

10. A data requirement intelligent transformation design system based on large models and knowledge graphs, applicable to the data requirement intelligent transformation design method based on large models and knowledge graphs as described in any one of claims 1-9, characterized in that, include: The hierarchical knowledge repository construction module is used to execute step S1; The demand sorting list generation module is used to execute step S2; The warehouse dispatch recall module is used to execute step S3; The semantic quality inspection module is used to perform the S4 step; A standardized design generation module is used to execute the S5 steps; The output of the standardized design generation module can be fed back as new knowledge to the hierarchical knowledge repository construction module, forming a knowledge optimization closed loop.

Images