Method and system for dynamic construction of test data model based on multi-source heterogeneous data sources

Abstract

The application discloses a test data model dynamic construction method and system based on multi-source heterogeneous data sources, relates to the technical field of software testing and data management, and specifically comprises the following steps: constructing a unified cross-source metadata graph to understand the semantics of different data sources; receiving and analyzing the test intention of a user to generate a structured test intention descriptor; matching the test intention and the metadata graph through a dynamic semantic correlation engine, automatically discovering and correlating the semantic correlation data entities across the data sources; using a test intention driven model synthesizer to select and combine operators from a pre-defined model operator library to dynamically generate data processing logic meeting the test scene requirements; and finally executing the logic to construct a target test data model. The application realizes intelligent construction of the test data model, and significantly improves the efficiency and quality of test data preparation.

Description

Technical Field

[0001] This invention relates to the field of software testing and data management technology, specifically to a method and system for dynamically constructing test data models based on multi-source heterogeneous data sources. Background Technology

[0002] The statements in this section are provided only as background information in connection with this disclosure and may not constitute prior art.

[0003] As software systems become increasingly complex, the management and preparation of test data has become a critical aspect of ensuring software quality. Efficient testing relies on high-quality, high-coverage test data. However, in today's enterprise environments, test data is typically scattered across multiple heterogeneous data sources, such as traditional relational databases (Oracle, MySQL), in-memory databases (Redis), and distributed big data platforms (HBase, Hive, MongoDB, etc.). These data sources have different structures and protocols, creating "data silos." Existing test data management tools have significant shortcomings in data model building: 1. Static vs. Rigid: Test data models (or tables) are usually predefined statically by humans. When business logic changes or new scenarios need to be tested, data requirements must be re-analyzed manually, and complex SQL queries or ETL scripts must be written. This process is tedious, time-consuming, and prone to errors, making it impossible to quickly respond to changes in test requirements.

[0004] 2. Lack of intelligent association: Most existing tools can only operate on a single table explicitly specified by the user or multiple tables with known relationships, lacking the ability to perform intelligent semantic associations across data sources. When it is necessary to combine data from multiple heterogeneous data sources to build a complete test scenario data view, the operation is extremely difficult and highly dependent on the domain knowledge of data experts.

[0005] 3. Disconnect from Test Intent: The process of preparing test data is separate from the specific test case intent (such as boundary testing, abnormal flow testing, and performance testing). When building the data model, testers need to manually translate the test intent into complex data filtering, combination, and transformation logic, which is inefficient, has a high barrier to entry, and makes it difficult to guarantee that the data model accurately meets the test requirements.

[0006] Therefore, there is an urgent need in this field for a method and system that can intelligently understand testing requirements and automatically and dynamically construct an accurate and suitable test data model from multi-source heterogeneous data in order to overcome the shortcomings of existing technologies. Summary of the Invention

[0007] The purpose of this invention is to address the problems existing in current test data management tools, such as static and rigid model definitions, lack of intelligent semantic association between heterogeneous data sources, and a serious disconnect between the data preparation process and specific test intentions. This invention provides a method and system for dynamically constructing test data models based on multi-source heterogeneous data sources. It adopts a cross-source metadata graph to deeply integrate the structural and semantic information of heterogeneous data, and a dynamic semantic association matching and automatic model operator assembly mechanism driven by test intentions. This enables intelligent on-demand construction of test data models and automatic discovery of cross-source relationships, thereby overcoming the shortcomings of traditional manual script writing, which is inefficient and prone to errors. It significantly improves the flexibility, accuracy, and automation level of test data preparation when facing complex business scenarios.

[0008] The technical solution of the present invention is as follows: A method for dynamically constructing test data models based on multiple heterogeneous data sources includes: Step S1: Construct a cross-source metadata graph, integrate metadata from multiple heterogeneous data sources, and identify the semantic relationships between them; Step S2: Parse the user's input test intent and generate a test intent descriptor containing the test objective, test entity, and test conditions; Step S3: Using a dynamic semantic association engine, match the test intent descriptor with the cross-source metadata graph to automatically discover data entities and cross-source associations related to the test intent; Step S4: Based on the matching results, the model synthesizer driven by the test intent selects and combines one or more model operators from the model operator library to dynamically generate the construction logic of the target test data model; Step S5: Execute the construction logic to generate a target test data model that can be used by the test system.

[0009] Furthermore, this application proposes that the steps for constructing a cross-source metadata graph specifically include: connecting and scanning multiple configured heterogeneous data sources through a multi-source data adapter, automatically extracting the table structure, field names, data types, and primary and foreign key constraint information of the data sources as basic metadata; using natural language processing technology to perform word segmentation, word vectorization, and semantic similarity calculation on the table names and field names in the basic metadata to generate semantic metadata; and integrating the basic metadata and semantic metadata, using data entities as nodes and structural and semantic associations between entities as edges to construct a graph-structured cross-source metadata graph.

[0010] Furthermore, this application also proposes to provide a graphical user interface, which provides a test entity selection box, a test condition input box, and a test type drop-down menu; to receive selection and configuration information input by the user through the graphical user interface, and to map it into a test intent descriptor in the form of key-value pairs containing "test target", "core test entity", "test constraint condition" and "test scenario type".

[0011] Furthermore, this application proposes that the dynamic semantic association engine performs matching and discovery steps including: using the "core test entity" and "test constraint" in the test intent descriptor as initial query nodes, performing a breadth-first traversal in the cross-source metadata graph; during the traversal, calculating the confidence of the association path based on the edge weights (including structural association weights and semantic similarity weights); filtering out all association paths with confidence higher than a preset threshold, and the data entities and their relationships covered by the association path are the automatically discovered cross-source association relationships.

[0012] Furthermore, this application proposes that the model operator library includes at least data connection operators, data filtering operators, data aggregation operators, data derivation operators, and data anonymization / obfuscation operators; the model synthesizer determines the specific connection method of the data connection operators based on the discovered cross-source relationships; it assembles one or more data filtering operators according to the "test constraints" in the test intent descriptor; and it selects and assembles the corresponding data aggregation operators or data derivation operators according to the "test scenario type" to construct data deformation logic that conforms to the test scenario.

[0013] Furthermore, this application also proposes that the steps for dynamically generating the construction logic of the target test data model further include: after the model synthesizer generates the initial construction logic, a logic optimizer is called to optimize the initial logic. The optimization strategies include, but are not limited to, predicate pushdown, connection order rearrangement, and intermediate result caching. The logic optimizer selects the optimal optimization strategy based on the computing power, data volume, and network overhead of each data source to generate the final executable construction logic.

[0014] Furthermore, this application proposes that the steps for executing the build logic and generating the target test data model are as follows: the build logic is compiled into a task that can be executed on the target data computing engine, including Spark, Flink, or a native database engine; the execution engine drives the computing engine to run the task, materializes the result into a specific physical data table, or registers it as a virtual data view, and provides a unified access interface to the upstream testing system.

[0015] Furthermore, this application proposes a dynamic test data model construction system based on multi-source heterogeneous data sources, comprising: a metadata management module for constructing and maintaining a cross-source metadata graph; a test intent parsing module for receiving and parsing user input and generating test intent descriptors; a dynamic semantic association engine for matching and association discovery between test intents and the metadata graph; a model synthesizer for dynamically assembling model operators and generating model construction logic based on association discovery results; a model operator library for storing predefined data model construction operators of various types; an execution engine for executing the model construction logic and generating the target test data model; and a multi-source data adaptation layer for connecting to and accessing different types of data sources.

[0016] Furthermore, this application proposes that the test intent parsing module includes: a graphical interface unit for receiving user interaction operations; a natural language processing unit for parsing the natural language text input by the user; and a descriptor generation unit for standardizing the parsing results into test intent descriptors.

[0017] Furthermore, this application also proposes that the system further includes a logic optimizer connected between the model synthesizer and the execution engine, for optimizing the execution performance of the initial construction logic generated by the model synthesizer.

[0018] Compared with existing technologies, the advantages of this invention are: This invention achieves semantic association of multi-source data by constructing a cross-source metadata graph. By combining dynamic semantic matching and automatic combination technology of model operators, it solves the problems of static rigidity, lack of cross-source association capability, and low efficiency of test intent conversion in the existing test data model construction process. It has the advantages of dynamically adapting to changes in requirements, automatically discovering data association relationships, and improving the efficiency and accuracy of test data generation. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0020] Figure 1 This is a flowchart of the steps of the method for dynamically constructing a test data model based on multiple heterogeneous data sources according to the present invention; Figure 2 This is another step flowchart of the method for dynamically constructing test data models based on multi-source heterogeneous data sources in this invention; Figure 3 This is a system framework diagram of the method for dynamically constructing test data models based on multi-source heterogeneous data sources, as proposed in this invention. Detailed Implementation

[0021] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0022] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0023] Example 1 In existing technologies, software test data is typically scattered across multiple heterogeneous data sources, such as relational databases, in-memory databases, and distributed big data platforms. Traditional test data management tools suffer from static and rigid model building, lack of cross-source correlation capabilities, and difficulty in translating test intents. When test scenarios require building data views from multiple data sources, complex scripts often need to be written manually, and it is difficult to accurately match the specific requirements of test cases. For example, in the testing of financial trading systems, testers need to extract related data from Oracle transaction record tables, Redis cache logs, and Hive historical data, but existing tools cannot automatically identify the semantic relationships between different data sources, resulting in inefficient and error-prone data preparation.

[0024] To address the aforementioned issues, the applicant identified a key deficiency in existing technologies: a lack of dynamic semantic association capabilities and automated model building mechanisms. By analyzing the challenges of integrating multi-source data, a technical approach combining metadata graphs with dynamic semantic matching is proposed. First, a cross-source metadata graph is established to uniformly manage the structural and semantic information of heterogeneous data sources. Second, a test intent parsing mechanism is designed to capture user needs. Finally, automated model building is achieved through dynamic operator combinations. This approach breaks through the traditional fixed model relying on manually written scripts, forming a complete technology chain from understanding requirements to automatic execution.

[0025] Therefore, please refer to Figures 1-3 This application proposes a method for dynamically constructing a test data model based on multi-source heterogeneous data sources, including the following steps: Construct a cross-source metadata graph to integrate multi-source metadata and identify semantic relationships; parse user test intent to generate structured descriptors; associate intent with metadata through a dynamic semantic matching engine; dynamically synthesize model construction logic based on matching results; execute logic to generate target test data model.

[0026] The cross-source metadata graph refers to a graph-structured data model formed by integrating the structure, field constraints, and semantic information of multiple data tables. Specifically, it can use natural language processing techniques to vectorize table name fields and construct nodes and edges based on primary and foreign key relationships. The test intent descriptor transforms user-inputted test targets, entities, and conditions into structured data. Specifically, it can receive configuration information through a graphical interface and map it into key-value pairs. The dynamic semantic association engine is a component that matches intent with metadata based on graph traversal algorithms. Specifically, it can use a breadth-first search algorithm to calculate the confidence of association paths. The model synthesizer is a module that dynamically combines operators based on association results. Specifically, it can select operators such as join, filter, and aggregation from a predefined operator library to construct execution logic.

[0027] Specifically, this method first extracts metadata from heterogeneous data sources through a multi-source adapter, constructing a graph containing structural and semantic associations. When a user inputs test requirements through an interactive interface, the system transforms them into standardized test intent descriptors. The dynamic semantic engine traverses and matches relevant data entities in the metadata graph, filtering out association paths with sufficient confidence. The model synthesizer automatically assembles data connection, filtering, and transformation operators based on the matching results, generating executable construction logic. Finally, the computation engine transforms the logic into physical data tables or virtual views, providing a ready-to-use data model for the test system.

[0028] Compared to existing technologies, traditional methods require manual pre-definition of data models and cannot perform cross-source joins. In contrast, this method automatically discovers cross-source relationships through dynamic semantic matching, achieving automated transformation from test intent to data model. For example, in cross-database join query scenarios, existing technologies require manual writing of multi-table join statements, while this method can automatically identify semantic relationships and generate optimal join logic, significantly reducing operational complexity.

[0029] Through the above technical solution, this application achieves dynamic construction of test data models and intelligent cross-source data association, solving the problems of slow response speed and high operational threshold of traditional methods. Testers can quickly obtain cross-source associated data without manually writing complex queries, while ensuring that the generated data model accurately matches the test scenario requirements, effectively improving the efficiency and accuracy of test data preparation.

[0030] This application further proposes the following steps for constructing a cross-source metadata graph: connecting and scanning multiple configured heterogeneous data sources through a multi-source data adapter, automatically extracting the table structure, field names, data types, and primary and foreign key constraint information of the data sources as basic metadata; using natural language processing technology to perform word segmentation, word vectorization, and semantic similarity calculation on the table names and field names in the basic metadata to generate semantic metadata; and integrating the basic metadata and semantic metadata, using data entities as nodes and structural and semantic relationships between entities as edges to construct a graph-structured cross-source metadata graph.

[0031] Among them, multi-source data adapters refer to components that support connections to multiple data source protocols, specifically implemented using JDBC, ODBC, or NoSQL driver interfaces, and their function is to eliminate access differences between heterogeneous data sources. Natural language processing technology refers to methods for semantic parsing of text data, specifically implemented using BERT pre-trained models or the Word2Vec algorithm, and its function is to identify the business meanings implied in table and field names. Basic metadata refers to a set of attributes describing the physical structure of the data source, specifically including field types, primary key and foreign key definitions, and its function is to establish explicit relationships between data entities. Semantic metadata refers to additional information generated through semantic analysis, specifically including word vector representations and similarity matrices, and its function is to reveal potential semantic relationships across data sources. Graph structure refers to a network data model composed of nodes and edges, specifically implemented using Neo4j graph databases or in-memory graph computing frameworks, and its function is to intuitively express the multidimensional relationships between data entities.

[0032] Specifically, the multi-source data adapter first scans the configured heterogeneous data sources such as Oracle and Hive, extracting field names and primary / foreign key constraints from the table structures. For example, it retrieves the field definitions of the user table from MySQL and the primary key information of the order table from HBase. Subsequently, the natural language processing unit segments field names such as "customer_name" and "order_id" into word vectors and calculates the semantic similarity between "customer name" and "user_name". Foreign key constraints in the basic metadata are mapped to inter-entity connection edges in the graph structure, while fields with semantic similarity higher than a threshold (e.g., 0.85) generate semantic association edges across data sources. In the final cross-source metadata graph, user table nodes are connected to order table nodes through foreign key edges, and simultaneously associated with the same-named fields in the customer information table through semantic edges.

[0033] Compared to existing technologies, traditional methods rely on manually compiling data dictionaries to maintain metadata relationships, requiring data experts to analyze table structures and record relationships row by row. This solution, however, automatically extracts basic metadata and combines it with semantic analysis technology to automatically discover explicit constraints and potential semantic relationships across data sources. For example, in a MongoDB collection without declared foreign key constraints, relationships with relational database tables can still be established through the semantic similarity of field names, solving the challenge of integrating metadata from unstructured data sources.

[0034] Through the above technical solution, this application achieves automated metadata integration of multi-source heterogeneous data without the need for manually pre-defined data association rules. When testing requirements involve data entities across data sources, the system can quickly locate relevant data sources based on structural and semantic relationships in the graph, avoiding the problem of missing relationships caused by data silos in traditional methods. For example, in a performance testing scenario where it is necessary to simultaneously obtain basic user information and historical order records, the system can automatically identify relevant tables belonging to different databases and establish connection paths, significantly reducing the time cost of test data preparation.

[0035] This application further proposes to provide a graphical user interface, which provides a test entity selection box, a test condition input box, and a test type drop-down menu; it receives selection and configuration information input by the user through the graphical user interface and maps it into a test intent descriptor in the form of key-value pairs containing test objectives, core test entities, test constraints, and test scenario types.

[0036] The graphical user interface (GUI) refers to a human-computer interaction interface that receives user operations through visual components. It can be implemented using web forms, desktop application controls, or visual drag-and-drop panels to reduce user complexity. The test entity selection box is an interactive component that allows users to select target entities from a pre-defined list of data entities. It can be implemented using checkboxes or tree selectors and is used to clearly define the core data objects for testing purposes. The test condition input box is a text box that receives user input filtering condition expressions. It can be implemented using a syntax editor that supports logical operators and is used to define the constraints that test data must meet. The test type dropdown menu is a selection control that provides options for preset test scenario types. It can be implemented using an enumerated dropdown list and is used to identify the business characteristics of the test scenario. The key-value pair test intent descriptor converts user input information into structured data objects. It can be implemented using JSON or XML format and is used to standardize the expression of test requirements.

[0037] Specifically, users select target data tables or fields through the test entity selection box in the graphical user interface, fill in data filtering conditions in the test condition input box, and select scenario types such as performance testing or abnormal flow testing through the test type drop-down menu. The interface automatically converts the user's input selections and configuration information into a key-value pair structure containing test objectives, core test entities, test constraints, and test scenario types. For example, when a user selects the order table as the core test entity, enters the condition that the order amount is greater than 10,000, and selects the boundary test scenario type, the system generates a descriptor containing {"Test Objective":"Verify Large Order Processing Logic","Core Test Entity":"Order Table","Test Constraints":"Amount_10000","Test Scenario Type":"Boundary Test"}. This descriptor provides standardized input for subsequent semantic matching and model building.

[0038] Compared to existing technologies, current test data preparation tools require users to manually write SQL queries or script code to express test requirements. This solution, however, automatically converts user actions into structured intent descriptors through a graphical interface, avoiding the tedious process of manually writing complex query statements. For example, traditional tools require users to input SQL statements containing multi-table join conditions via command line, while this solution only requires users to select interface elements to express their intent, significantly lowering the operational threshold.

[0039] Through the above technical solution, this application solves the problem of the disconnect between the test data preparation process and the test intent, enabling testers to quickly and accurately define test requirements without needing complex data query language skills. The automatic generation of standardized descriptors ensures that downstream semantic matching and model building processes can accurately identify test targets, reducing data model deviations caused by human error.

[0040] This application further proposes a dynamic semantic association engine for matching and discovery steps, including: using the core test entities and test constraints in the test intent descriptor as initial query nodes, performing a breadth-first traversal in the cross-source metadata graph; during the traversal, calculating the confidence of the association path based on the edge weights; filtering out all association paths with confidence higher than a preset threshold, and the data entities and their relationships covered by the association path are the automatically discovered cross-source association relationships.

[0041] The core test entities refer to the key data objects that need to be manipulated in the user-specified test scenario. These can be obtained through the entity selection box in the graphical interface and are used to locate the starting node in the metadata graph. Test constraints refer to the filtering rules or data range restrictions entered by the user. These can be converted into structured query conditions through the condition input box and used to filter invalid paths during traversal. Edge weights include structural association weights and semantic similarity weights. Structural association weights can be calculated using the strength of primary and foreign key constraints, while semantic similarity weights can be calculated using word vector cosine similarity. These are used to comprehensively evaluate the reliability of associations between entities. Association path confidence refers to the weighted average of the weights of each edge in the path. A dynamic threshold adjustment algorithm can be used to quantify the credibility of cross-source associations.

[0042] Specifically, after a user selects the core test entity and inputs test constraints through the interface, the dynamic semantic association engine maps it to an initial node in the metadata graph. Using a breadth-first search algorithm, the search scope expands outward from the initial node. During the traversal, the confidence score of each path is calculated by combining structural association weights and semantic similarity weights. For example, when the core test entity is the order table, the traversal might discover structural paths related to the user table via foreign keys, or semantic paths related to the logistics table via semantically similar fields. Finally, paths with confidence scores higher than a preset threshold are selected; for example, only paths with confidence scores greater than 0.8 are retained. This determines the entities and relationships related to the test intent across data sources.

[0043] Compared to existing technologies, traditional methods rely on manually predefined data association rules, failing to dynamically discover implicit relationships across heterogeneous data sources. This solution, however, combines structural constraints with weighted calculations of semantic similarity to automatically uncover potential association paths in multi-source metadata graphs. For example, it can discover relationships between fields with different names but similar semantics across different data sources, thereby reducing reliance on domain expert experience.

[0044] By employing the aforementioned technical solution, this application addresses the problem of low efficiency in discovering cross-source relationships during the construction of test data models. Through automated path traversal and confidence filtering, it can quickly identify data entities that meet the testing intent and their relationships. For example, in performance testing scenarios, it can automatically associate transaction tables and time-series fields in log tables of distributed databases, avoiding the need to manually write complex cross-database connection logic and significantly improving the efficiency and accuracy of test data model construction.

[0045] This application further proposes a method for dynamically constructing test data models based on multi-source heterogeneous data sources. The model operator library includes at least data connection operators, data filtering operators, data aggregation operators, data derivation operators, and data anonymization or obfuscation operators. The model synthesizer determines the specific connection method of the data connection operators based on the discovered cross-source relationships. Based on the test constraints in the test intent descriptor, one or more data filtering operators are assembled. Based on the test scenario type, the corresponding data aggregation operators or data derivation operators are selected and assembled to construct data deformation logic that conforms to the test scenario.

[0046] The data join operator is an operational unit that establishes data associations between different data sources. It can be implemented using SQL JOIN statements based on primary and foreign keys or API calls to establish physical or logical connections between cross-source data entities. The data filtering operator is an operational unit that filters target data based on conditions. It can be implemented by generating WHERE clauses or filtering functions using a conditional expression parser to extract subsets of data from the data source that meet test constraints. The data aggregation operator is an operational unit that performs statistical calculations on multi-dimensional data. It can be implemented using aggregation functions such as SUM and AVG or custom calculation logic to generate summary data that conforms to performance testing scenarios. The data derivation operator is an operational unit that generates new data fields based on existing fields. It can be implemented using a mathematical formula parser or rule engine to construct abnormal data or data in specific formats required for boundary testing. The data anonymization or obfuscation operator is an operational unit that protects sensitive information. It can be implemented using data replacement, encryption algorithms, or format obfuscation techniques to meet the test data generation requirements under privacy protection requirements.

[0047] Specifically, when a relationship is identified between two data entities in the cross-source metadata graph, the model synthesizer automatically selects the corresponding data join operator to establish a cross-source join. For example, when a foreign key relationship is detected between the order table and the user table, a JOIN join logic based on the user ID is generated. Time ranges or numerical thresholds in test constraints are converted into combinations of filter operators; for example, "transaction amount greater than 10000" is converted into a WHERE amount_10000 filter condition. For performance testing scenarios, the model synthesizer automatically assembles aggregation operators containing GROUP BY clauses to generate a transaction volume statistics view; while for abnormal flow testing scenarios, it selects derived operators to generate test data containing negative values ​​or excessively long strings.

[0048] Compared to existing technologies, traditional test data construction requires manually writing complete SQL queries or ETL scripts, and operators need to have a prior understanding of the structural relationships and business rules of all data sources. This solution, however, uses a predefined model operator library and an automated assembly mechanism to automatically generate a construction scheme containing complex logic such as joins, filters, and transformations based on dynamically identified data relationships and test scenario types. This avoids the tedious process of manual coding while ensuring a precise match between the data model and the test intent.

[0049] Through the above technical solution, this application solves the problems of complex manual operations and poor scenario adaptability in the process of test data model construction, and realizes the ability to automatically combine data operation operators according to the test scenario type. Testers do not need to manually write cross-source connection logic or data transformation code. The system can automatically convert test constraints into combinations of filtering operators and select the corresponding aggregation or derived operators according to different scenario requirements such as performance testing and boundary testing, which significantly improves the efficiency of test data model construction and scenario coverage.

[0050] This application further proposes that the steps for dynamically generating the construction logic of the target test data model include: after the model synthesizer generates the initial construction logic, it calls the logic optimizer to optimize the initial logic. The optimization strategies include, but are not limited to, predicate pushdown, connection order rearrangement, and intermediate result caching. Based on the computing power, data volume, and network overhead of each data source, the logic optimizer selects the optimal optimization strategy to generate the final executable construction logic.

[0051] The logic optimizer is a component that optimizes the performance of the initial logic. This can be implemented using cost-based optimization algorithms, dynamically adjusting the execution order and allocation of computation nodes by analyzing differences in the computing power of the data source, data distribution characteristics, and network transmission costs. Predicate pushdown involves pushing filtering conditions down to the execution stage as close to the data source as possible. This can be achieved by parsing the filtering conditions in the initial logic and embedding them into data join operators, thereby reducing the amount of data transmitted across the network. Join order reordering adjusts the order of multi-table joins based on table size and indexing of related fields. This can be achieved using dynamic programming algorithms to calculate the intermediate result size of different join paths and select the path that produces the minimum amount of intermediate data. Intermediate result caching temporarily stores frequently used intermediate datasets in memory or on disk. This can be achieved by marking common sub-expressions during logic execution and reusing their calculation results, avoiding resource waste caused by repeated calculations.

[0052] Specifically, after the initial build logic is generated, the logic optimizer first performs syntax parsing on the initial logic to identify optimizable operation nodes. For example, when multiple table join operations are detected, the optimizer, based on the table size statistics of each data source, prioritizes joining smaller tables with medium-sized tables, and then joins the results with larger tables, thereby reducing memory usage during the join process. Simultaneously, if the same filter condition is found to be repeatedly applied in multiple subqueries, the optimizer advances the execution of this condition to the data reading stage, reducing the amount of data processed subsequently through predicate pushdown. For complex calculations involving cross-data source data, the optimizer, based on the characteristics of the target data calculation engine, pushes down some calculations to databases with local computing capabilities, transmitting only the final results to reduce network overhead. The final executable build logic will dynamically select whether to cache intermediate results in memory or persistent storage based on the real-time load of the data source.

[0053] Compared to existing technologies, current test data construction tools typically employ fixed execution plans, failing to adjust the logical execution order based on the dynamic characteristics of the data source. This results in the transmission of a large amount of redundant data and a waste of computational resources. In contrast, this solution, through the dynamic decision-making mechanism of the logic optimizer, can automatically select the optimal execution strategy based on the storage scale, computing power differences, and network environment of different data sources, significantly reducing latency and resource consumption in cross-source data processing.

[0054] Through the above technical solution, this application solves the problem of low efficiency caused by fixed execution logic in the construction of existing test data models, realizes dynamic optimization of execution path according to the characteristics of heterogeneous data sources, effectively reduces network transmission volume and computing resource occupation when operating across data sources, and enables the generated test data model to quickly adapt to the needs of test scenarios of different scales and complexities.

[0055] This application further proposes the following steps for executing the build logic and generating the target test data model: the build logic is compiled into a task that can be executed on the target data computing engine, including Spark, Flink, or a native database engine; the execution engine drives the computing engine to run the task, materializes the result into a specific physical data table, or registers it as a virtual data view, and provides a unified access interface to the upstream testing system.

[0056] The target data computation engine refers to the execution environment that supports distributed computing or local processing. Specifically, it can use Spark for large-scale parallel data processing, Flink for integrated stream and batch processing, or directly call the database's built-in engine to execute native SQL logic, thus adapting to the execution characteristics of different data sources. The physical data table refers to the physical dataset that persistently stores the computation results. This can be achieved by writing to the HDFS file system or database storage media, supporting repeated access to stable datasets by the test system. The virtual data view refers to a dynamically generated logical collection of data on demand. This can be achieved by registering metadata information and lazy loading data content, reducing storage overhead and supporting real-time data updates. The unified access interface refers to a standardized service layer that encapsulates the data access protocol. This can be implemented using JDBC drivers or REST APIs, eliminating the perception differences of upstream test systems regarding different data storage formats.

[0057] Specifically, after the model building logic is generated, the system selects the appropriate computing engine based on the deployment environment of the target data source. For example, for data stored in Hive, it prioritizes compilation into Spark tasks to leverage distributed computing capabilities; for transactional data in MySQL, it directly generates stored procedures to call the native engine. During execution, the system selects the materialization method based on the test scenario requirements: if the test case needs to reuse a fixed dataset, it triggers a full write operation to the physical table; if the test case needs to obtain the latest data status in real time, it dynamically generates a virtual view that only retains the data association rules. Finally, a unified interface exposes the data access entry point to the upstream system, so that the test scripts do not need to distinguish between physical tables and virtual views at the underlying level.

[0058] Compared to existing technologies, traditional test data tools only support generating physical data tables in a fixed format, failing to dynamically select materialization strategies based on the scenario, resulting in wasted storage resources and delayed data updates. This solution, however, supports flexible output of both physical tables and virtual views, satisfying both high-stability testing scenarios and real-time testing needs. Furthermore, by shielding underlying differences through a unified interface, it significantly reduces the adaptation costs of the test system.

[0059] Through the above technical solutions, this application solves the problem of low resource utilization caused by the single execution engine of the test data model. It realizes the automatic selection of the optimal computing engine according to the characteristics of the data source, reduces storage overhead through the collaborative use of physical tables and virtual views, and simplifies the data call complexity of the test system with the help of a unified interface, thereby improving the efficiency of test data preparation and enhancing the scalability of the system.

[0060] This application further proposes a dynamic test data model construction system based on multi-source heterogeneous data sources, including a metadata management module, a test intent parsing module, a dynamic semantic association engine, a model synthesizer, a model operator library, an execution engine, and a multi-source data adaptation layer. The metadata management module is used to build and maintain a cross-source metadata graph; the test intent parsing module is used to receive and parse user input and generate test intent descriptors; the dynamic semantic association engine is used to match and discover associations between test intents and the metadata graph; the model synthesizer is used to dynamically assemble model operators and generate model construction logic based on the association discovery results; the model operator library is used to store predefined data model construction operators of various types; the execution engine is used to execute the model construction logic and generate the target test data model; and the multi-source data adaptation layer is used to connect to and access different types of data sources.

[0061] The metadata management module is a component that integrates metadata from multiple heterogeneous data sources to build a unified graph. This can be achieved using graph database storage technology combined with natural language processing algorithms. Its function is to eliminate data silos and establish cross-source semantic relationships. The test intent parsing module transforms user-inputted test requirements into a structured description. This can be achieved through a graphical interface working in conjunction with a natural language processing unit. Its function is to accurately capture the core elements of the test scenario. The dynamic semantic association engine performs cross-source data association analysis based on semantic similarity. This can be achieved through graph traversal algorithms and weight calculation models. Its function is to discover hidden cross-source data relationships. The model synthesizer dynamically combines data processing operators based on relationships. This can be achieved through a rule engine and a logic optimizer working together. Its function is to automatically generate data processing flows adapted to test requirements. The multi-source data adaptation layer is middleware that supports access to multiple data source protocols. This can be achieved through integration with JDBC, ODBC, and NoSQL drivers. Its function is to shield the heterogeneity of the underlying data sources.

[0062] Specifically, the metadata management module obtains structural information from heterogeneous data sources by connecting to the multi-source data adaptation layer, constructing a graph structure containing data entities and their relationships. The test intent parsing module receives test requirements input by users through a graphical interface or natural language, transforming them into a structured descriptor containing test objectives, core entities, and constraints. The dynamic semantic association engine performs semantic matching in the metadata graph based on this descriptor, identifying cross-data source association paths. The model synthesizer selects join, filter, and aggregation operators from the model operator library based on the discovered relationships, combining them to generate executable build logic. The execution engine compiles the logic into distributed computing tasks, accesses the actual data sources through the multi-source data adaptation layer, and ultimately generates physical data tables or virtual views for the test system to call.

[0063] Compared to existing technologies, traditional test data management systems require manual writing of complex scripts to integrate cross-source data and cannot dynamically respond to changing requirements. This system automatically establishes cross-source relationships through metadata graphs and dynamically generates data processing logic using semantic matching technology, eliminating the need for predefined fixed rules in the test data model construction process. Existing tools rely on expert experience for manual association when processing multi-source heterogeneous data, while this system automatically discovers cross-source relationships through a dynamic semantic association engine, significantly reducing operational complexity.

[0064] Through the above technical solution, this application achieves automation and intelligence in the test data model construction process. The system can automatically integrate heterogeneous data scattered across relational databases, in-memory databases, and big data platforms, and dynamically generate suitable data processing flows based on test intentions. This effectively solves the problems of cumbersome manual operations, difficulties in cross-source correlation, and the disconnect between test intentions and data models in traditional methods. Testers can quickly obtain test datasets that meet scenario requirements without manually writing complex query statements, significantly improving the efficiency and accuracy of test data preparation.

[0065] This application further proposes a test intent parsing module comprising a graphical user interface unit, a natural language processing unit, and a descriptor generation unit. The graphical user interface unit is used to receive user interaction operations; the natural language processing unit is used to parse the natural language text input by the user; and the descriptor generation unit is used to standardize the parsing results into test intent descriptors.

[0066] The graphical user interface (GUI) unit refers to interactive components that provide visual operation elements, specifically drag-and-drop controls and form input boxes, used to reduce user operation complexity and standardize input format. The natural language processing (NLP) unit refers to text semantic parsing components, specifically implemented using a BERT-based pre-trained model, used to accurately extract entity elements and constraints from user input. The descriptor generation unit refers to structured data transformation components, specifically implemented using a JSON Schema template mapping mechanism, used to transform unstructured input into a standardized descriptive format that can be processed by machines.

[0067] Specifically, when a user selects a test entity and inputs a natural language description through the graphical interface, the natural language processing unit performs entity recognition and dependency parsing on the input text to extract the core test object and its associated attributes. The descriptor generation unit then integrates the parsed semantic elements with the configuration information collected from the graphical interface, generating a standardized test intent descriptor according to a preset key-value pair structure. For example, when a user selects "order system" as the test entity and inputs "needs to include abnormal transaction data with an amount greater than 10,000 yuan," the system generates a structured descriptor containing the test target, core entity, and monetary constraints.

[0068] Compared to existing technologies, traditional test data configuration tools only provide input forms with fixed formats, which cannot handle complex test requirements described in natural language. This solution integrates graphical interaction with natural language understanding technology, enabling non-technical personnel to quickly express their test intentions using natural language, while ensuring the structured and standardized nature of the input information.

[0069] Through the above technical solution, this application achieves diversified and intelligent processing of test requirement inputs, effectively solving the problems of misunderstanding and cumbersome operation that exist when manually converting test intentions. By working collaboratively with the natural language processing unit and the graphical interface, it retains the flexibility of user expression while ensuring accurate parsing of input data in subsequent processing stages, providing high-quality input assurance for dynamically constructing test data models.

[0070] This application further proposes a dynamic test data model construction system based on multi-source heterogeneous data sources. The system also includes a logic optimizer connected between the model synthesizer and the execution engine, which is used to optimize the execution performance of the initial construction logic generated by the model synthesizer.

[0071] The logic optimizer is a component used to optimize the execution efficiency of data model building logic. It can be implemented using rule-based or cost-based optimization algorithms, such as selecting a better execution path by analyzing the operation order and resource consumption in the logic execution plan. The initial construction logic refers to the raw data operation flow generated by the model synthesizer based on test intent and metadata relationships. This can be represented as a combination of data join, filter, and aggregation operators, and its execution efficiency may be affected by data source characteristics or computation order. Execution performance optimization refers to reducing computational resource consumption or shortening execution time by adjusting the logic execution order or introducing caching mechanisms. This can be achieved using predicate pushdown, join order rearrangement, or intermediate result caching strategies. For example, filtering operations can be moved to the data reading stage to reduce the amount of data processed subsequently.

[0072] Specifically, after the model synthesizer generates the initial construction logic, the logic optimizer receives this logic and analyzes its execution plan. By traversing the operator nodes in the logic, it identifies optimizable operation steps, such as pushing filtering conditions to the data source for execution, or adjusting the connection order based on the data volume. Simultaneously, considering the computing power of the data source and network transmission costs, it dynamically selects the caching location for intermediate results to avoid redundant calculations. The optimized logic is then transformed into a task executable on the target computing engine, ultimately driven by the execution engine to generate the test data model.

[0073] Compared to existing technologies, current test data management tools typically execute manually written static logic directly, lacking the ability to dynamically optimize execution plans, resulting in low resource utilization or insufficient execution efficiency. This solution, however, introduces a logic optimizer that automatically adjusts execution strategies based on data source characteristics and runtime environment, reducing redundant computation and data transfer overhead, thereby improving the efficiency of complex cross-source data processing.

[0074] Through the above technical solution, this application solves the problems of low execution efficiency and high resource consumption in the existing technology of test data model construction logic. By dynamically optimizing the initial construction logic, the computational cost of cross-source data connection and processing can be effectively reduced, the test data generation time can be shortened, and the dependence on the performance of the underlying data source can be reduced, thereby improving the system's adaptability in complex heterogeneous environments.

[0075] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

[0076] This background section is provided to generally present the context of the invention. The work of the currently named inventors, the work to the extent described in this background section, and aspects of this section that did not constitute prior art at the time of application are neither expressly nor impliedly acknowledged as prior art to the invention.

Claims

1. A method for dynamically constructing test data models based on multi-source heterogeneous data sources, characterized in that, include: Step S1: Construct a cross-source metadata graph, integrate metadata from multiple heterogeneous data sources, and identify the semantic relationships between them; Step S2: Parse the user's input test intent and generate a test intent descriptor containing the test objective, test entity, and test conditions; Step S3: Using a dynamic semantic association engine, match the test intent descriptor with the cross-source metadata graph to automatically discover data entities and cross-source associations related to the test intent; Step S4: Based on the matching results, the model synthesizer driven by the test intent selects and combines one or more model operators from the model operator library to dynamically generate the construction logic of the target test data model; Step S5: Execute the construction logic to generate a target test data model that can be used by the test system.

2. The method for dynamically constructing a test data model based on multiple heterogeneous data sources according to claim 1, characterized in that, In step S1, the construction of the cross-source metadata graph specifically includes the following steps: Step S11: Connect to and scan multiple configured heterogeneous data sources through a multi-source data adapter, and automatically extract the table structure, field names, data types, and primary and foreign key constraint information of the data sources as basic metadata; Step S12: Use natural language processing technology to perform word segmentation, word vectorization, and semantic similarity calculation on the table names and field names in the basic metadata to generate semantic metadata; Step S13: Integrate the basic metadata with semantic metadata, and construct a cross-source metadata graph with data entities as nodes and structural and semantic associations between entities as edges.

3. The method for dynamically constructing a test data model based on multiple heterogeneous data sources according to claim 2, characterized in that, A graphical user interface is provided, which includes a test entity selection box, a test condition input box, and a test type drop-down menu. The system receives selection and configuration information input by the user through the graphical user interface and maps it into a test intent descriptor in the form of key-value pairs containing test objectives, core test entities, test constraints, and test scenario types.

4. The method for dynamically constructing a test data model based on multiple heterogeneous data sources according to claim 3, characterized in that, In step S3, the matching and discovery through the dynamic semantic association engine specifically includes: Using the core test entities and test constraints in the test intent descriptor as initial query nodes, a breadth-first traversal is performed in the cross-source metadata graph; During the traversal, the confidence of the associated path is calculated based on the edge weights; the edge weights include: structural association weights and semantic similarity weights. All associated paths with a confidence level higher than a preset threshold are selected. The data entities and their relationships covered by the associated paths are the automatically discovered cross-source associations.

5. The method for dynamically constructing a test data model based on multiple heterogeneous data sources according to claim 4, characterized in that, The model operator library includes at least data connection operators, data filtering operators, data aggregation operators, data derivation operators, and data desensitization / obfuscation operators; The model synthesizer determines the specific connection method of the data connection operator based on the discovered cross-source association relationship; Based on the test constraints in the test intent descriptor, assemble one or more data filtering operators; Based on the test scenario type, select and assemble the corresponding data aggregation operator or data derivation operator to construct data transformation logic that conforms to the test scenario.

6. The method for dynamically constructing a test data model based on multiple heterogeneous data sources according to claim 5, characterized in that, In step S4, the step of dynamically generating the construction logic of the target test data model further includes: After the model synthesizer generates the initial construction logic, it calls a logic optimizer to optimize the initial logic. The optimization strategies include: predicate pushdown, connection order rearrangement, and intermediate result caching. The logic optimizer selects the optimal optimization strategy based on the computing power, data volume, and network overhead of each data source to generate the final executable build logic.

7. The method for dynamically constructing a test data model based on multiple heterogeneous data sources according to claim 1, characterized in that, Step S5 includes: The build logic is compiled into a task that can be executed on the target data computing engine, including Spark, Flink, or a native database engine. The execution engine drives the computing engine to run tasks, materializes the results into a specific physical data table, or registers them as a virtual data view, and provides a unified access interface to the upstream testing system.

8. A dynamic construction system for test data models based on multi-source heterogeneous data sources, characterized in that, The method for dynamically constructing a test data model based on a multi-source heterogeneous data source, as described in any one of claims 1-7, includes: The metadata management module is used to build and maintain cross-source metadata graphs; The test intent parsing module is used to receive and parse user input and generate test intent descriptors; A dynamic semantic association engine is used to match and associate test intents with metadata graphs. The model synthesizer is used to dynamically assemble model operators and generate model building logic based on the association discovery results. The model operator library stores predefined operators for building various data models. The execution engine is used to execute the model building logic and generate the target test data model; The multi-source data adaptation layer is used to connect to and access different types of data sources.

9. The dynamic construction system for test data models based on multi-source heterogeneous data sources according to claim 8, characterized in that, The test intent parsing module includes: A graphical user interface unit for receiving user interaction operations; A natural language processing unit is used to parse the natural language text input by the user; A descriptor generation unit is used to standardize the parsing results into the test intent descriptor.

10. The dynamic construction system for test data models based on multi-source heterogeneous data sources according to claim 1, characterized in that, It also includes a logic optimizer connected between the model synthesizer and the execution engine, used to optimize the execution performance of the initial build logic generated by the model synthesizer.

Images