Data factory-based data production method, system and equipment

Abstract

The invention provides a data factory-based data production method, system and device, and relates to the technical field of data engineering.The method comprises the steps of obtaining business process data and a configuration instruction uploaded by a user, the business process data including BPMN node semantics, field constraints and process logic of a business process of the user; determining a parameter expression corresponding to the business process according to the configuration instruction; according to BPMN node semantics, field constraints and process logic, in combination with the parameter expression, generating a test data scene of the business process; packaging and executing interface calling according to the test data scene to obtain test data of the business process; wherein in the output process of the output test data, according to the preset period, the test data scene is dynamically updated according to the test data output in the preset period. According to the method, business process semantics, parameter expressions and periodic updating are used as an overall link of data production, so that the data output speed and quality are improved, and the business fitting degree is improved.

Description

Technical Field

[0001] This invention relates to the field of data engineering technology, and more specifically, to a data production method, system, and equipment based on a data factory. Background Technology

[0002] Against the backdrop of continuous testing system standardization and the rapid development of data-driven architecture, data factories are used to achieve automated management and efficient processing of the entire data lifecycle. Existing data factories generally use static parameter templates combined with open-source scheduling engines to output data. This involves manually configuring the API call order, manually constructing form data, and using traditional tools to perform simple data transformation and delivery. Such methods typically operate within a single environment, with test data and business processes maintained independently, and execution driven by scripts or simple rules.

[0003] In related technologies, as the complexity of business processes increases, static parameter mechanisms cannot adapt to dynamic changes during runtime, resulting in insufficient scheduling intelligence. At the same time, due to the lack of semantic understanding capabilities of open-source scheduling engines, a gap is formed between test instructions and business process nodes, resulting in data and processes being in a weakly coupled state for a long time. The preparation of test data is difficult to cover edge scenarios, which in turn affects the data output speed, quality, and matching degree between output data and processes of the data factory. Summary of the Invention

[0004] The problem addressed by this invention is how to improve the data output of a data factory.

[0005] To address the aforementioned problems, this invention provides a data production method, system, and equipment based on a data factory.

[0006] In a first aspect, the present invention provides a data production method based on a data factory, comprising: Obtain business process data and configuration instructions uploaded by the user. The business process data includes the BPMN node semantics, field constraints, and process logic of the user's business process. Based on the configuration instructions, determine the parameter expression corresponding to the business process; Based on the BPMN node semantics, the field constraints, and the process logic, combined with the parameter expressions, test data scenarios for the business process are generated. The test data scenario is encapsulated and the interface is called to obtain the test data of the business process; wherein, during the production of the test data, the test data scenario is dynamically updated according to the test data produced within the preset period.

[0007] Optionally, determining the dynamic parameter expression corresponding to the business process based on the configuration instruction includes: Lexical-syntax analysis is performed on the configuration instructions to obtain the database field references, environment variable injections, and service interface call instructions in the configuration instructions; The database field reference, the environment variable injection, and the service interface call instruction are respectively mapped to the first placeholder, the second placeholder, and the third placeholder; Based on the field constraints of the business process, the first placeholder, the second placeholder, and the third placeholder are processed respectively to obtain the real-time field value, the runtime configuration value, and the external return value. The real-time field value, the runtime configuration value, and the external return value are input into a preset expression template to obtain the parameter expression corresponding to the business process.

[0008] Optionally, the step of generating test data scenarios for the business process based on the BPMN node semantics, the field constraints, and the process logic, combined with the parameter expressions, includes: Construct a directed topology graph of the business process based on the semantics of the BPMN nodes; Based on the field constraints, generate a JSON-Schema fragment for each graph vertex in the directed topology graph; The real-time field value, the runtime configuration value, and the external return value in the parameter expression are injected into the JSON-Schema fragment to obtain a dynamically instantiated template. The execution probability of each edge in the directed topology graph is calculated based on the reinforcement learning model, and the Mock rule tree is determined based on the execution probability. Based on the Mock rule tree and the dynamically instantiated template, the test data scenario for the business process is generated.

[0009] Optionally, the step of encapsulating the test data scenario and executing the interface call to obtain the test data of the business process includes: The test data scenario is packaged into a task package; The task package is assigned to the execution node in a preset distributed cluster by a work-stealing algorithm. The task package is executed in parallel through the execution node, and the interface call is called to obtain the test data of the business process.

[0010] Optionally, determining the execution node of the task package in a preset distributed cluster using a work-stealing algorithm and scheduling the task package to the execution node includes: Based on the topology information of the preset distributed cluster, construct a node-core-thread distance matrix; Based on the distance matrix, all nodes in the preset distributed cluster are filtered to determine the candidate execution nodes for the task package; The work-stealing algorithm performs lock-free CAS stealing in the local queue of the candidate execution node until the execution node is determined, and then the task package is scheduled and migrated to the execution node.

[0011] Optionally, the step of dynamically updating the test data scenario according to the test data generated within the preset period during the production of the test data includes: The real-time test data generated by the execution node within the preset period is obtained, and the difference measure of the real-time test data is obtained based on the difference between the real-time test data and the preset test data expected value. The difference metric is input into the closed-loop feedback engine to incrementally correct the real-time field value, the runtime configuration value, and the external return value in the parameter expression, thereby obtaining the incrementally corrected parameter expression. Based on the parameter expression after incremental correction, the test data scenario of the business process is incrementally updated to obtain the updated test data scenario; The updated test data scenario is repackaged into a task package, and the interface call is executed again in the next cycle.

[0012] Optionally, the step of inputting the difference metric into the closed-loop feedback engine to incrementally correct the real-time field value, the runtime configuration value, and the external return value in the parameter expression to obtain the incrementally corrected parameter expression includes: Based on the difference metric, the correction weights of the first placeholder, the second placeholder, and the third placeholder are determined; Based on the correction weights of the first placeholder, the second placeholder, and the third placeholder, weighted incremental updates are performed on the real-time field value, the runtime configuration value, and the external return value, respectively. The updated real-time field value, the updated runtime configuration value, and the updated external return value are then filled back into the parameter expression to form the incrementally corrected parameter expression.

[0013] Optionally, the scenario of incrementally updating the test data of the business process based on the incrementally corrected parameter expression includes: The incrementally corrected parameter expression is injected into the directed topology graph of the business process. The JSON-Schema fragment of each graph vertex is re-instantiated, and the real-time field value, runtime configuration value, and external return value in the incrementally corrected parameter expression are injected into the re-instantiated JSON-Schema fragment to obtain the updated dynamic instantiation template. Based on the values ​​of the re-instantiated JSON-Schema fragment, the Mock rule tree is partially updated to obtain the partially updated Mock rule tree; The locally updated Mock rule tree and the updated dynamic instantiation template are merged to obtain the updated test data scenario.

[0014] Secondly, the present invention provides a data production system based on a data factory, comprising: The data receiving module is used to acquire business process data and configuration instructions uploaded by the user. The business process data includes the BPMN node semantics, field constraints and process logic of the user's business process. The parameter generation module is used to determine the parameter expression corresponding to the business process based on the configuration instructions. The scenario generation module is used to generate test data scenarios for the business process based on the BPMN node semantics, the field constraints, and the process logic, combined with the parameter expressions. The execution module is used to encapsulate and execute interface calls based on the test data scenario to obtain the test data of the business process; An update module is used to dynamically update the test data scenario according to the test data produced within the preset period during the process of producing the test data.

[0015] Thirdly, the present invention provides an electronic device, including a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the data production method based on a data factory as described above when executing the computer program.

[0016] This invention relates to a data factory-based data production method, system, and equipment. By integrating business process semantics, dynamic parameter expressions, and periodic closed-loop updates into the overall data production chain, it upgrades the data factory from static template-based data creation to adaptive data growth, thereby improving the speed, quality, and business relevance of data output. First, through explicit input of BPMN node semantics, field constraints, and process logic, previously isolated test requirements are precisely bound to the business process. This ensures that the variable values ​​and path branches at each subsequent step have clear business basis, resolving the weak coupling between data and process in related technologies and avoiding the inefficiency and errors caused by the disconnect between business process and data generation in traditional methods. Subsequently, the parameter expressions corresponding to the business process are determined based on configuration instructions. By parsing database field references, environment variable injections, and service interface call instructions in the configuration instructions, dynamic parameter expressions are generated, achieving flexibility and adaptability in test data generation. Compared to static parameter templates, dynamic parameter expressions can adjust in real time according to changes in the business process, better meeting complex and ever-changing test requirements and reducing the problem of insufficient test coverage caused by rigid parameters. Then, by combining BPMN node semantics, field constraints, process logic, and parameter expressions, test data scenarios for the business process are generated. Utilizing the node semantics and field constraints in the business process data, it is ensured that the generated test data not only meets the format requirements but also maintains logical consistency with the business process. This effectively improves the quality and reliability of the test data, reducing test errors and repeated corrections caused by data mismatch with business logic. Finally, during the test data generation process, the test data scenarios are dynamically updated according to a preset cycle based on the generated test data. This periodic update mechanism allows the data factory to provide real-time feedback on test results and automatically adjust the test data scenarios to adapt to the evolution of business processes and changes in test requirements. This not only accelerates the iteration speed of test data but also enhances the intelligence and automation level of the testing process, reducing manual intervention and maintenance costs. In summary, this solution, by tightly integrating business process data with the test data generation process and introducing dynamic parameter expressions and a periodic scenario update mechanism, comprehensively improves the data output effect of the data factory. It not only accelerates the data generation speed but also significantly improves the data quality and its matching degree with business processes, supporting efficient and accurate test data. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the data production method based on a data factory according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a data production system based on a data factory according to an embodiment of the present invention. Detailed Implementation

[0018] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0019] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0020] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0021] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0022] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties. The collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0023] Combination Figure 1 As shown, an embodiment of the present invention provides a data production method based on a data factory, which includes: Obtain business process data and configuration instructions uploaded by the user. The business process data includes the BPMN node semantics, field constraints, and process logic of the user's business process.

[0024] Specifically, the process first acquires the business process data and configuration instructions uploaded by the user. The business process data includes BPMN node semantics, field constraints, and process logic. Specifically, BPMN node semantics defines each node in the business process and their relationships; field constraints specify the format and range of data required for each node; and process logic specifies the execution order and conditions of the business process. This information collectively constitutes the business process, thus providing a precise business context for the generation of subsequent test data. In a preferred embodiment of the invention, business process data and configuration instructions can be uploaded through a visual orchestration layer (Vue). The visual orchestration layer provides an intuitive interface that allows users to define the BPMN node semantics, field constraints, and process logic of the business process, and to input configuration instructions, such as database field references, environment variable injection, and service interface call instructions. This information collectively constitutes the business process, providing a precise business context for the generation of subsequent test data.

[0025] Based on the configuration instructions, determine the parameter expression corresponding to the business process.

[0026] Specifically, the configuration instructions include database field references, environment variable injection, and service interface call instructions. By parsing the configuration instructions, the parameter expressions corresponding to the business process can be determined. Specifically, the parameter expressions map database fields, environment variables, and service interface calls to placeholders, and then, combined with the field constraints of the business process, calculate the real-time field values, runtime configuration values, and external return values. Therefore, the parameter expressions are dynamic and can be adjusted in real time according to changes in the business process, thus providing flexibility and adaptability for test data generation. In a preferred embodiment of the present invention, in the visual orchestration layer, the configuration instructions uploaded by the user are subjected to lexical-syntax parsing to accurately extract the database field references, environment variable injection, and service interface call instructions.

[0027] Based on the BPMN node semantics, the field constraints, and the process logic, combined with the parameter expressions, test data scenarios for the business process are generated.

[0028] Specifically, a directed topology graph of the business process is constructed based on the semantics of BPMN nodes. Each node in the directed topology graph represents a business operation, and each edge represents the flow relationship between operations. A JSON-Schema fragment is generated for each node based on field constraints, defining the data format and range. Real-time field values, runtime configuration values, and external return values ​​are injected into the JSON-Schema fragment along with parameter expressions to form a dynamically instantiated template. In embodiments of this invention, a reinforcement learning model can be used to calculate the execution probability of each edge in the topology graph, determine the Mock rule tree, and then generate test data scenarios based on the Mock rule tree and the dynamically instantiated template. This ensures that the generated test data scenarios highly match the business process, meet testing requirements, significantly improve the quality and reliability of test data, and reduce test errors and repeated corrections caused by mismatches between data and business logic.

[0029] The test data scenario is encapsulated and the interface is called to obtain the test data of the business process; wherein, during the production of the test data, the test data scenario is dynamically updated according to the test data produced within the preset period.

[0030] Specifically, the encapsulation of test data scenarios and the execution of API calls are key steps in achieving data production goals. First, test data scenarios are transformed into task packages, which contain all the information needed to execute API calls, such as request parameters and expected responses. This encapsulation ensures that test data scenarios can be effectively transmitted and executed, providing a clear instruction set for API calls. Upon receiving the task package, the execution node executes API calls in parallel, allowing multiple requests to be sent simultaneously, thus improving the efficiency of test data production. The result of executing the API calls is the test data for the business process, which will be used for subsequent business process verification and optimization. Second, the dynamic updating of test data scenarios is the core mechanism for ensuring the quality and relevance of test data. During test data production, produced test data is collected according to a preset cycle, and test data scenarios are adjusted based on this data. This update process allows for the identification and adaptation to changes in business processes based on the actual produced data, ensuring that test data scenarios continuously reflect the latest business status. This enables the data to maintain its validity and accuracy in a constantly changing business environment, thereby improving the adaptability and output effectiveness of the entire data factory. In a preferred embodiment of the present invention, during the execution of interface calls, the generated test data can be adapted through a BPM protocol adapter to ensure that the test data can be correctly connected with the business process in accordance with the BPMN specification, supporting the execution and verification of the business process.

[0031] This invention presents a data production method based on a data factory. By integrating business process semantics, dynamic parameter expressions, and periodic closed-loop updates into the overall data production chain, it upgrades the data factory from static template-based data creation to adaptive data growth, thereby improving the speed, quality, and business relevance of data output. First, through explicit input of BPMN node semantics, field constraints, and process logic, previously isolated test requirements are precisely bound to the business process. This ensures that the variable values ​​and path branches at each subsequent step have clear business basis, resolving the weak coupling between data and process in related technologies and avoiding the inefficiency and errors caused by the disconnect between business process and data generation in traditional methods. Then, based on configuration instructions, parameter expressions corresponding to the business process are determined. By parsing database field references, environment variable injections, and service interface call instructions in the configuration instructions, dynamic parameter expressions are generated, achieving flexibility and adaptability in test data generation. Compared to static parameter templates, dynamic parameter expressions can adjust in real time according to changes in the business process, better meeting complex and ever-changing test requirements and reducing the problem of insufficient test coverage caused by rigid parameters. Then, by combining BPMN node semantics, field constraints, process logic, and parameter expressions, test data scenarios for the business process are generated. Utilizing the node semantics and field constraints in the business process data, it is ensured that the generated test data not only meets the format requirements but also maintains logical consistency with the business process. This effectively improves the quality and reliability of the test data, reducing test errors and repeated corrections caused by data mismatch with business logic. Finally, during the test data generation process, the test data scenarios are dynamically updated according to a preset cycle based on the generated test data. This periodic update mechanism allows the data factory to provide real-time feedback on test results and automatically adjust the test data scenarios to adapt to the evolution of business processes and changes in test requirements. This not only accelerates the iteration speed of test data but also enhances the intelligence and automation level of the testing process, reducing manual intervention and maintenance costs. In summary, this solution, by tightly integrating business process data with the test data generation process and introducing dynamic parameter expressions and a periodic scenario update mechanism, comprehensively improves the data output effect of the data factory. It not only accelerates the data generation speed but also significantly improves the data quality and its matching degree with business processes, supporting efficient and accurate test data.

[0032] Optionally, determining the dynamic parameter expression corresponding to the business process based on the configuration instruction includes: Lexical-syntax analysis is performed on the configuration instructions to obtain the database field references, environment variable injections, and service interface call instructions in the configuration instructions; The database field reference, the environment variable injection, and the service interface call instruction are respectively mapped to the first placeholder, the second placeholder, and the third placeholder; Based on the field constraints of the business process, the first placeholder, the second placeholder, and the third placeholder are processed respectively to obtain the real-time field value, the runtime configuration value, and the external return value. The real-time field value, the runtime configuration value, and the external return value are input into a preset expression template to obtain the parameter expression corresponding to the business process.

[0033] Specifically, the configuration instructions first require lexical-syntax parsing to accurately extract database field references, environment variable injections, and service interface call instructions. Database field references ensure the accuracy of the data source, environment variable injection allows test data to adapt to different runtime environments, and service interface call instructions guarantee data real-time performance and interactivity with external systems. Next, these instructions are mapped to first, second, and third placeholders, laying the foundation for subsequent dynamic data generation. This placeholder mapping method allows for flexible insertion and replacement of actual field values, configuration values, and interface return values ​​in parameter expressions, facilitating subsequent and reverse dynamic updates of parameter expressions. Then, based on the field constraints of the business process, the placeholders are processed to obtain real-time field values, runtime configuration values, and external return values. The application of field constraints ensures that the generated data conforms to business rules, avoiding the generation of invalid data. Finally, these processed values ​​are input into a preset expression template to generate parameter expressions that precisely match the business process. This method, based on instruction parsing and placeholder mapping, not only improves the flexibility and adaptability of parameter expressions, but also ensures that the generated test data accurately reflects the needs of the business process.

[0034] In a preferred embodiment of the present invention, when implementing the data production method of the present invention, when a user uploads business process data and configuration instructions, the business process data describes an order processing flow, including BPMN node semantics (such as start node, approval node, and end node), field constraints (such as order amount must be greater than 0, customer name cannot be empty), and process logic (such as proceeding to the shipment stage upon approval). The configuration instructions include database field references (such as order_id and customer_name in the order table), environment variable injection (such as tax rate tax_rate), and service interface calls (such as inventory query interface). First, the configuration instructions are lexical-syntax parsed to extract the database field references, environment variable injection, and service interface call instructions. Then, these instructions are mapped to a first placeholder (such as db:order_id), a second placeholder (such as {env:tax_rate}), and a third placeholder (such as api:stock_check). Then, based on the field constraints of the business process, these placeholders are processed to obtain real-time field values ​​(such as order_id read from the database), runtime configuration values ​​(such as tax_rate of the current environment), and external return values ​​(such as stock_status returned by the inventory query interface). Finally, these values ​​are input into a preset expression template to generate dynamic parameter expressions (such as total={db:order_amount}* (1+ ${env:tax_rate}), thus obtaining the parameter expression corresponding to the business process.

[0035] In this embodiment of the invention, by mapping database field references, environment variable injections, and service interface call instructions to placeholders, and processing them according to field constraints to dynamically generate parameter expressions closely related to business processes, test data can adapt to changes in business processes in real time, reducing manual intervention and maintenance costs, while improving the efficiency and quality of test data generation.

[0036] Optionally, the step of generating test data scenarios for the business process based on the BPMN node semantics, the field constraints, and the process logic, combined with the parameter expressions, includes: Construct a directed topology graph of the business process based on the semantics of the BPMN nodes; Based on the field constraints, generate a JSON-Schema fragment for each graph vertex in the directed topology graph; The real-time field value, the runtime configuration value, and the external return value in the parameter expression are injected into the JSON-Schema fragment to obtain a dynamically instantiated template. The execution probability of each edge in the directed topology graph is calculated based on the reinforcement learning model, and the Mock rule tree is determined based on the execution probability. Based on the Mock rule tree and the dynamically instantiated template, the test data scenario for the business process is generated.

[0037] Specifically, firstly, a directed topology graph of the business process is constructed based on the semantics of BPMN nodes. This involves identifying and processing the various nodes in the business process and their interrelationships. Each node represents a specific business operation or decision point, while the connections between nodes indicate the flow direction and sequence of the business process. This graphically represents complex business logic, providing a clear process framework for the subsequent generation of test data. Next, JSON-Schema fragments are generated for each node based on field constraints, defining the data format and validation rules for each business process node. Field constraints ensure that the generated test data not only meets expectations in structure but also satisfies the specific requirements of the business process in content. Simultaneously, the JSON-Schema fragments detail the type, range, and required fields of each data field, ensuring the accuracy and consistency of the test data. Then, real-time field values, runtime configuration values, and external return values ​​from parameter expressions are injected into the JSON-Schema fragments to form dynamically instantiated templates. Parameter expressions provide flexibility and dynamism, allowing test data scenarios to adapt to different testing conditions and environments. In this way, test data matching actual business scenarios can be generated, improving test coverage and effectiveness. Finally, the execution probability of each edge in the topology graph is calculated using a reinforcement learning model to determine the Mock rule tree. Based on the logic of the business process and historical data, the Mock rule tree simulates possible business scenarios and data changes, providing richer and more diverse content for test data generation. This ensures that the test data covers various possible business situations, thereby improving the comprehensiveness and reliability of the tests.

[0038] In a preferred embodiment of the present invention, when a user uploads business process data for an order processing flow, which includes BPMN node semantics (such as start node, approval node, and end node), field constraints (such as order amount must be greater than 0, customer name cannot be empty), and process logic (such as proceeding to the shipment stage upon approval), a directed topology graph of the business process is first constructed based on the BPMN node semantics to identify each node and its relationships. Next, a JSON-Schema fragment is generated for each node based on the field constraints, defining the data format and validation rules. For example, the JSON-Schema fragment for an order node includes the types and constraints of fields such as order ID, customer name, and order amount. Then, real-time field values, runtime configuration values, and external return values ​​from the parameter expressions are injected into the JSON-Schema fragment to form a dynamically instantiated template. For example, real-time field values ​​come from the order table in the database, runtime configuration values ​​come from the tax rate in the environment variables, and external return values ​​come from the inventory query interface. Finally, a reinforcement learning model is used to calculate the execution probability of each edge in the topology graph, determine the Mock rule tree, and generate test data scenarios based on the Mock rule tree and the dynamically instantiated template. The generated test data scenarios will include various possible order processing situations, such as normal orders, approved orders, and failed orders, to ensure that the test data can fully cover all possible situations in the business process.

[0039] In this embodiment of the invention, by constructing a directed topology graph, generating JSON-Schema fragments, dynamically instantiating templates, and determining a Mock rule tree, the automated and intelligent generation of test data scenarios is achieved. This not only improves the efficiency and accuracy of test data generation but also ensures that the generated test data comprehensively covers all possible scenarios in the business process.

[0040] Optionally, the step of encapsulating the test data scenario and executing the interface call to obtain the test data of the business process includes: The test data scenario is packaged into a task package; The task package is assigned to the execution node in a preset distributed cluster by a work-stealing algorithm. The task package is executed in parallel through the execution node, and the interface call is called to obtain the test data of the business process.

[0041] Specifically, the task package encapsulation process involves integrating test data scenarios and their related parameters and configuration information into an executable unit, ensuring that the task package can be effectively transmitted and executed in a distributed environment. Next, a work-stealing algorithm is used to determine the execution node of the task package in a pre-defined distributed cluster. The work-stealing algorithm is an efficient load balancing strategy that filters candidate execution nodes by constructing a node-core-thread distance matrix and performs lock-free CAS stealing in the local queues of these nodes until the final execution node is determined. This algorithm can effectively utilize cluster resources, reduce task waiting time, and improve task execution efficiency. After the task package is scheduled to the determined execution node, the execution node executes interface calls in parallel. Parallel execution allows multiple interface calls to be performed simultaneously, significantly improving the speed of test data generation. Finally, the response data obtained through the interface calls becomes the test data for the business process, which will be used to verify and optimize the business process. In a preferred embodiment of the invention, after receiving the task package, the execution node schedules the task through a distributed scheduling engine that uses a work-stealing algorithm to determine the execution node of the task package in a pre-defined distributed cluster. If the cluster has multiple nodes, the work-stealing algorithm selects the most suitable node for task scheduling based on the node's load and the task package's requirements. After the task package is scheduled to the selected execution node, the node executes API calls in parallel, such as simultaneously calling the order creation API, inventory query API, and payment API. The response data from these API calls is collected and integrated to generate test data for the order processing flow. This generated test data is used to verify the correctness and performance of the order processing flow, ensuring its reliability and stability in actual business operations.

[0042] In this embodiment of the invention, test data scenarios are encapsulated into task packages, and a work-stealing algorithm is used to efficiently schedule these task packages in a distributed cluster, enabling rapid generation of test data. This ensures the integrity and executability of the test data scenarios, and intelligent scheduling improves task execution efficiency and cluster resource utilization, thereby generating a large amount of high-quality test data in a short time.

[0043] Optionally, determining the execution node of the task package in a preset distributed cluster using a work-stealing algorithm and scheduling the task package to the execution node includes: Based on the topology information of the preset distributed cluster, construct a node-core-thread distance matrix; Based on the distance matrix, all nodes in the preset distributed cluster are filtered to determine the candidate execution nodes for the task package; The work-stealing algorithm performs lock-free CAS stealing in the local queue of the candidate execution node until the execution node is determined, and then the task package is scheduled and migrated to the execution node.

[0044] Specifically, in distributed systems, the efficiency of task scheduling directly impacts the overall system performance and resource utilization. Using a work-stealing algorithm to determine the execution node for a task package is an efficient load balancing strategy. First, a node-core-thread distance matrix is ​​constructed based on the pre-defined topology of the distributed cluster. This matrix reflects the hardware affinity between nodes in the cluster, such as the difference in memory access latency between nodes in a NUMA architecture. By filtering all nodes in the cluster based on this matrix, candidate execution nodes for task packages can be quickly identified, thus avoiding execution on unsuitable nodes and reducing cross-node communication overhead. Subsequently, a lock-free CAS (Compare-And-Swap) stealing operation is performed in the local queue of the candidate execution nodes. This lock-free mechanism effectively reduces competition and synchronization overhead between threads, improving the efficiency of task stealing. In this way, the system can quickly find the most suitable node for executing the task package in the cluster and schedule and migrate the task package to that node for execution, thereby achieving efficient load balancing and task scheduling.

[0045] In this embodiment of the invention, by combining a work-stealing algorithm with a node-core-thread distance matrix and a lock-free CAS stealing mechanism, the efficiency of task scheduling and resource utilization in a distributed cluster can be significantly improved. First, node filtering based on topology information ensures that task packets are scheduled to the node most closely matching the task requirements, reducing performance loss caused by inter-node communication latency. Second, the lock-free CAS stealing mechanism avoids the thread blocking problem in traditional locking mechanisms, making task stealing operations more efficient and further reducing task waiting time. Finally, this efficient scheduling mechanism enables the distributed cluster to maintain good performance in high-concurrency scenarios, improving the overall throughput and response speed of the system.

[0046] Optionally, the step of dynamically updating the test data scenario according to the test data generated within the preset period during the production of the test data includes: The real-time test data generated by the execution node within the preset period is obtained, and the difference measure of the real-time test data is obtained based on the difference between the real-time test data and the preset test data expected value. The difference metric is input into the closed-loop feedback engine to incrementally correct the real-time field value, the runtime configuration value, and the external return value in the parameter expression, thereby obtaining the incrementally corrected parameter expression. Based on the parameter expression after incremental correction, the test data scenario of the business process is incrementally updated to obtain the updated test data scenario; The updated test data scenario is repackaged into a task package, and the interface call is executed again in the next cycle.

[0047] Specifically, in the test data generation process, dynamically updating the test data scenario is a crucial step to ensure that the test data closely aligns with the business process. By acquiring real-time test data generated by execution nodes within a preset period and comparing it with preset expected test data values, a difference metric can be calculated. This difference metric reflects the degree of deviation between the generated test data and the expected target. Subsequently, the difference metric is input into a closed-loop feedback engine to incrementally correct real-time field values, runtime configuration values, and external return values ​​in the parameter expressions. This incremental correction mechanism dynamically adjusts the parameter expressions based on real-time feedback, ensuring that the generated test data better adapts to changes in the business process. Finally, the test data scenario is incrementally updated based on the corrected parameter expressions, and the updated test data scenario is repackaged into a task package for continued execution of interface calls in the next period. The closed-loop feedback mechanism enables adaptive adjustment of test data generation, ensuring that the test data scenario continuously reflects the latest business status.

[0048] In a preferred embodiment of the present invention, in a test data generation scenario of an order processing system, test data needs to be generated based on real-time business data. Within each preset period (e.g., hourly), the system obtains real-time test data from the execution node, such as order amount, customer name, and inventory status. The system compares this real-time test data with preset expected test data values ​​(e.g., order amount range, customer name format, inventory status threshold, etc.) and calculates a difference metric. For example, if the order amount in the real-time test data exceeds the expected range, the system inputs this difference metric into the closed-loop feedback engine. The closed-loop feedback engine incrementally corrects the real-time field values ​​(e.g., order amount) in the parameter expression based on the difference metric, for example, by adjusting the generated range of order amount. The corrected parameter expression is used to update the test data scenario and generate new test data. The updated test data scenario is repackaged into a task package and the interface call continues to be executed in the next period to generate test data that better meets business requirements. In this way, the system can continuously optimize the test data generation process, ensuring that the test data always remains consistent with the business process, thereby improving the accuracy and reliability of the tests.

[0049] In this embodiment of the invention, firstly, by comparing real-time test data with preset expected values, deviations in the test data generation process can be quickly identified, and these deviations can be quantified through difference measurement. Secondly, the closed-loop feedback engine can perform precise incremental corrections to the parameter expressions based on the difference measurement, avoiding the performance overhead and complexity caused by full updates.

[0050] Optionally, the step of inputting the difference metric into the closed-loop feedback engine to incrementally correct the real-time field value, the runtime configuration value, and the external return value in the parameter expression to obtain the incrementally corrected parameter expression includes: Based on the difference metric, the correction weights of the first placeholder, the second placeholder, and the third placeholder are determined; Based on the correction weights of the first placeholder, the second placeholder, and the third placeholder, weighted incremental updates are performed on the real-time field value, the runtime configuration value, and the external return value, respectively. The updated real-time field value, the updated runtime configuration value, and the updated external return value are then filled back into the parameter expression to form the incrementally corrected parameter expression.

[0051] Specifically, firstly, based on the difference measure between real-time test data and preset expected values, the correction weight of each placeholder in the parameter expression (such as database field references, environment variable injection, and service interface calls) is determined. By quantifying the difference measure, a weight value is assigned to each placeholder, thereby determining the importance and adjustment range of each parameter during the correction process. Subsequently, based on the correction weight, weighted incremental updates are performed on real-time field values, runtime configuration values, and external return values, respectively. This weighted update mechanism can flexibly adjust parameter values ​​according to dynamic changes in business needs, ensuring that the generated test data is closer to the actual business scenario. Finally, the updated parameter values ​​are filled back into the parameter expression to form the incrementally corrected parameter expression. This process not only improves the adaptability and accuracy of test data but also realizes the automation and intelligent adjustment of test data generation through a closed-loop feedback mechanism. In a preferred embodiment of the present invention, real-time test data, such as order amount, tax rate, and inventory status, is obtained from the execution node within each preset period (e.g., hourly). These real-time test data are compared with preset expected test data values ​​(such as order amount range, tax rate range, and inventory status threshold), and the difference measure is calculated. If the order amount in the real-time test data exceeds the expected range, the correction weight of the database field reference (such as the order amount) will be determined based on the difference metric. Assuming a correction weight of 0.8, it indicates that adjusting the order amount is relatively important. A weighted incremental update is performed on the real-time field value (such as the order amount) according to the correction weight, for example, adjusting the order amount from 100 to 120. Simultaneously, the runtime configuration value (such as the tax rate) will be adjusted based on the correction weights of other placeholders (such as a tax rate correction weight of 0.5), for example, adjusting the tax rate from 10% to 11%. Finally, the updated parameter values ​​(such as order amount 120, tax rate 11%) are backfilled into the parameter expression, forming the incrementally corrected parameter expression. The updated parameter expression is used to generate new test data scenarios, and the API calls will continue to be executed in the next cycle to generate test data that better meets business needs. In this way, the test data generation process can be continuously optimized, ensuring that the test data always remains consistent with the business process, improving the accuracy and reliability of the tests.

[0052] In this embodiment of the invention, by determining the correction weight through difference measurement, it is possible to accurately identify which parameters need to be adjusted and by how much, avoiding errors caused by blind correction. Furthermore, the weighted incremental update mechanism allows for flexible adjustment of parameter values ​​based on changes in business needs, ensuring that the generated test data more closely reflects actual business scenarios.

[0053] Optionally, the scenario of incrementally updating the test data of the business process based on the incrementally corrected parameter expression includes: The incrementally corrected parameter expression is injected into the directed topology graph of the business process. The JSON-Schema fragment of each graph vertex is re-instantiated, and the real-time field value, runtime configuration value, and external return value in the incrementally corrected parameter expression are injected into the re-instantiated JSON-Schema fragment to obtain the updated dynamic instantiation template. Based on the values ​​of the re-instantiated JSON-Schema fragment, the Mock rule tree is partially updated to obtain the partially updated Mock rule tree; The locally updated Mock rule tree and the updated dynamic instantiation template are merged to obtain the updated test data scenario.

[0054] Specifically, in the process of dynamically updating test data scenarios, this invention updates the test data scenarios of the business process through incrementally corrected parameter expressions. First, the incrementally corrected parameter expressions are injected into the directed topology graph of the business process, and the JSON-Schema fragment of each graph vertex is re-instantiated to ensure that the data format and validation rules of each business process node can be adjusted according to the latest parameter values. Next, the incrementally corrected real-time field values, runtime configuration values, and external return values ​​are injected into the re-instantiated JSON-Schema fragments to form an updated dynamic instantiation template. This template can dynamically generate test data that meets the requirements of the business process. Subsequently, based on the values ​​of the re-instantiated JSON-Schema fragments, the Mock rule tree is partially updated. The Mock rule tree is generated based on business process logic and historical data to simulate business scenarios and data changes. Partial updates ensure that the Mock rule tree can quickly adapt to parameter changes without requiring a full regeneration. Finally, the partially updated Mock rule tree is merged with the updated dynamic instantiation template to obtain the updated test data scenario. This process, through partial updates and dynamic instantiation, enables efficient and adaptive updates of test data scenarios, ensuring that test data always remains consistent with business processes.

[0055] In this embodiment of the invention, by re-instantiating the JSON-Schema fragment and injecting the latest parameter values, the data format and validation rules of each business process node are ensured to be dynamically adjusted, thereby generating test data that meets the latest business requirements. Utilizing a partially updated Mock rule tree allows for rapid adaptation to parameter changes, reducing the performance overhead and complexity of full updates. Finally, by merging the partially updated Mock rule tree and the dynamically instantiated template, the test data generation process can be continuously optimized in each cycle, ensuring that the test data scenario always remains consistent with the business process.

[0056] In summary, in the preferred embodiment of the present invention, the user first uploads business process data and configuration instructions through a visual orchestration layer. The business process data details the order processing flow, including BPMN node semantics (e.g., start node, order creation node, approval node, shipping node, end node), field constraints (e.g., order amount must be greater than 0, customer name cannot be empty, approval status must be "Approved" or "Rejected"), and process logic (e.g., if approval is granted, the shipping process begins; otherwise, the process ends). The configuration instructions include database field references (e.g., order_id, customer_name, order_amount from the order table), environment variable injection (e.g., tax rate tax_rate), and service interface calls (e.g., inventory query interface stock_check, payment interface payment_process). Subsequently, the user-uploaded configuration instructions are lexical-syntactic parsed to accurately extract database field references, environment variable injection, and service interface call instructions. These instructions are mapped to placeholders, such as db:order_id, env:tax_rate, and api:stock_check. Based on the field constraints of the business process, these placeholders are processed to obtain real-time field values ​​(e.g., order_amount read from the database is 100), runtime configuration values ​​(e.g., tax_rate of the current environment is 0.1), and external return values ​​(e.g., stock_status returned by the inventory query interface is "In Stock"). These values ​​are then fed into a preset expression template to generate dynamic parameter expressions, such as total_amount = {db:order_amount} * (1 + {env:tax_rate}) and stock_status ={api:stock_check}.

[0057] Next, a directed topology graph of the business process is constructed based on the semantics of BPMN nodes. Each node represents a business operation, and each edge represents the flow relationship between operations. For example, the process starts from the start node, proceeds to the order creation node, then to the approval node, and finally, the approval status determines whether to proceed to the shipping node or the end node. A JSON-Schema fragment is generated for each node based on field constraints, defining the data format and scope. For example, the JSON-Schema fragment for the order creation node includes the types and constraints of fields such as order ID, customer name, and order amount. Real-time field values, runtime configuration values, and external return values ​​are injected into the JSON-Schema fragment using parameter expressions to form a dynamically instantiated template. A reinforcement learning model is used to calculate the execution probability of each edge in the topology graph, determining the Mock rule tree. Finally, test data scenarios are generated based on the Mock rule tree and the dynamically instantiated template. This ensures that the generated test data scenarios highly match the business process, meeting testing requirements and significantly improving the quality and reliability of the test data, reducing test errors and repeated corrections caused by data mismatch with business logic.

[0058] The generated test data scenarios are transformed into task packages containing all the information required to execute interface calls, such as request parameters and expected responses. These task packages are scheduled by a distributed scheduling engine using a work-stealing algorithm. Based on the pre-defined topology of the distributed cluster, a node-core-thread distance matrix is ​​constructed to filter candidate execution nodes. Lock-free CAS stealing is then performed in the local queues of these nodes until the final execution node is determined. After the task package is scheduled to the determined execution node, the node executes interface calls in parallel, such as simultaneously calling the order creation interface, inventory query interface, and payment interface. The results of these interface calls constitute the test data for the business process, which will be used for subsequent business process verification and optimization. During the test data production process, the produced test data is collected at pre-defined intervals (e.g., hourly), and the test data scenarios are dynamically updated based on this data. By identifying and adapting to changes in the business process based on the actual produced data, the test data scenarios continuously reflect the latest business status. In this way, the validity and accuracy of test data can be maintained in a constantly changing business environment, thereby improving the adaptability and output effectiveness of the entire data factory. Ultimately, the BPM protocol adapter adapts the generated test data to the BPM system, ensuring that the test data can correctly interface with the business process according to the BPMN specification, supporting the execution and verification of the business process, and completing the entire data production process.

[0059] Combination Figure 2 As shown, another embodiment of the present invention provides a data production system based on a data factory, comprising: The data receiving module is used to acquire business process data and configuration instructions uploaded by the user. The business process data includes the BPMN node semantics, field constraints and process logic of the user's business process. The parameter generation module is used to determine the parameter expression corresponding to the business process based on the configuration instructions. The scenario generation module is used to generate test data scenarios for the business process based on the BPMN node semantics, the field constraints, and the process logic, combined with the parameter expressions. The execution module is used to encapsulate and execute interface calls based on the test data scenario to obtain the test data of the business process; An update module is used to dynamically update the test data scenario according to the test data produced within the preset period during the process of producing the test data.

[0060] The data production system based on the data factory of the present invention has the same advantages over the prior art as the data production method based on the data factory described above, and will not be repeated here.

[0061] Another embodiment of the present invention provides an electronic device including a memory and a processor; The memory is used to store computer programs; The processor is configured to implement the data production method based on a data factory as described above when executing the computer program.

[0062] The electronic device of the present invention has the same advantages over the prior art as the data production method based on the data factory described above, and will not be repeated here.

[0063] In another embodiment of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored on the storage medium, and when the computer program is executed by a processor, the data production method based on a data factory as described above is implemented.

[0064] Alternatively, a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, causes the processor to perform the following operations: Obtain business process data and configuration instructions uploaded by the user. The business process data includes the BPMN node semantics, field constraints, and process logic of the user's business process. Based on the configuration instructions, determine the parameter expression corresponding to the business process; Based on the BPMN node semantics, the field constraints, and the process logic, combined with the parameter expressions, test data scenarios for the business process are generated. The test data scenario is encapsulated and the interface is called to obtain the test data of the business process; wherein, during the production of the test data, the test data scenario is dynamically updated according to the test data produced within the preset period.

[0065] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A data production method based on a data factory, characterized in that, include: Obtain business process data and configuration instructions uploaded by the user. The business process data includes the BPMN node semantics, field constraints, and process logic of the user's business process. Based on the configuration instructions, determine the parameter expression corresponding to the business process; Based on the BPMN node semantics, the field constraints, and the process logic, combined with the parameter expressions, test data scenarios for the business process are generated. The test data scenario is encapsulated and the interface is called to obtain the test data of the business process; wherein, during the production of the test data, the test data scenario is dynamically updated according to the test data produced within the preset period.

2. The data production method based on a data factory according to claim 1, characterized in that, The step of determining the dynamic parameter expression corresponding to the business process according to the configuration instruction includes: Lexical-syntax analysis is performed on the configuration instructions to obtain the database field references, environment variable injections, and service interface call instructions in the configuration instructions; The database field reference, the environment variable injection, and the service interface call instruction are respectively mapped to the first placeholder, the second placeholder, and the third placeholder; Based on the field constraints of the business process, the first placeholder, the second placeholder, and the third placeholder are processed respectively to obtain the real-time field value, the runtime configuration value, and the external return value. The real-time field value, the runtime configuration value, and the external return value are input into a preset expression template to obtain the parameter expression corresponding to the business process.

3. The data production method based on a data factory according to claim 2, characterized in that, The step of generating test data scenarios for the business process based on the BPMN node semantics, the field constraints, and the process logic, combined with the parameter expressions, includes: Construct a directed topology graph of the business process based on the semantics of the BPMN nodes; Based on the field constraints, generate a JSON-Schema fragment for each graph vertex in the directed topology graph; The real-time field value, the runtime configuration value, and the external return value in the parameter expression are injected into the JSON-Schema fragment to obtain a dynamically instantiated template. The execution probability of each edge in the directed topology graph is calculated based on the reinforcement learning model, and the Mock rule tree is determined based on the execution probability. Based on the Mock rule tree and the dynamically instantiated template, the test data scenario for the business process is generated.

4. The data production method based on a data factory according to claim 1, characterized in that, The step of encapsulating the test data scenario and executing the interface call to obtain the test data for the business process includes: The test data scenario is packaged into a task package; The task package is assigned to the execution node in a preset distributed cluster by a work-stealing algorithm. The task package is executed in parallel through the execution node, and the interface call is called to obtain the test data of the business process.

5. The data production method based on a data factory according to claim 4, characterized in that, The step of determining the execution node of the task package in a preset distributed cluster using a work-stealing algorithm and scheduling the task package to the execution node includes: Based on the topology information of the preset distributed cluster, construct a node-core-thread distance matrix; Based on the distance matrix, all nodes in the preset distributed cluster are filtered to determine the candidate execution nodes for the task package; The work-stealing algorithm performs lock-free CAS stealing in the local queue of the candidate execution node until the execution node is determined, and then the task package is scheduled and migrated to the execution node.

6. The data production method based on a data factory according to claim 3, characterized in that, During the process of generating the test data, the test data scenario is dynamically updated according to a preset period based on the test data generated within the preset period, including: The real-time test data generated by the execution node within the preset period is obtained, and the difference measure of the real-time test data is obtained based on the difference between the real-time test data and the preset test data expected value. The difference metric is input into the closed-loop feedback engine to incrementally correct the real-time field value, the runtime configuration value, and the external return value in the parameter expression, thereby obtaining the incrementally corrected parameter expression. Based on the parameter expression after incremental correction, the test data scenario of the business process is incrementally updated to obtain the updated test data scenario; The updated test data scenario is repackaged into a task package, and the interface call is executed again in the next cycle.

7. The data production method based on a data factory according to claim 6, characterized in that, The step of inputting the difference metric into the closed-loop feedback engine to incrementally correct the real-time field value, the runtime configuration value, and the external return value in the parameter expression to obtain the incrementally corrected parameter expression includes: Based on the difference metric, the correction weights of the first placeholder, the second placeholder, and the third placeholder are determined; Based on the correction weights of the first placeholder, the second placeholder, and the third placeholder, weighted incremental updates are performed on the real-time field value, the runtime configuration value, and the external return value, respectively. The updated real-time field value, the updated runtime configuration value, and the updated external return value are then filled back into the parameter expression to form the incrementally corrected parameter expression.

8. The data production method based on a data factory according to claim 6, characterized in that, The scenario of incrementally updating the test data of the business process based on the incrementally corrected parameter expression includes: The incrementally corrected parameter expression is injected into the directed topology graph of the business process. The JSON-Schema fragment of each graph vertex is re-instantiated, and the real-time field value, runtime configuration value, and external return value in the incrementally corrected parameter expression are injected into the re-instantiated JSON-Schema fragment to obtain the updated dynamic instantiation template. Based on the values ​​of the re-instantiated JSON-Schema fragment, the Mock rule tree is partially updated to obtain the partially updated Mock rule tree; The locally updated Mock rule tree and the updated dynamic instantiation template are merged to obtain the updated test data scenario.

9. A data production system based on a data factory, characterized in that, include: The data receiving module is used to acquire business process data and configuration instructions uploaded by the user. The business process data includes the BPMN node semantics, field constraints and process logic of the user's business process. The parameter generation module is used to determine the parameter expression corresponding to the business process based on the configuration instructions. The scenario generation module is used to generate test data scenarios for the business process based on the BPMN node semantics, the field constraints, and the process logic, combined with the parameter expressions. The execution module is used to encapsulate and execute interface calls based on the test data scenario to obtain the test data of the business process; An update module is used to dynamically update the test data scenario according to the test data produced within the preset period during the process of producing the test data.

10. An electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement the data production method based on a data factory as described in claims 1-8 when executing the computer program.

Images