A method and system for constructing dynamic data across the entire securities business process
By employing a full-link dynamic data construction method, the challenges of environmental dependence, cross-database integration, and T+N day data simulation in securities business have been solved, enabling efficient, secure, and flexible application of data construction and improving the efficiency and accuracy of data simulation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-19
- Publication Date
- 2026-05-26
AI Technical Summary
Existing data construction methods in the securities business field suffer from problems such as low reusability due to changes in environmental databases, insufficient demand for cross-database data construction, difficulty in simulating T+N day settlement data, low efficiency of integration testing, and insufficient adaptability.
It adopts a full-link dynamic data construction method, which realizes cross-system and cross-database data construction by building a data construction model, establishing an environment mapping relationship network, assembling the model in a visual interface, parameter-driven modules and execution engines, supporting continuous construction of business data for T+N days, and ensuring data security through permission isolation and monitoring modules.
It improves the utilization and accuracy of data construction, resolves environmental dependency risks, enhances cross-database data integration capabilities, shortens data simulation cycles, improves integration testing efficiency and data sharing flexibility, and adapts to complex business needs.
Smart Images
Figure CN122089462A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of data construction methods, specifically relating to a full-link dynamic data construction method and system for the securities business field. Background Technology
[0002] Currently, various data construction tools and platforms exist across the securities market to support the data construction needs of different systems and complete data addition, deletion, modification, and query operations during data simulation. The main processes of existing technologies are as follows: Figure 1 As shown: (1) For different databases, each system implements atomic design and writing, and executes by specifying the database address; (2) Mainly uses interface to create atomic data and database SQL to create atomic data, respectively to implement operations such as adding, deleting, modifying, and querying; (3) Data construction between multiple systems in the whole link adopts segmented construction, and the input and output of each system are constructed independently; (4) The data construction of the whole business link also adopts segmented construction. To complete the data construction of a business from day T to day T+N, more than N atomic designs and step-by-step operations are adopted, which play a certain role in the data construction of each system.
[0003] With the deepening of market reforms and the continuous advancement of the securities industry, the industry is setting sail towards specialization and differentiation. The trading systems that support the core business of the securities industry are experiencing increasingly diverse business types, more complex business logic, and more frequent system changes and upgrades. Effectively mitigating the system data simulation risks brought about by rapid business development has become an inevitable challenge for all securities firms. Existing data construction methods can generally meet the needs of single-system, single-data construction applications, but in the complex and ever-changing field of securities business, the following problems still exist:
[0004] 1. Due to the constant changes in the environment database, the traditional method of specifying the database for execution makes it difficult to reuse data construction atoms, resulting in reduced utilization.
[0005] 2. The need for cross-database data construction poses challenges to existing segmented construction methods, and the problems of insufficient data correlation and simulation become more apparent;
[0006] 3. In the simulation of transaction business data in the securities field, the requirement to construct full-chain data for T+N day settlement is difficult to achieve in a single-threaded mode, especially in time-sensitive project applications;
[0007] 4. The continuous improvement of integration testing efficiency requires cross-system end-to-end data construction to support data flow verification between multiple systems;
[0008] 5. Existing technologies are insufficient in adaptability to special scenarios, and a modular and parameterized data construction system is needed to improve reusability. Summary of the Invention
[0009] This invention addresses the problems existing in the prior art by providing a method and system for constructing dynamic data across the entire securities business process. It enables dynamic data construction through precise data construction model building, efficient algorithm execution, flexible parameter control, and data sharing modules during data simulation applications in securities business, and can be widely used in securities trading, products, accounts, and other fields.
[0010] To address the above technical problems, this invention provides the following technical solution: A method for constructing dynamic data across the entire securities business process, comprising the following steps:
[0011] S1. Construct a data construction model, which includes atomic operation units and model association relationships. The atomic operation units support adding, deleting, modifying, and querying operations on different types of databases.
[0012] S2. Establish an environment mapping relationship network, and map and associate the data construction model with databases of multiple different environments to achieve environment decoupling;
[0013] S3. Based on business needs, assemble and associate multiple data construction models in a visual interface using flowcharts to form a full-link data construction process across systems and databases.
[0014] S4. Receive external parameter configurations through the parameter-driven module to dynamically adjust the execution logic and data content of the data construction model;
[0015] S5. The execution engine sequentially calls the corresponding atomic operation units according to the full-link data construction process to complete the simulation construction of business data across nodes and dates.
[0016] Furthermore, the aforementioned atomic operation units include interface data generation atoms and database SQL data generation atoms, which are used to construct data through business interfaces and direct database operations, respectively.
[0017] Furthermore, the aforementioned environment mapping network stores the mapping relationship between different environments and database addresses through a configuration table, and dynamically selects the corresponding database connection based on the target environment during execution.
[0018] Furthermore, the aforementioned end-to-end data construction process supports the continuous construction of business data from day T to day T+N. Through node-based modeling and consistent association between models, a dataset spanning multiple days can be generated in a single execution process.
[0019] Furthermore, the aforementioned parameter-driven module supports passing parameters via interface input or configuration files. The parameters include at least one of the following: business type, date range, account identifier, and product code.
[0020] This invention also provides a full-link dynamic data construction system for the securities business field, comprising:
[0021] The model building module is used to create and edit data construction models through a visual interface, supporting the definition of atomic operation units and the association and reference between models;
[0022] The environment management module is used to maintain the environment mapping relationship network and realize the dynamic binding and switching of data construction models with multiple environment databases;
[0023] The process design module is used to assemble multiple data construction models in a flowchart format to form a full-link construction process that spans systems and databases.
[0024] The parameter-driven module is used to receive and parse external parameters and dynamically configure the execution behavior of the data construction model.
[0025] The execution engine module is used to parse and execute the entire construction process, sequentially calling the corresponding atomic operation units to complete the data construction.
[0026] Furthermore, the aforementioned system also includes a permission isolation module, which is used to control data operation permissions for different users or roles, ensuring the security of cross-database data access.
[0027] Furthermore, the aforementioned system also includes a monitoring log module, which records the execution status, exception information, and operation flow of the data construction process, supporting the traceability of the execution process.
[0028] Furthermore, the aforementioned atomic operation unit supports multiple database types, including relational databases, non-relational databases, and distributed databases.
[0029] Furthermore, the aforementioned execution engine module supports concurrent execution and transaction rollback mechanisms to ensure the efficiency and consistency of data construction.
[0030] Compared with the prior art, the beneficial technical effects of the present invention using the above technical solution are as follows:
[0031] This invention solves several data construction and simulation application problems currently faced in the securities business field, and has the following characteristics: 1. Supports environment decoupling and secure data sharing; 2. Supports integrated data construction across databases and nodes; 3. Supports complex T+N business end-to-end data construction and system end-to-end data construction with long call chains, significantly compressing the data simulation construction cycle and ensuring efficient and reliable data utilization; 4. Supports parameter-driven control, making the system flexible and configurable, and agilely responding to changes in requirements; Compared with existing technologies, the advantages created by this invention are as follows:
[0032] (1) It mitigates the application risk of data construction being dependent on the environment by using databases in different environments within the same system.
[0033] With the full establishment and dynamic invocation of the mapping relationship network, the system can flexibly switch environments to execute data construction, improving the utilization rate of data construction and the consistency of data construction in multiple environments. Without this function, data construction personnel would have to repeatedly build data construction models in different environments each time. The models contain environmental attributes and characteristics, which would seriously restrict the utilization rate and success rate of the models. The data construction system of this invention effectively mitigates this risk and opens up a data sharing channel in multiple environments.
[0034] (2) Solve the problem of integrating and constructing data distributed across different types of databases and different service databases, through practical...
[0035] The current flowchart-style association reference model with a visual interface allows for the selection of appropriate data construction strategies to meet specific business needs. It also fully considers data security issues between multiple databases through permission isolation and monitoring and identification control. From the perspective of application promotion, it has also greatly reduced the threshold for business data construction, improved the accuracy of data construction, reduced the generation of dirty data, and played a positive role in database management.
[0036] (3) The data construction of the entire business chain and the entire system chain has been implemented. The entire business chain refers to the fact that most securities businesses are...
[0037] The T+N day settlement process is completed, simulating the efficient operation of the entire business chain. This process is often lengthy and complex, involving fund holdings, contract flow changes, interest settlement, and end-of-day data processing. The data construction of this invention builds a node-based model of the business process and completes the consistency construction between multiple models, such as the consistency association between T-day data and T+N data. The integrated model is executed and implemented in 1 second, forming a complete construction of a cross-day dataset. Each application system in the chain can take what it needs from the dataset and can also transfer data to each other. In this process, the data maintains continuity and the system completes the functional simulation of business data flow.
[0038] (4) Parameter-driven control is the flexible touch of the system, which can sense changes in demand and adapt to dynamic adjustments. It enables the model to adapt to different business needs through external configuration or interface parameter settings without code-level modification. Users can customize the behavior of the model according to their own needs, thereby obtaining a personalized user experience and enabling the model to have polymorphic applications. This is of great significance for the continuous maintenance and promotion of this method and system. Attached Figure Description
[0039] Figure 1 This is a simplified diagram of the existing technical architecture.
[0040] Figure 2 This is the logical architecture diagram of TDCP, a full-link dynamic data construction system used in the securities business field.
[0041] Figure 3 This is a logic diagram for model construction.
[0042] Figure 4 This is a simplified diagram of the demand matrix.
[0043] Figure 5 This is a parameter-driven and integrated application framework diagram.
[0044] Figure 6 This is a schematic diagram of transaction business applications. Detailed Implementation
[0045] To better understand the technical content of the present invention, specific embodiments are described below in conjunction with the accompanying drawings.
[0046] In this invention, various aspects of the invention are described with reference to the accompanying drawings, in which numerous illustrative embodiments are shown. Embodiments of the invention are not limited to those depicted in the drawings. It should be understood that the invention is implemented through any of the various concepts and embodiments described above, as well as the concepts and embodiments described in detail below, because the concepts and embodiments disclosed herein are not limited to any particular implementation. Furthermore, some aspects of the invention disclosed may be used alone or in any suitable combination with other aspects of the invention disclosed.
[0047] This invention provides a method and system for constructing dynamic data across the entire securities business process, such as... Figure 2 The TDCP logical architecture diagram shown demonstrates how, in the data simulation application of securities business, dynamic data construction is achieved through the precise construction of data construction models, efficient algorithm execution, flexible parameter control, and data sharing modules. It is widely used in securities trading, products, accounts, and other fields.
[0048] In this embodiment, a method for constructing dynamic data across the entire securities business process is provided, characterized by the following steps:
[0049] S1. Construct a data construction model, which includes atomic operation units and model association relationships. The atomic operation units support adding, deleting, modifying, and querying operations on different types of databases.
[0050] S2. Establish an environment mapping relationship network, and map and associate the data construction model with databases of multiple different environments to achieve environment decoupling;
[0051] S3. Based on business needs, assemble and associate multiple data construction models in a visual interface using flowcharts to form a full-link data construction process across systems and databases.
[0052] S4. Receive external parameter configurations through the parameter-driven module to dynamically adjust the execution logic and data content of the data construction model;
[0053] S5. The execution engine sequentially calls the corresponding atomic operation units according to the full-link data construction process to complete the simulation construction of business data across nodes and dates.
[0054] As a preferred approach, the data construction model and strategy selection mainly include three steps: content selection, lineage analysis, and model determination. Figure 3 As shown:
[0055] (1) Content selection: By obtaining the after-hours data of the production business of the trading system and the request and response messages of the interface traffic, and then combining the pseudocode logic of the interface itself to dynamically capture the input and output parameters, we comprehensively analyze the account type, market category, trading direction, order attributes, etc. that the data construction requirements themselves will involve, and select and confirm the target key tables and field data of the data construction.
[0056] (2) Lineage analysis: sort out the lineage relationships of related fields, tables, nodes, etc. that the target content data depends on, and obtain the enumeration value list of core fields from the pseudocode maintenance system to establish a lineage relationship graph of the relevant table fields of a certain business, so as to provide accurate data support for model construction. Finally, flexible dynamic control parameters are injected based on lineage relationships, which are multiple combination application scenarios of enabling and disabling dynamic parameters for relationship adaptation.
[0057] (3) Determine the model, select different data construction generation strategies, and generate different data construction models. The cross-database strategy data construction relationship sorting includes factors of different types of databases. Through the visual model construction operation interface, different atoms are connected in series or parallel according to time sequence. Each atom retains database attributes, environment attributes, and parameter attributes. Each atom is independent but related and integrated. The business full-link strategy includes the reference parameter passing relationship between each atom. The business data on day T and the business on day T+N are kept consistent in terms of initialization date, transaction code, account, etc. by referencing global variables. The calculation logic of continuous business changes is maintained in terms of money, securities, interest, frozen amount, etc. For example, the construction value of the amount of new stock subscription payment on day T+3 is the number of subscriptions on day T+1 * code issuance price. The ordinary strategy adopts atomic SQL creation and atomic assembly. In addition, in the process of building and maintaining the data construction model, a sensitive monitoring module is set up. The sensitive field library is used to perform security monitoring and review of the fields in the data construction model to ensure the data security of the data construction. Finally, a real-time, accurate, and secure data construction model is formed, which effectively solves the different data construction needs of different securities businesses.
[0058] Model execution efficiency is crucial for its application. The complexity of securities business architecture determines the complexity of its database logic relationships. Furthermore, securities business often involves massive amounts of data, including basic securities information, transaction orders, transaction records, market data, investor information, holding funds, and permission flags. This data is constantly being generated and updated, requiring reliable and efficient execution capabilities to construct the model. Analyzing the data characteristics, which are distributed across different nodes and users, the model employs a basic selection sort algorithm to determine the master database, which has a time complexity of [missing information]. To improve the performance of multi-table join queries, the Block Nested-Loop Join algorithm was introduced. This algorithm stores rows / result sets from the outer loop in a join buffer, and compares each row of the inner loop with all records in the buffer, thus reducing the number of loop iterations. If the sum of the column lengths of tables t1 and t2 participating in the join is s, and c is the number of combinations, then the number of times table t3 is scanned is: The number of scans of t3 decreases as join_buffer_size increases until the join buffer can accommodate all combinations of t1 and t2. This ultimately reduces memory and CPU consumption during large-scale data processing, continuously improving query efficiency. It also enables efficient execution and output of the data construction model under numerous conditional controls, including related shareholder accounts, asset accounts, order tables, fund tables, and contract tables.
[0059] 3. Parameter-driven control: Parameter-driven control is the capability injected into this invention to solve the problem of flexible reuse of data construction. Addressing the issues of low success rate, low reuse rate, and limited application scenarios caused by various factors such as environmental dependence, strong binding to data construction scenarios, data asynchrony between databases, and obstacles to integrated applications, the parameter-driven control module mainly includes the following parts:
[0060] (1) Decoupling from the environment is achieved through parameter control architecture design. Since the inherent value of data construction determines that it needs to provide services in different environments, such as test environment, simulation environment, production environment, etc., in order to decouple the binding relationship between data construction and environment, the production environment database structure of the system is used as the basis to establish the corresponding relationship between other environments in the data construction system, and environment variable identifiers ${env}, ${env1}, ${env2}, etc. are established in the application space of different systems. During the execution of the data construction model, the corresponding whole set of environment database users and service addresses of each table node are dynamically matched according to the switched environment variable identifiers to complete the corresponding data construction behavior. In addition, for the situation that the database cannot achieve real-time synchronization of data in some environments, the system automatically enables the Database Link database execution mode according to the status of the ${dblink_on} and ${dblink_off} parameters in each environment, automatically builds multiple assembly methods, and uses the internal algorithm of the model to obtain the optimal path of the execution mode, thereby decoupling the data construction from the environment. The independent development, maintenance and evolution of the data construction system, multi-environment deployment and switching applications greatly improve the reusability of the data construction model plugin.
[0061] (2) Special scenario applications are achieved through personalized parameter control. The parameter control module set up in this system plays an important role in dealing with special scenarios of securities trading. It is mentioned in the model construction that the model itself reserves parameter control placeholders, as shown in the following formula: where branch_no is the business department number, real_seat_no is the customer's real seat, and real_seat_no is the output of the user-defined function (there are multiple judgment logics for the customer's real seat, so the user-defined function is used as a common function in the system):
[0062] As illustrated in the formula above, parameters are dynamically combined with the data construction model. During the execution of the API interface called by the model, the ${op_type} parameter or the [Mode Control] in the visual interface is used to control whether the above regular expression function is enabled. If it is not enabled, it will be executed in the normal mode. The execution in each mode is isolated from each other and does not affect each other, solving the problem of multi-terminal shared scenarios. Furthermore, the ${op_branch_no}, ${op_seat_no} parameters or the [Personalized Parameters] in the visual interface are used to control whether the parameters in the above regular expression function are replaced with personalized values. Through the above parameter control logic, the data construction model has richer application scenarios and value. This is crucial for the complex needs of securities business. One example of parameter control behavior mentioned above can achieve precise control of the customer scope in high-frequency trading business, such as specifying a branch, a seat, a market, or a quote. See Figure 4 Simplified diagram of the demand matrix. The data construction in this invention efficiently and agilely adapts to various such demands by adopting personalized parameter control.
[0063] (3) Parameter-driven settings enable the system to have stronger integration capabilities, see Figure 5 The parameter-driven and integrated application framework diagram, combined with the above-mentioned environment decoupling and personalized parameter control, enables data construction in automated testing, performance testing, and integration with other systems. By configuring different parameters, it can generate data constructions for various test scenarios, thereby improving scenario test coverage. Furthermore, the parameter control module can easily achieve integration and calling with other systems. By configuring parameters to adjust the integration and calling behavior, the interoperability of the data construction system is guaranteed.
[0064] In the end-to-end application of securities business, characterized by long links and complex data flow, the efficiency and quality of data construction in the integrated application of multiple systems directly affect the coverage, accuracy, and execution speed of tests. This invention addresses this bottleneck by creating a highly efficient and available data construction model using a link architecture that focuses on the real-time performance, continuity, accuracy, and reusability of end-to-end data. The application is mainly divided into stages such as data construction model building, data construction model output, and application system usage. Figure 6 The following is a schematic diagram of the transaction business application:
[0065] (1) The data construction model fully considers the data flow relationship between the various systems during the construction process. The account model is associated with product information to ensure that an account needs to have a certain product code holding and the permissions of the business. The transaction data model is associated with accounts, products and market data to ensure that the transaction data is associated with and corresponds to them. The asset analysis model is associated with upper-level accounts and transaction settlement data to ensure the continuity and accuracy of asset analysis data. The construction scenario is flexibly associated and integrated to form an efficient and usable data simulation model system.
[0066] (2) Data construction model output: The data of each model is completely output through the data processing and data sharing platform. While ensuring data security, the construction of related data is also completed, forming a data set with consistency and accuracy that can be used by different systems.
[0067] (3) The system is used for collaborative full-link application. The system consumes data according to the business flow order to construct models and output data. The simulation verification of the real business flow is completed in a short time, and finally the end-to-end data verification process is realized.
[0068] The implementation and application of the above model can significantly improve the testing efficiency of multi-system end-to-end integrated applications in securities business. It not only ensures the continuity and accuracy of data, but also improves the reusability of data through data modeling and sharing mechanisms, reduces repetitive work, and improves the overall efficiency and quality of testing.
[0069] While the present invention has been described above with reference to preferred embodiments, it is not intended to limit the invention. Those skilled in the art can make various modifications and refinements without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention shall be determined by the claims.
Claims
1. A full-link dynamic data construction method for the field of securities business, characterized in that, The steps are as follows: S1, a data construction model is constructed, the model includes an atomic operation unit and a model association relationship, the atomic operation unit supports increment, deletion, modification and query operations on different types of databases; S2, an environment mapping relationship network is established, the data construction model is mapped and associated with databases in multiple different environments, and environment decoupling is realized; S3, according to business requirements, a plurality of data construction models are assembled and associated in a visual interface through a flowchart mode to form a full-link data construction process across systems and databases; S4, an external parameter configuration is received through a parameter driving module, and the execution logic and data content of the data construction model are dynamically adjusted; S5, an execution engine sequentially calls corresponding atomic operation units according to the full-link data construction process to complete business data simulation construction across nodes and dates.
2. The full-link dynamic data construction method for the field of securities business according to claim 1, characterized in that, The atomic operation unit includes an interface atomic number and a database SQL atomic number, which are respectively used to realize data construction through a business interface and direct database operation.
3. The full-link dynamic data construction method for the field of securities business according to claim 1, characterized in that, The environment mapping relationship network stores the mapping relationship between different environments and database addresses through a configuration table, and dynamically selects the corresponding database connection according to the target environment during execution.
4. The full-link dynamic data construction method for the field of securities business according to claim 1, characterized in that, The full-link data construction process supports continuous construction of business data from T day to T+N day, and generates a multi-day data set in a single execution process through node modeling and consistent association between models.
5. The full-link dynamic data construction method for the field of securities business according to claim 1, characterized in that, The parameter driving module supports parameter transmission through interface input or configuration file, and the parameters include at least one of business type, date range, account identifier and product code.
6. A full-link dynamic data construction system for the securities business field, characterized by, It includes: A model construction module for creating and editing data construction models through a visual interface, supporting definition of atomic operation units and association between models; An environment management module for maintaining an environment mapping relationship network, realizing dynamic binding and switching of data construction models and multi-environment databases; A process design module for assembling a plurality of data construction models in a flowchart mode to form a full-link construction process across systems and databases; A parameter driving module for receiving and analyzing external parameters to dynamically configure the execution behavior of the data construction model; An execution engine module for parsing and executing the full-link construction process to sequentially call corresponding atomic operation units to complete data construction.
7. The system of claim 6, wherein, The system further includes a permission isolation module for controlling data operation permissions of different users or roles to ensure the security of cross-database data access.
8. The system of claim 6, wherein, The system further includes a monitoring log module for recording the execution state, exception information and operation sequence of the data construction process to support traceability of the execution process.
9. The system of claim 6, wherein, The atomic operation unit supports multiple database types, including relational databases, non-relational databases and distributed databases.
10. The system of claim 6, wherein, The execution engine module supports concurrent execution and transaction rollback mechanism to ensure the efficiency and consistency of data construction.