Financial data processing system and method

Abstract

The invention relates to a financial data processing system and method, and relates to the technical field of data processing. The system is characterized in that a server is used for dividing a target task in a task request of a client into a plurality of sub-tasks; determining a first metadata block of each sub-task in the plurality of sub-tasks; according to the first metadata block, processing a subtask corresponding to the first metadata block to obtain a first business data block; respectively encrypting each first metadata block and each first business data block to obtain a plurality of second metadata blocks and a plurality of second business data blocks, and transmitting the second metadata blocks and the second business data blocks to the client; the client is used for selecting a target operation plug-in associated with the target system type and selecting a target database plug-in associated with the target database type so as to write the second business data block into a local target database based on the target operation plug-in and the target database plug-in; and the management and control end is used for monitoring the operation states of a plurality of preset nodes in the server and the client in real time.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a financial data processing system and method. Background Technology

[0002] With the rapid development of the national capital market and the continued deepening of digital transformation in financial institutions, the volume of data in the financial sector is experiencing exponential growth. In multiple financial markets, including bonds, stocks, futures, and funds, relevant financial institutions and information services provide various types of data such as market information, news, research reports, in-depth analysis, and pricing benchmarks. Investors utilize this financial data for investment decisions and risk management, highlighting its paramount importance. Beyond its high importance, the diverse market and sectoral nature of financial data presents characteristics of large volume and high accuracy requirements. In these business scenarios, the demand for building a secure, efficient, and stable data synchronization mechanism is increasingly strong, serving as an essential path to unlocking the value of financial data and improving the quality and efficiency of financial data services.

[0003] In related technologies, data services in the financial sector typically involve transmitting file streams. After receiving the stream, clients manually process the data, performing tasks such as format conversion, validation, and quality checks. This file stream-based data service lacks simple and effective data file download and parsing capabilities. When users manually select and download data files, specialized software or tools are required to parse the data, converting it to the client's desired format before storing it in a relational database or spreadsheet. This process is not only time-consuming and labor-intensive, but also inefficient and complex, and prone to errors, thus affecting data integrity and accuracy. Summary of the Invention

[0004] In view of this, the present invention provides a financial data processing system and method to at least solve the problems of low operational efficiency, complex operation, and low accuracy of data services in related technologies. The technical solution of the present invention is as follows.

[0005] According to a first aspect of the present invention, a financial data processing system is provided. The system includes: a client, a server, and a control terminal. The client includes multiple operating system plugins and multiple database plugins. A synchronous bidirectional channel and an asynchronous unidirectional channel are established between the client and the server. The server is configured to: receive a task request sent by the client; divide the target task in the task request into multiple subtasks; determine a first metadata block for each subtask; process the subtask corresponding to the first metadata block according to the first metadata block to obtain a first business data block, where each first business data block and its corresponding first metadata block include the same target asset identifier and task sequence number; encrypt each first metadata block and each first business data block to obtain multiple second metadata blocks and multiple second business data blocks; and send each second metadata block to the client in real time through the synchronous bidirectional channel, as well as data represented by task sequence number. The process involves sequentially transmitting each second business data block to the client via an asynchronous unidirectional channel; the target asset identifier represents the data source resource ownership; the client is configured to: determine the target system type of the client's operating system and the target database type to which the second business data is to be stored; select the target operation plugin associated with the target system type from multiple operation plugins, and select the target database plugin associated with the target database type from multiple database plugins; during client runtime, automatically load the target operation plugin into a preset program interface to obtain the target program interface; call the target program interface to automatically create and verify the local table structure based on the second data block, and verify the received second business data block based on the local table structure; and write the successfully verified second business data block into the target database based on the target database plugin; the management terminal is configured to: monitor the running status of multiple preset nodes in the server and client in real time.

[0006] The database side includes multiple local databases corresponding to the client. The target database is a local database.

[0007] In the above implementation, the second metadata block is sent to the client first, and then each second business data block is sent to the client.

[0008] The aforementioned target operation plugin is used to shield the underlying differences between different operating systems, allowing financial data transmission business logic to be free from environmental details.

[0009] Specifically, common operational functions across different operating systems (including those with identical tools, processes, and steps for performing the same function) are encapsulated in a pre-defined program interface. Different operational functions between different operating systems (including those with different tools, processes, or steps for performing the same function) are encapsulated in the corresponding target operation plugins. In some implementations, the target task in a task request is divided into multiple subtasks based on task nodes and schedulable traffic according to task type. Different subtasks correspond to different task nodes, and these task nodes belong to pre-defined nodes, facilitating monitoring of the operational status of each task node by the management and control end, thereby facilitating the location of faults in each task node.

[0010] As one implementation, the server includes: a data source management module, used to determine the attribute information of the data source executing subtasks based on the first metadata block, and, based on the attribute information, identify the resources of the data source to obtain the asset identifier of the resources used by the subtasks; a data management module, used to process client task requests, and to split the tasks in the task requests into multiple subtasks; a data storage module, including multiple data execution engines and multiple databases of different types, used to call the target data execution engine among the multiple data execution engines to access and obtain business data in the multiple databases associated with the server; a data assembly module, used to embed the preset information associated with the first metadata block into the corresponding first business data block, the preset information including an asset watermark representing the asset identifier, a structure version number, and a task sequence number representing a global continuous sequence number; and to generate tasks, submit task submissions, update task status, and update task metrics; and an access control module, used to authenticate the access task requests, dynamically encrypt the access link of the task requests, and perform frame extraction verification on the second metadata block and the second business data in the transmission link.

[0011] The resources mentioned above include a data execution engine and a database.

[0012] As another implementation method, the control terminal is specifically configured to: track the asset transmission path of business data according to the asset identifier; generate a visualized asset full-link diagram based on the asset transmission path; and generate a visualized data full-link diagram based on multiple preset nodes, asset transmission paths and operating status; the data full-link diagram is a topology structure.

[0013] As another implementation, the client includes multiple connection pools; the database plugin includes a syntax conversion plugin and a connection pool plugin; the client is specifically configured to: call the target syntax conversion plugin to perform syntax conversion on the second business data block; determine the target connection pool that is compatible with the target database type based on the connection pool plugin; and store the syntax-converted second business data block in the target database based on the target connection pool.

[0014] As another implementation, asset identifiers are distributed in the form of watermarks in the second metadata block and the second business data block. The client is configured to: determine whether the asset identifier in the received second business data block is completely watermarked; if it is determined that the asset identifier in the received second business data block is not completely watermarked, send a retransmission request to the server to retransmit the second business data block with the incomplete watermark, and send the watermark information of the incomplete watermark to the management end, so that the management end can determine the asset flow path of the second business data block in the asset full-link diagram based on the watermark information, and highlight the asset flow path.

[0015] As another implementation, the client is also configured to write the second business data block that failed verification to the exception database and send the verification failure result to the management terminal; the management terminal is also configured to receive the verification failure result, determine the target link node in the data full link diagram, and highlight the target link node.

[0016] As another implementation method, the control terminal is also configured to: determine the data transmission volume and schedulable network resource volume of the transmission link, and adjust the data transmission adoption rate.

[0017] As another implementation, the control terminal is also configured to: determine whether the multiple task sequence numbers of all received second metadata blocks and second business data blocks are consecutive and complete; if they are not consecutive or complete, send a retransmission request to the server to retransmit the non-consecutive or incomplete second metadata blocks and / or non-consecutive or incomplete second business data blocks, and send the non-consecutive or incomplete task sequence number information to the control terminal; and determine whether the task sequence number of each received second business data block is consecutive with the other task sequence numbers of other previously received second business data blocks; if the task sequence numbers are not consecutive, send a retransmission request to the server to retransmit the second business data blocks with non-consecutive task sequence numbers, and send the non-consecutive task sequence number information to the control terminal.

[0018] As another implementation method, the access control module is specifically configured as follows: sending an authentication command to the client, responding to the user input information returned by the client, and determining whether the user input information matches the authentication command; and, on the access link, when accessing the server's database, sending a dynamic code to the access node, and determining whether the dynamic code returned by the access node within a preset time is correct.

[0019] According to a second aspect of the present invention, a financial data processing method is provided. The method includes: invoking a server to perform the following operations: receiving a task request sent by a client; dividing a target task in the task request into multiple subtasks; determining a first metadata block for each of the multiple subtasks; processing the subtask corresponding to the first metadata block according to the first metadata block to obtain a first business data block, wherein each first business data block and the corresponding first metadata block include the same target asset identifier and the task sequence number; encrypting each of the first metadata blocks and each of the first business data blocks to obtain multiple second metadata blocks and multiple second business data blocks; sending each of the second metadata blocks to the client in real time through the synchronous bidirectional channel; and sequentially transmitting each of the second business data blocks to the client in blocks according to the order represented by the task sequence number. The client is invoked to perform the following operations: determine the target system type of the operating system executed by the client and the target database type to which the second business data is to be stored; select the target operation plugin associated with the target system type from multiple operation plugins, and select the target database plugin associated with the target database type from multiple database plugins; automatically load the target operation plugin into a preset program interface when the client is running to obtain the target program interface; call the target program interface to automatically create and verify the local table structure according to the second metadata block, and verify the received second business data block according to the local table structure; write the successfully verified second business data block into the target database based on the target database plugin; and invoke the management terminal to perform the following operations: monitor the running status of multiple preset nodes in the server and the client in real time.

[0020] According to a third aspect of the present invention, a financial data processing system is provided, which is configured to perform a financial data processing method as described in the second aspect and any possible implementation thereof.

[0021] According to a fourth aspect of the present invention, a code generation apparatus is provided, comprising: a processor and a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement a financial data processing method as described in the second aspect and any possible implementation thereof.

[0022] According to a fifth aspect of the present invention, a computer-readable storage medium is provided, on which instructions are stored, such that when the instructions in the computer-readable storage medium are executed by a processor of a code generation device, the code generation device is able to perform a financial data processing method as described in the second aspect and any possible implementation thereof.

[0023] According to a sixth aspect of the present disclosure, a computer program product is provided, the computer program product including computer instructions, which, when executed on a service terminal, cause the service terminal to perform the financial data processing method of the second aspect and any possible implementation thereof described above.

[0024] The technical solutions provided by the embodiments of this invention offer at least the following beneficial effects: This application proposes a system operation mode with a separate design for the client, server, and management terminal. The server provides basic services such as acquiring transmission data, preparing verification data, chunked transmission, and security mechanisms. Supported by operating system and database plugins, the client receives secure business data and verification data transmitted from the server, automatically completing file download and parsing, business data verification, and data storage in the client-side database without client intervention. This efficiently solves the problems of data transmission landing and verification, and can autonomously adapt to complex client-side operating environments. Simultaneously, the management terminal enables real-time monitoring of the data transmission service, facilitating rapid location of transmission anomalies.

[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0027] Figure 1 This is a framework diagram of a financial data processing system according to an exemplary embodiment;

[0028] Figure 2 This is a schematic diagram of a financial data processing device according to an exemplary embodiment. Detailed Implementation

[0029] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0030] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0031] Before providing a detailed introduction to the financial data processing system provided in the embodiments of this application, let's first briefly introduce the application scenarios involved in the embodiments of this application.

[0032] With the rapid development of the national capital market and the continued deepening of digital transformation in financial institutions, the volume of data in the financial sector is experiencing exponential growth. In various financial markets such as bonds, stocks, futures, and funds, relevant financial institutions and information services provide the market with a variety of data, including market information, news, research reports, in-depth analysis, and pricing benchmarks. Investors utilize this financial data for investment decisions and risk management, making the importance of data self-evident. Besides its high importance, financial data across multiple markets and sectors is characterized by its large volume and high accuracy requirements. In these business scenarios, the demand for building a secure, efficient, and stable data synchronization mechanism is increasingly strong, as it is essential to unlock the value of financial data and improve the quality and efficiency of financial services.

[0033] Currently, there are three main data service models in the financial sector. The first is through financial information terminals provided by various financial institutions or information networks. These terminals serve as platforms for providing financial information and services, allowing users to access visualized data such as market quotes, data, and analyses from various financial markets. The second is securities market trading system services, which provide front-end application systems. These systems typically include modules for market display, order placement, and account management, helping financial institutions achieve automated trading management. The third is through online financial information services, relying on online media platforms such as websites, public accounts, and Douyin (TikTok), primarily providing financial information through articles and videos.

[0034] In addition, some institutions have begun to explore the construction of data transmission systems for the financial sector to achieve diversified data utilization. This will promote the healthy development of the financial market while meeting user needs, making financial information data a powerful engine for industry innovation and investment decisions. However, existing solutions generally suffer from problems such as low operational efficiency, insufficient customization, weak security capabilities, and opaque data service processes.

[0035] Research has revealed the following main problems in the relevant technologies.

[0036] First, the functionality is complex and the solution is inefficient. Current information service models and existing dedicated data transmission primarily transmit data file streams, lacking simple and effective data file download and parsing capabilities. If users manually select and download data files, specialized software or tools are needed to parse the data, converting it to the customer's required format before storing it in a relational database or spreadsheet. This process is not only time-consuming and labor-intensive, inefficient, and complex, but also prone to errors, affecting data integrity and accuracy. Some solutions deploy a service application locally to receive and process data files; however, any changes require the customer to redeploy the latest service version, resulting in a complex operational mechanism and insufficient customer service friendliness. Furthermore, existing solutions typically store data files on local hard drives or removable storage devices. These devices have limited capacity, making it difficult to store large amounts of data files, and users need to manage the directory structure and naming rules for easy retrieval and use. These tasks increase the user's burden and reduce data availability and security.

[0037] Secondly, there is insufficient customization, particularly in support for direct data connection and multi-terminal operation. As mentioned earlier, users typically can only download files. If synchronization to a database is required, the user must manually read and parse the file before connecting to the database for writing. Some solutions explore directly writing data to the customer's database, but any changes to this information necessitate the development of a new service version and redeployment on the customer's local machine, failing to achieve flexible and seamless switching. Furthermore, the diversity of customer database types and the variations in system operating terminals make ensuring a high degree of adaptability in data transmission service models and meeting diverse customer needs crucial. However, the lack of customization in existing solutions significantly hinders the release of data value and fails to fully realize the potential and role of data.

[0038] Third, security protection capabilities are low, and data leakage and tampering are prone to occur during transmission via the internet. In the information society, the internet, as the main channel for data and information transmission, plays a crucial role in promoting data flow. In the financial sector, due to the sensitivity, importance, and confidentiality of data, stronger security protection capabilities are required. Most existing solutions only provide user login authentication, lacking sufficient security protection during data transmission. This makes them susceptible to interception and subsequent reproduction of the original sensitive data, as well as the transmission of tampered data to the client after interception. While existing financial institutions prioritize building security protection capabilities for transaction terminals, the security protection capabilities for data transmission devices remain insufficient.

[0039] Fourth, the data service process is opaque, with insufficient visualization, monitoring, and handling capabilities. The entire financial data service process covers multiple stages, including data location, data computation, data assembly, data transmission, data reception, and data storage. The handling models and technical solutions for each stage differ significantly, and the entire data service process currently remains opaque, with low levels of monitoring and handling capabilities. If data is transmitted normally, existing solutions can function normally. However, if data loss or transmission failures occur during the service process, it significantly hinders the identification of the root cause and the implementation of technical solutions, making it impossible to support rapid problem identification and efficient handling.

[0040] Fifth, data is a critical asset for companies, and existing solutions lack the ability to manage and control this asset. In the information society, defining, inventorying, and planning disorganized data resources, providing a globally unified data asset portal, developing and authorizing rich, efficient, and secure data services, and building a data asset management system with clear themes, comprehensive services, and well-defined responsibilities are key areas that enterprises need to focus on. Data transmission devices, as a crucial channel for releasing the value of data assets, currently lack the definition of data assets and continuous tracking of their value release, failing to facilitate refined and process-oriented management of data assets.

[0041] Therefore, to address the aforementioned issues, this application proposes a financial data processing system. This system employs a separate operating model with client, server, and control terminals. The server provides basic services such as acquiring transmission data, preparing verification data, chunked transmission, and security mechanisms. Supported by operating system and database plugins, the client receives secure business data and verification data from the server, automatically completing file download and parsing, business data verification, and data storage in the client-side database. No client intervention is required, efficiently solving the challenges of data transmission and verification, and enabling autonomous adaptation to complex client-side operating environments. Simultaneously, the control terminal allows for real-time monitoring of the data transmission service, facilitating rapid identification of transmission anomalies.

[0042] For ease of understanding, the financial data processing method provided in this application will be described in detail below with reference to the accompanying drawings.

[0043] Figure 1 This is a schematic block diagram illustrating a financial data processing system according to an exemplary embodiment. The system includes a client 1, a server 2, and a control terminal 3.

[0044] Client 1 includes multiple operating system plugins and multiple database plugins; synchronous bidirectional channels and asynchronous unidirectional channels are set up between client 1 and server 2.

[0045] Server 2 is configured to: receive task requests sent by client 1; divide the target task in the task request into multiple subtasks; determine the first metadata block for each subtask; process the subtask corresponding to the first metadata block according to the first metadata block to obtain a first business data block, where each first business data block and its corresponding first metadata block include the same target asset identifier and task sequence number; encrypt each first metadata block and each first business data block to obtain multiple second metadata blocks and multiple second business data blocks; send each second metadata block to client 1 in real time through a synchronous bidirectional channel, and sequentially transmit each second business data block to client 1 in blocks through an asynchronous unidirectional channel according to the order represented by the task sequence number; the target asset identifier represents the data source resource ownership.

[0046] Client 1 is configured to: determine the target system type of the operating system to be executed by Client 1 and the target database type to which the second business data is to be stored; select the target operation plugin associated with the target system type from multiple operation plugins, and select the target database plugin associated with the target database type from multiple database plugins; automatically load the target operation plugin into the preset program interface when Client 1 is running to obtain the target program interface; call the target program interface to automatically create and verify the local table structure based on the second metadata block, and verify the received second business data block based on the local table structure; and write the successfully verified second business data block into the target database based on the target database plugin.

[0047] The control terminal 33 is configured to monitor the running status of multiple preset nodes in server 2 and client 1 in real time.

[0048] The database side includes multiple local databases corresponding to client 1. The target database is a local database.

[0049] In the above implementation, the second metadata block is sent to client 1 first, and then each second service data block is sent to client 1.

[0050] The aforementioned target operation plugin is used to shield the underlying differences between different operating systems, allowing financial data transmission business logic to be free from environmental details.

[0051] Specifically, common operating functions across different operating systems (including those that perform the same function using the same tools, procedures, and steps) are encapsulated in a pre-defined program interface. Different operating functions between different operating systems (including those that perform the same function using different tools, procedures, or steps) are encapsulated in the corresponding target operating plugins.

[0052] In some implementations, the target task in a task request is divided into multiple subtasks based on task nodes and schedulable traffic according to task type. Different subtasks correspond to different task nodes, and different task nodes belong to preset nodes, which facilitates the monitoring of the operating status of each task node by the control terminal 3, thereby facilitating the location of faults in each task node.

[0053] The system described above implements a system operation mode with separate design for client 1, server 2, and management / control terminal 3. The server provides basic services such as acquiring transmission data, preparing verification data, chunked transmission, and security mechanisms. Client 1, supported by operating system and database plugins, receives secure business data and verification data transmitted from the server, automatically completes file download and parsing, business data verification, and data storage in the client-side database without client intervention. This efficiently solves the problems of data transmission and verification, and can autonomously adapt to complex client-side operating environments. Simultaneously, the management / control terminal 3 enables real-time monitoring of the data transmission service, facilitating rapid location of transmission anomalies.

[0054] Control terminal 3 is also known as the management terminal.

[0055] In one implementation, the server 2 includes the following modules.

[0056] The data source control module 21 is used to determine the attribute information of the data source that executes the subtask according to the first metadata block, and, based on the attribute information, identify the resources of the data source and obtain the asset identifier of the resources used by the subtask.

[0057] The data management module 22 is used to process the task requests from client 1, and to split the tasks in the task requests into multiple sub-tasks.

[0058] The data storage module 23 includes multiple data execution engines and multiple databases of different types; it is used to call the target data execution engine among the multiple data execution engines to access and retrieve business data from the multiple databases associated with the server 2.

[0059] The data assembly module 24 is used to embed the preset information associated with the first metadata block into the corresponding first business data block. The preset information includes an asset watermark representing the asset identifier, a structure version number, and a task number representing a global continuous sequence number; as well as generating tasks, submitting task submissions, updating task status, and updating task indicators.

[0060] The access control module 25 is used to authenticate the access task requests, dynamically encrypt the access link of the task requests, and perform frame extraction verification on the second metadata block and the second service data in the transmission link.

[0061] The resources mentioned above include a data execution engine and a database.

[0062] In the above implementation, the server 2 consists of five modules: data source management module 21, data management module 22, data storage module 23, data assembly module 24, and access control module 25.

[0063] The data source management module 21 provides two services: metadata and data source. Based on a layered design, the metadata service provides attribute information of the data source through metadata, supports resource identification, evaluation and tracking of changes of data source data, thereby enabling simple and efficient management of a large amount of networked data source data; the data source service provides external data source request services and internal data source management services.

[0064] The data management module 22 provides three main types of module services: the invocation module, the service management module, and the asset access control module. The invocation module receives data retrieval requests from clients and forwards the information to the service management module, performing four main functions: workbench historical job synchronization, job status management, log management, and task allocation. The asset access control module provides unified invocation functions for access management and security control for the service management module and client identity security services. Through this multi-level design scheme, based on task decomposition, the insufficient customization issue in problem two can be effectively addressed.

[0065] Data storage module 23 is the data storage part inside the financial institution. It supports unified management and control of various types of data storage devices such as Oracle, MySQL, Hive, DB2, and Ignite, and can retrieve data from data storage devices by flexibly calling data execution engines such as Kettle, Sqoop, and DataX.

[0066] The data assembly module 24 comprises five parts: task service, task executor, task generation module, task scheduling module, and result output. The task service receives service requests from the calling modules within the service management module and promptly reports heartbeat information to the service management module, sending task calls to the task executor. The task executor initiates specific task calls to the task scheduling module. The task scheduling module manages task execution through the task scheduler and provides services such as task generation, task submission, task status updates, and task metric updates. Task generation relies on related services from the task generation module, and the task scheduling module achieves the lowest-level data manipulation by calling the execution engine service capabilities in the data storage module 23. The task generation module includes a task job generator and task job context management, providing more refined and accurate task control. The result output module assembles data into messages and corresponding data files, interacting with external systems through the calling modules within the data management module. The layered design and combined calling of the data source management module 21, data management module 22, data storage module 23, and data assembly module 24 effectively address the complex functional operations raised in Question 1 and provide a technical service foundation for data asset management in Question 5.

[0067] The access control and verification module 25 provides security control services for access requests and output content. It offers password verification, verification code verification, and certificate verification functions for access task requests. Based on dynamic tokens, it supports dynamic encryption of the access link, enhancing security control capabilities. It also provides message encryption services for the transmission link and data file encryption services for the transmitted content, ensuring the security of the transmission link. Furthermore, it provides frame extraction verification services, which can sensitively identify whether there is eavesdropping or message interception and forgery in the outgoing link. The coordinated use of multiple security solutions effectively addresses the problem of low security protection capabilities in the third area.

[0068] As another implementation method, the control terminal 3 is specifically configured to: track the asset transmission path of business data according to the asset identifier; generate a visualized asset full-link diagram according to the asset transmission path; and generate a visualized data full-link diagram according to multiple preset nodes, asset transmission paths and operating status; the data full-link diagram is a topology structure.

[0069] In another implementation, client 1 includes multiple connection pools; the database plugin includes a syntax conversion plugin and a connection pool plugin.

[0070] Client 1 is specifically configured to: invoke the target syntax conversion plugin to perform syntax conversion on the second business data block; determine the target connection pool that matches the target database type based on the connection pool plugin; and store the syntax-converted second business data block into the target database based on the target connection pool.

[0071] In some embodiments, client 1 also consists of five parts: identity security, data operation, operating system plugin, database plugin, and runtime data collection. Identity security provides four main services: terminal login, encrypted information transmission, dynamic credential refresh, and certificate management. A server-side permission management module ensures terminal security. The data operation module implements data stream reception, file assembly, data integrity verification, and corresponding data and log file storage management services, making it a key module for effectively transmitting data to the client. The operating system plugin automatically adapts the terminal to mainstream terminal operating systems such as Windows, Linux, MacOS, and UOS, masking differences in terminal operating environments. The database plugin provides plugin support for the terminal to ultimately insert data into the client-side database, offering plugins for Oracle, MySQL, TDSQL, DB2, OceanBase, and TiDB, automatically applying the corresponding plugin according to the client 1's database information, masking differences in client databases. Runtime data collection provides the ability to upload terminal operation status to the management terminal 33, building a full-process visualized management capability, and providing four types of functions: data collection, data integration, terminal computing, and data upload. The database plugin, operating system plugin, and data operation module effectively address the insufficient customization issue in problem two; the identity security module solves the insufficient security protection in problem three; and the runtime data collection provides data support for solving problem four.

[0072] The database component of client 1 supports various database types, including Oracle, MySQL, TDSQL, DB2, TiDB, and OceanBase, based on the plugins in client 1. Furthermore, with the enrichment of client database plugins, the number of database types supported by the entire device can be further expanded.

[0073] Correspondingly, in another embodiment, the control terminal 33 primarily addresses problem four, namely the black box nature of the data service process and insufficient visualization, monitoring, and handling capabilities. The control terminal 33 is based on uploaded operational data, where the operational data acquisition module of client 1 and the calling module in the server-side data control module provide operational data upload functions for both ends. Supported by the operational data, the control terminal 33 provides management services including process visualization, user management, permission management, asset group management, link monitoring, link reset, data retransmission, breakpoint resumption, data statistics, traffic management, and plugin installation. Simultaneously, user management, permission management, and asset group management provide key capability support to address problem five, the insufficient data asset management capabilities.

[0074] As an alternative implementation, asset identifiers are distributed in the form of watermarks in the second metadata block and the second business data block.

[0075] Client 1 is configured to: determine whether the asset identifier in the received second business data block is completely watermarked; if it is determined that the asset identifier in the received second business data block is not completely watermarked, send a retransmission request to server 2 to retransmit the second business data block with incomplete watermark, and send the watermark information of the incomplete watermark to management terminal 3, so that management terminal 3 can determine the asset flow path of the second business data block in the asset full-link diagram based on the watermark information, and highlight the asset flow path.

[0076] As another implementation, client 1 is also configured to: write the second business data block that failed verification into the exception database and send the verification failure result to management terminal 3; management terminal 3 is also configured to: receive the verification failure result, determine the target link node in the data full link diagram where the verification failure result is located, and highlight the target link node.

[0077] As another implementation, the control terminal 3 is also configured to: determine the data transmission volume and schedulable network resource volume of the transmission link, and adjust the data transmission adoption rate.

[0078] In another implementation, the control terminal 3 is further configured to: determine whether the multiple task sequence numbers of all received second metadata blocks and second service data blocks are consecutive and complete. If they are not consecutive or complete, a retransmission request for retransmitting the non-consecutive or incomplete second metadata blocks and / or non-consecutive or incomplete second service data blocks is sent to the server terminal 2, and the non-consecutive or incomplete task sequence number information is sent to the control terminal 3; and determine whether the task sequence number of each received second service data block is consecutive with other task sequence numbers of other previously received second service data blocks. If the task sequence numbers are not consecutive, a retransmission request for retransmitting the second service data blocks with non-consecutive task sequence numbers is sent to the server terminal 2, and the non-consecutive task sequence number information is sent to the control terminal 3.

[0079] As another implementation method, the access control module is specifically configured as follows: sending an authentication command to client 1, responding to the user input information returned by client 1, and determining whether the user input information matches the authentication command; and, when accessing the database of server 2 on the access link, sending a dynamic code to the access node, and determining whether the dynamic code returned by the access node within a preset time is correct.

[0080] As one specific implementation method, the implementation process of the above system is described as follows through the implementation flow.

[0081] Firstly, the client 1 installation and initial startup take effect.

[0082] First, the client obtains the data synchronization client 1. During the installation process, client 1 software identifies the client terminal's operating system using the `adapt()` method. The operating system type data is stored as environment metadata in client 1's relational data storage engine, which then schedules the operating system plugin to complete the integration, installation, and activation. For example, in a Windows environment, the Windows Add-in is scheduled. Client 1 automatically completes the plugin integration and environment activation.

[0083] Secondly, upon initial startup of Client 1, the client uses its database maintenance function to configure the database type and connection string. This data is stored as metadata in Client 1's relational data storage engine, triggering the integration and activation of database plugins, such as mysql-addin. Client 1 automatically verifies database connectivity and activates the SQL statement adaptation template.

[0084] Secondly, client 1 initiates a data retrieval request.

[0085] Client 1's data operation module and server 2's data management module establish a data transmission channel. This channel consists of three parts: a synchronous channel responsible for transmitting metadata such as table structure and data row counts, built using WebSocket to create a long-lived connection for real-time bidirectional interaction; an asynchronous channel for transmitting specific business data, constructed using a message queue; and a module for information exchange between the two channels, including metadata verification and exception handling such as data flow control. During business data transmission, the system supports automatic selection of the appropriate transmission channel based on data type, strengthening the control and management of core metadata while ensuring efficient business data transmission.

[0086] Client 1 sends a business data request to server 2 through the data operation module. This data is then transmitted to the data management module of server 2 via a real-time channel.

[0087] After receiving the data request from client 1, the data management module of server 2 plans the corresponding data acquisition task according to the built-in task splitting rules, such as splitting by business product group, splitting by table, and splitting by row count, and then passes it to the data assembly module. At the same time, the data management module, as the security hub of the entire device, provides client 1, server 2, and management terminal 3 with a series of dynamic and flexible security protection services, including security keys, link encryption, and transmission auditing.

[0088] The server-side 2 data assembly module receives task information data and sequentially generates and schedules tasks through the task service, task generator, task executor, and task scheduler, updating task status and related indicators. Based on the task information, it retrieves data source management information from the data source management module and, according to the data source management information, retrieves relevant business data, table structure, data volume, and other metadata from the data storage module as required. The server-side 2 data storage module manages various storage data sources for financial data and integrates different execution engines based on the differences in data sources to complete data discovery, scheduling, and extraction, returning the data to the data assembly module. The server-side 2 data source management module extracts relevant management information from the data sources, such as data storage volume, data storage mode, and database type, improving data acquisition efficiency. Through a two-tier architecture design of request service + management service, it provides efficient data source acquisition services. Related result data is transmitted via synchronous and asynchronous transmission channels built by the data management module, selecting different channels depending on whether it is business data or metadata.

[0089] Third, client 1 receives data.

[0090] The client 1 data operation module receives metadata information such as the table structure and data volume of the data to be transmitted from the metadata long connection channel. Based on the table structure information, it calls the client 1 database plugin, automatically completes the syntax adjustment mapping through the SQL statement adaptation function, and calls the database connection to complete the automatic creation and validation of the table structure. Simultaneously, it calls the client 1's local built-in lightweight relational database to complete the table structure creation and validation.

[0091] The client 1 data operation module retrieves business data from the asynchronous message queue channel and automatically verifies the consistency between the actual business data structure and the table structure in the metadata, checking for anomalies such as excessive length or missing data. Abnormal data is then stored in the client 1's built-in relational database.

[0092] Normal data is transmitted and stored by establishing a connection between client 1 and the client's local database.

[0093] After the data transmission is completed, the data volume information in the metadata is used to check whether the transmitted data matches, and abnormal data is checked against server 2.

[0094] Fourth, monitoring and handling of data transmission processes.

[0095] The monitoring and handling module of the control terminal 3 processes the operational data from both ends, draws a complete data transmission service flow, monitors the data service status, and provides various types of control services such as plug-in management and emergency operations. It is responsible for the synchronous management of different data files, allowing for the setting of multiple synchronization strategies and the selection of different strategies for each type of data file, as well as the option to enable or disable synchronization. For data that is not currently needed, users can choose to disable synchronization. Once a problem is identified on the visual operational graph, the control terminal 3 can drill down to retrieve the operational logs of that part of the client 1's function, directly performing automatic problem analysis and handling on the control terminal 3.

[0096] The above implementation first proposes a four-layer architecture for the client-side database of client 1, consisting of server 2, client 1, control terminal 33, and client 1. Data extraction, processing, and encryption protection are handled by server 2, and data is transmitted to the client's local database via client 1. Control terminal 33 provides visualized end-to-end control capabilities. This layered and decoupled design effectively solves the problem of operational complexity. Simultaneously, an integrated platform + plug-in construction model is adopted, comprehensively considering the differences in operating system environment and database type among clients, and abstracting common characteristics to ensure terminal installation, startup, and stable operation. The terminal can independently complete operating system construction and operating system plug-in integration to support terminal startup. After startup, it automatically completes the subsequent database integration and access plug-in installation based on the client's database configuration, thus solving the biggest adaptation problem. At the same time, a synchronous bidirectional and asynchronous unidirectional data transmission channel is built between server 2 and client 1 to ensure the effective transmission of core metadata and synchronously ensure the stable flow of business data. A server-side data source control module, data control module, data storage module, data assembly module, and permission control and verification module are built to address the two key requirements of data transmission and scheduling operation. Secondly, the client-side modules for identity security and data operations, serving as the final conduit for security protection and data transmission, provide secure, orderly, and customizable data transmission services with the support of the security hub and data operation hub. Finally, the control terminal 3, as a key component for eliminating process black boxes, improving emergency response, and strengthening asset management, integrates relevant service functions based on the data uploaded from the server-side 2 and the data terminal, providing control capabilities throughout the entire data transmission lifecycle.

[0097] Therefore, by building client-side identity security, data operation, operating system plugins, runtime data collection, and database plugin capabilities, enabling efficient collaborative transmission of metadata and business data through separate channels, and establishing server-side data management, data assembly, data storage, data source management, and access control capabilities, coupled with database capabilities on both the management and client sides, a complete operational technical solution is formed. Based on the targeted solution design and orderly process operation of each module, this solution effectively addresses five major technical issues: complex functional operations, low operational efficiency, insufficient customized support for direct data connection and multi-terminal operation, low security protection capabilities, susceptibility to data leakage and tampering through internet-based transmission, opaque data service processes, insufficient visualization, monitoring, and handling capabilities, and a lack of data asset management capabilities. This effectively meets the secure and efficient data transmission requirements of the financial sector, achieving dual guarantees of security and efficiency.

[0098] To achieve the above functions, the financial data processing system includes corresponding hardware structures and / or software modules for performing each function. Those skilled in the art will readily recognize that, based on the algorithmic steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] Regarding the system in the above embodiments, the specific manner in which each unit module performs operations has been described in detail in the embodiments related to the method, and will not be elaborated here.

[0100] Figure 2 This is a schematic diagram of a financial data processing device provided in this application. Figure 2 The financial data processing device 50 may include at least one processor 501 and a memory 503 for storing processor-executable instructions. The processor 501 is configured to execute the instructions in the memory 503 to implement the financial data processing method in the following embodiments.

[0101] This financial data processing equipment is also called a financial data processing device, such as a server.

[0102] In addition, the financial data processing device 50 may also include a communication bus 502, at least one communication interface 504, an input device 506, and an output device 505.

[0103] The processor 501 may be a processor (central processing unit, CPU), a microprocessor unit, an ASIC, or one or more integrated circuits for controlling the execution of the program of the present application.

[0104] The communication bus 502 may include a path for transmitting information between the aforementioned components.

[0105] Communication interface 504 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area networks (WLAN), etc.

[0106] Input device 506 is used to receive input signals and output device 505 is used to output signals.

[0107] Memory 503 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory may exist independently and be connected to the processing unit via a bus. Memory may also be integrated with the processing unit.

[0108] The memory 503 stores instructions for executing the scheme of this application, and the processor 501 controls the execution. The processor 501 executes the instructions stored in the memory 503 to implement the functions of the method of this application.

[0109] In a specific implementation, as one example, the processor 501 may include one or more CPUs, for example... Figure 2 CPU0 and CPU1 in the CPU.

[0110] In a specific implementation, as one example, the financial data processing device 50 may include multiple processors, such as... Figure 2Processors 501 and 507 are shown in the diagram. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0111] The financial data processing equipment, such as Figure 2 The diagram includes a processor 501 and a memory 503 for storing executable instructions of the processor 501; wherein the processor 501 is configured to execute executable instructions to implement the financial data processing method as described in any of the possible embodiments above. Furthermore, it achieves the same technical effects, and to avoid repetition, will not be described further here.

[0112] This application also provides a computer-readable storage medium, which, when the instructions in the computer-readable storage medium are executed by the processor of a financial data processing device or a financial data processing apparatus, enables the financial data processing apparatus to perform a financial data processing method as described in any of the possible embodiments above. And it can achieve the same technical effect; to avoid repetition, it will not be described again here.

[0113] This application also provides a computer program product, including a computer program or instructions, which are executed by a processor as a financial data processing method according to any of the possible implementations described above. Furthermore, it achieves the same technical effects, and to avoid repetition, it will not be described again here.

[0114] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0115] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A financial data processing system, characterized in that, The system includes a client, a server, and a management terminal. The client includes multiple operating system plugins and multiple database plugins. The client and the server are connected by a synchronous bidirectional channel and an asynchronous unidirectional channel. The server is configured to: receive a task request sent by the client; divide the target task in the task request into multiple subtasks; determine a first metadata block for each subtask; process the subtask corresponding to the first metadata block according to the first metadata block to obtain a first business data block, wherein each first business data block and its corresponding first metadata block include the same target asset identifier and the task sequence number; encrypt each first metadata block and each first business data block to obtain multiple second metadata blocks and multiple second business data blocks; send each second metadata block to the client in real time through the synchronous bidirectional channel, and sequentially transmit each second business data block to the client in blocks according to the order represented by the task sequence number; the target asset identifier represents the data source resource ownership; The client is configured to: determine the target system type of the operating system to be executed by the client and the target database type to which the second business data is to be stored; select a target operation plugin associated with the target system type from multiple operation plugins, and select a target database plugin associated with the target database type from multiple database plugins; automatically load the target operation plugin into a preset program interface to obtain a target program interface when the client is running; call the target program interface to automatically create and verify a local table structure based on the second metadata block, and verify the received second business data block based on the local table structure; and write the successfully verified second business data block into the target database based on the target database plugin. The control terminal is configured to monitor the running status of multiple preset nodes in the server and the client in real time.

2. The financial data processing system according to claim 1, characterized in that, The server includes: The data source management module is used to determine the attribute information of the data source that executes the subtask according to the first metadata block, and, based on the attribute information, identify the resources of the data source and obtain the asset identifier of the resources used by the subtask. The data management module is used to process the task requests from the client, and to split the tasks in the task requests into the multiple sub-tasks; The data storage module includes multiple data execution engines and multiple databases of different types; it is used to call the target data execution engine among the multiple data execution engines to access and retrieve business data from the multiple databases associated with the server. The data assembly module is used to embed the preset information associated with the first metadata block into the corresponding first business data block. The preset information includes an asset watermark representing the asset identifier, a structure version number, and a task number representing a global continuous sequence number; as well as generating a task, submitting the task submission, updating the task status, and updating the task indicators. The access control module is used to authenticate the accessed task request, dynamically encrypt the access link of the task request, and perform frame extraction verification on the second metadata block and the second service data in the transmission link.

3. The financial data processing system according to claim 2, characterized in that, The control terminal is specifically configured as follows: Based on the asset identifier, trace the asset transmission path of the business data; A visualized asset end-to-end graph is generated based on the asset transmission path; and a visualized data end-to-end graph is generated based on the multiple preset nodes, the asset transmission path, and the operating status; the data end-to-end graph is a topology structure.

4. The financial data processing system according to claim 2, characterized in that, The client includes multiple connection pools; the database plugin includes a syntax conversion plugin and a connection pool plugin; the client is specifically configured as follows: Invoke the target syntax conversion plugin to perform syntax conversion on the second business data block; Based on the connection pool plugin, a target connection pool that is compatible with the target database type is determined; Based on the target connection pool, the second business data block after syntax conversion is stored in the target database.

5. The financial data processing system according to claim 2, characterized in that, The asset identifier is distributed in the form of a watermark in the second metadata block and the second business data block; the client is configured as follows: Determine whether the asset identifier in the received second business data block is completely watermarked; If it is determined that the asset identifier in the received second business data block is an incomplete watermark, a retransmission request for retransmitting the second business data block with the incomplete watermark is sent to the server, and the watermark information of the incomplete watermark is sent to the management terminal, so that the management terminal can determine the asset flow path of the second business data block in the asset full-link diagram based on the watermark information, and highlight the asset flow path.

6. The financial data processing system according to claim 3, characterized in that, The client is also configured to: write the second business data block that failed verification into the exception database, and send the verification failure result to the control terminal; The control terminal is also configured to: receive the verification failure result, determine that the verification failure result is located at the target link node in the data full link diagram, and highlight the target link node.

7. The financial data processing system according to any one of claims 1 to 6, characterized in that, The control terminal is also configured to: determine the data transmission volume and schedulable network resource volume of the transmission link, and adjust the data transmission adoption rate.

8. The financial data processing system according to any one of claims 1 to 6, characterized in that, The control terminal is also configured as follows: Determine whether the multiple task sequence numbers of all received second metadata blocks and second service data blocks are consecutive and complete; if they are not consecutive or complete, send a retransmission request to the server to retransmit the non-consecutive or incomplete second metadata blocks and / or the non-consecutive or incomplete second service data blocks, and send the non-consecutive or incomplete task sequence number information to the control terminal. In addition, it determines whether the task sequence number of each received second service data block is consecutive to the task sequence numbers of other previously received second service data blocks. If the task sequence numbers are not consecutive, it sends a retransmission request to the server for retransmission of the second service data blocks with non-consecutive task sequence numbers, and sends the non-consecutive task sequence number information to the control terminal.

9. The financial data processing system according to any one of claims 2 to 6, characterized in that, The permission control module is specifically configured to: send an authentication command to the client, respond to the user input information returned by the client, and determine whether the user input information matches the authentication command; and, when accessing the database of the server on the access link, send a dynamic code to the access node and determine whether the dynamic code returned by the access node within a preset time is correct.

10. A financial data processing method, characterized in that, The method, applied to a financial data processing system, includes: The server executes the following operations: receiving a task request from the client; dividing the target task in the task request into multiple subtasks; determining the first metadata block for each subtask; processing the subtask corresponding to the first metadata block to obtain a first business data block, wherein each first business data block and its corresponding first metadata block include the same target asset identifier and the task sequence number; encrypting each first metadata block and each first business data block to obtain multiple second metadata blocks and multiple second business data blocks; transmitting each second metadata block to the client in real time through the synchronous bidirectional channel, and transmitting each second business data block to the client in blocks according to the order represented by the task sequence number; the target asset identifier represents the data source resource ownership. The client executes the following operations: determining the target system type of the operating system to be executed by the client and the target database type to which the second business data is to be stored; selecting the target operation plugin associated with the target system type from multiple operation plugins, and selecting the target database plugin associated with the target database type from multiple database plugins; automatically loading the target operation plugin into a preset program interface to obtain a target program interface during client runtime; calling the target program interface to automatically create and verify the local table structure based on the second metadata block, and verifying the received second business data block based on the local table structure; and writing the successfully verified second business data block into the target database based on the target database plugin. The control terminal is invoked to perform the following operation: real-time monitoring of the running status of multiple preset nodes in the server and the client.

Images