A financial big data management system and method

Abstract

The application is a kind of financial big data management system and method, relating to the technical field of financial data management, comprising: a data collection dynamic optimization module configured to adapt the timeout threshold based on the adjustment factor, parallel task scheduling and connection pool management, solving the problem of script timeout; real-time data verification and transmission module; full-link monitoring and self-healing module; intelligent performance diagnosis and optimization module; emergency data recovery and fault tolerance module. In the application, the adjustment factor of the dynamic optimization data collection module is combined with the Sigmoid function to realize intelligent expansion and contraction of the timeout threshold with the system load, combined with hierarchical task scheduling and connection pool management to solve the script timeout problem; the real-time data verification and transmission module ensures the integrity and security of data transmission through strict verification, hierarchical transmission strategy and encrypted cache, effectively reducing data gaps and loss, and providing high-quality data basis for financial business analysis.

Description

Technical Field

[0001] This invention relates to the field of financial data management technology, and in particular to a financial big data management system and method. Background Technology

[0002] With the rapid development of the financial industry, financial data is growing explosively, which places higher demands on the management of financial big data.

[0003] Currently, there are many problems in the management of financial big data: script timeouts are prone to occur during data collection, leading to data loss; during data transmission, incomplete data is often caused by insufficient verification or unreasonable transmission strategies; the entire data flow lacks effective monitoring and self-healing mechanisms, which easily leads to gaps in monitoring charts; system performance is difficult to continuously optimize, and problems such as script timeouts occur repeatedly; in the face of extreme situations, there is a lack of effective emergency recovery and fault tolerance measures, which cannot guarantee the continuity and security of financial data.

[0004] These problems seriously affect the reliability of financial data, which in turn has an adverse impact on risk monitoring and decision-making in financial operations.

[0005] Therefore, a financial big data management system and methodology are needed to address the problems mentioned above. Summary of the Invention

[0006] The purpose of this invention is to provide a financial big data management system and method to solve the above-mentioned problems.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] A financial big data management system, comprising:

[0009] The data acquisition dynamic optimization module is configured to adapt timeout thresholds based on adjustment factors, schedule parallel tasks, and manage connection pools to solve the timeout problem of acquisition scripts.

[0010] The real-time data verification and retransmission module is configured to rely on real-time verification, hierarchical retransmission strategies, and encrypted caching to solve the problem of incomplete data upload and ensure data integrity and transmission reliability.

[0011] The end-to-end monitoring and self-healing module is configured to eliminate gaps in monitoring charts by leveraging multi-dimensional monitoring, data self-healing mechanisms, and multi-level alarms.

[0012] The intelligent performance diagnosis and tuning module is configured to continuously optimize system performance through script bottleneck analysis, dynamic resource allocation, and periodic optimization suggestions;

[0013] The emergency data recovery and fault tolerance module is configured to handle data loss in extreme situations by leveraging breakpoint rollback, multi-level backup, and dual-active switching.

[0014] Preferably, the data acquisition dynamic optimization module specifically includes:

[0015] Runtime data of data acquisition tasks are extracted from the database, analyzed to obtain adjustment factors, and the data acquisition timeout threshold is optimized based on the adjustment factors to cope with subsequent acquisition tasks.

[0016] Determine the orchestration configuration for Kubernetes containers and prioritize subtasks;

[0017] The management of the data connection pool is determined, including investigating invalid connections after the parameter configuration of the connection pool is determined.

[0018] Preferably, the process of obtaining the regulation factor includes:

[0019] Extract the time periods corresponding to each data collection task from the database, and extract the time periods corresponding to the data gaps from the database, and record them as marked time periods;

[0020] Get the CPU utilization rate within the marked time period, preset the CPU utilization rate threshold, and divide the obtained CPU utilization rate by the CPU utilization rate threshold to obtain the utilization percentage;

[0021] Get the time it takes for the script to read performance files in the disk directory, and divide the obtained time by the preset standard read time to get the read percentage.

[0022] Obtain the congestion time of the receiving queue of the monitoring platform, extract the longest congestion time from it, divide it by the allowed congestion time, and get the excess limit.

[0023] The reference coefficient is obtained by weighting and summing the occupancy, read occupancy, and over-limit.

[0024] Obtain the timeout threshold and reference coefficient for each marked time period;

[0025] Extract the data collection task from the database, and record the time period without any data blank areas as the normal time period. Obtain the reference coefficient corresponding to the normal time period and record it as the standard reference coefficient.

[0026] The standard reference coefficients for each normal time period are obtained sequentially, and the average value is calculated to obtain the standard reference average coefficient.

[0027] The preliminary adjustment factor is obtained by dividing the reference coefficient of each marked time period by the standard reference coefficient; the adjustment factor is obtained by averaging all the preliminary adjustment factors.

[0028] Preferably, a nonlinear relationship function is constructed based on the adjustment factor and the timeout threshold, and the timeout threshold is optimized accordingly;

[0029] Nonlinear relational functions: ;

[0030] in The optimized timeout threshold time;

[0031] The timeout threshold before optimization;

[0032] k is the slope parameter; e is the natural constant; It is a regulatory factor.

[0033] Preferably, the real-time data verification and retransmission module specifically includes:

[0034] The system performs integrity checks on the fields of the data. If any anomalies are found during the check, the system will send detailed information about the abnormal data to the data acquisition module.

[0035] A tiered retransmission strategy is implemented based on subtask priority; a combination of mirroring and striping is used to divide the data into two halves, which are stored in two sets of mirror disks respectively; and the cached data is encrypted.

[0036] Preferably, the end-to-end monitoring and self-healing module specifically includes:

[0037] The system retrieves monitoring metrics data from the exporter of each module at preset time intervals.

[0038] When extracting data from log backtracking, threads are dynamically created based on the size of the data, and each thread handles the extraction task of one log file.

[0039] The extracted data is compared with the monitoring data in adjacent time periods to calculate the deviation rate. If the absolute value of the deviation rate exceeds 5%, the data is considered abnormal and needs to be extracted and verified again. For the re-extracted data, the deviation rate is calculated again. If it still exceeds 5%, it is marked as suspicious data and notified for manual review.

[0040] Preferably, the intelligent performance diagnosis and optimization module specifically includes:

[0041] When the script starts, the Profiler agent is loaded via the javaagent parameter to record the entry and exit times, parameter values, and return values ​​of each function.

[0042] The recorded data is stored in a local file in binary format, with one file generated every hour, and the data is converted into visual charts.

[0043] When the script's memory usage continues to exceed the preset value for a specified period of time, HPA will calculate the number of additional copies needed based on the current memory usage.

[0044] ARIMA is used for resource use forecasting, processing time series data with periodicity and trends.

[0045] Preferably, the emergency data recovery and fault tolerance module specifically includes:

[0046] When sending a supplementary data collection request to the data source, the request parameters include the page number, the amount of data per page, the start time, and the end time.

[0047] After receiving the request, the data source server queries the corresponding data based on these parameters and includes the total number of pages and the data of the current page in the response. After receiving the response, the system checks whether the data of the current page is complete. If it is complete, it continues to request the next page until all data is obtained.

[0048] The data is distributed and stored on the hard drive, and parity information is calculated and stored at the same time.

[0049] A financial big data management method includes the following steps:

[0050] Dynamically optimize data acquisition: use load data to calculate adjustment factors to dynamically adjust thresholds, and hierarchically schedule tasks and manage connection pools;

[0051] Ensure data transmission integrity: verify data, perform tiered retransmission, and use mirrored encrypted caching to ensure transmission integrity and data security;

[0052] End-to-end monitoring and self-healing: pull metrics and store them for a preset duration, backtrack and verify logs, and trigger alarms for deviations to achieve end-to-end monitoring and self-healing;

[0053] Intelligent diagnostics and performance tuning: identifying bottlenecks through data entry points and optimizing matching schemes; dynamic scaling and resource prediction, generating reports to track iterations and continuously improve performance;

[0054] Emergency recovery and fault tolerance: paginated data acquisition and disaster recovery storage, quarterly test recovery; primary and backup heterogeneous complementarity, automatic update synchronization.

[0055] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0056] 1. This invention achieves intelligent scaling of timeout thresholds based on system load by dynamically optimizing the adjustment factor and Sigmoid function of the data acquisition module. Combined with hierarchical task scheduling and connection pool management, it solves the script timeout problem. The real-time data verification and retransmission module ensures the integrity and security of data transmission through strict verification, hierarchical transmission strategies, and encrypted caching, effectively reducing data gaps and loss, and providing a high-quality data foundation for financial business analysis.

[0057] 2. This invention achieves visualized tracking of the entire data flow process and automatic response to anomalies through a full-link monitoring and self-healing module, eliminating monitoring gaps; the intelligent performance diagnosis and optimization module continuously analyzes bottlenecks and dynamically optimizes resources, reducing failures at their root; the emergency recovery and fault tolerance module ensures continuous data availability under extreme conditions through multi-level backup, primary / backup switching and regular testing, comprehensively improving the system's ability to cope with complex loads and sudden risks. Attached Figure Description

[0058] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0059] Figure 1 This is a system structure diagram of the present invention. Detailed Implementation

[0060] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0061] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein. Example 1:

[0062] Its specific implementation method is combined with the appendix Figure 1 Please provide a detailed explanation.

[0063] Appendix Figure 1 This invention provides a structural block diagram of a financial big data management system, which shows the connection relationship between the data acquisition dynamic optimization module and the emergency data recovery and fault tolerance module, and marks the main functional interaction flow of each module.

[0064] In this embodiment, it includes:

[0065] The data acquisition dynamic optimization module is configured to adapt timeout thresholds based on adjustment factors, schedule parallel tasks, and manage connection pools to solve the timeout problem of acquisition scripts and improve the efficiency and stability of periodic acquisition tasks.

[0066] Specifically, it includes:

[0067] Runtime data of data acquisition tasks are extracted from the database, analyzed to obtain adjustment factors, and the data acquisition timeout threshold is optimized based on the adjustment factors to cope with subsequent acquisition tasks.

[0068] The process of obtaining the regulatory factor includes:

[0069] Extract the time periods corresponding to each data collection task from the database, and extract the time periods corresponding to the data gaps from the database, and record them as marked time periods;

[0070] Get the CPU utilization rate within the marked time period, preset the CPU utilization rate threshold, and divide the obtained CPU utilization rate by the CPU utilization rate threshold to obtain the utilization percentage;

[0071] Converting CPU load into a calculable percentage (e.g., CPU utilization = 1 when running at full capacity and close to 0 when idle) clearly reflects the computational resource pressure during data acquisition.

[0072] Get the time it takes for the script to read performance files in the disk directory, and divide the obtained time by the preset standard read time to get the read percentage.

[0073] The fluctuation in the time taken to read files by the quantification script (e.g., when the disk is busy, the read occupancy > 1 indicates that the read speed has slowed down);

[0074] Obtain the congestion time of the receiving queue of the monitoring platform, extract the longest congestion time from it, divide it by the allowed congestion time, and get the excess limit.

[0075] Break down the vague reasons for the data gaps into specific quantitative indicators of CPU, disk, and queue to facilitate pinpointing which link caused the data collection failure.

[0076] The reference coefficient is obtained by weighting and summing the occupancy, read occupancy, and over-limit.

[0077] The reference coefficient is calculated by multiplying the usage extent, reading extent, and over-limit extent by their corresponding weight factors.

[0078] Obtain the timeout threshold and reference coefficient for each marked time period;

[0079] Extract the data collection task from the database, and record the time period without any data blank areas as the normal time period. Obtain the reference coefficient corresponding to the normal time period and record it as the standard reference coefficient.

[0080] Using the standard reference coefficient for a normal time period as a benchmark, the adjustment factor can compare the difference between the load during data blank periods and the normal load.

[0081] The adjustment factor is a dynamic value based on historical normal data and current load quantification, which better reflects the actual operating status of the system;

[0082] The standard reference coefficients for each normal time period are obtained sequentially, and the average value is calculated to obtain the standard reference average coefficient.

[0083] Divide the reference coefficient for each marked time period by the standard reference coefficient to obtain the preliminary adjustment factor;

[0084] The adjustment factor is obtained by averaging all the initial adjustment factors;

[0085] Reduce false positives for blank data (the threshold is reasonably widened under high load, and more real data is collected), and also reduce false negatives (the threshold is tightened under low load, and abnormalities in the collection script are detected in time).

[0086] Improved data collection completeness and accuracy provide a more reliable data foundation for downstream analysis (such as risk monitoring and business decision-making);

[0087] The system automatically optimizes itself, eliminating the need for frequent manual adjustments to timeout thresholds (traditional methods rely on operational experience and cannot respond to load changes in real time).

[0088] A nonlinear relationship function is constructed based on the adjustment factor and the timeout threshold, and the timeout threshold is optimized accordingly.

[0089] Nonlinear relational functions: ;

[0090] in The optimized timeout threshold time;

[0091] The timeout threshold before optimization;

[0092] k is the slope parameter (usually taken as 1~5, which controls the steepness of the curve; the larger k is, the steeper the curve).

[0093] e is the natural constant (approximately 2.718);

[0094] It is a regulatory factor;

[0095] Financial systems often exhibit a critical point effect (e.g., once CPU utilization reaches 80%, task execution time increases exponentially due to resource contention). The S-shaped curve of the Sigmoid function perfectly captures this non-linear relationship.

[0096] When the load is normal (adjustment factor) ≈1): The slope of the curve is gentle in the middle section, and the threshold changes little (e.g., the threshold is only slightly adjusted when the adjustment factor fluctuates from 1.0 to 1.1), which avoids frequent threshold oscillations caused by small load fluctuations and ensures system stability;

[0097] When the load is extreme (adjustment factor) ≥ or ≤1): The slope of the curve is steep at both ends, and the threshold is adjusted quickly (e.g., the adjustment factor jumps from 1.5 to 2.0, and the threshold is significantly relaxed), which can adapt to the increase in task time under extreme load in a timely manner and reduce the probability of normal tasks being misjudged as timeout due to the threshold being too strict.

[0098] Under extreme loads, the threshold in traditional linear models may become uncontrollable due to excessively large or small adjustment factors (e.g., linear amplification leading to unlimited threshold growth; the sigmoid function naturally possesses saturation characteristics).

[0099] No matter how extreme the regulatory factor is tending toward infinity or As the threshold approaches 0, it will eventually approach 1.5. Or 0.5 Limiting extreme thresholds from a mathematical perspective prevents the system from crashing due to unreasonable thresholds.

[0100] Financial transactions (such as payments and clearing) have extremely low tolerance for false timeouts: if the threshold is too strict, normal transactions may be judged as timeout failures due to high instantaneous load, leading to customer complaints or financial losses. The Sigmoid model automatically relaxes the threshold under high load, ensuring that tasks with reasonable actual time consumption are not falsely penalized, thus guaranteeing a smooth transaction process.

[0101] Under low load (such as nighttime batch processing), the Sigmoid function will slightly tighten the threshold. When the threshold is <1, the slope of the middle section of the curve is gentle, and the threshold is appropriately reduced, which promotes faster task execution, reduces resource idleness, and improves the overall efficiency of the system.

[0102] Financial systems have high requirements for model interpretability (to meet regulatory requirements for traceability of risk control). The Sigmoid function has clear parameters (slope k, natural constant e), and the adjustment factor can be calculated through intuitive indicators such as CPU utilization and queue length, making it easier for the technical team to explain the logic of threshold changes and also easier to pass compliance audits;

[0103] The slope k is a configurable control knob: if the financial system prioritizes stability (such as the core trading system), k can be increased (e.g., set to 2) to make the curve steeper, and the threshold converges to the safe range faster under extreme loads; if the system needs to respond sensitively to loads (such as real-time market data systems), k can be decreased (e.g., set to 0.8) to make the middle section of the curve smoother, balancing threshold stability and response speed.

[0104] Determine the orchestration configuration for Kubernetes containers and prioritize subtasks;

[0105] Specific configuration for Kubernetes container orchestration: Each subtask container is limited to 0.5 CPU cores and 512MB of memory, with a requested CPU core limit of 0.2 cores and a requested memory limit of 256MB; container health checks are implemented by executing a script that checks the running status of the subtask every 5 seconds. If the subtask is found to be unresponsive after 3 consecutive checks, the container is automatically restarted.

[0106] Subtask priority classification criteria: P0 level subtasks correspond to data collection involving changes in customer funds, such as deposits, withdrawals, and transfers; P1 level subtasks correspond to the collection of customer account information, such as account balances and transaction records; P2 level subtasks correspond to the collection of system operation logs. During resource scheduling, P0 level subtasks have a 30% higher scheduling priority than P1 level subtasks, and P1 level subtasks have a 20% higher priority than P2 level subtasks.

[0107] Determine the management of the data connection pool, including investigating invalid connections after determining the parameter configuration of the connection pool;

[0108] Apache CommonsDBCP connection pool parameter configuration: set maxWait to 10000 milliseconds, meaning that when there are no available connections in the connection pool, the maximum waiting time for the request thread is 10 seconds; set testOnBorrow to true, meaning that the connection validity test is performed every time a connection is obtained from the connection pool; set timeBetweenEvictionRunsMillis to 300000 milliseconds, meaning that the idle connection recycling thread runs every 5 minutes.

[0109] When an invalid connection is detected, the system automatically records information such as the data source IP address, port number, and failure time corresponding to the connection, and generates a troubleshooting work order. After receiving the work order, the operations and maintenance personnel remotely log in to the data source server to check the network connection, database service status, etc. After identifying the problem, they report the handling results back to the system, which records the results in the knowledge base for future reference.

[0110] The real-time data verification and retransmission module is configured to rely on real-time verification, hierarchical retransmission strategies, and encrypted caching to solve the problem of incomplete data upload and ensure data integrity and transmission reliability.

[0111] Specifically, it includes:

[0112] The system performs integrity checks on the data fields. For date fields, it verifies whether the format is "yyyy-MM-ddHH:mm:ss" and checks whether the date is within a reasonable range (e.g., not earlier than the system's launch date). For string fields, it verifies whether the length is within the specified range, such as the customer name field, which is limited to 1-50 characters. If any anomalies are found during the verification process, the system will send detailed information about the abnormal data (including field name, error type, original value, etc.) to the data acquisition module.

[0113] Implement a tiered retransmission strategy based on subtask priority;

[0114] Network configuration for the P0-level dedicated data retransmission channel: This channel uses a dedicated network interface card (NIC), with an IP address range of 192.168.100.0 / 24, a gateway of 192.168.100.1, and DNS servers of 8.8.8.8 and 8.8.4.4; the network bandwidth is guaranteed to be 100Mbps, and QoS (Quality of Service) settings ensure that traffic from this channel is prioritized over other channels.

[0115] The specific operation of batch compression and transmission is as follows: P1 and P2 level data are batch compressed every 30 seconds. The system automatically scans the local cache directory for unuploaded P1 and P2 level data files, groups them according to their creation time, and each group contains a maximum of 100 files. During compression, the ZIP compression level is set to 5 levels (0~9 levels, the higher the level, the higher the compression ratio but the longer the time). The compressed package generated after compression contains timestamp information and level information.

[0116] By using a combination of mirroring and striping, the data is divided into two halves and stored in two sets of mirror disks. This configuration improves read and write speeds by about 3 times compared to a single SSD. At the same time, if one disk fails, the data can be quickly recovered through the mirror disks, with a recovery time of no more than 5 minutes.

[0117] And the cached data is encrypted;

[0118] For example, the AES-256 encryption algorithm uses a Hardware Security Module (HSM) for key storage and management, and the key is automatically changed every 7 days. During each encryption and decryption operation, the system retrieves the key from the HSM, and immediately destroys the key information in memory after the operation is complete. Only authenticated system modules can obtain the key by calling the HSM interface, and each call is recorded in the audit log.

[0119] The end-to-end monitoring and self-healing module is configured to eliminate gaps in monitoring charts and achieve visualized tracking and anomaly response throughout the entire data flow process by leveraging multi-dimensional monitoring, data self-healing mechanisms, and multi-level alarms.

[0120] Specifically, it includes:

[0121] Monitoring metrics data is retrieved from the exporters of each module at preset time intervals, with a storage duration of 30 days. The configuration file defines various monitoring rules, such as a histogram of script runtime (buckets set to [1,2,3,4,5,6,7,8,9,10] seconds) and a gauge for data upload success rate. Multiple dashboards are created in the Grafana visualization tool, each corresponding to a module, allowing users to select different time ranges and data sources via dropdown menus. Thresholds for the monitoring metrics are also set.

[0122] The threshold for monitoring metrics is set based on the following: the threshold for data upload success rate is derived from historical data statistics over the past three months.

[0123] For example, over these three months, the average data upload success rate was 95%, and the lowest was 88%. Setting a 90% threshold allows for timely detection of anomalies while maintaining a certain level of fault tolerance. Thresholds for other metrics are set using a similar method, determined by combining historical data and business needs.

[0124] When extracting data from log backtracking, threads are dynamically created based on the size of the data, and each thread handles the extraction task of one log file. The thread pool adopts the rejection policy of CallerRunsPolicy, that is, when the queue is full, the thread that submitted the task will execute the task itself to avoid task loss.

[0125] The extracted data is compared with the monitoring data in adjacent time periods, and the deviation rate is calculated. The formula for calculating the deviation rate is (extracted data - adjacent data) / adjacent data × 100%. If the absolute value of the deviation rate exceeds 5%, the data is considered abnormal and needs to be extracted and verified again. For the data after re-extraction, the deviation rate is calculated again. If it still exceeds 5%, it is marked as suspicious data and notified for manual review.

[0126] Call the open APIs of WeChat Work and DingTalk to encapsulate alarm information into a specific format and send it to the corresponding group or individual; for example, the alarm information sent to WeChat Work includes fields such as title, alarm level, occurrence time, detailed description and handling suggestions, where the alarm level is indicated by red (severe), yellow (warning) and blue (hint);

[0127] The system records the response time and processing completion time for each alarm and generates an alarm response report monthly.

[0128] The intelligent performance diagnosis and tuning module is configured to continuously optimize system performance through script bottleneck analysis, dynamic resource allocation, and periodic optimization suggestions, thereby reducing the occurrence of script timeouts and other problems from the root cause.

[0129] Specifically, it includes:

[0130] When the script starts, the Profiler agent is loaded via the javaagent parameter. This agent will record the entry and exit times, parameter values, and return values ​​of each function during the script's execution.

[0131] The recorded data is stored in local files in binary format, with one file generated every hour, each file being approximately 100MB in size. These data can be converted into visual charts using specialized analysis tools, such as function call graphs where nodes represent functions, edges represent call relationships, and the thickness of the edges represents the number of calls.

[0132] The analysis tool first calculates the total time and percentage for each function, identifying functions with a percentage exceeding 10% as key optimization targets. Then, based on the function type and implementation, it matches corresponding optimization solutions from best practices in the codebase.

[0133] For example, for time-consuming JSON parsing code, analysis tools will compare the performance data of the Jackson and Gson libraries. The Jackson library is about 20% faster than the Gson library when parsing large JSON data, so it is recommended to use the Jackson library instead of the Gson library.

[0134] When the script's memory usage continues to exceed the preset value for a specified period of time, HPA will calculate the number of additional replicas needed based on the current memory usage rate. The calculation formula is: target number of replicas = current number of replicas × (current memory usage rate / target memory usage rate), rounded up.

[0135] For example: when the peak CPU utilization exceeds 90% for 3 minutes, increase the CPU resource quota of the Pod by 0.1 cores each time until the CPU utilization is below 80%;

[0136] Resource use forecasting was performed using ARIMA (Autoregressive Integral Moving Average), a model that can handle time series data with periodicity and trends well.

[0137] The model parameters were determined using a grid search method, with p=3, d=1, q=2, and a prediction step size of 168 hours (1 week). The model was updated daily based on the actual resource usage data from the previous day to improve prediction accuracy.

[0138] In addition to historical data and optimization suggestions, it also includes performance comparison charts before and after optimization, resource and time estimates required to implement the optimization suggestions, potential risks and countermeasures;

[0139] For example, the section recommending starting the backup data collection node 30 minutes in advance during peak hours will show the script timeout rate decrease curve after starting the backup node, as well as the server resources and manpower costs required to start the backup node;

[0140] Each optimization suggestion has a unique number. The system records the implementation time, implementers, and post-implementation performance data. By comparing the performance data before and after implementation, the system calculates the optimization effect, such as the percentage reduction in script timeout rate and the percentage increase in data upload success rate. For suggestions with poor implementation results, the system analyzes the reasons, which may be that the suggestion itself is unreasonable or there are problems in the implementation process. Based on the analysis results, the system will formulate a new optimization plan.

[0141] The emergency data recovery and fault tolerance module is configured to handle data interruptions in extreme situations by leveraging breakpoint rollback, multi-level backup, and active-active switching, ensuring the continuity and security of financial data.

[0142] Specifically, it includes:

[0143] When sending a supplementary data collection request to the data source, the request parameters include the page number, the amount of data per page, the start time, and the end time.

[0144] After receiving the request, the data source server queries the corresponding data based on these parameters and includes the total number of pages and the data of the current page in the response. After receiving the response, the system checks whether the data of the current page is complete. If it is complete, it continues to request the next page until all data is obtained.

[0145] A token bucket algorithm is used to control the access frequency of the data source. The token bucket has a capacity of 100 and generates 10 tokens per second. Each time a request is sent to the data source, a token needs to be obtained from the token bucket. If no token is available, the request waits until a token becomes available or the waiting time exceeds 10 seconds (at which point the request is abandoned and a failure message is recorded). In this way, the number of requests per second is ensured to not exceed 10, which meets the access frequency limit of the data source.

[0146] Data is distributed and stored on hard drives according to a certain algorithm, and parity information is calculated and stored at the same time. When one of the hard drives fails, the system can recover the data from the failed hard drive based on the data from the other hard drives and the parity information.

[0147] Each quarter, a non-peak business period is selected for data recovery testing. A random backup point in time is chosen, and data is restored from local SSD cache, distributed file system, and cloud cold backup. After recovery, the consistency between the restored data and the original data is compared to check the data integrity and accuracy. Recovery time, success rate, and other metrics are recorded during the test, and any problems found are rectified promptly.

[0148] The main script is developed in Java, based on the Spring Boot framework, and uses multithreading for data acquisition; the backup script is developed in Python, based on the Scrapy framework, and uses asynchronous I / O for data acquisition. The two technical architectures each have their advantages when handling different types of data sources. The Java script has more stable performance when handling high-concurrency database access, while the Python script has higher development efficiency when handling web crawler data sources.

[0149] When the main script is updated, developers commit code to the Git repository. The system automatically detects the code change and triggers a synchronization update process for the backup script. First, the updated content of the main script is converted into the corresponding code in the backup script (using a custom code conversion tool). Then, the converted code is compiled and unit tested. After passing the tests, it is deployed to the backup script's runtime environment. After deployment, compatibility testing is performed to simulate the main-backup switchover process, ensuring that the backup script can collect data normally. Example 2:

[0150] A financial big data management method includes the following components:

[0151] Dynamically optimize data acquisition: use load data to calculate adjustment factors to dynamically adjust thresholds, and hierarchically schedule tasks and manage connection pools;

[0152] Ensure data transmission integrity: verify data, perform tiered retransmission, and use mirrored encrypted caching to ensure transmission integrity and data security;

[0153] End-to-end monitoring and self-healing: pull metrics and store them for a preset duration, backtrack and verify logs, and trigger alarms for deviations to achieve end-to-end monitoring and self-healing;

[0154] Intelligent diagnostics and performance tuning: identifying bottlenecks through data entry points and optimizing matching schemes; dynamic scaling and resource prediction, generating reports to track iterations and continuously improve performance;

[0155] Emergency recovery and fault tolerance: paginated data acquisition and disaster recovery storage, quarterly test recovery; primary and backup heterogeneous complementarity, automatic update synchronization.

[0156] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0157] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0158] It should be noted that, in this document, the use of relational terms such as "first" and "second" is merely for distinguishing one entity or operation from another, and does not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0159] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0160] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0161] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0162] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0163] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0164] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0165] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A financial big data management system, characterized in that, include: The data acquisition dynamic optimization module is configured to adapt timeout thresholds based on adjustment factors, schedule parallel tasks, and manage connection pools to solve the timeout problem of acquisition scripts. A nonlinear relationship function is constructed based on the adjustment factor and the timeout threshold, and the timeout threshold is optimized accordingly. Nonlinear relational functions: ; in The optimized timeout threshold time; The timeout threshold before optimization; k is the slope parameter; e is the natural constant; It is a regulatory factor; The system retrieves monitoring metrics data from the exporter of each module at preset time intervals. When extracting data from log backtracking, threads are dynamically created based on the size of the data, and each thread handles the extraction task of one log file. The extracted data is compared with the monitoring data in adjacent time periods, and the deviation rate is calculated. The formula for calculating the deviation rate is (extracted data - adjacent data) / adjacent data × 100%. If the absolute value of the deviation rate exceeds 5%, the data is considered abnormal and needs to be extracted and verified again. For the data after re-extraction, the deviation rate is calculated again. If it still exceeds 5%, it is marked as suspicious data and notified for manual review. The real-time data verification and retransmission module is configured to rely on real-time verification, hierarchical retransmission strategies, and encrypted caching to solve the problem of incomplete data upload and ensure data integrity and transmission reliability. The end-to-end monitoring and self-healing module is configured to eliminate gaps in monitoring charts by leveraging multi-dimensional monitoring, data self-healing mechanisms, and multi-level alarms. The intelligent performance diagnosis and tuning module is configured to continuously optimize system performance through script bottleneck analysis, dynamic resource allocation, and periodic optimization suggestions; The emergency data recovery and fault tolerance module is configured to handle data loss in extreme situations by leveraging breakpoint rollback, multi-level backup, and dual-active switching.

2. The financial big data management system according to claim 1, characterized in that, The data acquisition dynamic optimization module specifically includes: Runtime data of data acquisition tasks are extracted from the database, analyzed to obtain adjustment factors, and the data acquisition timeout threshold is optimized based on the adjustment factors to cope with subsequent acquisition tasks. Determine the orchestration configuration for Kubernetes containers and prioritize subtasks; The management of the data connection pool is determined, including investigating invalid connections after the parameter configuration of the connection pool is determined.

3. A financial big data management system according to claim 2, characterized in that, The process of obtaining the regulatory factor includes: Extract the time periods corresponding to each data collection task from the database, and extract the time periods corresponding to the data gaps from the database, and record them as marked time periods; Get the CPU utilization rate within the marked time period, preset the CPU utilization rate threshold, and divide the obtained CPU utilization rate by the CPU utilization rate threshold to obtain the utilization percentage; Get the time it takes for the script to read performance files in the disk directory, and divide the obtained time by the preset standard read time to get the read percentage. Obtain the congestion time of the receiving queue of the monitoring platform, extract the longest congestion time from it, divide it by the allowed congestion time, and get the excess limit. The reference coefficient is obtained by weighting and summing the occupancy, read occupancy, and over-limit. Obtain the timeout threshold and reference coefficient for each marked time period; Extract the data collection task from the database, and record the time period without any data blank areas as the normal time period. Obtain the reference coefficient corresponding to the normal time period and record it as the standard reference coefficient. The standard reference coefficients for each normal time period are obtained sequentially, and the average value is calculated to obtain the standard reference average coefficient. The preliminary adjustment factor is obtained by dividing the reference coefficient of each marked time period by the standard reference coefficient; the adjustment factor is obtained by averaging all the preliminary adjustment factors.

4. A financial big data management system according to claim 1, characterized in that, The real-time data verification and retransmission module specifically includes: The system performs integrity checks on the fields of the data. If any anomalies are found during the check, the system will send detailed information about the abnormal data to the data acquisition module. A tiered retransmission strategy is implemented based on subtask priority; a combination of mirroring and striping is used to divide the data into two halves, which are stored in two sets of mirror disks respectively; and the cached data is encrypted.

5. A financial big data management system according to claim 1, characterized in that, The intelligent performance diagnosis and optimization module specifically includes: When the script starts, the Profiler agent is loaded via the javaagent parameter to record the entry and exit times, parameter values, and return values ​​of each function. The recorded data is stored in a local file in binary format, with one file generated every hour, and the data is converted into visual charts. When the script's memory usage continues to exceed the preset value for a specified period of time, HPA will calculate the number of additional copies needed based on the current memory usage. ARIMA is used for resource use forecasting, processing time series data with periodicity and trends.

6. A financial big data management system according to claim 1, characterized in that, The emergency data recovery and fault tolerance module specifically includes: When sending a supplementary data collection request to the data source, the request parameters include the page number, the amount of data per page, the start time, and the end time. After receiving the request, the data source server queries the corresponding data based on these parameters and includes the total number of pages and the data of the current page in the response. After receiving the response, the system checks whether the data of the current page is complete. If it is complete, it continues to request the next page until all data is obtained. The data is distributed and stored on the hard drive, and parity information is calculated and stored at the same time.

7. A financial big data management method, comprising a financial big data management system according to any one of claims 1-6, characterized in that, Includes the following steps: Dynamically optimize data acquisition: use load data to calculate adjustment factors to dynamically adjust thresholds, and hierarchically schedule tasks and manage connection pools; Ensure data transmission integrity: verify data, perform tiered retransmission, and use mirrored encrypted caching to ensure transmission integrity and data security; End-to-end monitoring and self-healing: pull metrics and store them for a preset duration, backtrack and verify logs, and trigger alarms for deviations to achieve end-to-end monitoring and self-healing; Intelligent diagnostics and performance tuning: identifying bottlenecks through data entry points and optimizing matching schemes; dynamic scaling and resource prediction, generating reports to track iterations and continuously improve performance; Emergency recovery and fault tolerance: paginated data acquisition and disaster recovery storage, quarterly test recovery; primary and backup heterogeneous complementarity, automatic update synchronization.

Images