Financial big data management system and method

Abstract

The invention particularly relates to a financial big data management system and method, and relates to the technical field of financial data management, and the method comprises the steps: a data collection dynamic optimization module is configured to carry out adaptation, parallel task scheduling and connection pool management on a timeout threshold based on a regulatory factor, and solves a problem of timeout of a collection script; a real-time data verification and supplementary transmission module; a full-link monitoring and self-healing module; an intelligent performance diagnosis and tuning module; and an emergency data recovery and fault tolerance module. According to the method, the regulation factor and the Sigmoid function of the data acquisition module are dynamically optimized, intelligent expansion and contraction of the overtime threshold along with the system load are realized, and the script overtime problem is solved by combining hierarchical task scheduling and connection pool management; the real-time data verification and supplementary transmission module guarantees the integrity and security of data transmission through strict verification, a hierarchical transmission strategy and encrypted caching, effectively reduces data blank and loss, and provides a high-quality data basis for financial service analysis.

Description

Technical Field

[0001] The present invention relates to the technical field of financial data management, and in particular to a financial big data management system and method. Background Art

[0002] With the rapid development of the financial industry, financial data has shown explosive growth, which has put higher requirements on the management of financial big data.

[0003] At present, there are many problems in the management of financial big data: script timeouts are prone to occur during data collection, resulting in data loss; during data transmission, data is often incomplete due to insufficient verification or unreasonable transmission strategies; there is a lack of effective monitoring and self-healing mechanisms for the entire data flow link, and monitoring charts are prone to interruptions; system performance is difficult to continuously optimize, and problems such as script timeouts occur repeatedly; in the face of extreme situations, there is a lack of effective emergency recovery and fault tolerance measures, and the continuity and security of financial data cannot be guaranteed.

[0004] These problems seriously affect the reliability of financial data, and in turn have an adverse impact on risk monitoring and decision-making in financial operations.

[0005] Therefore, a financial big data management system and method are needed to address the above-mentioned problems. Summary of the Invention

[0006] The purpose of the present invention is to propose a financial big data management system and method in order to solve the above problems.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions: A financial big data management system, comprising: The data collection dynamic optimization module is configured to adapt the timeout threshold based on the adjustment factor, parallel task scheduling and connection pool management to solve the collection script timeout problem; The real-time data verification and retransmission module is configured to rely on real-time verification, hierarchical retransmission strategy and encrypted cache to solve the problem of incomplete data upload and ensure data integrity and transmission reliability; The full-link monitoring and self-healing module is configured to eliminate gaps in monitoring charts with the help of multi-dimensional monitoring, data self-healing mechanisms, and multi-level alarms; The intelligent performance diagnosis and tuning module is configured to continuously optimize system performance through scripted bottleneck analysis, dynamic resource allocation, and periodic optimization suggestions; The emergency data recovery and fault-tolerance module is configured to cope with data interruptions in extreme situations by relying on breakpoint backtracking, multi-level backup, and active-active switching.

[0008] Preferably, the data acquisition dynamic optimization module specifically includes: Extract the running time data of data collection tasks from the database, analyze it to obtain the adjustment factor, and optimize the data collection timeout threshold based on the adjustment factor to cope with subsequent collection tasks; Determine the orchestration configuration of Kubernetes containers and prioritize subtasks; Determine the management of the data connection pool, including troubleshooting invalid connections after determining the connection pool parameter configuration.

[0009] Preferably, the process of obtaining the regulating factor includes: Extract the time period corresponding to each data collection task from the database, and extract the time period corresponding to the interval with blank data from it, and record it as the marked time period; Get the CPU usage during the marked time period, preset a CPU usage threshold, and divide the obtained CPU usage by the CPU usage threshold to get the usage percentage; Get the time it takes for the script to read the performance files in the disk directory, and divide the obtained time by the preset standard reading time to get the reading occupancy; Obtain the congestion time of the receiving queue of the monitoring platform, extract the maximum congestion time from it, and divide it by the allowed congestion time to obtain the excess limit; The reference coefficient is obtained by weighted summing the usage occupancy, reading occupancy and over-limit; Obtain the timeout threshold and reference coefficient corresponding to each marked time period; Extract the time period with heavy data collection tasks and no data blank area from the database, record it as the normal time period, and obtain the reference coefficient corresponding to the normal time period, record it as the standard reference coefficient; Obtain the standard reference coefficients of each normal time period in turn, and calculate the mean to obtain the standard reference mean coefficient; The reference coefficient of each marked time period is divided by the standard reference coefficient to obtain a preliminary adjustment factor; the adjustment factor is obtained by averaging all the preliminary adjustment factors.

[0010] Preferably, a nonlinear relationship function is constructed based on the adjustment factor and the timeout threshold time, and the timeout threshold is optimized accordingly; Nonlinear relationship function: ; in The optimized timeout threshold time; The timeout threshold before optimization; k is the slope parameter; e is a natural constant; is the regulating factor.

[0011] Preferably, the real-time data verification and retransmission module specifically includes: Perform integrity checks on data fields. If any anomalies are found during the verification process, the system will send detailed information about the abnormal data to the data acquisition module. A hierarchical retransmission strategy is implemented based on the priority of the subtasks. A mirroring and striping combination is used to split the data into two halves, storing them in two sets of mirror disks respectively. The cached data is encrypted.

[0012] Preferably, the full-link monitoring and self-healing module specifically includes: Pull monitoring indicator data from the exporter of each module at a preset time interval; When extracting data from log files, threads are dynamically created based on the amount of data, with each thread handling the extraction task of one log file. The extracted data is compared with the monitoring data of the adjacent time period and its deviation rate is calculated. If the absolute value of the deviation rate exceeds 5%, the data is considered abnormal and needs to be re-extracted and verified. For the re-extracted data, the deviation rate is calculated again. If it still exceeds 5%, it is marked as suspicious data and notified for manual review.

[0013] Preferably, the intelligent performance diagnosis and tuning module specifically includes: When the script starts, the Profiler agent is loaded through the javaagent parameter to record the entry and exit time, parameter values, and return value information of each function; The recorded data is stored in a local file in binary format. One file is generated every hour and the data is converted into a visual chart. When the script memory usage exceeds the preset value for a specified period of time, HPA will calculate the number of replicas that need to be added based on the current memory usage; ARIMA is used to forecast resource usage and process time series data with periodicity and trend.

[0014] Preferably, the emergency data recovery and fault tolerance module specifically includes: When sending a supplementary data collection request to a data source, the request parameters include page number, amount of data per page, start time, and end time; After receiving the request, the data source server queries the corresponding data based on these parameters and includes the total number of pages and the current page data in the response. After receiving the response, the system checks whether the current page data is complete. If complete, it continues to request the next page until all the data is obtained. Data is distributed and stored on the hard disk, and parity information is calculated and stored.

[0015] A financial big data management method comprises the following steps: Dynamically optimize data collection: Use load data to calculate adjustment factors, dynamically adjust thresholds, hierarchically schedule tasks, and manage connection pools; Ensure data transmission integrity: verify data, retransmit in stages, and use encrypted mirroring cache to ensure transmission integrity and data security; Full-link monitoring and self-healing: Pull indicators and store them for a preset period of time, perform log backtracking verification, and trigger alarms for deviations, enabling full-link monitoring and self-healing. Intelligent diagnosis and performance tuning: locating points to identify bottlenecks and optimize matching solutions; dynamic scaling and resource prediction, reporting and tracking iterations, and continuous performance improvement; Emergency recovery and fault tolerance: paging recovery and disaster recovery storage, quarterly recovery testing; primary and standby heterogeneous complementation, and automatic synchronization of updates.

[0016] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention dynamically optimizes the adjustment factor and Sigmoid function of the data acquisition module to achieve intelligent scaling of the timeout threshold with system load. Combined with hierarchical task scheduling and connection pool management, it solves the script timeout problem. The real-time data verification and retransmission module ensures the integrity and security of data transmission through strict verification, hierarchical transmission strategy and encrypted caching, effectively reducing data gaps and loss, and providing a high-quality data foundation for financial business analysis.

[0017] 2. The present invention uses a full-link monitoring and self-healing module to achieve visual tracking of all aspects of data flow and automatic response to anomalies, eliminating monitoring gaps; the intelligent performance diagnosis and tuning module continuously analyzes bottlenecks and dynamically optimizes resources to reduce failures from the root; the emergency recovery and fault-tolerant module uses multi-level backup, master-slave switching and regular testing to ensure continuous data availability in extreme situations, comprehensively improving the system's ability to cope with complex loads and sudden risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 This is a system structure diagram of the present invention. DETAILED DESCRIPTION

[0019] Several embodiments of the present application will be described in more detail below with reference to the accompanying drawings so that those skilled in the art can implement the present application. The present application can be embodied in many different forms and for many different purposes and should not be limited to the embodiments described herein. These embodiments are provided to make the present application comprehensive and complete and to fully convey the scope of the present application to those skilled in the art. The embodiments do not limit the present application.

[0020] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which this application belongs. It will be further understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the relevant art and / or the context of this specification, and will not be interpreted in an idealized or overly formal sense unless explicitly defined as such herein. Example 1:

[0021] The specific implementation method is combined with the attached Figure 1 Provide detailed explanation.

[0022] Attachment Figure 1 A structural block diagram of a financial big data management system provided by an embodiment of the present invention shows the connection relationship between the data acquisition dynamic optimization module and the emergency data recovery and fault tolerance module, and marks the main functional interaction process of each module.

[0023] In this embodiment, it includes: The data collection dynamic optimization module is configured to adapt the timeout threshold based on the adjustment factor, parallel task scheduling and connection pool management, solve the collection script timeout problem, and improve the efficiency and stability of periodic collection tasks; Specifically include: Extract the running time data of data collection tasks from the database, analyze it to obtain the adjustment factor, and optimize the data collection timeout threshold based on the adjustment factor to cope with subsequent collection tasks; The process of obtaining the adjustment factor includes: Extract the time period corresponding to each data collection task from the database, and extract the time period corresponding to the interval with blank data from it, and record it as the marked time period; Get the CPU usage during the marked time period, preset a CPU usage threshold, and divide the obtained CPU usage by the CPU usage threshold to get the usage percentage; Convert the CPU load into a calculable proportional value (e.g., CPU usage = 1 when fully utilized, close to 0 when idle), clearly reflecting the computing resource pressure during data collection. Get the time it takes for the script to read the performance files in the disk directory, and divide the obtained time by the preset standard reading time to get the reading occupancy; Quantify the fluctuations in the time it takes for scripts to read performance files (for example, if the read usage is > 1 when the disk is busy, it means that the reading is slow); Obtain the congestion time of the receiving queue of the monitoring platform, extract the maximum congestion time from it, and divide it by the allowed congestion time to obtain the excess limit; Break down the ambiguous reasons for data gaps into specific quantitative indicators of CPU, disk, and queues, making it easier to pinpoint the link that causes data collection failure. The reference coefficient is obtained by weighted summing the usage occupancy, reading occupancy and over-limit; Preset weight factors for usage occupancy, read occupancy, and over-limit, and multiply usage occupancy, read occupancy, and over-limit by their corresponding weight factors to obtain a reference coefficient; Obtain the timeout threshold and reference coefficient corresponding to each marked time period; Extract the time period with heavy data collection tasks and no data blank area from the database, record it as the normal time period, and obtain the reference coefficient corresponding to the normal time period, record it as the standard reference coefficient; Use the standard reference coefficient of the normal time period as a benchmark so that the adjustment factor can compare the difference between the load during the data blank period and the normal load; The adjustment factor is a dynamic value based on historical normal data and current load quantification, which is more in line with the actual operating status of the system; Obtain the standard reference coefficients of each normal time period in turn, and calculate the mean to obtain the standard reference mean coefficient; The reference coefficient of each marked time period was divided by the standard reference coefficient to obtain the preliminary adjustment factor; The adjustment factor is obtained by averaging all the preliminary adjustment factors; Reduce false positives due to data gaps (thresholds are relaxed during high loads to collect more real data) and false positives due to data gaps (thresholds are tightened during low loads to detect collection script anomalies promptly). Improved integrity and accuracy of data collection provide a more reliable data foundation for downstream analysis (such as risk monitoring and business decision-making); There is no need to frequently adjust the timeout threshold manually (the traditional method relies on operation and maintenance experience and cannot respond to load changes in real time). The system automatically optimizes. A nonlinear relationship function is constructed based on the adjustment factor and the timeout threshold time, and the timeout threshold is optimized accordingly; Nonlinear relationship function: ; in The optimized timeout threshold time; The timeout threshold before optimization; k is the slope parameter (usually 1 to 5, controlling the steepness of the curve; the larger k is, the steeper the curve is); e is a natural constant (about 2.718); is a regulating factor; Financial system loads often exhibit a critical mass (e.g., after CPU utilization reaches 80%, task duration increases exponentially due to resource competition). The S-shaped curve of the Sigmoid function perfectly captures this nonlinear relationship: When the load is normal (regulation factor ≈1): The slope of the middle section of the curve is gentle, and the threshold value changes slightly (for example, when the adjustment factor fluctuates from 1.0 to 1.1, the threshold value is only slightly adjusted). This avoids frequent threshold fluctuations caused by small load fluctuations and ensures system stability. When the load is extreme (regulation factor ≥ or ≤1): The slopes of the curve are steep at both ends, and the threshold adjusts quickly (for example, when the adjustment factor jumps from 1.5 to 2.0, the threshold is significantly relaxed). This allows for timely adaptation to the increased task duration under extreme loads and reduces the probability of normal tasks being mistakenly timed out due to overly strict thresholds. In traditional linear models, under extreme loads, the threshold may be out of control due to excessively large or small adjustment factors (for example, linear amplification causes the threshold to increase without an upper limit; the Sigmoid function naturally has saturation characteristics: No matter how extreme the regulating factor Goes to infinity or tends to 0, and the threshold will eventually approach 1.5 or 0.5 ; Limit extreme thresholds from a mathematical perspective to avoid system crashes due to unreasonable thresholds; Financial transactions (such as payments and clearing) have a very low tolerance for false positives. If the threshold is too strict, normal transactions may be deemed to have timed out due to high load, leading to customer complaints or financial losses. The Sigmoid model automatically relaxes the threshold under high load to ensure that legitimate, time-consuming tasks are not mistakenly terminated, ensuring a smooth transaction process. During low load (such as batch processing at night), the Sigmoid function will slightly tighten the threshold ( When the value is less than 1, the slope of the middle section of the curve is gentle and the threshold is moderately lowered), which promotes faster task execution, reduces idle resources, and improves the overall efficiency of the system; The financial system places high demands on model interpretability (to meet regulatory requirements for risk control traceability). The Sigmoid function has clear parameters (slope k, natural constant e), and the adjustment factor can be calculated using intuitive metrics such as CPU usage and queue length. This makes it easier for technical teams to explain the logic of threshold changes and pass compliance audits. The slope k is a configurable control knob. If the financial system requires stability (such as a core trading system), you can increase k (for example, set it to 2) to make the curve steeper and allow the threshold to converge to a safe range more quickly under extreme loads. If the system requires sensitive response to loads (such as a real-time market information system), you can decrease k (for example, set it to 0.8) to make the middle section of the curve flatter, balancing threshold stability and response speed. Determine the orchestration configuration of Kubernetes containers and prioritize subtasks; Specific configuration of Kubernetes container orchestration: Each subtask container has a CPU limit of 0.5 cores, a memory limit of 512MB, a requested CPU of 0.2 cores, and a requested memory of 256MB. Container health checks are implemented by executing a script that checks the running status of the subtask every 5 seconds. If the subtask is found to be unresponsive after three consecutive checks, the container is automatically restarted. Subtask priority classification criteria: P0-level subtasks correspond to data collection related to customer fund changes, such as deposits, withdrawals, and transfers; P1-level subtasks correspond to the collection of customer account information, such as account balances and transaction records; P2-level subtasks correspond to the collection of system operation logs. During resource scheduling, P0-level subtasks have a scheduling priority 30% higher than P1-level subtasks, and P1-level subtasks have a scheduling priority 20% higher than P2-level subtasks. Determine the management of the data connection pool, including troubleshooting invalid connections after determining the connection pool parameter configuration; Parameter configuration of Apache Commons DBCP connection pool: Set maxWait to 10000 milliseconds, that is, when there is no available connection in the connection pool, the maximum waiting time of the request thread is 10 seconds; set testOnBorrow to true, that is, each time a connection is obtained from the connection pool, the connection validity test is performed; set timeBetweenEvictionRunsMillis to 300000 milliseconds, that is, the idle connection recovery thread is run every 5 minutes; When an invalid connection is detected, the system automatically records the corresponding data source IP address, port number, failure time, and other information, and generates a troubleshooting work order. After receiving the work order, the operation and maintenance personnel remotely log in to the data source server to check the network connection, database service status, etc. After troubleshooting the problem, they will feedback the processing results to the system, and the system will record the results in the knowledge base for subsequent reference. The real-time data verification and retransmission module is configured to rely on real-time verification, hierarchical retransmission strategy and encrypted cache to solve the problem of incomplete data upload and ensure data integrity and transmission reliability; Specifically include: Perform integrity checks on data fields. For date fields, verify that the format is "yyyy-MM-ddHH:mm:ss" and that the date is within a reasonable range (e.g., not earlier than the system launch date). For string fields, verify that the length is within the specified range. For example, the length of the customer name field is limited to 1 to 50 characters. If any anomalies are found during the verification process, the system will send detailed information about the abnormal data (including field name, error type, original value, etc.) to the data collection module. Implement corresponding hierarchical retransmission strategy based on subtask priority; Network configuration for the dedicated P0 data transmission channel: This channel uses a separate network card with an IP address range of 192.168.100.0 / 24, a gateway of 192.168.100.1, and DNS servers of 8.8.8.8 and 8.8.4.4. The guaranteed network bandwidth is 100 Mbps, and QoS (Quality of Service) settings ensure that traffic on this channel takes precedence over other channels. Batch compression and transmission: P1 and P2 data are batch compressed every 30 seconds. The system automatically scans the local cache directory for unuploaded P1 and P2 data files and groups them in chronological order, with each group containing up to 100 files. During compression, the ZIP compression level is set to 5 (0 to 9, with higher levels increasing the compression rate but increasing the time required). The compressed package generated after compression contains timestamp and level information. Using a combination of mirroring and striping, data is divided into two halves and stored in two sets of mirrored disks. This configuration increases read and write speeds by approximately three times compared to a single SSD. Furthermore, if one of the disks fails, data can be quickly restored through the mirrored disks in less than five minutes. And encrypt the cached data; For example, the AES-256 encryption algorithm's key is stored and managed in a hardware security module (HSM), with the key automatically rotated every seven days. During each encryption and decryption operation, the system retrieves the key from the HSM and immediately destroys the key information in memory upon completion. Only authenticated system modules can access the key by calling the HSM's interface, and each call is recorded in the audit log. The full-link monitoring and self-healing module is configured to eliminate gaps in monitoring charts and achieve visual tracking and abnormal response of the entire data flow through multi-dimensional monitoring, data self-healing mechanism and multi-level alarm; Specifically include: Monitoring metric data is pulled from each module's exporter at preset intervals and stored for 30 days. The configuration file defines various monitoring rules, such as a histogram of script runtime (buckets set to [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] seconds) and a gauge for data upload success rates. Multiple dashboards are created in the Grafana visualization tool, each corresponding to a module. A drop-down menu allows you to select different time ranges and data sources for viewing. Thresholds for monitoring metrics are also set. The basis for setting the monitoring indicator threshold is as follows: The threshold for the data upload success rate is based on the historical data statistics of the past three months; For example, during the past three months, the average data upload success rate was 95%, with the lowest being 88%. Setting a threshold of 90% ensures a certain level of fault tolerance while also detecting anomalies in a timely manner. Thresholds for other indicators are set in a similar manner, combining historical data with business needs. When extracting data from log files, threads are dynamically created based on the amount of data, with each thread handling the extraction task of a log file. The thread pool uses the CallerRunsPolicy rejection policy, meaning that when the queue is full, the thread that submitted the task automatically executes it to avoid task loss. The extracted data is compared with the monitoring data from adjacent time periods to calculate the deviation rate. The deviation rate is calculated as (extracted data - adjacent data) / adjacent data × 100%. If the absolute value of the deviation rate exceeds 5%, the data is considered abnormal and needs to be re-extracted and verified. The deviation rate of the re-extracted data is calculated again. If it still exceeds 5%, it is marked as suspicious data and notified for manual review. Call the open APIs of WeChat Work and DingTalk to encapsulate alarm information into a specific format and send it to the corresponding group or individual. For example, the alarm information sent to WeChat Work includes fields such as title, alarm level, occurrence time, detailed description, and handling suggestions. The alarm level is indicated by red (critical), yellow (warning), and blue (prompt). The system will record the response time and processing completion time of each alarm and generate an alarm response report every month; The intelligent performance diagnosis and tuning module is configured to continuously optimize system performance through script bottleneck analysis, dynamic resource allocation, and periodic optimization suggestions, thereby reducing the occurrence of problems such as script timeouts from the root. Specifically include: When the script is started, the Profiler agent is loaded through the javaagent parameter. The agent records the entry and exit time, parameter values, and return value information of each function during the script execution. The recorded data is stored in a local file in binary format, with one file generated every hour. Each file is approximately 100MB in size. Dedicated analysis tools can be used to convert this data into visual graphs, such as a function call relationship graph, which uses nodes to represent functions and edges to represent call relationships. The thickness of the edges indicates the number of calls. The analysis tool first calculates the total time consumption and proportion of each function, identifying functions with a proportion exceeding 10% as key optimization targets. It then matches the corresponding optimization solution based on the function type and implementation method, drawing on best practices in the codebase. For example, for JSON parsing code that takes a long time, the analysis tool will compare the performance data of the Jackson library and the Gson library. The Jackson library is about 20% faster than the Gson library when parsing large JSON data. Therefore, it is recommended to use the Jackson library instead of the Gson library. When the script memory usage exceeds the preset value for a specified period of time, HPA will calculate the number of replicas to be added based on the current memory usage. The calculation formula is: target number of replicas = current number of replicas × (current memory usage / target memory usage), rounded up; For example, if the CPU usage peak exceeds 90% for 3 minutes, increase the Pod's CPU resource quota by 0.1 cores at a time until the CPU usage falls below 80%. ARIMA (Autoregressive Integrated Moving Average) is used to forecast resource usage. This model can better handle time series data with periodicity and trend. The model parameters were determined using a grid search method: p = 3, d = 1, q = 2, and a prediction step of 168 hours (1 week). The model was updated daily based on the previous day's actual resource usage data to improve prediction accuracy. In addition to historical data and optimization suggestions, it also includes performance comparison charts before and after optimization, estimated resources and time required to implement the optimization suggestions, possible risks, and countermeasures. For example, in the section that recommends starting a backup collection node 30 minutes in advance during peak hours, the chart shows the decrease in script timeout rate after starting the backup node, as well as the server resources and labor costs required to start the backup node. Each optimization suggestion has a unique number. The system records its implementation time, implementer, and post-implementation performance data. By comparing performance data before and after implementation, the optimization effect is calculated, such as the percentage reduction in script timeout rate and the percentage increase in data upload success rate. For suggestions with poor implementation results, the system analyzes the reasons, which may be unreasonable suggestions themselves or problems in the implementation process, and then formulates a new optimization plan based on the analysis results. The emergency data recovery and fault-tolerance module is configured to handle data interruptions in extreme situations by leveraging breakpoint backtracking, multi-level backup, and active-active switching, ensuring the continuity and security of financial data. Specifically include: When sending a supplementary data collection request to a data source, the request parameters include page number, amount of data per page, start time, and end time; After receiving the request, the data source server queries the corresponding data based on these parameters and includes the total number of pages and the current page data in the response. After receiving the response, the system checks whether the current page data is complete. If complete, it continues to request the next page until all the data is obtained. A token bucket algorithm is used to control the frequency of data source access. The token bucket has a capacity of 100 and generates 10 tokens per second. Each time a request is sent to a data source, a token must be obtained from the token bucket. If no token is available, the request waits until a token becomes available or the wait time exceeds 10 seconds (at which point the request is abandoned and a failure message is logged). This ensures that the number of requests per second does not exceed 10, complying with the data source's access frequency limit. Data is distributed and stored on the hard disks according to a certain algorithm, and parity information is calculated and stored at the same time. When one of the hard disks fails, the system can recover the data of the failed hard disk based on the data and parity information of the other hard disks. Data recovery testing is conducted quarterly during a non-peak business period. Data is restored from the local SSD cache, distributed file system, and cloud cold backups at a random backup time. After recovery is complete, the recovered data is compared with the original data for consistency and accuracy. Metrics such as recovery time and recovery success rate are recorded during the test, and any issues identified are promptly corrected. The main script is developed in Java, based on the SpringBoot framework, and uses multithreading for data collection. The backup script is developed in Python, based on the Scrapy framework, and uses asynchronous IO for data collection. Both technical architectures have their own advantages when processing different types of data sources. Java scripts have more stable performance when handling highly concurrent database access, while Python scripts are more efficient when processing web crawler data sources. When the master script is updated, the developer submits the code to the Git repository. The system automatically detects the code change and triggers the synchronization update process of the backup script. First, the updated content of the master script is converted into the corresponding code of the backup script (using a custom code conversion tool). The converted code is then compiled and unit tested. After passing the test, it is deployed to the runtime environment of the backup script. After the deployment is complete, compatibility testing is performed to simulate the master-slave switching process to ensure that the backup script can collect data normally. Example 2:

[0024] A financial big data management method includes the following parts: Dynamically optimize data collection: Use load data to calculate adjustment factors, dynamically adjust thresholds, hierarchically schedule tasks, and manage connection pools; Ensure data transmission integrity: verify data, retransmit in stages, and use encrypted mirroring cache to ensure transmission integrity and data security; Full-link monitoring and self-healing: Pull indicators and store them for a preset period of time, perform log backtracking verification, and trigger alarms for deviations, enabling full-link monitoring and self-healing. Intelligent diagnosis and performance tuning: locating points to identify bottlenecks and optimize matching solutions; dynamic scaling and resource prediction, reporting and tracking iterations, and continuous performance improvement; Emergency recovery and fault tolerance: paging recovery and disaster recovery storage, quarterly recovery testing; primary and standby heterogeneous complementation, and automatic synchronization of updates.

[0025] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters in the formulas are set by technicians in this field according to actual conditions.

[0026] The above description is merely illustrative of certain exemplary embodiments of the present invention. It goes without saying that those skilled in the art will be able to modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and description are illustrative in nature and should not be construed as limiting the scope of protection of the claims.

[0027] It should be noted that, in this document, if there are relational terms such as first and second, etc., they are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device that includes the element.

[0028] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0029] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0030] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0031] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0032] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0033] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

[0034] The above description is merely illustrative of certain exemplary embodiments of the present invention. It goes without saying that those skilled in the art will be able to modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and description are illustrative in nature and should not be construed as limiting the scope of protection of the claims.

Claims

1. A financial big data management system, characterized in that: include: The data collection dynamic optimization module is configured to adapt the timeout threshold based on the adjustment factor, parallel task scheduling and connection pool management to solve the collection script timeout problem; The real-time data verification and retransmission module is configured to rely on real-time verification, hierarchical retransmission strategy and encrypted cache to solve the problem of incomplete data upload and ensure data integrity and transmission reliability; The full-link monitoring and self-healing module is configured to eliminate gaps in monitoring charts with the help of multi-dimensional monitoring, data self-healing mechanisms, and multi-level alarms; The intelligent performance diagnosis and tuning module is configured to continuously optimize system performance through scripted bottleneck analysis, dynamic resource allocation, and periodic optimization suggestions; The emergency data recovery and fault-tolerance module is configured to cope with data interruptions in extreme situations by relying on breakpoint backtracking, multi-level backup, and active-active switching.

2. A financial big data management system according to claim 1, characterized in that: Data acquisition dynamic optimization module, specifically including: Extract the running time data of data collection tasks from the database, analyze it to obtain the adjustment factor, and optimize the data collection timeout threshold based on the adjustment factor to cope with subsequent collection tasks; Determine the orchestration configuration of Kubernetes containers and prioritize subtasks; Determine the management of the data connection pool, including troubleshooting invalid connections after determining the connection pool parameter configuration.

3. A financial big data management system according to claim 2, characterized in that: The process of obtaining the adjustment factor includes: Extract the time period corresponding to each data collection task from the database, and extract the time period corresponding to the interval with blank data from it, and record it as the marked time period; Get the CPU usage during the marked time period, preset a CPU usage threshold, and divide the obtained CPU usage by the CPU usage threshold to get the usage percentage; Get the time it takes for the script to read the performance files in the disk directory, and divide the obtained time by the preset standard reading time to get the reading occupancy; Obtain the congestion time of the receiving queue of the monitoring platform, extract the maximum congestion time from it, and divide it by the allowed congestion time to obtain the excess limit; The reference coefficient is obtained by weighted summing the usage occupancy, reading occupancy and over-limit; Obtain the timeout threshold and reference coefficient corresponding to each marked time period; Extract the time period with heavy data collection tasks and no data blank area from the database, record it as the normal time period, and obtain the reference coefficient corresponding to the normal time period, record it as the standard reference coefficient; Obtain the standard reference coefficients of each normal time period in turn, and calculate the mean to obtain the standard reference mean coefficient; The reference coefficient of each marked time period is divided by the standard reference coefficient to obtain a preliminary adjustment factor; the adjustment factor is obtained by averaging all the preliminary adjustment factors.

4. A financial big data management system according to claim 3, characterized in that: A nonlinear relationship function is constructed based on the adjustment factor and the timeout threshold time, and the timeout threshold is optimized accordingly; Nonlinear relationship function: ; in The optimized timeout threshold time; The timeout threshold before optimization; k is the slope parameter; e is a natural constant; is the regulating factor.

5. A financial big data management system according to claim 1, characterized in that: Real-time data verification and retransmission module, specifically including: Perform integrity checks on data fields. If any anomalies are found during the verification process, the system will send detailed information about the abnormal data to the data acquisition module. A hierarchical retransmission strategy is implemented based on the priority of the subtasks. A mirroring and striping combination is used to split the data into two halves, storing them in two sets of mirror disks respectively. The cached data is encrypted.

6. A financial big data management system according to claim 1, characterized in that: Full-link monitoring and self-healing module, including: Pull monitoring indicator data from the exporter of each module at a preset time interval; When extracting data from log files, threads are dynamically created based on the amount of data, with each thread handling the extraction task of one log file. The extracted data is compared with the monitoring data of the adjacent time period and its deviation rate is calculated. If the absolute value of the deviation rate exceeds 5%, the data is considered abnormal and needs to be re-extracted and verified. For the re-extracted data, the deviation rate is calculated again. If it still exceeds 5%, it is marked as suspicious data and notified for manual review.

7. A financial big data management system according to claim 1, characterized in that: Intelligent performance diagnosis and tuning module, including: When the script starts, the Profiler agent is loaded through the javaagent parameter to record the entry and exit time, parameter values, and return value information of each function; The recorded data is stored in a local file in binary format. One file is generated every hour and the data is converted into a visual chart. When the script memory usage exceeds the preset value for a specified period of time, HPA will calculate the number of replicas that need to be added based on the current memory usage; ARIMA is used to forecast resource usage and process time series data with periodicity and trend.

8. A financial big data management system according to claim 1, characterized in that: Emergency data recovery and fault tolerance module, specifically including: When sending a supplementary data collection request to a data source, the request parameters include page number, amount of data per page, start time, and end time; After receiving the request, the data source server queries the corresponding data based on these parameters and includes the total number of pages and the current page data in the response. After receiving the response, the system checks whether the current page data is complete. If complete, it continues to request the next page until all the data is obtained. Data is distributed and stored on the hard disk, and parity information is calculated and stored.

9. A financial big data management method, according to any one of claims 1 to 8, characterized in that: The following steps are involved: Dynamically optimize data collection: Use load data to calculate adjustment factors, dynamically adjust thresholds, hierarchically schedule tasks, and manage connection pools; Ensure data transmission integrity: verify data, retransmit in stages, and use encrypted mirroring cache to ensure transmission integrity and data security; Full-link monitoring and self-healing: Pull indicators and store them for a preset period of time, perform log backtracking verification, and trigger alarms for deviations, enabling full-link monitoring and self-healing. Intelligent diagnosis and performance tuning: locating points to identify bottlenecks and optimize matching solutions; dynamic scaling and resource prediction, reporting and tracking iterations, and continuous performance improvement; Emergency recovery and fault tolerance: paging recovery and disaster recovery storage, quarterly recovery testing; primary and standby heterogeneous complementation, and automatic synchronization of updates.

Images