Bucket dividing regulation and control method and device for financial data warehouse and electronic equipment

Abstract

The invention discloses a bucket dividing regulation and control method and device for a financial data warehouse and electronic equipment, and relates to the technical field of artificial intelligence or other related fields, and the method comprises the steps: collecting a time sequence performance set of the financial data warehouse through a distributed collection system; multi-level feature extraction is carried out on the time sequence performance set by adopting a sliding window mechanism to obtain a time sequence feature set, and the categories of time sequence features comprise instantaneous fluctuation features, periodic trend features and steady state baseline features; performing association analysis on the time sequence feature set based on a decision tree integration algorithm by taking an expected query performance requirement as a target to obtain a bucket-dividing adjustment strategy; and bucket index reconstruction is carried out on the financial data warehouse based on the bucket division adjustment strategy, and the financial data warehouse after index updating is obtained. Through the method and the device, the technical problem that the system performance is degraded due to the fact that a static bucket dividing mechanism of a real-time data warehouse cannot dynamically adapt to space-time imbalance of data loads in a financial service scene in the related technology is solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology or other related fields. Specifically, it relates to a binning control method and apparatus, and electronic equipment for a financial data warehouse. Background Technology

[0002] In the field of financial data warehousing, traditional bucketing mechanisms generally adopt a static bucketing strategy based on historical data distribution. Data is evenly distributed into a fixed number of storage buckets through predefined hash rules. This relies on the capacity planning model designed in the early stages of business operations and adjusts the bucketing parameters through periodic (usually daily or weekly) offline analysis.

[0003] However, financial business scenarios exhibit significant spatiotemporal imbalances in data load: on the one hand, different business lines experience pulsed write peaks during trading hours, such as a surge in order data during market opening; on the other hand, complex analytical queries during regulatory report generation periods consume concentrated computing resources. Traditional static bucketing mechanisms cannot detect dynamically changing load characteristics, leading to a double waste of resources: I / O contention in hot buckets and idle resources in cold buckets. This, in turn, causes system performance degradation issues such as increased query response latency and decreased write throughput.

[0004] Existing technologies attempt to alleviate the above problems by increasing hardware resource redundancy or manually adjusting parameters, but these methods have fundamental limitations: hardware expansion solutions can only temporarily mask performance bottlenecks and cannot solve the core contradiction of mismatch between data distribution and access patterns; manual parameter adjustment is difficult to implement due to the real-time requirements of financial business and relies on the experience and judgment of maintenance personnel, resulting in significant adjustment delays; more importantly, none of these methods have established a dynamic correlation mechanism between bucket parameters and real-time performance indicators, and therefore cannot capture the minute-level or even second-level load fluctuation characteristics in typical financial scenarios such as high-frequency trading and real-time risk control.

[0005] The aforementioned technical deficiencies mean that existing data warehouse systems, when faced with the spatiotemporal load imbalances unique to financial businesses, cannot guarantee the stable performance required by service level agreements, nor can they maximize the effective utilization of infrastructure resources.

[0006] There is currently no effective solution to the above problems. Summary of the Invention

[0007] The main purpose of this application is to provide a binning control method, device, and electronic device for financial data warehouses, so as to at least solve the technical problem that the static binning mechanism of real-time data warehouses in related technologies cannot dynamically adapt to the spatiotemporal imbalance of data load in financial business scenarios, resulting in system performance degradation.

[0008] To achieve the above objectives, according to one aspect of this application, a bucketing control method for a financial data warehouse is provided. The method includes: collecting a time-series performance set of the financial data warehouse through a distributed acquisition system, wherein the time-series performance set records performance indicators for M time steps, and the categories of the performance indicators include: business system performance, data warehouse table performance, and computing resource performance, where M is a preset value; performing multi-level feature extraction on the time-series performance set using a sliding window mechanism to obtain a time-series feature set, wherein the categories of time-series features recorded in the time-series feature set include: instantaneous fluctuation features, periodic trend features, and steady-state baseline features; performing association analysis on the time-series feature set based on a decision tree ensemble algorithm, with the expected query performance requirements as the target, to obtain a bucketing adjustment strategy; and reconstructing the bucket index of the financial data warehouse based on the bucketing adjustment strategy to obtain the index-updated financial data warehouse.

[0009] Furthermore, the step of collecting the time-series performance set of the financial data warehouse through a distributed acquisition system includes: performing data acquisition according to a preset period using a lightweight acquisition agent pre-deployed in the financial data warehouse to obtain raw acquisition data, wherein the lightweight acquisition agent is configured to monitor the data query engine of the financial data warehouse, as well as the financial business system and resource manager connected to the financial data warehouse; preprocessing the raw acquisition data to obtain the time-series performance set, wherein the preprocessing includes the following operations: time window aggregation, data format conversion, and unit of measurement normalization.

[0010] Further, the step of using a sliding window mechanism to perform multi-level feature extraction on the time-series performance set to obtain a time-series feature set includes: obtaining N preset calculation parameter groups corresponding to the expected query performance requirements and the sliding window mechanism, wherein each preset calculation parameter group indicates a different calculation granularity, and each preset calculation parameter group includes at least a time window length and a sliding calculation step size, and N is greater than or equal to 3; calculating the standard deviation of the query response time of the financial data warehouse based on the first parameter group and the time-series performance set, and using the standard deviation as the instantaneous fluctuation feature; based on the second parameter group and the... The time-series performance set is used to fit the data write rate of the financial data warehouse using the least squares method to obtain the trend curve of the data write rate. The slope change value of the trend curve is used as the periodic trend feature. The second parameter group indicates a larger computational granularity than the first parameter group. Based on the third parameter group and the time-series performance set, the resource utilization rate of each computing node in the financial data warehouse is calculated. The resource moving average is calculated based on the resource utilization rate. The resource moving average is used as the steady-state baseline feature. The third parameter group indicates a larger computational granularity than the second parameter group.

[0011] Furthermore, the step of performing association analysis on the time-series feature set based on the decision tree ensemble algorithm to obtain the bucketing adjustment strategy, with the expected query performance requirements as the target, includes: obtaining an association rule document, wherein the association rule document records the mapping relationship between each query performance indicator and the corresponding time-series feature; calculating the expected time-series feature based on the expected query performance requirements and the association rule document; analyzing the degree of deviation between the expected time-series feature and each time-series feature in the time-series feature set based on the gradient boosting decision tree ensemble algorithm to obtain the deviation analysis result; and determining the bucketing adjustment strategy based on the deviation analysis result.

[0012] Furthermore, the deviation analysis results record the cosine similarity between the expected time series feature and the time series feature.

[0013] Further, the step of determining the bucketing adjustment strategy based on the deviation analysis results includes: for each time-series feature, if the cosine similarity meets a first threshold range, triggering a horizontal splitting instruction generation process to obtain horizontal splitting strategy elements; or, if the cosine similarity meets a second threshold range, triggering a vertical expansion splitting instruction generation process to obtain vertical expansion strategy elements, wherein the value of the second threshold range is lower than the value of the first threshold range; or, if the cosine similarity meets a third threshold range, triggering a cross-node migration instruction generation process to obtain cross-node migration strategy elements, wherein the value of the third threshold range is lower than the value of the second threshold range; integrating all the strategy elements to obtain the bucketing adjustment strategy.

[0014] Furthermore, the step of reconstructing the bucket index of the financial data warehouse based on the bucket adjustment strategy to obtain the updated index of the financial data warehouse includes: parsing the bucket adjustment strategy to obtain operation instructions; executing the bucket structure adjustment operation according to the operation instructions; updating the global index mapping table of the financial data warehouse, and recording the adjusted bucket distribution status.

[0015] Further, the step of parsing the bucket adjustment strategy to obtain operation instructions includes: when the bucket adjustment strategy is a horizontal splitting strategy, reading the horizontal splitting strategy elements to obtain a horizontal splitting operation instruction, wherein the horizontal splitting operation instruction is used to instruct the following steps: creating a new bucket with the same structure as the source bucket, allocating the specified splitting data in the source bucket to the new bucket, and establishing the association between the new bucket and the source bucket; when the bucket adjustment strategy is a vertical expansion strategy, reading the vertical expansion strategy elements to obtain a vertical expansion operation instruction, wherein the vertical expansion operation instruction is used to instruct the following steps: determining the computing resource quota of the new bucket based on the resource occupancy status of the source bucket, and creating the new bucket based on the computing resource quota; when the bucket adjustment strategy is a cross-node migration strategy, reading the cross-node migration strategy elements to obtain a data migration instruction, wherein the data migration instruction is used to instruct the following steps: selecting a target migration node that meets the migration conditions, and migrating the bucket data to be migrated to the target migration node.

[0016] To achieve the above objectives, according to another aspect of this application, a bucketing control device for a financial data warehouse is also provided. This device includes: a collection unit, configured to collect a time-series performance set of the financial data warehouse through a distributed collection system, wherein the time-series performance set records performance indicators for M time steps, and the categories of the performance indicators include: business system performance, data warehouse table performance, and computing resource performance, where M is a preset value; an extraction unit, configured to perform multi-level feature extraction on the time-series performance set using a sliding window mechanism to obtain a time-series feature set, wherein the categories of time-series features recorded in the time-series feature set include: instantaneous fluctuation features, periodic trend features, and steady-state baseline features; an analysis unit, configured to perform association analysis on the time-series feature set based on a decision tree ensemble algorithm, with the expected query performance requirements as the target, to obtain a bucketing adjustment strategy; and a reconstruction unit, configured to reconstruct the bucket index of the financial data warehouse based on the bucketing adjustment strategy, to obtain the financial data warehouse with updated indexes.

[0017] Furthermore, the acquisition unit includes: an acquisition module, used to perform data acquisition according to a preset period through a lightweight acquisition agent pre-deployed in the financial data warehouse to obtain raw acquisition data, wherein the lightweight acquisition agent is configured to monitor the data query engine of the financial data warehouse, as well as the financial business system and resource manager connected to the financial data warehouse; and a preprocessing module, used to preprocess the raw acquisition data to obtain the time-series performance set, wherein the preprocessing includes the following operations: time window aggregation, data format conversion, and unit of measurement normalization.

[0018] Further, the extraction unit includes: a first acquisition module, used to acquire N preset calculation parameter groups corresponding to the expected query performance requirements and the sliding window mechanism, wherein each preset calculation parameter group indicates a different calculation granularity, and each preset calculation parameter group includes at least a time window length and a sliding calculation step size, and N is greater than or equal to 3; a first calculation module, used to calculate the standard deviation of the query response time of the financial data warehouse based on the first parameter group and the time series performance set, and use the standard deviation as the instantaneous fluctuation feature; and a fitting module, used to fit the data based on the second parameter group and the time series performance set. The data write rate of the financial data warehouse is fitted using the least squares method to obtain a trend curve of the data write rate, and the slope change value of the trend curve is used as the periodic trend feature. The second parameter group indicates a larger computational granularity than the first parameter group. The second calculation module is used to calculate the resource utilization rate of each computing node in the financial data warehouse based on the third parameter group and the time-series performance set, and calculate the resource moving average based on the resource utilization rate. The resource moving average is used as the steady-state baseline feature. The third parameter group indicates a larger computational granularity than the second parameter group.

[0019] Further, the analysis unit includes: a second acquisition module for acquiring an association rule document, wherein the association rule document records the mapping relationship between each query performance indicator and the corresponding time-series feature; a third calculation module for calculating the expected time-series feature based on the expected query performance requirement and the association rule document; an analysis module for analyzing the degree of deviation between the expected time-series feature and each time-series feature in the time-series feature set based on the gradient boosting decision tree ensemble algorithm, and obtaining the deviation analysis result; and a determination module for determining the bucketing adjustment strategy based on the deviation analysis result.

[0020] Furthermore, the deviation analysis results record the cosine similarity between the expected time series feature and the time series feature.

[0021] Further, the determining module includes: a first triggering submodule, configured to, for each of the temporal features, trigger a horizontal splitting instruction generation process to obtain horizontal splitting strategy elements when the cosine similarity meets a first threshold range; a second triggering submodule, configured to, when the cosine similarity meets a second threshold range, trigger a vertical expansion splitting instruction generation process to obtain vertical expansion strategy elements, wherein the value of the second threshold range is lower than the value of the first threshold range; a third triggering submodule, configured to, when the cosine similarity meets a third threshold range, trigger a cross-node migration instruction generation process to obtain cross-node migration strategy elements, wherein the value of the third threshold range is lower than the value of the second threshold range; and an integration submodule, configured to integrate all the strategy elements to obtain the bucketing adjustment strategy.

[0022] Furthermore, the reconstruction unit includes: a parsing module for parsing the bucketing adjustment strategy to obtain operation instructions; an execution module for executing the bucketing structure adjustment operation according to the operation instructions; and a recording module for updating the global index mapping table of the financial data warehouse and recording the adjusted bucketing distribution status.

[0023] Further, the parsing module includes: a first reading submodule, used to read horizontal splitting strategy elements and obtain horizontal splitting operation instructions when the bucketing adjustment strategy is a horizontal splitting strategy, wherein the horizontal splitting operation instructions are used to instruct the following steps: creating a new bucket with the same structure as the source bucket, allocating the specified splitting data in the source bucket to the new bucket, and establishing an association between the new bucket and the source bucket; a second reading submodule, used to read vertical expansion strategy elements and obtain vertical expansion operation instructions when the bucketing adjustment strategy is a vertical expansion strategy, wherein the vertical expansion operation instructions are used to instruct the following steps: determining the computing resource quota of the new bucket based on the resource occupancy status of the source bucket, and creating the new bucket based on the computing resource quota; a third reading submodule, used to read cross-node migration strategy elements and obtain data migration instructions when the bucketing adjustment strategy is a cross-node migration strategy, wherein the data migration instructions are used to instruct the following steps: selecting a target migration node that meets the migration conditions, and migrating the bucket data to be migrated to the target migration node.

[0024] To achieve the above objectives, according to another aspect of this application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the bucket control method for the financial data warehouse described in any one of the above claims.

[0025] To achieve the above objectives, according to another aspect of this application, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the bucket control method for the financial data warehouse described in any of the above claims.

[0026] To achieve the above objectives, according to another aspect of this application, a computer program product is also provided, including computer instructions, wherein when the computer instructions are executed by a processor, they implement the steps of the bucket control method for the financial data warehouse described in any of the above claims.

[0027] This invention proposes a bucketing control method for a financial data warehouse. First, a distributed acquisition system collects the time-series performance set of the financial data warehouse. This set records performance indicators for M time steps, categorized as business system performance, data warehouse table performance, and computing resource performance, with M being a preset value. Then, a sliding window mechanism is used to extract multi-level features from the time-series performance set, resulting in a time-series feature set. This feature set includes instantaneous fluctuation features, periodic trend features, and steady-state baseline features. Next, based on the expected query performance requirements, a decision tree ensemble algorithm is used to perform correlation analysis on the time-series feature set to obtain a bucketing adjustment strategy. Finally, the bucket index of the financial data warehouse is reconstructed based on the bucketing adjustment strategy, resulting in an index-updated financial data warehouse.

[0028] This invention employs a distributed real-time monitoring and intelligent analysis approach. By integrating multi-level time-series feature extraction methods that combine business system performance, data warehouse table performance, and computing resource performance, it achieves the goal of accurately understanding the dynamic load changes of the financial data warehouse, thereby realizing the technical effect of intelligent dynamic bucketing control. Specifically, this invention utilizes a distributed, lightweight agent for continuous monitoring, collecting and analyzing performance indicators of a financial data warehouse at different time scales. It identifies and extracts instantaneous fluctuations, periodic trends, and steady-state baseline features through a sliding window mechanism. Further analysis using a decision tree ensemble algorithm, with expected query performance requirements as the optimization objective, generates a dynamic bucketing adjustment strategy. Finally, based on the strategy, the bucket index of the financial data warehouse is reconstructed and optimized, ensuring balanced data distribution and improved query performance. This not only greatly improves the adaptability and flexibility of the data warehouse but also significantly enhances its processing capabilities when facing spatiotemporal imbalances in data load. It effectively avoids system performance bottlenecks and data skew problems caused by static bucketing mechanisms. In real-time data analysis scenarios for financial businesses, it ensures stable and efficient system operation even during peak business periods. This solves the technical problem in related technologies where static bucketing mechanisms for real-time data warehouses cannot dynamically adapt to the spatiotemporal imbalances in data load under financial business scenarios, leading to system performance degradation. Attached Figure Description

[0029] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0030] Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing a bucket control method for financial data warehouses is shown.

[0031] Figure 2 This is a flowchart of an optional bucket control method for a financial data warehouse according to an embodiment of the present invention;

[0032] Figure 3 This is a schematic diagram of the structure of an optional incremental update financial data storage system based on LSTM and XGBoost models in a real-time financial data warehouse scenario according to an embodiment of the present invention.

[0033] Figure 4 This is a schematic diagram of an optional bucket control device for a financial data warehouse according to an embodiment of the present invention;

[0034] Figure 5 This is a structural block diagram of an electronic device that performs a bucket control method for a financial data warehouse according to an embodiment of the present invention. Detailed Implementation

[0035] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0037] To facilitate understanding of the present invention by those skilled in the art, some terms or nouns involved in the various embodiments of the present invention are explained below:

[0038] A real-time data warehouse is a data warehouse system capable of receiving, processing, and analyzing real-time data streams. It allows users to query and analyze based on the latest data without waiting for traditional batch processing cycles to complete. In the financial sector, real-time data warehouses are frequently used in applications such as high-frequency trading, risk management, and market monitoring.

[0039] LSTM, or Long Short-Term Memory, is a special type of recurrent neural network (RNN) architecture used to learn and predict long-term dependencies in time series data. LSTM effectively captures long-term patterns in data through its unique gating mechanism and is widely used in fields such as load forecasting and time series analysis.

[0040] XGBoost, or Extreme Gradient Boosting, is an efficient, flexible, and scalable gradient boosting decision tree-based machine learning algorithm. It is known for its excellent performance and accuracy and is particularly suitable for processing datasets with multiple features and complex relationships. It is often used for predictive modeling and feature importance analysis.

[0041] Data skew: In distributed systems, data skew occurs when certain data partitions or nodes bear a disproportionately large amount of data or processing load. This can lead to degraded system performance, uneven resource utilization, and reduced overall processing efficiency. In big data processing, effectively managing and mitigating data skew is one of the key challenges in improving system performance and reliability.

[0042] AIOps, or Artificial Intelligence for IT Operations, refers to the use of artificial intelligence, machine learning, and big data technologies to optimize and automate IT operations processes. Its aim is to improve operational efficiency, reduce manual intervention, and solve complex operational challenges intelligently. AIOps is applied to various IT operations scenarios, including performance monitoring, anomaly detection, fault diagnosis, and predictive maintenance, with the goal of rapidly responding to and resolving IT issues.

[0043] It should be noted that the binning control method and apparatus for financial data warehouses in this application can be used in the field of artificial intelligence technology for performance optimization and dynamic resource allocation of large-scale real-time data analysis systems, and can also be used in any field other than artificial intelligence technology for performance optimization and dynamic resource allocation of large-scale real-time data analysis systems. This application does not limit the application field of the binning control method and apparatus for financial data warehouses.

[0044] It should be noted that all relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, and displayed data) involved in this application are information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, processing, transmission, provision, disclosure, use, and handling of such data comply with the laws, regulations, and standards of the relevant regions, necessary confidentiality measures have been taken, and they do not violate public order and good morals. Corresponding access points are provided for users to choose to authorize or refuse access. For example, this system has interfaces with relevant users or organizations. Before obtaining relevant information, a request to obtain the information needs to be sent to the aforementioned user or organization through the interface, and the relevant information is obtained only after receiving consent from the aforementioned user or organization.

[0045] The information collection (e.g., user voice, video, and text collection) and analysis operations involved in this application have provided users with corresponding operation entry points during execution, allowing users to choose to agree to or reject the automated decision results; if the user chooses to reject, the process will proceed to the expert decision-making process.

[0046] The following embodiments of the present invention can be applied to various systems / applications / devices that require real-time data analysis, performance optimization, and dynamic resource allocation, enabling intelligent dynamic bucket index management and performance improvement in financial real-time data warehouse scenarios. The present invention uses an LSTM model to perform multi-level feature extraction and short-term prediction on time-series performance data. Then, an XGBoost model comprehensively analyzes and makes decisions based on the prediction results and the current system state, enabling better prediction of the spatiotemporal trends of data load and dynamic adjustment of the bucket index strategy for the near-real-time financial data management warehouse according to expected query performance requirements.

[0047] Meanwhile, this invention also accurately locates data skew and performance bottlenecks through real-time performance monitoring and intelligent operation and maintenance, and dynamically optimizes and intelligently adjusts the data table structure in the near real-time financial data management warehouse. This makes the performance optimization process faster and more efficient, and the bucket index adjustment strategy more flexible and accurate, effectively responding to data fluctuations and changes in query patterns in financial business scenarios, and ensuring that the system can maintain a high level of performance and stability even in complex and changing environments.

[0048] The present invention will now be described in detail with reference to various embodiments.

[0049] Example 1

[0050] According to an embodiment of the present invention, an embodiment of a bucket control method for a financial data warehouse is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0051] The binning control method for financial data warehouses provided in Embodiment 1 of the present invention can be executed on mobile terminals, computer terminals or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a bucketing control method in a financial data warehouse is shown. Figure 1 As shown, the computer terminal 10 (or mobile device) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0052] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0053] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the bucketing control method of the financial data warehouse in this embodiment of the application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned bucketing control method of the financial data warehouse. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0054] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0055] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0056] Under the above operating environment, the present invention provides, as follows: Figure 2The illustrated bucketing control method for financial data warehouses is implemented using a real-time financial data warehouse system. This method, combined with Intelligent Operations and Maintenance (AIOps) technology, is used in real-time big data analysis scenarios, particularly to address performance bottlenecks and uneven resource allocation caused by short-term data surges. Through dynamic bucketing strategies and machine learning prediction methods, it aims to dynamically and intelligently adjust the hash-based bucketing indexing mechanism in the financial data storage system supporting incremental updates within a real-time financial data warehouse scenario. This optimizes data distribution and resource utilization, thereby significantly improving system performance and stability.

[0057] Specifically, the following steps are included:

[0058] Real-time performance monitoring and data collection: Continuously collect data table performance indicators, cluster resource utilization and upstream business system load data, and preprocess the data, including cleaning, feature extraction and format conversion.

[0059] LSTM model time series prediction: Utilize the LSTM model to learn the time series characteristics of historical performance data and predict performance indicators for a future period, including query latency, data ingestion rate, etc., to provide decision-making basis for intelligent operation and maintenance.

[0060] Multi-factor analysis of the XGBoost model: The XGBoost model is used to analyze various performance factors in the current system state, such as the number of buckets, data distribution, and hardware resources, and to predict the optimal bucket index configuration.

[0061] Intelligent decision-making of the decision engine: The decision engine module integrates the outputs of LSTM and XGBoost models, combines business rules and historical strategy performance, and automatically generates optimization strategies, such as dynamically adjusting the number of buckets and prioritizing partitions.

[0062] Intelligent data redistribution and strategy execution: The execution module progressively performs data migration and adjusts data table parameters in the real-time financial data warehouse according to the instructions of the decision engine, minimizing the impact on online services and ensuring data consistency.

[0063] Continuous optimization and feedback loop: The execution effect of the bucket expansion strategy is monitored through a real-time feedback mechanism, and the threshold and strategy parameters are dynamically adjusted to continuously improve the accuracy and efficiency of decision-making.

[0064] The above steps, through AI and machine learning-based intelligent operation and maintenance technologies, enable the financial real-time data warehouse system to automatically adapt to fluctuations in data volume, especially during peak periods of short-term data surges, thereby achieving efficient resource utilization and maintaining high system availability and performance.

[0065] The embodiments of the present invention will now be described in detail with reference to each specific step.

[0066] Figure 2 This is a flowchart of an optional bucket control method for a financial data warehouse according to an embodiment of the present invention, such as... Figure 2 As shown, the method includes the following steps:

[0067] Step S201: Collect the time-series performance set of the financial data warehouse through the distributed acquisition system. The time-series performance set records the performance indicators of M time steps. The categories of performance indicators include: business system performance, data warehouse data table performance and computing resource performance. M is a preset value.

[0068] Specifically, the distributed acquisition system is based on a high-performance data acquisition framework, used to collect and integrate key performance indicators from multiple data sources in real time. It can be seamlessly integrated into existing IT infrastructure to capture system status information in a low-latency, high-throughput manner, ensuring the comprehensiveness and accuracy of the data.

[0069] The distributed data acquisition system comprises the following core components: a data agent, specifically a lightweight acquisition agent distributed across the implementation system cluster nodes, responsible for capturing and preprocessing local data, including log files, monitoring metrics, and system events; a data collection center, i.e., a central data collector, responsible for aggregating data from various agents, performing preliminary aggregation and cleaning to ensure data consistency and format uniformity; a data stream processing engine, primarily utilizing a stream processing framework to analyze and process real-time data streams, extracting key features and performing preliminary analysis; and data storage, using a time-series database or distributed storage system to store and manage the collected performance data, supporting historical queries and trend analysis.

[0070] The time-series performance set of a financial data warehouse is used to store a series of dynamically updated performance metrics that reflect the operating status and efficiency of the financial data warehouse at different points in time. Specifically, it records the performance of the data warehouse when processing key business operations such as financial transactions, risk analysis, and customer activities. The performance metrics of M time steps represent the performance overview of the financial business system within M consecutive time intervals. For example, it can be set to one time step per minute, with M being 60, covering one hour of performance data.

[0071] Among the performance metrics categories, business system performance refers to indicators reflecting the load and responsiveness of upstream business systems, such as transaction volume, transaction processing speed, data generation speed, system peak load, data latency, system response time, and number of online users, used to understand changes in data generation rate and business needs; data warehouse table performance focuses on the internal workings of the data warehouse, including query latency, write throughput, data distribution, number of files, and file size, used to evaluate data storage and access efficiency; computing resource performance involves the utilization of the underlying computing and storage resources supporting data warehouse operation, such as CPU load, memory utilization, disk I / O, and network bandwidth utilization, used to identify resource bottlenecks and optimize resource allocation strategies.

[0072] By continuously monitoring the above performance indicators, we can promptly capture changes in the load of business systems, performance fluctuations in data warehouses, and the usage of computing resources, which is beneficial for subsequent AI-based analysis and decision-making.

[0073] Optionally, in the bucket control method for a financial data warehouse provided in this embodiment of the invention, the step of collecting the time-series performance set of the financial data warehouse through a distributed collection system includes: performing data collection according to a preset cycle through a lightweight collection agent pre-deployed in the financial data warehouse to obtain raw collected data, wherein the lightweight collection agent is configured to monitor the data query engine of the financial data warehouse, as well as the financial business system and resource manager connected to the financial data warehouse; preprocessing the raw collected data to obtain the time-series performance set, wherein the preprocessing includes the following operations: time window aggregation, data format conversion, and unit of measurement normalization.

[0074] In the financial data warehouse bucket control method provided in this embodiment of the invention, collecting the time-series performance set of the financial data warehouse through a distributed acquisition system is a key preliminary step, providing a data foundation for subsequent data analysis and strategy formulation.

[0075] The lightweight acquisition agent is pre-deployed on various nodes of the financial data warehouse as part of a distributed acquisition system. It can monitor the internal operating status of the data warehouse, the performance indicators of the data query engine, and key data of the financial business system and resource manager in real time.

[0076] The lightweight data acquisition agent is configured to listen for and capture the following data sources:

[0077] Data query engine: Monitors metrics such as query latency, query rate, and number of concurrent queries;

[0078] Financial business system: collects indicators such as transaction volume, user activity, and data generation rate;

[0079] Resource Manager: Monitors system computing resource usage such as CPU utilization, memory usage, disk I / O, and network bandwidth;

[0080] The lightweight data acquisition agent performs data acquisition tasks at preset intervals, such as once per minute, to ensure that the latest status of performance metrics is captured in a timely manner.

[0081] In one optional embodiment, the raw collected data needs to be preprocessed to transform it into a time-series performance dataset suitable for machine learning model analysis. Specifically, the preprocessing operations include: time window aggregation, which involves converting the raw data collected at specific time points into aggregated data statistically analyzed by time windows, such as calculating the average query latency every 5 minutes or the total transaction volume per hour; data format conversion, which ensures that all collected data can be processed uniformly, including converting text logs into structured data and binary data into readable formats; and unit normalization, which involves normalizing the units of performance metrics from different sources and categories (e.g., converting all latency metrics to milliseconds) to avoid data analysis errors caused by unit differences.

[0082] Through the aforementioned data collection and preprocessing steps, the lightweight data acquisition agent can promptly capture system state changes, ensuring the real-time nature of performance indicators. The preprocessing steps further improve data accuracy and consistency, providing a reliable basis for subsequent analysis. The agent's lightweight design reduces system resource consumption, and the time window aggregation and unit normalization in the preprocessing flow effectively reduce data processing overhead, improving overall resource utilization efficiency. High-frequency performance data collection enables the system to quickly detect anomalies, such as sudden increases in resource utilization or unexpected query delays, thus providing timely warnings.

[0083] Step S202: A sliding window mechanism is used to extract multi-level features from the time series performance set to obtain a time series feature set. The types of time series features recorded in the time series feature set include: instantaneous fluctuation features, periodic trend features, and steady-state baseline features.

[0084] Specifically, the sliding window mechanism is widely used in time series data analysis. It analyzes the data within a fixed-length time window by sliding it across the data stream, extracting short-term dynamic features. In real-time financial data warehouse scenarios, using the sliding window mechanism to process time series performance datasets can capture instantaneous fluctuations and recent trends in the data stream. A sliding window consists of a series of consecutive time points, and the window size (i.e., time length) and sliding step size (i.e., the time interval between window movements) can be adjusted according to the characteristics of the data. For example, a window size of 1 hour and a step size of 5 minutes can capture data fluctuations every 5 minutes and the average trend within 1 hour.

[0085] Within each time window, statistical analysis and feature engineering can be performed on performance indicators, such as calculating statistical indicators like the mean, median, and standard deviation, or more complex feature extraction, such as calculating frequency features and periodic patterns, which can fully explore the inherent information of the data.

[0086] Multi-level feature extraction refers to extracting features at different time scales from a time-series performance set, including instantaneous fluctuation features, periodic trend features, and steady-state baseline features. This helps machine learning models to understand the system's operating state more comprehensively and make more accurate predictions and decisions.

[0087] Among them, the instantaneous fluctuation feature focuses on the drastic changes in performance indicators in a short period of time, such as changes in query latency or write throughput within minutes or seconds. In this embodiment of the invention, the instantaneous fluctuation feature can be taken as the statistical quantity within the sliding window. For example, the degree of fluctuation of the performance indicator can be measured by calculating the standard deviation or variance within the sliding window. Specifically, the larger the standard deviation, the greater the data fluctuation, and vice versa. For another example, the difference between the minimum and maximum values ​​within the window can also be calculated to reflect the extreme value changes in a short period of time.

[0088] Cyclical trend features are used to capture long-term patterns of change, such as the periodic load variations of a business system within a week or month. Embodiments of this invention can predict seasonal peaks or troughs by analyzing cyclical trends. Specifically, a sliding window combined with periodic analysis algorithms (such as Fourier transform or spectral analysis) can be used to identify periodic patterns, converting time-series data to the frequency domain to identify the main cyclical frequencies, and further analyzing the correlation between these cyclical frequencies and business activities. For example, analyzing data within a month can identify high transaction volume patterns on weekends or specific holidays.

[0089] Steady-state baseline characteristics reflect average performance levels under conditions unaffected by significant external influences, such as daily average CPU utilization, memory usage, or disk I / O rate. Steady-state baseline characteristics can be constructed through statistical analysis of historical data (e.g., calculating long-term averages or medians), providing a normal operating reference standard for real-time monitoring and anomaly detection. For example, calculating the average daily query latency over the past month forms the system's steady-state baseline.

[0090] By extracting features at multiple levels and combining short-term sensitive responses with long-term robust predictions, we can gain a deeper understanding of the operational status of real-time financial data warehouses. This allows us to maintain high adaptability and stability in a constantly changing business environment, enabling us to promptly identify performance bottlenecks and resource allocation issues, as well as predict future load trends, thus providing data support for intelligent operation and maintenance decisions.

[0091] Optionally, in the bucket control method for financial data warehouses provided in this embodiment of the invention, the step of using a sliding window mechanism to perform multi-level feature extraction on the time-series performance set to obtain a time-series feature set includes: obtaining N preset calculation parameter groups corresponding to the expected query performance requirements and the sliding window mechanism, wherein each preset calculation parameter group indicates a different calculation granularity, and each preset calculation parameter group includes at least a time window length and a sliding calculation step size, and N is greater than or equal to 3; calculating the standard deviation of the query response time of the financial data warehouse based on the first parameter group and the time-series performance set, and using the standard deviation as an instantaneous fluctuation feature; performing least squares fitting on the data write rate of the financial data warehouse based on the second parameter group and the time-series performance set to obtain a trend curve of the data write rate, and using the slope change value of the trend curve as a periodic trend feature, wherein the calculation granularity indicated by the second parameter group is greater than that of the first parameter group; calculating the resource utilization rate of each computing node in the financial data warehouse based on the third parameter group and the time-series performance set, and calculating the resource moving average based on the resource utilization rate, and using the resource moving average as a steady-state baseline feature, wherein the calculation granularity indicated by the third parameter group is greater than that of the second parameter group.

[0092] In this embodiment of the invention, using a sliding window mechanism to perform multi-level feature extraction on the time-series performance set to obtain more detailed performance insights is an important component of constructing an intelligent bucketing adjustment strategy.

[0093] This invention allows for the pre-design of three sets of preset calculation parameters with different granularities to analyze multi-level features such as instantaneous fluctuations, periodic trends, and steady-state baselines. The time window length specifies the continuous time interval used for calculating the features; for example, the window for instantaneous fluctuations might be a few minutes, while the steady-state baseline might require several days. The sliding calculation step size specifies the basic time unit for the sliding window movement, such as 1 minute or 1 hour. The step size should be selected based on the frequency of data changes and the purpose of the analysis.

[0094] In the feature extraction step, instantaneous fluctuation feature extraction refers to calculating the standard deviation of query response times in the time-series performance set using the time window length and sliding calculation step size specified by the first parameter group. This reflects the degree of fluctuation in query response time over a short period (e.g., within a few minutes). A larger standard deviation indicates more severe instantaneous fluctuations in query response time, and vice versa. This instantaneous fluctuation feature helps identify changes in system pressure during high-concurrency queries.

[0095] When extracting periodic trend features, the second parameter group has a longer time window and a larger computational granularity than the first parameter group, making it suitable for analyzing data write rate changes over a longer period. This invention can use the least squares method to fit a curve showing the trend of data write rate over time. The slope of the curve provides information on the periodic trend of the data write rate. Analyzing the slope reveals periodic patterns in write operations, such as high write rates during the day and low write rates at night.

[0096] During the steady-state baseline feature extraction process, the computational granularity indicated by the third parameter group is further amplified, making it suitable for long-term moving averages of computational resource utilization. This embodiment of the invention can use statistical analysis to calculate the moving average of resource utilization for each computing node over a longer period as a steady-state baseline feature, reflecting the long-term average workload of the system under conditions free from special business events.

[0097] Through the above-mentioned multi-level feature extraction steps, this embodiment of the invention adopts a sliding window mechanism combined with multi-level feature extraction, while paying attention to short-term fluctuations and long-term trends to provide comprehensive data support for dynamic bucketing strategies and enhance predictive capabilities. In particular, instantaneous fluctuation features enhance the ability to quickly respond to sudden query pressures, periodic trend features allow for advance prediction of write rate peaks and troughs, reserving time for resource allocation and bucketing adjustments, and steady-state baseline features help identify resource utilization inefficiencies or potential performance bottlenecks.

[0098] Step S203: With the expected query performance requirements as the goal, the time-series feature set is analyzed based on the decision tree ensemble algorithm to obtain the bucketing adjustment strategy.

[0099] Specifically, the expected query performance requirements are the targets set for future system performance in a financial real-time data warehouse scenario based on historical data analysis and business forecasting. They are predetermined and typically revolve around the following core metrics:

[0100] Query latency: Ensures rapid response to data queries in high-concurrency scenarios, avoiding long waiting times; Throughput: Guarantees the system's ability to handle large amounts of data without congestion during short-term data surges, maintaining efficient real-time data ingestion and processing performance; Resource utilization: Maintains effective use of computing and storage resources, avoiding overconsumption or idleness; Data consistency: Ensures the atomicity of data operations, guaranteeing data accuracy and consistency even under high-concurrency writes. These performance metrics serve as important references when creating bucketing adjustment strategies, ensuring that the performance optimization of the near real-time financial data management warehouse matches business needs.

[0101] Another point to note is that decision tree ensemble algorithms are machine learning methods that improve the accuracy and robustness of a model by constructing multiple decision tree models and integrating their predictions. In this embodiment of the invention, a decision tree ensemble algorithm is used to perform association analysis on a time-series feature set. Specifically, multiple decision trees are constructed through random sampling to form a random forest. Average aggregation techniques are used to reduce overfitting and improve decision diversity. Sequential learning techniques are also employed to construct decisions tree by tree, with each tree attempting to correct the errors of the previous tree, creating a gradient boosting effect to capture complex data relationships and improve prediction accuracy.

[0102] Furthermore, a subset of features can be randomly selected for training before constructing each decision tree. This not only reduces feature dimensionality and improves training speed but also avoids excessive influence of a single feature on the final decision. Adaptive learning can also be employed, adjusting subsequent modeling strategies based on each prediction error to ensure continuous algorithm optimization and adaptation to changing business scenarios. Additionally, the prediction process of the decision tree model can be explained through rule chains, which helps in understanding how the model makes decisions about bucketing adjustments based on input features.

[0103] Furthermore, the bucketing adjustment strategy refers to the process of dynamically adjusting the bucketing configuration of near-real-time financial data management warehouse data tables based on the analysis results of the decision tree ensemble algorithm to optimize system performance. The strategy encompasses dynamic adjustment of the number of buckets, priority bucketing, intelligent redistribution, and threshold adaptation.

[0104] Dynamic bucket number adjustment refers to automatically increasing or decreasing the number of buckets when changes in data volume or query patterns are detected, maintaining a uniform distribution of data and avoiding data skew; priority bucketing refers to allocating more buckets to certain high-priority partitions or partitions susceptible to sudden data surges, optimizing partition performance; intelligent redistribution refers to intelligently redistributing data when bucket number adjustment is performed, ensuring data balance while minimizing the impact on online services; threshold adaptation refers to dynamically adjusting the threshold that triggers bucket number adjustment based on feedback from the real-time performance monitoring module, more accurately responding to changes in performance requirements.

[0105] By analyzing the time-series feature set using the decision tree ensemble algorithm, this invention can accurately understand the current system state and future performance requirements, thereby formulating an effective bucketing adjustment strategy to optimize the performance of the near real-time financial data management warehouse and ensure that the real-time financial data warehouse can maintain high efficiency and stability in the face of constantly changing business environments.

[0106] Optionally, in the bucketing control method for financial data warehouses provided in this embodiment of the invention, the step of performing association analysis on the time-series feature set based on the decision tree ensemble algorithm to obtain the bucketing adjustment strategy, with the expected query performance requirements as the target, includes: obtaining an association rule document, wherein the association rule document records the mapping relationship between each query performance indicator and the corresponding time-series feature; calculating the expected time-series feature based on the expected query performance requirements and the association rule document; analyzing the degree of deviation between the expected time-series feature and each time-series feature in the time-series feature set based on the gradient boosting decision tree ensemble algorithm to obtain the deviation analysis result; and determining the bucketing adjustment strategy based on the deviation analysis result.

[0107] In the financial data warehouse bucketing control method provided in this embodiment of the invention, the step of performing correlation analysis on the time series feature set based on the decision tree ensemble algorithm with the expected query performance requirements as the target, and then obtaining the bucketing adjustment strategy, is a key link in realizing intelligent resource allocation and performance optimization.

[0108] The acquired association rule document records the mapping relationship between query performance metrics and corresponding time-series features, and is derived from the analysis of historical data. These mapping relationships indicate which time-series features have a direct impact on query performance and the specific manifestations of this impact. For example, the document states that "when the query latency exceeds 200ms, the data skewness exceeds 0.6," indicating that data skewness is one of the important factors affecting query latency.

[0109] In the step of calculating expected time-series features based on expected query performance requirements and association rule documents, the expected query performance requirements are determined by business objectives. The implementation system calculates the desired time-series feature values ​​based on the expected performance requirements and historical association rules through reverse reasoning or model prediction. For example, the expected query performance requirements stipulate that the query latency should be within 100ms and the throughput should reach 10,000 records / s. To achieve a query latency of 100ms, it is calculated that under the expected business load, the data skew should be controlled below 0.5 and the number of active users should be less than 7,000.

[0110] In the step of analyzing deviation using the gradient boosting decision tree ensemble algorithm, the algorithm analyzes the degree of deviation between each temporal feature in the temporal feature set and the expected feature. It quantifies the gap between the current state and the target state by calculating the deviation between the expected feature and the actual feature in the temporal feature set. For example, if the actual data skewness is 0.7, while the expected value is 0.5, the impact of this deviation on query performance is evaluated, and it is determined whether the bucketing strategy needs to be adjusted.

[0111] Finally, based on the deviation analysis results, a bucket adjustment strategy is determined, including whether to increase, decrease, or keep the number of buckets unchanged, and when to implement these adjustments. For example, if the data skew is found to be much higher than expected, a strategy to increase the number of buckets is triggered; if a significant increase in future business volume is predicted, the number of buckets is pre-expanded to prepare for high loads.

[0112] Optionally, in the binning control method for financial data warehouses provided in this embodiment of the invention, the deviation analysis result records the cosine similarity between the expected time-series feature and the time-series feature. The step of determining the binning adjustment strategy based on the deviation analysis result includes: for each time-series feature, if the cosine similarity meets a first threshold range, triggering a horizontal splitting instruction generation process to obtain horizontal splitting strategy elements; or, if the cosine similarity meets a second threshold range, triggering a vertical expansion splitting instruction generation process to obtain vertical expansion strategy elements, wherein the value of the second threshold range is lower than the value of the first threshold range; or, if the cosine similarity meets a third threshold range, triggering a cross-node migration instruction generation process to obtain cross-node migration strategy elements, wherein the value of the third threshold range is lower than the value of the second threshold range; integrating all strategy elements to obtain the binning adjustment strategy.

[0113] In this embodiment of the invention, cosine similarity can be used as the metric for two feature vectors to compare the similarity between time-series features and expected features. The current state and expected state of each time-series feature are represented as vectors, and then the cosine value of the angle between the two vectors is calculated to obtain the degree of deviation of the feature value.

[0114] When the cosine similarity value falls within a relatively high range of the first threshold, it indicates that the current feature value is close to the expected value, but there may be a slight imbalance in the distribution. At this time, it is necessary to trigger the horizontal splitting instruction generation process to further refine the data distribution and improve query performance.

[0115] If the cosine similarity value is within a lower range of the second threshold, it indicates that the feature value deviates significantly from the expected value and the data is severely skewed. At this time, the vertical expansion split instruction generation process is triggered to balance the data distribution by increasing the number of buckets and to deal with the situation of short-term data surge.

[0116] When the cosine similarity value decreases further and falls into a lower third threshold range, it indicates extreme data skew, possibly due to excessive load on some nodes or partitions. At this point, a cross-node migration instruction generation process is triggered to redistribute data and ensure a balanced distribution across the entire cluster.

[0117] The preferred strategy element generation process in this embodiment of the invention involves a horizontal splitting instruction generation process that further divides large, relatively evenly distributed buckets into smaller sub-buckets to improve query locality and parallel processing capabilities. It can analyze data skewness and query patterns to determine which buckets need to be horizontally split and generate corresponding strategy elements.

[0118] Additionally, the vertical scaling instruction generation process primarily focuses on increasing the total number of buckets to alleviate data skew by distributing data. Strategy elements include the number of new buckets, a data migration plan, and a timeline. The final cross-node migration instruction generation process works as follows: when data is extremely unevenly distributed across nodes, a data migration strategy is generated based on resource usage and node performance to migrate some data from overloaded nodes to underloaded nodes.

[0119] Finally, all generated strategy elements are integrated into a complete bucketing adjustment strategy, including detailed operational guidelines for horizontal splitting, vertical scaling, and cross-node migration, to guide the subsequent data redistribution and index reconstruction process.

[0120] This invention provides a dynamic and intelligent bucketing control method by employing cosine similarity analysis and strategy element generation process. This method can effectively address the performance and resource management challenges of financial data warehouses when processing large-scale real-time data, and significantly improve the stability and cost-effectiveness of the system.

[0121] Step S204: Reconstruct the bucket index of the financial data warehouse based on the bucket adjustment strategy to obtain the financial data warehouse with updated index.

[0122] Specifically, bucket index refactoring refers to optimizing the bucket index structure of a near-real-time financial data management warehouse based on a bucketing adjustment strategy. This involves reorganizing existing data and updating the indexes, including determining the target number of buckets, data redistribution, and index updates. Specifically, based on the analysis results of the decision tree ensemble algorithm, a new number of buckets is determined. Data in the table is then redistributed to different buckets according to the new number. After data redistribution, the bucket indexes are updated to reflect the new bucket distribution, old index information is deleted, and new index entries are generated. Bucket index refactoring is a key means of dynamically responding to changes in business needs. By continuously adjusting the bucket index strategy, performance can be continuously optimized to adapt to the challenges of large-scale data processing and high-concurrency queries.

[0123] Index updates involve the following key steps: temporary index generation, consistency checks, index switching, and old index cleanup. Specifically, after the new data distribution is determined, a temporary index is first generated and stored in memory or a temporary file system. An integrity check is performed on the temporary index to ensure that each data record is accurately located in the correct bucket. After ensuring the temporary index is correct, an atomic operation replaces the temporary index with the production environment index, removing old index information, freeing up storage space, and reducing the impact of redundant indexes on system performance.

[0124] When performing bucket index reconstruction and index updates, distributed computing capabilities can be used to perform data redistribution tasks on multiple nodes simultaneously. Fine-grained data locking and version control mechanisms can be used during data redistribution and index updates. For data being written, an incremental update strategy can be adopted to directly allocate new data to the new bucket structure. After the index update is completed, a health check can be automatically performed to ensure that the new bucket index structure can work normally and that all query and write operations can achieve the expected performance.

[0125] Optionally, in the bucketing control method for a financial data warehouse provided in this embodiment of the invention, the step of reconstructing the bucket index of the financial data warehouse based on the bucketing adjustment strategy to obtain the updated financial data warehouse includes: parsing the bucketing adjustment strategy to obtain operation instructions; executing the bucketing structure adjustment operation according to the operation instructions; updating the global index mapping table of the financial data warehouse, and recording the adjusted bucketing distribution status.

[0126] In the financial data warehouse bucketing control method provided in this embodiment of the invention, reconstructing the bucket index based on the bucketing adjustment strategy generated by the decision is a key step to ensure system performance and reasonable resource allocation.

[0127] First, parse the bucketing adjustment strategy document to extract specific operation instructions, which may include, but are not limited to, increasing the number of buckets, decreasing the number of buckets, horizontal or vertical splitting of buckets, and cross-node migration of data. Each instruction includes: operation type (such as EXPAND, SPLIT, or MIGRATE), specific operation parameters (such as target number of buckets, source and target node IDs), and time window (i.e. when to start executing the operation).

[0128] Dependency analysis can be performed before any operation is executed to determine the order of operations and ensure data integrity and consistency. For example, it ensures that data migration is completed before index updates to prevent index invalidation due to changes in data location.

[0129] In the process of performing bucketing structure adjustment, the API of the distributed computing framework is first called according to the operation instructions to decompose the bucketing structure adjustment task into multiple sub-tasks, which are then distributed to different nodes in the cluster for parallel execution, thereby reducing the overall operation time.

[0130] When a strategy requires horizontal or vertical bucket splitting, data needs to be migrated from the original bucket to the newly created bucket. An incremental data migration strategy can be used to migrate only the changed data, retain the original index mapping, and update the index to point to the new bucket position after the migration is completed.

[0131] During the bucket structure adjustment process, resource usage is monitored in real time, and resource allocation is dynamically adjusted to ensure that all nodes have sufficient computing and storage resources to complete tasks and avoid resource bottlenecks.

[0132] In the step of updating the global index mapping table, once all bucketing structure adjustments are complete, the global index mapping table is updated, involving at least updating index entries to ensure that the index accurately points to the new bucket distribution, reflecting the latest data distribution and bucket positions. After the index mapping update, a consistency check is performed to verify the validity of the index and the rationality of the data distribution, avoiding data loss or query anomalies due to index update errors.

[0133] While updating the global index mapping table, record the adjusted bucket distribution status, including the number of buckets, the amount of data in each bucket and its distribution, as well as any key parameter adjustments that may affect performance.

[0134] By adopting the above steps, this invention not only solves the core problem of performance optimization when implementing the binning control method for financial data warehouses, but also achieves additional beneficial effects in terms of resource utilization, automated operation, and business continuity, providing a strong guarantee for the efficient and stable operation of financial data warehouses.

[0135] Optionally, in the bucketing control method for a financial data warehouse provided in this embodiment of the invention, the step of parsing the bucketing adjustment strategy to obtain operation instructions includes: when the bucketing adjustment strategy is a horizontal splitting strategy, reading the horizontal splitting strategy elements to obtain a horizontal splitting operation instruction, wherein the horizontal splitting operation instruction is used to instruct the following steps: creating a new bucket with the same structure as the source bucket, allocating the specified splitting data in the source bucket to the new bucket, and establishing an association between the new bucket and the source bucket; when the bucketing adjustment strategy is a vertical expansion strategy, reading the vertical expansion strategy elements to obtain a vertical expansion operation instruction, wherein the vertical expansion operation instruction is used to instruct the following steps: determining the computing resource quota of the new bucket based on the resource occupancy status of the source bucket, and creating a new bucket based on the computing resource quota; when the bucketing adjustment strategy is a cross-node migration strategy, reading the cross-node migration strategy elements to obtain a data migration instruction, wherein the data migration instruction is used to instruct the following steps: selecting a target migration node that meets the migration conditions, and migrating the bucket data to be migrated to the target migration node.

[0136] In the financial data warehouse bucketing control method provided in this embodiment of the invention, parsing the bucketing adjustment strategy and generating specific operation instructions directly determines the subsequent resource allocation and data redistribution operations.

[0137] Horizontal splitting strategies aim to address data skew by creating additional buckets to distribute data. In the case of horizontal splitting, the strategy element file is read to extract the following information: Source bucket structure: including the current number, size, and data distribution of buckets; Specified split data: data that needs to be split from the source buckets based on performance monitoring results and business needs; in this embodiment, this refers to the data with uneven distribution or excessive load; New bucket creation information: new buckets with the same structure as the source buckets are created based on the strategy elements, and the new bucket creation information is obtained to ensure a smooth transition of data to the new buckets; Data allocation information: the specified split data is migrated from the source buckets to the new buckets using data migration tools, while maintaining data consistency and integrity to ensure seamless data migration.

[0138] Vertical scaling strategies focus on increasing the total number of buckets to cope with increased data volume and improve query performance. Under a vertical scaling strategy, the following information is obtained from the strategy element file: Source bucket resource usage status: Analyzing the current data volume, resource utilization, and performance metrics of each bucket to determine which buckets need scaling; New bucket computing resource quota: Calculating the computing resource quota for new buckets based on resource usage status and expected query performance requirements, including CPU, memory, and storage space; Bucket creation and resource configuration: Creating new buckets based on the computing resource quota and allocating resources to the newly created buckets.

[0139] Cross-node migration strategies are used to optimize data distribution across nodes and alleviate the problem of uneven node resources. The strategy element file is read and the following content is parsed: Migration conditions: The conditions for data migration defined in the strategy elements, such as triggering migration when node resource utilization exceeds a certain threshold or data skew exceeds a threshold; Target migration node: Automatically selects a suitable target migration node to receive the data to be migrated based on resource usage and node performance, ensuring balanced node load after data migration; Data migration execution information: Uses distributed data movement tools to move the bucketed data to be migrated from the source node to the target migration node, ensuring the safety and integrity of the data during the migration process.

[0140] The horizontal splitting strategy in this embodiment of the invention effectively alleviates the data skew problem and improves query performance by creating additional buckets and redistributing data in overloaded source buckets; the vertical scaling strategy allows for dynamic adjustment of the number of buckets and computing resource quotas based on the growth of data volume and query demand, ensuring effective resource utilization and stable performance; the cross-node migration strategy achieves dynamic data balancing among nodes by automatically selecting target migration nodes and performing data migration, reducing resource bottlenecks and improving the overall stability and efficiency of the system.

[0141] Through steps S201 to S204 above, the time-series performance set of the financial data warehouse can first be collected through a distributed acquisition system. The time-series performance set records performance indicators for M time steps. The categories of performance indicators include: business system performance, data warehouse data table performance, and computing resource performance. M is a preset value. Then, a sliding window mechanism is used to perform multi-level feature extraction on the time-series performance set to obtain a time-series feature set. The categories of time-series features recorded in the time-series feature set include: instantaneous fluctuation features, periodic trend features, and steady-state baseline features. Then, with the expected query performance requirements as the target, the time-series feature set is analyzed for correlation based on the decision tree ensemble algorithm to obtain a bucketing adjustment strategy. Finally, the bucket index of the financial data warehouse is reconstructed based on the bucketing adjustment strategy to obtain the index-updated financial data warehouse.

[0142] In this embodiment of the invention, a distributed real-time monitoring and intelligent analysis approach is adopted. By integrating multi-level time-series feature extraction methods that integrate business system performance, data warehouse table performance, and computing resource performance, the goal of accurately understanding the dynamic load changes of the financial data warehouse is achieved, thereby realizing the technical effect of intelligent dynamic bucketing control. Specifically, this invention utilizes a distributed, lightweight agent for continuous monitoring, collecting and analyzing performance indicators of the financial data warehouse at different time scales. A sliding window mechanism identifies and extracts instantaneous fluctuations, periodic trends, and steady-state baseline features. Further analysis using a decision tree ensemble algorithm, with expected query performance requirements as the optimization objective, generates a dynamic bucketing adjustment strategy. Finally, based on the strategy, the bucket index of the financial data warehouse is reconstructed and optimized to ensure balanced data distribution and improved query performance. This not only greatly improves the adaptability and flexibility of the data warehouse but also significantly enhances its processing capabilities when facing spatiotemporal imbalances in data load. It effectively avoids system performance bottlenecks and data skew problems caused by static bucketing mechanisms. In real-time data analysis scenarios for financial businesses, it ensures stable and efficient system operation even during peak business periods. This solves the technical problem in related technologies where static bucketing mechanisms for real-time data warehouses cannot dynamically adapt to the spatiotemporal imbalances in data load under financial business scenarios, leading to system performance degradation.

[0143] The present invention will now be described in conjunction with another specific embodiment.

[0144] Figure 3 This is a schematic diagram of an optional incremental update financial data storage system based on LSTM and XGBoost models in a real-time financial data warehouse scenario, according to an embodiment of the present invention. Figure 3 As shown, the system includes a real-time performance monitoring module, a dynamic bucket expansion strategy module, and an intelligent data redistribution module. The real-time performance monitoring module performs data collection and preprocessing, as well as performance indicator analysis, and transmits the performance data to the dynamic bucket expansion strategy module. The dynamic bucket expansion strategy module performs LSTM time series prediction, XGBoost multi-factor analysis, and uses a decision engine to generate scaling decisions, which are then transmitted to the intelligent data redistribution module. The intelligent data redistribution module executes and monitors these decisions based on a task scheduler, and feeds back the execution results to the real-time performance monitoring module and the dynamic bucket expansion strategy module. The real-time performance monitoring module then captures the real-time status based on the execution feedback and synchronously feeds it back to the dynamic bucket expansion strategy module, thereby adjusting the decisions and forming a closed-loop positive feedback loop to ensure system flexibility and real-time performance.

[0145] The invention will now be described in conjunction with another alternative embodiment.

[0146] Example 2

[0147] This invention also provides a bucket control device for a financial data warehouse. It should be noted that the bucket control device for a financial data warehouse in this invention includes multiple implementation units, which can be used to execute the bucket control method for a financial data warehouse provided in the first embodiment above. Each implementation unit corresponds to each implementation step in the first embodiment above.

[0148] Figure 4 This is a schematic diagram of an optional bucket control device for a financial data warehouse according to an embodiment of the present invention, as shown below. Figure 4 As shown, the device may include: a data acquisition unit 41, an extraction unit 42, an analysis unit 43, and a reconstruction unit 44.

[0149] The acquisition unit 41 is used to acquire the time-series performance set of the financial data warehouse through a distributed acquisition system. The time-series performance set records the performance indicators of M time steps. The categories of performance indicators include: business system performance, data warehouse data table performance and computing resource performance. M is a preset value.

[0150] Extraction unit 42 is used to perform multi-level feature extraction on time series performance set using a sliding window mechanism to obtain time series feature set. The types of time series features recorded in the time series feature set include: instantaneous fluctuation features, periodic trend features and steady-state baseline features.

[0151] Analysis unit 43 is used to perform association analysis on the time series feature set based on the decision tree ensemble algorithm with the expected query performance requirements as the target, and obtain the bucketing adjustment strategy.

[0152] Reconstruction unit 44 is used to reconstruct the bucket index of the financial data warehouse based on the bucket adjustment strategy, so as to obtain the financial data warehouse with updated index.

[0153] The aforementioned bucketing control device for the financial data warehouse can first collect the time-series performance set of the financial data warehouse through the collection unit 41 via a distributed collection system. The time-series performance set records performance indicators for M time steps, and the categories of performance indicators include: business system performance, data warehouse data table performance, and computing resource performance, where M is a preset value. Then, the extraction unit 42 uses a sliding window mechanism to perform multi-level feature extraction on the time-series performance set to obtain a time-series feature set. The categories of time-series features recorded in the time-series feature set include: instantaneous fluctuation features, periodic trend features, and steady-state baseline features. Then, the analysis unit 43 uses the expected query performance requirements as the target and performs correlation analysis on the time-series feature set based on the decision tree ensemble algorithm to obtain the bucketing adjustment strategy. Finally, the reconstruction unit 44 reconstructs the bucket index of the financial data warehouse based on the bucketing adjustment strategy to obtain the index-updated financial data warehouse.

[0154] In this embodiment of the invention, a distributed real-time monitoring and intelligent analysis approach is adopted. By integrating multi-level time-series feature extraction methods that integrate business system performance, data warehouse table performance, and computing resource performance, the goal of accurately understanding the dynamic load changes of the financial data warehouse is achieved, thereby realizing the technical effect of intelligent dynamic bucketing control. Specifically, this invention utilizes a distributed, lightweight agent for continuous monitoring, collecting and analyzing performance indicators of the financial data warehouse at different time scales. A sliding window mechanism identifies and extracts instantaneous fluctuations, periodic trends, and steady-state baseline features. Further analysis using a decision tree ensemble algorithm, with expected query performance requirements as the optimization objective, generates a dynamic bucketing adjustment strategy. Finally, based on the strategy, the bucket index of the financial data warehouse is reconstructed and optimized to ensure balanced data distribution and improved query performance. This not only greatly improves the adaptability and flexibility of the data warehouse but also significantly enhances its processing capabilities when facing spatiotemporal imbalances in data load. It effectively avoids system performance bottlenecks and data skew problems caused by static bucketing mechanisms. In real-time data analysis scenarios for financial businesses, it ensures stable and efficient system operation even during peak business periods. This solves the technical problem in related technologies where static bucketing mechanisms for real-time data warehouses cannot dynamically adapt to the spatiotemporal imbalances in data load under financial business scenarios, leading to system performance degradation.

[0155] Furthermore, the acquisition unit includes: an acquisition module, used to perform data acquisition according to a preset period through a lightweight acquisition agent pre-deployed in the financial data warehouse to obtain raw acquisition data, wherein the lightweight acquisition agent is configured to monitor the data query engine of the financial data warehouse, as well as the financial business system and resource manager connected to the financial data warehouse; and a preprocessing module, used to preprocess the raw acquisition data to obtain a time-series performance set, wherein the preprocessing includes the following operations: time window aggregation, data format conversion, and unit of measurement normalization.

[0156] Further, the extraction unit includes: a first acquisition module, used to acquire N preset calculation parameter groups corresponding to the expected query performance requirements and sliding window mechanism, wherein each preset calculation parameter group indicates a different calculation granularity, and each preset calculation parameter group includes at least the time window length and the sliding calculation step size, and N is greater than or equal to 3; a first calculation module, used to calculate the standard deviation of the query response time of the financial data warehouse based on the first parameter group and the time series performance set, and use the standard deviation as an instantaneous fluctuation feature; a fitting module, used to perform least squares fitting on the data write rate of the financial data warehouse based on the second parameter group and the time series performance set to obtain the trend curve of the data write rate, and use the slope change value of the trend curve as a periodic trend feature, wherein the calculation granularity indicated by the second parameter group is greater than that of the first parameter group; a second calculation module, used to calculate the resource utilization rate of each computing node in the financial data warehouse based on the third parameter group and the time series performance set, and calculate the resource moving average based on the resource utilization rate, and use the resource moving average as a steady-state baseline feature, wherein the calculation granularity indicated by the third parameter group is greater than that of the second parameter group.

[0157] Furthermore, the analysis unit includes: a second acquisition module for acquiring association rule documents, wherein the association rule documents record the mapping relationship between each query performance indicator and the corresponding time-series feature; a third calculation module for calculating the expected time-series feature based on the expected query performance requirements and the association rule document; an analysis module for analyzing the degree of deviation between the expected time-series feature and each time-series feature in the time-series feature set based on the gradient boosting decision tree ensemble algorithm, and obtaining the deviation analysis result; and a determination module for determining the bucketing adjustment strategy based on the deviation analysis result.

[0158] Furthermore, the deviation analysis results record the cosine similarity between the expected time series features and the time series features.

[0159] Further, the determining module includes: a first triggering submodule, used to trigger a horizontal splitting instruction generation process for each temporal feature, where the cosine similarity meets a first threshold range, to obtain horizontal splitting strategy elements; a second triggering submodule, used to trigger a vertical expansion splitting instruction generation process, where the cosine similarity meets a second threshold range, to obtain vertical expansion strategy elements, wherein the value of the second threshold range is lower than the value of the first threshold range; a third triggering submodule, used to trigger a cross-node migration instruction generation process, where the cosine similarity meets a third threshold range, to obtain cross-node migration strategy elements, wherein the value of the third threshold range is lower than the value of the second threshold range; and an integration submodule, used to integrate all strategy elements to obtain a bucketing adjustment strategy.

[0160] Furthermore, the reconstruction unit includes: a parsing module, used to parse the bucketing adjustment strategy and obtain operation instructions; an execution module, used to execute the bucketing structure adjustment operation according to the operation instructions; and a recording module, used to update the global index mapping table of the financial data warehouse and record the adjusted bucketing distribution status.

[0161] Furthermore, the parsing module includes: a first reading submodule, used to read the horizontal splitting strategy elements and obtain horizontal splitting operation instructions when the bucketing adjustment strategy is a horizontal splitting strategy. The horizontal splitting operation instructions instruct the following steps: creating a new bucket with the same structure as the source bucket, allocating the specified splitting data from the source bucket to the new bucket, and establishing an association between the new bucket and the source bucket; a second reading submodule, used to read the vertical expansion strategy elements and obtain vertical expansion operation instructions when the bucketing adjustment strategy is a vertical expansion strategy. The vertical expansion operation instructions instruct the following steps: determining the computing resource quota of the new bucket based on the resource occupancy status of the source bucket, and creating a new bucket based on the computing resource quota; a third reading submodule, used to read the cross-node migration strategy elements and obtain data migration instructions when the bucketing adjustment strategy is a cross-node migration strategy. The data migration instructions instruct the following steps: selecting a target migration node that meets the migration conditions, and migrating the bucket data to be migrated to the target migration node.

[0162] It should be noted that the acquisition unit 41, extraction unit 42, analysis unit 43, and reconstruction unit 44 mentioned above correspond to steps S201 to S204 in Embodiment 1. The instances and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules or units can be hardware or software components stored in memory (e.g., memory 104) and processed by one or more processors (e.g., processors 102a, 102b, ..., 102n). The above modules or units can also run as part of the device in the computer terminal 10 provided in Embodiment 1.

[0163] The invention will now be described in conjunction with another alternative embodiment.

[0164] Example 3

[0165] The present invention can also provide an electronic device. Figure 5 This is a structural block diagram of an electronic device for implementing a binning control method for a financial data warehouse according to an embodiment of the present invention, as shown below. Figure 5 As shown, the electronic device may include: one or more ( Figure 5 Only one of the components is shown: processor 502, memory 504, memory controller, and peripheral interface, wherein the peripheral interface is connected to the radio frequency module, audio module, and display.

[0166] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the binning control method and apparatus for the financial data warehouse in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned binning control method for the financial data warehouse. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0167] The processor can access information and applications stored in memory via a transmission device to execute the following steps: First, collect a time-series performance set of the financial data warehouse through a distributed acquisition system. This time-series performance set records performance metrics for M time steps, categorized as: business system performance, data warehouse table performance, and computing resource performance, where M is a preset value. Second, perform multi-level feature extraction on the time-series performance set using a sliding window mechanism to obtain a time-series feature set. This time-series feature set contains time-series features categorized as: instantaneous fluctuation features, periodic trend features, and steady-state baseline features. Third, perform association analysis on the time-series feature set based on a decision tree ensemble algorithm, targeting expected query performance requirements, to obtain a bucketing adjustment strategy. Finally, reconstruct the bucket index of the financial data warehouse based on the bucketing adjustment strategy to obtain an index-updated financial data warehouse.

[0168] The processor can also invoke information and applications stored in the memory via the transmission device to perform the following steps: Data collection is performed according to a preset cycle using a lightweight acquisition agent pre-deployed in the financial data warehouse to obtain raw collected data. The lightweight acquisition agent is configured to monitor the data query engine of the financial data warehouse, as well as the financial business systems and resource managers connected to the financial data warehouse. The raw collected data is then pre-processed to obtain a time-series performance set. The pre-processing includes the following operations: time window aggregation, data format conversion, and unit of measurement normalization.

[0169] The processor can also invoke information and applications stored in memory via a transmission device to perform the following steps: Obtain N preset calculation parameter sets corresponding to the expected query performance requirements and the sliding window mechanism, wherein each preset calculation parameter set indicates a different calculation granularity, and each preset calculation parameter set includes at least the time window length and the sliding calculation step size, and N is greater than or equal to 3; calculate the standard deviation of the query response time of the financial data warehouse based on the first parameter set and the time-series performance set, and use this standard deviation as an instantaneous fluctuation feature; perform least squares fitting on the data write rate of the financial data warehouse based on the second parameter set and the time-series performance set to obtain a trend curve of the data write rate, and use the slope change value of this trend curve as a periodic trend feature, wherein the calculation granularity indicated by the second parameter set is greater than that of the first parameter set; calculate the resource utilization rate of each computing node in the financial data warehouse based on the third parameter set and the time-series performance set, and calculate the resource moving average based on the resource utilization rate, using the resource moving average as a steady-state baseline feature, wherein the calculation granularity indicated by the third parameter set is greater than that of the second parameter set.

[0170] The processor can also call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain the association rule document, wherein the association rule document records the mapping relationship between each query performance index and the corresponding time series feature; calculate the expected time series feature based on the expected query performance requirements and the association rule document; analyze the degree of deviation between the expected time series feature and each time series feature in the time series feature set based on the gradient boosting decision tree ensemble algorithm to obtain the deviation analysis result; and determine the bucketing adjustment strategy based on the deviation analysis result.

[0171] The processor can also invoke information and applications stored in memory via a transmission device to execute the following steps: For each time-series feature, if the cosine similarity meets a first threshold range, trigger a horizontal splitting instruction generation process to obtain horizontal splitting strategy elements; or, if the cosine similarity meets a second threshold range, trigger a vertical expansion splitting instruction generation process to obtain vertical expansion strategy elements, wherein the value of the second threshold range is lower than the value of the first threshold range; or, if the cosine similarity meets a third threshold range, trigger a cross-node migration instruction generation process to obtain cross-node migration strategy elements, wherein the value of the third threshold range is lower than the value of the second threshold range; integrate all strategy elements to obtain a bucketing adjustment strategy.

[0172] The processor can also access information and applications stored in memory via a transmission device to perform the following steps: parse the bucketing adjustment strategy to obtain operation instructions; execute bucketing structure adjustment operations according to the operation instructions; update the global index mapping table of the financial data warehouse and record the adjusted bucketing distribution status.

[0173] The processor can also access information and applications stored in memory via a transmission device to execute the following steps: When the bucketing adjustment strategy is a horizontal splitting strategy, read the horizontal splitting strategy elements to obtain a horizontal splitting operation instruction, which instructs the following steps: create a new bucket with the same structure as the source bucket, allocate the specified splitting data from the source bucket to the new bucket, and establish an association between the new bucket and the source bucket; When the bucketing adjustment strategy is a vertical scaling strategy, read the vertical scaling strategy elements to obtain a vertical scaling operation instruction, which instructs the following steps: determine the computing resource quota for the new bucket based on the resource occupancy status of the source bucket, and create a new bucket based on the computing resource quota; When the bucketing adjustment strategy is a cross-node migration strategy, read the cross-node migration strategy elements to obtain a data migration instruction, which instructs the following steps: select a target migration node that meets the migration conditions, and migrate the bucket data to be migrated to the target migration node.

[0174] This invention provides a bucketing control scheme for a financial data warehouse. By employing distributed real-time monitoring and intelligent analysis, and integrating multi-level time-series feature extraction methods based on business system performance, data warehouse table performance, and computing resource performance, it achieves precise insight into dynamic load changes in the financial data warehouse, thus realizing the technical effect of intelligent dynamic bucketing control. Specifically, this invention utilizes a distributed, lightweight agent for continuous monitoring, collecting and analyzing performance indicators of a financial data warehouse at different time scales. It identifies and extracts instantaneous fluctuations, periodic trends, and steady-state baseline features through a sliding window mechanism. Further analysis using a decision tree ensemble algorithm, with expected query performance requirements as the optimization objective, generates a dynamic bucketing adjustment strategy. Finally, based on the strategy, the bucket index of the financial data warehouse is reconstructed and optimized, ensuring balanced data distribution and improved query performance. This not only greatly improves the adaptability and flexibility of the data warehouse but also significantly enhances its processing capabilities when facing spatiotemporal imbalances in data load. It effectively avoids system performance bottlenecks and data skew problems caused by static bucketing mechanisms. In real-time data analysis scenarios for financial businesses, it ensures stable and efficient system operation even during peak business periods. This solves the technical problem in related technologies where static bucketing mechanisms for real-time data warehouses cannot dynamically adapt to the spatiotemporal imbalances in data load under financial business scenarios, leading to system performance degradation.

[0175] Those skilled in the art will understand that Figure 5 The structure shown is for illustrative purposes only. Electronic devices can also be smartphones, tablets, handheld computers, mobile internet devices (MIDs), PADs, and other terminal devices. Figure 5This does not limit the structure of the aforementioned electronic device. For example, electronic devices may also include components that are more... Figure 5 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 5 The different configurations shown.

[0176] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0177] The invention will now be described in conjunction with another alternative embodiment.

[0178] Example 4

[0179] This invention also provides a computer-readable storage medium. Optionally, in this invention, the computer-readable storage medium can be used to store the program code executed by the bucketing control method for the financial data warehouse provided in Embodiment 1.

[0180] Optionally, in this embodiment of the invention, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0181] This invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing the steps of a bucketing adjustment method for a financial data warehouse: The program collects a time-series performance set of the financial data warehouse through a distributed acquisition system, wherein the time-series performance set records performance indicators for M time steps, and the categories of performance indicators include: business system performance, data warehouse table performance, and computing resource performance, where M is a preset value; a sliding window mechanism is used to perform multi-level feature extraction on the time-series performance set to obtain a time-series feature set, wherein the categories of time-series features recorded in the time-series feature set include: instantaneous fluctuation features, periodic trend features, and steady-state baseline features; based on the expected query performance requirements, a decision tree ensemble algorithm is used to perform correlation analysis on the time-series feature set to obtain a bucketing adjustment strategy; and based on the bucketing adjustment strategy, the bucket index of the financial data warehouse is reconstructed to obtain an index-updated financial data warehouse.

[0182] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0183] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0184] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of units or modules may be electrical or other forms.

[0185] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0186] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0187] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0188] The above are merely preferred embodiments of this application. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A binning control method for a financial data warehouse, characterized in that, include: The time-series performance set of the financial data warehouse is collected through a distributed acquisition system. The time-series performance set records performance indicators for M time steps. The categories of the performance indicators include: business system performance, data warehouse data table performance, and computing resource performance. M is a preset value. A sliding window mechanism is used to perform multi-level feature extraction on the time series performance set to obtain a time series feature set. The types of time series features recorded in the time series feature set include: instantaneous fluctuation features, periodic trend features, and steady-state baseline features. With the expected query performance requirements as the objective, the time-series feature set is analyzed based on the decision tree ensemble algorithm to obtain a bucketing adjustment strategy. The financial data warehouse is reconstructed using a bucket adjustment strategy to obtain the updated financial data warehouse.

2. The binning control method for a financial data warehouse according to claim 1, characterized in that, The steps for collecting time-series performance data from a financial data warehouse using a distributed acquisition system include: By pre-deploying a lightweight acquisition agent in the financial data warehouse, data acquisition is performed according to a preset cycle to obtain raw acquisition data. The lightweight acquisition agent is configured to monitor the data query engine of the financial data warehouse, as well as the financial business system and resource manager connected to the financial data warehouse. The original collected data is preprocessed to obtain the time-series performance set. The preprocessing includes the following operations: time window aggregation, data format conversion, and unit of measurement normalization.

3. The binning control method for a financial data warehouse according to claim 1, characterized in that, The steps of performing multi-level feature extraction on the time-series performance set using a sliding window mechanism to obtain the time-series feature set include: Obtain N preset calculation parameter groups corresponding to the expected query performance requirements and the sliding window mechanism, wherein each preset calculation parameter group indicates a different calculation granularity, and each preset calculation parameter group includes at least the time window length and the sliding calculation step size, and N is greater than or equal to 3; The standard deviation of the query response time of the financial data warehouse is calculated based on the first parameter set and the time series performance set, and the standard deviation is used as the instantaneous fluctuation feature. The data write rate of the financial data warehouse is fitted using the least squares method based on the second parameter set and the time-series performance set to obtain the trend curve of the data write rate. The slope of the trend curve is used as the periodic trend feature. The calculation granularity indicated by the second parameter set is larger than that of the first parameter set. The resource utilization rate of each computing node in the financial data warehouse is calculated based on the third parameter group and the time-series performance set, and the resource moving average is calculated based on the resource utilization rate. The resource moving average is used as the steady-state baseline feature, wherein the calculation granularity indicated by the third parameter group is greater than that of the second parameter group.

4. The binning control method for a financial data warehouse according to claim 1, characterized in that, To determine the bucketing adjustment strategy based on the expected query performance requirements, the steps include: (1) Performing association analysis on the time-series feature set using the decision tree ensemble algorithm; (2) Analyzing the data to obtain the bucketing adjustment strategy. Obtain the association rule document, wherein the association rule document records the mapping relationship between each query performance indicator and the corresponding time-series feature; Calculate the expected time-series features based on the expected query performance requirements and the associated rule documents; The deviation of the expected time series features from each time series feature in the time series feature set is analyzed based on the gradient boosting decision tree ensemble algorithm to obtain the deviation analysis results; The bucket adjustment strategy is determined based on the deviation analysis results.

5. The binning control method for a financial data warehouse according to claim 4, characterized in that, The deviation analysis results record the cosine similarity between the expected time-series feature and the time-series feature. The step of determining the bucketing adjustment strategy based on the deviation analysis results includes: For each of the aforementioned temporal features, if the cosine similarity meets the first threshold range, a horizontal splitting instruction generation process is triggered to obtain the horizontal splitting strategy elements; or... If the cosine similarity meets the second threshold range, a vertical expansion splitting instruction generation process is triggered to obtain vertical expansion strategy elements, wherein the value of the second threshold range is lower than the value of the first threshold range; or, If the cosine similarity meets the third threshold range, the cross-node migration instruction generation process is triggered to obtain cross-node migration strategy elements, wherein the value of the third threshold range is lower than the value of the second threshold range. By integrating all the aforementioned strategy elements, the bucket adjustment strategy is obtained.

6. The binning control method for a financial data warehouse according to claim 1, characterized in that, The steps for reconstructing the bucket index of the financial data warehouse based on the bucket adjustment strategy to obtain the index-updated financial data warehouse include: The bucketing adjustment strategy is analyzed to obtain the operation instructions; Execute the bucket structure adjustment operation according to the operation instructions; Update the global index mapping table of the financial data warehouse and record the adjusted bucket distribution status.

7. The binning control method for a financial data warehouse according to claim 6, characterized in that, The steps for parsing the bucket adjustment strategy and obtaining operation instructions include: When the bucketing adjustment strategy is a horizontal splitting strategy, the horizontal splitting strategy elements are read to obtain the horizontal splitting operation instruction, wherein the horizontal splitting operation instruction is used to instruct the following steps: create a new bucket with the same structure as the source bucket, allocate the specified splitting data in the source bucket to the new bucket, and establish the association relationship between the new bucket and the source bucket; When the bucket adjustment strategy is a vertical expansion strategy, the vertical expansion strategy elements are read to obtain the vertical expansion operation instruction, wherein the vertical expansion operation instruction is used to instruct the following steps: determine the computing resource quota of the new bucket based on the resource occupancy status of the source bucket, and create the new bucket based on the computing resource quota; When the bucket adjustment strategy is a cross-node migration strategy, the cross-node migration strategy elements are read to obtain a data migration instruction, wherein the data migration instruction is used to instruct the following steps: select a target migration node that meets the migration conditions, and migrate the bucket data to be migrated to the target migration node.

8. A bin control device for a financial data warehouse, characterized in that, include: The acquisition unit is used to acquire the time-series performance set of the financial data warehouse through a distributed acquisition system. The time-series performance set records performance indicators for M time steps. The categories of the performance indicators include: business system performance, data warehouse data table performance, and computing resource performance. M is a preset value. The extraction unit is used to perform multi-level feature extraction on the time series performance set using a sliding window mechanism to obtain a time series feature set. The types of time series features recorded in the time series feature set include: instantaneous fluctuation features, periodic trend features, and steady-state baseline features. The analysis unit is used to perform correlation analysis on the time-series feature set based on the decision tree ensemble algorithm, with the expected query performance requirements as the target, to obtain the bucketing adjustment strategy. The reconstruction unit is used to reconstruct the bucket index of the financial data warehouse based on the bucket adjustment strategy, so as to obtain the financial data warehouse with updated index.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the bucket control method for the financial data warehouse according to any one of claims 1 to 7.

10. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the bucket control method for a financial data warehouse as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The system includes computer instructions, wherein when executed by a processor, the computer instructions implement the steps of the bucket control method for the financial data warehouse as described in any one of claims 1 to 7.

Images