Financial data distributed storage management method and system

Abstract

The invention discloses a financial data distributed storage management method and system, and relates to the technical field of financial data management, and the method comprises the steps: accessing logs based on historical financial information, extracting historical financial data fragments, marking a predefined business label for each historical financial data fragment, generating business label mapping data, and storing the business label mapping data in a database; combining the service tags, associating the combined service tags with the correspondingly set task tags, extracting the periodic task tags, setting the periodic task tags as periodic task tags, and analyzing corresponding periodic task metadata; according to the method, the financial data fragments are labeled with the business labels, the corresponding task label and business label combination is defined to correspond to the historical financial data fragments according to the accessed records of the historical financial data fragments and the corresponding business labels, and the historical access records of the historical financial data associated with the business labels and the task labels are analyzed; the system can understand the access mode of the financial data fragment corresponding to each service under different task conditions.

Description

Technical Field

[0001] This invention relates to the field of financial data management technology, specifically to a distributed storage management method and system for financial data. Background Technology

[0002] Current distributed storage management methods or systems for financial data typically rely on real-time or periodic monitoring of node resources (such as CPU, memory, storage space, and request volume). When a certain indicator exceeds a preset static threshold, a balancing operation (such as replica migration or data partition adjustment) is triggered. Alternatively, they may rely solely on a single indicator—the access frequency of financial data—to identify "hot" data and allocate it to balance node resources. This makes it impossible for existing technologies to understand why financial resources are accessed so frequently, and to establish a logical relationship between financial data access and specific business operations. Consequently, distributed storage management cannot optimize the allocation and management of financial data based on actual business needs. Summary of the Invention

[0003] The purpose of this invention is to provide a distributed storage management method and system for financial data to address the aforementioned shortcomings in the prior art.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a distributed storage management method for financial data, comprising the following steps:

[0005] S1. Based on historical financial information access logs, extract historical financial data fragments and assign predefined business tags to each historical financial data fragment to generate business tag mapping data. Then, combine the business tags and associate them with the corresponding task tags, extract periodic task tags and set them as periodic task tags, and analyze the corresponding periodic task metadata. Among them, business tags include: accounting entity (such as company code, department, etc.), time period (such as fiscal year, accounting period, quarter, etc.), data source or type (such as voucher type, account set, data table, etc.), business cycle (such as revenue cycle, purchase payment cycle, expense cycle, etc.), business type (such as sales revenue, fixed asset purchase, employee salary, etc.), and business status (such as posting status, audit status, whether associated with settlement, etc.).

[0006] S2. Based on historical financial information access logs, obtain the timestamps of each historical financial data segment and the timestamps when it was accessed, and generate a financial access time series, which includes a daily access time series and multiple periodic task access time series.

[0007] S3. Based on the historical financial information access log, obtain the resource information and corresponding timestamp of each node in the distributed storage system when accessing each historical financial data shard, and generate a node resource information sequence, which includes a daily node resource information sequence and multiple periodic task node resource information sequences.

[0008] S4. Collect the nodes where historical financial data is stored in segments, generate node storage information data, and further, obtain data distribution snapshots from the metadata service of the distributed storage system at a set frequency to generate snapshot time, ID of historical financial data segment and its corresponding node ID.

[0009] S5. Based on the business tag mapping data, financial access time series, node resource information series and node storage information data, construct an objective function to analyze the variance of the resource occupancy rate of each node when each historical financial data fragment is stored in each node, according to the time distribution characteristics of the access of each historical financial data fragment in the financial access time series.

[0010] S6. Minimize the objective function to obtain the minimum variance of resource occupancy rates for each node and the storage status of corresponding historical financial data shards within each node, generating a recommended storage scheme for financial data shards. Commercial solvers (such as Gurobi or CPLEX) can be used for minimization, including genetic algorithms, simulated annealing, and tabu search. The obtained storage status of historical financial data shards within each node represents the storage status of the corresponding node. It contains fragments of historical financial data;

[0011] S7. Manage the distributed storage system based on the recommended storage scheme for the financial data shards. For example, introduce an intelligent scheduler to load the corresponding recommended scheme generated in S6 according to the current time or event, compare it with the current distribution, generate a detailed data migration work order, and migrate the historical financial data shards according to the generated data migration work order.

[0012] Furthermore, S1 includes the following steps:

[0013] S1.1 Based on historical financial information access logs, extract historical financial data fragments, and assign predefined business tags to each historical financial data fragment to generate business tag mapping data;

[0014] S1.2. Based on the business tag mapping data, perform search processing on the business tags corresponding to each historical financial data in the financial access time series to generate a corresponding financial data business tag combination;

[0015] S1.3. Associate the same financial data business tags with a predefined task tag to generate business task mapping data;

[0016] S1.4 Obtain the timestamp of the historical financial data slice corresponding to each task tag when it is accessed within a continuous time period of the set duration, and generate a task execution timestamp sequence;

[0017] S1.5. Based on the Fourier transform and autocorrelation analysis of the task execution timestamp sequence, analyze whether the corresponding task label meets the requirements for setting the task periodicity, and analyze the task period length of the task label that meets the requirements for setting the task periodicity (such as 1 day, 1 week, 1 month, 1 quarter or 1 year, etc.), and set the corresponding task label as a periodic task label.

[0018] S1.6 For each periodic task tag, based on the timestamp of the access to the corresponding historical financial data shard, analyze the typical start time, duration and load curve of the periodic task tag, and generate the corresponding periodic task window profile.

[0019] For example, during each task execution, the actual start time (first related query) and end time (last related query) can be determined based on the continuous access logs generated, and the duration of each execution can be calculated. Statistics can be compiled for all historical execution windows: average / latest start time: e.g., "Monthly closing tasks average start at 20:05 on the 25th of each month," average / longest duration: e.g., "Average duration 3.5 hours, historical longest 6 hours." The load curve within the window can be analyzed to identify whether execution is stable or has specific peaks. The typical start time can be selected as the time when 90% / 92% / 95% of tasks have started. For example, if 92% of all monthly closing tasks start at or before 20:15, then 20:15 can be selected as the typical start time for that monthly closing task.

[0020] S1.7 Collect and combine the periodic task window profile, financial data business tag combination, and historical financial data fragments corresponding to each periodic task tag to generate the corresponding periodic task metadata.

[0021] Furthermore, after the initial automatic identification of periodic tasks, they must be jointly reviewed by financial business experts and system administrators to confirm the accuracy of their cycle, window, and associated data, preventing misjudgments. Additionally, by acquiring access information for new financial data shards in real time, the window profiles of periodic tasks, objective functions, and nodes storing historical financial data shards can be updated periodically.

[0022] Furthermore, the analysis of whether the corresponding task tags meet the requirements for setting task periodicity includes the following steps:

[0023] Obtain the autocorrelation coefficient sequence from the autocorrelation analysis of the task execution timestamp sequence corresponding to the task tag;

[0024] Check whether each autocorrelation coefficient in the autocorrelation coefficient sequence is greater than the set statistical confidence interval;

[0025] If so, then determine whether the corresponding autocorrelation coefficient matches the set calendar period;

[0026] If so, add the calendar matching enhancement factor to the autocorrelation coefficient to obtain the task periodicity intensity data;

[0027] Select a task periodicity intensity data that is greater than the set task periodicity intensity threshold;

[0028] If so, set the corresponding task label as a periodic task label and select the time lag value corresponding to the autocorrelation coefficient as the task period length of the periodic task label.

[0029] Furthermore, S2 includes the following steps:

[0030] S2.1 Based on historical financial information access logs, obtain the timestamps of each historical financial data segment and the timestamps when it was accessed, and generate a financial access time series;

[0031] S2.2 Obtain the historical financial data fragments and timestamps corresponding to the task tags in the financial access time series, and generate the daily access time series;

[0032] S2.3. Obtain the historical financial data slices and timestamps corresponding to each periodic task label in the financial access time series, and generate the corresponding periodic task access time series.

[0033] Furthermore, S3 includes the following steps:

[0034] S3.1 Based on the historical financial information access log, obtain the resource information and corresponding timestamp of each node in the distributed storage system when accessing each historical financial data shard, and generate a node resource information sequence;

[0035] S3.2 Search the node resource information sequence for node resource information whose timestamps are the same as the timestamps of the daily access time sequence, and generate the daily node resource information sequence;

[0036] S3.3 Search the node resource information sequence for node resources whose timestamps are the same as the timestamps of the periodic task access time sequence, and generate the periodic task node resource information sequence.

[0037] Furthermore, S5 includes the following steps:

[0038] S5.1, The resource utilization rate of node j in scenario S is calculated using the following formula. Perform the calculation:

[0039] ,

[0040] Among them, scenario S includes daily access scenarios corresponding to daily objective functions and multiple periodic task access scenarios corresponding to periodic task objective functions; This represents the security threshold of resource R for node j, where resource R includes I / O and CPU. include and , This represents the disk I / O resource utilization rate of node j in scenario S. This represents the CPU resource utilization rate of node j in scenario S; Let represent the historical financial data shard i stored within node j, and let represent the storage distribution of the historical financial data shard in the distributed storage system. This represents the average number of times historical financial data shard i is accessed per unit time under scenario S; This represents the average R resource consumption expected to occur when accessing historical financial data shard i once in scenario S.

[0041] S5.2, The overall load rate of node j in scenario S is calculated using the following formula. Perform the calculation:

[0042] ,

[0043] in, and These are the weighting coefficients for IO resources and CPU resources, respectively. This is used to adjust the weight of IO and CPU load in the overall evaluation based on business importance or system bottleneck tendencies;

[0044] S5.3, based on each node in each scenario The variance of the overall load rate of each node in daily scenarios was analyzed. The variance of the overall load rate of each node under various periodic task scenarios Perform calculations and configure the variance weights accordingly. and each By performing a weighted summation, we obtain the objective function O:

[0045] ,

[0046] in, and They represent and the mth The weight, M represents The quantity of periodic task scenarios can be represented as: , and ;

[0047] Furthermore, S5.3 uses the setting of variance weights to configure the pair. and each Performing a weighted sum involves the following steps:

[0048] S5.31. Obtain the typical start time and duration of each periodic task based on the periodic task metadata, calculate the duration of each periodic task, and advance the start time of each periodic task's duration to set the periodic task preparation time to obtain the corresponding periodic task scenario duration.

[0049] S5.32 Determine whether the current time is within a periodic task scenario period;

[0050] S5.33. If not, then all Set the weight to zero, and The weight is set to 1; if so, the periodic task scenario time corresponding to the current time is assigned. The weights are set to the predefined variance weights of the periodic task scenario for this periodic task, and the rest... The weight is set to zero, and the value obtained by subtracting the variance weight of the periodic task scenario from 1 is set to... The weights of the variance in the objective function are further adjusted at the start of each time period corresponding to each scenario.

[0051] Furthermore, the following constraints are also included when performing the minimization solution in S6:

[0052] Each historical financial data shard must be assigned to a specified number (predefined value) of replica nodes;

[0053] The total size of the historical financial data shards stored on each node is less than or equal to the capacity of that node;

[0054] Resource utilization of each node in the sharded recommended storage scheme for the obtained financial data. Within the corresponding resource R security zone.

[0055] A distributed storage management system for financial data includes a distributed storage system, a processor, a display, and an input port;

[0056] The distributed storage system is used to store computer programs and distributed storage of historical financial data fragments.

[0057] The processor is used to execute the computer program to implement a distributed storage management method for financial data; the processor may also include an intelligent scheduler.

[0058] The display is used for visualizing data;

[0059] The input port is used to collect external data.

[0060] 1. Compared with the prior art, the present invention provides a distributed storage management method and system for financial data. By tagging financial data fragments with business tags, and defining corresponding task tags and business tag combinations to correspond with historical financial data fragments based on the historical access records of financial data fragments and corresponding business tags, and by analyzing the historical access records of historical financial data associated with business tags and task tags, it is possible to analyze whether there is a periodic pattern in the access of each historical financial data fragment and analyze the corresponding period duration. This enables the system to understand the access patterns of financial data fragments corresponding to each business under different task conditions.

[0061] 2. Compared with the prior art, the financial data distributed storage management method and system provided by the present invention constructs an objective function that can be dynamically adjusted according to different task scenarios (such as daily sporadic access scenarios or periodic task scenarios). This allows the storage scheme of the distributed storage system to adjust the distribution of storage nodes of historical financial data shards according to different task scenarios corresponding to the current time. This ensures that the resources of each node can be evenly allocated under different scenarios, achieving the effect of optimizing and managing the storage distribution scheme of the distributed storage system according to task and business needs. Attached Figure Description

[0062] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0063] Figure 1 This is a flowchart illustrating the method steps provided in an embodiment of the present invention;

[0064] Figure 2 This is a system structure block diagram provided for an embodiment of the present invention. Detailed Implementation

[0065] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0066] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0067] Exemplary embodiments will be described more fully below with reference to the accompanying drawings; however, these exemplary embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will enable those skilled in the art to fully understand the scope of this disclosure.

[0068] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0069] As used herein, the term “and / or” includes any and all combinations of one or more of the associated enumerated entries. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded.

[0070] The embodiments described herein can be described with reference to plan views and / or cross-sectional views using the ideal schematic diagrams of this disclosure. Therefore, the exemplary illustrations can be modified according to manufacturing techniques and / or tolerances. Therefore, the embodiments are not limited to those shown in the drawings, but include modifications to configurations formed based on manufacturing processes. Therefore, the areas illustrated in the drawings are schematic in nature, and the shapes of the areas shown in the figures illustrate specific shapes of areas of an element, but are not intended to be limiting. Unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art.

[0071] Please see Figure 1 A distributed storage management method for financial data includes the following steps:

[0072] S1. Based on historical financial information access logs, extract historical financial data fragments and assign predefined business tags to each historical financial data fragment to generate business tag mapping data. Then, combine the business tags and associate them with the corresponding task tags, extract periodic task tags and set them as periodic task tags, and analyze the corresponding periodic task metadata. Among them, business tags include: accounting entity (such as company code, department, etc.), time period (such as fiscal year, accounting period, quarter, etc.), data source or type (such as voucher type, account set, data table, etc.), business cycle (such as revenue cycle, purchase payment cycle, expense cycle, etc.), business type (such as sales revenue, fixed asset purchase, employee salary, etc.), and business status (such as posting status, audit status, whether associated with settlement, etc.).

[0073] Specifically, the following steps are included:

[0074] S1.1 Based on historical financial information access logs, extract historical financial data fragments, and assign predefined business tags to each historical financial data fragment to generate business tag mapping data;

[0075] S1.2 Based on the business tag mapping data, search and process the business tags corresponding to each historical financial data in the financial access time series to generate the corresponding financial data business tag combination;

[0076] S1.3. Associate the same financial data business tags with a predefined task tag to generate business task mapping data;

[0077] S1.4 Obtain the timestamp of the historical financial data slice corresponding to each task tag when it is accessed within a continuous time period of the set duration, and generate a task execution timestamp sequence;

[0078] S1.5. Based on Fourier transform and autocorrelation analysis of the task execution timestamp sequence, analyze whether the corresponding task label meets the requirements for setting the task periodicity, and analyze the task period length of the task label that meets the requirements for setting the task periodicity (such as 1 day, 1 week, 1 month, 1 quarter or 1 year, etc.), and set the corresponding task label as a periodic task label.

[0079] S1.6 For each periodic task tag, based on the timestamp of the access to the corresponding historical financial data shard, analyze the typical start time, duration and load curve of the periodic task tag, and generate the corresponding periodic task window profile.

[0080] For example, during each task execution, the actual start time (first related query) and end time (last related query) can be determined based on the continuous access logs generated, and the duration of each execution can be calculated. Statistics can be compiled for all historical execution windows: average / latest start time: e.g., "Monthly closing tasks average start at 20:05 on the 25th of each month," average / longest duration: e.g., "Average duration 3.5 hours, historical longest 6 hours." The load curve within the window can be analyzed to identify whether execution is stable or has specific peaks. The typical start time can be selected as the time when 90% / 92% / 95% of tasks have started. For example, if 92% of all monthly closing tasks start at or before 20:15, then 20:15 can be selected as the typical start time for that monthly closing task.

[0081] S1.7 Collect and combine the periodic task window profile, financial data business tag combination, and historical financial data fragments corresponding to each periodic task tag to generate the corresponding periodic task metadata.

[0082] Furthermore, after the initial automatic identification of periodic tasks, they must be jointly reviewed by financial business experts and system administrators to confirm the accuracy of their cycle, window, and associated data, preventing misjudgments. Additionally, by acquiring access information for new financial data shards in real time, the window profiles of periodic tasks, objective functions, and nodes storing historical financial data shards can be updated periodically.

[0083] Furthermore, S1.5 analyzes whether the corresponding task tags meet the set task periodicity requirements, including the following steps:

[0084] S1.51 Obtain the autocorrelation coefficient sequence from the autocorrelation analysis of the task execution timestamp sequence corresponding to the task label;

[0085] S1.52. Check whether each autocorrelation coefficient in the autocorrelation coefficient sequence is greater than the set statistical confidence interval;

[0086] S1.53 If yes, then determine whether the corresponding autocorrelation coefficient matches the set calendar period;

[0087] S1.54. If so, add the calendar matching enhancement factor to the autocorrelation coefficient to obtain the task periodicity intensity data.

[0088] S1.55, Select task periodicity intensity data that is greater than the set task periodicity intensity threshold;

[0089] S1.56. If so, set the corresponding task label as a periodic task label and select the time lag value corresponding to the autocorrelation coefficient as the task period length of the periodic task label.

[0090] S2. Based on historical financial information access logs, obtain the timestamps of each historical financial data segment and the timestamps when it was accessed, and generate a financial access time series. The financial access time series includes daily access time series and multiple periodic task access time series, including the following steps:

[0091] S2.1 Based on historical financial information access logs, obtain the timestamps of each historical financial data segment and the timestamps when it was accessed, and generate a financial access time series;

[0092] S2.2 Obtain the historical financial data fragments and timestamps corresponding to the task tags in the financial access time series, and generate the daily access time series;

[0093] S2.3. Obtain the historical financial data slices and timestamps corresponding to each periodic task label in the financial access time series, and generate the corresponding periodic task access time series.

[0094] S3. Based on historical financial information access logs, obtain the resource information and corresponding timestamps of each node in the distributed storage system when accessing each historical financial data shard, and generate a node resource information sequence. The node resource information sequence includes a daily node resource information sequence and multiple periodic task node resource information sequences, including the following steps:

[0095] S3.1 Based on the historical financial information access log, obtain the resource information and corresponding timestamp of each node in the distributed storage system when accessing each historical financial data shard, and generate a node resource information sequence;

[0096] S3.2 Search for node resource information in the search node resource information sequence whose timestamps are the same as the timestamps of the daily access time sequence, and generate the daily node resource information sequence;

[0097] S3.3 Search for node resource information in the search node resource information sequence whose timestamps are the same as the timestamps of the periodic task access time sequence, and generate the periodic task node resource information sequence.

[0098] S4. Collect the nodes where historical financial data is stored in segments, generate node storage information data, and further, obtain data distribution snapshots from the metadata service of the distributed storage system at a set frequency to generate snapshot time, ID of historical financial data segment and its corresponding node ID.

[0099] S5. Based on business tag mapping data, financial access time series, node resource information series, and node storage information data, construct an objective function to analyze the variance of node resource utilization when simulating access to historical financial data fragments, assuming each historical financial data fragment is stored within each node, according to the time distribution characteristics of access to each historical financial data fragment in the financial access time series. The specific steps are as follows:

[0100] S5.1, The resource utilization rate of node j in scenario S is calculated using the following formula. Perform the calculation:

[0101] ,

[0102] Among them, scenario S includes daily access scenarios corresponding to daily objective functions and multiple periodic task access scenarios corresponding to periodic task objective functions; This represents the security threshold of resource R for node j, where resource R includes I / O and CPU. include and , This represents the disk I / O resource utilization rate of node j in scenario S. This represents the CPU resource utilization rate of node j in scenario S; Let represent the historical financial data shard i stored within node j, and let represent the storage distribution of the historical financial data shard in the distributed storage system. This represents the average number of times historical financial data shard i is accessed per unit time under scenario S; This represents the average R resource consumption expected to occur when accessing historical financial data shard i once in scenario S.

[0103] S5.2, The overall load rate of node j in scenario S is calculated using the following formula. Perform the calculation:

[0104] ,

[0105] in, and These are the weighting coefficients for IO resources and CPU resources, respectively. This is used to adjust the weight of IO and CPU load in the overall evaluation based on business importance or system bottleneck tendencies;

[0106] S5.3, based on each node in each scenario The variance of the overall load rate of each node in daily scenarios was analyzed. The variance of the overall load rate of each node under various periodic task scenarios Perform calculations and configure the variance weights accordingly. and each By performing a weighted summation, we obtain the objective function O:

[0107] ,

[0108] in, and They represent and the mth The weight, M represents The quantity of periodic task scenarios can be represented as: , and ;

[0109] Among them, the variance weight configuration is used. and each Weighted summation is achieved through the following steps:

[0110] S5.31. Obtain the typical start time and duration of each periodic task based on the periodic task metadata, calculate the duration of each periodic task, and advance the start time of each periodic task's duration to set the periodic task preparation time to obtain the corresponding periodic task scenario duration.

[0111] S5.32 Determine whether the current time is within a periodic task scenario period;

[0112] S5.33. If not, then all Set the weight to zero, and The weight is set to 1; if so, the periodic task scenario time corresponding to the current time is assigned. The weights are set to the predefined variance weights of the periodic task scenario for this periodic task, and the rest... The weight is set to zero, and the value obtained by subtracting the variance weight of the periodic task scenario from 1 is set to... The weights of the variance in the objective function are further adjusted at the start of each time period corresponding to each scenario.

[0113] S6. Minimize the objective function to obtain the minimum variance of resource utilization for each node and the corresponding storage status of historical financial data shards within each node, generating a recommended storage scheme for financial data shards. Commercial solvers (such as Gurobi or CPLEX) can be used for minimization, including genetic algorithms, simulated annealing, and tabu search. The obtained storage status of historical financial data shards within each node represents the storage scheme for that node. It contains fragments of historical financial data.

[0114] During the minimization process, the following constraints must also be satisfied:

[0115] 1. Each historical financial data shard must be assigned to a specified number (predefined value) of replica nodes.

[0116] 2. The total size of the historical financial data shards stored on each node cannot exceed the capacity of that node.

[0117] 3. Resource utilization rate of each node in the recommended storage scheme for the obtained financial data sharding. Within the corresponding resource R security range.

[0118] S7. Manage the distributed storage system by recommending storage solutions based on financial data shards. For example, by introducing an intelligent scheduler, the system loads the corresponding recommended solution generated by S6 based on the current time or event, compares it with the current distribution, generates detailed data migration work orders, and migrates historical financial data shards according to the generated data migration work orders.

[0119] Please see Figure 2 The present invention also provides a distributed storage management system for financial data, including a distributed storage system, a processor, a display, and an input port;

[0120] Distributed storage systems are used to store computer programs and distributed storage of historical financial data shards;

[0121] The processor is used to execute computer programs to implement the distributed storage and management method for financial data provided by the present invention; the processor may also include an intelligent scheduler.

[0122] A monitor is used to visualize data;

[0123] The input port is used to collect external data.

[0124] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A distributed storage management method for financial data, characterized in that, Includes the following steps: S1. Based on historical financial information access logs, extract historical financial data fragments and assign predefined business tags to each historical financial data fragment to generate business tag mapping data. Then, combine the business tags and associate them with the corresponding task tags, extract the periodic task tags and set them as periodic task tags, and analyze the corresponding periodic task metadata. S2. Based on historical financial information access logs, obtain the timestamps of each historical financial data segment and the timestamps when it was accessed, and generate a financial access time series, which includes a daily access time series and multiple periodic task access time series. S3. Based on the historical financial information access log, obtain the resource information and corresponding timestamp of each node in the distributed storage system when accessing each historical financial data shard, and generate a node resource information sequence, which includes a daily node resource information sequence and multiple periodic task node resource information sequences. S4. Collect the nodes where historical financial data is stored in segments and generate node storage information data; S5. Based on the business tag mapping data, financial access time series, node resource information series and node storage information data, construct an objective function to analyze the variance of the resource occupancy rate of each node when each historical financial data fragment is stored in each node, according to the time distribution characteristics of the access of each historical financial data fragment in the financial access time series. S6. Minimize the objective function to obtain the minimum variance of resource utilization rate of each node and the storage status of corresponding historical financial data shards in each node, and generate a recommended storage scheme for financial data shards. S7. Based on the financial data, recommend storage schemes and manage the distributed storage system.

2. The distributed storage management method for financial data according to claim 1, characterized in that, S1 includes the following steps: S1.1 Based on historical financial information access logs, extract historical financial data fragments, and assign predefined business tags to each historical financial data fragment to generate business tag mapping data; S1.

2. Based on the business tag mapping data, perform search processing on the business tags corresponding to each historical financial data in the financial access time series to generate a corresponding financial data business tag combination; S1.

3. Associate the same financial data business tags with a predefined task tag to generate business task mapping data; S1.4 Obtain the timestamp of the historical financial data slice corresponding to each task tag when it is accessed within a continuous time period of the set duration, and generate a task execution timestamp sequence; S1.

5. Based on the Fourier transform and autocorrelation analysis of the task execution timestamp sequence, analyze whether the corresponding task label meets the set task periodicity requirement, and analyze the task period length of the task label that meets the set task periodicity requirement, and set the corresponding task label as a periodic task label. S1.6 For each periodic task tag, based on the timestamp of the access to the corresponding historical financial data shard, analyze the typical start time, duration and load curve of the periodic task tag, and generate the corresponding periodic task window profile. S1.7 Collect and combine the periodic task window profile, financial data business tag combination, and historical financial data fragments corresponding to each periodic task tag to generate the corresponding periodic task metadata.

3. The distributed storage and management method for financial data according to claim 2, characterized in that, The analysis of whether the corresponding task tags meet the set task periodicity requirements includes the following steps: Obtain the autocorrelation coefficient sequence from the autocorrelation analysis of the task execution timestamp sequence corresponding to the task tag; Check whether each autocorrelation coefficient in the autocorrelation coefficient sequence is greater than the set statistical confidence interval; If so, then determine whether the corresponding autocorrelation coefficient matches the set calendar period; If so, add the calendar matching enhancement factor to the autocorrelation coefficient to obtain the task periodicity intensity data; Select a task periodicity intensity data that is greater than the set task periodicity intensity threshold; If so, set the corresponding task label as a periodic task label and select the time lag value corresponding to the autocorrelation coefficient as the task period length of the periodic task label.

4. The distributed storage management method for financial data according to claim 1, characterized in that, S2 includes the following steps: S2.1 Based on historical financial information access logs, obtain the timestamps of each historical financial data segment and the timestamps when it was accessed, and generate a financial access time series; S2.2 Obtain the historical financial data fragments and timestamps corresponding to the task tags in the financial access time series, and generate the daily access time series; S2.

3. Obtain the historical financial data slices and timestamps corresponding to each periodic task label in the financial access time series, and generate the corresponding periodic task access time series.

5. The distributed storage management method for financial data according to claim 1, characterized in that, S3 includes the following steps: S3.1 Based on the historical financial information access log, obtain the resource information and corresponding timestamp of each node in the distributed storage system when accessing each historical financial data shard, and generate a node resource information sequence; S3.2 Search the node resource information sequence for node resource information whose timestamps are the same as the timestamps of the daily access time sequence, and generate the daily node resource information sequence; S3.3 Search the node resource information sequence for node resources whose timestamps are the same as the timestamps of the periodic task access time sequence, and generate the periodic task node resource information sequence.

6. The distributed storage management method for financial data according to claim 1, characterized in that, S5 includes the following steps: S5.1, The resource utilization rate of node j in scenario S is calculated using the following formula. Perform the calculation: ， Among them, scenario S includes daily access scenarios corresponding to daily objective functions and multiple periodic task access scenarios corresponding to periodic task objective functions; This represents the security threshold of resource R for node j, where resource R includes disk I / O and CPU. include and , This represents the disk I / O resource utilization rate of node j in scenario S. This represents the CPU resource utilization rate of node j in scenario S; This represents the historical financial data shard i stored within node j; This represents the average number of times historical financial data shard i is accessed per unit time under scenario S; This represents the average R resource consumption expected to occur when accessing historical financial data shard i once in scenario S. S5.2, The overall load rate of node j in scenario S is calculated using the following formula. Perform the calculation: ， in, and These are the weighting coefficients for IO resources and CPU resources, respectively. ; S5.3, based on each node in each scenario The variance of the overall load rate of each node in daily scenarios was analyzed. The variance of the overall load rate of each node under various periodic task scenarios Perform calculations and configure the variance weights accordingly. and each We perform a weighted summation to obtain the objective function O.

7. A distributed storage management method for financial data according to claim 6, characterized in that, S5.3 uses the setting of variance weights to configure pairs. and each Performing a weighted sum involves the following steps: S5.

31. Based on the metadata of periodic tasks, obtain the typical start time and duration of each periodic task, calculate the duration of each periodic task, and advance the start time of the duration of each periodic task to set the preparation time of the periodic task to obtain the corresponding periodic task scenario duration. S5.32 Determine whether the current time is within a periodic task scenario period; S5.

33. If not, then all Set the weight to zero, and The weight is set to 1; if so, the periodic task scenario time corresponding to the current time is assigned. The weights are set to the predefined variance weights of the periodic task scenario for this periodic task, and the rest... The weight is set to zero, and the value obtained by subtracting the variance weight of the periodic task scenario from 1 is set to... The weight.

8. A distributed storage management method for financial data according to claim 6, characterized in that, The following constraints are also included when performing the minimization solution in S6: Each historical financial data shard must be assigned to a specified number (predefined value) of replica nodes; The total size of the historical financial data shards stored on each node is less than or equal to the capacity of that node; Resource utilization of each node in the sharded recommended storage scheme for the obtained financial data. Within the corresponding resource R security zone.

9. A distributed storage management system for financial data, used to execute the distributed storage management method for financial data as described in any one of claims 1-8, characterized in that: Includes distributed storage systems, processors, displays, and input ports; The distributed storage system is used to store computer programs and distributed storage of historical financial data fragments. The processor is used to execute the computer program to implement the financial data distributed storage management method according to any one of claims 1-8; The display is used for visualizing data; The input port is used to collect external data.

Images