Fund asset data record distributed receiving and storing method and system

Abstract

The invention belongs to the technical field of distributed storage, and provides a fund asset data record distributed receiving storage method and system, and the method specifically comprises the steps: carrying out the recognition in a data storage environment, obtaining each server node and a data label, obtaining the label effectiveness state of the data label from the server node, and obtaining the label effectiveness state of the data label; and calculating a label thermal strategy coefficient according to the label efficacy, identifying a thermal label from each server node by using the label thermal strategy coefficient, and finally recording and storing data in combination with a thermal label judgment result. The hot fragmentation risk caused by accumulation of the content key imbalance phenomenon of each server node under the same data label is effectively quantified, so that the key distinguishing capability of the server node on the storage content and the data content is improved; the low-efficiency performance risk that individual server nodes become functional bottlenecks in a distributed storage system under the influence of fund data high business is reduced, and the service performance and efficiency of fund asset data are improved.

Description

Technical Field

[0001] The present invention belongs to the field of distributed storage technology, and in particular relates to a method and system for distributed receiving and storing data records of fund assets. Background Art

[0002] In traditional financial markets, fund asset data records often rely on centralized data processing systems, which often have the risk of single-point failure of data storage and data leakage caused by network attacks. Therefore, DLT distributed ledger technology and blockchain technology have been derived. At the same time, these technologies have gradually been used to effectively expand nodes in distributed systems to cope with the conflict between the sudden increase in fund asset data and maintaining system complexity in a relatively stable state. Existing technologies often use data sharding mechanisms to resolve this trade-off competition. By horizontally splitting data to different nodes, the scalability of distributed systems can be improved, which is an important means to cope with massive data processing. However, shards are usually divided according to certain rules, such as hash or range sharding by fund ID. This static division does not take into account the dynamic changes of data access. Data access in the fund market is highly dynamic and uncertain. The trading volume of some funds may surge in a specific period of time, causing the load of these shards to increase rapidly and form a hot sharding phenomenon. That is, under high load conditions in a distributed system, due to the uneven load distribution of data shards, some shards carry significantly higher access requests than other shards, resulting in the performance bottleneck of related nodes. In particular, high-frequency trading currently accounts for an increasingly high proportion of daily trading volume, which will further lead to increased request processing delays and even node downtime, directly affecting the overall performance of the system, especially the response speed, which is quite important in fund asset data processing.

[0003] However, the hot sharding phenomenon under high load in distributed systems is generally reduced by server load balancing algorithms, ignoring the hot sharding phenomenon caused by the accumulation of critical imbalance of the content of each node itself. The critical imbalance effect is caused because the data carried by different data tags or shards has different business importance. For example, high-frequency transactions and asset data of popular funds may be read and written more frequently or have more burst characteristics than other data. Even if the load generated by them is generally balanced in time observation, due to the high business impact of these contents, some nodes are more likely to become bottlenecks of distributed storage systems. In addition, traditional hash or range sharding strategies usually only consider the distribution of data, but cannot dynamically respond to changes in market demand. Therefore, when the statically divided data suddenly increases in load, it is impossible to consider the sudden high demand of the business, resulting in excessive load on the shards of specific data. Summary of the invention

[0004] The purpose of the present invention is to propose a distributed receiving and storing method and system for fund asset data records to solve one or more technical problems existing in the prior art and at least provide a beneficial choice or create conditions.

[0005] In order to achieve the above object, according to one aspect of the present invention, a method for distributed receiving and storing data records of fund assets is provided, the method comprising the following steps: Configure server nodes and data tags in the data storage environment; obtain the tag validity state of the data tag from the server node; calculate the tag thermal strategy coefficient according to the tag validity state, and use the tag thermal strategy coefficient to identify the thermal system tags from each server node; record and store data in combination with the thermal system tag determination results; Further, the method for identifying and obtaining each server node and data label in the data storage environment is as follows: the data storage environment includes a plurality of server nodes, referred to as nodes; The data records of fund assets are sharded and stored in each node through the data sharding mechanism. The data records are composed of several data tags. The several data tags of the data records are stored in different nodes respectively. Each node stores the fund asset data of each data tag.

[0006] The data labels of fund asset data types are the shards processed by the regularized data sharding algorithm; or the data labels are fund type, fund name, fund manager, fund unit net value, dividend information, net asset value or investment grouping, etc.

[0007] Furthermore, a method for obtaining the label effectiveness state of a data label from a server node is: a preset monitoring interval rg, rg∈[0.5, 24] hours, with a default value of 1 hour; obtaining the label effectiveness state of the data label every rg, wherein the label effectiveness state of the data label is the product of the operation order value and the label pressure, and the calculation method of the operation order value is: the average value of the CPU occupancy rate of the server node within the monitoring interval is recorded as the occupancy, the minimum value of the occupancy of all nodes is recorded as the occupancy base value, then the occupancy of any node is subtracted from the occupancy base value, and the ratio of the difference to the occupancy base value is taken as its operation order value; the calculation method of the label pressure is: in a node, the number of read and write times of the same data label is taken as the unit read and write amount, and the value set consisting of the corresponding read and write amounts of each data label under the node is minmax normalized, and the normalized value of the unit read and write amount of the data label is defined as its label pressure.

[0008] Furthermore, a method for calculating the tag thermal strategy coefficient according to the tag effectiveness state is: using any data tag as a determination tag and a pre-set time interval as a monitoring interval; Alternatively, for the judgment label, the moment when the label effective state value is zero is taken as the zero point; any node searches in the reverse time direction from the current moment to obtain the first zero point as the current zero point of the node, and the number of moments between the current moment and the current zero point is taken as the zero point span of the node; from the set composed of the zero point span of each node, the median value Zps.mid, the interquartile range Zps.ir and the maximum value Zps.mx of the set are obtained respectively, and the zero point interval Zps_rg is constructed, Zps_rg=min{(Zps.mid+Zps.ir),Zps.mx}; the period from the current moment to the Zps_rg in the reverse time direction is taken as the monitoring interval; The thermal tag is identified based on the tag validity state of the tag within the monitoring range.

[0009] Furthermore, the method for identifying the thermal system label according to the label validity state of the label determined in the monitoring interval is: in the monitoring interval, respectively obtain the minimum value of the label validity state of the same node at different times and at different nodes at the same time, and record them as the first validity state quasi-value and the second validity state quasi-value; define the difference between the label validity state and the first validity state quasi-value as the quasi-node difference dimension SLbv; For any node, if the first effective state quasi-value and the second effective state quasi-value corresponding to any non-zero label effective state are both zero, then the time corresponding to the label effective state is recorded as the effective state valley point; The role of the effective state valley point can be regarded as the key feature on the label effective state curve. By analyzing these effective state valley points, we can directly capture the local fluctuations of load changes in the time series, especially those sudden drops. Therefore, it can play a role similar to a warning line in the time series, marking the inflection point of low-value data. The inflection point, as a benchmark for relative steady state, provides a more scientific metric for quantifying the changing trajectory of the system load.

[0010] Construct all effective state valley points at all times into a set Fn. The number of elements in the set Fn is recorded as Sfn. The difference between the label effective state of any effective state valley point and the minimum value of the label effective state in the set Fn is recorded as the node difference dimension of the effective state valley point. The principle of constructing the effective state valley points as a set to calculate the quasi-node difference dimension is that combining the difference of the time series can enable the algorithm to quickly analyze the critical fluctuations of the load with relatively low computational complexity, so that the system is more sensitive to sudden changes in the load.

[0011] The label effectiveness at any moment is taken as the valley threshold. The ratio of the number of elements whose label effectiveness is less than the valley threshold to the number of elements whose label effectiveness is greater than the valley threshold in the set Fn is recorded as the effectiveness variability Eff at that moment. If the label effectiveness of all elements in the set Fn is greater than the valley threshold, or less than the valley threshold, then the effectiveness variability at that moment is 1 / Sfn. The label thermal strategy coefficient Ltsc calculated according to the effectiveness valley is calculated as follows: ; Where j1 is the accumulated variable, SLbv j1 and Lbv j1 are the quasi-node difference dimension and node difference dimension of the j1th effective state valley point, exp() represents the exponential function with the natural constant e as the base, max() is the function for obtaining the maximum value, and lg() is the logarithmic function with 10 as the base; Since the label thermal strategy coefficient calculated by using the node difference dimension focuses on the basic effective state difference identification, the hot sharding risk caused by the accumulation of critical imbalance of the content of each node itself can be quickly quantified in real time. However, since this method has a strong dependence on the original data, it will lead to the overfitting problem of quantitative analysis, especially when there are a large number of high-frequency label effective states that suddenly increase. However, the existing technology cannot solve the overfitting problem induced by the strong dependence of the real-time quantization process on the original data. In order to better solve this problem and eliminate the phenomenon of data distortion caused by overfitting, the present invention proposes a more preferred solution as follows: Preferably, the method for identifying thermal system tags based on determining the tag effectiveness state of the tag within the monitoring interval can be replaced by: within the monitoring interval, if a tag effectiveness state value at a moment is greater than that at the previous moment and the tag effectiveness state value at the previous moment is not zero, then the moment is defined as an effectiveness state increase; the average value of each effectiveness state increase is recorded as the effectiveness state phase value, and if the tag effectiveness state at any moment is greater than the effectiveness state phase value, it is determined that it meets the first phase value condition; An increase-position binary is formed by any effective state increase position and its corresponding label effective state. The set constructed by the increase-position binary is interpolated by cubic spline using plrep and splev functions to obtain a smooth curve. The difference between the curve value of any ineffective state increase position and the label effective state is the increase-position estimation value. When the increase-position estimation value is less than zero, the ineffective state increase position is inefficiently estimated, otherwise it is effectively estimated. In the fund asset management system, high-frequency trading may lead to uneven data collection density. Therefore, the fitting curve constructed by the effective state phase value can be used to fill in the missing data points of the density. Especially in the scenario of a sudden increase in load, the fitting curve can accurately estimate the load peak through the fluctuation trend of the effective states of the previous and next labels.

[0012] Searching in reverse time direction from any effective state increment position, obtaining the first non-continuous effective state increment position as its reference position, and taking the time period between the effective state increment position and its reference position as an estimated reference interval; eliminating the estimated reference interval containing the zero point position; The strategic sensitivity value Psenv of the estimated reference interval is calculated according to the incremental estimation: Psenv = Len × lg (1 + eff.rp × eff.dsv); where eff.rp represents the proportion of the moments with valid estimation between the estimated reference interval and the first zero point in the reverse time direction, and eff.dsv is the maximum value of the incremental estimation label effectiveness state of each moment with valid estimation in the estimated reference interval; Len is the proportion of the number of moments in the estimated reference interval to the total number of moments in the monitoring interval, and lg is the logarithmic sign with 10 as the base; If the number of effective estimates in the estimation reference interval is greater than the number of inefficient estimates, the estimation reference interval is considered to meet the second phase condition; the first phase condition and the second phase condition are both used as phase conditions, and the number of phase conditions that are met at the current moment is recorded as the phase condition achievement value Glv; Calculate the label thermal strategy coefficient Ltsc based on each estimated reference interval: ; Where e is a natural constant, i1 is the serial number of the estimated reference interval, MPs represents the average value of the strategy sensitivity value corresponding to all estimated reference intervals, noz is the number of estimated reference intervals, avg{} is the average function, and Psenv i1 is the strategy sensitivity value of the i1th estimated reference interval, exp() is an exponential function with the natural constant e as the base, rk() is the ranking coefficient function, and the return value obtained by the ranking coefficient function rk(i1) is: obtain the maximum value of the label efficacy state in the i1th estimated reference interval, and take the percentile value of the obtained maximum value in all label efficacy states in the monitoring interval as the return value of the ranking coefficient function.

[0013] Beneficial effects: Since the label thermal strategy coefficient is calculated based on the estimated reference interval, it can accurately mark and downgrade the location where the load pressure collapse occurs, thereby forming a staged defect analysis of the cumulative effect of thermal risk. Therefore, it can effectively quantify the thermal sharding risk caused by the accumulation of content criticality imbalance phenomena of each server node under the same data label, improve the server node's ability to distinguish the criticality of its storage content and data content, and reduce the performance inefficiency risk of individual server nodes becoming functional bottlenecks in the distributed storage system under the high business impact of fund data.

[0014] Furthermore, the method of using the label thermal strategy coefficient to identify the thermal label from each server node is: for any data label, define the median value of the label thermal strategy coefficient corresponding to each node as the strategy division variable. If the label thermal strategy coefficient of the node is greater than the strategy division variable, then the data label is considered to be a thermal label at the node, otherwise it is a non-thermal label.

[0015] Furthermore, a method for storing data records in combination with the thermal label determination result is: any data record to be written contains several data labels, and thermal risk exclusion storage is performed for each of the data labels, specifically: each storage node of the current data label is obtained, if the current data label in any storage node is a thermal label, the node is defined as a risk node, and the thermal risk exclusion return value is obtained: when all storage nodes are risk nodes, the return value is considered to be FALSE, and a storage node is reselected for the new current data label, otherwise the return value is TRUE; through thermal risk exclusion storage, the thermal risk exclusion return values ​​of all data labels are TRUE, and the data record to be written is stored.

[0016] Preferably, all undefined variables in the present invention, if not clearly defined, can be manually set thresholds.

[0017] The present invention also provides a distributed receiving and storing system for data records of fund assets, the distributed receiving and storing system for data records of fund assets comprising: a processor, a memory, and a computer program stored in the memory and executable on the processor, the processor implementing the steps in the distributed receiving and storing method for data records of fund assets when executing the computer program, the distributed receiving and storing system for data records of fund assets can be run in computing devices such as desktop computers, notebook computers, PDAs, and cloud data centers, the executable system may include, but is not limited to, a processor, a memory, and a server cluster, the processor executing the computer program runs in the following system units: A distributed storage environment arrangement unit is used to identify and obtain each server node and data label in the data storage environment; A real-time monitoring unit, used to obtain the label validity status of the data label from the server node; A thermal label dynamic identification unit is used to calculate the label thermal strategy coefficient according to the label effectiveness state, and use the label thermal strategy coefficient to identify the thermal label from each server node; The new data storage unit is used to record and store data in combination with the thermal tag determination results.

[0018] The beneficial effects of the present invention are as follows: the present invention provides a distributed receiving and storage method and system for data records of fund assets, forms a phased defect analysis on the cumulative effect of thermal risk, effectively quantifies the thermal sharding risk caused by the accumulation of content criticality imbalance phenomena of each server node under the same data label, thereby improving the server node's ability to distinguish the criticality of its storage content and data content, reducing the performance inefficiency risk of individual server nodes becoming functional bottlenecks in the distributed storage system under the high business impact of fund data, providing a more stable distributed storage strategy for fund assets in a high-frequency reading and writing data environment, and improving the service performance and efficiency of fund asset data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features of the present invention will become more obvious by describing in detail the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings of the present invention represent the same or similar elements. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work. In the accompanying drawings: Figure 1 Shown is a flow chart of a distributed receiving and storing method for data records of fund assets; Figure 2 Shown is the structure diagram of the distributed receiving and storage system for fund asset data records. DETAILED DESCRIPTION

[0020] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0021] like Figure 1 The flowchart of the distributed receiving and storing method of fund asset data records is shown below. Figure 1 The distributed receiving and storing method of fund asset data records according to an embodiment of the present invention is described below. The method comprises the following steps: Configure server nodes and data tags in the data storage environment; obtain the tag validity state of the data tag from the server node; calculate the tag thermal strategy coefficient according to the tag validity state, and use the tag thermal strategy coefficient to identify the thermal system tags from each server node; record and store data in combination with the thermal system tag determination results; Further, the method for identifying and obtaining each server node and data label in the data storage environment is as follows: the data storage environment includes a plurality of server nodes, referred to as nodes; The data records of fund assets are sharded and stored in each node through the data sharding mechanism. The data records are composed of several data tags. The several data tags of the data records are stored in different nodes respectively. Each node stores the fund asset data of each data tag.

[0022] The data labels include fund type, fund name, fund manager, fund unit net value, dividend information, net asset value and investment grouping.

[0023] Furthermore, the method of obtaining the label status of the data label from the server node is: preset the monitoring interval rg, rg is 1 hour hour; The label effectiveness state of the data label is obtained every rg, where the label effectiveness state of the data label is the product of the operation order value and the label pressure. The calculation method of the operation order value is as follows: the average value of the CPU occupancy rate of the server node in the monitoring interval is recorded as the occupancy, and the minimum value of the occupancy of all nodes is recorded as the occupancy base value. The occupancy of any node is subtracted from the occupancy base value, and the ratio of the difference to the occupancy base value is taken as its operation order value; the calculation method of the label pressure is as follows: in a node, the number of read and write times of the same data label is taken as the unit read and write amount, and the value set consisting of the corresponding read and write amounts of each data label under the node is minmax normalized, and the normalized value of the unit read and write amount of the data label is defined as its label pressure.

[0024] Furthermore, a method for calculating the label thermal strategy coefficient according to the label effectiveness state is: taking any data label as a judgment label; Alternatively, for the judgment label, the moment when the label effective state value is zero is taken as the zero point; any node searches in the reverse time direction from the current moment to obtain the first zero point as the current zero point of the node, and the number of moments between the current moment and the current zero point is taken as the zero point span of the node; from the set composed of the zero point span of each node, the median value Zps.mid, the interquartile range Zps.ir and the maximum value Zps.mx of the set are obtained respectively, and the zero point interval Zps_rg is constructed, Zps_rg=min{(Zps.mid+Zps.ir),Zps.mx}; the period from the current moment to the Zps_rg in the reverse time direction is taken as the monitoring interval; The interquartile range is the difference between the upper quartile and the lower quartile, and min{} is the minimum function.

[0025] The thermal tag is identified based on the tag validity state of the tag within the monitoring range.

[0026] Furthermore, the method for identifying the thermal system label according to the label validity state of the judgment label in the monitoring interval is: in the monitoring interval, respectively obtain the minimum value of the label validity state of the same node at different times and at different nodes at the same time, and record them as the first validity state quasi-value and the second validity state quasi-value; define the difference between the label validity state and the first validity state quasi-value as the quasi-node difference dimension SLbv; the label validity state only adopts the corresponding value under the judgment label; For any node, if the first effective state quasi-value and the second effective state quasi-value corresponding to any non-zero label effective state are both zero, then the time corresponding to the label effective state is recorded as the effective state valley point; Construct all effective state valley points at all times into a set Fn. The number of elements in the set Fn is recorded as Sfn. The difference between the label effective state of any effective state valley point and the minimum value of the label effective state in the set Fn is recorded as the node difference dimension of the effective state valley point. The label effectiveness at any moment is taken as the valley threshold. The ratio of the number of elements whose label effectiveness is less than the valley threshold to the number of elements whose label effectiveness is greater than the valley threshold in the set Fn is recorded as the effectiveness variability Eff at that moment. If the label effectiveness of all elements in the set Fn is greater than the valley threshold, or less than the valley threshold, then the effectiveness variability at that moment is 1 / Sfn. The label thermal strategy coefficient Ltsc calculated according to the effectiveness valley is calculated as follows: ; Where j1 is the accumulated variable, SLbv j1 and Lbv j1 are the quasi-node difference dimension and node difference dimension of the j1th effective state valley point, exp() represents the exponential function with the natural constant e as the base, max() is the function for obtaining the maximum value, and lg() is the logarithmic function with 10 as the base; Preferably, the method for identifying thermal system tags based on determining the tag effectiveness state of the tag within the monitoring interval can be replaced by: within the monitoring interval, if a tag effectiveness state value at a moment is greater than that at the previous moment and the tag effectiveness state value at the previous moment is not zero, then the moment is defined as an effectiveness state increase, otherwise it is a non-effective state increase; the average value of each effectiveness state increase is recorded as the effectiveness state phase value, and if the tag effectiveness state at any moment is greater than the effectiveness state phase value, it is determined that it meets the first phase value condition; The label validity state only adopts the corresponding value under the judgment label; An increase-position binary is formed by any effective-state increase and its corresponding label effective-state. The set constructed by the increase-position binary is interpolated by cubic spline using the plrep and splev functions to obtain a smooth curve. The fitting value on the smooth curve obtained at each moment is used as the curve value at the corresponding moment. The difference between the curve value of any non-effective-state increase and the label effective-state is the increase-position estimation value. When the increase-position estimation value is less than zero, the non-effective-state increase is inefficiently estimated, otherwise it is effectively estimated. Search in reverse time direction from any effective state increment to obtain the first non-continuous effective state increment as its reference position, and use the time period between the effective state increment and its reference position as an estimated reference interval; exclude the estimated reference interval containing the zero point; wherein the continuously occurring effective state increment refers to a number of effective state increments that are continuous in time sequence with the effective state increment of the search start position, so each continuous effective state increment belongs to the estimated reference interval formed by the effective state increment that is closest to the current moment in time; The strategic sensitivity value Psenv of the estimated reference interval is calculated according to the incremental estimation: Psenv = Len × lg (1 + eff.rp × eff.dsv); where eff.rp represents the proportion of the moments with valid estimation between the estimated reference interval and the first zero point in the reverse time direction, and eff.dsv is the maximum value of the incremental estimation label effectiveness state of each moment with valid estimation in the estimated reference interval; Len is the proportion of the number of moments in the estimated reference interval to the total number of moments in the monitoring interval, and lg is the logarithmic sign with 10 as the base; When zs.rp cannot obtain the first zero point in the reverse time direction, the farthest moment in the monitoring interval is used as the first zero point obtained by the search; If the number of effective estimates in the estimation reference interval is greater than the number of inefficient estimates, the estimation reference interval is considered to meet the second phase condition; the first phase condition and the second phase condition are both used as phase conditions, and the number of phase conditions that are met at the current moment is recorded as the phase condition achievement value Glv; If any estimation reference interval cannot be included at any moment, the first estimation reference interval searched in the reverse time direction is inherited to correspond to the second phase value condition; the number of inefficient estimations refers to the number of inefficient state increments that occur, and the same is true for the number of effective estimations; the label thermal strategy coefficient Ltsc is calculated based on each estimation reference interval: ; Where e is a natural constant, i1 is the serial number of the estimated reference interval, MPs represents the average value of the strategy sensitivity value corresponding to all estimated reference intervals, noz is the number of estimated reference intervals, avg{} is the average function, and Psenv i1 is the strategy sensitivity value of the i1th estimated reference interval, exp() is an exponential function with the natural constant e as the base, rk() is the ranking coefficient function, and the return value obtained by the ranking coefficient function rk(i1) is: obtain the maximum value of the label efficacy state in the i1th estimated reference interval, and take the percentile value of the obtained maximum value in all label efficacy states in the monitoring interval as the return value of the ranking coefficient function.

[0027] Furthermore, the method of using the label thermal strategy coefficient to identify the thermal label from each server node is: for any data label, define the median value of the label thermal strategy coefficient corresponding to each node as the strategy division variable. If the label thermal strategy coefficient of the node is greater than the strategy division variable, then the data label is considered to be a thermal label at the node, otherwise it is a non-thermal label.

[0028] Furthermore, a method for storing data records in combination with the thermal label determination result is: any data record to be written contains several data labels, and thermal risk exclusion storage is performed for each of the data labels, specifically: each storage node of the current data label is obtained, if the current data label in any storage node is a thermal label, the node is defined as a risk node, and the thermal risk exclusion return value is obtained: when all storage nodes are risk nodes, the return value is considered to be FALSE, and a storage node is reselected for the new current data label, otherwise the return value is TRUE; through thermal risk exclusion storage, the thermal risk exclusion return values ​​of all data labels are TRUE, and the data record to be written is stored.

[0029] The current data tag of a data record has a thermal risk, which means that the storage state of the data tag is prone to accumulate hot node risks. Too many data tags with such hot node risks will affect the load balancing between server nodes. The data storage system of the entire fund assets has the risk of uneven load in terms of content popularity or content criticality.

[0030] The distributed receiving and storing system for fund asset data records provided by the embodiment of the present invention is as follows: Figure 2 Shown is a structural diagram of a distributed receiving and storing system for data records of fund assets of the present invention. The distributed receiving and storing system for data records of fund assets of this embodiment comprises: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned distributed receiving and storing method for data records of fund assets are implemented.

[0031] The system comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to run in the following units of the system: A distributed storage environment arrangement unit is used to identify and obtain each server node and data label in the data storage environment; A real-time monitoring unit, used to obtain the label validity status of the data label from the server node; A thermal label dynamic identification unit is used to calculate the label thermal strategy coefficient according to the label effectiveness state, and use the label thermal strategy coefficient to identify the thermal label from each server node; The new data storage unit is used to record and store data in combination with the thermal tag determination results.

[0032] The distributed receiving and storage system for data records of fund assets can be run in computing devices such as desktop computers, laptop computers, PDAs, and cloud servers. The distributed receiving and storage system for data records of fund assets can include, but is not limited to, processors and memories. Those skilled in the art can understand that the examples are only examples of distributed receiving and storage systems for data records of fund assets, and do not constitute a limitation on the distributed receiving and storage system for data records of fund assets. The system can include more or fewer components than the examples, or combine certain components, or different components. For example, the distributed receiving and storage system for data records of fund assets can also include input and output devices, network access devices, buses, etc.

[0033] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the data record distributed receiving storage system operating system of the fund assets, and uses various interfaces and lines to connect various parts of the data record distributed receiving storage system operating system of the entire fund assets.

[0034] The memory can be used to store the computer program and / or module, and the processor realizes various functions of the data record distributed receiving storage system of the fund assets by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0035] Although the description of the present invention has been quite detailed and has been described in particular with respect to several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, so as to effectively cover the intended scope of the present invention. In addition, the present invention is described above with the embodiments foreseeable by the inventors, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that are not currently foreseen may still represent equivalent changes of the present invention.

Claims

1. A distributed receiving and storing method for fund asset data records, characterized in that: The method comprises the following steps: configuring server nodes and data tags in a data storage environment; obtaining the tag effectiveness state of the data tags from the server nodes; calculating the tag thermal strategy coefficient according to the tag effectiveness state, and using the tag thermal strategy coefficient to identify the thermal system tags from each server node; and recording and storing data in combination with the thermal system tag determination result; The method for calculating the label thermal strategy coefficient according to the label effective state is: within a preset time period, horizontally compare the label effective state time series to obtain the first effective state quasi-value, horizontally compare the label effective state nodes to obtain the second effective state quasi-value, calculate the quasi-node difference dimension through the first effective state quasi-value, identify the effective state valley point according to the first effective state quasi-value and the second effective state quasi-value, obtain the effective state derivative according to the occurrence frequency of the effective state valley point, and calculate the label thermal strategy coefficient using the quasi-node difference dimension, the node difference dimension and the effective state derivative.

2. The distributed receiving and storing method for fund asset data records according to claim 1, characterized in that: The method for identifying and obtaining each server node and data label in a data storage environment is as follows: the data storage environment includes a plurality of server nodes, referred to as nodes; The data records of fund assets are sharded and stored in each node through the data sharding mechanism. The data records are composed of several data tags. The several data tags of the data records are stored in different nodes respectively. Each node stores the fund asset data of each data tag.

3. The distributed receiving and storing method for fund asset data records according to claim 1, characterized in that: The method for obtaining the label validity state of the data label from the server node is: preset a monitoring interval rg, rg∈[0.5, 24] hours, and obtain the label validity state of the data label every rg, wherein the label validity state of the data label is the product of the operation order value and the label pressure, and the calculation method of the operation order value is: the average value of the CPU occupancy rate of the server node within the monitoring interval is recorded as the occupancy, and the minimum value of the occupancy of all nodes is recorded as the occupancy base value, then the occupancy of any node is subtracted from the occupancy base value, and the ratio of the difference to the occupancy base value is used as its operation order value; the calculation method of the label pressure is: in a node, the number of read and write times of the same data label is taken as the unit read and write amount, and the value set consisting of the corresponding read and write amounts of each data label under the node is minmax normalized, and the normalized value of the unit read and write amount of the data label is defined as its label pressure.

4. The distributed receiving and storing method for fund asset data records according to claim 1, characterized in that: The method for calculating the label thermal strategy coefficient according to the label effectiveness state is: taking any data label as the judgment label and taking a preset time interval as the monitoring interval; and identifying the thermal system label according to the label effectiveness state of the judgment label within the monitoring interval.

5. The distributed receiving and storing method for fund asset data records according to claim 4 is characterized in that: The method of identifying the thermal system label according to the label validity state of the label determined in the monitoring interval is: respectively obtain the minimum value of the label validity state at different times of the same node and at different nodes at the same time, and record them as the first validity state quasi-value and the second validity state quasi-value respectively; define the difference between the label validity state and the first validity state quasi-value as the quasi-node difference dimension SLbv; for any node, if the first validity state quasi-value and the second validity state quasi-value corresponding to any non-zero label validity state are both zero, then the moment corresponding to the label validity state is recorded as the validity state valley point; construct each validity state valley point at all times into a set Fn, the number of elements in the set Fn is recorded as Sfn, and the difference between the label validity state of any validity state valley point and the minimum value of the label validity state in the set Fn is recorded as the node difference dimension of the validity state valley point; The label effectiveness at any moment is taken as the valley threshold. The ratio of the number of elements whose label effectiveness is less than the valley threshold to the number of elements whose label effectiveness is greater than the valley threshold in the set Fn is recorded as the effectiveness variability Eff at that moment. If the label effectiveness of all elements in the set Fn is greater than the valley threshold, or less than the valley threshold, then the effectiveness variability at that moment is 1 / Sfn. The label thermal strategy coefficient Ltsc calculated according to the effectiveness valley is calculated as follows: ； Where j1 is the accumulated variable, SLbv j1 and Lbv j1 They are the quasi-nodal difference direction and nodal difference direction of the j1th effective valley point respectively.

6. The distributed receiving and storing method for fund asset data records according to claim 4, characterized in that: The method of identifying thermal system tags according to the tag effectiveness state of the tag determined within the monitoring interval can be replaced as follows: if the tag effectiveness state value at a moment is greater than that at the previous moment and the tag effectiveness state value at the previous moment is not zero, then the moment is defined as an effectiveness state increase; the average value of each effectiveness state increase is recorded as the effectiveness state phase value, and if the tag effectiveness state at any moment is greater than the effectiveness state phase value, then it is determined that it meets the first phase value condition; An increase-position binary is formed by any effective-state increase position and its corresponding label effective state. The set constructed by the increase-position binary is interpolated by cubic spline using the plrep and splev functions to obtain a smooth curve. The difference between the curve value of any ineffective-state increase position and the label effective state is the increase-position estimation value. When the increase-position estimation value is less than zero, the ineffective-state increase position is inefficiently estimated, otherwise it is effectively estimated. From any effective state increase, search in reverse time direction to obtain the first non-continuous effective state increase as its reference position, and use the time period between the effective state increase and its reference position as an estimated reference interval; eliminate the estimated reference interval containing the zero point; calculate the strategy sensitivity value of the estimated reference interval based on the increase estimate; if the number of valid estimates in the estimated reference interval is greater than the number of inefficient estimates, the estimated reference interval is considered to meet the second phase condition; both the first phase condition and the second phase condition are used as phase conditions, and the number of phase conditions that meet the current moment is recorded as the phase condition achievement value; calculate the label thermal strategy coefficient based on the phase condition achievement value and each estimated reference interval.

7. The distributed receiving and storing method for fund asset data records according to claim 1, characterized in that: The method of using the label thermal strategy coefficient to identify the hot label from each server node is: for any data label, define the median value of the label thermal strategy coefficient corresponding to each node as the strategy partition variable. If the label thermal strategy coefficient of the node is greater than the strategy partition variable, then the data label is considered to be a hot label at the node, otherwise it is a non-hot label.

8. The distributed receiving and storing method for fund asset data records according to claim 1, characterized in that: The method for storing data records in combination with the result of thermal label determination is: any data record to be written contains several data labels, and thermal risk exclusion storage is performed for each data label, specifically: each storage node of the current data label is obtained, if the current data label in any storage node is a thermal label, the node is defined as a risk node, and the thermal risk exclusion return value is obtained: when all storage nodes are risk nodes, the return value is considered to be FALSE, and the storage node is reselected for the new current data label, otherwise the return value is TRUE; By using thermal risk exclusion storage, the thermal risk exclusion return values ​​of all data tags are TRUE, and the data records to be written are stored.

9. The distributed receiving and storage system for fund asset data records is characterized by: The distributed receiving and storage system for data records of the fund assets comprises: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the distributed receiving and storage method for data records of the fund assets described in any one of claims 1 to 8 are implemented. The distributed receiving and storage system for data records of the fund assets runs on computing devices such as desktop computers, laptop computers, PDAs, and cloud data centers.

Images