Risk measurement method and system based on duckdb database memory calculation

Abstract

The present invention relates to the field of data processing technology, and in particular to a risk measurement method and system based on duckdb database memory calculation. The method comprises: obtaining duckdb database data and code data for historical risk calculation, using the database data and code data to obtain basic data, frequent item sets and expressions; obtaining set importance according to the number, support and usage of basic data in the frequent item sets where the basic data in the frequent item sets are located in the code data; obtaining important factors according to the usage frequency, support and set importance of each expression in each frequent item set; combining the usage and important factors of the calculation result values ​​obtained from the basic data in each frequent item set of each expression in the code data, screening common sub-expressions and their corresponding frequent item sets, and then establishing indexes and storing them in the database. The present invention improves the efficiency of risk measurement.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a risk measurement method and system based on duckdb database memory calculation. Background Art

[0002] Risk measurement is the process of quantitatively evaluating the probability of a potential risk and the losses it may cause. In fields such as finance, investment, and project management, risk measurement is essential for making the right decisions. When building a risk calculation model and performing risk measurement calculations, a large amount of historical data queries may be involved, and a large amount of data and complex calculations need to be processed. Risk managers need to frequently query historical data in the database to assess the impact and probability of a specific risk. These query operations are essential for retrieving and analyzing data. For example, when it comes to calculating the volatility of multiple assets, the same basic data may be used multiple times, which will result in a large amount of repeated calculations, which in turn leads to low efficiency in risk measurement. Summary of the invention

[0003] In order to solve the problem of low efficiency of existing methods in risk measurement, the purpose of the present invention is to provide a risk measurement method and system based on duckdb database memory calculation. The technical solution adopted is as follows:

[0004] In a first aspect, the present invention provides a risk measurement method based on duckdb database memory calculation, the method comprising the following steps:

[0005] Obtain duckdb database data and code data for historical risk calculation, use the duckdb database data to obtain basic data and frequent item sets, and use the code data to obtain expressions;

[0006] According to the number, support and usage of basic data in the frequent itemset in each frequent itemset, the set importance of each frequent itemset is obtained; according to the usage frequency, support and set importance of each expression in each frequent itemset, the importance factor of each expression in each frequent itemset is obtained;

[0007] Combine the usage of the calculation result value obtained from the basic data of each expression in each frequent item set in the code data and the important factors to screen the common sub-expressions and their corresponding frequent item sets;

[0008] An index is created based on the common sub-expressions and their corresponding frequent item sets and stored in a database.

[0009] Preferably, obtaining the set importance of each frequent itemset according to the number, support and usage of basic data in the frequent itemset where the basic data in each frequent itemset is located in the code data includes:

[0010] According to the support of each frequent item set where the basic data to be analyzed is located, the usage frequency of the basic data to be analyzed in the code data and the total number of basic data in all frequent item sets where the basic data to be analyzed is located, an important index of the basic data to be analyzed is obtained, wherein the support and the total number are both positively correlated with the important index, and the usage frequency is negatively correlated with the important index; the basic data to be analyzed is any basic data;

[0011] Obtaining the importance of the basic data to be analyzed according to the usage frequency of the basic data to be analyzed in the code data and the important indicators;

[0012] The overall distribution of the importance of the use of all basic data in the candidate frequent itemsets is comprehensively considered to determine the set importance of the candidate frequent itemsets, where the candidate frequent itemsets are any frequent itemsets.

[0013] Preferably, obtaining the importance of the use of the basic data to be analyzed according to the use frequency of the basic data to be analyzed in the code data and the important index includes:

[0014] Calculate the normalized value of the usage frequency of the basic data to be analyzed in the code data;

[0015] The usage importance of the basic data to be analyzed is obtained according to the normalized value of the usage frequency and the normalized value of the important index of the basic data to be analyzed, and both the normalized value of the usage frequency and the normalized value of the important index are positively correlated with the usage importance.

[0016] Preferably, the step of determining the set importance of the candidate frequent itemsets based on the overall distribution of the usage importance of all basic data in the candidate frequent itemsets comprises: taking the sum of the usage importance of all basic data in the candidate frequent itemsets as the set importance of the candidate frequent itemsets.

[0017] Preferably, obtaining the importance factor of each expression in each frequent itemset according to the usage frequency, support and importance of the set of each expression in each frequent itemset includes:

[0018] Calculate the ratio between the number of times the expression to be evaluated is used in the candidate frequent item set and the support of the candidate frequent item set;

[0019] According to the ratio and the set importance of the candidate frequent item sets, an importance factor of the expression to be evaluated in the candidate frequent item sets is obtained; the expression to be evaluated is any expression.

[0020] Preferably, obtaining the importance factor of the expression to be evaluated in the candidate frequent itemset according to the ratio and the set importance of the candidate frequent itemset includes:

[0021] The normalized result of the product of the ratio and the set importance of the candidate frequent itemsets is used as the importance factor of the expression to be evaluated in the candidate frequent itemsets.

[0022] Preferably, the method of combining the usage of the calculation result value obtained from the basic data of each expression in each frequent item set in the code data and the important factors to screen the common sub-expressions and their corresponding frequent item sets includes:

[0023] According to the important factors of the expression to be evaluated in the candidate frequent item set and the number of times the calculation result value obtained from the basic data of the expression to be evaluated in the candidate frequent item set is used in the code data, the reference feature value when the expression to be evaluated uses the basic data in the candidate frequent item set is obtained;

[0024] If the reference feature value is greater than a preset reference threshold, the expression to be evaluated is determined to be a common sub-expression, and the candidate frequent item set is the frequent item set corresponding to the expression to be evaluated.

[0025] Preferably, the step of obtaining a reference feature value when the expression to be evaluated uses the basic data in the candidate frequent itemset according to the important factors of the expression to be evaluated in the candidate frequent itemset and the number of times the calculation result value obtained from the basic data of the expression to be evaluated in the candidate frequent itemset is used in the code data comprises:

[0026] The sum of the constant 1 and the important factors of the expression to be evaluated in the candidate frequent item set is recorded as the first eigenvalue;

[0027] The product of the first feature value and the number of times the calculation result value obtained from the basic data of the candidate frequent item set of the expression to be evaluated is used in the code data is determined as the reference feature value when the expression to be evaluated uses the basic data in the candidate frequent item set.

[0028] Preferably, the step of obtaining basic data and frequent itemsets by using the duckdb database data includes:

[0029] Using the risk data in the duckdb database data as basic data, determining a basic data vector set;

[0030] According to the basic data vector set, the FP-tree algorithm is used to determine the frequent item sets where the basic data are located.

[0031] In a second aspect, the present invention provides a risk measurement system based on duckdb database memory calculation, the system comprising:

[0032] A data collection module is used to obtain duckdb database data and code data for historical risk calculation, obtain basic data and frequent item sets using the duckdb database data, and obtain expressions using the code data;

[0033] The important factor calculation module is used to obtain the set importance of each frequent item set according to the number, support and usage of basic data in the frequent item set where the basic data in each frequent item set is located in the code data; and obtain the important factor of each expression in each frequent item set according to the usage frequency, support and set importance of each expression in each frequent item set;

[0034] A screening module, for screening common sub-expressions and their corresponding frequent itemsets, based on the usage of the calculation result values ​​obtained from the basic data of each expression in each frequent itemset in the code data and the important factors;

[0035] The index building module is used to build an index based on the common sub-expressions and their corresponding frequent item sets and store them in a database.

[0036] The present invention has at least the following beneficial effects:

[0037] The present invention takes into account that when measuring risks, due to the existence of a large amount of data that needs to be calculated multiple times, the amount of calculation is large, resulting in low efficiency of risk measurement. The present invention first obtains basic data, frequent item sets and expressions based on duckdb database data and code data used for historical risk calculation, and then obtains important factors of expressions in frequent item sets according to the number, support and usage of basic data in the frequent item sets where the basic data in each frequent item set is located in the code data, evaluates the importance of the data, and combines the important factors and the usage of the calculation result values ​​obtained from the basic data in each frequent item set of each expression in the code data, screens common sub-expressions and their corresponding frequent item sets, further establishes indexes based on the screened common sub-expressions and their corresponding frequent item sets and stores them in the database, determines the data that needs to be stored, reduces the query operation of the database, reduces the usage of the database memory, and improves the efficiency of risk measurement. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A flow chart of a risk measurement method based on duckdb database memory calculation provided by an embodiment of the present invention;

[0040] Figure 2 This is a structural block diagram of a risk measurement system based on duckdb database memory calculation provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the risk measurement method and system based on duckdb database memory calculation proposed by the present invention are described in detail as follows in combination with the accompanying drawings and preferred embodiments.

[0042] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0043] The specific scheme of the risk measurement method and system based on duckdb database memory calculation provided by the present invention is described in detail below with reference to the accompanying drawings.

[0044] Example of risk measurement method based on duckdb database memory calculation:

[0045] The specific scenario targeted by this embodiment is: when performing risk measurement, a large number of queries are required to complete the data calculation. When a large number of data are queried, the same basic data is stored and used, resulting in a large number of repeated calculations. When the CSE technology in the duckdb database is used to optimize the data in the database, the call status of the data determined by the basic data in the database is not taken into account. Therefore, it is necessary to determine the index status of the data determined by the current CSE according to the frequent item sets of the basic data in the database and the call status of the expression data, so as to reduce the query operation of the database and reduce the memory usage.

[0046] This embodiment proposes a risk measurement method based on duckdb database memory calculation, such as Figure 1 As shown, the risk measurement method based on duckdb database memory calculation in this embodiment includes the following steps:

[0047] Step S1, obtaining duckdb database data and code data for historical risk calculation, using the duckdb database data to obtain basic data and frequent item sets, and using the code data to obtain expressions.

[0048] First, we obtain duckdb database data and code data used for historical risk calculation. Then, we obtain database risk data in the code data using SQL queries or database-specific API calls, and use this risk data as basic data to annotate these basic data.

[0049] Since it is necessary to reduce the query on basic data in the database, the data that needs to be stored for the current common sub-expression is determined. There may be multiple basic data calculated by the same expression, and the calculation result obtained by the expression may be used multiple times. Therefore, the frequent item set of the current basic data can be determined according to the frequency of occurrence of the basic data, and the common sub-expression is further identified according to the expression in the frequent item set to determine the data that needs to be stored.

[0050] After obtaining the basic data, determine the expressions and functions used by these basic data, and mark these expressions or functions for statistical purposes. It should be noted that in programming and computer science, functions are also a type of expression, so functions and expressions are collectively referred to as expressions. After determining the expression that uses the basic data, obtain the basic data used in an expression and store it in a vector, for example: correlation_AB=corr(returns_A, returns_B), where corr is an expression, returns_A and returns_B represent basic data, and the basic data [returns_A, returns_B] in the expression is stored in the vector.

[0051] The basic data vectors of all expressions in the code data that have used the basic data are stored in the basic data vector set. The FP-tree algorithm is used to determine the frequent item sets of the basic data vector set. This embodiment sets the minimum support k of the FP-tree to 5. In specific applications, the implementer can set it according to the specific situation. The frequent item sets with support greater than the minimum support are determined as the frequent item sets where the basic data appears, that is, the frequent item sets where the basic data is located. The FP-tree algorithm is a prior art and will not be described in detail here.

[0052] So far, basic data, frequent itemsets and expressions have been obtained.

[0053] Step S2, obtaining the set importance of each frequent item set according to the number, support and usage of the basic data in the frequent item set where the basic data in each frequent item set is located in the code data; obtaining the importance factor of each expression in each frequent item set according to the usage frequency, support and the set importance of each expression in each frequent item set.

[0054] Generally speaking, if the data in the database is frequently queried, it means that the data is more important; if the basic data is not used synchronously with other data each time it is used, it means that the current basic data needs to be extracted from the database separately, and the interaction with other basic data is relatively small, and its importance will be relatively reduced.

[0055] In a frequent item set, the more basic data there are, the more cases in which the current basic data appear at the same time and the more cases involved in the calculation; at the same time, the more times the frequent item set appears, the more important the basic data is.

[0056] Based on this, according to the support of the basic data to be analyzed in each frequent item set, the usage frequency of the basic data to be analyzed in the code data and the total number of basic data in all frequent item sets where the basic data to be analyzed is located, the important indicators of the basic data to be analyzed are obtained, the support and the total number are both positively correlated with the important indicators, and the usage frequency is negatively correlated with the important indicators; the basic data to be analyzed is any basic data.

[0057] Among them, a positive correlation means that the dependent variable will increase as the independent variable increases, and the dependent variable will decrease as the independent variable decreases. It can be an additive relationship, a multiplicative relationship, etc., which is determined by actual application; a negative correlation means that the dependent variable will decrease as the independent variable increases, and the dependent variable will increase as the independent variable decreases. It can be a subtractive relationship, a division relationship, etc., which is determined by actual application.

[0058] In this embodiment, a specific calculation formula of the important index is given, and the important index of the vth basic data can be expressed as:

[0059]

[0060] in, represents the important index of the vth basic data, Indicates the number of frequent item sets where the vth basic data is located, represents the support of the vth basic data in the ith frequent item set, Indicates the usage frequency of the vth basic data in the code data, It represents the number of basic data of the vth basic data in the ith frequent item set.

[0061] Represents the total number of basic data in all frequent item sets where the vth basic data is located. It is used to reflect the proportion of the total usage frequency of the vth basic data in the frequent item set to the current overall usage frequency; when the total number of basic data in all frequent item sets where the vth basic data is located is larger and the total usage frequency of the vth basic data in the frequent item set accounts for the current overall usage frequency, the importance index of the vth basic data is larger.

[0062] By adopting the above method, important indicators of each basic data can be obtained.

[0063] Next, this embodiment is described by taking a frequent item set as an example, and other frequent item sets can be processed by the method provided in this embodiment.

[0064] Any frequent item set is recorded as a candidate frequent item set, and the usage importance of the basic data to be analyzed is obtained according to the usage frequency of the basic data to be analyzed in the code data and the important indicators; the overall distribution of the usage importance of all basic data in the candidate frequent item set is comprehensively considered to determine the set importance of the candidate frequent item set.

[0065] Specifically, the normalized value of the usage frequency of the basic data to be analyzed in the code data is calculated; the usage importance of the basic data to be analyzed is obtained according to the normalized value of the usage frequency and the normalized value of the important index of the basic data to be analyzed, and the normalized value of the usage frequency and the normalized value of the important index are both positively correlated with the usage importance.

[0066] Among them, the positive correlation means that the dependent variable will increase as the independent variable increases, and the dependent variable will decrease as the independent variable decreases. It can be an additive relationship, a multiplicative relationship, etc., which is determined by practical applications.

[0067] In this embodiment, a specific calculation formula for the use importance is given, and the use importance of the vth basic data can be expressed as:

[0068]

[0069] in, Indicates the importance of the use of the vth basic data, Indicates the usage frequency of the vth basic data in the code data, Indicates the maximum value of the usage frequency of all basic data in the code data. represents the important index of the vth basic data, Represents the normalization function.

[0070] It means that the usage frequency of the vth basic data in the code data is normalized to reflect the usage importance of the vth basic data. When the normalized result of the usage frequency of the vth basic data in the code data is larger and the important index of the vth basic data is also larger, it means that the usage importance of the vth basic data is greater, that is, the usage importance of the vth basic data is greater.

[0071] By adopting the above method, the usage importance of each basic data in the candidate frequent item set can be obtained, and the sum of the usage importance of all basic data in the candidate frequent item set is taken as the set importance of the candidate frequent item set.

[0072] By using the above method, the set importance of each frequent item set can be obtained.

[0073] Since the same basic data is used in different code segments when performing risk calculations, this will lead to a large number of repeated calculations. If the calculation process and the same basic data are used multiple times, it means that the calculation result determined by the corresponding expression is more important. If the calculation result is also frequently used in other functions or codes, the calculation result will be directly stored in the database, which can greatly reduce the operations and calculations on the basic data in the database during risk calculations.

[0074] The same basic data can be called multiple times by the same expression or used by different expressions, so the number of times it is used needs to be determined separately. For example, in the frequent item set {A, B}, the support is 6. Among them, there is an expression fun1 that uses basic data A, B, C 4 times, indicating that the number of times fun1 expression is used in the frequent item set is 4 times; there is an expression fun2 that uses basic data A and B 2 times, indicating that the number of times fun1 expression is used is 4 times.

[0075] If an expression uses the same basic data as a parameter for calculation multiple times in the same calculation process, it means that multiple repeated calculations have occurred in the calculation process. The more times the expression is used in all the parameters in this frequent item set, the more important the expression is in the risk calculation process. Therefore, the important factor of each expression in each frequent item set is determined based on the frequency of use, support, and set importance of each expression in each frequent item set.

[0076] In this embodiment, an expression is taken as an example for description, and other expressions can be processed by the method provided in this embodiment.

[0077] Specifically, any expression is recorded as an expression to be evaluated, and the ratio between the number of times the expression to be evaluated is used in a candidate frequent item set and the support of the candidate frequent item set is calculated; the importance factor of the expression to be evaluated in the candidate frequent item set is obtained according to the ratio and the set importance of the candidate frequent item set; and the normalized result of the product of the ratio and the set importance of the candidate frequent item set is used as the importance factor of the expression to be evaluated in the candidate frequent item set.

[0078] In this embodiment, a specific calculation formula of the important factor is given, and the important factor of the vth basic data in the uth frequent item set can be expressed as:

[0079]

[0080] in, represents the important factor of the sth expression in the uth frequent itemset, represents the set importance of the u-th frequent itemset, represents the number of times the sth expression is used in the uth frequent itemset, represents the support of the u-th frequent itemset, Represents the normalization function.

[0081] It is used to reflect the importance of the sth expression in the uth frequent itemset. The larger the ratio, the more important the sth expression is, and the more necessary it is to store the calculation result of the expression in the database. When the set importance of the uth frequent itemset is greater and the ratio of the number of times the sth expression is used in the uth frequent itemset to the support of the uth frequent itemset is greater, it means that the sth expression is more important in the uth frequent itemset, that is, the importance factor of the sth expression in the uth frequent itemset is greater.

[0082] Using the above method, the important factor of each expression in each frequent item set can be obtained.

[0083] Step S3, combining the usage of the calculation result value obtained from the basic data of each expression in each frequent item set in the code data and the important factors, screening common sub-expressions and their corresponding frequent item sets.

[0084] After determining the important factors of each expression in each frequent item set, obtain the number of times the calculation result value obtained from the basic data of each expression in each frequent item set is used in the code data, for example: result = fun1 (A, B), where result is the calculation result value of the expression fun1. After determining the calculation result value, obtain the number of times the calculation result is used in the entire code segment, and then combine the number of times it is used and the important factor to update the corresponding importance of the expression, that is, obtain the corresponding reference feature value.

[0085] Next, the expression to be evaluated and the candidate frequent item set are still used as an example for description. The method provided in this embodiment can be used to process other expressions or other frequent item sets.

[0086] Specifically, according to the important factor of the expression to be evaluated in the candidate frequent item set and the number of times the calculated result value obtained from the basic data of the expression to be evaluated in the candidate frequent item set is used in the code data, a reference eigenvalue when the expression to be evaluated uses the basic data in the candidate frequent item set is obtained; specifically, the sum of the constant 1 and the important factor of the expression to be evaluated in the candidate frequent item set is recorded as the first eigenvalue; the product of the first eigenvalue and the number of times the calculated result value obtained from the basic data of the expression to be evaluated in the candidate frequent item set is used in the code data is determined as the reference eigenvalue when the expression to be evaluated uses the basic data in the candidate frequent item set.

[0087] In this embodiment, a specific calculation formula for the reference eigenvalue is given. The reference eigenvalue of the s-th expression using the basic data in the u-th frequent item set can be expressed as:

[0088]

[0089] in, It represents the reference eigenvalue when the s-th expression uses the basic data in the u-th frequent itemset. Indicates the number of times the calculation result value of the s-th expression obtained from the basic data in the u-th frequent item set is used in the code data, represents the important factor of the sth expression in the uth frequent itemset, represents the first eigenvalue.

[0090] When the calculation result value obtained by the s-th expression from the basic data in the u-th frequent item set is used more times in the code data and the importance factor of the s-th expression in the u-th frequent item set is larger, it means that the s-th expression is more important to the u-th frequent item set, that is, the reference feature value of the s-th expression when using the basic data in the u-th frequent item set is larger.

[0091] By adopting the above method, the reference feature value of each expression using the basic data in each frequent item set can be obtained. The larger the reference feature value, the more important the expression is. Therefore, if the reference feature value is greater than the preset reference threshold, the corresponding expression is determined to be a common sub-expression, and the corresponding frequent item set is determined to be a frequent item set corresponding to the common sub-expression. In this embodiment, the preset reference threshold is 0.7. In specific applications, the implementer can set it according to specific circumstances.

[0092] So far, the above method can be used to screen out multiple common sub-expressions and their corresponding frequent item sets.

[0093] Step S4: create an index based on the common sub-expressions and their corresponding frequent item sets and store it in a database.

[0094] After filtering out the common sub-expressions and their corresponding frequent item sets, the data calculated by the frequently accessed common sub-expressions need to be stored in the database for quick query access from the database during data interaction. The specific steps are as follows:

[0095] 1. Obtain the basic data in the frequent itemsets corresponding to the common sub-expressions, use the common sub-expressions to perform calculations, and determine the calculation results;

[0096] 2. Sort the calculation results according to the usage frequency of the frequent itemsets required by the common sub-expressions;

[0097] 3. Create a new table in the current database and insert the calculation results into the new table;

[0098] 4. Create indexes for the sorted data to ensure that each data point can be accessed quickly.

[0099] So far, the risk measurement of duckdb database memory calculation has been completed by using the method provided in this embodiment.

[0100] This embodiment takes into account that when measuring risks, due to the large amount of data that needs to be calculated multiple times, the amount of calculation is large, resulting in low efficiency of risk measurement. This embodiment first obtains basic data, frequent item sets and expressions based on duckdb database data and code data used for historical risk calculation, and then obtains important factors of expressions in frequent item sets according to the number, support and usage of basic data in the frequent item sets where the basic data in each frequent item set is located in the code data, evaluates the importance of the data, and screens common sub-expressions and their corresponding frequent item sets in combination with the important factors and the usage of the calculation result values ​​obtained from the basic data in each frequent item set of each expression in the code data. Further, based on the screened common sub-expressions and their corresponding frequent item sets, indexes are established and stored in the database, the data that needs to be stored is determined, the query operations of the database are reduced, the usage of the database memory is reduced, and the efficiency of risk measurement is improved.

[0101] Example of risk measurement system based on duckdb database memory calculation:

[0102] See also Figure 2 , which shows a structural block diagram of a risk measurement system based on duckdb database memory calculation provided by an embodiment of the present invention. The system may include a data collection module, an important factor calculation module, a screening module and an index establishment module.

[0103] The data acquisition module is used to obtain duckdb database data and code data for historical risk calculation, obtain basic data and frequent item sets using the duckdb database data, and obtain expressions using the code data;

[0104] The important factor calculation module is used to obtain the set importance of each frequent item set according to the number, support and usage of basic data in the frequent item set where the basic data in each frequent item set is located in the code data; and obtain the important factor of each expression in each frequent item set according to the usage frequency, support and set importance of each expression in each frequent item set;

[0105] A screening module, for screening common sub-expressions and their corresponding frequent itemsets, based on the usage of the calculation result values ​​obtained from the basic data of each expression in each frequent itemset in the code data and the important factors;

[0106] The index building module is used to build an index based on the common sub-expressions and their corresponding frequent item sets and store them in a database.

[0107] It should be understood that Figure 2The structural block diagram of the risk measurement system based on duckdb database memory calculation and its modules shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in the memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art can understand that the above methods and systems can be implemented using computer executable instructions and / or included in processor control codes, such as carrier media such as disks, CDs or DVD-ROMs, programmable memories such as read-only memories (firmware), or data carriers such as optical or electronic signal carriers. Such codes are provided on. The system and its modules of this specification can not only be implemented by hardware circuits such as ultra-large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above hardware circuits and software (for example, firmware).

[0108] For more details about the above modules, please refer to other places in this manual and will not be repeated here.

[0109] In other embodiments, a risk measurement device based on duckdb database memory calculation is also provided, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, so that the device executes the above-mentioned risk measurement method based on duckdb database memory calculation. The device can be specifically a chip, a component or a module, and the chip may include a connected processor and a memory; wherein the memory is used to store instructions, and when the processor calls and executes the instructions, the chip can execute the risk measurement method based on duckdb database memory calculation provided in the above embodiment.

[0110] In other embodiments, a computer program product is also provided. When the computer program product is run on a computer, the computer is enabled to execute the above-mentioned related steps to implement the risk measurement method based on duckdb database memory calculation provided in the above embodiment.

[0111] In other embodiments, a computer-readable storage medium is also provided, in which a computer program code is stored. When the computer program code is executed on a computer, the computer executes the above-mentioned related method steps to implement the risk measurement method based on duckdb database memory calculation provided in the above embodiment.

[0112] Among them, the provided system, electronic device, computer program product, and computer-readable storage medium are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0113] It should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A risk measurement method based on duckdb database memory calculation, characterized in that: The method comprises the following steps: Obtain duckdb database data and code data for historical risk calculation, use the duckdb database data to obtain basic data and frequent item sets, and use the code data to obtain expressions; According to the number, support and usage of basic data in the frequent itemset in each frequent itemset, the set importance of each frequent itemset is obtained; according to the usage frequency, support and set importance of each expression in each frequent itemset, the importance factor of each expression in each frequent itemset is obtained; Combine the usage of the calculation result value obtained from the basic data of each expression in each frequent item set in the code data and the important factors to screen the common sub-expressions and their corresponding frequent item sets; Establishing an index based on the common sub-expression and its corresponding frequent item set and storing it in a database; The step of obtaining the set importance of each frequent itemset according to the number, support and usage of the basic data in the frequent itemset where the basic data in each frequent itemset is located in the code data includes: According to the support of each frequent item set where the basic data to be analyzed is located, the usage frequency of the basic data to be analyzed in the code data and the total number of basic data in all frequent item sets where the basic data to be analyzed is located, an important index of the basic data to be analyzed is obtained, wherein the support and the total number are both positively correlated with the important index, and the usage frequency is negatively correlated with the important index; the basic data to be analyzed is any basic data; Obtaining the importance of the basic data to be analyzed according to the usage frequency of the basic data to be analyzed in the code data and the important indicators; Determine the set importance of the candidate frequent itemset by comprehensively analyzing the overall distribution of the importance of the use of all basic data in the candidate frequent itemset, wherein the candidate frequent itemset is any frequent itemset; The important factor of each expression in each frequent item set is obtained according to the usage frequency, support and importance of each expression in each frequent item set, including: Calculate the ratio between the number of times the expression to be evaluated is used in the candidate frequent item set and the support of the candidate frequent item set; According to the ratio and the set importance of the candidate frequent item set, an important factor of the expression to be evaluated in the candidate frequent item set is obtained; the expression to be evaluated is any expression; The method of combining the usage of the calculation result value obtained from the basic data of each expression in each frequent item set in the code data and the important factors to screen the common sub-expressions and their corresponding frequent item sets includes: According to the important factors of the expression to be evaluated in the candidate frequent item set and the number of times the calculation result value obtained from the basic data of the expression to be evaluated in the candidate frequent item set is used in the code data, the reference feature value when the expression to be evaluated uses the basic data in the candidate frequent item set is obtained; If the reference feature value is greater than a preset reference threshold, the expression to be evaluated is determined to be a common sub-expression, and the candidate frequent item set is the frequent item set corresponding to the expression to be evaluated.

2. The risk measurement method based on duckdb database memory calculation according to claim 1 is characterized in that: The obtaining of the importance of the basic data to be analyzed based on the usage frequency of the basic data to be analyzed in the code data and the important index includes: Calculate the normalized value of the usage frequency of the basic data to be analyzed in the code data; The usage importance of the basic data to be analyzed is obtained according to the normalized value of the usage frequency and the normalized value of the important index of the basic data to be analyzed, and both the normalized value of the usage frequency and the normalized value of the important index are positively correlated with the usage importance.

3. The risk measurement method based on duckdb database memory calculation according to claim 1 is characterized in that: The step of determining the set importance of the candidate frequent itemsets by synthesizing the overall distribution of the usage importance of all basic data in the candidate frequent itemsets includes: taking the sum of the usage importance of all basic data in the candidate frequent itemsets as the set importance of the candidate frequent itemsets.

4. The risk measurement method based on duckdb database memory calculation according to claim 1 is characterized in that: The step of obtaining the importance factor of the expression to be evaluated in the candidate frequent item set according to the ratio and the set importance of the candidate frequent item set includes: A normalized result of the product of the ratio and the set importance of the candidate frequent itemsets is used as an importance factor of the expression to be evaluated in the candidate frequent itemsets.

5. The risk measurement method based on duckdb database memory calculation according to claim 1 is characterized in that: The step of obtaining a reference feature value when the expression to be evaluated uses the basic data in the candidate frequent itemset according to the important factors of the expression to be evaluated in the candidate frequent itemset and the number of times the calculation result value obtained from the basic data of the expression to be evaluated in the candidate frequent itemset is used in the code data comprises: The sum of the constant 1 and the important factors of the expression to be evaluated in the candidate frequent item set is recorded as the first eigenvalue; The product of the first feature value and the number of times the calculation result value obtained from the basic data of the candidate frequent item set of the expression to be evaluated is used in the code data is determined as the reference feature value when the expression to be evaluated uses the basic data in the candidate frequent item set.

6. The risk measurement method based on duckdb database memory calculation according to claim 1 is characterized in that: The method of obtaining basic data and frequent itemsets by using the duckdb database data includes: Using the risk data in the duckdb database data as basic data, determining a basic data vector set; According to the basic data vector set, the FP-tree algorithm is used to determine the frequent item sets where the basic data are located.

7. A risk measurement system based on duckdb database memory calculation, characterized in that: For implementing the method described in claim 1, the system comprises: A data collection module is used to obtain duckdb database data and code data for historical risk calculation, obtain basic data and frequent item sets using the duckdb database data, and obtain expressions using the code data; The important factor calculation module is used to obtain the set importance of each frequent item set according to the number, support and usage of basic data in the frequent item set where the basic data in each frequent item set is located in the code data; and obtain the important factor of each expression in each frequent item set according to the usage frequency, support and set importance of each expression in each frequent item set; A screening module, for screening common sub-expressions and their corresponding frequent itemsets, based on the usage of the calculation result values ​​obtained from the basic data of each expression in each frequent itemset in the code data and the important factors; The index building module is used to build an index based on the common sub-expressions and their corresponding frequent item sets and store them in a database.

Images