Index similarity judgment method, system, terminal device and computer storage medium
By judging and disassembling the similarity between the new indicators and the existing indicators, we automatically identify and avoid duplicate indicators, and solve the problem of the lack of similarity judgment of the new indicators in the indicator system, improve the accuracy of judgment and reduce labor costs, and achieve Standardized management of the indicator system.
Patent Information
- Application Number
- CN202111354644.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-16
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-11-16
AI Technical Summary
The new indicators in the index system in the existing technology lack a similarity judgment stage, which leads to high manual review costs and inaccurate results, and there are subjective factors.
By judging the similarity between the newly added indicators and the existing indicators, if the similarity is less than the preset threshold, it will be added to the indicator system, use the indicator dictionary to disassemble the indicators, calculate the similarity between the basic indicators and dimensions, and automatically identify the repeated indicators and Avoid duplicate construction.
It reduces labor costs, improves the accuracy of judging indicator similarity, avoids confusion in indicator architecture, and realizes standardized management of enterprise-level unified data indicator system.
Smart Images

Figure CN114049016B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of indicator analysis, and in particular to an indicator similarity judgment method, an indicator similarity judgment system, a terminal device, and a computer-readable storage medium. Background Art
[0002] The similarity judgment of indicators is particularly important for the construction of the indicator system (Indication System-IS). At present, the similarity judgment of indicators mainly relies on manual judgment based on business and technical calibers, and is usually carried out manually when the construction of the basic indicator system has been completed. There is no similarity judgment stage for the newly added indicators of the indicator system. The workload consumed by manual review in the above process is huge, and the indicator similarity judgment results obtained based on manual verification cannot exclude subjective factors, resulting in inaccurate indicator similarity judgment results. Summary of the invention
[0003] The present disclosure provides an indicator similarity judgment method, system, terminal device and computer-readable storage medium to at least solve the current problems of lack of a similarity judgment stage for newly added indicators of the indicator system, high labor costs caused by manual verification, and inaccurate judgment results.
[0004] To achieve the above object, the present disclosure provides an indicator similarity judgment method, comprising:
[0005] Determine whether new indicators are received;
[0006] If a new indicator is received, the similarity between the new indicator and each first stock indicator in the indicator system is calculated to obtain a first similarity calculation result;
[0007] Based on the similarity calculation result, it is determined whether the similarities between the newly added indicator and each first stock indicator are less than a preset threshold;
[0008] If both are smaller than the preset threshold, the new indicator is added to the indicator system.
[0009] In one implementation, calculating the similarity between the newly added indicator and each first stock indicator in the indicator system includes:
[0010] Decomposing the newly added indicator and each first stock indicator respectively to obtain a decomposition result of the newly added indicator and a decomposition result of each first stock indicator;
[0011] Based on the decomposition result of the newly added indicator and the decomposition result of each first stock indicator, the similarity between the newly added indicator and each first stock indicator in the indicator system is calculated.
[0012] In one embodiment, the method further comprises:
[0013] Establishing an indicator dictionary, wherein the indicator dictionary is divided into basic indicators and combined indicators, wherein the combined indicators include basic indicators and dimensions and attributes that modify the basic indicators;
[0014] The newly added indicators and the first stock indicators are respectively disassembled, including:
[0015] The newly added indicator and each first stock indicator are respectively decomposed based on the indicator dictionary.
[0016] In one embodiment, calculating the similarity between the newly added indicator and each first stock indicator in the indicator system based on the decomposition result of the newly added indicator and the decomposition result of each first stock indicator includes:
[0017] Determine whether the newly added indicator is a basic indicator based on the decomposition result of the newly added indicator;
[0018] If it is not a basic indicator, then the basic indicator, dimension and attribute of the newly added indicator are obtained from the disassembly result of the newly added indicator, and the basic indicator, dimension and attribute of each second stock indicator that is not a basic indicator in each first stock indicator are obtained from the disassembly result of each first stock indicator;
[0019] Filter out third stock indicators that are the same as the basic indicator of the newly added indicator from the second stock indicators, and obtain dimensions and attributes of the third stock indicators;
[0020] Calculating the similarity between the dimension and attribute of the basic indicator and the dimension and attribute of each third stock indicator respectively to obtain a second similarity calculation result; and,
[0021] A first similarity calculation result is obtained based on the second similarity calculation result.
[0022] In one embodiment, after judging whether the newly added indicator is a basic indicator based on the decomposition result of the newly added indicator, the method further includes:
[0023] If it is a basic indicator, the newly added indicator is determined to be a repeated indicator, and the step of judging whether the similarity between the newly added indicator and each first stock indicator is less than a preset threshold based on the similarity calculation result is skipped.
[0024] In one implementation, after determining whether a new indicator is received, the method further includes:
[0025] If no new indicator is received, it is determined whether there are duplicate indicators in each of the first stock indicators in the indicator system, and if so, the duplicate indicators are removed.
[0026] In one embodiment, the determining whether there are duplicate indicators in each first stock indicator in the indicator system includes:
[0027] Obtaining the underlying structured query language SQL language of each first stock indicator;
[0028] Splitting the SQL language of each first stock indicator to obtain the splitting result of each first stock indicator; and,
[0029] Based on the split results of the first stock indicators, it is determined whether there are duplicate indicators in the first stock indicators.
[0030] To achieve the above purpose, the present disclosure also provides an indicator similarity judgment system, including:
[0031] A first judgment module, which is configured to judge whether a new indicator is received;
[0032] A calculation module, configured to calculate the similarity between the newly added indicator and each first stock indicator in the indicator system when the judgment module determines that the newly added indicator is received, to obtain a first similarity calculation result;
[0033] A second judgment module is configured to judge whether the similarities between the newly added indicator and each of the first stock indicators are less than a preset threshold based on the similarity calculation result;
[0034] The adding module is configured to add the newly added indicator to the indicator system when the second judging module judges that they are all less than a preset threshold.
[0035] To achieve the above-mentioned purpose, the present disclosure also provides a terminal device, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor runs the computer program stored in the memory, the processor executes the indicator similarity judgment method.
[0036] To achieve the above-mentioned purpose, the present disclosure also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the indicator similarity judgment method.
[0037] According to the indicator similarity judgment method, system, terminal device and computer-readable storage medium provided by the present disclosure, it is determined whether a new indicator is received; if a new indicator is received, the similarity between the new indicator and each first stock indicator in the indicator system is calculated to obtain a first similarity calculation result; based on the similarity calculation result, it is determined whether the similarity between the new indicator and each first stock indicator is less than a preset threshold; if both are less than the preset threshold, the new indicator is added to the indicator system. When the new indicator is received in the pre-application stage, the present disclosure automatically identifies the similarity between the new indicator and the stock indicator to avoid duplication of indicators, reduce labor costs, and improve the accuracy of similarity judgment.
[0038] Other features and advantages of the present disclosure will be described in the following description, and partly become apparent from the description, or be understood by practicing the present disclosure. The purpose and other advantages of the present disclosure can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] The accompanying drawings are used to provide further understanding of the technical solution of the present disclosure and constitute a part of the specification. Together with the embodiments of the present disclosure, they are used to explain the technical solution of the present disclosure and do not constitute a limitation on the technical solution of the present disclosure.
[0040] Figure 1 A flowchart of an indicator similarity judgment method provided in an embodiment of the present disclosure;
[0041] Figure 2 for Figure 1 Schematic diagram of the process of step S12;
[0042] Figure 3 A flowchart of an indicator similarity determination method provided by another embodiment of the present disclosure;
[0043] Figure 4 A flowchart of an indicator similarity judgment method provided by another embodiment of the present disclosure;
[0044] Figure 5 A schematic diagram of the structure of an indicator similarity judgment system provided by an embodiment of the present disclosure;
[0045] Figure 6 A schematic diagram of the structure of a terminal device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0046] To make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described here is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.
[0047] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence; and, in the absence of conflict, the embodiments in the present disclosure and the features in the embodiments can be arbitrarily combined with each other.
[0048] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure. The singular forms of "a", "said" and "the" used in the embodiments of the present disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings.
[0049] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present disclosure, and have no specific meanings. Therefore, "module", "component" or "unit" can be used in a mixed manner.
[0050] An indicator system refers to an organism composed of several interrelated statistical indicators. The establishment of an indicator system is the premise and basis for predictive or evaluation research. It is a process of decomposing an abstract research object into a behavioral and operational structure according to the identification of a certain aspect of its essential attributes and characteristics, and assigning corresponding weights to each constituent element (i.e., indicator) in the indicator system. At present, there may be similar indicators and repeated indicators in the indicator system and when new indicators are added to the indicator system, resulting in confusion in the indicator system structure. To this end, this embodiment proposes a method for judging the similarity of indicators. For the similarity detection of indicators, first, in the pre-application stage, the system automatically distinguishes to avoid the situation of repeated construction of indicators; second, for the existing indicators in the indicator system, according to the actual technical skills, i.e., processing logic, the repeatability of the indicators is judged to screen out repeated indicators, so as to finally realize the standardized management in the process of establishing an enterprise-level unified data indicator system.
[0051] Please refer to Figure 1 , Figure 1 A flowchart of an indicator similarity judgment method provided in an embodiment of the present disclosure, the method comprising steps S11-S14.
[0052] In step S11, it is determined whether a new indicator is received. If a new indicator is received, step S12 is executed, otherwise, the process ends.
[0053] In some embodiments, if no new indicators are received, the existing indicators of the indicator system are checked for duplication to filter out duplicate indicators in the indicator system. For details, please refer to the embodiments described below and will not be repeated here.
[0054] In step S12, the similarity between the newly added indicator and each first stock indicator in the indicator system is calculated to obtain a first similarity calculation result.
[0055] It should be noted that the stock index in this embodiment refers to the index currently existing in the index system. Among them, the first stock index, the second stock index and the third stock index are only used to distinguish similar objects and have no other meanings. In some embodiments, the first stock index, the second stock index and the third stock index can be the same index or different indexes, which depends on the actual situation.
[0056] In one implementation, all indicators are decomposed and then similarity calculation of the indicators is performed based on the decomposition results, which improves the similarity calculation efficiency and the similarity accuracy. Figure 2 As shown, step S12 includes the following steps:
[0057] S121, respectively disassembling the newly added indicator and each first stock indicator to obtain a disassembly result of the newly added indicator and a disassembly result of each first stock indicator;
[0058] S122. Calculate the similarity between the new indicator and each first stock indicator in the indicator system based on the decomposition result of the new indicator and the decomposition result of each first stock indicator.
[0059] In step S13, based on the similarity calculation result, it is determined whether the similarities between the newly added indicator and each first stock indicator are all less than a preset threshold. If they are all less than the preset threshold, step S14 is executed, otherwise the process is terminated, and the newly added indicator is temporarily determined as a duplicate indicator and is no longer added to the indicator system to avoid the addition of duplicate indicators. Alternatively, in some implementations, the indicator is output, and other methods or staff are used to further determine whether the indicator is a duplicate indicator.
[0060] In step S14, the newly added indicator is added to the indicator system.
[0061] Before adding a new indicator to the indicator system in this embodiment, the similarity between the new indicator and the first stock indicators in the indicator system is first calculated, and based on the similarity calculation result, it is decided whether to add the new indicator to the indicator system, thereby replacing the manual verification method and intelligently screening out whether the new indicator is a duplicate indicator or a similar indicator. On the basis of reducing labor costs, it avoids the problem of inaccurate judgment caused by subjective factors, and effectively maintains the structural order of the indicator system, so as to ultimately achieve standardized management in the process of establishing an enterprise-level unified data indicator system.
[0062] Please refer to Figure 3 , Figure 3 A flowchart of an indicator similarity judgment method provided for another embodiment of the present disclosure is provided. On the basis of the previous embodiment, this embodiment establishes an indicator dictionary, provides a basis for indicator decomposition, and sets dimension, attribute and other decomposition methods to solve the process of complex indicators being difficult to confirm whether they exist, improve the efficiency of indicator decomposition, and thus improve the efficiency of indicator similarity judgment. In the similarity judgment process, the weighted judgment method of considering dimension similarity and dimension value similarity is conducive to the establishment of a blood relationship of the subsequent indicator system. Specifically, compared with the previous embodiment, the method also includes step S31, and step S121 is further divided into step S1211.
[0063] In step S31, an indicator dictionary is established, wherein the indicator dictionary is divided into basic indicators and combined indicators, wherein the combined indicators include basic indicators and dimensions and attributes that modify the basic indicators.
[0064] In some implementations, the indicator dictionary may include other indicators in addition to basic indicators and combined indicators, such as comprehensive management indicators, where indicators are usually statistical data, such as "number of 5G package users", rather than individual users or several pieces of data.
[0065] Basic indicators: atomic units of indicators that cannot be further divided, such as "number of users";
[0066] Attribute: A word that modifies a basic indicator, such as "active" in "active users", which is an attribute, because activity needs to be further defined, unlike 5G packages, broadband and other entities, which can be directly obtained and correspond to the subject as dimension-dimension value;
[0067] Dimension: Modify the basic indicators, which can narrow the scope of basic indicator statistics. For example, "number of users" is the basic indicator, "5G active users" is the combined indicator, and "5G" is the dimension value corresponding to the dimension "network standard". The dimension value limits the scope of statistical users to the scope of 5G services. There are several basic dimensions in the dimension, including time dimension and geographical dimension, including year, month, week, day, hour, province, city, district, county, town, etc. It can be understood that the dimension is to modify the classification of the basic indicators in some way, and the actual modification of the indicator is the dimension value.
[0068] Composite indicator: An indicator formed by modifying basic indicators in multiple dimensions and continuously limiting the scope, or an indicator calculated by basic indicators and composite indicators through addition, subtraction, multiplication, division and other operations.
[0069] Comprehensive management indicators: combined indicators or basic indicators, indicators calculated through logical operations of "and", "or" and "exclusive or".
[0070] In one embodiment, the above indicator dictionary can not only be used to decompose indicators, but also to establish an indicator system. Specifically, according to the definitions in the above indicator dictionary, the existing basic indicators, dimension values, combined indicators and comprehensive management indicators are sorted out to establish a basic indicator system. Each indicator in the indicator system may include indicator name, indicator meaning, indicator underlying processing SQL code and other contents.
[0071] In step S1211, the newly added indicator and each first stock indicator are respectively decomposed based on the indicator dictionary.
[0072] In this embodiment, the indicator is decomposed according to the indicator dictionary, wherein the indicator decomposition can rely on the existing word segmentation toolkit, such as the "Jieba" word segmentation tool, and the dimension-dimension value and basic indicator are used as the word segmentation dictionary (the part in the dictionary is no longer refined). For example:
[0073] The indicator "Number of active integrated package users in Beijing in the past year" is transformed into:
[0074] Time dimension Space dimension attributes Other dimension attributes Basic indicators
[0075] Number of active integrated package users in Beijing in the past year
[0076] In the above word segmentation content, the fusion package is the dimension value of the dimension "package". To facilitate search and analysis, in one implementation, each indicator is stored in a nested format of {key: value}, and the attributes can be sorted by pinyin letters, and the dimension values are also sorted by the dimensions to which they belong and the pinyin letters of their own names. For example: indicator name: {time dimension: {recent year}; space dimension: {Beijing City}; attribute: {active, bill}, other dimensions: {fusion package}, basic indicator: {number of users}}.
[0077] It should be noted here that when judging the duplication of indicators, the time dimension and the space dimension can be ignored, and the two dimensions have no impact on the core business meaning of the indicators.
[0078] Further, based on the decomposition result of the newly added indicator and the decomposition result of each first stock indicator, the similarity between the newly added indicator and each first stock indicator in the indicator system is calculated (step S122), including the following steps:
[0079] Determine whether the newly added indicator is a basic indicator based on the decomposition result of the newly added indicator;
[0080] If it is not a basic indicator, then the basic indicator, dimension and attribute of the newly added indicator are obtained from the disassembly result of the newly added indicator, and the basic indicator, dimension and attribute of each second stock indicator that is not a basic indicator in each first stock indicator are obtained from the disassembly result of each first stock indicator;
[0081] Filter out third stock indicators that are the same as the basic indicator of the newly added indicator from the second stock indicators, and obtain dimensions and attributes of the third stock indicators;
[0082] Calculating the similarity between the dimension and attribute of the basic indicator and the dimension and attribute of each third stock indicator respectively to obtain a second similarity calculation result; and,
[0083] A first similarity calculation result is obtained based on the second similarity calculation result.
[0084] Specifically, when a user needs to add a new indicator, it is determined whether a similar indicator already exists for the new indicator. The following is one implementation of this embodiment:
[0085] First, remove the time dimension and space dimension from the word segmentation result of the newly added indicator I_new and convert it into a vector: indicator name: {attribute: {attribute 1, attribute 2}, other dimensions: {dimension value 1, dimension value 2}, basic indicator: {indicator 1}}. For ease of calculation, it can be converted into a formula: I_new = {property: {property_i}, dim: {dim_i}, base: {ind_i}}; i is a natural number greater than or equal to 1.
[0086] A. If the newly added indicator is a basic indicator, it will be judged as a duplicate indicator and cannot be added.
[0087] B. In case of non-A:
[0088] According to the structure of the dimension, according to the dimension value, find the dimension corresponding to the dimension value; the formula is expressed as: key refers to the dimension, val refers to the dimension value
[0089] I_new = {property:{property_i}, dim:{dim_i_key:dim_i_val}, base:{ind_i}}; i is a natural number greater than or equal to 1. For example, dimension value: Fusion Package, the corresponding dimension is "Package Type"
[0090] C. According to the basic indicators, dimensions and attributes involved in the indicator, traverse all non-basic indicators (second stock indicators) in the indicator system, extract the stock indicators with basic indicators (third stock indicators), and record the third stock indicator set as Ic, with a total of c indicators in the set; for each indicator in Ic, base Ii, i = 1, 2, 3, ..., c,
[0091] The attribute similarity value is calculated according to the following formula:
[0092] Sim1=count(I_new.property∩Ii.property) / count(I_new.property+Ii.property)
[0093] Dimension similarity is calculated according to the following formula:
[0094] Sim2=count(I_new.dim_i_val∩Ii.dim_i_val) / count(I_new.dim_i_val+Ii.dim_i_val)
[0095] If the value Sim2! = 0,
[0096] Similarity SIM = (Sim1 + Sim2) / 2
[0097] If Sim2=0, calculate Sim3,
[0098] Sim3=count(I_new.dim_i_key∩Ii.dim_i_key) / count(I_new.dim_i_key+Ii.dim_i_key).
[0099] Similarity Sim = Sim1*0.5+Sim3*0.25
[0100] Finally, the final calculated Sim is output to the user, and the indicators with Sim values greater than 0.7 (threshold is adjustable) are displayed so that the demand personnel can check whether there are completely similar indicators or highly similar indicators.
[0101] It can be understood that the above Sim1, Sim2, and Sim3 are the second similarity calculation results, and SIM and Sim are the first similarity calculation results calculated in two different situations.
[0102] Further, when the newly added indicator is judged to be a basic indicator, the indicator is directly judged to be a repeated indicator without further calculation of indicator similarity, thereby improving the efficiency of indicator similarity judgment. Specifically, after judging whether the newly added indicator is a basic indicator based on the decomposition result of the newly added indicator, the following steps are also included:
[0103] If it is a basic indicator, the newly added indicator is determined to be a repeated indicator, and the step of judging whether the similarity between the newly added indicator and each first stock indicator is less than a preset threshold based on the similarity calculation result is skipped.
[0104] Please refer to Figure 4 , Figure 4 A flowchart of an indicator similarity judgment method provided by another embodiment of the present disclosure is provided. Based on the above embodiment, when no new indicator is received, this embodiment performs a repeated self-check on the stock indicators in the indicator system to automatically obtain indicators with consistent processing logic but inconsistent names in the existing indicator library to standardize the enterprise-level data indicator system. Specifically, after determining whether the new indicator is received (S11), it also includes:
[0105] In step S41, if no new indicator is received, it is determined whether there are duplicate indicators in the first stock indicators in the indicator system. If so, step S42 is executed, otherwise the process ends.
[0106] In step S42, the duplicate indicators are removed.
[0107] Furthermore, this embodiment disassembles the word table implementation process according to the underlying SQL code of the stock indicators to obtain the processing logic of the indicators and screen out the seemingly different but essentially identical duplicate indicators. Specifically, the step of determining whether there are duplicate indicators in each of the first stock indicators in the indicator system (S41) includes the following steps:
[0108] Obtaining the underlying structured query language SQL language of each first stock indicator;
[0109] Splitting the SQL language of each first stock indicator to obtain the splitting result of each first stock indicator; and,
[0110] Based on the split results of the first stock indicators, it is determined whether there are duplicate indicators in the first stock indicators.
[0111] In this embodiment, the main processing indicators have been established, but for some indicators, the indicator names are different, but the essential processing logic is the same. Such indicators are screened out to standardize the indicator system.
[0112] In the existing indicator system, each indicator stores the corresponding SQL (Structured Query Language) language, and the indicator processing logic can be obtained based on the SQL language:
[0113] (1) In the database, you can obtain the SQL execution plan by using the EXPLAIN keyword. The calculation process is generally as follows:
[0114] select [count | ave | sum] (data field) from table / view
[0115] join (left, right, full) table / view on table / view A.field = table / view table B.field
[0116] where filter condition
[0117] group by
[0118] having count filter condition
[0119] For example, if a certain indicator has many screening conditions, the database engine will adjust the actual execution order according to its optimal mode. Its execution plan is the actual execution order.
[0120] (2) According to the execution plan, split the indicator SQL language to obtain the main body and restrictions of the indicator:
[0121] 1. Basic indicators: data fields and previous metrics, i.e. basic indicators ---- "[count|ave|sum](data field)"
[0122] 2. Extraction of attributes and dimensions:
[0123] The extraction of attributes and dimensions is relatively complex. In practical applications, it is not about one-to-one correspondence, but only about extracting the main processing logic. Including:
[0124] (1) Associated table / view: {name1, name2};
[0125] (2) Basic restrictions (in the where clause): {subject: value range; subject: value range}; the subject condition is set to an and relationship by default. For example, if A>3, the subject is A, and the value range is (3, +inf).
[0126] (3) Whether there is an or statement. If there is, divide the or statement into multiple groups of value contents according to the processing position of or. For example, select A.name from A join B on A.id=B.id where A.col1>3and(B.col1>6orB.col1<3) then there will be two groups of restrictions, one is {A.col1:(3,+inf); B.col1:(6,+inf)}, and the other is {A.col1:(3,+inf); B.col1:(-inf,3)}
[0127] (4) Grouping restrictions: {subject} group by corresponding statement
[0128] (5)Having count restriction: {subject: value range}
[0129] According to the above method, the logic of processing an indicator can be disassembled into a series of data items for storage, and then analyzed and identified according to the disassembly results. Specifically, all the first stock indicators are disassembled into the main body (basic indicators) and restrictions in the SQL language according to the above method, and then the tables and columns with the same basic indicators are divided into multiple subsets with the same basic indicators according to the stored main body (tables and columns) to form a subset C, C = {C1, C2, ..., Cn}, each Ci represents an indicator set with the same basic indicators; then for each Ci, operations are performed to compare all indicators of Ci, first grouping according to group by and having count; finally, the restricted content in each group is extracted, including indicators with exactly the same restricted fields, value ranges, etc. These indicators are repeated indicators, and these indicators are output.
[0130] Compared with the related art, this embodiment first avoids the inability to extract the same indicators due to differences in SQL statements, splits the SQL statements into actual subjects (basic indicators), restrictions and value ranges, shields details, extracts key content for judgment, and realizes efficient duplication checking of indicators.
[0131] Based on the same technical concept, the embodiment of the present disclosure also provides an indicator similarity judgment system, such as Figure 5 As shown, the system comprises:
[0132] A first determination module 51, configured to determine whether a new indicator is received;
[0133] A calculation module 52, which is configured to calculate the similarity between the newly added indicator and each first stock indicator in the indicator system to obtain a first similarity calculation result when the judgment module determines that the newly added indicator is received;
[0134] A second judgment module 53, which is configured to judge whether the similarities between the newly added indicator and each first stock indicator are less than a preset threshold based on the similarity calculation result;
[0135] The adding module 54 is configured to add the newly added indicator to the indicator system when the second judging module determines that they are all less than a preset threshold.
[0136] In one implementation, the calculation module 52 includes:
[0137] A disassembling unit, configured to respectively disassemble the newly added indicator and each first stock indicator to obtain a disassembling result of the newly added indicator and a disassembling result of each first stock indicator;
[0138] A calculation unit is configured to calculate the similarity between the new indicator and each first stock indicator in the indicator system based on the disassembly result of the new indicator and the disassembly result of each first stock indicator.
[0139] In one embodiment, the system further comprises:
[0140] Establishing an indicator dictionary, wherein the indicator dictionary is divided into basic indicators and combined indicators, wherein the combined indicators include basic indicators and dimensions and attributes that modify the basic indicators;
[0141] The separate disassembling unit is specifically configured to separately disassemble the newly added indicator and each first stock indicator based on the indicator dictionary.
[0142] In one embodiment, the computing unit comprises:
[0143] A judging element, configured to judge whether the newly added indicator is a basic indicator based on the disassembly result of the newly added indicator;
[0144] An acquisition component is configured to, when the component is determined to be not a basic indicator, acquire the basic indicator, dimension and attribute of the newly added indicator from the disassembly result of the newly added indicator, and acquire the basic indicator, dimension and attribute of each second stock indicator that is not a basic indicator in each first stock indicator from the disassembly result of each first stock indicator;
[0145] A screening component, which is configured to screen out third stock indicators that are the same as the basic indicator of the newly added indicator from the second stock indicators, and obtain dimensions and attributes of the third stock indicators;
[0146] a calculation element, configured to respectively calculate the similarity between the dimension and attribute of the basic indicator and the dimension and attribute of each third stock indicator to obtain a second similarity calculation result; and
[0147] The calculation element is further configured to obtain a first similarity calculation result based on the second similarity calculation result.
[0148] In one embodiment, the system further comprises:
[0149] The judgment jump-out module is configured to, when the judgment element judges that it is a basic indicator, judge the newly added indicator as a repeated indicator, and jump out of the second judgment module 53 to judge whether the similarity between the newly added indicator and each first stock indicator is less than a preset threshold based on the similarity calculation result.
[0150] In one embodiment, the system further comprises:
[0151] The judgment and elimination module is configured to judge whether there are duplicate indicators in each first stock indicator in the indicator system when the first judgment module 51 judges that no new indicator is received, and if so, eliminate the duplicate indicators.
[0152] In one implementation, the judging and rejecting module includes:
[0153] An acquisition unit, configured to acquire the underlying structured query language SQL language of each first stock indicator;
[0154] a splitting unit, configured to split the SQL language of each first stock indicator to obtain a split result of each first stock indicator; and
[0155] A duplication judgment unit is configured to judge whether there are duplicate indicators in each of the first stock indicators based on the split results of each of the first stock indicators.
[0156] Based on the same technical concept, the embodiment of the present disclosure also provides a terminal device, such as Figure 6 As shown, the terminal device includes a memory 61 and a processor 62, wherein the memory 61 stores a computer program, and when the processor 62 runs the computer program stored in the memory 61, the processor executes the indicator similarity judgment method.
[0157] Based on the same technical concept, the embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the indicator similarity judgment method.
[0158] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those of ordinary skill in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.
[0159] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present disclosure, rather than to limit them. Although the present disclosure has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present disclosure.
Claims
1. A method for judging index similarity, characterized in that: include: Determine whether new indicators are received; If a new indicator is received, the similarity between the new indicator and each first stock indicator in the indicator system is calculated to obtain a first similarity calculation result; Based on the similarity calculation result, determining whether the similarities between the newly added indicator and each first stock indicator are all less than a preset threshold; If both are less than the preset threshold, the newly added indicator is added to the indicator system; The calculating the similarity between the newly added indicator and each first stock indicator in the indicator system includes: Decomposing the newly added indicator and each first stock indicator respectively to obtain a decomposition result of the newly added indicator and a decomposition result of each first stock indicator; Calculating the similarity between the newly added indicator and each first stock indicator in the indicator system based on the disassembly result of the newly added indicator and the disassembly result of each first stock indicator; Also includes: Establishing an indicator dictionary, wherein the indicator dictionary is divided into basic indicators and combined indicators, wherein the combined indicators include basic indicators and dimensions and attributes that modify the basic indicators; The separately disassembling of the newly added indicators and the first stock indicators includes: Based on the indicator dictionary, the newly added indicator and each first stock indicator are respectively disassembled; The calculating the similarity between the newly added indicator and each first stock indicator in the indicator system based on the decomposition result of the newly added indicator and the decomposition result of each first stock indicator includes: Determine whether the newly added indicator is a basic indicator based on the decomposition result of the newly added indicator; If it is not a basic indicator, then the basic indicator, dimension and attribute of the newly added indicator are obtained from the disassembly result of the newly added indicator, and the basic indicator, dimension and attribute of each second stock indicator that is not a basic indicator in each first stock indicator are obtained from the disassembly result of each first stock indicator; Filter out third stock indicators that are the same as the basic indicator of the newly added indicator from the second stock indicators, and obtain dimensions and attributes of the third stock indicators; Calculating the similarity between the dimension and attribute of the basic indicator and the dimension and attribute of each third stock indicator respectively to obtain a second similarity calculation result; and, A first similarity calculation result is obtained based on the second similarity calculation result.
2. The method according to claim 1, characterized in that: After judging whether the newly added indicator is a basic indicator based on the decomposition result of the newly added indicator, the method further includes: If it is a basic indicator, the newly added indicator is determined to be a repeated indicator, and the step of judging whether the similarity between the newly added indicator and each first stock indicator is less than a preset threshold based on the similarity calculation result is skipped.
3. The method according to claim 1, characterized in that After determining whether a new indicator has been received, the following steps are also included: If no new indicator is received, it is determined whether there are duplicate indicators in each of the first stock indicators in the indicator system, and if so, the duplicate indicators are removed.
4. The method according to claim 3, characterized in that The determining whether there are duplicate indicators in each first stock indicator in the indicator system includes: Obtaining the underlying structured query language SQL language of each first stock indicator; Splitting the SQL language of each first stock indicator to obtain the splitting result of each first stock indicator; and, Based on the split results of the first stock indicators, it is determined whether there are duplicate indicators in the first stock indicators.
5. An indicator similarity judgment system, characterized in that: include: A first judgment module, which is configured to judge whether a new indicator is received; A calculation module, configured to calculate the similarity between the newly added indicator and each first stock indicator in the indicator system when the judgment module determines that the newly added indicator is received, to obtain a first similarity calculation result; A second judgment module is configured to judge whether the similarities between the newly added indicator and each of the first stock indicators are less than a preset threshold based on the similarity calculation result; An adding module, configured to add the newly added indicator to the indicator system when the second judging module judges that the newly added indicator is less than a preset threshold; The calculation module comprises: A disassembling unit, configured to respectively disassemble the newly added indicator and each first stock indicator to obtain a disassembling result of the newly added indicator and a disassembling result of each first stock indicator; A calculation unit, configured to calculate the similarity between the newly added indicator and each first stock indicator in the indicator system based on the decomposition result of the newly added indicator and the decomposition result of each first stock indicator; The system further comprises: An indicator dictionary establishing unit, which is configured to establish an indicator dictionary, wherein the indicator dictionary is divided into basic indicators and combined indicators, wherein the combined indicators include basic indicators and dimensions and attributes that modify the basic indicators; The disassembling unit is specifically configured to respectively disassemble the newly added indicator and each first stock indicator based on the indicator dictionary; The computing unit comprises: A judging element, configured to judge whether the newly added indicator is a basic indicator based on the disassembly result of the newly added indicator; An acquisition component is configured to, when the component is determined to be not a basic indicator, acquire the basic indicator, dimension and attribute of the newly added indicator from the disassembly result of the newly added indicator, and acquire the basic indicator, dimension and attribute of each second stock indicator that is not a basic indicator in each first stock indicator from the disassembly result of each first stock indicator; A screening component, which is configured to screen out third stock indicators that are the same as the basic indicator of the newly added indicator from the second stock indicators, and obtain dimensions and attributes of the third stock indicators; A calculation element, which is configured to respectively calculate the similarity between the dimension and attribute of the basic indicator and the dimension and attribute of each third stock indicator to obtain a second similarity calculation result; and The calculation element is further configured to obtain a first similarity calculation result based on the second similarity calculation result.
6. A terminal device, characterized in that: It comprises a memory and a processor, wherein the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the indicator similarity judgment method according to any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the processor executes the indicator similarity judgment method according to any one of claims 1 to 4.
