A financial variable retrieval method based on semantic-statistical dual space fusion and polysemy abbreviation closed loop disambiguation verification

Abstract

The application provides a financial variable retrieval method based on semantic-statistical dual-space fusion and polysemy abbreviation closed-loop disambiguation verification. The application can realize high-precision retrieval of financial variables by constructing a semantic vector retrieval space and a statistical metadata retrieval space. At the same time, the financial abbreviation polysemy disambiguation module is used to accurately analyze the multiple meanings of abbreviations combined with context information, solving the problem of abbreviation ambiguity and variable retrieval fragmentation in traditional methods. Through a bidirectional closed-loop consistency verification mechanism, the system uses the retrieval results to verify the meaning of the abbreviation in reverse, dynamically updates, and ensures the statistical consistency of variable matching. Specifically, the application includes receiving user queries and preprocessing, disambiguating abbreviations and generating retrieval intent, parallel retrieval in dual space, fusion scoring and sorting of results, reverse verification and updating of abbreviation meaning, and finally outputting accurate variable candidates and their statistical attributes. The application can effectively reduce financial variable mismatching and abbreviation misinterpretation, improve the reliability, interpretability and reproducibility of retrieval results, and has a wide application prospect, especially suitable for retrieval fields assisted by big data and large language models.

Description

[0001] A financial variable retrieval method based on semantic-statistical dual-space fusion and polysemous abbreviation closed-loop disambiguation verification is characterized by comprising two stages: offline data index construction and online retrieval. The offline data index construction stage sequentially executes the following steps:

[0002] Step A1: Standardize the financial variable records from statistical databases, commercial databases, research team data dictionaries, and literature variable definition tables, mapping the original fields to a unified metadata schema to form a variable metadata record structure that includes variable name, statistical scope, unit, frequency, and time coverage. At the same time, perform unit standardization on the unit field and create a unit conversion chart.

[0003] Step A2: Construct a statistical signature based on the variable metadata record structure. ,in Encoding the data source, For frequency coding, Unit coding, For statistical caliber coding, For seasonal adjustment indicators, For discount indication, For logarithmic transformation identifier, Let be the difference order. For time coverage interval, For regional encoding, Encode the sample range, then build an inverted index and a hash bucket index on the statistical signature, and construct a statistical metadata database;

[0004] Step A3: Segment, clean, and standardize the terminology of the descriptive text of financial variables, data dictionary descriptions, database field annotation texts, and variable usage paragraphs in the literature. After adding structured prompts such as variable names, aliases, statistical definitions, and frequencies to the text fragments, perform vectorization processing to build a semantic vector index library.

[0005] Step A4: Construct a polysemous abbreviation graph ,in A set of financial abbreviation nodes. For a set of meaningful nodes, For the mapping edge between abbreviations and their meanings, configure feature parameters such as the full meaning, domain label vector, trigger word set, and method association word set for each meaning node to establish a pre-mapping relationship between financial abbreviations and candidate meanings;

[0006] The online search phase executes the following steps in sequence:

[0007] Step S1: Receive the user's natural language query for financial variable retrieval, perform text normalization processing on the query text to unify capitalization, normalize punctuation, convert full-width / half-width characters, and clean up whitespace characters to obtain a standardized query. At the same time, extract lexical features, syntactic features, frequency of inclusion, unit, statistical caliber, time range statistical clue features, domain clue features, and a set of financial abbreviations from the query text.

[0008] Step S2: For each financial abbreviation, obtain the corresponding set of candidate meanings from the abbreviation polysemy graph. Based on the query context, and combined with keyword contribution features, method word matching features, variable category preference matching features, syntactic position matching features, domain label matching features, and historical context preference features, calculate the context matching score for each candidate meaning.

[0009] Step S3: Based on the standardized query, the extracted features, the candidate meaning set of financial abbreviations, and the context matching score of each candidate meaning, construct a structured retrieval intent object that uniformly represents semantic retrieval information, abbreviation candidate meaning information, statistical constraint information, and domain context information;

[0010] Step S4: Based on the structured retrieval intent object, construct a semantic subquery for each candidate meaning of financial abbreviation and generate a corresponding vector representation. Perform nearest neighbor recall in the semantic vector index library for each semantic subquery. Take the union of the recall results to obtain a semantic candidate result set. Then, calculate the semantic retrieval score of each record in the semantic candidate result set by weighted average method. The weight of the weighted average is based on the conditional probability of the candidate meaning and the context similarity of the query.

[0011] Step S5: Based on the structured retrieval intent object, perform structured matching and statistical signature pre-screening in the statistical metadata database, and calculate the statistical compatibility score of each record in the statistical candidate result set;

[0012] Step S6: The semantic candidate result set and the statistical candidate result set are fused, scored, and ranked using a gating fusion method, through a gating function. Generate the gating coefficients, where Here, k is the sigmoid function, and k is the gate steepness parameter. To calculate the compatibility threshold, To adjust the parameters, use the formula. Calculate the final fusion score for each candidate record, where Score(v) is the final fusion score. For semantic retrieval scores, For semantic weighting coefficients in scenarios with low statistical compatibility, To calculate the compatibility score weights, the candidate records are sorted in descending order based on the final fusion score.

[0013] Step S7: Perform reverse consistency verification on the meanings of financial abbreviations. Extract evidence sets including domain labels, method association labels, variable category labels, and statistical attribute patterns from the top K records of the candidate record ranking results. Calculate the support score for each candidate meaning of a financial abbreviation using a Top-K voting method, and then apply the formula... Calculate the difference D(a) between the context matching score of forward disambiguation and the support of reverse evidence, where ε is a smoothing term. When the difference D(a) exceeds the preset threshold δ, update the candidate meaning ranking and context matching score based on the support, reconstruct the structured retrieval intent object, and re-execute steps S4 to S6. The iteration termination condition is reaching the maximum number of iterations, the candidate meaning ranking remains unchanged in two adjacent rounds, the ranking of the first K candidate records in two adjacent rounds is stable, or the difference D(a) does not exceed the preset threshold δ.

[0014] Step S8: Output the candidate results of financial variables sorted by the final fusion score, and output the explanation results of financial abbreviations, semantic matching keywords, statistical compatibility sub-item scores, matching evidence and confidence information corresponding to the candidate results. The explanation results of financial abbreviations are determined by the candidate meaning with the highest support.

[0015] 2. The financial variable retrieval method according to claim 1, wherein the unified metadata model in step A1 further includes at least one of variable category label, method association label, variable transformation path description, source version number, missing rate, and revision risk score.

[0016] 3. The variable retrieval method according to claim 1, characterized in that, in step S2, each candidate meaning The context matching score is calculated using the following formula:

[0017]

[0018] in To score for keywords, For method word matching score, Scoring the variable category preference. Scoring is given based on syntactic position or role matching. Scoring based on domain tag matching, Score the historical context preference. These are the weight parameters.

[0019] 4. The financial variable retrieval method according to claim 1, characterized in that, in step S5, each compatibility level is assigned a value according to the following rules: complete consistency is assigned a value of 1, convertible / partially compatible is assigned a value between 0 and 1, and incompatible is assigned a value of 0.

[0020] 5. The variable retrieval method according to claim 1, wherein the statistical compatibility score in step S5 is expressed as:

[0021]

[0022] in For frequency compatibility; Unit compatibility; To ensure compatibility of statistical standards; For time range compatibility; For the corresponding weights.

[0023] 6. The financial variable retrieval method according to claim 1, wherein the Top-K voting calculation method in step S7 is as follows: when the candidate record evidence satisfies at least one matching rule of the candidate meaning, the number of support votes is accumulated for the corresponding candidate meaning, and the support is the total number of support votes obtained by the candidate meaning.

[0024] A Financial Variable Retrieval Method Based on Semantic-Statistical Dual-Space Fusion and Polysemous Abbreviation Closed-Loop Disambiguation Validation Technical Field

[0025] This invention relates to the fields of artificial intelligence, large model retrieval, and finance, and particularly to a variable retrieval and terminology understanding enhancement method for financial scenarios. Specifically, it relates to a method that jointly models semantic retrieval, statistical metadata retrieval, and financial abbreviation disambiguation and improves retrieval accuracy through closed-loop consistency verification. Background Technology

[0026] With the widespread application of Large Language Models (LLMs) in the financial field, more and more researchers and developers are beginning to try to fine-tune large models in specific domains to improve their performance in real-world tasks. Financial data is complex and structurally diverse; therefore, how to efficiently and accurately fine-tune large models has become an important research direction in this field.

[0027] Currently, common methods for fine-tuning large models mainly include three approaches: basic fine-tuning, self-supervised learning, and retrieval-enhanced generation (RAG). Basic fine-tuning methods, such as LoRA, adapt the model to a specific task by training it directly on task-related data. However, a major problem with basic fine-tuning is that as the model size increases, the training cost and required computational resources rise sharply, making it unaffordable for many financial institutions in practical applications. Self-supervised learning enhances the model's generalization ability by pre-training on large-scale unlabeled data, but this method consumes a large amount of training data and computational resources. Furthermore, in the financial field, many data and problems are specialized and domain-specific, making precise fine-tuning impossible. Retrieval-enhanced generation (RAG), on the other hand, combines external knowledge sources (such as databases, literature repositories, and report repositories) with the model's generation capabilities through a "retrieval-enhancing-generation" process. The system first performs semantic retrieval based on the user's question, recalling relevant text fragments or evidence, then inputting this as supplementary knowledge into the model. Finally, the model generates an answer under the constraints of external evidence, improving the accuracy of the generated results in a manner similar to an "external knowledge base." RAGs require no training, are convenient and quick, inexpensive, and suitable for any large model. These excellent characteristics make them widely used in the field of fine-tuning.

[0028] Despite the widespread application of RAG, its limitations lie in the fact that current RAG methods typically rely on text similarity to retrieve information, failing to consider task-specific statistical attributes and data consistency, especially in the processing of financial data. For example, nominal GDP and real GDP are completely different in data format and statistical caliber; simply relying on text similarity matching can easily lead to mismatches and misinterpretations, resulting in misleading decision-making and research outputs in the financial field.

[0029] Furthermore, the ambiguity of financial terms and abbreviations has long plagued the accuracy of retrieval. In the financial field, many terms and abbreviations have multiple meanings, and their interpretation often depends on the context. For example, "FE" in finance and economics may refer to "Fixed Effects" or "Frontier Efficiency," while "IV" may represent "Instrumental Variables" or "Implied Volatility." Existing methods for handling terminology abbreviations, such as simple mapping dictionary methods, cannot resolve the context-dependent problem of these ambiguous abbreviations, leading to comprehension biases in the model during processing.

[0030] Furthermore, existing technologies typically process abbreviation explanations and variable retrieval separately, lacking a linked verification mechanism: abbreviation disambiguation results fail to effectively participate in variable statistical attribute matching, and retrieval results cannot reverse-verify the reasonableness of abbreviation explanations. When the initial explanation of an abbreviation is biased, the error may be amplified in subsequent retrieval and generation processes.

[0031] To overcome the aforementioned problems, this invention introduces a semantic-statistical dual-space retrieval and an abbreviation polysemy disambiguation mechanism. During the retrieval process, it jointly utilizes contextual relevance and statistical consistency to effectively avoid errors caused by inconsistent data definitions or terminological polysemy. Specifically, in abbreviation disambiguation, this invention employs a forward discrimination method based on contextual matching scores to compare candidate meanings, rather than relying on independent probability distribution modeling. Simultaneously, the retrieval results can be used to verify the reasonableness of abbreviation interpretations, thus forming a closed-loop optimization mechanism combining forward disambiguation and reverse verification.

[0032] This invention enables simultaneous context-dependent disambiguation of polysemous abbreviations, joint retrieval of semantic relevance and statistical attribute consistency, candidate result fusion and ranking, and reverse closed-loop verification of abbreviation interpretations in natural language query processing. This improves the accuracy, statistical consistency, and interpretability of financial variable retrieval and terminology understanding. It is simple, efficient, and widely applicable to any large language model, solving the problems of statistical attribute bias and semantic ambiguity inherent in traditional retrieval methods in the financial field with extremely low computational cost. Summary of the Invention

[0033] The purpose of this invention is to provide a variable retrieval method for financial search scenarios based on semantic-statistical dual-space fusion and closed-loop disambiguation verification of polysemous abbreviations. This scheme constructs a semantic vector retrieval space and a statistical metadata retrieval space to jointly model the semantic requirements and statistical attribute constraints of variables in user natural language queries. Furthermore, through a polysemous abbreviation disambiguation module and a reverse verification mechanism for search results, it achieves closed-loop consistency optimization between abbreviation semantic interpretation and variable retrieval results. This reduces problems such as variable mismatch, mixed statistical definitions, frequency mismatch, and misinterpretation of abbreviations in financial search scenarios, improving the statistical consistency, interpretability, reproducibility, and reliability of downstream large-scale model applications of the search results.

[0034] To achieve the above objectives, this invention provides a variable retrieval method based on semantic-statistical dual-space fusion and polysemous abbreviation closed-loop disambiguation verification. The method includes an offline data index construction stage and an online retrieval stage.

[0035] In some implementations, the offline data indexing construction phase includes the following steps:

[0036] Step A1 involves standardizing variable records from data sources such as statistical databases, commercial databases, research team data dictionaries, and literature variable definition tables, mapping the original fields to a unified metadata schema, and forming a variable metadata record structure.

[0037] Step A2: Construct a statistical signature based on the variable metadata record structure, and establish an inverted index and a hash bucket index to support subsequent fast candidate pre-screening based on statistical constraints;

[0038] Step A3 involves segmenting, cleaning, standardizing and vectorizing variable description text, data dictionary descriptions, database field annotation text, and variable usage paragraphs in the literature to construct a semantic vector index.

[0039] Step A4: Construct an abbreviation polysemous graph, establish the mapping relationship between abbreviation nodes and multiple candidate meaning nodes, and configure context matching feature parameters for each candidate meaning.

[0040] In some implementations, the unified metadata schema in step A1 includes, in addition to fields such as variable name, variable description text, data source, unit, frequency, statistical caliber, and time coverage, at least one of the following: variable category label, method association label, variable transformation path description, source version number, missing rate, and revision risk score.

[0041] In some implementations, step A1 further includes performing unit standardization on the unit field and establishing a unit conversion graph to support determinations such as "completely consistent", "convertible", and "incompatible" in unit compatibility calculations.

[0042] In some implementations, the statistical signature described in step A2 is used to encode statistical attributes such as the source, frequency, unit, statistical scope, seasonal adjustment identifier, deflator identifier, logarithmic transformation identifier, difference order, time coverage interval, regional range, and sample range of variables, so as to facilitate rapid filtering and compatibility prediction during retrieval.

[0043] In some implementations, when constructing the semantic vector index in step A3, structured prompt information can be added to the original text fragment before encoding, so that the vector representation can simultaneously absorb variable names, aliases, statistical definitions, frequencies, and explanatory text information, thereby improving the quality of semantic retrieval recall.

[0044] In some implementations, in step A4, for each candidate meaning node of an abbreviation, at least one or more of the following information is stored: full meaning, domain tag vector, trigger word set, method association word set, and variable category preference information; the feature parameters are used to subsequently calculate the context matching score of the candidate meaning in conjunction with the query context.

[0045] In some implementations, the online retrieval phase includes the following steps:

[0046] Step S1: Receive the user's natural language query and perform preprocessing and feature encoding to obtain a standardized query and query features; the query features include at least lexical features, syntactic features, statistical cue features, domain cue features, and a set of abbreviations.

[0047] Step S2: Perform forward polysemy disambiguation on the abbreviations, obtain the candidate meaning set corresponding to each abbreviation from the abbreviation polysemy graph, and calculate the context matching score of each candidate meaning based on the query context.

[0048] Step S3: Construct a structured retrieval intent object based on the standardized query, query features, candidate meaning set corresponding to each abbreviation, and context matching score. The structured retrieval intent object is used to uniformly represent semantic retrieval information, abbreviation candidate meaning information, statistical constraint information, and domain context information.

[0049] Step S4: Based on the structured retrieval intent object, perform semantic retrieval in the semantic vector space using a recall-then-ranking approach to obtain a set of semantic candidate results and corresponding semantic scores.

[0050] Step S5: Based on the structured retrieval intent object, perform structured matching, statistical signature pre-screening, and statistical compatibility scoring in the statistical metadata space to obtain a set of statistical candidate results.

[0051] Step S6: The semantic candidate result set and the statistical candidate result set are fused, scored, and sorted to obtain the candidate record sorting result;

[0052] Step S7: Based on the candidate record ranking results, perform reverse consistency verification on the meaning of the abbreviation, compare the difference between the context matching result obtained by forward disambiguation and the reverse evidence support result, and trigger at least one re-retrieval when the difference exceeds a preset threshold, so as to form a closed-loop iterative process of "forward disambiguation - dual-space retrieval - reverse verification".

[0053] Step S8: Output the final candidate record results, as well as the abbreviation explanations, semantic matching information, statistical compatibility information, and conflict warning information corresponding to the final candidate record results.

[0054] In some implementations, the preprocessing in step S1 includes at least one of the following: case unification, punctuation normalization, full-width / half-width conversion, and whitespace character cleanup; the statistical clue features include at least one of the following: frequency prompts, unit prompts, statistical caliber prompts, time range prompts, preprocessing status prompts, regional range prompts, and sample range prompts.

[0055] In some implementations, in step S2, for each candidate meaning, a context matching score is calculated based on at least one of keyword contribution features, method word matching features, record category preference matching features, syntactic position or role matching features, domain label matching features, and historical context preference features.

[0056] In some implementations, the statistical constraint information in step S3 includes at least one or more of the following fields: target frequency, target unit, target statistical caliber, target time range, preprocessing state constraint, regional range constraint, and sample range constraint; the structured retrieval intent object may also include standardized queries, lexical features, syntactic features, abbreviation set, candidate meaning set, and their context matching scores.

[0057] In some implementations, in step S4, for queries with multiple abbreviation meanings, a semantic subquery is constructed for each candidate meaning and a corresponding vector representation is generated; each semantic subquery performs nearest neighbor recall in the semantic vector index, forms a union of candidate records, and then combines the context matching score of the candidate meaning to rearrange the recall results to obtain a semantic score.

[0058] In some implementations, the statistical compatibility score in step S5 incorporates at least two or more of the following compatibility indicators: frequency compatibility, unit compatibility, statistical caliber compatibility, time range compatibility, preprocessing state compatibility, regional range compatibility, sample range compatibility, source quality score, and revision risk-related score.

[0059] In some implementations, the statistical compatibility score in step S5 adopts a weighted summation model, with the compatibility value ranging from [0,1], and is assigned a value based on preset rules; where complete consistency is assigned a high score, convertible / partially compatible is assigned a medium score, and incompatibility is assigned a low score or zero score.

[0060] In some implementations, when performing fusion scoring and sorting on candidate records in step S6, any one of gated fusion, linear fusion, or ranking-level fusion is used; wherein in the gated fusion method, a gating function is generated using statistical compatibility scores to dynamically reduce the weight of semantic scores, thereby preventing candidate records that are "semantically highly related but statistically obviously incompatible" from entering the high-ranking order.

[0061] In some implementations, the reverse consistency check in step S7 includes: extracting a set of evidence related to the abbreviation meaning discrimination from the Top-K candidate records; calculating the support of each candidate meaning based on the set of evidence; comparing the difference between the support result and the context matching score obtained in step S2; and when the difference exceeds a preset threshold, reconstructing the retrieval intent object based on the updated candidate meaning ranking or context matching score and performing a second retrieval.

[0062] In some implementations, the evidence set in step S7 includes at least one of the following: domain / subdomain labels, method association labels, variable category labels, and statistical attribute patterns; the support calculation can adopt a Top-K voting method, wherein when the candidate record evidence satisfies at least one matching rule of the candidate meaning, the number of support votes is accumulated for the corresponding candidate meaning.

[0063] In some implementations, the triggering conditions in step S7 include at least one of the following: 1. The maximum number of iterations is reached; 2. The candidate meaning ranking remains unchanged in two adjacent iterations; 3. The ranking of the top K candidate records remains stable in two adjacent iterations; 4. The difference between the forward disambiguation ranking result and the reverse evidence ranking result does not exceed a preset threshold.

[0064] In some implementations, the explanation information output in step S8 includes at least one of the following: abbreviation explanation result, semantic matching keywords / similar segments / similarity intervals, statistical compatibility sub-item score, conflict warning information, and result confidence warning; wherein the abbreviation explanation result can be determined by the candidate meaning with the highest support.

[0065] This invention also provides a variable retrieval system based on semantic-statistical dual-space fusion and polysemous abbreviation closed-loop disambiguation verification, for implementing the above method, including:

[0066] The offline data preparation and index building module is used to perform variable metadata standardization, statistical signature construction, semantic vector index construction, and abbreviation polysemous graph construction.

[0067] The user query preprocessing and feature encoding module is used to receive user natural language queries and extract query features;

[0068] The abbreviation polysemy disambiguation module is used to generate candidate meanings and calculate the context matching score of the candidate meanings;

[0069] The search intent building module is used to generate structured search intent objects;

[0070] The semantic vector retrieval module is used to perform semantic similarity retrieval and output a set of semantic candidate results;

[0071] The statistical metadata retrieval module is used to perform structured matching and calculate statistical compatibility scores based on variable metadata and statistical signatures.

[0072] The fusion scoring and ranking module is used to fuse and score the semantic candidate result set and the statistical candidate result set.

[0073] The closed-loop consistency verification module is used to reverse verify the meaning of abbreviations using the candidate record sorting results and trigger a second search when the conditions are met.

[0074] The explanation output module is used to output the final candidate results of variables and explanatory information.

[0075] In some implementations, the statistical metadata retrieval module performs structured matching based on the variable metadata record structure and statistical signature; the variable metadata record structure includes at least one of the following: unique variable identifier, variable name, variable description text, data source, statistical caliber, unit, frequency, time coverage, preprocessing status, variable category label, method association label, variable transformation path description, source version number, missing rate, revision risk score, and source quality score.

[0076] The present invention also provides an electronic device, including a processor, a memory, and a computer program stored in the memory, wherein when the computer program is executed by the processor, the processor performs the steps of the variable retrieval method as described in any of the above embodiments.

[0077] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the variable retrieval method as described in any of the above embodiments.

[0078] Compared with the prior art, the present invention has at least the following beneficial effects:

[0079] (1) By jointly retrieving and merging the semantic vector space and the statistical metadata space, statistical consistency constraints are introduced while maintaining semantic relevance, which can effectively reduce the problems of record mismatch, mixed statistical standards and frequency mismatch.

[0080] (2) By using the polysemous abbreviation graph and context-driven matching scoring mechanism, the context-related judgment of polysemous abbreviations in financial variable scenarios is carried out to reduce the risk of misinterpretation of professional terms;

[0081] (3) Through the closed-loop consistency mechanism of "forward disambiguation - dual-space retrieval - reverse verification - re-retrieval", the retrieval results can be reverse-constrained for abbreviation interpretation, thereby improving the overall robustness and retrieval accuracy of the system.

[0082] (4) By using statistical signature pre-screening, unit conversion charts and computable statistical compatibility scoring rules, factors such as frequency, unit, statistical scope, time coverage, and preprocessing status are quantified to improve retrieval efficiency, interpretability and auditability.

[0083] (5) The output of this invention includes a candidate record list, abbreviation explanation results, semantic matching information, statistical compatibility information and conflict prompt information. It can be used as a pre-retrieval enhancement module for large language models to improve the reliability of downstream variable description, research design suggestions, code generation and report writing tasks.

[0084] Furthermore, the method of this invention focuses on retrieval enhancement and rule / lightweight model collaboration. It can achieve high-precision variable retrieval and terminology understanding enhancement in financial empirical scenarios without the need for high-cost parameter fine-tuning of large models or only low-cost adaptation. It has the advantages of flexible deployment, strong adaptability and low computational cost. Attached Figure Description

[0085] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0086] Figure 1 This is a schematic diagram of the overall system architecture provided in an embodiment of the present invention;

[0087] Figure 2 This document provides Chinese explanations and examples of key variables during the online retrieval process of this invention.

[0088] Figure 3 A flowchart illustrating a financial variable retrieval method based on semantic-statistical dual-space fusion and abbreviation polysemy disambiguation provided in an embodiment of the present invention;

[0089] Figure 4 A flowchart illustrating the offline data index construction process provided in this embodiment of the invention;

[0090] Figure 5 The graph shows the performance comparison results of the method of this invention with pure semantic retrieval, traditional RAG and loop-free dual-space methods in financial variable retrieval tasks. It includes Top-k retrieval accuracy, statistical consistency, abbreviation disambiguation accuracy, false retrieval rate and performance improvement brought by loop iteration.

[0091] Figure 6 This is a heatmap showing the statistical compatibility of candidate variables in an embodiment of the present invention. The example target query is 'Please provide the actual variable definition of China's quarterly GDP year-on-year growth'. The graph shows the degree of compatibility of different candidate variables in terms of statistical constraints such as frequency, unit, statistical scope, transformation method, time coverage and regional scope. The darker the color, the higher the compatibility.

[0092] Figure 7 The figures show the ablation experiment results of key modules in this embodiment of the invention. The bar chart represents the Top-1 retrieval accuracy, and the line chart represents the statistical consistency hit rate and abbreviation disambiguation accuracy. As can be seen from the figures, the complete model has the best overall performance, indicating that the statistical space, abbreviation disambiguation, closed-loop verification, and gating fusion modules can all effectively improve the retrieval effect of financial variables. Detailed Implementation

[0093] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0094] This invention addresses the challenges of fine-tuning large-scale models in the financial sector, including issues such as "semantic similarities of variables but incompatible statistical attributes," "misunderstandings due to multiple meanings of financial abbreviations," and "insufficient reliability caused by the disconnect between abbreviation disambiguation and the retrieval process." It proposes a unified technical solution: "forward disambiguation—semantic / statistical dual-space retrieval—fusion ranking—reverse closed-loop verification."

[0095] In one implementation, the system can be deployed on a server cluster, the backend of a research platform, an institutional intranet retrieval system, or a local workstation. The system internally includes: a text processing component, an abbreviation graph component, a vector retrieval component, a metadata retrieval component, a fusion sorting component, a closed-loop verification component, and an interpretation output component. Each component can be implemented by a software module or by a combination of software and dedicated hardware.

[0096] I. Data Preparation and Offline Data Indexing Construction Module

[0097] Step A1: Standardization of Variable Records

[0098] In one implementation, the system accesses and standardizes various variable data sources during the offline phase. These data sources include, but are not limited to, national statistical databases, international organization databases, commercial databases, data dictionaries built by the research team, and variable definition tables from literature.

[0099] For each variable record, the system maps the original fields to a unified metadata schema, forming a variable metadata record structure. In addition to text fields, the following fields are added to enhance retrieval capabilities: variable_category_tag (variable category tag, including price, labor), method_association_tag (method association tag, including DID, IV, FE, GMM, SFA, event study method), transform_recipe (variable construction description, such as "original value → seasonal adjustment → logarithm → first difference"), source_version (source version number), missing_rate (missing rate), and revision_risk_score (revision risk score).

[0100] In one implementation, the system performs unit standardization on the unit field, converting "hundred million yuan", "trillion yuan", "%", "percentage point", "index (base period = 100)" and "dollar (current price / constant price)" into standardized unit codes, and establishes a unit conversion graph to support the "convertibility" determination in unit compatibility calculation.

[0101] Step A2: Constructing the statistical signature

[0102] In one implementation, the system constructs a statistical signature based on metadata fields for rapid filtering and compatibility prediction. The statistical signature can be represented as:

[0103]

[0104] in Encoding the data source, For frequency coding, Unit coding,

[0105] For statistical caliber coding, For seasonal adjustment indicators, For discount indication, For logarithmic transformation identifier, It is the difference order. For time coverage interval, For regional encoding, Encode the sample range.

[0106] In one implementation, the system builds an inverted index and a hash bucket index for the statistical signatures to quickly perform candidate pre-screening upon receiving a search intent. For example, when a user specifies "quarterly, actual value, provincial panel," priority can be given to filtering those that meet the criteria. The set of candidate records.

[0107] Step A3: Building the Semantic Vector Index Library

[0108] In one implementation, the system segments variable description text, variable definition tables, variable usage paragraphs in literature, and database field annotation text into text fragments. It then performs text cleaning, terminology normalization, and synonym expansion, calculates vector representations, and builds a vector retrieval index. To enhance the expression of financial terminology, structured hints are appended to the text fragments before encoding, expanding the original text fragments to: "Variable name: [display_name]; Aliases: [aliases]; Caliber: [caliber]; Frequency: [frequency]; Description: [description_text]"

[0109] By employing the above methods, the semantic sparsity problem caused by relying solely on natural language fragments can be reduced, and the vector representation can incorporate statistical attribute information, thereby improving the quality of semantic retrieval recall.

[0110] Step A4: Construction of the Polysemous Abbreviation Graph

[0111] In one implementation, the system constructs an abbreviation polysemy graph. ,in For the set of abbreviation nodes, For a set of meaningful nodes, This represents the mapping edge between the abbreviation and its meaning.

[0112] For each abbreviation node Each of its associated meaning nodes At least the following parameters can be stored: full name of meaning, domain label vector. Trigger word set (For example, the trigger words for "fixed effects" are "panel data, regression, control variables"), and a set of method-related terms. (For example, the related terms for "instrumental variables" are "2SLS, endogeneity, causal identification").

[0113] II. Online Search Module

[0114] Step S1: User query preprocessing and feature encoding

[0115] The system receives user query text. And perform normalization processing to obtain standardized queries. The normalization process can be represented as follows: . The text normalization function should include at least case unification, punctuation normalization, full-width / half-width character conversion, and whitespace character cleanup. In one implementation, the system will... Analysis into multiple features:

[0116]

[0117] in Lexical features (word segmentation results, keywords, n-grams). Syntactic features (parts of speech, dependency relations, word roles). Statistical clue characteristics (frequency, unit, scope, time range, etc.) Domain-specific clues (macro / financial / econometric / causal, etc.) This is a set of abbreviations extracted from the text.

[0118] Step S2: Abbreviation polysemy disambiguation

[0119] For each abbreviation The system obtains a set of candidate meanings from the polysemous abbreviation graph. The system defines the meaning of each candidate. Calculate the context matching score, which is a weighted average of multi-source features:

[0120]

[0121] in Contribute scores to keywords. For the method word matching score, To record category preference matching scores, Scoring is given based on syntactic position or role matching. Scoring based on domain tag matching, Score the historical context preference. These are the weight parameters.

[0122] Thus, the system obtains each abbreviation. The corresponding set of candidate meanings and the set of context matching scores for each candidate meaning And use it as input for the subsequent construction of structured retrieval intent objects.

[0123] Step S3: Constructing Structured Search Intent Objects

[0124] The system obtains the standardized query from step S1 Query feature set and the candidate meaning set corresponding to each abbreviation obtained in step S2. Context matching score of candidate meaning Construct a structured search intent object :

[0125]

[0126] in, This represents the set of all candidate meanings corresponding to the abbreviations. This represents the set of context matching scores corresponding to each candidate meaning. The structured retrieval intent object... It is used to uniformly represent semantic retrieval information, abbreviation candidate meaning information, statistical constraint information, and domain context information, and serves as input for subsequent semantic vector space retrieval and statistical metadata space retrieval.

[0127] Step S4: Semantic Vector Space Retrieval and Semantic Retrieval Scoring

[0128] The system is based on the structured search intent object obtained in step S3. Construct semantic subqueries and perform retrieval in the semantic vector space. Specifically, for each abbreviation... Candidate meaning set The system provides each candidate meaning Constructing semantic subqueries And generate the corresponding vector representation. Each semantic subquery Perform nearest neighbor search on the semantic vector index library already built in the offline phase to retrieve the corresponding set of records. The union of the recall results of all subqueries is used to form a semantic candidate record set: .

[0129] For the semantic candidate record set Each record in Its text representation vector is The system determines the meaning of each candidate. The context matching score is used to calculate the semantic score of the record:

[0130]

[0131] in The similarity function can be cosine similarity, inner product, or normalized distance metric. The system scores semantic similarity. For candidate records Sort the results to obtain a set of semantic results. , in This represents the set of candidate records recalled using semantic vector indexing. Represents the set of semantic candidate results. Indicates candidate records The semantic retrieval score is used as input for subsequent fusion ranking and closed-loop verification.

[0132] Step S5: Statistical metadata spatial retrieval and statistical compatibility scoring

[0133] The system retrieves candidate records by analyzing their metadata based on the statistical constraints (frequency, unit, scope, time range) specified in the user query. The process involves calculating a statistical compatibility score for each candidate record. This process aims to ensure that the retrieved records match the user's needs in terms of statistical attributes.

[0134] To simplify the calculation, we employ a weighted compatibility scoring model that comprehensively scores multiple statistical attributes (frequency, unit, statistical scope). Each statistical attribute is calculated using a simple compatibility function and matched against the user's query conditions.

[0135] The formula for calculating the statistical compatibility score is as follows:

[0136]

[0137] in For frequency compatibility; Unit compatibility; To ensure compatibility of statistical standards; For time range compatibility; These correspond to the weights. The compatibility values ​​range from [0,1], with larger values ​​indicating a better match. Compatibility can be assigned according to preset rules: 1 for perfect match, an intermediate value for convertible match, and 0 for incompatibility.

[0138] In one implementation, the compatibility of each sub-item in the statistical compatibility score is calculated using a simplified method. Frequency compatibility. The values ​​are assigned in segments based on the degree of consistency between the expected query frequency and the candidate record frequency: 1 for perfect consistency, and 0 for low-loss conversion. High-loss conversion can be achieved Incompatible with 0, where 1> > >0.

[0139] Unit compatibility Assign values ​​segmented according to unit matching relationships: 1 is assigned if units are completely identical, which can be converted using a unit conversion chart. Incompatible with taking 0, where 0 < <1.

[0140] Statistical caliber compatibility Values ​​are assigned in segments based on the degree of consistency in statistical reporting: 1 for complete consistency, and 1 for partial consistency or compatibility after adjustment. For obvious conflicts, take 0, where 0 < <1.

[0141] Time coverage compatibility Based on the query time range With candidate record time interval Calculation of overlap ratio:

[0142]

[0143] When the candidate records completely cover the query time range When partially covered, the value is between 0 and 1; when completely uncovered, the value is 0.

[0144] Step S6: Semantic-Statistical Fusion Scoring and Ranking

[0145] The system processes candidate records semantic score Compatibility score with statistics To achieve the final score, the two components are combined. The system adopts a gated fusion approach, and the gate function is defined as follows:

[0146]

[0147] in For the Sigmoid function, To calculate the compatibility threshold, This refers to the gating steepness parameter. Candidate records. The final fusion score is calculated using the following formula:

[0148]

[0149] in For semantic weighting coefficients in scenarios with low statistical compatibility, The statistical compatibility score is weighted accordingly. When statistical compatibility is high ( When the statistical compatibility is low, the semantic score is hardly reduced in weight, and the final score mainly depends on semantic matching; when the statistical compatibility is low ( When ), semantic scores are calculated by coefficient. The weighting is reduced, while the statistical compatibility score still contributes some weight to the final score, thus effectively preventing candidate records that are "semantically highly related but statistically obviously incompatible" from occupying high positions in the ranking.

[0150] Step S7: Closed-loop consistency verification and secondary search

[0151] Based on the final ranking result of the candidate records obtained in step S6, the system performs a reverse consistency check on the candidate meanings of the abbreviations, and performs a second search when a preset trigger condition is met, thus forming a closed-loop iterative process of "forward disambiguation - dual-space search - reverse verification". Specifically, the system obtains the previous... The candidate records constitute the first A set of candidate records in, Indicates that the order is first. The j-th candidate record. For each abbreviation... The system was previously Extract a set of evidence related to the determination of the candidate meaning from each candidate record. In one embodiment, the evidence set includes at least one of the following: domain labels, method association labels, record category labels, statistical attribute patterns, unit information, frequency information, or caliber information. (Regarding abbreviations...) Each candidate meaning The system is based on the previous The evidence set of each candidate record is calculated. support Defined as:

[0152]

[0153] in, This is an evidence matching function used to determine candidate records. Collection of evidence Does it support candidate meaning? In one implementation, when Satisfying candidate meaning When at least one matching rule is met, ;otherwise, Thus, the system obtains the abbreviation. Each candidate meaning The set of support:

[0154]

[0155] And a reverse evidence ranking result is generated based on the support set. On the other hand, step S2 is already an abbreviation. Each candidate meaning gives a set of context matching scores. }, the system according to The forward disambiguation sorting result is generated based on... The system generates a reverse evidence ranking result and compares the ranking differences between the two. In one implementation, the system can directly compare based on... The obtained candidate meaning ranking and based on The obtained candidate meanings are ranked; when the difference between the rankings exceeds a preset threshold, a second search is triggered. Specifically, the system uses the context matching score set obtained in step S2. and based on the previous The set of support scores obtained from the evidence of each candidate record Calculate abbreviations Difference:

[0156]

[0157] in, For smoothing terms. When If a significant difference exists between the forward disambiguation result and the reverse evidence result, a second search is triggered; otherwise, if they are considered essentially the same, the current iteration ends. In one implementation, .

[0158] When the triggering condition is met, the system uses the support set. For candidate meaning set The sorting and corresponding context matching score set Update the structured search intent object and reconstruct it based on the updated results. Repeat steps S4 to S6.

[0159] In one embodiment, the termination condition of the closed-loop iteration includes at least one of the following: 1. The maximum number of iterations is reached; 2. The candidate meaning ranking remains unchanged in two adjacent iterations; 3. The ranking of the top K candidate records remains stable in two adjacent iterations; 4. The difference between the forward disambiguation ranking result and the reverse evidence ranking result does not exceed a preset threshold.

[0160] Step S8: Generate Explanatory Output

[0161] The system uses the final score of the candidate records obtained in step S6 as a basis. (v) and the difference obtained in step S7 Generate the final output. Details are as follows:

[0162] For candidate record set By final score Sort, select The bars serve as the final output set:

[0163]

[0164] For each output record ,supply:

[0165] Semantic score

[0166] Statistical compatibility score

[0167] Interpretive information based on candidate record evidence (e.g., matching fragments, statistical constraints).

[0168] For each abbreviation To determine the final interpretation and meaning:

[0169]

[0170] For each record or abbreviation, such as degree of difference Exceeding the preset threshold Conflict warnings or low-confidence warnings can be added.

[0171] III. Typical Application Examples

[0172] Example 1: Scenarios with highly relevant semantics but conflicting statistical definitions

[0173] User query: "Please provide the quarterly variable definition for China's real GDP year-on-year growth."

[0174] In the semantic vector space, the system may recall multiple candidate records (v) with similar semantics, including "nominal GDP year-on-year", "real GDP year-on-year", "GDP deflator", "GDP annualized quarter-on-quarter rate", etc.

[0175] Because the aforementioned candidates are highly similar semantically, relying solely on semantic scores is insufficient for effective differentiation. Therefore, the system further calculates a statistical compatibility score for each candidate record within the statistical metadata space. Specifically, it includes:

[0176] Frequency compatibility (quarterly)

[0177] Price caliber compatibility (actual value)

[0178] Compatibility of conversion methods (year-on-year)

[0179] By matching and weighting each statistical sub-item, the system reduces the weight of candidate records that do not meet the constraints (such as "nominal GDP year-on-year" and "GDP annualized quarter-on-quarter rate"), thereby improving the overall score of the record corresponding to "real GDP year-on-year (quarterly)". .

[0180] This embodiment demonstrates that by introducing statistical metadata space for constraint matching, the shortcomings of pure semantic retrieval in variable differentiation can be effectively compensated, thereby significantly improving the accuracy of retrieval results.

[0181] Example 2: Closed-loop verification and error correction scenario

[0182] User query: "What variables are commonly used when measuring the efficiency of FE?"

[0183] The system initially interprets "FE" as "fixed effect" based on prior rules, and constructs the search intent accordingly, performing semantic search and statistical matching.

[0184] In one implementation, the system performs a consistency analysis on the initial search results. When statistical labels related to "frontier efficiency," such as "random frontier analysis (SFA)," "efficiency value," and "inefficiency term," frequently appear in the high-ranking candidate results, the system determines that there is an inconsistency between the current abbreviation interpretation and the search results.

[0185] Based on the above detection results, the system updates the posterior probability of the abbreviation meaning, improves the support for "FE→Frontier Efficiency", and triggers the intent reconstruction and secondary retrieval process.

[0186] In the secondary retrieval, the system regenerates the query vector and performs retrieval sorting, ultimately outputting a set of variables related to efficiency measurement.

[0187] Through this closed-loop verification and feedback mechanism, the system can use the search results to correct the abbreviation disambiguation results in reverse, thereby achieving dynamic error correction and improving the overall search accuracy.

[0188] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and alterations to the above embodiments within the scope of the present invention.

[0189] IV. Methods for Verifying Technical Effects

[0190] In one implementation, the system can evaluate the technical effectiveness using the following metrics:

[0191] Top-K variable retrieval accuracy (Accuracy@K);

[0192] Statistical Consistency@K;

[0193] Abbreviation disambiguation accuracy;

[0194] The improvement in accuracy after closed-loop iteration;

[0195] False search rate (especially the proportion of semantically related but statistically incompatible false searches).

[0196] Compared to baseline systems that rely solely on semantic retrieval, this invention can significantly reduce false retrieval rates and improve statistical consistency hit rates in financial query scenarios with a high density of polysemous abbreviations and clear statistical requirements, thereby enhancing the usability and reliability of retrieval results in financial empirical research.

[0197] The above embodiments are merely preferred embodiments of the present invention, used to illustrate the technical principles and effects of the present invention, and are not intended to limit the scope of protection of the present invention. Equivalent substitutions, improvements, or modifications made by those skilled in the art based on the disclosure of the present invention should all fall within the scope of protection of the present invention.

Claims

1. A financial variable retrieval method based on semantic-statistical dual-space fusion and polysemous abbreviation closed-loop disambiguation verification, characterized in that, The process includes two stages: offline data index building and online retrieval. The offline data index building stage performs the following steps in sequence: Step A1: Standardize the financial variable records from statistical databases, commercial databases, research team data dictionaries, and literature variable definition tables, mapping the original fields to a unified metadata schema to form a variable metadata record structure that includes variable name, statistical scope, unit, frequency, and time coverage. At the same time, perform unit standardization on the unit field and create a unit conversion chart. Step A2: Construct a statistical signature based on the variable metadata record structure. ,in Encoding the data source, For frequency coding, Unit coding, For statistical caliber coding, For seasonal adjustment indicators, For discount indication, For logarithmic transformation identifier, It is the difference order. For time coverage interval, For regional encoding, Encode the sample range, then build an inverted index and a hash bucket index on the statistical signature, and construct a statistical metadata database; Step A3: Segment, clean, and standardize the terminology of the descriptive text of financial variables, data dictionary descriptions, database field annotation texts, and variable usage paragraphs in the literature. After adding structured prompts such as variable names, aliases, statistical definitions, and frequencies to the text fragments, perform vectorization processing to build a semantic vector index library. Step A4: Construct a polysemous abbreviation graph ,in A set of financial abbreviation nodes. For a set of meaningful nodes, For the mapping edge between abbreviations and their meanings, configure feature parameters such as the full meaning, domain label vector, trigger word set, and method association word set for each meaning node to establish a pre-mapping relationship between financial abbreviations and candidate meanings; The online search phase executes the following steps in sequence: Step S1: Receive the user's natural language query for financial variable retrieval, perform text normalization processing on the query text to unify capitalization, normalize punctuation, convert full-width / half-width characters, and clean up whitespace characters to obtain a standardized query. At the same time, extract lexical features, syntactic features, frequency of inclusion, unit, statistical caliber, time range statistical clue features, domain clue features, and a set of financial abbreviations from the query text. Step S2: For each financial abbreviation, obtain the corresponding set of candidate meanings from the abbreviation polysemy graph. Based on the query context, and combined with keyword contribution features, method word matching features, variable category preference matching features, syntactic position matching features, domain label matching features, and historical context preference features, calculate the context matching score for each candidate meaning. Step S3: Based on the standardized query, the extracted features, the candidate meaning set of financial abbreviations, and the context matching score of each candidate meaning, construct a structured retrieval intent object that uniformly represents semantic retrieval information, abbreviation candidate meaning information, statistical constraint information, and domain context information; Step S4: Based on the structured retrieval intent object, construct a semantic subquery for each candidate meaning of financial abbreviation and generate a corresponding vector representation. Perform nearest neighbor recall in the semantic vector index library for each semantic subquery. Take the union of the recall results to obtain a semantic candidate result set. Then, calculate the semantic retrieval score of each record in the semantic candidate result set by weighted average method. The weight of the weighted average is based on the conditional probability of the candidate meaning and the context similarity of the query. Step S5: Based on the structured retrieval intent object, perform structured matching and statistical signature pre-screening in the statistical metadata database, and calculate the statistical compatibility score of each record in the statistical candidate result set; Step S6: The semantic candidate result set and the statistical candidate result set are fused, scored, and ranked using a gating fusion method, through a gating function. Generate the gating coefficients, where Here, k is the sigmoid function, and k is the gate steepness parameter. To calculate the compatibility threshold, To adjust the parameters, use the formula. Calculate the final fusion score for each candidate record, where Score(v) is the final fusion score. For semantic retrieval scores, For semantic weighting coefficients in scenarios with low statistical compatibility, To calculate the compatibility score weights, the candidate records are sorted in descending order based on the final fusion score. Step S7: Perform reverse consistency verification on the meanings of financial abbreviations. Extract evidence sets including domain labels, method association labels, variable category labels, and statistical attribute patterns from the top K records of the candidate record ranking results. Calculate the support level for each candidate meaning of a financial abbreviation using a Top-K voting method, and then apply the formula... Calculate the difference D(a) between the context matching score of forward disambiguation and the support of reverse evidence, where ε is a smoothing term. When the difference D(a) exceeds the preset threshold δ, update the candidate meaning ranking and context matching score based on the support, reconstruct the structured retrieval intent object, and re-execute steps S4 to S6. The iteration termination condition is reaching the maximum number of iterations, the candidate meaning ranking remains unchanged in two adjacent rounds, the ranking of the first K candidate records in two adjacent rounds is stable, or the difference D(a) does not exceed the preset threshold δ. Step S8: Output the candidate results of financial variables sorted by the final fusion score, and output the explanation results of financial abbreviations, semantic matching keywords, statistical compatibility sub-item scores, matching evidence and confidence information corresponding to the candidate results. The explanation results of financial abbreviations are determined by the candidate meaning with the highest support.

2. The financial variable retrieval method according to claim 1, characterized in that, The unified metadata schema described in step A1 also includes at least one of the following: variable category label, method association label, variable transformation path description, source version number, missing rate, and revision risk score.

3. The variable retrieval method according to claim 1, characterized in that, In step S2, each candidate meaning The context matching score is calculated using the following formula: in To score for keywords, For method word matching score, Scoring the variable category preference. Scoring is given based on syntactic position or role matching. Scoring based on domain tag matching, Score the historical context preference. These are the weight parameters.

4. The financial variable retrieval method according to claim 1, characterized in that, In step S5, each compatibility level is assigned a value according to the following rules: completely consistent is assigned a value of 1, convertible / partially compatible is assigned a value between 0 and 1, and incompatible is assigned a value of 0.

5. The variable retrieval method according to claim 1, characterized in that, The statistical compatibility score mentioned in step S5 is expressed as follows: in For frequency compatibility; Unit compatibility; To ensure compatibility of statistical standards; For time range compatibility; For the corresponding weights.

6. The financial variable retrieval method according to claim 1, characterized in that, The Top-K voting calculation method described in step S7 is as follows: when the candidate record evidence satisfies at least one matching rule of the candidate meaning, the number of support votes is accumulated for the corresponding candidate meaning, and the support is the total number of support votes obtained by the candidate meaning.

Images