Method and device for constructing financial large model evaluation set

Abstract

The present disclosure provides a method and device for constructing a financial large model evaluation set, the method comprising: obtaining original evaluation materials derived from multiple financial business scenarios, and constructing a task type system for evaluating the capabilities of a financial large model and a business dimension system for defining the financial business scope associated with the capability evaluation; based on the task type system and the business dimension system, labeling each original evaluation sample in the original evaluation materials to obtain an intermediate evaluation set with task labels and business labels; quality control and difficulty assessment are performed on the intermediate evaluation set to obtain a target evaluation set with a multi-dimensional label structure. Thus, the evaluation set covers diversified financial business scenarios and capability task types, improves the systematicness, representativeness and interpretability of the evaluation set, and at the same time, provides a solid data foundation for subsequent on-demand screening of evaluation subsets of specific capability dimensions and precise evaluation of the comprehensive performance of the financial large model in specific business scenarios.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a method and apparatus for constructing a large financial model evaluation set. Background Technology

[0002] With the rapid development of artificial intelligence technology, the application of large language models in the financial field is constantly deepening, and they are now widely used in diverse business scenarios such as financial question answering, investment analysis, risk identification, compliance review, customer service, and robo-advisory. To ensure the reliable operation of these models in professional, rigorous, and high-risk financial environments, it is urgent to establish a scientific and systematic capability assessment mechanism. Among these, constructing high-quality evaluation datasets has become a crucial foundation for measuring the comprehensive capabilities of large financial models in areas such as professional knowledge understanding, logical reasoning, and complex decision-making. Summary of the Invention

[0003] This disclosure provides a method and apparatus for constructing a large financial model evaluation set, thereby at least partially addressing one of the technical problems in related technologies. The technical solution of this disclosure is as follows: According to a first aspect of the present disclosure, a method for constructing a financial large-scale model evaluation set is provided, comprising: acquiring original evaluation materials from multiple financial business scenarios, and constructing a task type system and a business dimension system for evaluating the capabilities of the financial large-scale model; wherein, the task type system is used to define the capability task types and task subtypes for evaluating the financial large-scale model; the business dimension system is used to define the financial business dimensions and business sub-dimensions covered by the evaluation; based on the task type system and the business dimension system, labeling each original evaluation sample in the original evaluation materials to obtain an intermediate evaluation set with task labels and business labels; and performing quality control and difficulty assessment on the intermediate evaluation set to obtain a target evaluation set with a multi-dimensional label structure; wherein, the multi-dimensional label structure includes task labels, business labels, and difficulty level labels.

[0004] According to a second aspect of the present disclosure, an apparatus for constructing a financial large-scale model evaluation set is provided, comprising: an acquisition module, configured to acquire original evaluation materials from multiple financial business scenarios, and construct a task type system and a business dimension system for evaluating the capabilities of the financial large-scale model; wherein, the task type system is used to define the capability task types and task subtypes for evaluating the financial large-scale model; and the business dimension system is used to define the financial business dimensions and business sub-dimensions covered by the evaluation. The annotation module is used to label each original evaluation sample in the original evaluation materials based on the task type system and the business dimension system to obtain an intermediate evaluation set with task tags and business tags; the processing module is used to perform quality control and difficulty assessment on the intermediate evaluation set to obtain a target evaluation set with a multi-dimensional tag structure; wherein, the multi-dimensional tag structure includes task tags, business tags and difficulty level tags.

[0005] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement the method for constructing a financial large model evaluation set as described in the first aspect of the present disclosure.

[0006] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided that, when instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform a method for constructing a financial large model evaluation set as described in the first aspect of the present disclosure.

[0007] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the method for constructing a financial large model evaluation set as described in the first aspect of the present disclosure.

[0008] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: In this technical solution, raw evaluation materials are obtained from multiple financial business scenarios, and a task type system and a business dimension system are constructed to achieve structured organization and semantic alignment of the evaluation content. Then, this dual system is used to label the original evaluation samples, forming an intermediate evaluation set with clear task and business labels. Finally, through quality control and difficulty assessment, a multi-dimensional label structure target evaluation set containing task labels, business labels, and difficulty level labels is obtained. This multi-dimensional label structure enables the evaluation set to not only cover diverse financial business scenarios and capability task types, but also to achieve fine-grained characterization in the three dimensions of task, business, and difficulty, thereby significantly improving the systematicity, representativeness, and interpretability of the evaluation set. Simultaneously, it provides a solid data foundation for subsequent on-demand selection of evaluation subsets for specific capability dimensions and accurate evaluation of the comprehensive performance of large financial models in specific business scenarios. Specifically, when generating the target evaluation set, multi-stage quality control and systematic difficulty assessment of the intermediate evaluation set are performed to form a target evaluation set containing three labels: task, business, and difficulty, which can support performance-based evaluation. The flexible combination and targeted assessment of capabilities, business functions, or difficulty enhances the comprehensiveness, accuracy, and practicality of financial model evaluation. Furthermore, capability task types are defined based on cognitive and reasoning behaviors in actual model applications, and further refined into sub-types within financial business scenarios, forming a capability-oriented task type system. Simultaneously, a business dimension system covering core functions is constructed based on financial business knowledge, further subdivided into business sub-dimensions. Thus, the task type system and business dimension system together provide a unified and scalable structured framework, enabling evaluations to accurately reflect the model's capabilities in real financial scenarios, thereby improving the comprehensiveness and relevance of the assessment.

[0009] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0010] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0011] Figure 1 This is a flowchart illustrating the method for constructing a financial large model evaluation set as shown in the first embodiment of this disclosure; Figure 2 This is a flowchart illustrating the method for constructing a financial large model evaluation set as shown in the second embodiment of this disclosure; Figure 3 This is a flowchart illustrating the method for constructing a financial large model evaluation set as shown in the third embodiment of this disclosure; Figure 4This is a flowchart illustrating the method for constructing a financial large model evaluation set as shown in the fourth embodiment of this disclosure; Figure 5 This is a flowchart illustrating the method for constructing a financial large model evaluation set as shown in the fifth embodiment of this disclosure; Figure 6 This is a flowchart illustrating the method for constructing a financial large model evaluation set as shown in the sixth embodiment of this disclosure; Figure 7 This is a flowchart illustrating the method for constructing a financial large model evaluation set as shown in the seventh embodiment of this disclosure; Figure 8 This is a schematic diagram illustrating the principle of the method for constructing a financial large model evaluation set as shown in the embodiments of this disclosure; Figure 9 This is a schematic diagram of the structure of the apparatus for constructing a financial large model evaluation set as shown in the eighth embodiment of this disclosure; Figure 10 This is a schematic diagram of the structure of an electronic device shown in an exemplary embodiment of the present disclosure. Detailed Implementation

[0012] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0013] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0014] It should be noted that the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in the technical solution disclosed herein are all carried out with the consent of the user, and all comply with the provisions of relevant laws and regulations, and do not violate public order and good morals.

[0015] In related technologies, evaluation datasets suffer from limited business coverage. Most datasets focus on general financial knowledge or fixed task types, lacking systematic modeling of the real-world business systems of financial institutions. In actual financial operations, companies typically encompass multiple business dimensions, including customer management, product management, risk control, compliance review, marketing operations, channel services, and transaction processing. Each business dimension may involve various task types such as information extraction, rule judgment, multi-hop reasoning, and process decision-making. Because existing evaluation datasets fail to effectively characterize the hierarchical structure of business dimensions and task types, they struggle to comprehensively reflect the differences in model capabilities across different business scenarios, thus weakening the representativeness and reference value of the evaluation results.

[0016] Furthermore, existing evaluation data generally lack a unified task classification system and business semantic structure, resulting in uneven distribution of evaluation samples in terms of task type, knowledge scope, and reasoning complexity. This makes it difficult to fully cover diverse actual business needs, thus affecting the comprehensiveness and accuracy of model capability assessment. On the other hand, most evaluation sets lack a systematic difficulty level division mechanism, making it impossible to accurately measure the model's performance under tasks of different complexity, especially in high-order capabilities such as multi-hop reasoning, business process understanding, and comprehensive analysis, where effective differentiation is difficult.

[0017] To address any of the above problems, this disclosure proposes a method and apparatus for constructing a large financial model evaluation set.

[0018] The following describes, with reference to the accompanying drawings, a method and apparatus for constructing a financial large model evaluation set according to embodiments of the present disclosure.

[0019] Figure 1 This is a flowchart illustrating the method for constructing a financial large model evaluation set as shown in the first embodiment of this disclosure.

[0020] It should be noted that, in this embodiment of the disclosure, the method for constructing a financial large model evaluation set is configured in a financial large model evaluation set construction device. This financial large model evaluation set construction device can be applied to an electronic device so that the electronic device can perform the function of constructing a financial large model evaluation set.

[0021] like Figure 1 As shown, the method for constructing this large financial model evaluation set includes: Step 110: Obtain original evaluation materials from multiple financial business scenarios, and construct a task type system for assessing the capabilities of the financial big model, as well as a business dimension system for defining the scope of financial business associated with the capability assessment.

[0022] The task type system is used to define the capability task types and task subtypes for evaluating the financial big data model; the business dimension system is used to define the financial business dimensions and business sub-dimensions covered by the evaluation.

[0023] To ensure the representativeness of the evaluation content, one possible approach is to collect diverse original evaluation materials from real financial business practices. These original evaluation materials can cover multiple financial business scenarios such as credit approval, investment advisory, compliance review, customer service, and risk management.

[0024] To achieve a systematic and refined assessment of the capabilities of large financial models, as a possible approach, a task type system for assessing the capabilities of large financial models and a business dimension system for defining the scope of financial business associated with the capability assessment are constructed.

[0025] The task type system, starting from the perspective of model cognition and reasoning behavior, defines the types of capability tasks that the evaluation needs to cover, and further subdivides them into specific task subtypes. This comprehensively portrays the capabilities that the model should possess at different operational levels. For example, capability task types may include, but are not limited to: financial knowledge Q&A, financial business understanding and analysis, financial fact reasoning and calculation, and financial compliance and risk control. Among these, financial knowledge Q&A may include, but is not limited to, subtypes such as business rule knowledge Q&A and financial industry standard knowledge Q&A; financial business understanding and analysis may include, but is not limited to, subtypes such as financial report interpretation, financial information extraction, financial text generation, and user intent recognition; and financial fact reasoning and calculation may include, but is not limited to, financial data inspection, numerical calculation and verification, and financial ratio analysis. Task sub-types include indicator trend inference and cash flow estimation; financial compliance and risk control may include, but are not limited to, tasks such as financial compliance review and financial risk control; business dimensions may include, but are not limited to, customer management, customer service, marketing management, and credit management. Customer management may include, but are not limited to, customer information management, customer management and maintenance, customer lifecycle management, customer value operation, customer evaluation and segmentation, customer contact management, and customer feedback management. Customer service may include, but are not limited to, customer service management, customer service provision, customer interaction and service, and dispute resolution. Marketing management may include, but are not limited to, marketing management and marketing activity management. Credit management may include, but is not limited to, corporate loans, personal loans, and general loans.

[0026] In some embodiments, in step 110, such as Figure 2 As shown, the task type system and business dimension system for evaluating the capabilities of the financial big data model can be constructed using the following steps: Step 1101: Based on the cognitive and reasoning behaviors required for the financial big data model in practical applications, define multiple capability task types.

[0027] To accurately characterize the capabilities of large financial models, one possible approach is to identify the basic cognitive processes and reasoning patterns that large financial models rely on when dealing with problems, such as information recognition, rule understanding, logical deduction, numerical calculation, or comprehensive judgment, based on the core intelligent activities that the models need to perform in real financial transactions. Furthermore, based on the cognitive and reasoning behaviors that large financial models need to complete in practical applications, multiple capability task types are abstracted.

[0028] Step 1102: For any capability task type, based on the business requirements of each financial business scenario under any capability task type, determine the task subtype under any capability task type.

[0029] To achieve a deep integration of task classification with financial business practices, and to further refine the task type of this capability based on the specific application needs in different financial business scenarios, for example, under the "rule understanding" capability task type, corresponding task subtypes can be defined according to the differences in rule expression and judgment logic in scenarios such as credit approval, compliance review, or product configuration.

[0030] Step 1103: Construct a task type system based on each capability task type and its corresponding task subtype.

[0031] Furthermore, all capability task types and their subordinate task subtypes are systematically integrated to form a clearly hierarchical and logically consistent task type system. It should be noted that this system can adopt a hierarchical structure, with the upper level representing capability task types and the lower level representing detailed task formats tailored to specific contexts. Thus, the task type system provides a standardized task classification framework for the entire evaluation set, enabling evaluation samples from different sources to be categorized, compared, and combined within the same semantic system, effectively improving the consistency, scalability, and maintainability of the evaluation content.

[0032] Step 1104: Based on business knowledge related to the financial field, determine multiple financial business dimensions, and further subdivide each financial business dimension based on the business knowledge to obtain financial business sub-dimensions under each financial business dimension.

[0033] To ensure comprehensive coverage of the actual business landscape of the financial industry, the evaluation content is based on business knowledge related to the financial sector. Key business areas to be covered are identified as high-level financial business dimensions, such as customer service, product management, risk management, compliance operations, and transaction processing. Furthermore, based on the functional divisions, regulatory requirements, or operational procedures within each dimension, the financial business dimensions are further subdivided into specific financial business sub-dimensions. The business knowledge related to the financial sector refers to a widely recognized and applied structured professional knowledge system developed through long-term practice in the financial industry, including regulatory norms, industry standards, internal functional divisions, and typical business processes.

[0034] Step 1105: Construct a business dimension system based on each financial business dimension and its corresponding sub-dimension.

[0035] Furthermore, the identified financial business dimensions and their subdivided business dimensions are organized to construct a structured business dimension system. This business dimension system is presented in a hierarchical manner, which not only reflects the overall architecture of the financial business but also retains the professional characteristics of each sub-field.

[0036] In summary, based on the cognitive and reasoning behaviors required by the model in practical applications, the ability task types are defined and further refined into task subtypes according to the specific needs of various financial business scenarios, thus forming a task type system with clear hierarchy and explicit capability orientation. Simultaneously, based on business knowledge in the financial field, financial business dimensions covering the core functions of the industry are identified, and each financial business dimension is further subdivided into specific business sub-dimensions according to its internal functional division, regulatory requirements, or operational logic. This constructs a complete and semantically rigorous business dimension system, providing a unified and scalable structured framework for the subsequent labeling, screening, and analysis of evaluation samples. The resulting evaluation foundation can accurately reflect the model's performance in diverse and real-world financial scenarios, thereby improving the comprehensiveness of the assessment.

[0037] Step 120: Based on the task type system and business dimension system, label each original evaluation sample in the original evaluation materials to obtain an intermediate evaluation set with task labels and business labels.

[0038] Furthermore, based on the task type system and business dimension system, each original evaluation sample in the original evaluation materials is labeled, transforming the originally unstructured raw materials into data units with clear semantic identifiers, forming a preliminary structured intermediate evaluation set. It should be noted that this intermediate evaluation set not only retains the semantic information of the original content but also embeds the dual dimensions of task capability and business context, providing a reliable basis for subsequent quality optimization and difficulty grading.

[0039] In some embodiments, such as Figure 3 As shown, step 120 may include the following steps: Step 1201: Based on the cognitive and reasoning behaviors involved in any original evaluation sample, determine the corresponding ability task type and task subtype from the task type system.

[0040] To achieve accurate classification of evaluation samples in terms of model capability dimensions, one possible approach is to conduct an in-depth analysis of the cognitive processes and reasoning logic that the content of each original evaluation sample relies on or examines, such as whether it is necessary to identify key entities, understand rule constraints, perform multi-step derivation, or complete numerical calculations. Based on the constructed task type system, the highest-level capability task type and its subordinate task subtypes that best meet the capability requirements of the sample are matched.

[0041] Step 1202: Based on the financial business scenario associated with any original evaluation sample, determine the corresponding financial business dimension and business sub-dimension from the business dimension system.

[0042] Meanwhile, in order to accurately position the evaluation samples in the business context, as a possible approach, we can identify the specific financial business background from which each original evaluation sample originates or is applicable, such as whether its content involves customer service processes, credit product rules, compliance review requirements, or transaction settlement operations, and match the original evaluation sample with the corresponding financial business dimensions and detailed business sub-dimensions by referring to the established business dimension system.

[0043] Step 1203: Using the corresponding capability task type and task subtype, label any original evaluation sample with task tags, and using the corresponding financial business dimension and business sub-dimension, label any original evaluation sample with business tags, so as to obtain an intermediate evaluation set with task tags and business tags.

[0044] Furthermore, task tags (including capability task types and task subtypes) and business tags (including financial business dimensions and business sub-dimensions) are assigned to the original evaluation sample, completing dual labeling. All samples processed in this way constitute the intermediate evaluation set, with each sample in the intermediate evaluation set carrying structured tags for both task and business dimensions. Thus, this intermediate evaluation set not only retains the semantic information of the original content but also embeds dual tags of capability orientation and business affiliation, providing a data foundation for subsequent quality control, difficulty assessment, and targeted evaluation, thereby enhancing the systematicness and flexibility of the evaluation system.

[0045] In summary, based on the cognitive and reasoning behaviors exhibited by the samples, corresponding capability task types and subtypes are accurately matched from the task type system to ensure that their classification in the model capability dimension has clear behavioral orientation. Secondly, combined with the actual financial business scenarios associated with the samples, corresponding financial business dimensions and sub-dimensions are determined from the business dimension system, making the positioning of the samples in the business context realistic, specific, and in line with industry practice. On this basis, through dual-label annotation, each original evaluation sample is transformed into a structured data unit carrying both task and business labels, thereby forming a semantically rich and well-organized intermediate evaluation set. This intermediate evaluation set not only overcomes the problem of traditional evaluation data lacking unified classification standards and business alignment, but also provides a highly consistent and interpretable foundation for subsequent quality screening, difficulty grading, and fine-grained capability assessment, improving the systematicness, coverage, and practicality of financial large-scale model evaluation.

[0046] Step 130: Perform quality control and difficulty assessment on the intermediate evaluation set to obtain the target evaluation set with a multi-dimensional label structure.

[0047] The multidimensional tag structure includes task tags, business tags, and difficulty level tags.

[0048] To enhance the professionalism and practicality of the evaluation set, one possible approach is to implement two key processing steps on the intermediate evaluation set: quality control and difficulty assessment. Quality control identifies and eliminates duplicate, ambiguous, or insufficiently informative samples, ensuring that the retained evaluation content possesses clear semantic expression and a well-defined evaluation objective. Difficulty assessment determines the complexity of samples through comprehensive analysis of knowledge depth, reasoning steps, and language structure, assigning corresponding difficulty level labels. After these processing steps, each evaluation sample ultimately carries a three-dimensional structured label: a task label reflecting the capability type, a business label reflecting the business affiliation, and a difficulty level label representing the complexity. Based on these three-dimensional structured labels, a multi-dimensional label structured target evaluation set is constructed. This target evaluation set not only boasts high-quality and comprehensive content but also supports flexible combination and filtering based on dimensions such as task, business, or difficulty, thus providing an accurate and interpretable evaluation foundation for the capability assessment of large-scale financial models.

[0049] In some embodiments, such as Figure 4 As shown, step 130 may include the following steps: Step 1301: Perform multi-stage quality control on the intermediate evaluation set to obtain a high-quality intermediate evaluation set.

[0050] To achieve a comprehensive improvement in the professionalism, accuracy, and usability of evaluation data, one possible approach is to implement systematic multi-stage quality control on the intermediate evaluation set that has been labeled with task tags and business tags. This multi-stage quality control may include multiple progressive steps to filter and optimize the intermediate evaluation set at multiple levels, ultimately retaining evaluation samples with high information density and high semantic clarity to form a high-quality intermediate evaluation set.

[0051] Step 1302: Evaluate the difficulty of the high-quality intermediate evaluation set to obtain the target evaluation set with a multi-dimensional label structure.

[0052] To achieve a detailed characterization of the complexity of the evaluation samples, one possible approach is to conduct a difficulty assessment on a high-quality intermediate evaluation set. This difficulty assessment comprehensively evaluates multiple key factors affecting cognitive load, including but not limited to the depth and breadth of required financial knowledge, the number of reasoning steps (such as whether multi-hop reasoning is involved), the complexity of rule application, the abstractness of language expression, and the degree of certainty of the answer. Based on this, each sample is assigned a structured difficulty level label (e.g., divided into low, medium, and high levels). After the assessment, each sample can simultaneously possess task labels, business labels, and difficulty level labels, forming a complete multi-dimensional label structure, ultimately forming the target evaluation set.

[0053] In summary, by implementing multi-stage quality control on the intermediate evaluation set, a high-quality intermediate evaluation set can be obtained. Further, a systematic difficulty assessment of the high-quality intermediate evaluation set is conducted to form a target evaluation set with a multi-dimensional label structure. This not only retains the original task and business labels but also adds a difficulty dimension, constructing a multi-dimensional label structure containing three attributes: task, business, and difficulty. This target evaluation set can support flexible combination and targeted evaluation based on capability dimensions, business scenarios, or complexity levels, improving the comprehensiveness, accuracy, and practicality of financial large-scale model evaluation.

[0054] In some embodiments, such as Figure 5 As shown, step 1301 may include the following steps: Step 13011: Perform a duplicate detection on the intermediate evaluation set and remove duplicate samples from the intermediate evaluation set.

[0055] To eliminate redundant content generated during data collection or annotation in the evaluation set, one possible approach is to perform duplicate detection on all samples in the intermediate evaluation set, identify duplicate or near-duplicate samples, and remove them from the set. Duplicate samples can include completely identical text samples as well as semantically highly similar approximate samples.

[0056] As an example, cosine similarity is used to calculate the similarity between two samples. Samples with a similarity higher than a set similarity threshold are considered semantically repetitive or highly similar and are thus removed as duplicate samples. For instance, each evaluation sample can be encoded into a fixed-dimensional semantic vector using a pre-trained language model (such as BERT), and then the cosine similarity between any two sample vectors can be calculated. If the similarity exceeds a preset threshold (e.g., 0.95), the two samples are considered to be highly semantically overlapping, even if their surface wording differs, and are considered redundant content that should be removed from the evaluation set. This method can effectively identify samples that are not literally repetitive but semantically equivalent, improving the accuracy and robustness of deduplication.

[0057] Step 13012: Identify and remove low-information samples from the intermediate evaluation set after removing duplicate samples to obtain a quality-cleaned intermediate evaluation set.

[0058] Then, the remaining samples are evaluated for information content, and low-information samples that lack substantial examination value are identified and eliminated. Low-information samples are usually characterized by overly simple content (such as only containing common sense questions and answers), vague question descriptions, lack of clear task objectives, or answers that cannot reflect the true capabilities of the model (such as answers that are "yes / no" without contextual support). Low-information samples can be identified through rule matching, keyword density analysis, question complexity indicators, or information entropy estimation based on language models.

[0059] Step 13013: The intermediate evaluation set after quality purification is scored, and evaluation samples with quality scores higher than the set scoring threshold are selected to obtain a high-quality intermediate evaluation set.

[0060] Furthermore, after deduplication and low-information filtering, a comprehensive quality score is applied to the purified evaluation set. Then, a reasonable scoring threshold is set, and only samples with scores higher than the threshold are retained to form the final high-quality intermediate evaluation set.

[0061] In summary, by performing repeatability detection and removing duplicate samples, the evaluation bias caused by data redundancy is effectively eliminated. Secondly, low-information samples are identified and removed from the deduplicated set. Finally, the evaluation set after double purification is comprehensively scored, and samples with scores higher than the preset threshold are retained. The resulting high-quality intermediate evaluation set has less data redundancy, higher information content, and more reliable overall quality, and can more accurately reflect the true ability of the model.

[0062] In some embodiments, such as Figure 6 As shown, step 13013 may include the following steps: Step 130131: For any second evaluation sample in the quality-cleaned intermediate evaluation set, determine the response format type of any second evaluation sample.

[0063] The response format types include any one of the following: single choice, multiple choice, judgment, and question answer. Single choice is used to evaluate the model's ability to discriminate under a single correct option; multiple choice is used to evaluate the model's ability to identify multi-label financial rules; judgment is used to evaluate the model's ability to identify the truth or falsehood of financial statements; and question answer is used to evaluate the model's ability to generate free text that conforms to the semantics of financial business.

[0064] As one possible implementation, any second evaluation sample in the quality-cleaned intermediate evaluation set is assigned a response format type based on the expected output format. The response format types include four basic categories: single-choice, multiple-choice, judgment, and question-and-answer. Single-choice is used to evaluate the model's ability to discriminate based on a single correct option; multiple-choice is used to evaluate the model's ability to identify multi-label financial rules; judgment is used to evaluate the model's ability to distinguish the truth or falsehood of financial statements; and question-and-answer is used to evaluate the model's ability to generate free text that conforms to the semantics of financial transactions.

[0065] Step 130132: Based on the scoring criteria of each dimension in the quality scoring dimension set corresponding to the response format type, perform a quality score on any second evaluation sample to obtain the quality score of each dimension.

[0066] Then, the set of quality scoring dimensions pre-associated with the response format type is invoked, and the sample is scored in multiple dimensions according to the specific scoring criteria of each dimension.

[0067] For example, the set of quality scoring dimensions associated with single-choice questions includes dimensions such as question stem clarity, distractor quality, and single-choice uniqueness. Among them, the scoring criteria for question stem clarity are used to evaluate whether the question stem is clearly expressed, whether the semantics are clear, and whether there are any ambiguities or non-standard expressions. The scoring criteria for distractor quality are used to evaluate whether incorrect options are reasonable and misleading, thereby avoiding obvious errors or invalid distractors. The scoring criteria for single-choice uniqueness are used to evaluate whether there is a unique optimal answer, avoiding situations where multiple options are correct or the answer is controversial. For example, the set of quality scoring dimensions associated with multiple choice may include dimensions such as: clarity of the question stem, quality of distractors, and independence of options. Among them, the scoring criteria for clarity of the question stem are used to evaluate whether the description of the question stem is clear and unambiguous; the scoring criteria for quality of distractors are used to evaluate whether the incorrect options have a reasonable distracting effect; and the independence of options is used to evaluate whether there is a semantic inclusion relationship or overlap between the options, so as to avoid multiple options expressing the same meaning. For example, the set of quality scoring dimensions associated with judgment may include dimensions such as clarity of the question stem, absolute degree of reasonableness, and knowledge validity. The scoring criteria for clarity of the question stem are used to assess whether the question stem description is clear and unambiguous. The absolute degree of reasonableness is used to assess whether the question stem uses excessive absolute expressions such as "always," "certainly," and "never" to avoid misleading judgments. The scoring criteria for knowledge validity are used to assess whether the question is based on financial professional knowledge, rather than answers that can be obtained by common sense guessing. For example, the set of quality scoring dimensions associated with question and answer may include dimensions such as question clarity, task decidability, knowledge relevance, and answer constraint. Among them, the scoring criteria for question clarity are used to assess whether the question is clearly expressed and whether there is any ambiguity or incomplete description; the scoring criteria for task decidability are used to assess whether the question has a clear answer objective, thus enabling objective evaluation; the scoring criteria for knowledge relevance are used to assess whether the question involves financial business knowledge or business semantics, rather than a generalized question; and the scoring criteria for answer constraint are used to assess whether the question has a reasonable range of answers, thus avoiding an overly broad answer space.

[0068] Step 130133: Determine the overall quality score of any second evaluation sample based on the quality scores of each dimension.

[0069] Furthermore, multiple dimensions of scores are integrated into a unified comprehensive quality score through methods such as weighted averaging, rule fusion, or machine learning models. The weighting can be pre-set based on business importance (e.g., "accuracy" has a higher weight than "language fluency"), or dynamically adjusted through expert calibration. This comprehensive score fully reflects the overall quality level of the sample in terms of knowledge, logic, expression, and task suitability, and serves as the basis for subsequent screening.

[0070] In summary, for each second evaluation sample in the quality-purified intermediate evaluation set, its response format type is accurately identified, thereby clarifying its task form and output specifications. Then, a set of quality scoring dimensions pre-associated with that response format type is invoked, and multi-dimensional independent scoring is performed according to the scoring standards defined for each dimension, ensuring a high degree of adaptation between the scoring system and task characteristics. Finally, based on the quality scores of each dimension, a comprehensive quality score is generated through weighted fusion or rule aggregation, forming a quantifiable and comparable quality indicator. Thus, this mechanism abandons the traditional, singular, and subjective quality judgment method, improves the objectivity, professionalism, and consistency of evaluation sample selection, and effectively guarantees the overall level of the final target evaluation set in terms of task effectiveness, business relevance, and evaluation reliability, providing effective data support for the capability evaluation of large financial models.

[0071] In some embodiments, such as Figure 7 As shown, step 1302 may include the following steps: Step 13021: For any first evaluation sample in the high-quality intermediate evaluation set, score any evaluation sample on at least one difficulty assessment dimension; wherein, the difficulty assessment dimension includes at least one of financial knowledge level, reasoning depth, language and structural complexity, distractor discrimination and information integration complexity.

[0072] As one possible approach, for each evaluation sample (referred to as the "first evaluation sample") in the high-quality intermediate evaluation set, a quantitative analysis of the difficulty level is performed on the first evaluation sample from multiple preset cognitive and task complexity dimensions.

[0073] As an example, the financial knowledge level reflects the depth and specialization of financial expertise required for the sample; reasoning depth measures the number of logical deduction steps required to complete the task or whether multi-hop reasoning is involved; language and structural complexity assesses the sentence length, terminology density, and degree of grammatical nesting in the question statement; distractor discrimination (applicable to multiple-choice questions) measures the semantic or logical similarity between incorrect and correct options, with higher discrimination indicating greater difficulty; and information integration complexity examines whether the model needs to integrate information from multiple sources or types (such as tables, texts, and rules) to arrive at an answer. The system can generate difficulty scores for samples on one or more of the above dimensions based on rules, expert annotations, or model-assisted methods.

[0074] Step 13022: Determine the overall difficulty level of any evaluation sample based on the scores on each difficulty assessment dimension.

[0075] Furthermore, through a pre-defined fusion strategy, such as weighted averaging, threshold determination, decision tree rules, or machine learning models, the multi-dimensional scores are aggregated into a unified comprehensive difficulty level. This level is typically divided into several discrete levels, such as "low," "medium," and "high."

[0076] Step 13023: Based on the overall difficulty level, add a difficulty level label to any evaluation sample to obtain a target evaluation set with a multi-dimensional label structure.

[0077] Then, the overall difficulty level of each evaluation sample is attached to the sample's metadata as a "difficulty level tag". At this point, each sample has a task tag (from the task type system), a business tag (from the business dimension system), and a difficulty level tag. The three together constitute a complete multi-dimensional tag structure. The set of all samples processed in this way is the target evaluation set.

[0078] To flexibly select one or more sub-evaluation sets from an existing target evaluation set with a multi-dimensional label structure, the following methods can be employed. For example, if the evaluation objective is to assess the model's rule understanding ability under the "compliance review" business scenario, samples with the business sub-dimensional "compliance review" and task type "rule understanding" can be selected based on the "business dimension" label. If the objective is to test the model's performance in high-difficulty scenarios, "high" difficulty samples can be selected based on the "difficulty level" label. Multiple label dimensions can also be combined for cross-selection, such as "risk management + multi-hop inference + high difficulty." Furthermore, the selected sub-evaluation sets can be used to conduct targeted capability assessments of the large financial model. Standard evaluation metrics (such as accuracy, F1 score, and generation quality score) can quantify the model's performance under specific task types, business dimensions, or difficulty levels. This mechanism allows the evaluation process to move beyond overall average performance and delve into specific capability quadrants, achieving fine-grained, scenario-based, and interpretable capability assessments to support model deployment decisions and continuous optimization in actual financial business.

[0079] In summary, by acquiring raw evaluation materials from multiple financial business scenarios and constructing a task type system and a business dimension system, we achieved structured organization and semantic alignment of the evaluation content. Furthermore, we used this dual system to label the original evaluation samples, forming an intermediate evaluation set with clear task and business labels. Then, through quality control and difficulty assessment, we finally obtained a target evaluation set with a multi-dimensional label structure containing task labels, business labels, and difficulty level labels. This multi-dimensional label structure enables the evaluation set to not only cover diverse financial business scenarios and capability task types, but also achieve fine-grained characterization across the three dimensions of task, business, and difficulty, thus significantly improving the systematicity, representativeness, and interpretability of the evaluation set. Simultaneously, it provides a solid data foundation for subsequent on-demand selection of evaluation subsets for specific capability dimensions and accurate evaluation of the comprehensive performance of large-scale financial models in specific business scenarios.

[0080] The method for constructing a financial large-scale model evaluation set in this disclosure obtains original evaluation materials from multiple financial business scenarios and systematically constructs a task type system and a business dimension system, achieving structured organization and semantic alignment of the evaluation content. Based on this, the original evaluation samples are uniformly labeled using this dual system to form an intermediate evaluation set with clear task and business labels. Then, through quality control and difficulty assessment, a multi-dimensional label structure target evaluation set containing task labels, business labels, and difficulty level labels is finally obtained. This multi-dimensional label structure enables the evaluation set to not only cover diverse financial business scenarios and capability task types, but also achieve fine-grained characterization in the three dimensions of task, business, and difficulty, thereby significantly improving the systematicity, representativeness, and interpretability of the evaluation set. Simultaneously, it provides a solid data foundation for subsequent on-demand selection of evaluation subsets for specific capability dimensions and accurate evaluation of the comprehensive performance of the financial large-scale model in specific business scenarios.

[0081] Based on any of the above embodiments, such as Figure 8 As shown, the method for constructing a large financial model evaluation set according to this disclosure embodiment may include the following steps: 1. Organize the financial capability dimensions and business segment dimensions that the evaluation set needs to cover, and establish a two-layer dimension system; 2. All collected evaluation data will undergo quality screening. The overall screening process is as follows: Original evaluation data - duplicate data filtering - low-information data screening - data quality scoring based on large models - high-quality evaluation dataset The specific implementation process is as follows: (1) Duplicate data filtering First, duplicate detection is performed on the collected evaluation data to reduce questions that are semantically highly similar or test completely identical knowledge points, thereby avoiding the influence of sample redundancy on the evaluation results. In this embodiment, a method based on semantic vector similarity is used for duplicate data identification, specifically including: 1) Convert the question text into a semantic vector representation; 2) Calculate the cosine similarity between any two questions; The cosine similarity between question A and question B is calculated as follows: ; (2) Filtering low-information data After filtering duplicate data, low-information samples in the evaluation data are identified and removed to avoid questions that are too simple or do not require professional knowledge to answer affecting the evaluation results.

[0082] Low-information samples mainly include the following: 1) Questions that can be answered using only common sense. Questions for which the correct answer can be determined without requiring financial expertise; 2) Objective questions with clearly obvious answers For example, questions with obviously wrong distractors or questions that differ too much from the correct answer.

[0083] 3) Questions that do not reflect business semantic features For example, it only involves general common sense judgments, without involving financial business knowledge or business logic; By filtering samples with low information content, the sensitivity of evaluation data to differences in model capabilities can be improved.

[0084] (3) Data quality scoring based on large models After filtering duplicate data and screening low-information samples, a large language model is used to quantify and score the quality of the evaluation data, thereby helping to identify potential low-quality samples.

[0085] Specifically, corresponding data quality scoring dimensions are constructed for different question types, and a large model is used to conduct structured analysis and scoring of question quality: 1) Scoring of single-choice questions Clarity is used to assess whether the question stem is clear, whether the meaning is unambiguous, and whether there are any issues with the expression or non-standard wording. Distractor quality is used to evaluate whether incorrect options are reasonable and misleading, thereby avoiding obvious errors or invalid distractors; Single-choice uniqueness is used to assess whether there is a single optimal answer, avoiding situations where multiple options are true at the same time or there is controversy over the answer; 2) Scoring dimensions for multiple-choice questions Clarity is used to assess whether the description in the question stem is clear and unambiguous. Distractor quality is used to evaluate whether incorrect options have a reasonable distracting effect. Option independence is used to assess whether there is a semantic inclusion or overlap relationship between the options, in order to avoid multiple options expressing the same meaning; 3) Scoring Dimensions for True / False Questions Clarity is used to assess whether the wording of the question stem is accurate and standardized. The degree of absolute reasonableness is used to assess whether the question stem overuses absolute expressions such as "always," "certainly," and "never," thereby avoiding misleading judgments. Knowledge validity is used to assess whether a question is based on financial expertise, rather than an answer that can be obtained through common sense. 4) Scoring Dimensions for Questions and Answers Question clarity is used to assess whether the question is clearly expressed and whether there is any ambiguity or incomplete description. Task determinability is used to assess whether a question has a clear answer objective, thus enabling objective evaluation. Knowledge relevance is used to assess whether a question involves financial business knowledge or business semantics, rather than a generalized question. The degree of constraint on the answer is used to assess whether the question has a reasonable range of answers, thereby avoiding overly broad answer options; 3. Use multi-dimensional quantitative assessment methods for difficulty. In this embodiment, in order to realize the hierarchical construction of evaluation data and enable the evaluation data to effectively distinguish large models of different ability levels, a quantitative evaluation method for the difficulty of evaluation data based on multi-dimensional features is proposed to perform structured modeling and quantitative scoring of the complexity of evaluation samples. The difficulty assessment method is based on the cognitive complexity and task characteristics in the financial field. It breaks down the key factors affecting the difficulty of the questions into multiple quantifiable dimensions and achieves automatic difficulty classification of the evaluation data through comprehensive scoring results. The specific implementation process is as follows: (1) Construction of difficulty assessment dimensions In this embodiment, a multi-dimensional difficulty assessment index system is constructed based on the main factors affecting the complexity of solving objective problems in the financial field. The index system includes the following five dimensions: 1) Level of financial knowledge It is used to measure the complexity and depth of financial knowledge involved in the question.

[0086] The knowledge level reflects the degree of financial expertise required by the question, with different levels corresponding to different levels of cognitive complexity.

[0087] For example: lower-level questions mainly involve identifying basic financial concepts or judging simple definitions; intermediate-level questions involve understanding and applying basic knowledge; higher-level questions involve comprehensive analysis of multiple knowledge points or integration of cross-disciplinary knowledge. This dimension can be used to depict the differences in the level of expertise required for the evaluation data. 2) Depth of Reasoning Used to measure the number of logical reasoning steps required to solve a problem and the complexity of the reasoning; Reasoning depth reflects the logical reasoning ability required by a model when solving a problem, including: whether conditional judgments are required, whether multi-step reasoning is required, and whether there is an implicit reasoning process. For example, simple questions can be answered without reasoning; medium-difficulty questions require one or two steps of logical deduction; and high-difficulty questions require multiple steps of reasoning or implicit logical deduction by combining multiple conditions.

[0088] This dimension can be used to characterize the differentiation of evaluation data in terms of logical reasoning ability.

[0089] 3) Language and structural complexity This is used to measure the comprehension difficulty of the question text at the language expression level, including sentence complexity, information density, and semantic structural complexity. Language complexity is mainly reflected in: sentence length, number of nested clauses, complexity of conditional expression, and information density; For example, simple questions are usually expressed in a single sentence; complex questions may contain multiple conditions or nested structures; highly complex questions may have expressions with high information density or complex semantic relationships. This dimension is used to characterize the impact of language comprehension difficulty on model performance.

[0090] 4) Discrimination of interference items It is used to measure the similarity between incorrect options and correct answers in objective questions, thereby reflecting the question's ability to differentiate between them.

[0091] The higher the discrimination of the distractors, the greater the semantic or knowledge similarity between the incorrect options and the correct answer, thus increasing the difficulty of distinguishing between them.

[0092] For example, the distractors in low-discrimination questions are usually significantly different from the correct answer; the distractors in medium-discrimination questions have some superficial similarities; and the distractors in high-discrimination questions are usually constructed based on common misunderstandings or points of confusion.

[0093] 5) Information integration complexity Information integration complexity is used to measure whether a comprehensive analysis combining the information in the question stem and professional knowledge is required when answering a question. The complexity of information integration is mainly reflected in: whether it is necessary to rely on the information in the question stem, whether it is necessary to filter key information, whether it is necessary to integrate multiple information sources, and whether it is necessary to combine professional knowledge for reasoning; For example: low-complexity questions can be answered without relying on the provided materials; medium-complexity questions require extracting key conditions from the question stem; high-complexity questions require integrating multiple information points and combining them with professional knowledge for comprehensive analysis.

[0094] This dimension reflects the information processing ability required by the question.

[0095] (2) Multidimensional scoring method In this embodiment, each evaluation question is scored on the above five dimensions, with each dimension scoring from 1 to 5 points, where 1 point represents the lowest level of complexity and 5 points represent the highest level of complexity. The scoring process employs a structured analysis approach, which involves first analyzing the characteristics of the questions across various dimensions, and then determining the corresponding scores based on pre-set scoring criteria.

[0096] By scoring multiple dimensions separately, the bias caused by a single scoring standard can be avoided, thereby improving the stability and reliability of the difficulty assessment results.

[0097] (3) Calculation method of comprehensive difficulty After obtaining the scores for each dimension, the scores from multiple dimensions are combined and calculated to obtain the overall difficulty score of the question.

[0098] In this embodiment, the overall difficulty score can be calculated using a weighted summation or average method: Overall difficulty score = Functional mapping result of scores in each dimension Each dimension can be set with the same weight, or different weights can be set according to the evaluation requirements.

[0099] For example, reasoning depth can be given a higher weight, and financial knowledge level can be given a higher weight. Based on the comprehensive scoring results, the assessment data can be divided into multiple difficulty levels, for example: Easy, Medium, Hard This results in a hierarchical evaluation dataset.

[0100] (4) Application of difficulty classification results The above difficulty assessment method can achieve the following: 1) Control the distribution ratio of evaluation data across different difficulty levels; 2) Improve the ability of evaluation data to distinguish differences in model capabilities; 3) Supports assessment needs for models with different ability levels; 4) Supports the construction of hierarchical benchmark datasets; The evaluation dataset constructed using this method can more accurately reflect the performance of the model in tasks of different complexities, thereby enhancing the reference value of the evaluation results.

[0101] Corresponding to the method for constructing a financial large model evaluation set provided in the above embodiments, this disclosure also provides an apparatus for constructing a financial large model evaluation set. Since the apparatus for constructing a financial large model evaluation set provided in this disclosure corresponds to the method for constructing a financial large model evaluation set provided in the above embodiments, the implementation method of constructing a financial large model evaluation set is also applicable to the apparatus for constructing a financial large model evaluation set provided in this disclosure, and will not be described in detail in this disclosure.

[0102] Figure 9This is a schematic diagram of the structure of the apparatus for constructing a financial large model evaluation set as shown in the eighth embodiment of this disclosure.

[0103] like Figure 9 As shown, the construction device 900 for the financial large model evaluation set includes: an acquisition module 910, an annotation module 920, and a processing module 930.

[0104] The acquisition module 910 is used to acquire original evaluation materials from multiple financial business scenarios and construct a task type system for evaluating the capabilities of the financial large-scale model, as well as a business dimension system for defining the scope of financial businesses associated with the capability evaluation. The task type system is used to define the capability task types and task subtypes for evaluating the financial large-scale model; the business dimension system is used to define the financial business dimensions and business sub-dimensions covered by the evaluation. The annotation module 920 is used to label each original evaluation sample in the original evaluation materials based on the task type system and the business dimension system to obtain an intermediate evaluation set with task labels and business labels. The processing module 930 is used to perform quality control and difficulty assessment on the intermediate evaluation set to obtain a target evaluation set with a multi-dimensional label structure. The multi-dimensional label structure includes task labels, business labels, and difficulty level labels.

[0105] As one possible implementation, the processing module 930 is used to perform multi-stage quality control on the intermediate evaluation set to obtain a high-quality intermediate evaluation set; and to evaluate the difficulty of the high-quality intermediate evaluation set to obtain a target evaluation set with a multi-dimensional label structure.

[0106] As one possible implementation, the processing module 930 is used to perform duplicate detection on the intermediate evaluation set and remove duplicate samples from the intermediate evaluation set; identify and remove low-information samples from the intermediate evaluation set after removing duplicate samples to obtain a quality-cleaned intermediate evaluation set; score the quality of the quality-cleaned intermediate evaluation set and select evaluation samples with quality scores higher than a set score threshold to obtain a high-quality intermediate evaluation set.

[0107] As one possible implementation, the processing module 930 is used to score any first evaluation sample in the high-quality intermediate evaluation set on at least one difficulty assessment dimension; wherein, the difficulty assessment dimension includes at least one of financial knowledge level, reasoning depth, language and structural complexity, distractor discrimination degree and information integration complexity; based on the scores on each difficulty assessment dimension, the comprehensive difficulty level of any evaluation sample is determined; according to the comprehensive difficulty level, a difficulty level label is added to any evaluation sample to obtain a target evaluation set with a multi-dimensional label structure.

[0108] As one possible implementation, the processing module 930 is used to determine the response format type of any second evaluation sample in the intermediate evaluation set that has been cleaned up; to perform quality scoring on any second evaluation sample according to the scoring criteria of each dimension in the quality scoring dimension set corresponding to the response format type, so as to obtain the quality score of each dimension; and to determine the comprehensive quality score of any second evaluation sample based on the quality scores of each dimension.

[0109] As one possible implementation, the response format type includes any of the following types: single choice, used to evaluate the model's ability to discriminate under a single correct option; multiple choice, used to evaluate the model's ability to identify multi-label financial rules; judgment, used to evaluate the model's ability to identify the truth or falsehood of financial statements; question answering, used to evaluate the model's ability to generate free text that conforms to the semantics of financial business.

[0110] As one possible implementation, module 910 is used to define multiple capability task types based on the cognitive and reasoning behaviors required by the financial big model in practical applications; for any capability task type, based on the business requirements of various financial business scenarios under any capability task type, to determine the task subtypes under any capability task type; based on each capability task type and its corresponding task subtypes, to construct a task type system; based on business knowledge related to the financial field, to determine multiple financial business dimensions, and based on business knowledge, to further subdivide each financial business dimension to obtain the financial business sub-dimensions under each financial business dimension; based on each financial business dimension and its corresponding financial business sub-dimensions, to construct a business dimension system.

[0111] As one possible implementation, the annotation module 920 is used to determine the corresponding capability task type and task subtype from the task type system based on the cognitive and reasoning behaviors involved in any original evaluation sample; to determine the corresponding financial business dimension and business sub-dimension from the business dimension system based on the financial business scenario associated with any original evaluation sample; to annotate any original evaluation sample with task tags using the corresponding capability task type and task subtype, and to annotate any original evaluation sample with business tags using the corresponding financial business dimension and business sub-dimension, so as to obtain an intermediate evaluation set with task tags and business tags.

[0112] As one possible implementation, the financial large model evaluation set construction device 900 also includes an evaluation module.

[0113] The evaluation module is used to select at least one sub-evaluation set from the target evaluation set according to the set evaluation objectives. The sub-evaluation set is obtained by filtering based on at least one label dimension in the multi-dimensional label structure. The sub-evaluation set is used to evaluate the capabilities of the financial big model to obtain the performance indicators of the financial big model under the corresponding task type, business dimension or difficulty level.

[0114] The apparatus for constructing a financial large-scale model evaluation set in this disclosure acquires original evaluation materials from multiple financial business scenarios and constructs a task type system and a business dimension system, achieving structured organization and semantic alignment of the evaluation content. Then, using this dual system, the original evaluation samples are labeled to form an intermediate evaluation set with clear task and business labels. Finally, through quality control and difficulty assessment, a multi-dimensional label structure target evaluation set containing task labels, business labels, and difficulty level labels is obtained. This multi-dimensional label structure enables the evaluation set to not only cover diverse financial business scenarios and capability task types, but also achieve fine-grained characterization in the three dimensions of task, business, and difficulty, thereby significantly improving the systematicity, representativeness, and interpretability of the evaluation set. Simultaneously, it provides a solid data foundation for subsequent on-demand selection of evaluation subsets for specific capability dimensions and accurate evaluation of the comprehensive performance of the financial large-scale model in specific business scenarios.

[0115] In an exemplary embodiment, an electronic device is also proposed.

[0116] The electronic devices include: processor; Memory used to store processor-executable instructions; The processor is configured to execute instructions to implement the method for constructing a financial large model evaluation set as proposed in any of the foregoing embodiments.

[0117] As an example, Figure 10 This is a schematic diagram of the structure of an electronic device 1000 as shown in an exemplary embodiment of this disclosure, as follows: Figure 10 As shown, the aforementioned electronic device 1000 may further include: The memory 1010 and processor 1020 are connected by a bus 1030, which connects different components (including the memory 1010 and the processor 1020). The memory 1010 stores a computer program, and when the processor 1020 executes the program, it implements the method for constructing a financial large model evaluation set as described in this embodiment of the disclosure.

[0118] Bus 1030 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0119] Electronic device 1000 typically includes a variety of electronic device readable media. These media can be any available media that can be accessed by electronic device 1000, including volatile and non-volatile media, removable and non-removable media.

[0120] The memory 1010 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1040 and / or cache memory 1050. The electronic device 1000 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 1060 may be used to read and write non-removable, non-volatile magnetic media (…). Figure 10 Not shown; usually referred to as a "hard drive"). Although Figure 10 As not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 1030 via one or more data media interfaces. Memory 1010 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this disclosure.

[0121] A program / utility 1080 having a set (at least one) of program modules 1070 may be stored in, for example, memory 1010. Such program modules 1070 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 1070 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0122] Electronic device 1000 can also communicate with one or more external devices 1090 (e.g., keyboard, pointing device, display 1091, etc.), one or more devices that enable a user to interact with electronic device 1000, and / or any device that enables electronic device 1000 to communicate with one or more other computing devices (e.g., network card, modem, etc.). This communication can be performed via input / output (I / O) interface 1092. Furthermore, electronic device 1000 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1093. As shown, network adapter 1093 communicates with other modules of electronic device 1000 via bus 1030. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0123] The processor 1020 performs various functional applications and data processing by running programs stored in the memory 1010.

[0124] It should be noted that the implementation process and technical principles of the electronic device in this embodiment are explained in the foregoing description of the method for constructing the financial large model evaluation set in this disclosure embodiment, and will not be repeated here.

[0125] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory including instructions, which can be executed by a processor of an electronic device to complete the method for constructing the financial large model evaluation set proposed in any of the above embodiments. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0126] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions, characterized in that, when the computer program / instructions are executed by a processor, they implement the method for constructing the financial large model evaluation set proposed in any of the above embodiments.

[0127] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0128] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for constructing a large financial model evaluation set, characterized in that, include: The system acquires original evaluation materials from multiple financial business scenarios and constructs a task type system for evaluating the capabilities of the financial big model, as well as a business dimension system for defining the scope of financial businesses associated with the capability evaluation. The task type system is used to define the capability task types and task subtypes for evaluating the financial big model, and the business dimension system is used to define the financial business dimensions and business sub-dimensions covered by the evaluation. Based on the task type system and the business dimension system, each original evaluation sample in the original evaluation material is labeled to obtain an intermediate evaluation set with task labels and business labels; The intermediate evaluation set is subjected to quality control and difficulty assessment to obtain a target evaluation set with a multi-dimensional label structure; wherein, the multi-dimensional label structure includes task labels, business labels and difficulty level labels.

2. The method according to claim 1, characterized in that, The process of performing quality control and difficulty assessment on the intermediate evaluation set to obtain a target evaluation set with a multi-dimensional label structure includes: Multi-stage quality control is performed on the intermediate evaluation set to obtain a high-quality intermediate evaluation set; The difficulty level of the high-quality intermediate evaluation set is evaluated to obtain the target evaluation set with a multi-dimensional label structure.

3. The method according to claim 2, characterized in that, The multi-stage quality control of the intermediate evaluation set to obtain a high-quality intermediate evaluation set includes: The intermediate evaluation set is subjected to duplicate detection, and duplicate samples in the intermediate evaluation set are removed. Low-information samples in the intermediate evaluation set after removing duplicate samples are identified and removed to obtain a quality-cleaned intermediate evaluation set. The intermediate evaluation set that has undergone quality purification is scored, and evaluation samples with quality scores higher than a set scoring threshold are selected to obtain a high-quality intermediate evaluation set.

4. The method according to claim 2, characterized in that, The process of evaluating the difficulty of the high-quality intermediate evaluation set to obtain a target evaluation set with a multi-dimensional label structure includes: For any first evaluation sample in the high-quality intermediate evaluation set, the evaluation sample is scored on at least one difficulty assessment dimension; wherein, the difficulty assessment dimension includes at least one of financial knowledge level, reasoning depth, language and structural complexity, distractor discrimination and information integration complexity. Based on the scores on each of the aforementioned difficulty assessment dimensions, the overall difficulty level of any given evaluation sample is determined. Based on the overall difficulty level, a difficulty level label is added to any of the evaluation samples to obtain a target evaluation set with a multi-dimensional label structure.

5. The method according to claim 3, characterized in that, The quality scoring of the purified intermediate evaluation set includes: For any second evaluation sample in the quality-cleaned intermediate evaluation set, determine the response format type of the any second evaluation sample; Based on the scoring criteria of each dimension in the quality scoring dimension set corresponding to the response format type, a quality score is performed on any second evaluation sample to obtain the quality score of each dimension. Based on the quality scores of each dimension, a comprehensive quality score is determined for any second evaluation sample.

6. The method according to claim 5, characterized in that, The response format type includes any of the following types: Single choice is used to evaluate the model's ability to distinguish between the only correct option; Multiple options are used to evaluate the model's ability to identify multi-label financial rules; Judgment is used to assess the model's ability to identify the truth or falsehood of financial statements; Question answering is used to evaluate the model's ability to generate free text that conforms to the semantics of financial business.

7. The method according to claim 1, characterized in that, The system of task types and business dimensions for constructing the capability assessment of the financial big data model includes: Based on the cognitive and reasoning behaviors required for the financial big model in practical applications, multiple capability task types are defined. For any capability task type, based on the business requirements of each financial business scenario under that capability task type, determine the task subtype under that capability task type; Based on each of the aforementioned capability task types and their corresponding task subtypes, the task type system is constructed; Based on business knowledge related to the financial field, multiple financial business dimensions are identified, and each of the financial business dimensions is further subdivided based on the business knowledge to obtain financial business sub-dimensions under each of the financial business dimensions. The business dimension system is constructed based on each of the aforementioned financial business dimensions and their corresponding financial business sub-dimensions.

8. The method according to claim 1, characterized in that, Based on the task type system and the business dimension system, each original evaluation sample in the original evaluation materials is labeled to obtain an intermediate evaluation set with task labels and business labels, including: Based on the cognitive and reasoning behaviors involved in any original evaluation sample, determine the corresponding ability task type and task subtype from the task type system; Based on the financial business scenario associated with any of the original evaluation samples, determine the corresponding financial business dimension and business sub-dimension from the business dimension system; Using the corresponding capability task type and task subtype, task tags are applied to any original evaluation sample, and using the corresponding financial business dimension and business sub-dimension, business tags are applied to any original evaluation sample to obtain an intermediate evaluation set with task tags and business tags.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: Based on the set evaluation objectives, at least one sub-evaluation set is selected from the target evaluation set; wherein, the sub-evaluation set is obtained by filtering based on at least one label dimension in the multi-dimensional label structure; The financial big data model is evaluated using the sub-evaluation set to obtain its performance indicators under corresponding task types, business dimensions, or difficulty levels.

10. A device for constructing a large financial model evaluation set, characterized in that, include: The acquisition module is used to acquire original evaluation materials from multiple financial business scenarios and construct a task type system for evaluating the capabilities of the financial big model, as well as a business dimension system for defining the scope of financial businesses associated with the capability evaluation; wherein, the task type system is used to define the capability task types and task subtypes for evaluating the financial big model; the business dimension system is used to define the financial business dimensions and business sub-dimensions covered by the evaluation. The annotation module is used to annotate each original evaluation sample in the original evaluation material based on the task type system and the business dimension system, so as to obtain an intermediate evaluation set with task tags and business tags; The processing module is used to perform quality control and difficulty assessment on the intermediate evaluation set to obtain a target evaluation set with a multi-dimensional label structure; wherein, the multi-dimensional label structure includes task labels, business labels and difficulty level labels.

Images