Financial text large model performance evaluation method and device

Abstract

The invention discloses a financial text large model performance evaluation method and device. In the scheme, evaluation samples in a target evaluation sample library are input into a target model, and an output result and runtime performance data of the target model are obtained. An inference performance score is generated for the target model based on the runtime performance data. And performing basic semantic quality evaluation and field specialty evaluation on the target model based on the output result to generate a service quality score. And according to the application service scene type of the target model, distributing a weight ratio for the reasoning performance score and the service quality score. And generating a total performance score of the target model based on the weight ratio. According to the technical scheme, the model is evaluated from multiple dimensions including field specialty and the like according to special requirements in the financial field, and the performance of the large financial text model is comprehensively reflected. And the weight is dynamically adjusted according to the service scene type, so that the evaluation result better meets the actual service requirement. And comprehensive and accurate evaluation of the performance of the financial text large model is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence evaluation, in particular to a financial text large model performance evaluation method and device. BACKGROUND

[0002] With the deepening of the digital transformation of the financial industry, text generation large models are increasingly widely used in core business scenarios such as compliance report generation, real-time customer service response, and risk event analysis. Existing large model evaluation methods gradually expose limitations.

[0003] First, the general evaluation framework is difficult to adapt to the professional needs of financial businesses, there is a lack of depth evaluation dimensions in vertical fields, and the fixed evaluation framework is difficult to balance the differentiated requirements of different businesses for model performance. At the same time, for key indicators specific to financial scenarios, such as compliance and data security, existing solutions lack systematic evaluation methods, making it impossible to comprehensively and accurately measure the overall performance of models in actual financial businesses. These problems make it difficult for financial institutions to select, optimize, and risk control models, and there is an urgent need for a more targeted text generation large model evaluation method to promote the reliable application of large models in financial scenarios. SUMMARY

[0004] Based on the above problems, the present application provides a financial text large model performance evaluation method and device, aiming to comprehensively and accurately evaluate the performance of financial text large models and improve the reliability of text large models in financial scenarios.

[0005] The embodiments of the present application disclose the following technical solutions:

[0006] The first aspect of the present application provides a financial text large model performance evaluation method, which comprises:

[0007] inputting an evaluation sample in a target evaluation sample library into a target model to obtain an output result and runtime performance data of the target model; the target model is a text generation large model to be evaluated; the target evaluation sample library contains financial business text generation tasks and standard templates;

[0008] performing inference performance evaluation on the target model based on the runtime performance data to generate an inference performance score;

[0009] performing business quality evaluation on the target model based on the output result to generate a business quality score; the business quality evaluation includes basic semantic quality evaluation and domain professional evaluation; the domain professional evaluation includes structured output compliance evaluation, reply similarity evaluation of similar questions, business logic rationality evaluation, and output text compliance evaluation;

[0010] According to an application business scene type of the target model, a weight proportion of the inference performance score and the service quality score is determined;

[0011] Based on the weight proportion, the weighted service quality score and the inference performance score are fused to generate a performance total score of the target model.

[0012] In an optional implementation, the structured output compliance evaluation includes:

[0013] The structural features of the standard template and the output result are parsed;

[0014] The structural similarity score and the format standardization score of the output result and the standard template are calculated;

[0015] A structured output compliance score is generated based on the structural similarity score and the format standardization score.

[0016] In an optional implementation, the reply similarity evaluation of the similar question includes:

[0017] A reply similarity score under the similar question is calculated through semantic vector mapping.

[0018] In an optional implementation, the business logic rationality evaluation includes:

[0019] The output result of the target model is disassembled into a logic unit sequence;

[0020] The logic unit sequence is matched with a standard logic path in a pre-constructed financial business knowledge graph to generate a business logic rationality score.

[0021] In an optional implementation, the output text compliance evaluation includes:

[0022] Based on a preset financial compliance rule library, a rule violation item in the output result is scanned;

[0023] According to a rule violation type, a deduction operation is performed to generate an output text compliance score.

[0024] In an optional implementation, the method further includes:

[0025] When the scanned result satisfies any one of the following conditions, the evaluation is terminated:

[0026] There is a preset high-risk rule violation keyword;

[0027] Risk prompt content is completely missing;

[0028] Cumulative deduction reaches a preset threshold.

[0029] In an optional implementation, the service quality evaluation based on the output result on the target model generates a service quality score, including:

[0030] An evaluation index judgment matrix is constructed for each evaluation index in the service quality evaluation by using an analytic hierarchy process;

[0031] The weight of each evaluation index is calculated according to the judgment matrix;

[0032] The score of each evaluation index is fused based on the weight of each evaluation index to generate a service quality score.

[0033] In an optional implementation, the runtime performance data includes a first character response time, computing resource consumption data, and maximum concurrency; the inference performance evaluation based on the runtime performance data on the target model generates an inference performance score, including:

[0034] A timeliness score is generated based on the first character response time;

[0035] A resource utilization rate score is generated based on the computing resource consumption data;

[0036] A concurrency capability score is generated based on the maximum concurrency;

[0037] According to the application scenario type of the target model, the inference performance weight proportion of the timeliness score, the resource utilization rate score, and the concurrency capability score is determined;

[0038] The timeliness score, the resource utilization rate score, and the concurrency capability score are fused based on the inference performance weight proportion to generate an inference performance score.

[0039] In an optional implementation, the basic semantic quality evaluation includes:

[0040] The output result is subjected to a keyword coverage evaluation to generate a keyword coverage score;

[0041] The output result is subjected to a semantic similarity evaluation to generate a semantic similarity score.

[0042] The second aspect of the application provides a financial text large model performance evaluation device, which comprises:

[0043] An evaluation data acquisition module is configured to input an evaluation sample in a target evaluation sample library into a target model, and acquire an output result and runtime performance data of the target model; the target model is a text generation large model to be evaluated; the target evaluation sample library contains a financial service text generation task and a standard template;

[0044] an inference performance evaluation module configured to perform inference performance evaluation on the target model based on the runtime performance data, and generate an inference performance score;

[0045] a service quality evaluation module configured to perform service quality evaluation on the target model based on the output result, and generate a service quality score; the service quality evaluation comprises basic semantic quality evaluation and domain expertise evaluation; the domain expertise evaluation comprises structured output compliance evaluation, similar question and answer reply similarity evaluation, service logic rationality evaluation, and output text compliance evaluation;

[0046] a weight proportion allocation module configured to determine a weight proportion of the inference performance score and the service quality score according to an application service scenario type of the target model;

[0047] a performance evaluation module configured to fuse the weighted service quality score and the weighted inference performance score based on the weight proportion, and generate a performance total score of the target model.

[0048] Compared with the prior art, the present application has the following beneficial effects:

[0049] In the technical scheme of the present application, first, the evaluation samples in the target evaluation sample library are input into the target model, and the output result and runtime performance data of the target model are obtained; then, inference performance evaluation is performed on the target model based on the runtime performance data, and an inference performance score is generated; service quality evaluation, including basic semantic quality evaluation and domain expertise evaluation, is performed on the target model based on the output result, and a service quality score is generated; thereafter, a weight proportion of the inference performance score and the service quality score is allocated according to an application service scenario type of the target model; finally, the weighted service quality score and the weighted inference performance score are fused based on the weight proportion, and a performance total score of the target model is generated. As can be seen in the technical scheme of the present application, through multi-dimensional evaluation of the inference performance and service quality of the target model, not only the running efficiency of the model is considered, but also the model output content is evaluated from multiple aspects such as basic semantic quality and domain expertise, which can comprehensively reflect the performance of the financial text large model. And according to different financial service scenario types, the weight proportion of the inference performance score and the service quality score is dynamically adjusted, so that the evaluation result is more in line with the actual business needs, and the pertinence and practicality of the evaluation are improved. The comprehensive and accurate evaluation of the performance of the financial text large model is realized, which provides a scientific and quantitative basis for the optimization and selection of the text large model in the financial field, and significantly improves the reliability and risk controllability of the application of the large model in the financial field. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0051] Figure 1 A flow chart of a financial text large model performance evaluation method provided by the embodiments of the present application;

[0052] Figure 2 A structural schematic diagram of a financial text large model performance evaluation device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0053] As described above, the current large model evaluation method is difficult to adapt to the professional needs of financial business, there is a lack in the depth evaluation dimension of vertical fields, and the fixed evaluation framework is difficult to balance the differentiated requirements of different businesses on model performance. It leads to the inability to comprehensively and accurately measure the comprehensive performance of the model in the actual financial business, and the financial institutions face challenges in model selection, optimization and risk control.

[0054] The inventor has proposed a financial text large model performance evaluation method and device.

[0055] First, the evaluation samples in the target evaluation sample library are input into the target model to obtain output results and runtime performance data of the target model; then, inference performance evaluation is performed on the target model based on the runtime performance data to generate an inference performance score; business quality evaluation including basic semantic quality evaluation and field professional evaluation is performed on the target model based on the output results to generate a business quality score; thereafter, the inference performance score and the business quality score are assigned a weight ratio according to the application business scenario type of the target model; finally, the weighted business quality score and the inference performance score are fused based on the weight ratio to generate a performance total score of the target model. It can be seen that in the technical scheme of the present application, the inference performance and business quality of the target model are evaluated in multiple dimensions, not only considering the running efficiency of the model, but also evaluating the model output content from multiple aspects such as basic semantic quality and field professional quality, which can comprehensively reflect the performance of the financial text large model. And according to different financial business scenario types, the weight ratio of the inference performance score and the business quality score is dynamically adjusted, so that the evaluation result is more in line with the actual business demand, improving the pertinence and practicality of the evaluation. The comprehensive and accurate evaluation of the performance of the financial text large model is realized, which provides a scientific and quantitative basis for the optimization and selection of the text large model in the financial field, and significantly improves the reliability and risk controllability of the application of the large model in the financial field.

[0056] In order for those skilled in the art to better understand the technical scheme of the present application, the technical scheme in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0057] Referring to Figure 1 The figure is a flow chart of a financial text large model performance evaluation method provided by an embodiment of the present application. As Figure 1 The method comprises the following steps:

[0058] S101, input evaluation samples in a target evaluation sample library into a target model to obtain output results and runtime performance data of the target model.

[0059] In the embodiments of the present application, the target model refers to a financial field text generation large model to be evaluated.

[0060] In the embodiments of the present application, a target evaluation sample library is first constructed, which covers text generation tasks of multiple typical business scenarios in the financial industry, including but not limited to compliance report generation, risk event template extraction, wealth management document key point generation, etc. Each evaluation sample contains a business problem description, a standard answer, and a corresponding financial business standard template (such as a bank internal report format template, a compliance rhetoric template, etc.).

[0061] The evaluation samples in the constructed target sample library are input into the target model one by one, and the target model generates corresponding output results after processing each evaluation sample.

[0062] In the embodiments of the present application, a performance monitoring tool (such as Prometheus, Grafana, etc.) is used to collect runtime performance data of the model when reprocessing each evaluation sample in real time.

[0063] The collected data includes but is not limited to: first word response time (the time required from the end of the user sending the question to the user receiving the first token returned by the model), CPU / GPU occupancy, memory consumption, and maximum concurrency, etc. hardware efficiency indicators.

[0064] S102, based on the runtime performance data, the inference performance of the target model is evaluated, and an inference performance score is generated.

[0065] In the embodiments of the present application, inference performance evaluation refers to multi-dimensional evaluation of the hardware efficiency of the target model, including time characteristic dimension, resource consumption dimension, and concurrency capability dimension, etc.

[0066] Based on the runtime performance data collected in step S101, the inference performance of the target model is evaluated, and an inference performance score is generated

[0067] In the embodiments of the present application, each dimension is scored, and is converted to a unified score system score according to a preset, such as 0-100 points. The inference performance score is generated based on the score of each dimension.

[0068] S103, based on the output result, the business quality of the target model is evaluated, and a business quality score is generated.

[0069] In the embodiments of the present application, business quality evaluation includes basic semantic quality evaluation and field professional evaluation.

[0070] Basic semantic quality evaluation refers to the evaluation of the semantic accuracy and information integrity of the target model in general scenarios, avoiding basic understanding bias.

[0071] The field-specific evaluation refers to vertical field-specific evaluation indexes for financial industry norms, business logic and regulatory requirements, including structured output compliance evaluation, similar question reply similarity evaluation, business logic rationality evaluation and output text compliance evaluation.

[0072] In the embodiments of the present application, the structured output compliance evaluation refers to a mechanism for evaluating whether the model-generated content meets the established structure and format specifications for the structured reports, tables and other formats commonly used in the daily business of financial institutions, ensuring information transmission efficiency and normal operation of business processes.

[0073] The similar question reply similarity evaluation refers to, in the financial business scenarios (such as financial customer service and intelligent question answering), evaluating whether the model's reply to similar questions posed by users in different ways is consistent, avoiding user confusion caused by too large differences in replies, and reflecting the model's ability to understand and handle similar semantic problems.

[0074] The business logic rationality evaluation refers to evaluating whether the model-generated content meets the business logic sequence and rules specific to financial business.

[0075] The output text compliance evaluation refers to evaluating whether the model-generated text contains any non-compliant content based on financial industry regulatory requirements, internal compliance systems and data security specifications.

[0076] In the embodiments of the present application, each evaluation dimension is scored, and a unified score system score such as 0-100 is converted according to a preset. The business quality score is generated based on the score of each dimension.

[0077] In the embodiments of the present application, the weight proportions of the inference performance score and the business quality score are assigned according to the application business scenario type of the target model.

[0078] In the embodiments of the present application, first, the business scenario type can be divided into quality priority type business scenarios (such as compliance report generation scenarios and complex risk assessment scenarios), efficiency priority type business scenarios (such as real-time anti-fraud detection scenarios and remote bank question answering scenarios) and balanced type business scenarios (such as intelligent investment advice scenarios and customer credit scoring scenarios) according to the priority of quality and efficiency in financial business, and the business scenario type of the target model is labeled with a business label in advance.

[0079] In the embodiments of the present application, the business scenario type is automatically matched through the business label, and the weight proportions of the inference performance score and the business quality score are determined according to the business scenario type.

[0080] In the embodiments of the present application, for the quality priority type service scenario, the weight proportion of the service quality score as the dominant weight is set, for example, the service quality score weight is 70%, and the inference performance score weight is 30%; for the efficiency priority type service scenario, the weight proportion of the inference performance score as the dominant weight is set, for example, the service quality score weight is 20%, and the inference performance score weight is 80%; and for the balanced type service scenario, the weight of the inference performance score and the service quality score is set as 50% respectively. It should be noted that the specific weight proportion can be appropriately adjusted according to actual application needs, which is not limited herein.

[0081] In S105, based on the weight proportion, the weighted service quality score and the inference performance score are fused to generate the performance total score of the target model.

[0082] In the embodiments of the present application, based on the weight proportion determined in the foregoing step, the service quality score and the inference performance score of the target model generated in the foregoing step are fused and weighted to generate the performance total score of the target model, and the specific formula is as follows:

[0083] Performance total score = service quality score x V1 + inference performance score x V2

[0084] Wherein, V1 is the weight of the service quality score, V2 is the weight of the inference performance score, V1 + V2 = 1, and the values of V1 and V2 are determined by the foregoing step S104.

[0085] In an example implementation, taking the compliance report generation scenario as an example, V1 = 80%, V2 = 20%, the service quality score of the target model is 90 points, and the inference performance score is 85 points, then the performance total score of the target model is 90 x 0.8 + 85 x 0.2 = 89 points.

[0086] In the embodiments of the present application, the performance total score is used to comprehensively measure the overall performance of the target model in the financial field text generation task.

[0087] Optionally, the performance total score can be defined by interval performance.

[0088] In an example implementation, taking 100 points as an example, if the performance total score is between 85-100 points, it indicates that the performance of the target model is excellent, and the target model can be directly put into production; if the performance total score is between 70-85 points, it indicates that the target model has local short boards, and needs to be optimized (such as improving the concurrency capability of a certain scene); and if the performance total score is less than 70 points, it indicates that the core indicators of the target model are not up to standard, and the architecture needs to be retrained or adjusted.

[0089] The specific interval setting and performance definition can be set according to the performance requirements of the target model, which is not limited herein.

[0090] Optionally, an interactive report containing the scores of each dimension and the scene adaptation suggestions is generated for the joint analysis of the financial institution technical team and the business department.

[0091] In the embodiments of the present application, through multi-dimensional evaluation of the inference performance and business quality of the target model, not only the running efficiency of the model is considered, but also the model output content is evaluated from multiple aspects such as basic semantic quality and domain expertise, which can comprehensively reflect the performance of the financial text large model. According to different types of financial business scenarios, the weight proportion of the inference performance score and the business quality score is dynamically adjusted, so that the evaluation result is more in line with the actual business demand, and the pertinence and practicality of the evaluation are improved. The comprehensive and accurate evaluation of the performance of the financial text large model is realized, which provides a scientific and quantitative basis for the optimization and selection of the text large model in the financial field, and significantly improves the reliability and risk controllability of the application of the large model in the financial scenario.

[0092] Optionally, the structured output conformity evaluation in the step S103 comprises:

[0093] Step one, analyze the structural characteristics of the standard template and the output result.

[0094] In the embodiments of the present application, the standard template can be converted into a standard structure tree Tstrand by tree structure analysis technology, such as Python lxml library. At the same time, the output result of the model is parsed into an output structure tree Toutput.

[0095] Step two, calculate the structural similarity score and format standardization score of the output result and the standard template.

[0096] In an example implementation, the difference between the two tree structures parsed in step one is calculated based on the tree edit distance algorithm (such as the tree structure extension of Levenshtein distance), and the structural similarity score is obtained. The calculation formula of the structural similarity score is as follows:

[0097] S_distance=(1 - d / max_distance)×60);

[0098] Wherein, d is the tree edit distance of the standard structure tree Tstrand and the output structure tree Toutput, and the tree edit distance represents the minimum number of editing operations (node insertion, deletion, modification) required to convert Toutput into Tstrand. max_distance is the maximum edit distance of the two trees in the worst case (optionally, a reasonable upper limit value can be pre-set according to the complexity of the standard template).

[0099] In the embodiment of the present application, the output result of the target model is simultaneously subjected to format specification checking (such as font, font size, color, alignment, and other format attributes), and a format specification score is generated.

[0100] In the embodiment of the present application, a set of format checking items F = {f_1, f_2, …, f_n} is first defined, where each checking item corresponds to a format requirement of the standard template, for example, f_1 is whether the font is Songti, f_2 is whether the font size is 12 points, and so on. For the output result of the target model, if the checking item f_i is met, then v_i = 1, otherwise v_i = 0, i = 1, 2, …, n. The calculation formula of the format specification score is as follows:

[0101] ;

[0102] where n is the total number of checking items.

[0103] Step three, generate a structured output compliance score based on the structure similarity score and the format specification score.

[0104] In the embodiment of the present application, the structured output compliance score is the sum of the structure similarity score and the format specification score, that is:

[0105] Structured output compliance score = S_distance + S_format.

[0106] In the embodiment of the present application, the range of the structured output compliance score is set to 0-100, and the higher the score, the higher the degree of structural compliance of the output result of the target model with the standard template.

[0107] In the embodiment of the present application, the structured output compliance score is set to 0-100, and the higher the score, the higher the degree of structural compliance of the output result of the target model with the standard template.

[0108] Optionally, the reply similarity evaluation of similar questions in the above step S103 includes:

[0109] The reply similarity score under similar questions is calculated by semantic vector mapping.

[0110] In the embodiment of the present application, the reply text corresponding to similar questions is mapped into a fixed-dimensional semantic vector by using a pre-trained language model.

[0111] By calculating the Euclidean distance or cosine distance between vectors, the semantic difference between the reply texts is quantified, and combined with the set difference threshold, it is determined whether there is too large difference in the reply. Finally, the reply similarity score in the range of 0-100 is obtained, and the higher the score, the higher the similarity of the reply, and the smaller the difference.

[0112] In an example implementation, according to historical data and experiments, the difference threshold T is set, and the final reply similarity score is determined by the value relationship between the cosine distance D between vectors and the threshold. Table 1 is an example of a reply similarity score table provided by the embodiment of the application.

[0113] Table 1: Reply similarity score table

[0114]

[0115] In the embodiment of the application, the consistency of the reply is evaluated by the semantic vector distance, the logical fluctuation of the target model in the processing of similar problems (such as the process difference under different expressions of the same business) can be identified, the user experience is improved, and the decline of customer trust caused by contradictory replies is avoided.

[0116] Optionally, the business logic rationality evaluation in the step S103 comprises:

[0117] Step 1, the output result of the target model is decomposed into a logical unit sequence.

[0118] In the embodiment of the application, the output result of the target model is decomposed into a logical unit sequence.

[0119] In an example implementation, the output result is "evaluate customer credit risk → determine loan amount 500,000 yuan → verify customer identity information → generate loan contract".

[0120] Then the logical unit sequence is: [risk assessment, quota approval, identity verification, contract signing].

[0121] Step 2, match the logical unit sequence with the standard logical path in the pre-constructed financial business knowledge graph to generate a business logic rationality score.

[0122] In the embodiment of the application, the calculation formula of the business logic rationality score is as follows:

[0123] Business logic rationality score = number of matched logical units / total number of logical units x 100

[0124] In an example implementation, the standard logical path in the pre-constructed financial business knowledge graph is: identity verification → risk assessment → quota approval → contract signing.

[0125] The matching result is that the "identity verification" unit is in the wrong position (it should be in the first step, but it is actually in the third step), and the order of the remaining units is correct. Therefore, the business logic rationality score of the target model is 3 / 4*100=75.

[0126] In the embodiments of the present application, by structuring the business logic through the knowledge graph, the process reversal or omission (such as loaning first and then approving) in the output result of the target model can be accurately detected, the business risks (such as compliance loopholes or operation errors) caused by logical errors can be reduced, and it is ensured that the model generated content meets the financial business specifications.

[0127] Optionally, the output text compliance evaluation in the step S103 comprises:

[0128] Based on the preset financial compliance rule library, the rule violation items in the output result are scanned, a hierarchical deduction operation is performed according to the violation type, and an output text compliance score is generated.

[0129] In the embodiments of the present application, the preset financial compliance rule library comprises:

[0130] Prohibitive expressions such as "stable profit", "guaranteed return", and "zero risk";

[0131] Sensitive data such as ID number, account balance, and bank account number;

[0132] Regulatory points: the risk disclosure of the financial product must include "past performance does not represent future performance" and "investment has risks".

[0133] In the embodiments of the present application, the rule violation items in the output result are scanned based on the preset financial compliance rule library using a natural language processing tool (such as a regular expression or a keyword matching).

[0134] According to the violation type, a hierarchical deduction operation is performed to generate an output text compliance score. Specifically:

[0135]

[0136] In the embodiments of the present application, the base score of the output text compliance score is 100 points. The deduction standards corresponding to different violation types can be set according to actual needs, which are not limited here.

[0137] In one example implementation, the prohibitive expressions are triggered: deduct 15 points for each; sensitive data is leaked: deduct 20 points for each; and regulatory points are missing: deduct 10 points for each.

[0138] For example, the output result of the target model contains the following rule violation items:

[0139] The prohibitive expressions are triggered, the sensitive data is leaked, and the regulatory points are missing.

[0140] The output text compliance score of the target model = 100 - (15 + 20 + 10) = 55 points.

[0141] In an optional implementation, the evaluation is terminated when the scanning result of the aforementioned violation item meets any of the following conditions:

[0142] There is a preset high-risk violation keyword;

[0143] The risk prompt content is completely missing;

[0144] The accumulated deduction reaches a preset threshold.

[0145] In the embodiments of the present application, through automatic scanning and grading deduction, rapid identification of violation content is realized, especially rigid interception of high-risk keywords, to ensure that the model output meets the requirements of regulations such as “Commercial Bank Financial Business Supervision and Management Measures”, and to avoid financial institutions facing regulatory penalties or customer complaints due to violation content.

[0146] Optionally, the above step S103 comprises:

[0147] S1031, for each evaluation index in the business quality evaluation, a judgment matrix of the evaluation index is constructed by the analytic hierarchy process.

[0148] In the embodiments of the present application, the comprehensive evaluation of the business quality of the financial field generation model is taken as the target layer, that is, by determining the weight of each evaluation index, the accurate evaluation of the model generation quality is realized, and the needs of different financial business scenarios are met.

[0149] The plurality of evaluation indexes in the basic semantic quality evaluation and the field professional evaluation are taken as the criterion layer. These indexes measure the professionalism, integrity and compliance of the model generation content from different dimensions.

[0150] In the embodiments of the present application, for different financial business scenarios, the importance of each evaluation index in the criterion layer with respect to the model generation content in the field is compared pairwise. The 1-9 scale method is adopted, and the comparison results are constructed into a judgment matrix.

[0151] After completing the pairwise comparison and assignment of all indexes, a judgment matrix is formed. The judgment matrix is as follows:

[0152] ;

[0153] Wherein, the number of rows of the matrix represents the number of evaluation indexes, a ij represents the value of the importance of the i-th evaluation index to the j-th index, i = 1, 2, 3, 4, 5, 6, j = 1, 2, 3, 4, 5, 6.

[0154] Table 2 provides a plurality of 1-9 scale value tables for the embodiments of the present application.

[0155] Table 2 1-9 scale value table

[0156]

[0157] In an example implementation, for the indicators A and B, if in the financial knowledge question and answer scene, the indicator A is slightly more important than the indicator B, then 3 is assigned to the corresponding position of the judgment matrix; if both are equally important, then 1 is assigned.

[0158] S1032, calculate the weight of each evaluation index.

[0159] In the embodiments of the present application, the weight vector of each evaluation index is obtained after normalization processing of the eigenvector corresponding to the maximum eigenvalue of the judgment matrix constructed in the previous step.

[0160] Consistency check is performed to ensure that the consistency of the judgment matrix is acceptable and the weight vector is effective.

[0161] The calculation formula of the consistency ratio CR is as follows:

[0162] ;

[0163] wherein, is the maximum eigenvalue corresponding to the judgment matrix; n is the number of evaluation indexes; CI is a consistency index in the analytic hierarchy process, which is a quantitative index for measuring the consistency of the judgment matrix. RI is a random consistency index in the analytic hierarchy process, which is a reference value for measuring the consistency of the judgment matrix. It represents the consistency level that a judgment matrix of different orders can achieve in a random case. According to the number of evaluation indexes, the corresponding RI value is obtained from the average random consistency index table.

[0164] If CR is less than 0.1, it is considered that the consistency of the judgment matrix is acceptable; otherwise, the judgment matrix needs to be adjusted until the consistency check is passed.

[0165] In the embodiments of the present application, the weight of each evaluation index calculated by the analytic hierarchy process can scientifically and reasonably reflect the importance of each index in the business quality evaluation, and provide an accurate basis for subsequent score fusion. The consistency check mechanism ensures the reliability of the weight result.

[0166] S1033, fuse the scores of each evaluation index based on the weight of each evaluation index to generate a business quality score.

[0167] In the embodiments of the present application, the scores of each evaluation index are fused based on the weight of each evaluation index determined in the foregoing steps to generate a business quality score.

[0168] Optionally, the runtime performance data includes first-word response time, computing resource consumption data, and maximum concurrency. The step 102 includes:

[0169] S1021, generating a timeliness score based on the first-word response time.

[0170] In the embodiments of the present application, the first-word response time refers to the response time required from the end of sending a question by a user to the reception of the first token returned by the target model by the user.

[0171] The corresponding timeliness score is generated according to the first-word response time. Specifically, the scoring standard can be set according to actual needs, which is not limited herein.

[0172] Table 3 is a first-word response time scoring table provided by the embodiments of the present application.

[0173] Table 3 First-word response time scoring table

[0174]

[0175] S1022, generating a resource utilization rate score based on the computing resource consumption data.

[0176] In the embodiments of the present application, the CPU / GPU occupancy rate and memory consumption data of the model running in real time are collected, and the resource consumption data (including benchmark average value, benchmark maximum value, and benchmark minimum value) of the open source same parameter order of magnitude large model are combined to normalize the resource usage of the model, calculate the resource utilization rate score, and the specific calculation formula is as follows:

[0177] Resource utilization rate score=(benchmark maximum value-measured value) / (benchmark maximum value-benchmark minimum value)×100.

[0178] S1023, generating a concurrency capability score based on the maximum concurrency.

[0179] Table 4 is a benchmark concurrency level classification table in the financial field provided by the embodiments of the present application.

[0180] Table 4 Benchmark concurrency level classification table in the financial field

[0181]

[0182] In the embodiments of the present application, the simulation tool is used to simulate the scenario of multiple users simultaneously initiating requests to the target model, gradually increasing the number of concurrent users, recording the response time and request processing success rate of the target model under different concurrency, determining the actual maximum concurrency of the target model, and determining the benchmark concurrency according to Table 4, calculating the concurrency capability score, and the specific formula is as follows:

[0183] The concurrency capability score = min (actual maximum concurrency / benchmark concurrency x 100, 100).

[0184] S1024, according to the application business scene type of the target model, the inference performance weight proportion is allocated to the time efficiency score, the resource utilization rate score and the concurrency capability score.

[0185] In the embodiment of the application, first, the business scene type is divided into high real-time business scene, resource sensitive scene, high concurrency peak scene and balanced scene, and the business label of the application business scene type of the target model is labeled in advance.

[0186] In the embodiment of the application, the business scene type is automatically matched by the business label, and the weight proportion of the time efficiency score, the resource utilization rate score and the concurrency capability score is determined according to the business scene type.

[0187] In the embodiment of the application, for the high real-time business scene, the time efficiency score weight is greater than the concurrency capability score weight which is greater than the resource utilization rate score weight; for the resource sensitive scene, the resource utilization rate score weight is greater than the resource utilization rate score weight which is greater than the concurrency capability score weight; for the high concurrency peak scene, the concurrency capability score weight is greater than the time efficiency score weight which is greater than the resource utilization rate score weight; for the balanced scene, the time efficiency score weight, the concurrency capability score weight and the resource utilization rate score weight are configured to be basically equal.

[0188] S1025, the inference performance score is generated by fusing the time efficiency score, the resource utilization rate score and the concurrency capability score based on the inference performance weight proportion.

[0189] According to the weight proportion determined in step S1024, the time efficiency score, the resource utilization rate score and the concurrency capability score are weighted and fused to generate the inference performance score.

[0190] In the embodiment of the application, through the multi-dimensional inference performance evaluation and the dynamic weight distribution combined with the business scene type, the inference performance of the model in different business scenes can be comprehensively and accurately reflected, and a scientific basis is provided for the financial institutions to select efficient and stable models.

[0191] Optionally, the basic semantic quality evaluation in the step S103 above comprises:

[0192] Step 11, the keyword coverage score is generated by evaluating the keyword coverage of the output result.

[0193] In the embodiment of the application, the financial key terms, data or core concepts related to the question are extracted from the standard answer of the target evaluation sample library, such as "annual yield" and "compound interest calculation".

[0194] Store these keywords in a list as the basis for evaluation.

[0195] Parse the target model output results and extract the keywords contained therein.

[0196] In an example implementation, natural language processing techniques such as word segmentation, part-of-speech tagging, etc. can be used to assist in extracting keywords.

[0197] Count the number of keywords contained in the output results and calculate their proportion in the total number of keywords in the standard answer.

[0198] Calculate the keyword coverage score, the specific formula is as follows:

[0199] Keyword coverage score = (number of keywords hit / total number of keywords in standard answer) x 100.

[0200] For example, the standard answer contains 5 keywords: "annual yield", "compound interest calculation", "risk level", "investment period", and "expected return". The target model output result contains 4 of them. Then the keyword coverage score = (4 / 5) x 100 = 80 points.

[0201] Step 12, perform semantic similarity evaluation on the output results to generate a semantic similarity score.

[0202] In the embodiments of the present application, the output results and the standard answer are converted into high-dimensional vectors using a pre-trained model. These vectors can capture the semantic information in the text, facilitating subsequent similarity calculation.

[0203] Based on the cosine similarity, the semantic correlation between the output results and the standard answer vectors is calculated. The cosine similarity has a value range of 0 to 1, and the higher the value, the stronger the semantic similarity.

[0204] The calculated cosine similarity value is multiplied by 100 to convert it into a semantic similarity score between 0 and 100, the specific formula is as follows:

[0205] Semantic similarity score = average cosine similarity x 100.

[0206] For example, the cosine similarity between the output results and the standard answer vectors is 0.85. Then the semantic similarity score = 0.85 x 100 = 85 points.

[0207] In the embodiments of the present application, through keyword coverage evaluation and semantic similarity evaluation, the basic semantic quality of the model output content can be comprehensively and accurately evaluated. These evaluation results not only help to understand the basic performance of the model in general natural language processing tasks, but also provide an important reference for subsequent field-specific evaluation.

[0208] The embodiment of the application constructs a target evaluation sample library containing a financial service text generation task and a standard template, inputs a sample into a target model to obtain an output result and runtime performance data containing a first word response time, a calculation resource consumption and a maximum concurrency; generates a score of inference performance based on the runtime performance data from the dimensions of time characteristics, resource consumption and concurrency capability, dynamically allocates weights for each dimension according to an application scenario type; generates a score of service quality based on the output result from basic semantic quality (keyword coverage, semantic similarity) and domain professionalism (structured output compliance, similar question reply similarity, business logic rationality, output text compliance), wherein the domain professionalism evaluation is implemented by tree edit distance, semantic vector distance, knowledge graph matching and compliance word library scanning, and the weight of the service quality layer is determined by an analytic hierarchy process; allocates a weight proportion for the score of inference performance and the score of service quality according to the scenario type, fuses a total performance score, and generates an evaluation report corresponding to different performance performances in a total score interval, so as to realize comprehensive and accurate evaluation of a financial large model and provide a scientific basis for model optimization and financial service landing.

[0209] Based on the financial text large model performance evaluation method provided in the foregoing embodiments, the application also provides a financial text large model performance evaluation device. Figure 2 A structural schematic diagram of a financial text large model performance evaluation device provided by the embodiment of the application is shown in FIG. 1. Figure 2 As shown in the figure, the financial text large model performance evaluation device includes:

[0210] An evaluation data acquisition module 201 is configured to input an evaluation sample in a target evaluation sample library into a target model, acquire an output result and runtime performance data of the target model, wherein the target model is a text generation large model to be evaluated, and the target evaluation sample library contains a financial service text generation task and a standard template;

[0211] An inference performance evaluation module 202 is configured to evaluate inference performance of the target model based on the runtime performance data and generate a score of inference performance;

[0212] A service quality evaluation module 203 is configured to evaluate service quality of the target model based on the output result and generate a score of service quality, wherein the service quality evaluation includes basic semantic quality evaluation and domain professionalism evaluation, and the domain professionalism evaluation includes structured output compliance evaluation, similar question reply similarity evaluation, business logic rationality evaluation and output text compliance evaluation;

[0213] A weight proportion allocation module 204 is configured to determine a weight proportion of the score of inference performance and the score of service quality according to an application service scenario type of the target model.

[0214] The performance evaluation module 205 is configured to generate a total performance score of the target model by fusing the weighted service quality score and the inference performance score based on the weight proportion.

[0215] Optionally, the service quality evaluation module 203 comprises a structured output compliance evaluation unit configured to:

[0216] analyze structural features of the standard template and the output result;

[0217] calculate a structural similarity score and a format specification score of the output result and the standard template;

[0218] generate a structured output compliance score based on the structural similarity score and the format specification score.

[0219] Optionally, the service quality evaluation module 203 comprises a reply similarity evaluation unit of similar questions, configured to:

[0220] calculate a reply similarity score of similar questions by semantic vector mapping.

[0221] Optionally, the service quality evaluation module 203 comprises a service logic rationality evaluation unit, configured to:

[0222] disassemble the output result of the target model into a sequence of logic units;

[0223] match the sequence of logic units with a standard logic path in a pre-constructed financial service knowledge graph to generate a service logic rationality score.

[0224] Optionally, the service quality evaluation module 203 comprises an output text compliance evaluation unit, configured to:

[0225] scan the output result for non-compliance items based on a pre-set financial compliance rule library;

[0226] perform a deduction operation according to the type of non-compliance to generate an output text compliance score.

[0227] Optionally, the apparatus further comprises an evaluation termination module configured to terminate the evaluation when the scanning result of the output text compliance evaluation unit meets any of the following conditions:

[0228] there is a pre-set high-risk non-compliance keyword;

[0229] risk prompt content is completely missing;

[0230] cumulative deductions reach a pre-set threshold.

[0231] Optionally, the service quality evaluation module 203 is specifically configured to:

[0232] By the analytic hierarchy process, for each evaluation index in the service quality evaluation, a judgment matrix of the evaluation index is constructed;

[0233] According to the judgment matrix, the weight of each evaluation index is calculated;

[0234] Based on the weight of each evaluation index, the scores of each evaluation index are fused to generate a service quality score.

[0235] Optionally, the runtime performance data includes first letter response time, computing resource consumption data and maximum concurrency, and the inference performance evaluation module 202 is specifically configured to:

[0236] Based on the first letter response time, a timeliness score is generated;

[0237] Based on the computing resource consumption data, a resource utilization rate score is generated;

[0238] Based on the maximum concurrency, a concurrency capability score is generated;

[0239] According to the application scene type of the target model, the inference performance weight proportion of the timeliness score, the resource utilization rate score and the concurrency capability score is determined;

[0240] Based on the inference performance weight proportion, the timeliness score, the resource utilization rate score and the concurrency capability score are fused to generate an inference performance score.

[0241] Optionally, the service quality evaluation module 203 includes a basic semantic quality evaluation unit, configured to:

[0242] Perform keyword coverage evaluation on the output result to generate a keyword coverage score;

[0243] Perform semantic similarity evaluation on the output result to generate a semantic similarity score.

[0244] It should be noted that each of the embodiments of the present specification is described in a progressive manner, and the same or similar parts between the embodiments can be mutually referred to, and each of the embodiments focuses on the differences from other embodiments. In particular, for the device embodiments, since they are basically similar to the method embodiments, they are described more simply, and the relevant parts can be referred to the part of the description of the method embodiments. The above-described device embodiments are only illustrative, and the units described as separate components can or can not be physically separated, and the components indicated as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiments according to the actual needs. Those skilled in the art can understand and implement it without creative labor.

[0245] The above describes only one specific embodiment of the present application, but the protection scope of the present application is not limited to this. Any skilled person in the art can easily think of changes or replacements within the technical range disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for evaluating the performance of a large financial text model, characterized in that, The method comprises: inputting an evaluation sample in a target evaluation sample library into a target model to obtain an output result and runtime performance data of the target model; the target model is a text generation large model to be evaluated; the target evaluation sample library comprises a financial business text generation task and a standard template; performing inference performance evaluation on the target model based on the runtime performance data to generate an inference performance score; performing business quality evaluation on the target model based on the output result to generate a business quality score; the business quality evaluation comprises basic semantic quality evaluation and domain professional evaluation; the domain professional evaluation comprises structured output compliance evaluation, reply similarity evaluation of similar questions, business logic rationality evaluation and output text compliance evaluation; determining a weight proportion of the inference performance score and the business quality score according to an application business scene type of the target model; based on the weight proportion, fusing the weighted business quality score and the inference performance score to generate a performance total score of the target model.

2. The method of claim 1, wherein, The structured output compliance evaluation comprises: analyzing structural features of the standard template and the output result; calculating a structural similarity score and a format specification score of the output result and the standard template; generating a structured output compliance score based on the structural similarity score and the format specification score.

3. The method of claim 1, wherein, The reply similarity evaluation of similar questions comprises: calculating a reply similarity score under similar questions through semantic vector mapping.

4. The method of claim 1, wherein, The business logic rationality evaluation comprises: disassembling the output result of the target model into a logic unit sequence; matching the logic unit sequence with a standard logic path in a pre-constructed financial business knowledge graph to generate a business logic rationality score.

5. The method of claim 1, wherein, The output text compliance evaluation comprises: scanning a rule violation item in the output result based on a pre-set financial compliance rule library; performing a deduction operation according to a rule violation type to generate an output text compliance score.

6. The method of claim 5, wherein, The method further comprises: terminating the evaluation when the scanning result satisfies any one of the following conditions: there is a pre-set high-risk rule violation keyword; risk prompt content is completely missing; cumulative deduction reaches a pre-set threshold.

7. The method of claim 1, wherein, The business quality evaluation based on the output result to generate a business quality score comprises: constructing a judgment matrix of evaluation indexes through an analytic hierarchy process for each evaluation index in the business quality evaluation; calculating weights of the evaluation indexes according to the judgment matrix; fusing scores of the evaluation indexes based on the weights of the evaluation indexes to generate a business quality score.

8. The method of claim 1, wherein, The runtime performance data comprises first character response time, calculation resource consumption data and maximum concurrency; the inference performance evaluation based on the runtime performance data to generate an inference performance score comprises: generating a timeliness score based on the first character response time; generating a resource utilization rate score based on the calculation resource consumption data; generating a concurrency capability score based on the maximum concurrency; According to the application scene type of the target model, a reasoning performance weight proportion of the timeliness score, the resource utilization rate score and the concurrency capability score is determined; Based on the reasoning performance weight proportion, the timeliness score, the resource utilization rate score and the concurrency capability score are fused to generate a reasoning performance score.

9. The method of claim 1, wherein, The basic semantic quality evaluation includes: A keyword coverage evaluation is performed on the output result to generate a keyword coverage score; A semantic similarity evaluation is performed on the output result to generate a semantic similarity score.

10. A financial text large model performance evaluation device, characterized by, The device includes: An evaluation data acquisition module configured to input an evaluation sample in a target evaluation sample library into a target model to acquire an output result and runtime performance data of the target model; the target model is a text generation model to be evaluated; the target evaluation sample library includes a financial business text generation task and a standard template; A reasoning performance evaluation module configured to perform a reasoning performance evaluation on the target model based on the runtime performance data to generate a reasoning performance score; A business quality evaluation module configured to perform a business quality evaluation on the target model based on the output result to generate a business quality score; the business quality evaluation includes a basic semantic quality evaluation and a domain professional evaluation; the domain professional evaluation includes a structured output compliance evaluation, a reply similarity evaluation of similar questions, a business logic rationality evaluation and an output text compliance evaluation; A weight proportion allocation module configured to determine a weight proportion of the reasoning performance score and the business quality score according to an application business scene type of the target model; A performance evaluation module configured to fuse the weighted business quality score and the reasoning performance score based on the weight proportion to generate a performance total score of the target model.

Images