A method and device for performing a performance evaluation on a financial institution

Abstract

The embodiment of the specification provides a method for performing a job evaluation on a financial institution, comprising: receiving a first request from a first device, wherein the first device corresponds to a first user, and the first request comprises an institution identifier of a target institution to be evaluated; responding to the first request by a large model, performing target processing on evidence materials of the target institution to obtain a target evaluation result; wherein the target processing comprises triggering a plurality of rounds of first interaction when a first condition is met, and any round of first interaction comprises: pushing a first interaction request to the first device, and receiving first interaction feedback containing supplementary materials sent by the first user in response to the first interaction request, and incorporating the supplementary materials into the evidence materials; the first condition comprises that the confidence of the current processing result of the large model is lower than a preset threshold; and sending the target evaluation result to the first device, so that the first device generates a job evaluation report for the target institution based on the target evaluation result.

Description

Technical Field

[0001] This specification relates to the field of data processing technology, and in particular to a method and apparatus for evaluating the performance of financial institutions. Background Technology

[0002] In the field of anti-money laundering (AML), regulators require financial institutions to conduct regular performance reviews to systematically examine whether their specific practices in key areas such as AML strategies, customer due diligence, transaction monitoring, and suspicious transaction reporting comply with relevant laws, regulations, and regulatory guidelines. Performance reviews of financial institutions are a core component of AML supervision and self-inspection, and the results directly affect the institution's compliance risks and reputation.

[0003] For large financial groups, which typically have numerous branches, subsidiaries, or business units spread across different regions, performance evaluations need to cover every single unit being evaluated. In related technologies, performance evaluations primarily rely on compliance experts or audit teams to conduct them manually. Specifically, the evaluation team needs to repeatedly execute a set of procedures for each evaluated institution, including collecting extensive documentation (such as policy documents, operating manuals, and training records), reviewing sampled transaction data, conducting personnel interviews, etc., and ultimately forming an evaluation conclusion based on the personal experience and professional judgment of the personnel performing the evaluation.

[0004] The aforementioned manual evaluation methods face several challenges in practice. As the number of evaluated organizations increases, the repetitive manual document collection, data review, and interviews consume a significant amount of time from experienced compliance personnel, leading to a substantial increase in human and time costs. Furthermore, the depth, breadth, and accuracy of the evaluation largely depend on the personal experience, professional level, and subjective judgment of the evaluators. Different evaluators often employ different inspection scales and judgment standards when faced with the same or similar review materials, resulting in a lack of comparability between evaluation results from different organizations and making it difficult to form a unified and objective risk view. In addition, manual evaluations are typically conducted periodically, making it difficult to respond promptly to sudden changes in regulatory policies or emerging risk patterns.

[0005] With the development of Large Language Model (LLM) technology, the industry has begun to explore its application in performance evaluation scenarios. One straightforward approach is to input supporting materials as prompts into the LLM, which then generates the evaluation conclusions. However, this method is often ill-suited to addressing the need for comprehensive analysis of multi-dimensional and cross-source information in performance evaluations, and its generalization is limited in scenarios like financial compliance, where the accuracy and comprehensiveness of conclusions are crucial.

[0006] Therefore, in the scenario of evaluating the performance of financial institutions, it is desirable to provide a solution that uses technological means to improve the efficiency and standardization of evaluations while taking into account the reliability of evaluation conclusions and the flexibility of human-machine collaboration. Summary of the Invention

[0007] This specification describes one or more embodiments of a method and apparatus for evaluating the performance of financial institutions, which can solve the above-mentioned technical problems.

[0008] Based on the first aspect, a method for evaluating the performance of financial institutions is provided, including:

[0009] A first request is received from a first device, wherein the first device corresponds to a first user, and the first request includes an institution identifier of the target institution to be evaluated, wherein the target institution is a financial institution.

[0010] The large model responds to the first request and performs target processing on the supporting materials of the target institution to obtain a target evaluation result; wherein, the target processing includes triggering several rounds of first interaction when a first condition is met, and any round of first interaction includes: pushing a first interaction request to a first device and receiving a first interaction feedback containing supplementary materials sent by a first user in response to the first interaction request, and incorporating the supplementary materials into the supporting materials; the first condition includes that the confidence level of the current processing result of the large model is lower than a preset threshold.

[0011] The target evaluation results are sent to the first device, which generates a performance evaluation report for the target organization based on the target evaluation results.

[0012] According to the second aspect, this specification provides an apparatus for evaluating the performance of financial institutions, comprising:

[0013] The receiving module is configured to receive a first request from a first device, wherein the first device corresponds to a first user, and the first request includes an institution identifier of the target institution to be evaluated, wherein the target institution is a financial institution.

[0014] The processing module is configured to respond to the first request through a large model, perform target processing on the supporting materials of the target institution, and obtain a target evaluation result; wherein, the target processing includes triggering several rounds of first interaction when a first condition is met, and any round of first interaction includes: pushing a first interaction request to a first device, and receiving a first interaction feedback containing supplementary materials sent by a first user in response to the first interaction request, and incorporating the supplementary materials into the supporting materials; the first condition includes that the confidence level of the current processing result of the large model is lower than a preset threshold.

[0015] The sending module is configured to send the target evaluation result to the first device, so that the device generates a performance evaluation report for the target organization based on the target evaluation result.

[0016] According to a third aspect, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the method described in the first aspect.

[0017] According to a fourth aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method described in the first aspect.

[0018] In summary, the methods and apparatus provided in the embodiments of this specification offer a method for evaluating the performance of financial institutions. During the analysis and processing of supporting materials using a large model, multiple rounds of interaction with users are triggered based on confidence level monitoring. Supplementary materials provided by the user are incorporated into the supporting materials, driving the large model to continue analysis based on more comprehensive information. This achieves a closed-loop collaboration between machine analysis capabilities and human judgment experience while retaining the large model as the core analysis engine. While improving the efficiency of performance evaluation, it retains key nodes for human supervision and confirmation in financial compliance scenarios, balancing automation with the reliability and compliance of evaluation conclusions. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0020] Figure 1 This is a schematic diagram illustrating the implementation framework of a performance evaluation method for financial institutions disclosed in this specification.

[0021] Figure 2 This document presents a flowchart of a method for evaluating the performance of financial institutions.

[0022] Figure 3 This is a schematic diagram of an apparatus for evaluating the performance of financial institutions, based on an embodiment of this specification. Detailed Implementation

[0023] The solutions provided in the embodiments of this specification will now be described with reference to the accompanying drawings.

[0024] With the rapid development of Large Language Model (LLM) technology, it has demonstrated powerful capabilities in natural language understanding, logical reasoning, and text generation. LLM typically employs a Transformer architecture based on attention mechanisms. Through pre-training on massive corpora, it can learn the statistical features and semantic relationships of language, exhibiting good generalization ability. In practical applications, LLM can be guided to perform specific tasks through prompts, including text analysis, information extraction, and question-answering reasoning. For ease of reference, LLM will be referred to simply as a large model below. In some embodiments, LLM can also possess the ability to understand data from other modalities; in this case, LLM can also be called a multimodal large model.

[0025] In evaluating the performance of financial institutions, evaluators need to collect and process various types of data, including policy documents, operation manuals, training records, transaction data reports, and suspicious transaction reports. These materials are collectively referred to as supporting documentation. Supporting documentation comes from a wide range of sources and comes in various formats, including native text formats as well as non-native text formats such as PDF, Word, Excel, and images. Based on this supporting documentation, evaluators need to examine the target institution's performance in key areas such as anti-money laundering strategies, customer due diligence, transaction monitoring, and suspicious transaction reporting, and then form an evaluation conclusion.

[0026] During the evaluation process, evaluators may need to make judgments and decisions at multiple stages. For example, when the information in the supporting materials is insufficient to support a clear conclusion for a certain checkpoint, supplementary materials or interviews with the evaluated organization are required; after the preliminary analysis is completed, the conclusions need to be confirmed. These stages require the intervention of human experts' experience, while traditional automated solutions often lack a mechanism to organically combine such experience-based judgment with machine analytical capabilities.

[0027] In view of this, the inventors propose a method for evaluating the performance of financial institutions in the embodiments of this specification. A large-scale model can be used as the core analysis engine to process the supporting materials of the financial institution to be evaluated. An interaction mechanism with the user is introduced during the processing. For example, when the confidence level of the current processing result of the large-scale model is lower than a preset threshold, the large-scale model can initiate a first interaction request to the user to obtain supplementary materials, and incorporate the supplementary materials provided by the user through interactive feedback into the subsequent processing flow. Finally, a performance evaluation report is generated based on the analysis results of the large-scale model, completing the performance evaluation of the financial institution. This combines the analytical capabilities of the large-scale model with the judgment experience of human experts in the same closed-loop process, improving the degree of automation while retaining key nodes for human supervision and confirmation, ensuring the rigor of the performance evaluation.

[0028] Figure 1A schematic diagram of the implementation framework of this method is shown. (See also...) Figure 1 In this embodiment, the first device corresponds to the first user, who can be a compliance officer or auditor responsible for evaluating the performance of the target organization. The first user initiates an evaluation request through the first device, and the evaluation request carries at least the organization identifier of the target organization to be evaluated; the target organization is a financial institution, such as a bank, securities company, insurance company, or its subordinate branches.

[0029] Upon receiving an evaluation request, supporting documentation from the target organization can be obtained. Supporting documentation refers to various forms of information carriers used to support the performance evaluation conclusions. The sources of supporting documentation will be detailed in subsequent embodiments and will not be elaborated upon here.

[0030] After obtaining supporting evidence, the evidence is processed using a large-scale model. This target processing involves using the large-scale model as the core cognitive engine to understand, analyze, and reason about the supporting evidence to form an evaluation conclusion for the target institution. Unlike simply inputting the material into the large-scale model all at once and directly obtaining the output, the target processing in this embodiment incorporates several interactive triggers and feedback loops.

[0031] Specifically, during the target processing, the large model can continuously monitor the confidence level of its current processing result. The processing result can be an intermediate analysis conclusion, an understanding of supporting evidence, or a preliminary judgment of a checkpoint. Confidence level measures the degree of certainty the large model has regarding its current processing result, and can be obtained through the probability distribution, log probability, or other confidence assessment mechanisms output by the large model. When the large model determines that the confidence level of its current processing result is lower than a preset threshold, it means that the large model is not certain about the current analysis conclusion, and there may be insufficient information or ambiguity in understanding. At this time, a first interaction request can be pushed to the first device to request the first user to provide supplementary materials to eliminate uncertainty.

[0032] After receiving the first interaction request, the first user can prepare supplementary materials to address the questions or requirements raised in the request and send the first interaction feedback containing the supplementary materials through the first device. Upon receiving the first interaction feedback, the system incorporates the supplementary materials into the supporting evidence, updates the context information of the large model, and continues to perform the target processing based on the updated supporting evidence. This process can be repeated several times until the large model deems the confidence level of the current processing result sufficient, or the preset evaluation process is completed.

[0033] After the target processing is completed, the target evaluation results are obtained. These results may include evaluation conclusions for each checkpoint of the target organization, corresponding evidence citations, and a comprehensive evaluation level or score. The target evaluation results are then sent to a first device, which can generate a performance evaluation report for the target organization based on these results. This performance evaluation report is typically a structured document that can be organized into a more readable and formal evaluation report according to a preset report template.

[0034] As can be seen from the above embodiments, the method provided in this specification, by embedding several rounds of first-stage interaction and feedback loops with the user during the large model's target processing, can promptly introduce the judgment and supplementary information of human experts when the analysis results are uncertain, without changing the large model as the core analysis engine. This overcomes the shortcomings of the single-call method of the large model, which can only passively output and cannot actively verify. Simultaneously, the supplementary materials in the interactive feedback are incorporated into supporting materials and participate in subsequent processing, forming a closed-loop process where material accumulation and conclusion iteration mutually promote each other, balancing the automation efficiency of the evaluation process with the reliability of the evaluation conclusions.

[0035] Based on the above technical framework Figure 2 A flowchart illustrating a method for performance evaluation of financial institutions according to embodiments of this specification is shown. It is understood that the method disclosed in these embodiments can be executed by any device, equipment, platform, or cluster of devices with computing and processing capabilities.

[0036] See Figure 2 In one embodiment, the method includes at least the following steps: Step S201: Receiving a first request from a first device, wherein the first device corresponds to a first user, the first request includes the institution identifier of the target institution to be evaluated, wherein the target institution is a financial institution. Step S203: Responding to the first request through a large model, performing target processing on the supporting materials of the target institution to obtain a target evaluation result; wherein the target processing includes triggering several rounds of first interaction when a first condition is met, and any round of first interaction includes: pushing a first interaction request to the first device, and receiving a first interaction feedback containing supplementary materials sent by the first user in response to the first interaction request, and incorporating the supplementary materials into the supporting materials; the first condition includes that the confidence level of the current processing result of the large model is lower than a preset threshold. Step S205: Sending the target evaluation result to the first device, so that it generates a performance evaluation report for the target institution based on the target evaluation result.

[0037] The above steps will be explained in detail below.

[0038] In step S201, a first request is received from a first device, wherein the first device corresponds to a first user, and the first request includes the institution identifier of the target institution to be evaluated, wherein the target institution is a financial institution.

[0039] The first device can be any computing terminal with network communication capabilities and a user interface, such as a desktop computer, laptop computer, tablet computer, or smart mobile terminal. The first user can initiate an evaluation request through the user interface on the first device. The first user can be, for example, a compliance officer within a financial institution, an auditor appointed by a higher-level regulatory agency, or a professional from a third-party compliance assessment agency. The first request can be carried using HTTP requests, remote procedure calls, or other network communication protocols, and its payload includes the target organization's identifier.

[0040] In one specific implementation, the target institution's institutional identifier can be a unique number registered in the compliance management system. In another specific implementation, the institutional identifier can include a combination of the institution's full name, abbreviation, and location, to accurately locate the evaluation target in a multi-institutional scenario. The target institution is typically a financial institution, including commercial banks, securities companies, insurance companies, trust companies, fund management companies, and their branches or subsidiaries at various levels. It should be noted that the specific type of the target institution determines the applicable regulatory framework and set of checkpoints for subsequent performance evaluation. Different types of financial institutions differ in anti-money laundering obligations, customer due diligence requirements, and transaction monitoring standards. Therefore, specifying the target institution's institutional identifier in this step not only helps locate the evaluation target but also provides constraints on the scope of the evaluation for subsequent processing.

[0041] In step S203, the large model responds to the first request and performs target processing on the supporting materials of the target organization to obtain a target evaluation result; wherein, the target processing includes triggering several rounds of first interaction when a first condition is met, and any round of first interaction includes: pushing a first interaction request to a first device and receiving a first interaction feedback containing supplementary materials sent by a first user in response to the first interaction request, and incorporating the supplementary materials into the supporting materials; the first condition includes that the confidence level of the current processing result of the large model is lower than a preset threshold.

[0042] In this step, the large model analyzes and processes the target organization to generate evaluation results. This step will be explained in detail below from several aspects, including the acquisition of supporting materials, the basic process of target processing, user interaction triggers, various implementation forms of interaction requests, and the pause and resume mechanism of the interaction loop.

[0043] First, let me introduce how to obtain supporting materials.

[0044] Before performing target processing, the large model needs to obtain supporting materials from the target organization. Supporting materials are various forms of information carriers used to support the performance evaluation conclusions, encompassing all types of documents and data related to the target organization's performance. Depending on the specific application scenario and system configuration, the sources of supporting materials may include, for example, being provided by the first user, and / or collected by the large model itself, and / or provided by the target organization.

[0045] In one embodiment, supporting materials can be provided by a first user through a first device. When initiating an evaluation request (i.e., the first request), the first user can upload various documents related to the target organization as supporting materials through the first device. These documents may include policy documents, operation manuals, training records, transaction data reports, suspicious transaction report submission records, etc. The supporting materials provided by the first user are actual documents generated and used by the target organization in its daily operations, which can constitute an important information foundation for the large model's analysis. The method of providing supporting materials by the first user implemented in this embodiment is generally applicable to the following scenarios: the target organization's internal documents are not connected to external data platforms or knowledge bases, so the large model cannot obtain them through automated means; or the first user has already mastered a set of core materials that have been screened and organized, and directly providing these materials helps to improve evaluation efficiency and avoids introducing irrelevant or low-quality noise information when the large model collects them on its own.

[0046] In another embodiment, a large model can automatically collect supporting materials for the target organization based on its organizational identifier. This approach is typically suitable for scenarios where the target organization's supporting materials are already integrated into a data platform or knowledge base accessible to the large model. This reduces the manual workload for the first user, and the large model can selectively retrieve and collect materials based on its understanding of performance evaluation needs, often uncovering relevant information sources that the first user might have overlooked.

[0047] In another embodiment, supporting materials can be provided by the target organization. As the party being evaluated, the target organization can use its self-deployed or habitually used large model to prepare supporting materials according to the list of materials required for the performance evaluation, and then transmit the supporting materials to the large model used to respond to the first request through a first user or a first device. In one specific implementation, after step S201, a material collection request can be sent to a second device corresponding to the target organization. This material collection request may include, for example, a description of the type or scope of the required supporting materials. In response to this material collection request, the target organization uses its deployed large model to retrieve and extract relevant materials from its internal document system or data platform, and returns the supporting materials through the second device. In another specific implementation, the target organization can also proactively organize and submit supporting materials using its large model after learning about the performance evaluation, without waiting for the material collection request to be triggered. This approach is typically suitable for scenarios where the target organization has a well-developed document management system and the ability to process large models. It leverages the target organization's management capabilities over its own internal documents and business data, allowing the evaluated party to complete the initial organization and summary of supporting materials, thereby reducing the workload of the first user in collecting supporting materials from each organization and item by item.

[0048] It should be understood that in the above embodiments, the large model used by the target organization can be the same model as the large model used to respond to the first request, or they can be different models. For example, the target organization can access the large model used to respond to the first request through a network interface to prepare supporting materials, thereby sharing the inference capabilities of the same large model; or the target organization can use a large model independently deployed internally or obtained through a third-party service to prepare supporting materials and transmit the results to the large model used to respond to the first request through the first user or the first device. This specification does not limit the specific ownership and deployment method of the large model used by the target organization in the embodiments.

[0049] According to one implementation, the large model can retrieve supporting materials from a database by generating and executing structured query statements. Specifically, the large model can first generate query text for the supporting materials in natural language, including the institution's identifier. This query text describes the large model's semantic understanding of the content and scope of the required data, such as "obtain the number of all suspicious transaction reports submitted by the target institution in the most recent evaluation period and the submission time of each report." When generating this query text, the large model can combine the target institution's identifier and preset evaluation checkpoint information to determine the specific semantic content of the query. For example, when the evaluation checkpoint involves the timely submission rate of suspicious transaction reports, the query text generated by the large model can focus on the time attribute of the suspicious transaction reports. Subsequently, based on this query text, the large model can generate structured query SQL statements. The process of converting the natural language query text into SQL statements can be achieved through the large model's own code generation capabilities. The large model constructs syntactically correct SQL statements based on the semantics of the query text and known database table structure information. For example, corresponding to the query text in the example above, the large model can generate SQL statements in the following form:

[0050] SELECT report_id,submission_time

[0051] FROM suspicious_transaction_reports

[0052] WHERE institution_id = 'Target Institution Identifier'

[0053] AND submission_time BETWEEN [period_start] AND [period_end]

[0054] Next, the large-scale model invokes the target tool to execute the SQL statement, retrieves the query results from the target database, and uses these results as part of the supporting documentation. In this way, the large-scale model can automatically transform abstract information retrieval needs expressed in natural language into precise queries against structured data sources, eliminating the need for the first user to manually write query statements or log into a data platform to export data, thus significantly reducing manual operation costs. Furthermore, since the SQL query results are a structured collection of data rows, the large-scale model can directly perform numerical comparisons, statistical analysis, and compliance assessments on the structured data in subsequent target processing, achieving higher processing efficiency and accuracy compared to extracting similar information from unstructured documents.

[0055] According to another implementation, the large model can obtain relevant document fragments as supporting material by performing semantic retrieval in a knowledge base. Specifically, based on the first feature of the target institution, the large model performs a similarity-based retrieval in a knowledge base containing several documents to obtain the target context. The first feature of the target institution is key semantic information describing the institution, such as its name, business type, regulatory jurisdiction, and main business scope. In one example, the first feature can be extracted by the large model from an institutional information database based on the institution's identifier; in another example, the first feature can be constructed by the large model based on the information carried in the first request. The knowledge base stores a large number of documents, which may include at least one of the following: policy documents, historical reports, operation manuals, training records, etc. These documents are stored in raw text form and have been pre-vectorized, meaning each document fragment is mapped to a vector representation in a high-dimensional semantic space. During retrieval, the large model also maps the first feature to a semantic vector and calculates the similarity, such as cosine similarity, between the vector corresponding to the first feature and the vectors of each document fragment in the knowledge base. Several document fragments with a semantic similarity greater than a preset threshold to the first feature are selected as the target context, and the target context is incorporated into the supporting materials.

[0056] The semantic retrieval approach described above is typically suitable for processing unstructured documents. In the field of financial compliance, a large number of regulatory policy documents, historical performance evaluation reports, and internal operating manuals are unstructured documents. Their content is expressed in natural language and cannot be directly obtained through structured queries such as SQL. Through semantic retrieval, large models can automatically locate the most relevant segments to the performance evaluation of the target institution from these documents. For example, when the first feature includes the target institution's business type as "securities company," semantic retrieval can prioritize recalling policy document segments related to anti-money laundering regulatory requirements in the securities industry, as well as historical performance evaluation reports of similar securities companies, thus providing targeted references for the current evaluation. This process does not require manual annotation or classification of documents; the accuracy of the retrieval depends on the quality of the vectorized representation and the effectiveness of semantic similarity calculation.

[0057] It should be noted that the two self-collection methods mentioned above—database query-based and semantic retrieval-based—can be applied individually or in combination. In a combined application example, the large model can first obtain the quantitative indicator data of the target institution through database query, and then obtain the relevant policy background and historical evaluation information through semantic retrieval, using both as supporting materials for self-collection. This combined approach can provide the large model with multi-dimensional and multi-source information support, helping to improve the comprehensiveness and depth of subsequent target processing and analysis.

[0058] After acquiring supporting evidence, the large-scale model can perform target processing on it. Target processing refers to the process by which the large-scale model, based on its pre-trained knowledge and reasoning capabilities, performs a series of operations on the supporting evidence, including semantic understanding, information extraction, logical verification, and compliance analysis, gradually forming an evaluation conclusion for the target institution. Target processing can be performed by the large-scale model in a step-by-step reasoning manner. In each step of reasoning, the large-scale model can focus on a specific checkpoint, extract information related to that checkpoint from the supporting evidence, compare the extracted information with applicable regulatory requirements, and form an analytical conclusion and corresponding confidence assessment for that checkpoint.

[0059] In one embodiment, the supporting materials may include first materials in a non-native text format. The first materials could be, for example, scanned documents stored in PDF format, data reports stored in Excel format, screenshots or photos stored in PNG or JPG format, etc. Since the core input of a large model is usually a text sequence, non-native text format materials cannot be directly read and analyzed by the large model without processing. In this case, the target processing also includes multimodal processing of the first materials. Specifically, the first materials can be converted into an image format to obtain a first image. For PDF format materials, they can be rendered page by page into high-resolution PNG images; for Excel format materials, they can be rendered worksheet by worksheet into images; for materials that are originally in image format, they can be directly used as the first image. This unified conversion of the first materials into image format preserves the complete visual information in the original materials, including unstructured visual elements such as text layout, table structure, charts, stamps, and handwritten annotations, avoiding information loss or formatting errors caused by using traditional text parsing libraries to extract content. Subsequently, the first image is processed using a visual model to obtain the first text corresponding to the first materials. The visual model, pre-trained on large-scale image and text data, is capable of optical character recognition (OCR) of text within images, while also understanding table structures, chart relationships, and layout. The first text output by the visual model not only includes the text extracted from the image but also structured semantic information such as descriptions of table structures and interpretations of chart data. After obtaining the first text, it can replace the initial supporting material, transforming non-native text formats that the large model cannot directly process into a text representation that the model can accept and understand, incorporating it as context into subsequent target processing steps. This multimodal processing enables the large model to be compatible with various common file formats such as PDF, Word, Excel, and images during performance evaluation, broadening the range of supporting material formats that automated evaluation can handle and reducing the workload of the initial user in pre-converting the materials.

[0060] During the target processing, the large model can monitor the confidence level of the current processing result. Confidence level is a quantitative indicator that measures the certainty of the large model regarding its current processing result. When generating the analysis conclusion at each step, the large model can usually simultaneously generate the probability distribution or log probability value corresponding to that conclusion. For example, when the large model judges a certain checkpoint as "compliant," it can output a confidence score for that judgment, which reflects the degree of certainty the large model has in making that judgment based on the current supporting materials. In one specific implementation, the confidence level can be obtained by normalizing the log probability output by the large model; in another specific implementation, the confidence level can be obtained by evaluating the consistency of the results from multiple samplings of the large model; in other implementations, a separate confidence level evaluation module can be used to predict the confidence level based on the hidden layer representation output by the large model.

[0061] The preset threshold is a parameter that can be adjusted according to actual application needs. A higher threshold means stricter requirements for the reliability of the analysis conclusions, and the frequency of triggering user interaction may increase accordingly; a lower threshold allows large models to output conclusions under slightly lower certainty, and the frequency of triggering user interaction decreases accordingly. In actual deployment, this threshold can be flexibly set according to factors such as the evaluation complexity of the target organization, available human resources, and timeliness requirements.

[0062] When the large model determines that the confidence level of the current processing result is lower than a preset threshold (i.e., the first condition is met), it means that the large model lacks certainty in its current analysis conclusion. Possible reasons include: the supporting materials lack information directly related to the current checkpoint; the supporting materials contain contradictory statements, making it impossible for the large model to determine which to follow; the large model has ambiguities in its understanding of certain industry-specific terms or business logic in the materials; or the judgment requires consideration of the target organization's specific business practices, and the current materials lack relevant explanations of these practices. In these cases, if the large model draws a conclusion under low confidence conditions, it may lead to inaccurate final evaluation results, thereby affecting the seriousness and authority of the performance evaluation. Therefore, when the first condition is met, a first round of interaction can be triggered.

[0063] Specifically, once the large model determines that the confidence level of the current processing result is below a preset threshold, the current processing can be paused. Pausing means saving the current inference state of the large model. The inference state can include: contextual information of the currently processed supporting evidence, completed inference steps and corresponding intermediate conclusions, the currently executing inference step and its progress, as well as the attention state and hidden layer representations within the large model. Pausing can be achieved by serializing and storing the above inference state in a cache or persistent storage medium so that it can be fully restored when processing resumes later.

[0064] After the pause, the large model can push a first interaction request to the first device. The first interaction request may include: a detailed description of the current insufficient confidence level to help the first user understand what obstacles the large model has encountered; and clear questions or requests to guide the first user to provide targeted supplementary materials.

[0065] After receiving the first interaction request, the first user can consult relevant materials, conduct internal communication with the target organization (personnel) or arrange interviews in response to the questions or requests in the first interaction request to obtain supplementary materials, and send the first interaction feedback containing the supplementary materials through the first device.

[0066] Upon receiving the initial feedback, supplementary materials can be incorporated into the supporting evidence to update the contextual information of the large model. In other words, the large model adds supplementary materials as new usable information to the existing supporting evidence. Afterward, the large model can resume its current processing, resuming execution from the pause point. At this point, the large model can reanalyze and reason about the current checkpoint based on the newly added supplementary materials. Because the supplementary materials directly address the aspects of the large model where certainty is insufficient, they often effectively eliminate the original information gaps or ambiguities, thereby increasing the confidence of the conclusions obtained after the large model's reanalysis.

[0067] The process of the first interaction described above can be repeated several times. The trigger condition for each round of interaction can be that the confidence level of the current processing result is lower than a preset threshold, as listed above. Supplementary materials from each round of interaction are incorporated into the supporting evidence, ensuring that the information base available for large-scale model analysis continuously enriches with each round of interaction. This cumulative mechanism allows subsequent processing of the large model to be based on more comprehensive information, thereby gradually increasing the confidence level of the analytical conclusions until the requirements are met or the preset performance evaluation process has been completed.

[0068] Besides the initial interaction described above, target processing can also include other forms of interaction with the user. The specific content and form of the interaction request included in each interaction can vary depending on the processing stage at which the interaction is triggered and the specific reason for triggering the interaction. Two typical interaction request implementation forms are described below.

[0069] The first type of interaction request may include follow-up questions generated by the large model based on the current processing results. These follow-up questions are derived from the large model's analysis of supporting materials. After analyzing the supporting materials, the large model may find that some key information is missing, the description is unclear, or further verification is needed. In this case, the large model can automatically generate targeted follow-up questions based on the information gap between the already known material content and the current analysis requirements. For example, when analyzing a branch's suspicious transaction monitoring process, the large model learns from the supporting materials that the branch has deployed a transaction monitoring system, but the materials do not specify whether the system's monitoring threshold parameter settings meet regulatory requirements. The large model can generate the follow-up question: "Please provide the threshold parameter configuration record of the branch's transaction monitoring system, including the current settings of the large transaction monitoring threshold and the suspicious transaction monitoring threshold, and the date of the most recent adjustment." This follow-up question generation process demonstrates the large model's understanding of evaluation criteria and its ability to deeply analyze material content; the large model not only identifies explicit information gaps but also correlates these gaps with the requirements of evaluation checkpoints, generating precise question statements to guide the first user to efficiently supplement the required information.

[0070] Furthermore, follow-up questions can also be used to guide the first user in conducting in-person interviews with the target institution. In financial compliance performance evaluation, in addition to reviewing written materials, interviewing relevant personnel at the evaluated institution is also an important means of obtaining information. The large-scale model can generate follow-up questions to guide in-person interviews based on any doubts or matters requiring verification discovered during its material analysis. For example, if the large-scale model finds that an institution's description of customer due diligence materials is too general, it can generate the following follow-up question: "Please confirm with the institution's compliance officer what specific steps are included in the enhanced due diligence measures implemented for high-risk clients, and the coverage rate of enhanced due diligence for high-risk clients in the most recent quarter." The first user can then conduct interviews based on this information and return the information obtained as supplementary material. This supplementary material can include information from the first user based on the in-person interview record, such as interview summary, key points of the interviewee's responses, or a summary of the situation compiled by the first user based on the interview content. In this way, the large-scale model's follow-up questions combine the model's knowledge with the communication skills of performance evaluation experts, effectively supplementing the written materials.

[0071] Secondly, the target processing may also include a second interaction triggered when the second condition is met. During target processing, after the large model completes a preliminary analysis of a certain checkpoint or all checkpoints based on existing supporting materials, it can generate a preliminary evaluation result, at which point the second condition is met. The preliminary evaluation result is an intermediate version of the target evaluation result, containing the large model's preliminary judgment on each checkpoint, the corresponding evidence cited, and confidence information. In this case, the second interaction can be triggered. The second interaction may include: pushing a second interaction request to the first device to display the preliminary evaluation result to the first user, enabling the first user to fully review the large model's analysis conclusions. For example, the first user can view the various contents of the preliminary evaluation result through the first device's interface, including the evaluation conclusion for each checkpoint, the supporting material fragments, and the confidence level indication given by the large model.

[0072] The first user, based on their professional experience, judges the accuracy of each conclusion. In the second interaction, the first user can provide confirmation information regarding this preliminary evaluation result. Confirmation information can include acceptance of correct conclusions, correction of incorrect conclusions, supplementary explanations for omitted checks, or corrections of inappropriate evidence citations, etc. In one example, the first user can check off each conclusion they approve or reject, and attach corrections to rejected conclusions; in another example, the first user can annotate the text of the preliminary evaluation result, and return the annotated version as confirmation information.

[0073] Through the above steps, a systematic verification of all analytical conclusions of the large model can be achieved in the second interaction, rather than asking questions point by point. Compared to the first interaction, which triggers questions one by one during the target processing, the method of confirming the preliminary evaluation results through the second interaction is usually suitable for scenarios where the analysis work is basically completed and overall review is required.

[0074] After the target processing is completed, the large model can generate target evaluation results. These results are the final evaluation conclusions formed by the large model after a comprehensive analysis of supporting materials, combined with supplementary materials obtained in several rounds of first-round interactions and / or confirmation information obtained in second-round interactions. The target evaluation results can include evaluation conclusions for various checkpoints of the target organization, such as item-by-item assessments of the soundness of anti-money laundering strategies, the quality of customer due diligence, the effectiveness of transaction monitoring, and the timeliness and accuracy of suspicious transaction reports. The conclusion for each checkpoint can be linked to specific segments of the supporting or supplementary materials on which it is based, providing evidentiary support. Furthermore, the target evaluation results may also include a comprehensive evaluation rating or score, which is a comprehensive quantitative reflection of the target organization's performance on each checkpoint.

[0075] Next, in step S205, the target evaluation result can be sent to the first device, so that it can generate a performance evaluation report for the target organization based on the target evaluation result.

[0076] A performance evaluation report is typically a structured document. Following a pre-defined report template, it organizes the conclusions of each checkpoint in the target evaluation results, evidence citations, and overall evaluation level into a highly readable document format. The report template may include a report title, evaluation period, basic information about the target organization, detailed evaluations of each checkpoint, overall evaluation level or score, and appendices (such as a list of evidence materials). The first device fills in the data from the target evaluation results into the corresponding positions in the report template, generating a complete performance evaluation report. The first user can view, download, print, or archive this performance evaluation report using the first device.

[0077] In one embodiment, the performance evaluation report includes information on the sources of evidence upon which the target evaluation results are based. For example, each evaluation conclusion in the report may be accompanied by a footnote or hyperlink pointing to specific excerpts of supporting or supplementary materials that form the basis of that conclusion. This traceability design allows reviewers of the report (such as higher-level regulatory agencies or internal audit departments) to quickly locate the sources of information for the evaluation conclusions, verify the objectivity and accuracy of the evaluation conclusions, thereby improving the transparency of the evaluation process and the credibility of the evaluation results.

[0078] The foregoing, based on one or more embodiments, details a method for performance evaluation of financial institutions. Using the method provided in the embodiments of this specification, during the analysis and processing of supporting materials using a large model, multiple rounds of interaction with users can be triggered based on confidence level monitoring. Supplementary materials provided by users are incorporated into the supporting materials, driving the large model to continue analysis based on more comprehensive information. This achieves a closed-loop synergy between machine analysis capabilities and human judgment experience while retaining the large model as the core analysis engine. While improving the efficiency of performance evaluation, it retains key nodes for human supervision and confirmation in financial compliance scenarios, balancing the degree of automation with the reliability and compliance of the evaluation conclusions.

[0079] In this specification, the terms "first" in phrases such as "first interactive request" and "first device," as well as the corresponding terms "second" and "third" (if any) in the text, are used merely for the convenience of distinction and description and do not have any limiting meaning.

[0080] The foregoing description describes specific embodiments of this specification; other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than those shown in the embodiments, and the desired result may still be achieved. Furthermore, the processes depicted in the drawings do not necessarily need to follow the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0081] Figure 3 This is a schematic diagram of an apparatus for evaluating the performance of financial institutions according to an embodiment of this specification. The apparatus 300 is deployed in a computing device, which can be implemented using any device, equipment, platform, or device cluster with computing and processing capabilities. The apparatus 300 includes:

[0082] The receiving module 301 is configured to receive a first request from a first device, wherein the first device corresponds to a first user, and the first request includes an institution identifier of the target institution to be evaluated, wherein the target institution is a financial institution.

[0083] The processing module 302 is configured to respond to the first request through a large model, perform target processing on the supporting materials of the target institution, and obtain a target evaluation result; wherein, the target processing includes triggering several rounds of first interaction when a first condition is met, and any round of first interaction includes: pushing a first interaction request to a first device, and receiving a first interaction feedback containing supplementary materials sent by a first user in response to the first interaction request, and incorporating the supplementary materials into the supporting materials; the first condition includes that the confidence level of the current processing result of the large model is lower than a preset threshold.

[0084] The sending module 303 is configured to send the target evaluation result to the first device, so that the device generates a performance evaluation report for the target organization based on the target evaluation result.

[0085] According to another embodiment, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the foregoing combinations. Figure 2 The steps of the method are described.

[0086] According to yet another embodiment, this specification also provides a computing device including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the foregoing combination. Figure 2 The steps of the method are described.

[0087] Those skilled in the art will recognize that the functions described in the embodiments of the present invention in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0088] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, or improvements made based on the technical solutions of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for evaluating the performance of financial institutions, comprising: Receive a first request from a first device, wherein the first device corresponds to a first user, and the first request includes an institution identifier of the target institution to be evaluated, wherein the target institution is a financial institution; The large model responds to the first request and performs target processing on the supporting materials of the target institution to obtain a target evaluation result; wherein, the target processing includes triggering several rounds of first interaction when a first condition is met, and any round of first interaction includes: pushing a first interaction request to a first device and receiving a first interaction feedback containing supplementary materials sent by a first user in response to the first interaction request, and incorporating the supplementary materials into the supporting materials; the first condition includes that the confidence level of the current processing result of the large model is lower than a preset threshold; The target evaluation results are sent to the first device, which generates a performance evaluation report for the target organization based on the target evaluation results.

2. The method according to claim 1, wherein, The first interaction request includes follow-up questions generated by the large model based on the current processing result; the current processing result is obtained by the large model based on the analysis of supporting materials.

3. The method according to claim 1, wherein, The supporting materials were provided by the first user through the first device.

4. The method according to claim 1, wherein, The supporting materials include first materials in a non-native text format; the target processing also includes: The first material is converted into an image format to obtain the first image; The first image is processed using a visual model to obtain the first text corresponding to the first material; Replace the first material with the first text.

5. The method according to claim 1, wherein the target processing further comprises: The second interaction is triggered when the second condition is met, the second condition including that the large model generates preliminary evaluation results; The second interaction includes: pushing a second interaction request to the first device to display the preliminary evaluation result to the first user, and receiving confirmation information from the first user regarding the preliminary evaluation result.

6. The method according to claim 1, wherein, Before performing target processing on the supporting materials of the target organization, the method further includes: Using a large model, supporting materials for the target organization are collected based on its organizational identifier.

7. The method according to claim 6, wherein, The supporting materials collected from the target organization include: Using the large model, query text for supporting materials is generated in natural language, including the institution's identifier; Based on the query text, generate a structured query SQL statement; The target tool was invoked to execute the SQL statement, and supporting evidence was obtained.

8. The method according to claim 6, wherein, The supporting materials collected from the target organization include: Using the large model, based on the first feature of the target organization, a similarity-based retrieval is performed in a knowledge base containing several documents to obtain the target context; the documents include at least one of the following: policy documents, historical reports, operation manuals, and training records; the target context includes several document fragments with a semantic similarity to the first feature greater than a preset threshold. The target context is incorporated into the supporting material.

9. The method according to claim 1, wherein, After pushing the first interaction request to the first device, the method further includes pausing the current processing of the large model; Furthermore, after incorporating the supplementary material into the supporting material, the current processing of the large model continues.

10. An apparatus for evaluating the performance of financial institutions, comprising: The receiving module is configured to receive a first request from a first device, wherein the first device corresponds to a first user, and the first request includes an institution identifier of the target institution to be evaluated, wherein the target institution is a financial institution. The processing module is configured to respond to the first request through a large model, perform target processing on the supporting materials of the target institution, and obtain a target evaluation result; wherein, the target processing includes triggering several rounds of first interaction when a first condition is met, and any round of first interaction includes: pushing a first interaction request to a first device, and receiving a first interaction feedback containing supplementary materials sent by a first user in response to the first interaction request, and incorporating the supplementary materials into the supporting materials; the first condition includes that the confidence level of the current processing result of the large model is lower than a preset threshold; The sending module is configured to send the target evaluation result to the first device, so that the device generates a performance evaluation report for the target organization based on the target evaluation result.

11. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1-9.

12. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-9.

Images