Unstructured cue optimization scoring method for smart credit audit assistants

Abstract

The application discloses an unstructured prompt word optimization scoring method for an intelligent credit audit assistant. The method comprises collecting unstructured credit audit text of an enterprise, wherein the credit audit text comprises credit audit materials, due diligence reports and post-loan inspection document data; the collected credit audit text is blocked and indexed to construct an intelligent audit vector knowledge base; a large language model is used to analyze the credit audit text, generate an audit opinion on the credit of a client, and give an enterprise risk score; the enterprise risk score is associated with an actual default risk index to construct a loss function for measuring the difference between the two, the large model prompt is regarded as a training parameter, and the prompt is iteratively optimized through back propagation; and the enterprise risk score is visually displayed through a visual means. The application can effectively solve the subjective problem in risk measurement, improve the performance of the model in tasks without standard answers, and provide a more reliable basis for intelligent credit audit.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of financial risk control, and particularly relates to an unstructured prompt word optimization scoring method for an intelligent credit audit assistant. BACKGROUND

[0002] In the current credit risk management field, traditional scoring methods are facing multi-dimensional challenges, and their limitations mainly lie in three core aspects: the limited structured data dimension leads to a single risk assessment perspective, the key risk signals in unstructured text are not effectively captured, and the subjective judgment bias in the manual audit process is difficult to eliminate. These pain points jointly restrict the accuracy and efficiency of risk assessment in credit audit.

[0003] In view of the above problems, there are explorations of risk assessment based on natural language processing in the prior art. For example, through the semantic understanding ability of a large language model, text information vectorization and risk feature extraction are realized. Although the large language model has a certain semantic understanding ability, enterprise credit audit and risk scoring belong to a typical "no standard answer" task, and its rationality needs to be verified by the actual default of the enterprise. In the prior art, the prompt word design of the large language model relies on manual experience, and there is also a lack of effective verification of the rationality of its evaluation results, resulting in a deviation between the subjective evaluation results of the model on enterprise credit audit and the objective situation of the enterprise. SUMMARY

[0004] The purpose of the present application is to provide an unstructured prompt word optimization scoring method for an intelligent credit audit assistant to overcome the deficiencies in the prior art.

[0005] To achieve the above purpose, the present application provides an unstructured prompt word optimization scoring method for an intelligent credit audit assistant, comprising:

[0006] Step 1, collecting credit audit materials, due diligence reports and post-loan inspection document data of enterprises;

[0007] Step 2, block and index the collected data, and construct an intelligent audit vector knowledge base;

[0008] Step 3, based on the intelligent audit vector knowledge base and using a large language model, analyzing the credit audit text of the current enterprise, generating an audit opinion on the client credit, and giving an enterprise risk score;

[0009] Step 4, associating the enterprise risk score with the actual default risk indicator, constructing a loss function to measure the difference between the two, regarding the large model prompt as a training parameter, and iteratively optimizing the prompt through back propagation;

[0010] Step 5, visually displaying the enterprise risk score through a visual means.

[0011] Further, the credit audit material includes the basic situation of the enterprise client, the pre-loan credit access situation, the loan withdrawal information, the early warning information, the overdue information, the post-loan collection situation and the lawsuit situation.

[0012] Further, the process of constructing the vector knowledge base in step 2 includes:

[0013] Based on the hierarchical structure and segment semantics of the credit audit text, the document is processed by block, then the pre-trained text embedding model is used to convert the document block into a dense text vector with unified dimensions, and finally the generated document block vector is structured labeled according to the customer unified number, credit unified code and timestamp, and stored in a vector database supporting efficient search.

[0014] Further, the step 3 specifically includes:

[0015] Based on the prompt engineering paradigm to design prompt templates, the large language model identifies and extracts risk points in enterprise credit, and forms an audit report, and gives a risk score of the enterprise based on the audit report .

[0016] Further, the loss function is:

[0017] ;

[0018] Wherein, is the output result of the loss function, is the actual default risk index of the enterprise, and the default loss rate is selected; represents the mean square error, which is used as a measurement index of the rationality of the enterprise risk score.

[0019] Further, the text gradient used to optimize the input prompt words is calculated by back propagation in step 4, and the gradient calculation method of the optimization process is as follows:

[0020] ;

[0021] Wherein, represents the first order partial derivative of the loss function to the risk score , represents the input prompt of the large language model, represents that the gradient is calculated and output by the large language model in the form of text, represents the optimization gradient of the input prompt word.

[0022] Further, the input prompt word is constantly iteratively optimized in combination with the text gradient and the existing input prompt word:

[0023] ;

[0024] wherein, represent the input prompt words after the first iterative optimization, represent the input prompt words after the first iterative optimization.

[0025] Further, a validation set val is set, and a stop criterion for prompt optimization is set to prevent overfitting, as follows:

[0026] ;

[0027] wherein, is the new prompt word loss on the validation set, is the current optimal loss set.

[0028] Further, the enterprise risk score is displayed through the score dashboard in step 5, and the trend curve of the score in previous years is displayed, so that the enterprise risk evaluation result is intuitively reflected.

[0029] Beneficial effects: 1. The present application combines a large amount of unstructured data in the field of credit audit (including but not limited to customer interview records, due diligence reports, post-loan inspection documents, etc.), realizes the evaluation of enterprise credit risk, and compared with the limitations of traditional methods relying on simple structured data, the application of multi-source unstructured data can help banks to more comprehensively evaluate the credit risk level of enterprises;

[0030] 2. The present application uses a large language model (LLM) to process credit audit text, effectively compensates for the defects of traditional models in the utilization of unstructured information, refers to the concept of prompt engineering paradigm in the process of designing prompt words, and uses role prompts, thought chain prompts and other methods to significantly improve the stability and accuracy of the output content of the large language model;

[0031] 3. The present application associates the subjective evaluation result of the large language model with the real default index of the enterprise, constructs a loss function and back propagates, realizes the joint optimization of the risk index and the prompt word; this mechanism effectively solves the subjective bias problem caused by relying on artificial interpretation or fixed prompt words in traditional credit audit, makes the evaluation result of the bank on the credit risk of the enterprise highly match the actual default situation of the enterprise, and significantly improves the reliability of the evaluation result in the scene without standard answer. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a structural schematic diagram of the unstructured prompt optimization scoring method for the intelligent credit audit assistant of the embodiment of the present application. DETAILED DESCRIPTION

[0033] The application will be further clarified below in conjunction with the accompanying drawings and specific embodiments, which are implemented on the premise of the technical solutions of the application. It should be understood that these embodiments are only used to illustrate the application and not to limit the scope of the application.

[0034] As Figure 1 shown, the embodiment of the application provides an unstructured prompt word optimization scoring method for an intelligent credit audit assistant, comprising:

[0035] Step 1, collect the credit audit materials, due diligence reports and post-loan inspection document data of the enterprise. Among them, the credit audit materials include enterprise customer basic information, pre-loan credit access information, in-loan withdrawal information, early warning information, overdue information, post-loan collection information, legal action information and other data.

[0036] Step 2, block and index the collected data, and construct an intelligent audit vector knowledge base. Specifically, the credit audit text collected in step 1 is used as the core input, and a hybrid blocking method of "semantic coherence limit + fixed length bottom-up" is adopted. First, the natural paragraph structure of the document is used for preliminary splitting to ensure that each text block contains a complete semantic unit. For long paragraphs that exceed the preset length, further splitting is performed according to punctuation marks such as periods and semicolons. For text blocks that still exceed the length, strictly cut according to the maximum length limit to ensure that the length of a single text block meets the requirements; further, a pre-trained text embedding model (such as Zhipu Embedding 3 or gte-Qwen2) is used to convert the text block into a vector; a vector database (such as Mivus, FAISS) supporting approximate nearest neighbor (ANN) search is selected as the storage carrier, and the vectorized text segments are batch stored in the vector database supporting efficient search in the structure of "timestamp-client unified number-credit unified code-text block vector-original text block".

[0037] Step 3, based on the intelligent audit vector knowledge base and using a large language model, analyze the current enterprise's credit audit text, generate an audit opinion on the client credit, and give an enterprise risk score. Specifically, based on the prompt engineering paradigm, a prompt template is designed to let the large language model identify and extract risk points in enterprise credit and form an audit report, and based on the audit report, an enterprise risk score is given . According to the unstructured prompt word, the cosine similarity is used to find relevant knowledge and data from the intelligent audit vector knowledge base, and the large model is based on the retrieved knowledge to output the audit opinion of the entire client credit according to the self-recurrence logic. The calculation method of the cosine similarity is as follows:

[0038] ;

[0039] Among them, is the embedding vector of the user question. An embedded word vector of one of the intelligent audit knowledge bases; The module length of the word vector in the high-dimensional semantic space.

[0040] Calculate The cosine similarity of all From the intelligent audit vector knowledge base, filter out the top N most relevant knowledge pieces, and integrate the "user question + top N relevant knowledge" into an enhanced prompt word. The enhanced prompt word needs to be encoded by the deep learning model architecture Transformer into a "dynamic semantic representation", that is, by capturing the contextual dependency between tokens, a semantic vector strongly bound to the specific context is generated, which can accurately reflect the true semantics of the text in different scenarios. The final output of Transformer is the "probability distribution of the next token", and the model selects the token according to the probability, appends the new token to the context, and repeats the "modeling → generation" cycle until the complete credit approval opinion is output.

[0041] Step 4, associate the enterprise risk score with the actual default risk indicator, build a loss function to measure the difference between the two, and treat the large model prompt as a training parameter, and iterate the optimization of the prompt. Specifically, use the risk score output by the large language model in step 3 The actual default risk indicator of the enterprise is the default loss rate As the core variable, use the mean square error to build the loss function:

[0042] ;

[0043] Where, The output result of the loss function, The actual default risk indicator of the enterprise is the default loss rate; Represent the mean square error, which is a measure of the rationality of the enterprise risk score. Specifically, The default loss rate refers to the amount of loss suffered by the bank or other financial institutions when the borrower defaults on the loan, expressed as a percentage of the total risk exposure at the time of default. The Total amount of a financial institution is calculated using cumulative losses and risk exposure after evaluating all outstanding loans, and is calculated as follows:

[0044] ;

[0045] Where, The total amount of risk exposure refers to the total amount of outstanding exposure that the bank or financial institution assumes for the loan in the event of default. R is the total amount of recoverable funds, which refers to the total amount of funds actually recovered through the disposal of collateral, recovery, and other means after the loan defaults.

[0046] The input prompt of the large language model is taken as the parameter to be optimized. The text gradient of the prompt to the loss function is calculated by back propagation, as follows:

[0047] ;

[0048] wherein, represents the first-order partial derivative of the loss function to the risk score , represents the input prompt of the large language model, represents that the gradient is calculated by the large language model in the form of text, represents the optimization gradient of the input prompt.

[0049] The text gradient is transferred from the loss function layer to the prompt layer in reverse, and the dimensions that need to be optimized in the prompt are determined (such as focusing on the analysis dimension, refining the granularity of the score standard, etc.). Update the prompt based on the text gradient:

[0050] ;

[0051] wherein, represents the input prompt after the th iteration of optimization, represents the input prompt after the th iteration of optimization.

[0052] It also includes setting a validation set val and setting the stopping criteria for prompt optimization to prevent overfitting, as follows:

[0053] ;

[0054] wherein, is the loss of the new prompt on the validation set, is the current optimal loss set.

[0055] Step 5, visually display the enterprise risk score through visualization. Specifically, the enterprise risk score can be displayed through a score dashboard, and the trend curve of the score over the years can be displayed to intuitively reflect the results of enterprise risk evaluation.

[0056] For example:

[0057] XX Precision Machinery Co., Ltd. (hereinafter referred to as "XX Precision") is selected as a case enterprise. The selection is based on the general credit granting scenario for manufacturing enterprises in bank credit business, and strictly follows the "Enterprise Customer Basic Information" data collection standard. By setting specific background information and credit demand, an analysis sample close to the actual business of the bank is constructed.

[0058] The specific analysis process is:

[0059] 1. Data collection and preprocessing

[0060] Basic business information constitutes the core basis of enterprise qualification evaluation, covering key elements such as registered capital and equity structure, and can intuitively reflect enterprise size and governance structure. The financial statements of the past three years as standardized financial data carriers contain structured information such as balance sheets, profit tables and cash flow tables, providing data support for quantitative analysis of enterprise solvency, profitability and operational efficiency.

[0061] Synchronous collection of multiple types of unstructured data: audit materials include enterprise customer basic information, pre-loan credit access information, in-loan withdrawal information, early warning information, overdue information, post-loan collection information, legal action information, etc., covering the entire life cycle of enterprise credit; due diligence reports contain management interview records and industry risk analysis, which can mine enterprise strategic planning and industry environment adaptability; post-loan inspection documents reflect the dynamic changes of enterprise operating conditions through regular tracking records.

[0062] 2. Vector knowledge base construction and product description generation

[0063] With text as input, the mixed block method of "semantic coherence + fixed length bottom" is used for text splitting. This method first divides semantic units according to the internal logical structure of the document (such as chapter titles like "in-loan withdrawal record" and "post-loan collection record"), and then uses a fixed length constraint of 200-300 words as a bottom mechanism to ensure that the text blocks maintain semantic integrity and control the data size to adapt to the input requirements of subsequent vectorization models.

[0064] Secondly, the Zhipu Embedding 3 model is used to convert the split text blocks into 768-dimensional dense vectors. The 768-dimensional vector dimension design balances between semantic expression ability and computational efficiency, which can capture deep semantic features of text and reduce resource consumption in subsequent retrieval and calculation through dense vector format.

[0065] Finally, to achieve traceability and efficient retrieval of data, a three-dimensional labeling structure of "timestamp-customer unified number-credit unified code" is used to uniquely identify the vector data. Taking a case customer as an example, the labeling format is "20250831-KH2025001-XD20250831", where the timestamp (20250831) records the data processing time, the customer number (KH2025001) is associated with the enterprise subject information, and the credit code (XD20250831) corresponds to the specific credit application project. The labeled vector data is stored in the Milvus vector database, and by constructing a structured vector index system dedicated to the case customer, the conversion of unstructured data to a "semantic vector + structured metadata" dual-dimensional management mode is realized, providing low-latency and high-relevance feature retrieval support for the credit scoring model.

[0066] 3. Enterprise credit audit opinion generation and risk scoring

[0067] Prompt template design is a core link to ensure that the large model outputs meet the professional requirements of credit audit. The case-specific prompt template integrates role positioning, task boundaries, and output specifications through structured design, providing a clear operational framework for the large model, thereby improving the accuracy and consistency of risk identification.

[0068] Role definition module: Clearly define the professional identity and core responsibilities of the large model, set as "You are a senior credit auditor of the bank, need to identify risk points based on the provided materials". This role positioning enables the model to simulate the thinking paradigm of professional auditors, focusing on key risk dimensions in credit evaluation and avoiding interference from non-professional perspectives. By anchoring the large model in the "senior auditor" role, the depth of interpretation of complex financial data and industry information can be effectively improved, ensuring that the analysis process meets the professional standards of bank credit audit.

[0069] Task instruction module: Based on the thinking chain prompt paradigm, instruct the model to "extract risk points from solvency, operational stability, industry risk, and post-loan repayment enthusiasm, and label the severity (high / medium / low) and text block number for each risk point". This instruction ensures the comprehensiveness and traceability of the analysis through three constraints: first, limit the core dimensions of risk assessment, covering the main sources of enterprise credit risk; second, require risk level quantification to avoid ambiguous statements; third, force the association of original text references to provide clear clues for subsequent manual review. This structured task design effectively reduces the randomness of model output, making the risk identification process repeatable.

[0070] Output format module: specifies the standardized result presentation method, requires output in the fixed format of "audit opinion: [risk point description]; risk score (0-100 points): [score]; score basis: [text block reference]". This format realizes the readability and comparability of the results through the binding of three elements: the risk point description needs to accurately summarize the nature of the problem, the risk score realizes the fine distinction of the risk level through the continuous quantitative scale of 0-100 points, and the text block reference ensures the traceability of the scoring results.

[0071] For the precise credit audit scene of XX, the system extracts key information from the vector knowledge base: "2024Q2 inspection report shows that 'the main supplier cooperation agreement will expire in January 2025'" and other important related information, and inputs it into the large language model for analysis.

[0072] Based on the semantic understanding and risk factor extraction of the input text, the model outputs structured audit opinions according to the industry risk assessment framework. Specifically, its analysis dimensions include solvency, operating stability, industry risk, and post-loan repayment enthusiasm: in the solvency dimension, by comparing the enterprise's current ratio (1.8) with the industry average (1.5), the risk level is determined to be "low"; in the operating stability dimension, the model identifies that the expiration of the core supplier cooperation may cause a risk of raw material supply disruption, and the risk level is rated as "medium"; in the industry risk dimension, combined with the macro data of a 12% decline in downstream automobile manufacturing demand, the risk level is determined to be "high"; in the solvency dimension, the company has not responded to multiple collection calls, and the risk level is determined to be "high".

[0073] Based on multi-dimensional risk qualitative analysis, the model quantifies and integrates each risk by setting weights in the prompt words, and finally generates an initial risk score = 68 points.

[0074] 4. Prompt word iterative optimization

[0075] The actual default loss rate of this enterprise is 25%, corresponding to an actual risk score of 25 points. The mean square error (MSE) is used as the loss function. The initial output score of the model (68 points) and the actual risk score (25 points) are substituted into the MSE formula to calculate the loss function value.

[0076] Using the principle of backpropagation, the initial prompt words are used as parameters, and the text gradient of the loss function with respect to the prompt words is calculated by the large language model to accurately locate the source of the deviation. The analysis results show that "insufficient industry risk weight" is the core factor leading to the deviation of the model output, which means that the initial prompt words have not fully guided the model to conduct in-depth assessment of the industry environmental risk.

[0077] To address this issue, the large model automatically adjusts the prompt to: "Add 'industry demand fluctuation impact on enterprise revenue' analysis dimension, and require quantitative evaluation (such as 'downstream demand each down 1%, enterprise revenue expected to decrease X%'). By introducing specific quantitative indicators, the model begins to directly link industry dynamics with enterprise financial performance, initially improving the accuracy of the evaluation.

[0078] After 3 iterations of optimization, the final prompt forms a structured analysis framework:

[0079] From the perspectives of debt servicing ability, operational stability, industry risk, and post-loan repayment willingness, the industry risk score weight is increased to 40%.

[0080] 5. Visualize the results

[0081] The dashboard homepage highlights the customer's current real-time risk score of 48 points with a red warning symbol, which is significantly lower than the bank's set safety threshold of 60 points, directly triggering the high-risk warning mechanism. This design ensures that auditors can quickly identify the risk level of the customer when logging into the system, providing priority guidance for further in-depth review.

[0082] The line chart shows the score change trajectory from 2022 to 2024, with data showing that the customer's score has continuously declined from 72 points in 2022 to 48 points in 2024, with a 15% annual compound decline. The trend curve clearly presents the acceleration characteristics of risk deterioration: the decline from 2022 to 2023 is 16.7% (72→60), and the decline from 2023 to 2024 is further expanded to 20% (60→48), reflecting the increasing cumulative effect of risk factors.

[0083] The risk factor contribution pie chart visually displays the weight distribution of different core risk sources: industry risk accounts for 35%, supply chain risk accounts for 30%, debt servicing ability risk accounts for 25%, and post-loan repayment willingness risk accounts for 10%. Among them, industry risk as the primary contributing factor prompts auditors to focus on the periodic fluctuations, policy regulation influences, and market competition pattern changes of the customer's industry; supply chain risk points to the stability problems of the customer's upstream and downstream cooperation, which needs further verification of supplier dependence and logistics support capacity; debt servicing ability risk reflects the health status of enterprise cash flow, which needs to be cross-verified with financial indicators such as current ratio and quick ratio; post-loan repayment willingness needs immediate follow-up by the collection personnel.

[0084] The above is only the preferred embodiment of the present application, it should be noted that for ordinary skilled in the art, other not specifically described parts, belong to prior art or common knowledge. Without departing from the principles of the present application, a number of improvements and refinements can also be made, which should be considered as the protection scope of the present application.

Claims

1. An unstructured cue optimization scoring method for intelligent credit audit assistant characterized in that, The application relates to a credit audit method and device. Step 1, collecting credit audit materials, due diligence reports and post-loan inspection document data of an enterprise; Step 2, block indexing of the collected data, and construction of an intelligent audit vector knowledge base; Step 3, based on the intelligent audit vector knowledge base, and using a large language model to analyze the current enterprise credit audit text, generating an audit opinion on the client credit, and giving an enterprise risk score; Step 4, associating the enterprise risk score with the actual default risk index, constructing a loss function to measure the difference between the two, regarding the large model prompt as a training parameter, and iteratively optimizing the prompt through back propagation; Step 5, visually displaying the enterprise risk score through a visual means; The loss function is as follows: ； wherein, is the output result of the loss function, is the actual default risk indicator of the enterprise, and the selection is the default loss rate; represents the mean square error, which is a measurement indicator of the rationality of the enterprise risk score. In step 4, the text gradient used to optimize the input prompt words is calculated through back propagation, and the gradient calculation method of the optimization process is as follows: ； wherein, represents a first-order partial derivative of a loss function with respect to a risk score, represents an input prompt for a large language model, represents that the gradient is computed by the large language model in the form of text output, represents an optimization gradient of the input prompt word;​ Combine the text gradient with the existing input prompt words to iteratively optimize the input prompt words: ； wherein, represent the input prompt word after the first iterative optimization, represent the input prompt word after the first iterative optimization; Also includes setting a validation set val and setting a stop standard for prompt word optimization to prevent overfitting, as follows: ； wherein, is a new prompt word loss on the validation set, is a set current optimal loss.

2. The unstructured cue word optimization scoring method for intelligent credit audit assistant according to claim 1, wherein, The credit audit materials include enterprise client basic information, pre-loan credit access information, loan withdrawal information, early warning information, overdue information, post-loan collection information and lawsuit information.

3. The unstructured cue word optimization scoring method for intelligent credit audit assistant according to claim 1, wherein, The process of constructing the vector knowledge base in step 2 includes: Based on the hierarchical structure and segment semantics of the credit audit text, the document is block processed, then a pre-trained text embedding model is used to convert the document block into a vector, generating a dense text vector with unified dimensions, finally the generated document block vector is structuredly labeled according to the client unified number, credit unified code and timestamp, and is stored in a vector database supporting efficient search.

4. The unstructured cue word optimization scoring method for intelligent credit audit assistant according to claim 1, wherein, Step 3 specifically includes: Based on the prompt engineering paradigm, the prompt template is designed, the large language model is used to identify and extract risk points in enterprise credit, and an audit report is formed, and based on the audit report, a risk score of the enterprise is given .

5. The unstructured cue word optimization scoring method for intelligent credit audit assistant according to claim 1, wherein, In step 5, the enterprise risk score is displayed through a score dashboard, the score trend curve of the past years is displayed, and the enterprise risk evaluation result is intuitively reflected.

Images