A loan auditing method and system based on a multi-modal large model
Patent Information
- Application Number
- CN202611122811.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-28
- Publication Date
- 2026-10-30
AI Technical Summary
为处理此类文档，现有方案通常采用通用光学字符识别或独立的多模态大模型进行处理；然而，单纯的字符识别缺乏对复杂语境的语义推理能力，而独立的生成式大模型在解码阶段易产生概率性幻觉，导致生成内容与原始图像数据不一致；此外，大模型子词生成粒度与字符识别粒度存在对齐困难，且全图盲目识别方式计算资源消耗大，难以兼顾审核的灵活性与录入数据的精准度；因此，如何在多模态大模型的语义推理过程中有效引入视觉字符识别的硬约束，以抑制生成式幻觉并提升信贷审核的自动化精度与可靠性，成为亟需解决的技术问题
1、本发明通过将光学字符识别结果作为视觉硬约束注入多模态大模型的解码过程，有效抑制了生成式模型易产生的概率性幻觉问题；利用视觉激励信号动态修正原始语义逻辑回归，确保生成的审核报告在保持语义推理灵活性的同时，兼具字符识别的数据精准度，显著提升了金融审核场景下关键要素提取的可靠性；
Smart Images
Figure CN123016456A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and financial technology, specifically to a loan approval method and system based on a multimodal large model. Background Technology
[0002] In the current financial credit review scenario, the business process involves a large number of original credit documents such as bank statements, purchase and sale contracts and credit reports. These documents are usually in the form of high-resolution bitmaps, with complex layouts and diverse formats, containing key risk control elements such as loan amount and deadline, and the accuracy requirements for data extraction are extremely high. To process such documents, existing solutions typically employ general optical character recognition or independent multimodal large models. However, simple character recognition lacks semantic reasoning capabilities for complex contexts, while independent generative large models are prone to probabilistic illusions during the decoding stage, leading to inconsistencies between the generated content and the original image data. Furthermore, there are difficulties in aligning the granularity of word generation in large models with the granularity of character recognition, and the blind recognition method for the entire image consumes significant computational resources, making it difficult to balance the flexibility of the review process with the accuracy of the entered data. Therefore, how to effectively introduce hard constraints of visual character recognition into the semantic reasoning process of multimodal large models to suppress generative illusions and improve the automation accuracy and reliability of credit review has become an urgent technical problem to be solved. Summary of the Invention
[0003] The purpose of this invention is to provide a loan approval method and system based on a multimodal large model, which avoids the illusions produced by generative models in financial approval scenarios, and effectively integrates the flexibility of semantic reasoning with the determinism of optical character recognition, avoiding the blind recognition of the entire image with high computational consumption, thereby ensuring the accuracy of financial accounting data entry. Specifically, the technical solution of this invention is as follows: A loan approval method based on a multimodal large model includes: Receive the original credit document image; Load the preset multimodal large model and the preset optical character recognition model; Based on the original credit document image and the preset risk control element query vector, a semantic attention distribution matrix is generated; Based on the semantic attention distribution matrix, the average confidence of the identified text sequence and the entire sequence is determined. Based on a pre-defined multimodal large model, generate the original semantic logical value vector; By combining the recognized text sequence, the average confidence of the entire sequence, and the original semantic logical value vector, the final probability distribution vector of the output word is generated; Generate a loan approval report based on the final probability distribution vector of the output word units; If the average confidence of the entire sequence is less than the preset safety threshold or if there is a conflict between the semantic logic and the recognized text sequence, a closed-loop optimization process for manual review is triggered. If the average confidence level of the entire sequence is greater than or equal to the preset security threshold and there is no conflict between the semantic logic and the recognized text sequence, a loan approval report is output.
[0004] Preferably, the semantic attention distribution matrix is generated, including: The original credit document image is input into a preset multimodal large model encoder to extract key vector features at the image location; Calculate the dot product of the preset risk control element query vector and the key vector features; Based on the dot product result and the preset vector dimension scaling factor, normalized attention weights are generated. Based on the normalized attention weights, a semantic attention distribution matrix is generated; Among them, the semantic attention distribution matrix represents the probability distribution of the correlation between a specific risk control field and each region of the image.
[0005] Preferably, determining the average confidence level between the identified text sequence and the entire sequence includes: Calculate the mean and standard deviation of the total attention weights in the semantic attention distribution matrix. A dynamic threshold is constructed based on the mean of the full-image attention weight, the standard deviation of the full-image attention weight, and the preset sensitivity adjustment coefficient. Filter the coordinates of the semantic attention distribution matrix where the attention weight is higher than the dynamic threshold, and generate a set of high attention discrete points. Based on the high-attention discrete point set, the coordinates of the minimum bounding rectangle are calculated. The original credit document image is cropped based on the coordinates of the minimum bounding rectangle to generate a high-resolution fragmented image. Input high-resolution fragmented images into a preset optical character recognition model, and output the recognized text sequence and the average confidence of the entire sequence.
[0006] Preferably, generating the final probability distribution vector of the output word units includes: Identify the character to be verified in the text sequence; Construct a sparse bias vector pointing to the character to be verified; Visual excitation signals are generated based on the average confidence of the entire sequence, the preset visual constraint strength coefficient, and the sparse bias vector. The visual stimulus signal is superimposed onto the original semantic logic value vector to generate a modified logic value vector; The modified logical value vector is processed by a normalized exponential function to generate the final probability distribution vector of the output word.
[0007] Preferred options also include: Maintain alignment pointers to the recognized text sequence; In response to the fact that the word output at the previous time step of the preset multimodal large model matches the current character to be verified in the recognized text sequence, the alignment pointer is moved to the next position; In response to the average confidence level of the entire sequence approaching zero, the generation of visual stimulus signals is stopped.
[0008] The preferred closed-loop optimization process for manual review includes: Receive the manually corrected key region mask ground value and text ground value; The negative log-likelihood loss is calculated based on the final probability distribution vector of the output word and the text truth value. Based on the semantic attention distribution matrix and the ground truth of the key region mask, calculate the attention sum of squares loss; The total optimization loss is generated by combining the negative log-likelihood loss with the attention sum-of-squares loss. Based on the total optimization loss, update the preset risk control element query vector.
[0009] A loan approval system based on a multimodal large model includes: The data access module is used to receive images of raw credit documents; The semantic localization module is used to generate a semantic attention distribution matrix based on the original credit document image and the preset risk control element query vector; The visual extraction module is used to determine the average confidence of the identified text sequence and the entire sequence based on the semantic attention distribution matrix. The probability fusion module is used to combine the recognized text sequence, the average confidence of the whole sequence and the original semantic logical value vector to generate the final probability distribution vector of the output word. The closed-loop optimization module is used to trigger a manual review closed-loop optimization process in response to situations where the average confidence of the entire sequence is less than a preset safety threshold or there is a conflict between the semantic logic and the recognized text sequence.
[0010] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention effectively suppresses the probabilistic illusion problem that generative models are prone to by injecting optical character recognition results as visual hard constraints into the decoding process of a multimodal large model; it uses visual excitation signals to dynamically correct the original semantic logic regression, ensuring that the generated audit report maintains the flexibility of semantic reasoning while also having the data accuracy of character recognition, thus significantly improving the reliability of key element extraction in financial audit scenarios. 2. This invention utilizes a semantic attention distribution matrix to accurately locate high-value risk control areas, and combines a dynamic threshold screening mechanism to achieve intelligent cropping and local recognition of the original document. This method avoids the high computational consumption caused by traditional blind full-image recognition, effectively filters background noise interference, and significantly improves the system's credit document processing efficiency and response speed while adaptively processing documents of different resolutions. 3. This invention solves the technical problem of inconsistent word generation granularity and character recognition granularity in large models by constructing sparse bias vectors and maintaining alignment pointers; this mechanism realizes the gradual and accurate alignment of recognized text sequences and generated words, eliminates the cumulative error caused by granularity misalignment, and ensures the accuracy and consistency of sensitive accounting data such as amount and date by using strong constraints on the generation results through visual evidence. 4. This invention constructs a closed-loop optimization process that includes manual review. By calculating the negative log-likelihood loss and the sum of squared attention loss, the preset risk control element query vector is updated in a targeted manner. This mechanism can not only quickly correct model bias based on expert feedback, but also enable the model to adaptively learn the distribution characteristics of new samples, promote the continuous evolution of the core query vector, and realize the long-term stable operation and capability iteration of the intelligent review system. Attached Figure Description
[0011] The present invention will be further explained below with reference to the accompanying drawings and embodiments: Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a structural diagram of the system of the present invention. Detailed Implementation
[0012] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments.
[0013] Example 1:
[0014] Please see Figure 1 A loan approval method based on a multimodal large model includes: Receive the original credit document image; Load the preset multimodal large model and the preset optical character recognition model; Based on the original credit document image and the preset risk control element query vector, a semantic attention distribution matrix is generated; Based on the semantic attention distribution matrix, the average confidence of the identified text sequence and the entire sequence is determined. Based on a pre-defined multimodal large model, generate the original semantic logical value vector; By combining the recognized text sequence, the average confidence of the entire sequence, and the original semantic logical value vector, the final probability distribution vector of the output word is generated; Generate a loan approval report based on the final probability distribution vector of the output word units; If the average confidence of the entire sequence is less than the preset safety threshold or if there is a conflict between the semantic logic and the recognized text sequence, a closed-loop optimization process for manual review is triggered. If the average confidence level of the entire sequence is greater than or equal to the preset security threshold and there is no conflict between the semantic logic and the recognized text sequence, a loan approval report is output.
[0015] This embodiment discloses a loan approval method based on a multimodal large model. This method solves the technical problem of generative models easily generating illusions in financial approval scenarios by introducing hard constraints of optical character recognition during the decoding stage of the multimodal large model. The specific implementation steps are as follows: The system receives images of original credit documents; the data access module acquires images of financial instruments to be reviewed via scanners, document scanners, or electronic document stream interfaces, and records them as follows. The image is typically a high-resolution bitmap containing bank statements, sales contracts, or credit reports. Load a preset multimodal large model and a preset optical character recognition model; the preset multimodal large model is a deep neural network pre-trained on financial corpus and fine-tuned by instructions, which has the ability to align text and image features and semantic reasoning; the preset optical character recognition model is a high-precision network that focuses on text detection and recognition in specific regions, such as CRNN or a Transformer-based OCR model. Based on the original credit document image and the preset risk control element query vector, a semantic attention distribution matrix is generated; the system then uses the original credit document image... Input a visual encoder for a large multimodal model and utilize preset risk control element query vectors representing specific review elements, such as loan amount and deadline. By querying image features, a semantic attention distribution matrix representing the semantic relevance between each region of the image and risk control elements is calculated. ; Based on the semantic attention distribution matrix, the average confidence of the identified text sequence and the entire sequence is determined; the system then uses the semantic attention distribution matrix... The numerical distribution of the data is analyzed to locate high-interest regions in the image, and these regions are then locally cropped and identified to obtain the recognized text sequence. And the overall sequence average confidence level, which characterizes the overall recognition reliability of the sequence. ; Simultaneously, based on a pre-defined multimodal large model, an original semantic logical value vector is generated. The multimodal large model receives the global visual features of the original credit document image and the currently generated text sequence, and independently performs semantic reasoning according to the current context to calculate the original semantic logical value vector of the predicted word at the current time step. This vector represents the natural generation trend of the model without external visual constraints; The key lies in combining the recognized text sequence, the average confidence score of the entire sequence, and the original semantic logical value vector to generate the final probability distribution vector of the output word units; the system utilizes the Bayesian inference principle to recognize the text sequence As prior visual evidence, based on the average confidence level of the entire sequence The high and low dynamic adjustment of its effect on the original semantic logical value vector The correction strength is used to calculate the final probability distribution vector of the corrected output word. ; Based on the final probability distribution vector of the output tokens, a loan approval report is generated; the system then uses the corrected probability distribution vector... Sampling or greedy search is performed to generate the final review text word by word; during this process, if the average confidence level of the entire sequence is... Less than the preset safety threshold Or the semantic logic generated by a large model and the recognition of text sequences. If irreconcilable conflicts exist, the system will trigger a manual review and closed-loop optimization process; conversely, if the average confidence level of the entire sequence is... Greater than or equal to the preset safety threshold Furthermore, since there are no logical conflicts, the system will directly output a loan approval report; among which, a preset security threshold is set. It is set based on the statistical results of historical manual review data, such as taking the 5th percentile of the confidence value of historical error-free identification samples; Through the above steps, this application can effectively integrate the flexibility of semantic reasoning with the determinism of OCR recognition, ensuring the accuracy of financial accounting data entry.
[0016] Example 2:
[0017] Generate a semantic attention distribution matrix, including: The original credit document image is input into a preset multimodal large model encoder to extract key vector features at the image location; Calculate the dot product of the preset risk control element query vector and the key vector features; Based on the dot product result and the preset vector dimension scaling factor, normalized attention weights are generated. Based on the normalized attention weights, a semantic attention distribution matrix is generated; Among them, the semantic attention distribution matrix represents the probability distribution of the correlation between a specific risk control field and each region of the image.
[0018] This embodiment is a further detailed explanation of the step of generating the semantic attention distribution matrix in Embodiment 1; Specifically, the original credit document image is input into a preset multimodal large model encoder to extract key vector features at the image location; the system then processes the original credit document image... The image is preprocessed and input into a visual encoder to extract its coordinates in two-dimensional space. The key vector feature at point is denoted as The key vector feature contains deep visual semantic information of the corresponding pixel region; specifically, the visual encoder adopts a visual backbone network based on the Transformer architecture; key vector feature It is the output tensor of the Key projection layer extracted from the last self-attention mechanism of the visual encoder, or the visual feature mapping vector used in a multimodal large model before cross-attention calculation with the text modality; Calculate the dot product of the preset risk control element query vector and the key vector feature; the system calls the preset risk control element query vector corresponding to the currently pending field. This vector is obtained through supervised fine-tuning on a labeled financial document dataset and is used to represent the semantic features of specific risk control elements; the system calculates the vector one by one. Key vector features at various locations across the entire image The dot product of the two is used to quantify the similarity between them; Based on the dot product result and the preset vector dimension scaling factor Normalized attention weights are generated; to avoid the gradient vanishing problem in the subsequent Softmax function caused by excessively large dot product values, the system introduces a preset vector dimension scaling factor. This factor is typically the square root of the query vector dimension; Based on normalized attention weights, a semantic attention distribution matrix is generated; the system calculates image coordinates using the following Softmax function formula. Normalization focus weights : ; Among them, the denominator part Traversing all spatial locations of the image, i.e., summing and normalizing the features of the entire image; the resulting semantic attention distribution matrix. It accurately characterizes the correlation probability distribution between specific risk control fields and different regions of the image, and satisfies... It enables rapid focusing from the global image to key semantic regions.
[0019] Example 3:
[0020] Determine the average confidence of the identified text sequence and the entire sequence, including: Calculate the mean and standard deviation of the total attention weights in the semantic attention distribution matrix. A dynamic threshold is constructed based on the mean of the full-image attention weight, the standard deviation of the full-image attention weight, and the preset sensitivity adjustment coefficient. Filter the coordinates of the semantic attention distribution matrix where the attention weight is higher than the dynamic threshold, and generate a set of high attention discrete points. Based on the high-attention discrete point set, the coordinates of the minimum bounding rectangle are calculated. The original credit document image is cropped based on the coordinates of the minimum bounding rectangle to generate a high-resolution fragmented image. Input high-resolution fragmented images into a preset optical character recognition model, and output the recognized text sequence and the average confidence of the entire sequence.
[0021] This embodiment is a further detailed explanation of the steps in Embodiment 1 for determining the average confidence of the identified text sequence and the entire sequence; Calculate the mean and standard deviation of the full-image attention weights in the semantic attention distribution matrix; the system processes the semantic attention distribution matrix generated in the previous steps. Perform global statistical analysis and calculate the average attention weight across the entire map. and the standard deviation of the full-image attention weight This quantifies the background noise level of the current document. A dynamic threshold is constructed based on the mean of the full-image attention weight, the standard deviation of the full-image attention weight, and a preset sensitivity adjustment coefficient. To adapt to the extraction requirements of documents with different resolutions, the system constructs a dynamic threshold. Its calculation formula is Among them, the preset sensitivity adjustment coefficient By analyzing historical calibration sample sets The value is determined through statistical analysis. Specifically, it is determined by calculating the Mahalanobis distance between the attention levels of the foreground and background regions in historical samples and selecting the value that maximizes the separation between the foreground and background. The value is usually between 2.0 and 3.0. Filter coordinates in the semantic attention distribution matrix where the attention weight is higher than a dynamic threshold to generate a set of high-attention discrete points; systematically traverse the matrix. To satisfy all coordinates Including high-attention discrete point set ; Based on a set of highly relevant discrete points, the system calculates the coordinates of the minimum bounding rectangle; the system compares the point sets... By finding the extreme values of the horizontal and vertical coordinates, we can determine the smallest rectangular region that can completely cover the point set, thus obtaining the coordinates of the smallest bounding rectangle. ; Based on the coordinates of the minimum bounding rectangle In the original credit document image Perform affine transformations or slicing operations on the image to extract the corresponding image regions and generate high-resolution fragmented images. This step preserves the subtle texture features of the characters by physically cropping them from the original high-resolution bitmap, allowing subsequent OCR models to recognize them. High-resolution fragmented images are input into a pre-defined optical character recognition model, which outputs the recognized text sequence and the average confidence score of the entire sequence; the OCR model then... Perform recognition and output a sequence of recognized text characters. Then, calculate the arithmetic mean of the confidence scores of all characters in the sequence to obtain the average confidence score of the entire sequence. ; Through the above methods, this application can accurately locate and extract key information based on semantic guidance, avoiding blind identification of the entire map that consumes high computing power.
[0022] Example 4:
[0023] Generate the final probability distribution vector of the output word units, including: Identify the character to be verified in the text sequence; Construct a sparse bias vector pointing to the character to be verified; Visual excitation signals are generated based on the average confidence of the entire sequence, the preset visual constraint strength coefficient, and the sparse bias vector. The visual stimulus signal is superimposed onto the original semantic logic value vector to generate a modified logic value vector; The modified logical value vector is processed by a normalized exponential function to generate the final probability distribution vector of the output word. Also includes: Maintain alignment pointers to the recognized text sequence; In response to the fact that the word output at the previous time step of the preset multimodal large model matches the current character to be verified in the recognized text sequence, the alignment pointer is moved to the next position; In response to the average confidence level of the entire sequence approaching zero, the generation of visual stimulus signals is stopped.
[0024] This embodiment is a further detailed explanation of the step in Example 1 for generating the final probability distribution vector of the output word; Specifically, it involves identifying the current character to be verified in the text sequence; and generating the... The system uses the maintained alignment pointers to determine the time step for each lexical unit. From recognizing text sequences Lock the currently unverified character Simultaneously, the independent confidence level of the character output during the OCR recognition process is obtained. , which is the maximum probability value of the OCR network in the Softmax layer; Construct a sparse bias vector pointing to the current character to be verified; the system constructs a vector with dimensions equal to the vocabulary size. Consistent multi-hotspot sparse bias vector To address the inconsistency between the word generation granularity of the large model and the OCR character granularity, the system pre-processes the recognized text sequence using the large model's accompanying word segmenter. Convert to the corresponding token sequence as the baseline; or, when constructing the sparse bias vector; the system traverses the large model vocabulary. Search for all decoded first characters and the currently unverified character. The same lexical will result in vectors The values of the index positions corresponding to these word elements are set to 1, and the values of the other positions are set to 0, thereby solving the granularity alignment problem between character-level OCR and sub-word-level large model; Visual excitation signals are generated based on the average confidence level of the entire sequence, preset visual constraint strength coefficients, and sparse bias vectors; the system calculates the visual excitation signals. The formula is Among them, among them The independent confidence score of the current character replaces the average confidence score of the entire sequence to achieve fine-grained visual error correction, with a preset visual constraint strength coefficient. It is a hyperparameter used to balance semantic and visual weights, taking into account the original semantic logical value vector. The unnormalized Logit value typically ranges from negative infinity to positive infinity, while the average confidence level of the entire sequence... The preset visual constraint strength coefficient is a probability value between 0 and 1. It needs to be set to a scalar of the same order of magnitude as the Logit value, for example, a value ranging from 10.0 to 50.0. Alternatively, the original semantic logical value vector can be processed before generating the visual stimulus signal. A normalization process is performed to ensure that the visual stimulus signal has a substantial impact on the probability distribution; this is determined by performing a grid search on the validation set with the objective of maximizing the F1 score; this step ensures that the OCR confidence level... At higher levels, visual evidence exerts a strong constraint on the generated results; The visual stimulus signal is superimposed onto the original semantic logic value vector to generate a corrected logic value vector; the system then applies the visual stimulus signal... Compared with the original semantic logical value vector generated by the large model By performing element-by-element addition, we obtain the corrected logical value vector. ; Specifically, the original semantic logical value vector It is a multimodal large model without superimposed visual bias. In the case of relying solely on global image features That is, the global visual representation and historical lexical units output by the pre-set multimodal large model encoder. The calculated natural prediction probability distribution; The modified logical value vector is normalized using an exponential function to generate the final probability distribution vector of the output tokens. The system then uses the Softmax function to... Mapped to a probability distribution: ; In addition, the system also performs the following pointer maintenance logic: maintaining an alignment pointer to the recognized text sequence; responding to the fact that the tokens output at the previous time step of the preset multimodal large model are consistent with the currently verified characters in the recognized text sequence, i.e., when the generated... equal At that time, the system will align the pointers. Move to the next position, pointing to ; The response is that the average confidence level of the entire sequence approaches zero, specifically referring to , If the value is a very small positive number, such as 1e-5, the system will stop generating visual stimulus signals, allowing the model to smoothly degenerate into a pure semantic reasoning mode, thus preventing low-quality OCR results from interfering with the generation. Through the above mechanism, this application realizes the dynamic injection of OCR results as hard constraints into the large model decoding process, effectively suppressing probability illusions.
[0025] Example 5:
[0026] The manual review closed-loop optimization process includes: Receive the manually corrected key region mask ground value and text ground value; The negative log-likelihood loss is calculated based on the final probability distribution vector of the output word and the text truth value. Based on the semantic attention distribution matrix and the ground truth of the key region mask, calculate the attention sum of squares loss; The total optimization loss is generated by combining the negative log-likelihood loss with the attention sum-of-squares loss. Based on the total optimization loss, update the preset risk control element query vector.
[0027] This embodiment is a further detailed explanation of the manual review closed-loop optimization process in Embodiment 1; When the system triggers manual review, it receives the manually corrected key area mask truth value and text truth value; business experts correct the review results, and the system records the corrected text content as the text truth value. The correct region selected by the bounding box on the image is used as the ground truth of the key region mask. ;in, It is a binary matrix, with the target region having a value of 1 and the background region having a value of 0; Based on the final probability distribution vector of the output tokens and the true value of the text, the negative log-likelihood loss is calculated; the system also calculates the cross-entropy loss of the text generation task, i.e., the negative log-likelihood loss. The aim is to improve the model's ability to generate correct text; Simultaneously, based on the semantic attention distribution matrix and the ground truth of the key region mask, the attention sum of squares loss is calculated; considering the semantic attention distribution matrix... It is a normalized probability distribution with a sum of 1, while the true value of the key region mask... For a binary mask, the sum equals the area. The system... Normalization is performed to generate the target attention distribution. : ; in, To prevent local minima where the denominator is zero; the system calculates and The difference between them yields the corrected sum-of-squares attention loss. This step aims to force the model's attention mechanism to accurately fit the probability distribution characteristics of the manually focused region. The total optimization loss is generated by combining the negative log-likelihood loss and the attention sum-of-squares loss; the system constructs the total optimization loss through a weighted summation. ,in To assist in task weighting, the learning rates of the two tasks are balanced; based on the total optimization loss, the preset risk control element query vector is updated; the system uses the backpropagation algorithm to calculate the total optimization loss. Query vector for preset risk control elements The gradient, and update The parameters are used to enable the model to adaptively learn and optimize the features of new samples; It should be noted that, in order to ensure the effective execution of the backpropagation algorithm, this application adopts a split gradient backpropagation strategy: for negative log-likelihood loss... It is mainly used to update the decoder parameters of multimodal large models and the parameters of OCR recognition networks; for attention sum of squares loss Because it directly calculates the predicted semantic attention distribution matrix. With the true value of the critical region mask The difference between them avoids non-differentiable physical coordinate clipping operations, therefore The gradient can be directly backpropagated to the generator. The source, that is, directly used to update the preset risk control element query vector. The parameters ensure the directional evolution of the query vector for core risk control elements.
[0028] Example 6: Please see Figure 2 A loan approval system based on a multimodal large model includes: The data access module is used to receive images of raw credit documents; The semantic localization module is used to generate a semantic attention distribution matrix based on the original credit document image and the preset risk control element query vector; The visual extraction module is used to determine the average confidence of the identified text sequence and the entire sequence based on the semantic attention distribution matrix. The probability fusion module is used to combine the recognized text sequence, the average confidence of the whole sequence and the original semantic logical value vector to generate the final probability distribution vector of the output word. The closed-loop optimization module is used to trigger a manual review closed-loop optimization process in response to situations where the average confidence of the entire sequence is less than a preset safety threshold or there is a conflict between the semantic logic and the recognized text sequence.
[0029] This embodiment provides a loan approval system based on a multimodal large model, the system including: Data access module: configured to receive images of raw credit documents. And convert it into a digital tensor format suitable for model processing; Semantic localization module: Configured to generate a semantic attention distribution matrix based on the original credit document image and preset risk control element query vectors. To achieve semantic positioning of key information areas; Visual extraction module: Configured to crop images based on a semantic attention distribution matrix, using statistical thresholds, and calling an OCR model to determine the text sequence for recognition. Compared with the average confidence level of the whole sequence ; Probabilistic fusion module: configured to combine the recognized text sequence, the average confidence score of the entire sequence, and the original semantic logical value vector. The final probability distribution vector of the output word is generated through bias techniques. This enables the injection of hard visual constraints. Closed-loop optimization module: configured to respond when the average confidence level of the entire sequence is less than a preset safety threshold. If there is a conflict between the semantic logic and the recognized text sequence, a closed-loop optimization process of manual review is triggered, and the model parameters are updated using manual feedback data. Through the collaborative work of the above modules, this system achieves efficient, accurate, and continuously evolving credit approval.
[0030] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.
Claims
1. A loan approval method based on a multimodal large model, characterized in that, include: Receive the original credit document image; Load the preset multimodal large model and the preset optical character recognition model; Based on the original credit document image and the preset risk control element query vector, a semantic attention distribution matrix is generated; Based on the semantic attention distribution matrix, the average confidence of the identified text sequence and the entire sequence is determined. Based on a pre-defined multimodal large model, generate the original semantic logical value vector; By combining the recognized text sequence, the average confidence of the entire sequence, and the original semantic logical value vector, the final probability distribution vector of the output word is generated; Generate a loan approval report based on the final probability distribution vector of the output word units; If the average confidence of the entire sequence is less than the preset safety threshold or if there is a conflict between the semantic logic and the recognized text sequence, a closed-loop optimization process for manual review is triggered. If the average confidence level of the entire sequence is greater than or equal to the preset security threshold and there is no conflict between the semantic logic and the recognized text sequence, a loan approval report is output.
2. The loan approval method based on a multimodal large model according to claim 1, characterized in that, Generate a semantic attention distribution matrix, including: The original credit document image is input into a preset multimodal large model encoder to extract key vector features at the image location; Calculate the dot product of the preset risk control element query vector and the key vector features; Based on the dot product result and the preset vector dimension scaling factor, normalized attention weights are generated. Based on the normalized attention weights, a semantic attention distribution matrix is generated; Among them, the semantic attention distribution matrix represents the probability distribution of the correlation between a specific risk control field and each region of the image.
3. The loan approval method based on a multimodal large model according to claim 1, characterized in that, Determine the average confidence of the identified text sequence and the entire sequence, including: Calculate the mean and standard deviation of the total attention weights in the semantic attention distribution matrix. A dynamic threshold is constructed based on the mean of the full-image attention weight, the standard deviation of the full-image attention weight, and the preset sensitivity adjustment coefficient. Filter the coordinates of the semantic attention distribution matrix where the attention weight is higher than the dynamic threshold, and generate a set of high attention discrete points. Based on the high-attention discrete point set, the coordinates of the minimum bounding rectangle are calculated. The original credit document image is cropped based on the coordinates of the minimum bounding rectangle to generate a high-resolution fragmented image. Input high-resolution fragmented images into a preset optical character recognition model, and output the recognized text sequence and the average confidence of the entire sequence.
4. The loan approval method based on a multimodal large model according to claim 1, characterized in that, Generate the final probability distribution vector of the output word units, including: Identify the character to be verified in the text sequence; Construct a sparse bias vector pointing to the character to be verified; Visual excitation signals are generated based on the average confidence of the entire sequence, the preset visual constraint strength coefficient, and the sparse bias vector. The visual stimulus signal is superimposed onto the original semantic logic value vector to generate a modified logic value vector; The modified logical value vector is processed by a normalized exponential function to generate the final probability distribution vector of the output word.
5. The loan approval method based on a multimodal large model according to claim 4, characterized in that, Also includes: Maintain alignment pointers to the identified text sequence; In response to the fact that the word output at the previous time step of the preset multimodal large model matches the current character to be verified in the recognized text sequence, the alignment pointer is moved to the next position; In response to the average confidence level of the entire sequence approaching zero, the generation of visual stimulus signals is stopped.
6. The loan approval method based on a multimodal large model according to claim 1, characterized in that, The manual review closed-loop optimization process includes: Receive the manually corrected key region mask ground value and text ground value; The negative log-likelihood loss is calculated based on the final probability distribution vector of the output word and the text truth value. Based on the semantic attention distribution matrix and the ground truth of the key region mask, calculate the attention sum of squares loss; The total optimization loss is generated by combining the negative log-likelihood loss and the attention sum-of-squares loss. Based on the total optimization loss, update the preset risk control element query vector.
7. A loan approval system based on a multimodal large model, applied to the loan approval method based on a multimodal large model as described in any one of claims 1-6, characterized in that, include: The data access module is used to receive images of raw credit documents; The semantic localization module is used to generate a semantic attention distribution matrix based on the original credit document image and the preset risk control element query vector; The visual extraction module is used to determine the average confidence of the identified text sequence and the entire sequence based on the semantic attention distribution matrix. The probability fusion module is used to combine the recognized text sequence, the average confidence of the whole sequence and the original semantic logical value vector to generate the final probability distribution vector of the output word. The closed-loop optimization module is used to trigger a manual review closed-loop optimization process when the average confidence of the entire sequence is less than a preset safety threshold or when there is a conflict between the semantic logic and the recognized text sequence.