Financial risk identification method based on multi-granularity mixed mode retrieval

Abstract

The invention is suitable for the technical field of mixed modal retrieval, provides a financial risk identification method based on multi-granularity mixed modal retrieval, and is suitable for the field of mixed modal retrieval. The method comprises the following steps: constructing a multi-modal retrieval library, respectively encoding image and text modal information by using CLIP and RoBERTa models, and aligning features through a learnable projection matrix to form a searchable financial knowledge base; a multi-granularity retrieval scheme combining Dual-Encoder and Cross-Encoder is adopted, a Top10 candidate answer is rapidly positioned through coarse granularity, and a high-relevance candidate set is accurately screened through fine granularity; and finally, through a multi-input weighted integration strategy, integrating multiple rounds of prediction results of the large model and carrying out weighted summation based on association degrees to generate a final risk prediction result. According to the method, limitation of a single data source is broken through, comprehensive utilization of cross-modal information is realized, information quality and retrieval efficiency are ensured, model uncertainty is remarkably reduced, a risk identification system with robustness and accuracy is formed, and scientific decision support is provided for financial institutions.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of mixed modal retrieval technology, and particularly relates to a financial risk identification method based on multi-granularity mixed modal retrieval. BACKGROUND

[0002] Under the tide of economic structural transformation and consumption upgrading in China, the automobile consumption market is rapidly expanding, and the automobile finance industry is ushering in an opportunity for vigorous development. However, the financial risks accompanying market expansion present unprecedented complexity and concealment, posing a great challenge to the traditional risk management system.

[0003] Traditional financial risk early warning and prevention means mainly rely on expert experience or black list rule methods, and data-driven machine learning methods. Although the expert experience method has certain transparency and traceability, it is limited to a specific time frame and is difficult to adapt to the rapidly changing market environment. The black list method has the problems of insufficient timeliness and privacy protection, and cannot meet the real-time and accuracy requirements of modern financial scenarios for risk identification.

[0004] Compared with traditional rule methods, machine learning methods have improved risk identification efficiency, but they are highly dependent on a large number of labeled data sets and are significantly affected by input data quality. In the real situation of high data labeling cost and uneven data quality, the performance of machine learning methods is often limited.

[0005] With the rise of large model technology, it has shown significant advantages in handling diversified scenarios. However, in specific fields or scenarios, relying solely on a single response based on customer-related information input for risk early warning may have instability, affecting the accuracy of early warning. This instability is due to the limited ability of a single model or feature to capture complex financial risks, making it difficult to fully and accurately reflect the risk status of customers.

[0006] In view of this, the application proposes a financial risk identification method based on multi-granularity mixed modal retrieval. This method integrates the rich semantic information of multi-modal data, constructs a multi-modal external information library related to the field, and realizes the deep fusion of cross-modal information. At the same time, a multi-granularity retrieval scheme is adopted, combining a hierarchical mechanism of coarse-grained rapid positioning and fine-grained accurate screening to efficiently extract external information highly related to the client to be predicted. Finally, by integrating the multi-round prediction results of large models and based on the correlation degree weighted summation, the multiple preliminary prediction results are effectively integrated, significantly reducing the uncertainty of a single model or feature, and improving the accuracy and efficiency of risk early warning. The application aims to provide more accurate and reliable risk decision support for financial institutions, helping them to cope with the increasingly complex financial risk challenges. SUMMARY

[0007] The application provides a financial risk identification method based on multi-granularity mixed modal retrieval, aiming to solve the problems of existing financial risk early warning prevention and control means, such as time frame limitation of expert experience, timeliness and privacy problems of black list, dependence of machine learning on a large number of labeled data and influence of data quality, and instability of single reply of large model in early warning, which is difficult to comprehensively and accurately cope with the complex and hidden risk challenges in the automobile financial industry.

[0008] The application is implemented as follows: a financial risk identification method based on multi-granularity mixed modal retrieval, comprising:

[0009] Step one, multi-modal retrieval library construction:

[0010] The CLIP model is used to encode the image modal information, extract global and local features, and fuse them through the Transformer; the RoBERTa model is used to encode the text modal information, and the TF-IDF algorithm is used to extract risk keywords to construct a label vector; the vector mapping of the text modal label is performed through risk label semantic enhancement coding; the multi-modal features are aligned to a unified vector space through a learnable projection matrix to construct a searchable financial knowledge base;

[0011] Step two, associated content retrieval:

[0012] Multi-granularity screening is performed in combination with Dual-Encoder and Cross-Encoder:

[0013] ①. Coarse-grained retrieval: calculate the cosine similarity between the predicted client information and the retrieval library entries through the pre-trained language model, and select the top 10 candidate answers;

[0014] ②. Fine-grained retrieval: concatenate the candidate answers and the client information and input them into the Cross-Encoder, calculate the comprehensive relevance score through the multi-layer perceptron, and output the accurate candidate set;

[0015] Step three, risk identification result generation:

[0016] Generate the final prediction result through a multi-input weighted integration strategy:

[0017] ①. Multi-input generation: concatenate the candidate retrieval results and the client information and input them into the large model to generate z sets of probability distributions;

[0018] ②. Weighted integration: based on the candidate answer relevance score Weighted sum of probability distributions, input

[0019] Output the final risk prediction result risk.

[0020] Preferably, the multi-modal retrieval library construction comprises:

[0021] Image modal encoding: encode GPS trajectory pictures by res I ×res I Grid division, extract local features of each grid by CLIP encoder Spliced global features Post-input Layer Transformer generates image representation vector

[0022] Text modal encoding:

[0023] Structured data: split by delimiter and filter stop words, add risk label and encode by RoBERTa;

[0024] Unstructured data: process by rule segmentation and Chinese word segmentation, add risk label and encode by RoBERTa;

[0025] Feature alignment: introduce a learnable projection matrix and Calculate the aligned vector:

[0026]

[0027] Where After completion, store the alignment result in the vector database to build a retrieval library.

[0028] Preferably, risk label enhanced encoding includes:

[0029] Extract TF-IDF high-frequency words from risk customer credit investigation reports as risk keywords, and add no-risk label directly for other non-risk customers:

[0030] Encode risk labels by pre-training language model, map to risk label vector S, and splice with image encoding results:

[0031] S = ReLU (Linear (v tag ))

[0032]

[0033] Where is the output vector of the CLIP encoder.

[0034] Preferably, the text modal encoding is specifically:

[0035] Structured data: T struc Split by delimiter and filter stop words to get corresponding text sequence t seq ,

[0036] Encode by RoBERTa after adding risk label:

[0037]

[0038] Unstructured data: T unstruc Through rule segmentation and Chinese word segmentation processing, the corresponding text sequence t′ is obtained seq Encode by RoBERTa after adding risk label:

[0039]

[0040] Where v′ risk represents a fixed label vector of risk customer label.

[0041] Preferably, the coarse-grained retrieval is specifically: encoding the to-be-predicted customer information X by a pre-trained language model to generate a semantic vector v x , and calculating the cosine similarity with the retrieval library entry :

[0042]

[0043] The top 10 candidate answers with the highest similarity are screened to form a set C.

[0044] Preferably, the fine-grained retrieval is specifically: splicing the candidate answer v c and the customer information v x into a spliced vector:

[0045]

[0046] Input Cross-Encoder to generate interaction representation vector v inter , and calculate the comprehensive score by MLP:

[0047]

[0048] According to the score, a high correlation candidate set C′ is screened.

[0049] Preferably, the multi-input generation is specifically: splicing the candidate retrieval result C′ and the customer information v x into an input sequence C′={c1v x ,c2v x ,...,c z v x}, and inputting the large model to generate a probability distribution p l =p(y|C l =c l v x );

[0050] Weighted integration: normalized correlation degree score Post-weighted summation:

[0051]

[0052] Final risk prediction result is:

[0053] risk = MLP (p(y|(x, D)))

[0054] Wherein, MLP represents a multi-layer perception machine output result.

[0055] Compared with the prior art, the embodiments of the application have the following beneficial effects:

[0056] Firstly, the application integrates the loan customer's credit information, related picture information and vehicle GPS data by constructing a field-related multi-modal external information library, realizes the deep fusion of cross-modal information, breaks through the limitation of single data source, makes the risk identification be able to utilize multi-channel information, and significantly improves the comprehensiveness and efficiency of identification.

[0057] Secondly, the application adopts a multi-granularity retrieval scheme, combines coarse-grained key content retrieval and fine-grained key content screening, quickly locates and accurately screens the external information highly related to the client to be predicted; this hierarchical retrieval mechanism not only guarantees the retrieval efficiency, but also improves the information quality, and provides strong support for subsequent risk identification.

[0058] Thirdly, the application integrates the preliminary probability prediction results obtained by the large model multi-round prediction, and performs weighted summation based on the candidate answer association degree score, effectively integrates multiple preliminary prediction results, reduces the uncertainty influence of single model or feature; this answer integration strategy significantly improves the robustness and prediction accuracy of risk identification, and provides financial institutions with more scientific and reliable risk decision basis. BRIEF DESCRIPTION OF DRAWINGS

[0059] Fig. 1 is the overall process schematic diagram of the application;

[0060] Fig. 2 is the multi-modal retrieval library construction process schematic diagram of the application;

[0061] Fig. 3 is the associated content retrieval process schematic diagram of the application; DETAILED DESCRIPTION

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application. The terminology used in the description herein is for describing particular embodiments only and is not intended to be limiting of the application. The terms "comprising," "having," "including," and "containing" are to be construed as open-ended terms (i.e., meaning "including, but not limited to,") unless otherwise noted. Recitation of ranges of values herein are merely intended to serve as a shorthand method of referring individually to each separate value falling within the range, unless otherwise indicated herein and each separate value is incorporated into the specification as if it were individually recited herein. The use of any and all examples, or exemplary language (e.g., "such as") provided herein, is intended merely to better illuminate the application and does not pose a limitation on the scope of the application unless otherwise claimed. No language is such that it will be construed as indicating any non-claimed element as essential to the practice of the application.

[0063] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase "in an embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily all directed to the same embodiment, or to a single alternative embodiment. It is expressly understood that any of the embodiments described herein can be combined with any of the other embodiments.

[0064] The embodiment of the application provides a financial risk identification method based on multi-granularity mixed modal retrieval, as shown in the figure, comprising: Figs. 1-3

[0065] Step one, multi-modal retrieval library construction:

[0066] The CLIP model is used for encoding image modal information, global and local features are extracted and fused through the Transformer, the RoBERTa model is used for encoding text modal information, a label vector is constructed by combining the TF-IDF algorithm and extracting risk keywords, the text modal label is mapped to a vector through risk label semantic enhancement coding, and the multi-modal features are aligned to a unified vector space through a learnable projection matrix to construct a retrievable financial knowledge base;

[0067] Step two, associated content retrieval:

[0068] Combined with Dual-Encoder and Cross-Encoder for multi-granularity screening:

[0069] ①. Coarse-grained retrieval: the cosine similarity of the to-be-predicted customer information and the retrieval library items is calculated through the pre-trained language model, and the top 10 candidate answers are screened;

[0070] ②. Fine-grained retrieval: the candidate answers and the customer information are spliced and input into the Cross-Encoder, the comprehensive correlation score is calculated through the multilayer perceptron, and the accurate candidate set is output;

[0071] Step three, risk identification result generation:

[0072] ​The final prediction result is generated by a multi-input weighted integration strategy:

[0073] ①. Multi-input generation: splice the candidate search results and customer information respectively into the large model to generate z sets of probability distributions;

[0074] ②. Weighted integration: based on the candidate answer correlation score s Cl The probability distributions are weighted and summed, and the MLP output

[0075] The final risk prediction result risk.

[0076] It should be noted that, due to the limitations of existing financial risk early warning and prevention means, such as the time frame limitation of expert experience, the timeliness and privacy problems of blacklists, the dependence of machine learning on large amounts of labeled data and the influence of data quality, and the instability of single reply of large models, it is difficult to comprehensively and accurately cope with the complex and hidden risk challenges in the automobile finance industry. To solve this problem, the present scheme builds a multi-modal external information library related to the field, deeply integrates loan customer credit information, relevant pictures, vehicle GPS data and other multi-source heterogeneous information, breaks through the limitations of single data source to improve the comprehensiveness of risk identification; On this basis, a multi-granularity retrieval scheme is adopted to realize a hierarchical retrieval mechanism of coarse-grained rapid positioning and fine-grained accurate screening, and high-correlation external information is efficiently extracted; Finally, the multi-round prediction results of the integrated large model are weighted and summed based on the correlation degree, and multi-dimensional prediction information is integrated to reduce uncertainty; This series of innovative designs form a complete technical chain from data integration, information retrieval to result integration, not only significantly improve the efficiency and accuracy of risk identification, but also provide more scientific and reliable risk decision support for financial institutions through multi-level information fusion and decision optimization, effectively enhance the risk prevention and control capability.

[0077] Specifically, in the present embodiment, the present scheme builds a multi-modal retrieval library to provide a rich data basis for subsequent risk identification; During the construction of the multi-modal retrieval library, for image modal information, the CLIP model is used for encoding processing to extract global and local features of the image, and the Transformer structure is used to fuse these features to generate an image representation vector with rich semantic information; At the same time, for text modal information, the RoBERTa model is used for encoding, and the TF-IDF algorithm is used to extract risk keywords to construct a label vector to represent the risk attributes of the text. In order to realize the effective fusion and retrieval of different modal features, a learnable projection matrix is introduced to align the multi-modal features to a unified vector space, and a retrievable financial knowledge base is constructed;

[0078] Next, in the associated content retrieval stage, the method combines Dual-Encoder and Cross-Encoder for multi-granularity screening to improve the accuracy and efficiency of retrieval; coarse-grained retrieval calculates the cosine similarity of the to-be-predicted customer information and the retrieval library items through the pre-trained language model, quickly screens out the Top10 candidate answers highly related to the to-be-predicted customer information; then, fine-grained retrieval inputs the candidate answers and customer information into Cross-Encoder after splicing, calculates the comprehensive correlation score through a multi-layer perception, further outputs the accurate candidate set, and ensures the accuracy and relevance of the retrieval result.

[0079] Finally, in the risk identification result generation stage, the method generates the final risk prediction result by using a multi-input weighted integration strategy; specifically, the candidate retrieval result and the customer information are spliced and input into the large model to generate multiple probability distributions; then, the probability distributions are weighted and summed based on the candidate answer correlation score to form the final risk prediction result; this strategy effectively integrates multiple preliminary prediction results, reduces the uncertainty of a single prediction result, and improves the robustness and accuracy of risk identification.

[0080] In further preferred embodiments of the present application, as shown in Figs. 1-3 The multi-modal retrieval library construction includes:

[0081] Image modality encoding: encode the GPS trajectory picture according to res I ×res I Grid division, extract local features of each grid through a CLIP encoder Splice global features Input Layer Transformer generates an image representation vector

[0082] Text modality encoding:

[0083] Structured data: split according to the delimiter and filter stop words, append risk labels and encode through RoBERTa;

[0084] Unstructured data: process through rule segmentation and Chinese word segmentation, append risk labels and encode through RoBERTa;

[0085] Feature alignment: introduce a learnable projection matrix And Calculate the aligned vector:

[0086]

[0087] Wherein After completion, the alignment result is stored in the vector database to construct the retrieval library.

[0088] In this embodiment, the system divides the GPS trajectory image into grids and uses the CLIP encoder to extract local features from each grid. These local features are then concatenated with global features to form a complete image feature representation. The concatenated features are then fed into the layer Transformer to generate an image representation vector with rich semantic information. This process not only captures the image's detailed information but also ensures the semantic consistency of the image features, providing a solid foundation for subsequent retrieval and recognition.

[0089] At the same time, for text modal information, the system will adopt corresponding encoding strategies based on different data types;

[0090] For structured data, the system first segments it by delimiters and filters out stop words to reduce noise. It then appends a risk tag to the end of the data to clearly identify the data's potential risk attributes. The processed data is then encoded using the RoBERTa encoder to generate a text vector with semantic representation capabilities.

[0091] For unstructured data, the system converts it into a more manageable text form through rule-based segmentation and Chinese word segmentation. Risk labels are also added before RoBERTa encoding. This step ensures that regardless of the data structure, it can be effectively converted into text vectors that can be used for retrieval and recognition. After encoding the image and text modalities, the system introduces a learnable projection matrix to achieve effective fusion and retrieval of features from different modalities.

[0092] These projection matrices can align the feature vectors of different modalities, ensuring their semantic consistency in a unified vector space. By calculating the aligned vectors, the system can eliminate the semantic differences between the different modalities, facilitating subsequent retrieval and recognition.

[0093] Finally, the aligned multimodal feature vectors are stored in a vector database to build a comprehensive and searchable financial knowledge base. This knowledge base not only integrates rich multimodal information, but also uses an efficient retrieval mechanism to enable the system to quickly and accurately locate knowledge items related to the information to be queried. In application scenarios such as financial risk identification, this multimodal retrieval library can provide strong data support and decision-making basis, helping financial institutions better identify potential risks and make scientific decisions.

[0094] In a further preferred embodiment of the present invention, Figs. 1-3 As shown, the risk label enhancement coding includes:

[0095] The TF-IDF high-frequency words of the risk customer credit investigation report are extracted as risk keywords, and the other non-risk customers are directly marked with a non-risk label:

[0096] The risk label is encoded by a pre-trained language model, and after being mapped into a risk label vector S, it is spliced with the image encoding result:

[0097] S = ReLU (Linear (v tag ))

[0098]

[0099] Where is the output vector of the CLIP encoder.

[0100] In this embodiment, in the risk label enhanced encoding process, the credit investigation report of the risk customer is processed, and the TF-IDF high-frequency words are extracted as risk keywords. This step aims to identify and quantify the key risk information in the credit investigation report, ensuring that the core words related to customer risk can be accurately captured; for other non-risk customers, the system directly adds a non-risk label to simplify the subsequent processing process.

[0101] Next, the system uses a pre-trained language model to encode the extracted risk label, mapping it into a risk label vector S with semantic representation ability; This process ensures that the vector representation of the risk label can accurately reflect its semantic content, providing a basis for subsequent feature fusion and retrieval.

[0102] Finally, the system splices the risk label vector S with the image encoding result; The image encoding result here is usually a vector representation obtained by encoding the customer-related pictures through image encoders such as CLIP; The splicing operation combines the semantic information of the risk label with the spatial information of the image encoding result, forming a more comprehensive and rich feature representation; This spliced feature representation not only contains the visual information of the image itself, but also incorporates semantic information related to customer risk, thereby enhancing the discriminability and robustness of the feature representation.

[0103] In further preferred embodiments of the present application, as shown in Figs. 1-3 The text modal encoding is specifically:

[0104] Structured data: T struc Split by delimiter and filter stop words to get the corresponding text sequence t seq ,

[0105] After adding the risk label, it is encoded by RoBERTa:

[0106]

[0107] Unstructured data: T unstruc Through rule segmentation and Chinese word segmentation processing, the corresponding text sequence t′ is obtained seq , after appending the risk label, it is encoded by RoBERTa:

[0108]

[0109] where v′ risk A fixed token vector representing risky customer tokens.

[0110] In this embodiment, during the text modality encoding process, the system adopts differentiated processing strategies for different types of text data to ensure that key information in the text can be accurately and efficiently extracted and integrated with risk-related semantic content;

[0111] For structured data, the system first segments the text according to pre-defined delimiters to separate the individual segments. It then filters out stop words to produce a more refined and meaningful text sequence. Next, the system appends a risk tag to the end of the text sequence. This tag, typically a fixed tag vector, clearly identifies whether the text sequence is associated with a risky customer. Finally, the system encodes the processed text sequence using pre-trained language models such as RoBERTa to generate a text vector with semantic representation capabilities.

[0112] For unstructured data, since its format is more free and irregular, the system will adopt a method that combines rule segmentation with Chinese word segmentation processing; first, the system will perform a preliminary segmentation of the text based on certain rules to divide the various potentially meaningful fragments in the text; then, these fragments will be further segmented using Chinese word segmentation tools to obtain more detailed and accurate text sequences; similar to structured data processing, the system will also append risk labels to the end of the text sequence to clearly identify the risk attributes of the text; finally, the processed text sequence is encoded using pre-trained language models such as RoBERTa to generate text vectors.

[0113] In a further preferred embodiment of the present invention, Figs. 1-3 As shown in the figure, the coarse-grained retrieval is specifically as follows: the customer information to be predicted X is encoded by the pre-trained language model, and a semantic vector v is generated. x , calculate and retrieve library entries Cosine similarity of:

[0114]

[0115] The top 10 candidate answers with the highest similarity are selected to form set C.

[0116] In this embodiment, in the coarse-grained retrieval stage, the system first encodes the relevant information of the to-be-predicted customer through a pre-trained language model; the to-be-predicted customer information is converted from the original text form to a vector representation with semantic representation capability, that is, a semantic vector; the pre-trained language model can learn the deep semantic features of the text through the training of a large amount of text data, so as to ensure that the generated semantic vector can accurately reflect the core content and semantic information of the to-be-predicted customer information;

[0117] Next, the system will calculate the cosine similarity of the generated semantic vector and each item in the retrieval library; cosine similarity is a commonly used text similarity measurement method, which evaluates the similarity between two vectors by calculating the cosine value of the included angle between them; in coarse-grained retrieval, the system will use this method to quickly filter out the retrieval library items that are most similar in semantics to the to-be-predicted customer information;

[0118] Finally, the system will sort the retrieval library items according to the calculated cosine similarity, and select the top 10 highest similarity candidate answers to form a set C; this step ensures that the system can quickly locate the candidate answers highly related to the to-be-predicted customer information, providing an important reference for subsequent fine-grained retrieval and risk identification.

[0119] In further preferred embodiments of the application, as shown in Figs. 1-3 The fine-grained retrieval is specifically: concatenating the candidate answer v c and the customer information v x into a concatenated vector:

[0120]

[0121] Inputting the Cross-Encoder generates an interaction representation vector v inter , and calculating the comprehensive score through the MLP:

[0122]

[0123] According to the score, a high-relevance candidate set C' is selected.

[0124] In this embodiment, in the fine-grained retrieval stage, the system first concatenates the candidate answers selected in the coarse-grained retrieval stage and the relevant information of the to-be-predicted customer to form a concatenated vector; the candidate answers and the customer information are closely combined so that their relevance can be more accurately evaluated subsequently;

[0125] Next, the system inputs the concatenated vector into a Cross-Encoder; Cross-Encoder is a deep learning model that can capture the interaction features between input vectors and generate an interaction representation vector; this interaction representation vector contains deep semantic association information between the candidate answer and the customer information, which helps to more accurately evaluate the matching degree between them;

[0126] Then, the system inputs the generated interaction representation vector into a multi-layer perceptron; MLP is a neural network with multiple layers, which can perform nonlinear transformation on the input vector and calculate a comprehensive score; this comprehensive score reflects the relevance strength between the candidate answer and the customer information, and the higher the score, the stronger the relevance;

[0127] Finally, the system sorts the candidate answers according to the comprehensive score and selects a high-relevance candidate set; this step ensures that the system can accurately locate the candidate answers that are highly relevant to the customer information and have practical value, providing more reliable basis for subsequent risk identification and decision-making.

[0128] In further preferred embodiments of the present application, as shown in Figs. 1-3 The multiple inputs are generated by concatenating the candidate search results C' and the customer information v x as an input sequence C' = {c1v x ,c2v x ,...,c z v x}, and inputting the large model to generate a probability distribution p l =p(y|C l =c l v x );

[0129] Weighted integration: normalized correlation degree score Post-weighted summation:

[0130]

[0131] The final risk prediction result is:

[0132] risk = MLP(p(y|(x,D)))

[0133] Where MLP represents the output result of the multi-layer perceptron.

[0134] In the embodiment, the multi-input generation and weighted integration generates a probability distribution by constructing an input sequence containing candidate search results and customer information, and generates an accurate and reliable risk prediction result by weighted summation of the normalized correlation degree scores. This process not only improves the accuracy of risk identification, but also enhances the robustness of the prediction result, providing strong support for the risk management of financial institutions.

[0135] It should be noted that, for the foregoing embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0136] In several embodiments provided in the present application, it should be understood that the disclosed device can be implemented by other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units can have another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, which can be electrical or other form.

[0137] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0138] The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the protection scope of the application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on these embodiments, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art can still combine, add or delete or make other adjustments to the features of the embodiments of the present application according to the circumstances without conflict and creative labor, so as to obtain different other technical solutions which do not deviate from the concept of the present application in essence. These technical solutions also belong to the scope of protection of the present application.

Claims

1. A financial risk identification method based on multi-granularity mixed modal retrieval, characterized by: include: Step 1: Multimodal retrieval library construction: The CLIP model is used to encode image modal information, extract global and local features, and fuse them through the Transformer. The RoBERTa model is used to encode text modal information, and the TF-IDF algorithm is used to extract risk keywords and construct label vectors. The text modal labels are mapped to vectors through semantic enhancement encoding of risk labels. A learnable projection matrix is ​​used to align multimodal features to a unified vector space to construct a searchable financial knowledge base. Step 2: Related content retrieval: Combining Dual-Encoder and Cross-Encoder for multi-granularity screening: ①. Coarse-grained retrieval: Calculate the cosine similarity between the customer information to be predicted and the search database entries using a pre-trained language model, and select the top 10 candidate answers; ② Fine-grained retrieval: The candidate answers are concatenated with customer information and fed into a cross-encoder. A multi-layer perceptron calculates the comprehensive relevance score and outputs a precise candidate set. Step 3: Generate risk identification results: The final prediction result is generated through a multi-input weighted integration strategy: ①. Multi-input generation: The candidate search results and customer information are spliced ​​together and input into the large model to generate z groups of probability distributions; ②. Weighted integration: based on the candidate answer relevance score s Cl The probability distribution is weighted and summed, and the MLP input The final risk prediction result risk is obtained.

2. The financial risk identification method based on multi-granularity mixed modal retrieval according to claim 1, characterized in that: The construction of multimodal retrieval library includes: Image modality encoding: GPS track pictures are encoded in res I ×res I Mesh division, extracting local features of each mesh through CLIP encoder Stitching global features Post-input Layer Transformer generates image representation vector Text mode encoding: Structured data: Split by delimiter, filter stop words, append risk tags, and then use RoBERTa encoding; Unstructured data: This is processed through rule-based segmentation and Chinese word segmentation, with risk labels added and then encoded using RoBERTa. Feature alignment: Introducing a learnable projection matrix and Calculate the aligned vector: in After completion, the alignment results are stored in the vector database to build a retrieval library.

3. The financial risk identification method based on multi-granularity mixed modal retrieval according to claim 1, characterized in that: Risk tag enhancement codes include: For risky customers, we extract TF-IDF high-frequency words as risk keywords from the credit review report, while for other risk-free customers, we directly add the risk-free tag: The risk label is encoded through the pre-trained language model, mapped to the risk label vector S and then concatenated with the image encoding result: S=ReLU(Linear(v tag )) in is the output vector of the CLIP encoder.

4. The financial risk identification method based on multi-granularity mixed modal retrieval according to claim 1 is characterized in that: The text mode encoding is specifically: Structured data: T struc Split by delimiter and filter stop words to get the corresponding text sequence t seq , After appending the risk tag, use RoBERTa encoding: Unstructured data: T unstruc Through rule segmentation and Chinese word segmentation processing, the corresponding text sequence t′ is obtained seq , after appending the risk label, it is encoded by RoBERTa: where v′ risk A fixed token vector representing risky customer tokens.

5. The financial risk identification method based on multi-granularity mixed modal retrieval according to claim 1 is characterized in that: Coarse-grained retrieval is specifically as follows: Encode the customer information to be predicted X through the pre-trained language model to generate a semantic vector v x , calculate and retrieve library entries Cosine similarity of: The top 10 candidate answers with the highest similarity are selected to form set C.

6. The financial risk identification method based on multi-granularity mixed modal retrieval according to claim 1, characterized in that: Fine-grained retrieval is specifically as follows: The candidate answer v c With customer information x Concatenate into concatenated vectors: Input Cross-Encoder to generate interaction representation vector v inter , and calculate the comprehensive score through MLP: The high-correlation candidate set C′ is screened based on the score.

7. The financial risk identification method based on multi-granularity mixed modal retrieval according to claim 1 is characterized in that: The specific steps for multi-input generation are: Combine the candidate retrieval result C′ with the customer information v x Splice to input sequence C′={c1v x ,c2v x ,...,c z v x }, input the large model to generate the probability distribution p l =p(y|C l =c l v x ); Weighted Ensemble: Normalized Relevance Score Post-weighted summation: The final risk prediction results are: risk=MLP(p(y|(x,D))) Where MLP represents the output result of multi-layer perceptron.

Images