Automobile financial post-loan risk identification method based on multi-modal large model

By constructing a multimodal risk control model and utilizing credit review text, video, and GPS data, the accuracy and timeliness issues of auto finance risk identification in existing technologies have been resolved, enabling comprehensive and multi-faceted prevention and control of post-loan risks and precise early warning.

CN118917936APending Publication Date: 2024-11-08CHERY HUIYIN MOTOR FINANCE SERVICE CO LTD
View PDF 0 Cites 6 Cited by

Patent Information

Application Number
CN202410794344.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-06-19
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing methods for identifying risks in auto finance rely on the experience of business experts or historical blacklists, which are insufficient to address new types of financial risks. Furthermore, machine learning methods require large amounts of labeled data and their outputs lack specific analytical descriptions.

Method used

A multimodal risk control model is constructed, utilizing credit review text, credit review video, and vehicle GPS data. Through fusion coding, fine-tuning, and reinforcement learning methods, a description of risk level and risk points is generated.

Benefits of technology

It improves the accuracy and efficiency of post-loan risk identification in auto finance and provides precise risk warning decision support.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118917936A_ABST
    Figure CN118917936A_ABST
Patent Text Reader

Abstract

The invention discloses an automobile financial post-loan risk identification method based on a multi-modal large model. The method comprises the following steps: processing an input credit examination text, a credit examination video and vehicle GPS data by using an established multi-modal risk control large model, and predicting and outputting the risk degree and risk point description of a customer; the establishment of the multi-modal risk control large model comprises the following steps: step 1, based on a financial large language model, constructing the multi-modal risk control large model by using a credit review text, a credit review video and vehicle GPS data of an automobile loan; 2, carrying out fusion coding on multi-modal data, and training a large model through fine tuning; and 3, performing time sequence sampling on the training data set by adopting a reinforcement learning method, and performing fine adjustment and result evaluation on the multi-modal risk control large model again to obtain an optimal multi-modal risk control large model. According to the invention, multi-modal data in the field of automobile finance is processed, so that all-directional and multi-angle risk prevention and control of automobile loan are realized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of machine learning prediction of financial risks, and in particular to a method for identifying post-loan risks of automobile finance based on a multimodal large model. Background Art

[0002] In recent years, the penetration rate of the auto finance market has continued to rise, placing higher demands on the risk management capabilities of auto finance institutions. Against this backdrop, more and more auto finance institutions are using big data and artificial intelligence technologies to implement digital transformation of risk management. Generally speaking, the amount of auto consumer credit contracts is high, and fraudulent loan institutions will forge loan materials or trick credit-free households into applying for installment car purchases, and then immediately sell the vehicles through illegal channels after obtaining them, in order to defraud financial institutions of credit funds. In order to effectively identify auto finance risks, it is necessary to make full use of multimodal data at different stages, such as pre-loan credit review data (video, text, etc.) and post-loan vehicle GPS data.

[0003] Traditional risk prevention and control measures mainly adopt the risk control rules based on the experience of business experts, or use the historical blacklist matching method to carry out risk control. Such methods can generally detect possible risk behaviors based on historical experience or historical blacklists, and are interpretable. However, the experience of business experts is limited in time, and blacklists usually have timeliness and privacy issues, which leads to the fact that these methods also have great limitations and are difficult to deal with new financial risks. With the development of artificial intelligence technology, methods based on machine learning and deep learning are gradually being applied in the field of risk control to identify potential financial risks. Such methods can effectively improve the timeliness of risk control. However, such methods often require the use of a large amount of labeled data for model training, and the model only outputs the probability of the customer's risk, and it is difficult to output a specific analysis description of its prediction results. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method for identifying post-loan risks in automobile finance based on a multimodal large model. The method constructs a multimodal post-loan risk control large model based on a large language model, realizes the mining of multimodal and multi-dimensional user information, mines more hidden information of customers, improves the accuracy of post-loan risk identification and early warning efficiency, and provides automobile financial institutions with more accurate and effective risk early warning decision support.

[0005] In order to achieve the above-mentioned purpose, the technical solution adopted by the present invention is: a method for identifying post-loan risks of automobile finance based on a multimodal large model, which uses the established multimodal risk control large model to process the input credit review text, credit review video and vehicle GPS data to predict and output the risk level and risk point description of the customer; wherein the establishment of the multimodal risk control large model includes:

[0006] Step 1: Based on the financial language model, use the credit review text, credit review video and vehicle GPS data of auto loans to build a multimodal risk control model;

[0007] Step 2: Fusion encode multimodal data and train the large model through fine-tuning;

[0008] Step 3: Use reinforcement learning methods to perform time-series sampling on the training data set, fine-tune the multimodal risk control model again and evaluate the results to obtain the optimal multimodal risk control model.

[0009] In step one, the data input into the multimodal risk control model is preprocessed. The preprocessing of video data includes using adaptive sampling based on overlap rate and fixed sampling based on time series to obtain a certain number of video key frames as visual modality input data; the preprocessing of GPS data includes converting GPS data into feature input data of both visual and textual modalities.

[0010] In step one, the construction of a large risk control model based on multimodal data includes the processing of videos and images based on the large language model structure by adding a visual modal encoding structure, and adding a visual input mapping module to align the visual modal vector to the text modal vector space, thereby completing the construction of a large multimodal model.

[0011] The steps to build a multi-modal risk control model include:

[0012] Step 1. Preprocess the customer's credit review video and GPS data to obtain a video frame set {F j |j=1,2,3,...,L} and GPS visual images to construct a multimodal large model visual feature input set {V i |i=1,2,3,...,L+1}, and use the visual encoder to process to obtain the visual feature vector;

[0013] Step 2. Concatenate the customer's letter review text with the customer's GPS text, and then process the constructed text modal input based on the text feature encoding space of the large language model to construct a text feature vector;

[0014] Step 3. Build a learnable visual input mapper to process the visual feature vector and align the encoded features of the visual modality with the text feature space to obtain the aligned visual feature vector;

[0015] Step 4. Fuse the text feature vector with the aligned visual feature vector, input it into the backbone network based on the large language model for processing, and input the corresponding risk control results.

[0016] In step two, fine-tuning of the multimodal risk control large model includes encoding the input multimodal information using the coding structure built based on the large model after completing the construction of the multimodal risk control model, and training the large model through fine-tuning to generate prediction results for the risk level and corresponding risk point descriptions, including four steps: encoding of image input information, fusion of text input information, alignment of multimodal input vectors, and training and fine-tuning of the large model.

[0017] The encoding of visual input information includes: the input information of the visual modality includes two types: GPS images constructed based on the sampled GPS trajectory and a set of fixed-length video frames obtained after sampling processing; these two types of image information are encoded using a visual encoder ViT based on a pre-trained model.

[0018] Multimodal input vector alignment involves using an MLP-based mapping model to map the visual encoding vector to the text semantic encoding space to obtain the converted visual encoding vector v Tvp It is input into LLM together with the encoded text information input vector for training.

[0019] The steps of training and fine-tuning the large model include: iteratively updating the multimodal risk control large model through fine-tuning based on low-rank matrices, fine-tuning method based on low-rank matrices, by constructing and training low-rank matrices, based on the visual encoding vector and text encoding vector constructed in step 2, and taking the risk level assessment and risk point description annotated by experts as benchmark values, fine-tuning the trained large model to be more adapted to the domain content.

[0020] Step three includes adopting a variety of training sampling strategies to sample the training data set, and then iteratively updating the sampling strategy and the multimodal risk control model constructed in step two through the constructed reinforcement learning framework to obtain the optimal model results.

[0021] Building a reinforcement learning framework includes:

[0022] Step 1. Define the state space S t , action space A t And the reward function R(s t ,a t )

[0023] Step 2. Initialize state s0

[0024] Step 3. According to the current state s t Select action a t , using strategy π(a t |s t ) to select actions, where π is the policy function

[0025] Step 4. Execute action at And observe the reward r of the environment feedback t and the new state s t+1

[0026] Step 5. Update the strategy based on the reward and use the reinforcement learning algorithm Q-learning to update the policy function

[0027] Step 6. Repeat Step 3-5 until the stop condition is reached

[0028] in:

[0029] State space S t : It is defined as the accuracy of the large model's prediction of sample data in a certain time window [t, t+△t], which is designed as a continuous state space to more finely represent the performance changes of the model;

[0030] Action Space A t : Defined as different sampling methods, different sampling based on time series strategies are performed on the data set before the time window [t, t+△t];

[0031] Reward function R(s t ,a t ): Use the accuracy of the model as the reward signal, designed in the form of a Sigmoid function Among them A t represents the accuracy of the model in the time window [t, t+△t], A threshold is the preset accuracy threshold, and α is a hyperparameter that controls the slope of the reward function. When the model accuracy exceeds the threshold, the reward function will tend to 1, giving a positive reward, and when the model accuracy is lower than the threshold, it will tend to 0, giving a negative reward. In this way, the model is guided to learn and optimize towards high accuracy.

[0032] The policy function π(a t |s t ): Model optimization strategy selection function where a t represents the action selected at time step t; s t represents the state at time step t, that is, the accuracy of sample prediction in the time window [t, t+△t]; Q(s t ,a t ) represents the state-action pair (s t ,a t ) is the estimated value learned during the training process; β is a temperature parameter that controls the distribution of action selection probabilities. A higher β will make the probability distribution smoother, while a lower β will make the probability distribution more concentrated on the optimal action; N represents the size of the action space, that is, the number of selectable sampling methods.

[0033] The advantages of the present invention are: (1) based on the credit review text, credit review video and vehicle GPS data of auto loans, a multimodal risk control model is constructed, which can process multimodal data in the field of auto finance, thereby realizing all-round and multi-angle risk prevention and control of auto loans; (2) a customer risk prediction method based on a fine-tuned large model is designed, through multimodal and multi-dimensional customer information input, based on the fine-tuned financial language large model to mine more hidden information of the customer, output the customer's risk level and risk point description, and effectively improve the accuracy of risk warning; (3) a reinforcement learning method based on time series strategy sampling is designed to fine-tune the large model, which significantly improves the prediction accuracy of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The following is a brief description of the contents expressed in the drawings of the present invention and the symbols in the drawings:

[0035] Figure 1 This is a flow chart of the risk identification method of the present invention;

[0036] Figure 2 A schematic diagram of a multi-modal risk control model constructed for the present invention;

[0037] Figure 3 This is a flow chart for fine-tuning the multi-modal risk control large model of the present invention. DETAILED DESCRIPTION

[0038] The specific implementation of the present invention will be further explained in detail below by describing the optimal embodiment with reference to the accompanying drawings.

[0039] The financial risk prediction method based on the multimodal big model mainly includes three main steps: construction of the multimodal risk control big model, fine-tuning of the multimodal risk control big model, and sampling-based fine-tuning and result evaluation. First, based on the existing open source financial big language models (such as deepmoney, DISC-FinLLM, XuanYuan, etc.), the credit review text, credit review video and vehicle GPS data of auto loans are used to build a multimodal post-loan risk control big model; secondly, the multimodal data is fused and encoded, and the big model is trained through fine-tuning to generate prediction results for the risk level and risk point description; finally, the reinforcement learning method is used to perform appropriate time series sampling on the training data set to achieve further fine-tuning of the big model and result evaluation. The flow chart of the present invention is as follows: Figure 1 As shown, the specific steps are as follows:

[0040] Step 1: Multimodal risk control model construction. The multimodal risk control model construction includes three main parts: credit review video preprocessing, GPS data preprocessing, and risk control model construction based on multimodal data. The specific contents are as follows:

[0041] 1. Credibility-reviewed video preprocessing: When preprocessing the credibility-reviewed video, the present invention adopts an adaptive sampling method based on overlap rate and a fixed sampling method based on time series to obtain a certain number of video key frames to construct visual modality input data.

[0042] Given two video frames F i and F j , the overlap ratio The calculation formula is as follows:

[0043]

[0044] Among them, area(F i ) represents the video frame F i area(F i )∩area(F j ) represents the video frame F i and F j Overlap area. In a shorter period of time, the interval between video frames and overlap ratio The relationship can be expressed by a linear model. Therefore, given a sampled video frame F i Its previous sampled video frame F i-1 , assuming that the overlap rate and frame interval are and Assume that F i The next sampled video frame whose overlap rate meets the given threshold O is F i+1 , then F i+1 With F i Frame interval The following linear model can be used to estimate:

[0045]

[0046]

[0047] 0=w*1+b (4)Where w and b represent the learnable weight parameters. i With F i+1 The overlap rate Need to meet the given threshold O, frame interval Satisfies the following calculation formula:

[0048]

[0049] Since the frame interval is a positive integer and as few video frames as possible need to be sampled, the sampling interval is as follows:

[0050]

[0051] in, Represents rounding down. Based on the above sampling interval, adaptive sampling can be performed to obtain candidate video frames that meet the overlap rate threshold. The candidate video frame set obtained after repetition rate sampling {F i |i=1,2,3,...,m}. In order to obtain a fixed number of input video frames, the present invention further samples the candidate video frames using fixed sampling based on time series. Specifically, for a given fixed number of sampled frames L and a candidate video frame set {F i |i=1,2,3,...,m}, the video frame set is sorted in time sequence ( represents rounding up) intervals, and then randomly select a video frame from each time interval to finally form the credit-check video input video frame set {F j |j=1,2,3,...,L}.

[0052] 2. GPS data preprocessing. In order to effectively use GPS data, the present invention converts GPS data into visual and textual input forms, so as to achieve multi-angle and all-round utilization of GPS data. Given raw GPS data is usually given in the form of several pieces per second, one piece per second, or one piece per few seconds. This type of data has a lot of redundancy and noise, and contains less semantic information. Therefore, it is necessary to process the given raw GPS data to reduce redundancy and noise, while having more semantic information.

[0053] GPS stay points usually represent the geographical area where a user stays for a period of time. By analyzing the user's GPS stay points, we can understand the user's travel habits, preferences and activity patterns, which is of great value in the fields of customer trajectory mining, market research, urban planning, traffic management, etc. Therefore, the present invention constructs the features of the visual and textual modes of GPS data by mining the customer stay point information. The specific process is as follows:

[0054] Step 1. Use the stay point extraction method based on geometric characteristics to obtain the user's stay points in the past n days. That is, given a time threshold and a distance threshold, find all points that meet the given threshold conditions in all GPS point sequences of the user in the past n days as the user's candidate stay point set.

[0055] Step 2. Fuse the stay points in the user's candidate stay point set that are less than the distance threshold. That is, starting from the first stay point sorted based on the time series, the stay points within the same distance threshold range are averaged and fused as new stay points, and finally the user's stay point set for the past n days is obtained.

[0056] Step 3. Concatenate all elements in the set of the user's stay points in the past n days as the user's GPS text modal input; draw all elements in the set of the user's stay points in the past n days onto an image in the form of longitude and latitude coordinates as the user's GPS visual modal input.

[0057] 3. Construction of a risk control big model based on multimodal data. In order to effectively utilize multimodal data in automobile financial risk control, the present invention constructs a multimodal risk control big model to realize the processing of visual and textual information. The multimodal risk control big model mainly includes two parts: a visual feature extraction module and a large language model. It aims to understand the characteristics of credit review images, credit review texts, and GPS data, and generate corresponding risk results based on the understood content.

[0058] In order to realize the processing of the features of the letter review images, letter review texts and GPS data, the present invention is based on the large language model structure, by adding a visual modality encoding structure, to realize the processing of videos and images, and adding a visual input mapping module to align the visual modality vector to the text modality vector space, thereby completing the construction of a multimodal large model. Figure 2 The schematic diagram of the multi-modal risk control model shown in the figure has the following construction process:

[0059] Step 1. Preprocess the customer's credit review video and GPS data to obtain a video frame set {F j |j=1,2,3,...,L} and GPS visual images to construct a multimodal large model visual feature input set {V i |i=1,2,3,...,L+1} and process it with a visual encoder to obtain a visual feature vector.

[0060] Step 2. Concatenate the customer's letter review text with the customer's GPS text, and then process the constructed text modal input based on the text feature encoding space of the large language model to construct a text feature vector.

[0061] Step 3. Build a learnable visual input mapper, process the visual feature vector, align the encoded features of the visual modality with the text feature space to obtain the aligned visual feature vector.

[0062] Step 4. Fuse the text feature vector with the aligned visual feature vector, input it into the backbone network based on the large language model for processing, and input the corresponding risk control results.

[0063] Step 2: Fine-tune the multimodal risk control model. After completing the input data preprocessing and model building, in order to more effectively utilize the multimodal credit review information and GPS information obtained after processing, the input multimodal information is encoded using the coding structure built based on the large model, and the large model is trained through fine-tuning to generate prediction results describing the risk level and corresponding risk points. The main process of fine-tuning the multimodal risk control model is as follows: Figure 3 As shown in the figure, it mainly includes four steps: encoding image input information, fusing text input information, aligning multimodal input vectors, and training and fine-tuning the large model. The training set includes multimodal data and its corresponding prediction structure. The data set is constructed through historical data. The data in the data set is processed, trained, and fine-tuned according to the following steps:

[0064] 1. Encoding of visual input information. The input information of the visual modality includes two types: GPS images constructed based on the sampled GPS trajectory and a set of fixed-length video frames obtained after sampling. The resolutions of these two types of images are often not uniform. In order to fuse the semantic information they contain to facilitate the next step of processing, the visual encoder ViT (Vision Transformer) based on the pre-trained model is used to encode these two types of image information, and each image is encoded into a length of H of the same dimension. vision , width is W vision Semantic vector of GPS image is generated The sequence of vectors of video frame codes In actual implementation, L=9.

[0065]

[0066]

[0067] Where φ represents the encoder based on the pre-trained model, represents the i-th frame in the video frame sequence. Represents the encoding vector corresponding to the i-th frame. After encoding, the two vector encodings are concatenated to obtain the concatenated visual encoding vector v vision :

[0068]

[0069] in Represents a concatenation operation.

[0070] 2. Text input information encoding. For the pre-processed letter review text and text consisting of a sequence of sampled GPS trajectories Concatenate them, and then perform semantic encoding on them to obtain the concatenated text encoding vector v text :

[0071]

[0072] Among them, Encoder represents the encoder, Indicates the letter review text, Indicates GPS text, Represents a concatenation operation.

[0073] 3. Multimodal input fusion. Since the visual coding vector and the text coding vector are located in different coding spaces, a mapping model is needed to align the two vector spaces. The present invention uses an MLP-based mapping model to map the visual coding vector to the text semantic coding space to obtain the converted visual coding vector v Tvp :

[0074]

[0075] in represents the MLP layer. After the mapping is completed, the dimension obtained is d×h e The visual encoding vector In the text semantic coding space, it has more sufficient visual semantic information, which is input together with the encoded text information input vector into the multimodal risk control model for training.

[0076] 4. Train the large model based on the fine-tuning method. The large model in the general field can show relatively good cognitive and reasoning ability in common and general scenarios, but it often performs poorly in unknown scenarios or domain-specific scenarios. This type of general large model is relatively poorly adapted to specific fields. In the financial scenario, the cognitive ability of different scenarios has a greater impact on the prediction results of the model. Therefore, the present invention iteratively updates the financial language large model through a fine-tuning method based on a low-rank matrix. While ensuring the cognitive reasoning ability of the large model in the general field, it further optimizes its generalization in specific scenarios in the financial field, while reducing the difficulty in the process of training the large model, and effectively reducing the labor cost in the large model adaptation process.

[0077] The present invention uses a fine-tuning method based on a low-rank matrix. By constructing and training a low-rank matrix, based on the visual coding vector and text coding vector constructed in step 2 and step 3 of step 2, and taking the risk level assessment and risk point description annotated by experts as the benchmark value, the large model to be trained is fine-tuned to be more suitable for the domain content. Specifically, in the training stage, a bypass is first added next to the original large model, and a set of low-rank dimensionality reduction and dimensionality increase matrices are added to it, and the weight W0 of the large model is updated:

[0078] W=W0+△W (12)

[0079] △W=BA (13)W0 represents the initial parameters corresponding to the original financial language model, △W represents the parameters to be updated, and B∈R d×r Represents the dimension reduction matrix, A∈R r×k Represents a dimension-raising matrix, and the corresponding rank r∈min(d, k), d, k are the dimensions of the weight matrix W0. After the construction is completed, based on the obtained visual encoding vector v vision With the text encoding vector v text Generate the corresponding hidden vector h:

[0080] v = Concatenate(v vision ,v text ) (14)

[0081] h=W0v+Wv=W0v+BAv (15)

[0082] Where v represents the fused input representation vector, and Concatenate() represents the vertical concatenation operation. This model is fine-tuned to make it more suitable for the goals of specific tasks.

[0083] Step 3: Sampling-based fine-tuning and result evaluation. Fine-tuning large models using a reinforcement learning method based on temporal strategy sampling can significantly improve their performance. Different sampling methods have different effects on model training, and the sampling method based on temporal strategy shows superiority in this context. Combining reinforcement learning can effectively improve the performance of large models and make them perform better in complex tasks.

[0084] 1. Time series strategy sampling. Time series sampling can extract useful information, reduce noise and improve model performance in the field of risk control. Therefore, a time series-based sampling strategy is selected in the sampling strategy to fine-tune the model. The following four methods are mainly used:

[0085] 1) Sliding window sampling

[0086] Define a window width of W size If x(t) represents a data point at time t (where time t represents the data entry time point), then the data set of the sliding window at time t can be expressed as {x(t), x(t-1), ..., x(tW size +1)}, the window moves forward with time, each time moving 1 time unit (here the time unit is day).

[0087] 2) Exponentially weighted sliding window sampling

[0088] The importance of recently observed data is emphasized by giving greater weight to recent data, because the latest data in risk data is often more important than the old data. Let λ be the weight decay coefficient (0<λ≤1), x(t) is the data point at time t, and the weighted value y(t) = λx(t) + (1-λ)y(t-1). By appropriately selecting the value of λ, the attention paid to new and old data can be balanced.

[0089] 3) Time decay sampling

[0090] Different sampling weights are given to data according to their time distance. Let p(t) represent the sampling probability of the data point at time t. Where λ is the decay rate, which controls the decay speed, and T is the selected prediction time point. is a normalization constant that ensures that the sum of all p(t) is 1.

[0091] 4) Weighted random sampling

[0092] Let w(t) = μ (T-t) is the weight of the data point at time t, where μ is close to but less than 1, and T is the selected prediction time point, then the sampling probability is defined as

[0093] 2. Build a reinforcement learning framework. Reinforcement learning optimizes decision-making strategies by obtaining feedback through interaction with the environment. It can adapt to the ever-changing data characteristics and needs, thereby better capturing key information and trends. This can more accurately reflect the true value of the data, thereby improving the quality of the overall model. The overall steps for building the entire reinforcement learning framework are described as follows:

[0094] Step 1. Define the state space S t , action space A t And the reward function R(s t ,a t )

[0095] Step 2. Initialize state s0

[0096] Step 3. According to the current state s t Select action a t , using strategy π(a t |s t ) to select actions, where π is the policy function

[0097] Step 4. Execute action a t And observe the reward r of the environment feedback t and the new state s t+1

[0098] Step 5. Update the strategy based on the reward and use the reinforcement learning algorithm Q-learning to update the policy function

[0099] Step 6. Repeat Step 3-5 until the stop condition is reached

[0100] The detailed description of the relevant parameters are as follows:

[0101] State space S t : It is defined as the accuracy of the large model in predicting sample data in a certain time window [t, t+△t], and is designed as a continuous state space to more finely represent the performance changes of the model.

[0102] Action Space A t : Defined as different sampling methods, for example, different time series strategy-based sampling is performed on the data set before the time window [t, t+△t], such as the four time series sampling methods mentioned above.

[0103] Reward function R(s t ,a t ): Use the accuracy of the model as the reward signal, designed in the form of a Sigmoid function Among them A t represents the accuracy of the model in the time window [t, t+△t], A threshold is the preset accuracy threshold, and α is a hyperparameter that controls the slope of the reward function. When the model accuracy exceeds the threshold, the reward function will tend to 1, giving a positive reward, and when the model accuracy is lower than the threshold, it will tend to 0, giving a negative reward. In this way, the model is guided to learn and optimize towards high accuracy.

[0104] The policy function π(a t |s t ): Model optimization strategy selection function where a t represents the action selected at time step t. t It represents the state at time step t, that is, the accuracy of sample prediction in the time window [t, t+△t]. t ,a t ) represents the state-action pair (s t ,a t ) is the estimated value learned during the training process. β is a temperature parameter that controls the distribution of action selection probabilities. A higher β makes the probability distribution smoother, while a lower β makes the probability distribution more concentrated on the optimal action. N represents the size of the action space, that is, the number of sampling methods that can be selected.

[0105] The strategy function designed in this way can be based on the current state s tand the estimated value Q(s) for each sampling method in the action space t ,a t ) is used to calculate the selection probability of each sampling method. Sampling methods with higher probabilities are more likely to be selected. At the same time, the choice of temperature parameter β can affect the randomness of action selection, thereby adjusting the balance between exploration and exploitation of the strategy.

[0106] 3. Reinforcement learning update. After building the reinforcement learning framework, the sampling strategy and the model in step 2 are updated iteratively to obtain the optimal model results. The reinforcement learning update process uses the Q-learning algorithm:

[0107]

[0108] Where Q(s,a) is the state action value function, which represents the expected return of performing action a in state s. α is the learning rate, which controls the step size of each update to control the learning rate. γ is the discount factor, which is used to balance the importance of immediate rewards and future rewards and determines the importance of future rewards. t is in state s t Execute action a t Instant rewards after. Indicates that in the next state s t+1 The maximum expected reward for selecting action a.

[0109] Based on the accuracy of the output prediction of the large model trained in step 2, the most appropriate time series sampling strategy is selected through reinforcement learning, and the large model is further fine-tuned and evaluated.

[0110] The present invention is based on the existing open source financial big language model, uses the credit review text, credit review video and vehicle GPS data of auto loans to build a multimodal risk control big model; then based on the constructed multimodal risk control big model, the big model fine-tuning technology is used to perform domain adaptation to generate risk degree and risk point descriptions; finally, a reinforcement learning method based on temporal strategy sampling is used to sample the constructed supervised fine-tuning data set to fine-tune the risk control big model, and the fine-tuning model is evaluated and fed back to obtain a multimodal risk control big model with better performance.

[0111] Compared with the existing methods, the present invention proposes a customer risk prediction method based on a big model. By using a big model to mine multi-modal and multi-dimensional user information, more hidden information of the customers to be predicted can be mined, the accuracy of the prediction and the efficiency of the early warning can be improved, and more accurate and effective risk early warning decision support can be provided to financial institutions.

[0112] The technical innovations of the present invention include: (1) Based on the credit review text, credit review video and vehicle GPS data of auto loans, a multimodal risk control model is constructed, which can process multimodal data in the field of auto finance, thereby realizing all-round and multi-angle risk prevention and control of auto loans; (2) A customer risk prediction method based on a fine-tuned large model is designed. By inputting multimodal and multi-dimensional customer information, more hidden information of the customer is mined based on the fine-tuned financial language large model, and the customer's risk level and risk point description are output, which effectively improves the accuracy of risk warning; (3) A reinforcement learning method based on time series strategy sampling is designed to fine-tune the large model, which significantly improves the prediction accuracy of the large model.

[0113] Obviously, the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, they are all within the protection scope of the present invention.

Claims

1. A method for identifying post-loan risks in automobile finance based on a multimodal large model, characterized by: The established multi-modal risk control model is used to process the input credit review text, credit review video and vehicle GPS data to predict and output the customer's risk level and risk point description; The establishment of a multi-modal risk control model includes: Step 1: Based on the financial language model, use the credit review text, credit review video and vehicle GPS data of auto loans to build a multimodal risk control model; Step 2: Fusion encode multimodal data and train the large model through fine-tuning; Step 3: Use reinforcement learning methods to perform time-series sampling on the training data set, fine-tune the multimodal risk control model again and evaluate the results to obtain the optimal multimodal risk control model.

2. The method for identifying post-loan risks of automobile finance based on a multimodal large model as claimed in claim 1, characterized in that: In step one, the data input into the multimodal risk control model is preprocessed. The preprocessing of video data includes using adaptive sampling based on overlap rate and fixed sampling based on time series to obtain a certain number of video key frames as visual modality input data; the preprocessing of GPS data includes converting GPS data into feature input data of both visual and textual modalities.

3. The method for identifying post-loan risks of automobile finance based on a multimodal large model as claimed in claim 1, characterized in that: In step one, the construction of a large risk control model based on multimodal data includes the processing of videos and images based on the large language model structure by adding a visual modal encoding structure, and adding a visual input mapping module to align the visual modal vector to the text modal vector space, thereby completing the construction of a large multimodal model.

4. The method for identifying post-loan risks of automobile finance based on a multimodal large model as claimed in claim 3, characterized in that: The steps to build a multi-modal risk control model include: Step 1. Preprocess the customer's credit review video and GPS data to obtain a video frame set {F j |j=1,2,3,...,L} and GPS visual images to construct a multimodal large model visual feature input set {V i |i=1,2,3,...,L+1}, and use the visual encoder to process to obtain the visual feature vector; Step 2. Concatenate the customer's letter review text with the customer's GPS text, and then process the constructed text modal input based on the text feature encoding space of the large language model to construct a text feature vector; Step 3. Build a learnable visual input mapper to process the visual feature vector and align the encoded features of the visual modality with the text feature space to obtain the aligned visual feature vector; Step 4. Fuse the text feature vector with the aligned visual feature vector, input it into the backbone network based on the large language model for processing, and input the corresponding risk control results.

5. A method for identifying post-loan risks of automobile finance based on a multimodal large model as claimed in any one of claims 1 to 4, characterized in that: In step two, fine-tuning of the multimodal risk control large model includes encoding the input multimodal information using the coding structure built based on the large model after the multimodal risk control model is built, and training the large model through fine-tuning to generate prediction results for the risk level and corresponding risk point descriptions, including four steps: encoding of image input information, fusion of text input information, alignment of multimodal input vectors, and training and fine-tuning of the large model.

6. The method for identifying post-loan risks of automobile finance based on a multimodal large model as claimed in claim 5, characterized in that: The encoding of visual input information includes: the input information of the visual modality includes two types: GPS images constructed based on the sampled GPS trajectory and a set of fixed-length video frames obtained after sampling processing; these two types of image information are encoded using a visual encoder ViT based on a pre-trained model.

7. The method for identifying post-loan risks of automobile finance based on a multimodal large model as claimed in claim 5, characterized in that: Multimodal input vector alignment involves using an MLP-based mapping model to map the visual encoding vector to the text semantic encoding space to obtain the converted visual encoding vector v Tvp It is input into LLM together with the encoded text information input vector for training.

8. The method for identifying post-loan risks of automobile finance based on a multimodal large model as claimed in claim 5, characterized in that: The steps of training and fine-tuning the large model include: iteratively updating the multimodal risk control large model through fine-tuning based on low-rank matrices, fine-tuning method based on low-rank matrices, by constructing and training low-rank matrices, based on the visual encoding vector and text encoding vector constructed in step 2, and taking the risk level assessment and risk point description annotated by experts as benchmark values, fine-tuning the trained large model to be more adapted to the domain content.

9. A method for identifying post-loan risks of automobile finance based on a multimodal large model as claimed in any one of claims 1 to 8, characterized in that: Step three includes adopting a variety of training sampling strategies to sample the training data set, and then iteratively updating the sampling strategy and the multimodal risk control model constructed in step two through the constructed reinforcement learning framework to obtain the optimal model results.

10. The method for identifying post-loan risks of automobile finance based on a multimodal large model as claimed in claim 9, characterized in that: Building a reinforcement learning framework includes: Step 1. Define the state space S t , action space A t And the reward function R(s t ,a t ) Step 2. Initialize state s0 Step 3. According to the current state s t Select action a t , using strategy π(a t |s t ) to select actions, where π is the policy function Step 4. Execute action a t And observe the reward r of the environment feedback t and the new state s t+1 Step 5. Update the strategy based on the reward and use the reinforcement learning algorithm Q-learning to update the policy function Step 6. Repeat Step 3-5 until the stop condition is reached in: State space S t : It is defined as the accuracy of the large model's prediction of sample data in a certain time window [t, t+△t], which is designed as a continuous state space to more finely represent the performance changes of the model; Action Space A t : Defined as different sampling methods, different sampling based on time series strategies are performed on the data set before the time window [t, t+△t]; Reward function R(s t ,a t ): Use the accuracy of the model as the reward signal, designed in the form of a Sigmoid function Among them A t represents the accuracy of the model in the time window [t, t+△t], A threshold is the preset accuracy threshold, and α is a hyperparameter that controls the slope of the reward function. When the model accuracy exceeds the threshold, the reward function will tend to 1, giving a positive reward, and when the model accuracy is lower than the threshold, it will tend to 0, giving a negative reward. In this way, the model is guided to learn and optimize towards high accuracy. The policy function π(a t |s t ): Model optimization strategy selection function where a t represents the action selected at time step t; s t represents the state at time step t, that is, the accuracy of sample prediction in the time window [t, t+△t]; Q(s t ,a t ) represents the state-action pair (s t ,a t ) is the estimated value learned during the training process; β is a temperature parameter that controls the distribution of action selection probabilities. A higher β will make the probability distribution smoother, while a lower β will make the probability distribution more concentrated on the optimal action; N represents the size of the action space, that is, the number of selectable sampling methods.

Citation Information

Cited By

  • Method and device for identifying risk website

    CN119720089A

  • Method and apparatus for identifying a risky website

    CN119720089B

  • Tax risk assessment method, device and model and storage medium

    CN119887411A

  • Construction method of multi-modal large model for security risk identification

    CN120493188A

  • Automobile risk control time sequence prediction method based on large model technology

    CN120807128A