Business verification method and system based on multi-modal data fusion

Abstract

The embodiment of the invention provides a service verification method and system based on multi-modal data fusion, and the method comprises the steps: obtaining the multi-modal data of a to-be-verified target object, and enabling the multi-modal data to comprise structural data, text data, image data, and time series data; and according to the structured data, the text data, the image data and the time sequence data, respectively extracting a structured feature vector, a semantic feature vector, an image feature vector and a time sequence behavior feature vector corresponding to the multi-modal number. And performing classification prediction and regression prediction based on the structured feature vector, the semantic feature vector, the image feature vector and the time sequence behavior feature vector to obtain a verification result label and a verification result numerical value of the target object, and outputting the verification result label and the verification result numerical value of the target object. Therefore, the accuracy of service verification is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a business verification method and system based on multimodal data fusion. Background Art

[0002] Most current financial verification systems rely on rule-based templates or structured scorecards for risk assessment. These systems typically only process structured data, employ single-value algorithmic models, and lack the ability to understand unstructured data. For example, existing systems cannot effectively verify user transaction records, interview images, and textual descriptions of fund usage. Furthermore, current verification systems rely solely on structured data for verification, resulting in low accuracy rates due to the limited dimensionality of the data.

[0003] Therefore, how to improve the accuracy of business verification is an urgent problem that needs to be solved. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a business verification method and system based on multimodal data fusion to improve the accuracy of business verification.

[0005] In a first aspect, the present application provides a service verification method based on multimodal data fusion, comprising:

[0006] Acquire multimodal data of a target object to be verified, the multimodal data including structured data, text data, image data, and time series data, wherein the structured data and text data are derived from a verification request of the target object, the image data is derived from a face-to-face image of the target object, and the time series data is derived from historical transaction records of the target object;

[0007] Extracting, according to the structured data, the text data, the image data, and the time series data, a structural feature vector, a semantic feature vector, an image feature vector, and a temporal behavior feature vector corresponding to the multimodal data;

[0008] Performing classification prediction and regression prediction based on the structured feature vector, the semantic feature vector, the image feature vector, and the temporal behavior feature vector to obtain a verification result label and a verification result value of the target object;

[0009] Output the verification result label and the verification result value of the target object.

[0010] Optionally, performing classification prediction and regression prediction based on the structured feature vector, the semantic feature vector, the image feature vector, and the temporal behavior feature vector to obtain a verification result label and a verification result value of the target object includes:

[0011] splicing and fusing the structural feature vector, the semantic feature vector, the image feature vector, and the temporal behavior feature vector to obtain a verification factor feature vector of the target object;

[0012] Performing classification prediction on the verification factor feature vector to obtain the verification result label of the target object;

[0013] Perform regression prediction on the verification factor feature vector to obtain the verification result value of the target object.

[0014] Optionally, the splicing and fusing the structural feature vector, the semantic feature vector, the image feature vector, and the temporal behavior feature vector to obtain the verification factor feature vector of the target object includes:

[0015] splicing the semantic feature vector, the image feature vector, and the temporal behavior feature vector to obtain a first feature vector to be processed;

[0016] Using the first feature vector to be processed, the first weight matrix obtained by pre-training, and the first bias vector obtained by pre-training as input parameters of a rectified linear unit (ReLU) activation function, and generating a second feature vector to be processed through the ReLU activation function;

[0017] The structured feature vector and the second feature vector to be processed are concatenated to obtain the verification factor feature vector of the target object.

[0018] Optionally, extracting, based on the structured data, the text data, the image data, and the time series data, a structural feature vector, a semantic feature vector, an image feature vector, and a temporal behavior feature vector corresponding to the multimodal data, respectively, includes:

[0019] Inputting the structured data into a pre-trained gradient boosting decision tree model to extract the structured feature vector;

[0020] Inputting the text data into a deep learning model based on an attention mechanism to extract the semantic feature vector;

[0021] Inputting the image data into a convolutional neural network model to extract the image feature vector;

[0022] The time series data is input into a bidirectional long short-term memory network model to extract the time series behavior feature vector.

[0023] Optionally, before performing classification prediction and regression prediction based on the structured feature vector, the semantic feature vector, the image feature vector, and the temporal behavior feature vector to obtain the verification result label and verification result value of the target object, the method further includes a training method for a multi-layer perceptron model, wherein the multi-layer perceptron model is used to obtain the verification result label and verification result value of the target object based on the structured feature vector, the semantic feature vector, the image feature vector, and the temporal behavior feature vector, and the training method specifically includes:

[0024] Constructing an initial multilayer perceptron model and obtaining a training sample set for training the initial multilayer perceptron model, wherein the training sample set includes an input sample vector and sample annotations, wherein the input sample vector includes a sample structured feature vector, a sample semantic feature vector, a sample image feature vector, and a sample temporal behavior feature vector, and the sample annotations include a verification result annotation and a verification result numerical annotation, and the initial multilayer perceptron model uses a two-layer fully connected network as a prediction head to jointly perform regression prediction and classification prediction;

[0025] Inputting the training sample set into the initial multi-layer perceptron model for training, and determining the mean square error loss function of the regression prediction based on the number of training samples, the prediction verification result value of each training sample predicted and output by the initial multi-layer perceptron model, and the verification result value label of each training sample during the training process;

[0026] Determine, during the training process, a cross entropy loss function for the classification prediction based on the number of training samples, the probability that each training sample output by the initial multilayer perceptron model prediction is the first verification result label, and the verification result label of each training sample;

[0027] Using the preset first weight and second weight as weighted weights of the mean square error loss function and the cross entropy loss function, respectively, performing weighted fusion on the mean square error loss function and the cross entropy loss function to obtain the target loss function of the initial multi-layer perceptron model;

[0028] When the value of the target loss function is less than or equal to a preset loss function threshold, the training of the initial multi-layer perceptron model is stopped to obtain the multi-layer perceptron model.

[0029] Optionally, during the training of the multi-layer perceptron model, the method further includes:

[0030] In target iterative training of the initial multilayer perceptron model, randomly discarding target neurons corresponding to the discard probability in each fully connected layer of the initial multilayer perceptron model according to the discard probability;

[0031] Continuing the target iterative training based on the adjusted neuron state;

[0032] The step of randomly discarding target neurons corresponding to the discard probability in each fully connected layer of the initial multilayer perceptron model according to the discard probability specifically includes:

[0033] For each neuron set in the fully connected layer, generate a random number corresponding to each neuron in the neuron set, wherein the random number obeys a uniform distribution from 0 to 1;

[0034] Marking neurons whose random numbers are less than or equal to the discard probability as neurons to be discarded;

[0035] Resetting the output weight of the to-be-discarded neuron to zero and freezing the activation value transfer of the to-be-discarded neuron in the forward propagation;

[0036] During the back propagation process, the gradient update calculation of the to-be-discarded neurons is skipped.

[0037] Optionally, during the training of the multi-layer perceptron model, the method further includes:

[0038] Acquire a target sample subset from the training sample set;

[0039] Based on each feature dimension of the target sample subset, obtaining the mean and variance of each feature dimension in the target sample subset;

[0040] performing data normalization processing on each sample in the target sample subset according to the mean and the variance, wherein the mean of the samples in the target sample subset after normalization is 0 and the variance is 1;

[0041] According to the learnable scaling parameter and translation parameter, the normalized sample is linearly transformed to obtain a linearly transformed sample;

[0042] The linearly transformed samples are input into the initial multi-layer perceptron model to calculate the target loss function, and back propagation is performed according to the target loss function to update the parameters of the initial multi-layer perceptron model.

[0043] Optionally, after obtaining the multi-layer perceptron model, the method further includes:

[0044] Obtain a weight vector and an L1 norm of each neuron in the multilayer perceptron model, where the L1 norm is the sum of the absolute values ​​of all elements in the weight vector;

[0045] Obtaining the gradient of the target loss function with respect to the output of each neuron through a back-propagation algorithm, and determining the contribution of each neuron to the output of the multi-layer perceptron model based on the gradient corresponding to each neuron;

[0046] Determining an output contribution pruning threshold according to the output contribution corresponding to each neuron in the multilayer perceptron model;

[0047] Determining a parameter sparsity pruning threshold according to the L1 norm of each neuron in the multilayer perceptron model;

[0048] According to the output contribution pruning threshold and the parameter sparsity pruning threshold, redundant neurons in the multilayer perceptron model are deleted, and the model structure of the multilayer perceptron model is updated.

[0049] Optionally, after obtaining the multi-layer perceptron model, the method further includes:

[0050] Obtaining prediction results of the multilayer perceptron model in a validation set, wherein the prediction results include a validation result label set and a validation result value set;

[0051] Calculating a model prediction confidence index based on the distribution consistency of the verification result label set and the verification result value set;

[0052] When the model prediction confidence index is lower than a preset threshold, readjusting the fully connected layer dimension of the multi-layer perceptron model;

[0053] The calculation of the model prediction confidence index based on the distribution consistency of the verification result label set and the verification result value set specifically includes:

[0054] Dividing the verification result value set into a plurality of continuous intervals, each continuous interval corresponding to a verification result label;

[0055] Count the number of matches between the actual verification result label and the corresponding interval verification result label in each continuous interval;

[0056] Generating the model prediction confidence index according to the ratio of the number of matches to the total number of samples in the verification result value set;

[0057] The step of re-adjusting the fully connected layer dimension of the multi-layer perceptron model specifically includes:

[0058] Determining a dimensionality scaling factor of a fully connected layer based on a difference between the model prediction confidence index and the preset threshold;

[0059] Proportionally scaling the input feature dimension and the output feature dimension of the fully connected layer based on the dimension scaling factor to generate an adjusted fully connected layer structure;

[0060] Freeze all parameters except the fully connected layer in the multilayer perceptron model, and perform fine-tuning training on the adjusted fully connected layer based on the validation set.

[0061] In the second aspect, the present application provides a business verification system based on multimodal data fusion, which includes a machine-readable storage medium and a processor. The machine-readable storage medium stores machine-executable instructions. When the processor executes the machine-executable instructions, the business verification system based on multimodal data fusion implements the aforementioned business verification method based on multimodal data fusion.

[0062] In a third aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method as described in any one of the first aspects.

[0063] In a fourth aspect, the present application provides a computer program product, which, when executed by a processor, is used to implement the method as described in any one of the first aspects.

[0064] The business verification method and system based on multimodal data fusion provided by the present application obtains multimodal data of the target object to be verified, wherein the multimodal data includes structured data, text data, image data, and time series data. According to the structured data, the text data, the image data, and the time series data, the structural feature vector, semantic feature vector, image feature vector, and time series behavior feature vector corresponding to the multimodal data are extracted respectively. Based on the structural feature vector, the semantic feature vector, the image feature vector, and the time series behavior feature vector, classification prediction and regression prediction are performed to obtain the verification result label and verification result value of the target object, and output the verification result label and verification result value of the target object. This method can enrich the diversity of data representation used for verification by obtaining the multimodal data of the target object and extracting the corresponding various feature vectors. Moreover, these feature vectors can fully reflect the multidimensional characteristics of the target object in business verification, so that business verification based on the rich multidimensional characteristics of the target object can improve the analysis depth of the target object, thereby improving the accuracy of business verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0066] Figure 1 A flowchart of a service verification method based on multimodal data fusion provided in an embodiment of the present application;

[0067] Figure 2 A schematic diagram of the structure of a convolutional neural network model provided in an embodiment of the present application;

[0068] Figure 3 A schematic diagram of the structure of a bidirectional long short-term memory network model provided in an embodiment of the present application;

[0069] Figure 4 A flowchart of another service verification method based on multimodal data fusion provided in an embodiment of the present application;

[0070] Figure 5 A schematic diagram of the structure of a business verification system based on multimodal data fusion provided in an embodiment of the present application.

[0071] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0072] To help those skilled in the art better understand the present invention, the following will provide a clear and complete description of the technical solutions in the embodiments of the present invention, in conjunction with the accompanying drawings. It is apparent that the described embodiments are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by those skilled in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0073] The terms "first," "second," and so on, in the specification and claims of the present invention and the accompanying drawings are used to distinguish between different items, not to describe a specific order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, apparatus, product, or end comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed therein, or may optionally include other steps or elements inherent to such process, method, product, or end.

[0074] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute a separate or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0075] Figure 1 This is a flow chart of a business verification method based on multimodal data fusion provided in an embodiment of the present application. It should be understood that in other embodiments, the order of some steps in the business verification method based on multimodal data fusion in this embodiment can be shared with each other according to actual needs, or some steps can be omitted or maintained. Figure 1 As shown, the method may include the following steps:

[0076] Step S110: Acquire multimodal data of the target object to be verified.

[0077] Among them, multimodal data includes structured data, text data, image data, and time series data. Structured data and text data come from the verification request of the target object, image data comes from the face-to-face review image of the target object, and time series data comes from the historical transaction records of the target object.

[0078] In this embodiment, in an actual business verification scenario, when faced with the task of verifying the business of a target object, the first task is to collect its multimodal data. Taking a common financial credit business scenario as an example, the target object may be a customer applying for a loan. When the customer initiates a loan verification request, relevant structured data and text data can be obtained from the loan verification request.

[0079] Structured data is often presented in a tabular format and contains information across multiple dimensions. For example, a customer's basic identity information dimension might include numeric fields such as age, income, and occupation. These dimensions of structured data are interconnected and together form a preliminary description of the customer's basic situation.

[0080] Text data is the free text provided by customers when filling out loan applications. For example, it may describe the purpose of the loan, interview history, work experience, family status, etc. This text information can reflect the customer's needs and background at a more detailed level.

[0081] The face-to-face interview is a crucial step in the loan process, during which customer images are recorded. By collecting these interview images, we can generate image data. This image data contains a wealth of information, such as the customer's facial expression—confidence, anxiety, or hesitation—and body language—relaxation, tension, or other distinctive gestures. These image features can, to a certain extent, reflect the customer's psychological state and potential credit risk.

[0082] The target's historical transaction records are also a crucial data source. These records reflect a customer's financial transactions over a period of time and constitute time series data. They contain transaction information at different points in time, such as transaction amounts, frequency, and transaction partners. By analyzing this time series data, we can understand a customer's spending habits and capital flow patterns, providing valuable support for assessing their creditworthiness.

[0083] Step S120: extracting structural feature vectors, semantic feature vectors, image feature vectors, and temporal behavior feature vectors corresponding to multimodal data based on the structured data, text data, image data, and time series data.

[0084] In this embodiment, after obtaining the multimodal data of the target object, in order to better utilize these data for business verification, it is necessary to extract features from different types of data and convert the original data into more representative and computable feature vectors.

[0085] Step S121: input the structured data into a pre-trained gradient boosting decision tree model to extract a structured feature vector.

[0086] The gradient boosted decision tree model is highly effective at processing structured data. It is pre-trained using a large amount of structured data from historical loan customers. During training, the model continuously learns the relationships between different features and how these features relate to loan risk.

[0087] When the structured data of the current target object is input into a pre-trained gradient boosting decision tree model, the model processes the data according to its internal decision rules and node partitioning. The gradient boosting decision tree model is an ensemble model composed of multiple decision trees, each of which makes a decision on the input data.

[0088] Suppose structured data contains multiple feature dimensions, such as age, income level, and occupation. For the age feature, the decision tree within the model might learn the relationship between different age groups and loan risk based on historical data. For example, there might be a decision node that divides age into different ranges, such as youth, middle-aged, and elderly. When the input customer age is in the youth range, the decision tree will continue to make a decision along the corresponding branch.

[0089] For the income range feature, the model also uses historical data to determine the correlation between different income ranges and loan risk. There may be decision nodes that categorize income levels into low, medium, and high ranges. When the input customer's income falls into the medium income range, the decision tree will make a corresponding judgment.

[0090] Occupational type is also an important characteristic dimension. Different occupational types have different risk characteristics in credit assessment. Based on historical data, the model learns which occupational types are relatively high-risk and which are relatively low-risk. For example, some unstable occupational types may be considered high-risk, while some stable occupational types such as public servants may be considered low-risk.

[0091] In the process of the decision tree making decisions on each feature dimension, the data will be continuously divided and judged. The final output results of each decision tree will be combined to form a comprehensive decision result. This decision result will be converted into a vector form, namely a structured feature vector. This structured feature vector contains information of multiple dimensions, and each dimension represents the comprehensive evaluation result of the model on different features. For example, one dimension of the structured feature vector may represent the model's evaluation result on the relationship between the customer's age stage and loan risk, another dimension may represent the evaluation result on the relationship between the income level range and loan risk, and so on. In this way, the gradient boosting decision tree model converts the input structured data into a representative structured feature vector, providing strong support for subsequent business verification. Exemplarily, the structured data is input into the gradient boosting decision tree model, and the leaf node index where the structured data falls in the gradient boosting decision tree model is exported for One-hot encoding to form a sparse high-dimensional structured feature representation, namely a structured feature vector. For example, the structured feature vector can be a 1*100-dimensional feature vector .

[0092] Step S122: Input the text data into a deep learning model based on the attention mechanism to extract a semantic feature vector.

[0093] The deep learning model based on the attention mechanism can effectively process text data and extract semantic information from it. When customers fill out a text description of the loan purpose and personal situation in the loan application, this text data is input into the model.

[0094] Optionally, before inputting the text data into the deep learning model based on the attention mechanism, the text data can be cleaned first, for example, special characters and abnormal punctuation can be removed from the text data.

[0095] Deep learning models based on the attention mechanism typically consist of multiple layers, including an input layer, an embedding layer, an attention layer, and an output layer. In the input layer, text data is fed into the model as a sequence of characters. The embedding layer converts each character into a corresponding vector representation. This process converts discrete characters into points in a continuous vector space, facilitating model computation.

[0096] The attention layer is the core of the model, dynamically assigning weights to different characters or words based on the context of the text. When processing a customer's loan purpose description, the attention mechanism focuses on keywords and phrases related to loan risk. For example, if a customer mentions that the loan is for starting a business, the attention mechanism will focus on entrepreneurial-related terms such as "startup project" and "market prospects," assigning them higher weights. For the personal profile, the attention mechanism will also consider information about the customer's job stability and income source.

[0097] During the model's calculations, the attention mechanism continuously adjusts weight assignments based on contextual information. For example, when the phrase "stable job" appears in a text, the attention mechanism determines its importance to loan risk assessment based on the context and adjusts the weight of each word in the phrase accordingly.

[0098] The output layer integrates the vectors processed by the attention layer to generate a comprehensive semantic feature vector. This semantic feature vector contains semantic information related to loan risk in the text data. For example, one dimension of the semantic feature vector might represent the risk level of the loan purpose, while another dimension might represent the stability of the customer's personal situation. In this way, the deep learning model based on the attention mechanism transforms text data into a representative semantic feature vector, providing important semantic information for subsequent business verification.

[0099] For example, the text data can be segmented by a pre-trained word segmentation model and encoded to generate the corresponding token ID. By inputting the token ID into a deep learning model based on the attention mechanism (such as the bert-base-chinese model), the output of the classification tag ([CLS] Token) is taken as the representation of the entire text, and a 1*768-dimensional semantic feature vector is obtained. .

[0100] Step S123: Input the image data into the convolutional neural network model to extract the image feature vector.

[0101] Optionally, before inputting the image data into the convolutional neural network model, the image data can be scaled to a preset size (e.g., 224×224×3), normalized (e.g., the pixel values ​​of the image data are divided by 255), and stored in a tensor format.

[0102] Convolutional neural network models typically consist of convolutional layers, pooling layers, and fully connected layers. In the convolutional layers, the model uses multiple convolution kernels to perform sliding convolutions on the image. Each convolution kernel has its own weight parameters, and the convolution operation can extract different image features. For example, one convolution kernel might extract edge features in an image, while another might extract texture features.

[0103] When processing image data from customer interviews, the convolutional layer continuously performs convolution operations on the image to extract various features. For example, for an image of a customer's facial expression, the convolutional layer might extract features of areas like the eyes and mouth, which can reflect the customer's emotional state. For an image of a customer's body language, the convolutional layer might extract features like body posture and gestures.

[0104] The pooling layer downsamples the feature map output by the convolutional layer, reducing its dimensionality while retaining important feature information. Common pooling operations include max pooling and average pooling. Max pooling selects the maximum value in each region of the feature map as the representative value for that region, while average pooling selects the average value for that region as the representative value. Pooling can reduce the computational complexity of the model while improving its robustness.

[0105] The fully connected layer flattens the feature map output by the pooling layer and then performs calculations through a series of fully connected neurons. Each neuron in the fully connected layer is connected to all neurons in the previous layer. Through weighted summation and activation functions, the feature map is converted into a vector. This vector is the image feature vector, which contains characteristic information related to loan risk in the image data. For example, one dimension of the image feature vector may represent the degree of tension in the customer's facial expression, while another dimension may represent the confidence level of the customer's body language. In this way, the convolutional neural network model converts image data into a representative image feature vector, providing important image information for subsequent business verification.

[0106] For example, Figure 2 This is a structural diagram of a convolutional neural network model provided in an embodiment of the present application. Figure 2 As shown in the figure, the network structure of the convolutional neural network model includes:

[0107] Input layer: Input image of size 224×224×3.

[0108] Convolutional layer 1: Convolutional 2D (Conv2D) is performed using 32 3×3 kernels and a Rectified Linear Unit (ReLU) activation function.

[0109] Max Pooling Layer 1: Use a 2×2 window to perform a max pooling operation to reduce the data dimension.

[0110] Convolutional layer 2: performs two-dimensional convolution operations using 64 convolution kernels of size 3×3 and applies the ReLU activation function.

[0111] Max pooling layer 2: Again, a 2×2 window is used for the max pooling operation.

[0112] Convolutional layer 3: Two-dimensional convolution operation is performed using 128 convolution kernels of size 3×3, and the ReLU activation function is applied.

[0113] Global Average Pooling Layer: Apply the Global Average Pooling operation to average pool the feature maps to further reduce the data dimension.

[0114] Fully connected layer: A fully connected layer (FC) maps the 128-dimensional input to 64 dimensions and applies the ReLU activation function.

[0115] Output layer: output 1*64 dimensional image feature vector .

[0116] Step S124: Input the time series data into the bidirectional long short-term memory network model to extract the time series behavior feature vector.

[0117] In financial credit business scenarios, the bidirectional long short-term memory network model can effectively process time series data and extract temporal behavioral features. After obtaining a customer's historical transaction records as time series data, they are input into the model.

[0118] The bidirectional LSTM model consists of forward and reverse LSTM networks. LSTM is a special type of recurrent neural network that can handle long-term dependencies in sequential data. When processing time series data, the forward LSTM network processes the data in chronological order, starting with the oldest transaction and working backwards to the latest. The reverse LSTM network processes the data in reverse chronological order, starting with the latest transaction and working forwards to the oldest transaction.

[0119] During the forward and reverse LSTM computations, each time step updates the hidden state based on the input data of the current time step and the hidden state of the previous time step. LSTM networks control the flow of information through gating mechanisms, including input gates, forget gates, and output gates. The input gate determines how much information from the input data of the current time step is added to the hidden state; the forget gate determines how much information from the hidden state of the previous time step is retained; and the output gate determines how much information from the hidden state of the current time step is output.

[0120] When processing a customer's historical transaction records, the bidirectional long short-term memory network model focuses on information such as the chronological order of transactions, changes in transaction amounts, and transaction frequency. For example, if a transaction amount suddenly and significantly increases over a period of time, the model uses a gating mechanism to retain this information in a hidden state and factor it into subsequent calculations.

[0121] Ultimately, the bidirectional LSTM model merges the forward and reverse hidden states to generate a comprehensive temporal behavior feature vector. This temporal behavior feature vector contains temporal behavior information related to loan risk in the time series data. For example, one dimension of the temporal behavior feature vector might represent the changing trend of a customer's transaction amount, while another dimension might represent the stability of the customer's transaction frequency. In this way, the bidirectional LSTM model transforms time series data into a representative temporal behavior feature vector, providing important temporal information for subsequent business verification.

[0122] For example, we can collect the customer's transaction flow within the last three months as time series data, sort the time series data in ascending order by time, and construct a structured input vector for each transaction flow, including features such as transaction type, amount, account balance, time interval, whether it is a holiday, and the party where funds flow, encoded as a vector of dimension d (assuming the dimension is 10). Ultimately, each customer's transaction flow constitutes a T×d time series input (T is the number of transactions, assuming the dimension is 60). The model structure is as follows Figure 3 As shown in the figure, the final 1*256-dimensional temporal behavior feature vector is obtained. . Figure 3 This is a schematic diagram of the structure of a bidirectional long short-term memory network model provided in an embodiment of the present application. Figure 3 As shown in FIG, the network structure of the bidirectional long short-term memory network model includes:

[0123] Input layer: The input sequence (i.e., the input sequence corresponding to the time series data) is represented as d1, d2, d3, ..., dt.

[0124] Forward Long Short-Term Memory Layer: The input sequence passes through the forward long short-term memory (LSTM) layer, which computes and generates a forward hidden state sequence h0, h1, …, ht. The forward LSTM layer processes data from the beginning to the end of the input sequence, capturing dependencies from the past to the future.

[0125] Backward LSTM layer: Simultaneously, the input sequence passes through the backward LSTM layer, which computes and generates the backward hidden state sequence h0′, h1′, …, ht′. The backward LSTM layer processes data from the end to the beginning of the input sequence, capturing dependencies from the future to the past.

[0126] Hidden state merging: The hidden states of the forward and backward LSTM layers are merged at each time step. For example, ht and ht′ are merged to form the final sequence representation, which is a 1*256-dimensional temporal behavior feature vector. .

[0127] Step S130: Perform classification prediction and regression prediction based on the structured feature vector, semantic feature vector, image feature vector, and temporal behavior feature vector to obtain a verification result label and a verification result value of the target object.

[0128] In this embodiment, after extracting the feature vectors corresponding to the multimodal data, it is necessary to use these feature vectors to perform classification prediction and regression prediction to obtain the verification result label and verification result value of the target object.

[0129] Step S131: splicing and fusing the structural feature vector, semantic feature vector, image feature vector, and temporal behavior feature vector to obtain a verification factor feature vector of the target object.

[0130] In financial credit business scenarios, in order to fully utilize the feature information of different types of data, it is necessary to splice and fuse structured feature vectors, semantic feature vectors, image feature vectors and temporal behavior feature vectors.

[0131] Step S1311: concatenate the semantic feature vector, the image feature vector, and the temporal behavior feature vector to obtain a first feature vector to be processed.

[0132] First, a simple concatenation operation is performed on the semantic feature vector, image feature vector, and temporal behavior feature vector. These three feature vectors represent the feature information of text data, image data, and time series data, respectively. This concatenation operation integrates these different types of feature information to form a more comprehensive feature vector. For example, the semantic feature vector contains semantic information about the loan purpose and personal information, the image feature vector contains the image information of the customer's interview video, and the temporal behavior feature vector contains the temporal information of the customer's historical transaction records. After concatenation, the first feature vector to be processed contains the comprehensive feature information of these three types of data.

[0133] For example, the calculation formula of this step can be shown as the following formula (1):

[0134] (1)

[0135] in, is the first feature vector to be processed with dimension 1*1088, is a 1*768-dimensional semantic feature vector, is a 1*64-dimensional image feature vector, It is a 1*256-dimensional temporal behavior feature vector.

[0136] Step S1312: using the first feature vector to be processed, the first weight matrix obtained by pre-training, and the first bias vector obtained by pre-training as input parameters of a rectified linear unit (ReLU) activation function, and generating a second feature vector to be processed through the ReLU activation function.

[0137] After obtaining the first feature vector to be processed, it needs to be further processed. This process involves performing a matrix multiplication operation on the first feature vector to be processed and the first weight matrix obtained by pre-training. This process weights each dimension of the first feature vector to be processed. The purpose of weighting is to highlight certain important feature information while reducing the influence of less important feature information.

[0138] The result of the matrix multiplication is then added to the pre-trained first bias vector. The first bias vector can adjust the overall offset of the weighted result so that the model can better adapt to the distribution of the data.

[0139] Finally, the weighted and biased result is used as the input to the rectified linear unit (ReLU) activation function. The ReLU activation function is a commonly used activation function that sets input values ​​less than 0 to 0 and leaves input values ​​greater than 0 unchanged. The ReLU activation function introduces nonlinearity and enhances the model's expressiveness. After processing with the ReLU activation function, the second feature vector to be processed is obtained.

[0140] Exemplarily, the calculation formula of this step can be shown as the following formula (2):

[0141] (2)

[0142] in, is the second feature vector to be processed with dimension 1*100, is the first weight matrix, is the first bias vector.

[0143] Step S1313: concatenate the structured feature vector and the second feature vector to be processed to obtain a verification factor feature vector of the target object.

[0144] After obtaining the second unprocessed feature vector, it is concatenated with the structured feature vector. The structured feature vector represents the feature information of the structured data, while the second unprocessed feature vector contains the combined feature information of semantic, image, and time series data. By concatenating these two vectors, a verification factor feature vector containing the feature information of all multimodal data is obtained. This verification factor feature vector will serve as input data for subsequent classification and regression predictions, providing strong support for accurately obtaining the verification result label and verification result value of the target object.

[0145] Exemplarily, the calculation formula of this step can be shown as the following formula (3):

[0146] (3)

[0147] in, It is a 1*200-dimensional verification factor feature vector.

[0148] Step S132: Classify and predict the verification factor feature vector to obtain the verification result label of the target object.

[0149] In financial credit scenarios, a trained classification model is used to perform classification predictions on the validation factor feature vector. This classification model can be a multi-layer perceptron model, for example. The validation factor feature vector contains comprehensive feature information from the target object's multimodal data.

[0150] The classification model internally contains multiple neurons and connection weights. Through the forward propagation process, the validation feature vector is input into the model. The model processes the input validation feature vector based on its internal weights and neuron calculation rules. During this processing, the model learns the relationships between different features and how these features are associated with different classification labels.

[0151] For example, in the loan business, classification labels might include "loan granted" and "loan denied." The model determines which classification label best matches the target object based on the feature information in the verification factor feature vector. Ultimately, the model outputs a classification result, which is the verification result label for the target object. This label helps financial institutions quickly understand the target object's credit risk level and make appropriate decisions.

[0152] Step S133: Perform regression prediction on the verification factor feature vector to obtain the verification result value of the target object.

[0153] Similarly, in financial credit business scenarios, a trained regression model is used to perform regression prediction on the feature vector of the verification factor. The regression model can also be a multi-layer perceptron model, etc.

[0154] The goal of a regression model is to predict a continuous value. For example, in the loan business, this might be to predict the target's credit score or loan amount. During the regression prediction process, the validation factor feature vector is passed into the model as input. The model processes the input feature vector based on its internal weights and neuron calculation rules.

[0155] The model learns the relationship between the features in the validation factor's feature vector and the target value. By continuously adjusting its internal weights, the model's predictions are kept as close as possible to the true target value. Ultimately, the model outputs a predicted value, which serves as the validation result for the target object. This value can provide financial institutions with more specific reference information, helping them make more informed lending decisions.

[0156] Step S140: Output the verification result label and verification result value of the target object.

[0157] In financial credit business scenarios, after completing classification and regression predictions, the system outputs the target object's verification result label and value. This output can be presented to financial institution staff in the form of a report, or fed back to relevant business systems through a system interface.

[0158] Verification result labels intuitively inform staff of the target customer's credit risk level, such as "high-quality customer," "average customer," or "risky customer," or alternatively, "loan granted" or "loan denied." The numerical verification result provides more specific information, such as credit score and loan amount. Based on these output verification results, staff can quickly make loan approval decisions, including whether to approve the loan, the loan amount, and the interest rate. These results can also be used for subsequent risk management and customer relationship management, helping financial institutions better assess the risks associated with their businesses.

[0159] In one possible implementation, the aforementioned classification prediction and regression prediction of the verification factor feature vector can be implemented using a two-layer multilayer perceptron network model. For example, the verification factor feature vector can be input into the multilayer perceptron network model, and classification prediction and regression prediction can be performed simultaneously, thereby outputting the verification result label and verification result value of the target object. For example, the prediction process can be shown as follows: ,in, Is a two-dimensional vector, namely [verification result label, verification result value], is the corresponding weight matrix, is the corresponding bias vector.

[0160] The method provided in the embodiment of the present application obtains multimodal data of the target object to be verified, and the multimodal data includes structured data, text data, image data, and time series data. According to the structured data, text data, image data, and time series data, the structured feature vector, semantic feature vector, image feature vector, and time series behavior feature vector corresponding to the multimodal data are extracted respectively. Based on the structured feature vector, semantic feature vector, image feature vector, and time series behavior feature vector, classification prediction and regression prediction are performed to obtain the verification result label and verification result value of the target object, and output the verification result label and verification result value of the target object. This method can enrich the diversity of data representation used for verification by obtaining the multimodal data of the target object and extracting the corresponding various feature vectors. Moreover, these feature vectors can fully reflect the multidimensional characteristics of the target object in terms of business verification, so that business verification is performed based on the rich multidimensional characteristics of the target object, which can improve the analysis depth of the target object, thereby improving the accuracy of business verification.

[0161] Next, we will continue to introduce the training process of the above multi-layer perceptron model in detail. Figure 4 This is a flow chart of another service verification method based on multimodal data fusion provided in an embodiment of the present application. Figure 4 As shown, the method further includes:

[0162] Step S210: constructing an initial multi-layer perceptron model and obtaining a training sample set for training the initial multi-layer perceptron model.

[0163] Among them, the training sample set includes input sample vectors and sample annotations. The input sample vectors include sample structured feature vectors, sample semantic feature vectors, sample image feature vectors, and sample temporal behavior feature vectors. The sample annotations include verification result annotations and verification result numerical annotations. The initial multi-layer perceptron model uses a two-layer fully connected network as the prediction head to jointly perform regression prediction and classification prediction.

[0164] In financial credit scenarios, to accurately perform classification and regression predictions on target objects, a multilayer perceptron model must be built and trained. First, an initial multilayer perceptron model is constructed. This model consists of an input layer, a hidden layer, and an output layer. The input layer receives an input sample vector, which contains the sample's structural feature vector, the sample's semantic feature vector, the sample's image feature vector, and the sample's temporal behavior feature vector. During the construction process, it is important to ensure that the number of neurons in the input layer matches the dimensionality of the input sample vector to ensure that the data is correctly input into the model.

[0165] Hidden layers are the core of the multilayer perceptron model. There can be multiple hidden layers, each containing multiple neurons. Neurons are connected to each other via connection weights, which are continuously adjusted during training. The hidden layer performs nonlinear transformations on the input data, thereby extracting complex features from the data. In this embodiment, the number of neurons and the number of layers in the hidden layer need to be appropriately set based on the specific task and data characteristics to balance the model's complexity and generalization ability.

[0166] The output layer uses a two-layer fully connected network as the prediction head, which is used to jointly perform regression and classification predictions. The first layer of the fully connected network is responsible for regression prediction, outputting a continuous numerical value corresponding to the numerical value of the verification result in the sample annotation; the second layer of the fully connected network is responsible for classification prediction, outputting a classification label corresponding to the verification result label in the sample annotation. In this way, the model can perform regression and classification tasks simultaneously, improving the accuracy and comprehensiveness of predictions.

[0167] Next, we obtain a training sample set for training the initial multilayer perceptron model. This training sample set is based on a large amount of historical data covering different types of sample objects and their corresponding multimodal data. For each sample object, we first extract its structural feature vector, semantic feature vector, image feature vector, and temporal behavior feature vector using the methods described above. These feature vectors are then combined to form the input sample vector. Simultaneously, based on the actual situation in the historical data, each target object is annotated with a verification result label and a verification result value to form a sample annotation.

[0168] When collecting training samples, it's important to ensure consistent dimensionality and matching feature dimensions across different features. Because structured feature vectors, semantic feature vectors, image feature vectors, and temporal behavior feature vectors may come from different data types, their dimensions and dimensions may differ. Therefore, these feature vectors need to be preprocessed to ensure they have consistent dimensions and matching dimensions before being input into the model. For example, normalization or standardization can be used to unify the value ranges of different feature vectors to the same interval, thus avoiding model training instability caused by different dimensions.

[0169] Step S220: Input the training sample set into the initial multi-layer perceptron model for training, and during the training process, determine the mean square error loss function of the regression prediction based on the number of training samples, the prediction verification result value of each training sample predicted and output by the initial multi-layer perceptron model, and the verification result value label of each training sample.

[0170] The training process is an iterative optimization process, and the model will continuously adjust the internal connection weights to make the prediction results as close to the sample labels as possible.

[0171] During each training iteration, the model processes each training example in the training set and outputs a predicted verification value. This predicted verification value is derived from the model's internal neuron calculations and connection weights based on the input sample feature vector. The predicted verification value for each training example is then compared with the corresponding verification value annotation.

[0172] To measure the model's performance on regression prediction tasks, we need to determine the mean squared error loss function for regression prediction. The mean squared error loss function is calculated based on the number of training samples, the predicted verification result value for each training sample, and the verification result value label. The specific calculation process is as follows: First, for each training sample, the difference between its predicted verification result value and the verification result value label is calculated, and then this difference is squared. Next, the squared differences of all training samples are added together, and finally the result of the addition is divided by the number of training samples to obtain the mean squared error loss function value.

[0173] The mean squared error loss function (MSE) reflects the degree of error in the model's regression prediction task. Smaller values ​​indicate closer the model's predictions are to the actual validation results, indicating better model performance. During training, the MSE loss function is gradually reduced by continuously adjusting the model's connection weights, thereby improving the model's regression prediction capabilities.

[0174] Step S230: During the training process, a cross entropy loss function for classification prediction is determined based on the number of training samples, the probability that each training sample output by the initial multilayer perceptron model is the first verification result label, and the verification result label of each training sample.

[0175] In the model training process for financial credit business scenarios, in addition to focusing on the performance of regression prediction, it is also necessary to pay attention to the performance of classification prediction. For classification prediction tasks, it is necessary to determine the cross-entropy loss function.

[0176] During each training iteration, the model outputs the probability of each training example being labeled as the first validation result. This probability is derived from the model's internal neuron calculations and connection weights based on the input sample feature vector. Each training example is also annotated with a corresponding validation result, representing the true classification label.

[0177] The cross-entropy loss function is calculated based on the number of training samples, the probability that each training sample is labeled as the first validation result, and the validation result label. The specific calculation process is as follows: For each training sample, if its validation result label is labeled as the first validation result, then the cross-entropy loss function is calculated as the negative logarithm of the probability that the training sample is labeled as the first validation result; if its validation result label is not the first validation result, then the cross-entropy loss function is calculated as the negative logarithm of the probability that the training sample is not labeled as the first validation result. Then, the cross-entropy calculation terms of all training samples are added together, and finally the result of the addition is divided by the number of training samples to obtain the value of the cross-entropy loss function.

[0178] The cross-entropy loss function reflects the degree of error in the model's classification prediction task. A smaller value indicates that the model's classification prediction results are closer to the true classification label, and the model's performance is better. During training, by continuously adjusting the model's connection weights, the cross-entropy loss function value is gradually reduced, thereby improving the model's classification prediction ability.

[0179] Step S240: Using the preset first weight and second weight as weighted weights of the mean square error loss function and the cross entropy loss function, respectively, performing weighted fusion on the mean square error loss function and the cross entropy loss function to obtain the target loss function of the initial multi-layer perceptron model.

[0180] In model training for financial credit business scenarios, in order to comprehensively consider the performance of regression prediction and classification prediction, it is necessary to perform a weighted fusion of the mean square error loss function and the cross entropy loss function to obtain the target loss function of the initial multi-layer perceptron model.

[0181] The preset first and second weights are used to measure the importance of the mean squared error loss function and the cross entropy loss function in the target loss function, respectively. The settings of these two weights need to be adjusted according to specific business needs and data characteristics. For example, if the accuracy of regression predictions in financial credit business is more important, that is, the prediction accuracy of the verification result values ​​is required to be higher, then the first weight can be appropriately increased; if the accuracy of classification predictions is more important, that is, the classification accuracy of the verification result labels is required to be higher, then the second weight can be appropriately increased.

[0182] The specific process of weighted fusion is to multiply the mean squared error loss function by the first weight, the cross entropy loss function by the second weight, and then add these two weighted loss functions to obtain the value of the target loss function. In this way, the target loss function can comprehensively reflect the overall performance of the model in regression prediction and classification prediction tasks.

[0183] During the training process, the model's goal is to minimize the target loss function by adjusting the internal connection weights. This ensures that the model achieves good performance in both regression and classification prediction tasks, and improves the model's accuracy in verifying the target object.

[0184] Step S250: When the value of the target loss function is less than or equal to the preset loss function threshold, the training of the initial multi-layer perceptron model is stopped to obtain a multi-layer perceptron model.

[0185] During model training for financial credit scenarios, the initial multilayer perceptron model is continuously iterated. Each iteration adjusts the model's connection weights to gradually reduce the target loss function. The preset loss function threshold is a pre-set criterion used to determine whether the model has been trained sufficiently well.

[0186] When the target loss function value is less than or equal to the preset loss function threshold, it indicates that the overall error of the model in the regression prediction and classification prediction tasks has reached an acceptable range. At this point, the training of the initial multilayer perceptron model is stopped, and the current model state is determined as the final multilayer perceptron model.

[0187] This multi-layer perceptron model has been trained and optimized using a large number of training samples. It can accurately output the target object's verification result label and value based on the input sample feature vector. In subsequent business verification processes, this trained multi-layer perceptron model can be directly used to verify new target objects, improving the efficiency and accuracy of business verification.

[0188] In one possible implementation, the training process of the multilayer perceptron model may further include the following steps:

[0189] Step S260: In the target iterative training of the initial multilayer perceptron model, target neurons corresponding to the dropout probability are randomly dropped in each fully connected layer of the initial multilayer perceptron model according to the dropout probability.

[0190] In the financial credit business scenario, a dropout strategy is used during the training of the multi-layer perceptron model to prevent overfitting. During target iterative training, target neurons are randomly selected and dropped based on the dropout probability for each fully connected layer of the initial multi-layer perceptron model.

[0191] The specific operation is as follows: For each set of neurons in the fully connected layer, a random number is first generated for each neuron in the set. These random numbers follow a uniform distribution between 0 and 1. This means that each neuron has an equal probability of being selected. Then, these random numbers are compared with the drop probability. If the random number corresponding to a neuron is less than or equal to the drop probability, the neuron is marked as a neuron to be dropped.

[0192] For example, assuming the drop probability is a set ratio, for a set of neurons in a fully connected layer, the random number generated by each neuron is randomly distributed between 0 and 1. When the random number of a neuron falls within a range less than or equal to the drop probability, it is marked as a neuron to be dropped. This random selection method allows different neurons to be randomly dropped during different training iterations, improving the model's generalization ability.

[0193] Step S261: reset the output weights of the to-be-discarded neurons to zero, and freeze the activation value transfer of the to-be-discarded neurons in the forward propagation.

[0194] Once the neurons to be discarded have been identified, they need to be processed. First, the output weights of the neurons to be discarded are reset to zero. The output weights are the weights of the connections between neurons and neurons in the next layer. Setting them to zero means that the neuron will not affect the neurons in the next layer in subsequent calculations.

[0195] At the same time, the activation value propagation of the neurons to be dropped is frozen during the forward propagation. During forward propagation, neurons calculate activation values ​​based on input signals and pass these activation values ​​to the neurons in the next layer. For the neurons to be dropped, their activation values ​​are frozen, meaning they are not allowed to propagate to the next layer. This ensures that the neurons to be dropped do not participate in the model calculations during this training iteration, thus achieving a random dropout effect.

[0196] For example, in a fully connected layer, some neurons are marked as discarded neurons. During the forward propagation calculation, the output weights of these discarded neurons are zero, and their activation values ​​are not passed to the next layer. These neurons have no effect on the calculation results in this calculation.

[0197] Step S262: During the back propagation process, the gradient update calculation of the neurons to be discarded is skipped.

[0198] In the training process of the multilayer perceptron model, backpropagation is an important step for updating the model's connection weights. During the backpropagation process, the gradient of each neuron needs to be calculated and then the connection weights are adjusted based on the gradient.

[0199] For neurons marked as to be discarded, their gradient update calculations are skipped during the back-propagation process. This is because these neurons have been discarded in this iterative training and do not participate in the forward propagation calculations, so their gradient updates are meaningless.

[0200] For example, when calculating gradients, the neurons in each layer are traversed. When encountering neurons to be discarded, the gradient calculation step for them is skipped and their connection weights are not updated. This ensures that during model training, only the connection weights of neurons participating in the forward propagation are updated, thus avoiding overfitting.

[0201] Step S263: Continue target iterative training based on the adjusted neuron state.

[0202] After completing the processing of the neurons to be discarded in the initial multi-layer perceptron model, that is, after the output weights are reset to zero, the activation value transfer is frozen, and the gradient update calculation is skipped, the target iterative training continues based on the adjusted neuron state.

[0203] During subsequent training, the model performs forward propagation calculations on the input training samples based on the new neuron states to obtain prediction results. Then, the target loss function is calculated based on the prediction results and sample labels, and backpropagation is performed to update the connection weights of the neurons involved in the calculation.

[0204] As iterative training progresses, the model continuously adjusts the connection weights, gradually reducing the value of the target loss function. At the same time, thanks to the use of a dropout strategy, the model can avoid overfitting the training data, improving the model's generalization ability and enabling more accurate verification when faced with new target objects.

[0205] Optionally, the following steps may be included in the training process of the multilayer perceptron model:

[0206] Step S270: During the training process of the multi-layer perceptron model, a target sample subset is obtained from the training sample set.

[0207] During the multi-layer perceptron model training process in the financial credit business scenario, in order to improve the training efficiency and model stability, it is necessary to obtain a target sample subset from the training sample set.

[0208] The training sample set contains a large number of training samples. Directly using the entire training sample set for training may result in a waste of computing resources and an increase in training time. Therefore, a portion of samples are randomly selected from the training sample set to form the target sample subset.

[0209] The process of selecting the target sample subset must ensure both randomness and representativeness. Randomness ensures that each sample has an equal probability of being selected, thus avoiding sample bias. Representativeness requires that the target sample subset reflect the characteristic distribution of the entire training sample set.

[0210] For example, a random sampling method can be used to randomly select a certain proportion of samples from the training sample set as the target sample subset. This ensures that the target sample subset has similar feature distribution to the entire training sample set, allowing the model to learn more comprehensive feature information during training.

[0211] Step S271: Based on each feature dimension of the target sample subset, obtain the mean and variance of each feature dimension in the target sample subset.

[0212] After obtaining the target sample subset, it needs to be further processed. For each feature dimension in the target sample subset, its mean and variance are calculated respectively.

[0213] The mean is the average value of a set of data, reflecting the data's central tendency. The variance measures the degree of dispersion of the data. Calculating the mean and variance for each feature dimension can help us understand the distribution of the target sample subset along each feature dimension.

[0214] The specific calculation process is as follows: For each feature dimension, the values ​​of all samples in the target sample subset on that feature dimension are added together, and then divided by the number of samples to obtain the mean value of that feature dimension. Next, for each sample value on that feature dimension, the difference between its value and the mean is calculated, the difference is squared, and the squared differences of all samples are added together and divided by the number of samples to obtain the variance of that feature dimension.

[0215] For example, suppose that multiple samples in the target sample subset have values ​​for a certain feature dimension. First, add these values ​​and divide by the number of samples to obtain the mean for that feature dimension. Then, square the difference between each sample value and the mean, add these squared values, and divide by the number of samples to obtain the variance for that feature dimension. This calculation yields the mean and variance for each feature dimension in the target sample subset.

[0216] Step S272: performing data normalization processing on each sample in the target sample subset according to the mean and variance, wherein the mean of the samples in the target sample subset after normalization is 0 and the variance is 1.

[0217] After obtaining the mean and variance of each feature dimension of the target sample subset, data normalization is performed on each sample in the target sample subset. The purpose of data normalization is to unify data of different feature dimensions to the same scale, thereby avoiding model training instability caused by different feature scales.

[0218] The specific normalization method is to first subtract the mean of each feature dimension from the value of each sample in the target sample subset, and then divide it by the standard deviation of that feature dimension (the standard deviation is the square root of the variance). Through this process, the data of each feature dimension will be converted to a distribution with a mean of 0 and a variance of 1.

[0219] For example, for the value of a particular feature dimension of a sample in the target sample subset, first subtract the mean of that feature dimension to obtain a difference. Then, divide this difference by the standard deviation of that feature dimension to obtain the normalized value. This process is repeated for all samples in the target sample subset in all feature dimensions to complete the data normalization process.

[0220] Step S273: performing a linear transformation on the normalized samples according to the learnable scaling parameters and translation parameters to obtain linearly transformed samples.

[0221] After completing the data normalization, the normalized samples need to be linearly transformed. The linear transformation uses learnable scaling and translation parameters.

[0222] The scaling parameter is used to adjust the scale of the normalized data, and the translation parameter is used to adjust the position of the data. By performing a linear transformation on the normalized samples, the expressive power of the model can be increased, making the model more adaptable to different data distributions.

[0223] The specific linear transformation process is to multiply the normalized samples by the scaling parameter and then add the translation parameter to obtain the linearly transformed samples. The scaling and translation parameters are parameters that the model needs to learn during training and are continuously adjusted based on the characteristics of the training data and the performance of the model.

[0224] For example, for the value of a normalized sample in a certain feature dimension, multiply it by the scaling parameter and then add the translation parameter to obtain the value of the sample in that feature dimension after linear transformation. This process is repeated for all samples in all feature dimensions to obtain the linearly transformed samples.

[0225] Step S274: Input the linearly transformed samples into the initial multi-layer perceptron model to calculate the target loss function, and perform back propagation according to the target loss function to update the parameters of the initial multi-layer perceptron model.

[0226] After obtaining the linearly transformed sample, it is input into the initial multi-layer perceptron model. The model will perform forward propagation calculation on the input sample to obtain the prediction result.

[0227] Then, the target loss function is calculated based on the prediction results and sample labels. The target loss function comprehensively considers the errors of regression prediction and classification prediction, and reflects the performance of the model in its current state.

[0228] Next, backpropagation is performed based on the value of the target loss function. Backpropagation is an algorithm used to calculate gradients. It calculates the gradient of each parameter in the model based on the value of the target loss function. The gradient represents the rate of change of the target loss function with respect to each parameter.

[0229] Finally, the parameters of the initial multilayer perceptron model are updated based on the calculated gradient. This parameter update is achieved using the gradient descent algorithm, which adjusts the parameter values ​​in the opposite direction of the gradient, gradually reducing the target loss function. By continuously training the model, the parameters are continuously optimized, thereby improving model performance.

[0230] Optionally, after obtaining the multilayer perceptron model, redundant neurons in the multilayer perceptron model can be evaluated and pruned through feedforward pruning to reduce the parameter size and the number of computing nodes, thereby improving the prediction efficiency of the multilayer perceptron model. Specifically, the method may include the following steps:

[0231] Step S280: Obtain the weight vector and L1 norm of each neuron in the multilayer perceptron model.

[0232] The L1 norm is the sum of the absolute values ​​of all elements in the weight vector.

[0233] In the financial credit business scenario, after completing the training of the multi-layer perceptron model and obtaining the final multi-layer perceptron model, the model needs to be further optimized and analyzed. First, the weight vector and L1 norm of each neuron in the multi-layer perceptron model are obtained.

[0234] The weight vector is a vector of weights that connect neurons to neurons in the previous layer. Each neuron has a corresponding weight vector that reflects the strength of the connection between the neuron and the neurons in the previous layer.

[0235] The L1 norm is the sum of the absolute values ​​of all elements in the weight vector. The process of calculating the L1 norm is to take the absolute value of each element in the weight vector of each neuron and then add up these absolute values ​​to get the L1 norm of the weight vector of that neuron.

[0236] For example, suppose a neuron's weight vector contains multiple elements. Taking the absolute value of each element yields a set of absolute values. These absolute values ​​are then summed up, and the final result is the L1 norm of the neuron's weight vector. By obtaining the weight vector and L1 norm of each neuron, we can make a preliminary assessment of the neuron's importance in the model. The weight vector reflects the connection between the neuron and the neurons in the previous layer, while the L1 norm can, to a certain extent, reflect the parameter sparsity of the neuron. Neurons with higher parameter sparsity have relatively smaller L1 norms and may play a relatively weaker role in the model. Conversely, neurons with lower parameter sparsity have relatively larger L1 norms and may play a more important role in the model.

[0237] Step S281: Obtain the gradient of the target loss function for each neuron output through the back-propagation algorithm, and determine the output contribution of each neuron to the multi-layer perceptron model based on the gradient corresponding to each neuron.

[0238] After obtaining the weight vector and L1 norm of each neuron in the multilayer perceptron model, the next step is to determine the contribution of each neuron to the model output. This can be achieved through the backpropagation algorithm.

[0239] The backpropagation algorithm is an important method for calculating gradients. In a multilayer perceptron model, training samples are first fed into the model for forward propagation, resulting in the model's output. The target loss function is then calculated based on the model's output and the sample labels.

[0240] Next, we use the backpropagation algorithm, starting from the model's output layer, to calculate the gradient of the target loss function with respect to each neuron's output layer by layer. The gradient represents the rate of change of the target loss function with respect to the neuron's output. For each neuron, its corresponding gradient reflects the degree to which a change in that neuron's output affects the target loss function value.

[0241] For example, during backpropagation, starting with the neurons in the output layer, the gradient of each output layer neuron is calculated based on the rate of change of the target loss function with respect to the output of the output layer neurons. This gradient information is then passed to the neurons in the previous layer, and the gradient of the previous layer neurons is calculated based on the connection weights between the previous layer neurons and the output layer neurons. This calculation continues layer by layer until the gradient of the input layer neurons is calculated.

[0242] Based on the gradient corresponding to each neuron, its contribution to the output of the multilayer perceptron model can be determined. Generally speaking, the larger the absolute value of the gradient, the greater the impact of the neuron's output change on the target loss function value, and the higher the neuron's contribution to the model output; conversely, the smaller the absolute value of the gradient, the lower the neuron's contribution to the model output. In this way, the importance of each neuron in the model can be quantified, providing a basis for subsequent model optimization.

[0243] Step S282: Determine an output contribution pruning threshold based on the output contribution corresponding to each neuron in the multi-layer perceptron model.

[0244] After determining the output contribution of each neuron in the multilayer perceptron model, we need to determine the output contribution pruning threshold based on these output contributions. The output contribution pruning threshold is used to determine which neurons can be considered to have a low contribution to the model output and can be pruned during the model optimization process.

[0245] There are several ways to determine the output contribution pruning threshold. One common approach is to determine it based on the distribution of all neuron output contributions. First, the output contributions of all neurons are sorted, either from largest to smallest or from smallest to largest.

[0246] Then, based on the sorted output contribution sequence, a threshold is determined using a specific strategy. For example, the value at a specific position in the output contribution sequence can be selected as the threshold. If the output contribution of a certain proportion of neurons is selected as a reference, the output contribution value corresponding to that proportion can be used as the threshold. In other words, neurons with output contributions below the threshold are considered to have a low contribution to the model output and can be considered for pruning.

[0247] Another approach is to determine the threshold based on experience or experimental results. Experiment with different thresholds and observe the performance changes on the validation set. Select a threshold that maximizes the number of neurons with low contributions while maintaining stable model performance as the output contribution pruning threshold.

[0248] Determining the appropriate output contribution pruning threshold is a critical step, as it requires a balance between model complexity and performance. If the threshold is set too high, too many neurons that contribute little to the model output may be retained, resulting in an overly complex model. If the threshold is set too low, some neurons that contribute significantly to the model output may be pruned, affecting model performance.

[0249] Step S283: Determine the parameter sparsity pruning threshold according to the L1 norm of each neuron in the multilayer perceptron model.

[0250] After determining the output contribution pruning threshold, we also need to determine the parameter sparsity pruning threshold based on the L1 norm of each neuron. The parameter sparsity pruning threshold is used to determine which neurons have high parameter sparsity and can be pruned during the model optimization process.

[0251] Similar to determining the output contribution pruning threshold, there are several methods for determining the parameter sparsity pruning threshold. One method is to determine it based on the distribution of the L1 norm of all neurons. First, sort the L1 norms of all neurons, preferably from smallest to largest.

[0252] Then, based on the sorted L1 norm sequence, a threshold is determined using a specific strategy. For example, the value at a specific position in the L1 norm sequence can be selected as the threshold. If the L1 norm of a certain proportion of neurons is selected as a reference, the L1 norm value corresponding to this proportion can be used as the threshold. In other words, neurons with an L1 norm below the threshold are considered to have high parameter sparsity and can be considered for pruning.

[0253] Another approach is to determine the threshold based on experience or experimental results. Experiment with different thresholds and observe the performance changes on the validation set. Select a threshold that prunes as many neurons with high parameter sparsity as possible while maintaining basic model performance stability.

[0254] Determining the appropriate parameter sparsity pruning threshold also requires a balance between model complexity and performance. If the threshold is set too high, too many neurons with high parameter sparsity may be retained, resulting in a large number of redundant parameters in the model. If the threshold is set too low, some neurons with high parameter sparsity but still contributing to the model output may be pruned, affecting model performance.

[0255] Step S284: Delete redundant neurons in the multilayer perceptron model according to the output contribution pruning threshold and the parameter sparsity pruning threshold, and update the model structure of the multilayer perceptron model.

[0256] After determining the output contribution pruning threshold and parameter sparsity pruning threshold, the redundant neurons in the multilayer perceptron model can be deleted according to these two thresholds, and the model structure can be updated.

[0257] First, we can traverse each neuron in the model and compare the output contribution of each neuron with the output contribution pruning threshold, and compare the L1 norm of each neuron with the parameter sparsity pruning threshold. If the output contribution of a neuron is less than the output contribution pruning threshold and its L1 norm is less than the parameter sparsity pruning threshold, then the neuron is considered redundant.

[0258] For neurons that are determined to be redundant, they are deleted from the model. The operation of deleting redundant neurons includes removing the connection between the neuron and the neurons in the previous layer and the next layer, and deleting the weight parameters corresponding to the neuron.

[0259] After removing redundant neurons, the multilayer perceptron model's structure needs to be updated. This involves adjusting the number and connectivity of neurons in each layer. For example, if some neurons in a layer are removed, the connection matrix between that layer and the next layer needs to be adjusted accordingly.

[0260] After updating the model structure, you can also perform fine-tuning training on the model. Fine-tuning training involves performing a small number of iterative training iterations on the model using the training data based on the new model structure to further optimize the model parameters and improve its performance. By removing redundant neurons and updating the model structure, you can reduce the model's complexity, improve its computational efficiency, and, to a certain extent, avoid overfitting, making the model more concise and efficient.

[0261] Optionally, after obtaining the multi-layer perceptron model, the multi-layer perceptron model can be fine-tuned using the prediction results of the multi-layer perceptron model in the validation set. Specifically, the fine-tuning training process can include the following steps:

[0262] Step S290: Obtain the prediction results of the multi-layer perceptron model in the validation set, where the prediction results include a validation result label set and a validation result value set.

[0263] In financial credit scenarios, after obtaining a trained multi-layer perceptron model, its performance needs to be evaluated. To do this, the model's prediction results are obtained from a validation set. The validation set is a set of data different from the training set and is used to test the model's generalization ability on unseen data.

[0264] Each sample in the validation set is input into the multilayer perceptron model, which performs forward propagation and outputs a prediction result. The prediction result includes a validation result label set and a validation result value set. The validation result label set is the classification label obtained by the model for each sample. For example, in the credit business, this might be "high-quality customer," "average customer," or "risky customer." The validation result value set is the continuous value obtained by the model through regression prediction for each sample, such as credit score or loan amount.

[0265] For example, for a sample in the validation set, the model will output a classification label and a continuous value based on its input feature vector, through internal neuron calculations and the application of connection weights. This process is repeated for all samples in the validation set, resulting in a validation result label set and a validation result value set. By analyzing these predictions, we can understand the model's performance on the validation set and provide a basis for subsequent model adjustments.

[0266] Step S291: Calculate the model prediction confidence index based on the distribution consistency of the verification result label set and the verification result value set.

[0267] After obtaining the validation result labels and numerical values ​​for the multilayer perceptron model in the validation set, we need to calculate the model prediction confidence index based on the distribution consistency of these two sets. The model prediction confidence index is used to measure the reliability and accuracy of the model's prediction results.

[0268] First, divide the verification result value set into multiple continuous intervals, each of which corresponds to a verification result label. The method for dividing the continuous intervals can be determined based on specific business needs and data characteristics. For example, in a credit business, the verification result value set can be divided into different intervals based on the credit score range, and each interval corresponds to a different credit rating label, such as "high credit rating," "medium credit rating," "low credit rating," etc.

[0269] Then, count the number of matches between the actual verification result label and the corresponding verification result label within each continuous interval. For each continuous interval, traverse all samples within that interval and compare their actual verification result labels with the verification result label corresponding to that interval to see if they are consistent. If they are consistent, the prediction result for that sample is considered a match; if not, it is considered a mismatch. Count the number of matching samples.

[0270] Finally, the model prediction confidence index is generated based on the ratio of the number of matches to the total number of samples in the validation result set. The number of matches within each consecutive interval is summed to obtain the total number of matches. The total number of matches is then divided by the total number of samples in the validation result set. The resulting ratio is the model prediction confidence index. Higher values ​​indicate greater reliability and accuracy of the model's predictions; conversely, lower values ​​indicate lower reliability and accuracy of the model's predictions.

[0271] Step S292: When the model prediction confidence index is lower than a preset threshold, the dimension of the fully connected layer of the multi-layer perceptron model is readjusted.

[0272] After calculating the model prediction confidence index, it is compared with a preset threshold. The preset threshold is a standard set in advance based on specific business needs and model performance requirements.

[0273] When the model prediction confidence index falls below the preset threshold, it indicates that the model's prediction performance on the validation set is not ideal and the model needs to be adjusted. In this case, the fully connected layer dimensions of the multilayer perceptron model are resized.

[0274] The first step in rescaling the fully connected layer dimensions is to determine the dimensionality scaling factor based on the difference between the model's prediction confidence metric and a preset threshold. This dimensionality scaling factor adjusts the input and output feature dimensions of the fully connected layer. A large difference indicates that the model's performance is significantly below expectations, and a larger dimensionality adjustment may be necessary. A small difference indicates that the model's performance is relatively below expectations, and a smaller dimensionality adjustment can be appropriate.

[0275] For example, if the model prediction confidence index is far below the preset threshold, the dimension scaling factor may be set to a smaller value to reduce the dimension of the fully connected layer; if the model prediction confidence index is slightly below the preset threshold, the dimension scaling factor may be set to a value close to 1 to make a smaller adjustment.

[0276] The input feature dimension and output feature dimension of the fully connected layer are scaled proportionally based on the dimension scaling factor to generate the adjusted fully connected layer structure. For example, if the dimension scaling factor is 0.8, then the input feature dimension and output feature dimension of the fully connected layer will be multiplied by 0.8 to obtain the adjusted dimension.

[0277] After adjusting the fully connected layer structure, all parameters in the multilayer perceptron model, except for the fully connected layer, are frozen. This is because these parameters have already been well optimized during previous training. To avoid excessive impact on other parameters when adjusting the fully connected layer, they are frozen. Then, fine-tuning training is performed on the adjusted fully connected layer based on the validation set. During fine-tuning, the parameters of the adjusted fully connected layer are optimized using the validation set data to improve the model's predictive performance on the validation set. Through this adjustment and training, the model can better adapt to the data and improve the model's prediction confidence index, ensuring that it reaches or exceeds the preset threshold.

[0278] Optionally, after obtaining the multi-layer perceptron model, the model volume of the multi-layer perceptron model can be compressed by knowledge distillation, thereby improving the inference speed of the multi-layer perceptron model.

[0279] Specifically, for example, a student model can be constructed first. The structural design of the student model requires comprehensive consideration of multiple factors. In terms of the number of network layers, the student model can be appropriately reduced compared to the multilayer perceptron model (teacher model). For example, the teacher model (i.e., the multilayer perceptron model obtained above) may have multiple hidden layers, while the student model can retain only a few hidden layers. This reduces the number of model parameters and, therefore, the model size.

[0280] Regarding the number of neurons, the number of neurons in each layer can be reduced accordingly. For example, if a hidden layer in the teacher model has a large number of neurons, the number of neurons in the corresponding layer in the student model can be reduced by a certain ratio. At the same time, the dimensions of the student model's input and output layers must be consistent with those of the teacher model to ensure they can process the same type of input data and output predictions in the same format.

[0281] For example, in the financial credit business, the input data is the feature vector extracted from multimodal data, and the output is the verification result label and verification result value. Then the input layer dimension of the student model must be able to receive these feature vectors, and the output layer dimension must be able to correctly output the label and value.

[0282] Next, we define the distillation loss function. The distillation loss function measures the difference between the student model and the teacher model, as well as the difference between the student model's predictions and the true labels. It consists of two parts: the distillation loss and a traditional loss function (such as the cross-entropy loss for classification tasks and the mean squared error loss for regression tasks).

[0283] Distillation loss primarily measures the similarity between the output of the student model and the output of the teacher model. For classification tasks, the teacher model outputs a probability distribution corresponding to each category, and the student model also outputs a probability distribution. The difference between these two probability distributions can be measured using the Kullback-Leibler (KL) divergence. The KL divergence reflects the degree of deviation between the probability distribution of the student model and the probability distribution of the teacher model. The smaller the deviation, the closer the student model is to the teacher model's prediction.

[0284] For regression tasks, the teacher model outputs a continuous value, and the student model also outputs a continuous value. The difference between these two values ​​can be measured using the mean squared error (MSE). The smaller the MSE, the closer the student model's regression predictions are to those of the teacher model.

[0285] Traditional loss functions measure the difference between the student model's predictions and the true labels. In classification tasks, the cross-entropy loss function is used, which measures the difference between the probability distribution of the student model's output and the probability distribution of the true labels. In regression tasks, the mean squared error loss function is used, which measures the difference between the values ​​output by the student model and the true values.

[0286] The final distillation loss function is a weighted sum of the distillation loss and the traditional loss function, each weighted according to a specific weight. The weighting needs to be adjusted based on the specific task and data characteristics. For example, if the student model is more focused on learning the knowledge of the teacher model, the weight of the distillation loss can be set higher; if the student model's accuracy in predicting the true label is more important, the weight of the traditional loss function can be set higher.

[0287] Next, the student model undergoes knowledge distillation training. After constructing the student model and defining the distillation loss function, knowledge distillation training can begin. The training process is similar to traditional model training, but in each iteration, both the teacher model's output and the true label must be considered.

[0288] The training data is fed into the teacher model and the student model. The teacher model processes the input data based on its internal parameters and structure, outputting a classification probability distribution and regression value. The student model also processes the same input data and outputs its own classification probability distribution and regression value.

[0289] According to the definition of the distillation loss function, the values ​​of the distillation loss and the traditional loss function are calculated and weighted together to obtain the final loss value. For example, assuming the weight of the distillation loss is α, the weight of the traditional loss function is β, the distillation loss is L_distill, and the traditional loss is L_traditional, then the final loss value L = α*L_distill+β*L_traditional.

[0290] The backpropagation algorithm is used to calculate the gradient of the loss value with respect to the student model parameters. The backpropagation algorithm starts from the output layer and calculates the gradient of each parameter layer by layer. The gradient represents the rate of change of the loss value with respect to the parameter.

[0291] Based on the calculated gradient, an optimization algorithm (such as stochastic gradient descent) is used to update the student model's parameters. The optimization algorithm adjusts the parameter values ​​in the opposite direction of the gradient, gradually reducing the loss. Through continuous iterative training, the student model gradually learns the knowledge of the teacher model and its ability to predict true labels continues to improve.

[0292] After completing knowledge distillation training, the student model's performance needs to be evaluated. This can be done using a validation set or a test set. The validation or test set data is fed into the student model, which then outputs predictions, including a validation result label set and a validation result value set. A series of evaluation metrics are then calculated based on these predictions and the true labels.

[0293] For classification tasks, you can calculate metrics such as precision, recall, and F1 value. Precision represents the ratio of correctly predicted samples to the total number of samples; recall represents the ratio of samples that are actually of a certain category and correctly predicted to be of that category to the total number of samples of that category; and F1 value is the harmonic mean of precision and recall, which takes both into account.

[0294] For regression tasks, you can calculate metrics such as mean square error and mean absolute error. The mean square error represents the average of the squares of the differences between the predicted values ​​and the true values; the mean absolute error represents the average of the absolute values ​​of the differences between the predicted values ​​and the true values.

[0295] At the same time, it is also necessary to compare the performance of the student model and the teacher model on the same dataset. If the performance of the student model is similar to that of the teacher model, and the model size is significantly reduced and the inference speed is significantly improved, then it means that knowledge distillation has achieved good results.

[0296] If the performance evaluation results of the student model meet the requirements, the compressed student model can be used instead of the teacher model for inference. This can compress the model size of the multi-layer perceptron model and thus improve the inference speed of the multi-layer perceptron model.

[0297] Figure 5 Schematic diagram of a business verification system 100 based on multimodal data fusion provided in an embodiment of the present application. Figure 5 As shown, the processor 120 can be used in the business verification system 100 based on multimodal data fusion and is used to perform the functions of the present invention.

[0298] The service verification system 100 based on multimodal data fusion can be a general-purpose server or a special-purpose server, both of which can be used to implement the service verification method based on multimodal data fusion of the present invention. Although only one server is shown in the present invention, for convenience, the functions described in the present invention can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0299] For example, the business verification system 100 based on multimodal data fusion may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and storage media 140 in different forms, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the business verification system 100 based on multimodal data fusion may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present invention can be implemented according to these program instructions. The business verification system 100 based on multimodal data fusion also includes an input / output (I / O) interface 150 between the computer and other input / output devices.

[0300] For ease of explanation, only one processor is described in the business verification system 100 based on multimodal data fusion. However, it should be noted that the business verification system 100 based on multimodal data fusion in the present invention may also include multiple processors, so the steps performed by one processor described in the present invention may also be performed jointly or individually by multiple processors. For example, if the processor of the business verification system 100 based on multimodal data fusion executes step A and step B, it should be understood that step A and step B may also be executed jointly by two different processors or individually in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor execute steps A and B together.

[0301] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the business verification method based on multimodal data fusion of the aforementioned embodiment.

[0302] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the steps in the business verification method based on multimodal data fusion of the aforementioned embodiment.

[0303] The device embodiments described above are merely illustrative. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network modules. Some or all of these modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0304] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus the necessary general hardware platform, or of course, by means of hardware. Based on this understanding, the above technical solution, in essence, or the portion that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of storing or storing data.

[0305] Finally, it should be noted that what is disclosed above is only a preferred embodiment of the present invention, which is only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A service verification method based on multimodal data fusion, characterized in that: include: Acquire multimodal data of a target object to be verified, the multimodal data including structured data, text data, image data, and time series data, wherein the structured data and text data are derived from a verification request of the target object, the image data is derived from a face-to-face image of the target object, and the time series data is derived from historical transaction records of the target object; Extracting, according to the structured data, the text data, the image data, and the time series data, a structural feature vector, a semantic feature vector, an image feature vector, and a temporal behavior feature vector corresponding to the multimodal data; Performing classification prediction and regression prediction based on the structured feature vector, the semantic feature vector, the image feature vector, and the temporal behavior feature vector to obtain a verification result label and a verification result value of the target object; Output the verification result label and the verification result value of the target object.

2. The method according to claim 1, characterized in that The performing classification prediction and regression prediction based on the structured feature vector, the semantic feature vector, the image feature vector, and the temporal behavior feature vector to obtain a verification result label and a verification result value of the target object includes: splicing and fusing the structural feature vector, the semantic feature vector, the image feature vector, and the temporal behavior feature vector to obtain a verification factor feature vector of the target object; Performing classification prediction on the verification factor feature vector to obtain the verification result label of the target object; Perform regression prediction on the verification factor feature vector to obtain the verification result value of the target object.

3. The method according to claim 2, characterized in that The splicing and fusing of the structural feature vector, the semantic feature vector, the image feature vector, and the temporal behavior feature vector to obtain the verification factor feature vector of the target object includes: splicing the semantic feature vector, the image feature vector, and the temporal behavior feature vector to obtain a first feature vector to be processed; Using the first feature vector to be processed, the first weight matrix obtained by pre-training, and the first bias vector obtained by pre-training as input parameters of a rectified linear unit (ReLU) activation function, and generating a second feature vector to be processed through the ReLU activation function; The structured feature vector and the second feature vector to be processed are concatenated to obtain the verification factor feature vector of the target object.

4. The method according to claim 1, wherein The extracting, based on the structured data, the text data, the image data, and the time series data, the structural feature vector, the semantic feature vector, the image feature vector, and the temporal behavior feature vector corresponding to the multimodal data respectively includes: Inputting the structured data into a pre-trained gradient boosting decision tree model to extract the structured feature vector; Inputting the text data into a deep learning model based on an attention mechanism to extract the semantic feature vector; Inputting the image data into a convolutional neural network model to extract the image feature vector; The time series data is input into a bidirectional long short-term memory network model to extract the time series behavior feature vector.

5. The method according to any one of claims 1 to 4, characterized in that Before performing classification prediction and regression prediction based on the structured feature vector, the semantic feature vector, the image feature vector, and the temporal behavior feature vector to obtain the verification result label and the verification result value of the target object, the method further includes a training method for a multi-layer perceptron model, wherein the multi-layer perceptron model is used to obtain the verification result label and the verification result value of the target object based on the structured feature vector, the semantic feature vector, the image feature vector, and the temporal behavior feature vector, and the training method specifically includes: Constructing an initial multilayer perceptron model and obtaining a training sample set for training the initial multilayer perceptron model, wherein the training sample set includes an input sample vector and sample annotations, wherein the input sample vector includes a sample structured feature vector, a sample semantic feature vector, a sample image feature vector, and a sample temporal behavior feature vector, and the sample annotations include a verification result annotation and a verification result numerical annotation, and the initial multilayer perceptron model uses a two-layer fully connected network as a prediction head to jointly perform regression prediction and classification prediction; Inputting the training sample set into the initial multi-layer perceptron model for training, and determining the mean square error loss function of the regression prediction based on the number of training samples, the prediction verification result value of each training sample predicted and output by the initial multi-layer perceptron model, and the verification result value label of each training sample during the training process; Determine, during the training process, a cross entropy loss function for the classification prediction based on the number of training samples, the probability that each training sample output by the initial multilayer perceptron model prediction is the first verification result label, and the verification result label of each training sample; Using the preset first weight and second weight as weighted weights of the mean square error loss function and the cross entropy loss function, respectively, performing weighted fusion on the mean square error loss function and the cross entropy loss function to obtain the target loss function of the initial multi-layer perceptron model; When the value of the target loss function is less than or equal to a preset loss function threshold, the training of the initial multi-layer perceptron model is stopped to obtain the multi-layer perceptron model.

6. The method according to claim 5, characterized in that During the training process of the multi-layer perceptron model, the method further includes: In target iterative training of the initial multilayer perceptron model, randomly discarding target neurons corresponding to the discard probability in each fully connected layer of the initial multilayer perceptron model according to the discard probability; Continuing the target iterative training based on the adjusted neuron state; The step of randomly discarding target neurons corresponding to the discard probability in each fully connected layer of the initial multilayer perceptron model according to the discard probability specifically includes: For each neuron set in the fully connected layer, generate a random number corresponding to each neuron in the neuron set, wherein the random number obeys a uniform distribution from 0 to 1; Marking neurons whose random numbers are less than or equal to the discard probability as neurons to be discarded; Resetting the output weight of the to-be-discarded neuron to zero and freezing the activation value transfer of the to-be-discarded neuron in the forward propagation; During the back propagation process, the gradient update calculation of the to-be-discarded neurons is skipped.

7. The method according to claim 5, characterized in that During the training process of the multi-layer perceptron model, the method further includes: Acquire a target sample subset from the training sample set; Based on each feature dimension of the target sample subset, obtaining the mean and variance of each feature dimension in the target sample subset; performing data normalization processing on each sample in the target sample subset according to the mean and the variance, wherein the mean of the samples in the target sample subset after normalization is 0 and the variance is 1; According to the learnable scaling parameter and translation parameter, the normalized sample is linearly transformed to obtain a linearly transformed sample; The linearly transformed samples are input into the initial multi-layer perceptron model to calculate the target loss function, and back propagation is performed according to the target loss function to update the parameters of the initial multi-layer perceptron model.

8. The method according to claim 5, characterized in that After obtaining the multi-layer perceptron model, the method further includes: Obtain a weight vector and an L1 norm of each neuron in the multilayer perceptron model, where the L1 norm is the sum of the absolute values ​​of all elements in the weight vector; Obtaining the gradient of the target loss function with respect to the output of each neuron through a back-propagation algorithm, and determining the contribution of each neuron to the output of the multi-layer perceptron model based on the gradient corresponding to each neuron; Determining an output contribution pruning threshold according to the output contribution corresponding to each neuron in the multilayer perceptron model; Determining a parameter sparsity pruning threshold according to the L1 norm of each neuron in the multilayer perceptron model; According to the output contribution pruning threshold and the parameter sparsity pruning threshold, redundant neurons in the multilayer perceptron model are deleted, and the model structure of the multilayer perceptron model is updated.

9. The method according to claim 5, characterized in that After obtaining the multi-layer perceptron model, the method further includes: Obtaining prediction results of the multilayer perceptron model in a validation set, wherein the prediction results include a validation result label set and a validation result value set; Calculating a model prediction confidence index based on the distribution consistency of the verification result label set and the verification result value set; When the model prediction confidence index is lower than a preset threshold, readjusting the fully connected layer dimension of the multi-layer perceptron model; The calculation of the model prediction confidence index based on the distribution consistency of the verification result label set and the verification result value set specifically includes: Dividing the verification result value set into a plurality of continuous intervals, each continuous interval corresponding to a verification result label; Count the number of matches between the actual verification result label and the corresponding interval verification result label in each continuous interval; Generating the model prediction confidence index according to the ratio of the number of matches to the total number of samples in the verification result value set; The step of re-adjusting the fully connected layer dimension of the multi-layer perceptron model specifically includes: Determining a dimensionality scaling factor of a fully connected layer based on a difference between the model prediction confidence index and the preset threshold; Proportionally scaling the input feature dimension and the output feature dimension of the fully connected layer based on the dimension scaling factor to generate an adjusted fully connected layer structure; Freeze all parameters except the fully connected layer in the multilayer perceptron model, and perform fine-tuning training on the adjusted fully connected layer based on the validation set.

10. A business verification system based on multimodal data fusion, characterized in that: It includes a processor and a computer-readable storage medium, wherein the computer-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a computer, the business verification method based on multimodal data fusion according to any one of claims 1 to 9 is implemented.

Images