User willingness evaluation method and device for financial business handling

Abstract

The invention relates to the technical field of information security, and discloses a user willingness evaluation method and device for financial business handling, and the method comprises the steps: inputting to-be-detected audio and video data into a multi-modal fusion model, and obtaining a multi-modal fusion feature; calculating a negative emotion proportion and an emotion fluctuation degree based on the multi-modal fusion features; extracting face change data based on the to-be-detected audio and video data, and performing face behavior anomaly detection based on the face change data to obtain a face behavior anomaly score; extracting voice change data based on the to-be-detected audio and video data, and performing voice behavior anomaly detection based on the voice change data to obtain a voice behavior anomaly score; and determining a user willingness evaluation result based on the negative emotion proportion, the emotion fluctuation degree, the facial behavior anomaly score and the voice behavior anomaly score. According to the method and the system, dynamic capture of emotion change and behavior abnormity of the user in the business handling process is realized, and potential fraud risks can be found more accurately.

Description

Technical Field

[0001] The present invention relates to the field of information security technology, and in particular to a method and device for evaluating user willingness for handling financial business. Background Art

[0002] In the handling of financial services, the true intention of the customer is very important. The customer willingness can reflect whether the customer is truly willing to handle the business. The higher the true willingness, the more likely the customer is to handle the business out of their own needs. The lower the willingness, the more likely the customer is to be deceived or induced by someone, or to handle the financial business out of deliberate fraud. Therefore, the true willingness of the customer can be used to assist the financial industry in fraud risk detection.

[0003] When judging whether a customer is a fraud risk, we rely on structured data such as records provided by the customer or related to the customer, and lack analysis of multimodal data such as customer video, voice, and images. This results in the inability to detect customers in real time and the existence of lags. In addition, customers' performance in fraud situations usually includes non-verbal clues such as micro-expressions and abnormal intonation, and it is difficult to discover these details by relying solely on structured data analysis. Summary of the invention

[0004] In view of this, the present invention provides a method and device for evaluating user willingness for financial business processing, so as to solve the problems of lack of analysis of multimodal data in determining whether there is a risk of fraud, and difficulty in dynamically capturing user's emotional changes and abnormal behaviors.

[0005] In a first aspect, the present invention provides a method for evaluating user willingness for financial business handling, the method comprising: Acquire the audio and video data to be detected, input the audio and video data to be detected into the multimodal fusion model, and obtain the multimodal fusion features; Calculate the proportion of negative emotions and the degree of emotional fluctuation based on multimodal fusion features; Extracting facial change data based on the audio and video data to be detected, and performing facial behavior abnormality detection based on the facial change data to obtain a facial behavior abnormality score; Extracting voice change data based on the audio and video data to be detected, and performing voice behavior anomaly detection based on the voice change data to obtain a voice behavior anomaly score; The user willingness evaluation result is determined based on the negative emotion ratio, emotion fluctuation degree, facial behavior abnormality score and voice behavior abnormality score.

[0006] The present embodiment provides a user willingness evaluation method for financial business handling, which inputs the audio and video data to be detected into a multimodal fusion model to obtain multimodal fusion features, and calculates the negative emotion ratio and the degree of emotion fluctuation based on the multimodal fusion features; extracts facial change data based on the audio and video data to be detected, and performs facial behavior abnormality detection based on the facial change data to obtain a facial behavior abnormality score; extracts voice change data based on the audio and video data to be detected, and performs voice behavior abnormality detection based on the voice change data to obtain a voice behavior abnormality score; determines the user willingness based on the negative emotion ratio, the degree of emotion fluctuation, the facial behavior abnormality score and the voice behavior abnormality score. The system can detect changes in user emotions and abnormal facial and voice behaviors through modal fusion features, dynamically capture changes in user emotions and abnormal behaviors during business handling, and then build a real willingness evaluation calculation system based on the results of emotion and behavior analysis. It can more accurately detect potential fraud risks without the need for historical user relationship data. It supports real-time analysis of customer emotions and behaviors during transactions and calculates willingness. When risks are discovered, it can terminate business handling in advance to protect the rights and interests of both parties, avoid delays in risk detection, and effectively reduce losses caused by financial fraud.

[0007] In an optional implementation, the multimodal fusion model includes a speech branch, a video space branch, and a temporal branch; the audio and video data to be detected are input into the multimodal fusion model to obtain multimodal fusion features, including: The speech branch is used to extract speech features from the audio signal in the audio and video data to be detected, and the speech branch coding features are obtained; The video feature extraction is performed on the video frame information in the audio and video data to be detected by using the video space branch to obtain the video space branch coding feature; The timing branch is used to extract the timing features of the video frame information to obtain the video timing coding features; The speech branch coding features, video space branch coding features and video temporal coding features are fused to obtain multimodal fusion features.

[0008] The present embodiment provides a method for evaluating user willingness for financial business processing, which utilizes voice branches, video space branches, and timing branches to achieve accurate extraction of multimodal features in audio and video data to be detected, and performs feature fusion of multimodal features, namely, voice branch coding features, video space branch coding features, and video timing coding features, thereby laying a foundation for the subsequent dynamic capture of the user's emotional changes and behavioral abnormalities during the business processing process.

[0009] In an optional implementation, the speech branch coding features, the video space branch coding features and the video time sequence coding features are fused to obtain a multimodal fusion feature, including: The speech branch coding features, video space branch coding features and video temporal coding features are respectively input into the attention network to obtain the attention coefficient of each modality feature; The attention coefficient of each modal feature is used to perform weighted fusion on the speech branch coding features, video space branch coding features and video temporal coding features to obtain multimodal fusion features.

[0010] The present embodiment provides a method for evaluating user willingness for financial business processing, which utilizes an attention network to accurately calculate the attention coefficient of each modal feature, and then uses the attention coefficient of each modal feature to weightedly fuse the speech branch coding features, video space branch coding features, and video timing coding features, thereby achieving accurate calculation of the coding features containing spatial information, speech information, and timing information in the audio and video data to be detected. The obtained multimodal fusion features lay a foundation for the subsequent dynamic capture of the user's emotional changes and behavioral abnormalities during the business processing process.

[0011] In an optional implementation, the negative emotion ratio and the degree of emotion fluctuation are calculated based on the multimodal fusion features, including: Calculate the emotion probability sequence based on multimodal fusion features; The emotion classification results in the emotion probability sequence are counted, and the proportion of negative emotions is calculated based on the statistical results; The standard deviation of emotion probability is calculated based on the emotion probability sequence, and the degree of emotion fluctuation is calculated based on the standard deviation of emotion probability.

[0012] The present embodiment provides a method for evaluating user willingness for financial business processing, which calculates the emotion probability sequence by multimodal fusion features, thereby realizing accurate acquisition of customer emotion changes, and obtains the proportion of negative emotions by statistically analyzing the emotion classification results in the emotion probability sequence, and calculates the degree of emotion fluctuation based on the standard deviation of the emotion probability, thereby realizing quantification of the negative emotion proportion and the degree of emotion fluctuation.

[0013] In an optional implementation, facial change data is extracted based on the audio and video data to be detected, and facial behavior abnormality detection is performed based on the facial change data to obtain a facial behavior abnormality score, including: Extract the coordinates of the key points of the facial features, the tip of the nose, the chin and the eyes based on the audio and video data to be detected; Calculate the change in the coordinates of the key points of the facial features based on the coordinates of the key points of the facial features at adjacent moments; Calculate the relative displacement of the head based on the coordinates of the nose tip key point and the chin key point; Determine eye closure based on eye key point coordinates, and determine blink frequency based on the eye closure; Based on the coordinate changes of key points of facial features, relative displacement of the head and blinking frequency, a classification algorithm is used to calculate the facial behavior abnormality score.

[0014] The present embodiment provides a method for evaluating user willingness for handling financial business, which achieves accurate acquisition of user facial changes by extracting the coordinates of key points of facial features, the tip of the nose, the chin and the eyes, and then calculates the change in the coordinates of key points of facial features, the relative displacement of the head and the blinking frequency, and uses a classification algorithm to calculate the facial behavior abnormality score, thereby achieving the capture and scoring of abnormal behaviors such as facial feature changes, head movements, and blinking frequency.

[0015] In an optional implementation, voice change data is extracted based on the audio and video data to be detected, and voice behavior anomaly detection is performed based on the voice change data to obtain a voice behavior anomaly score, including: The audio signal in the audio and video data to be detected is segmented, and the basic speech features in each audio signal are extracted; the basic speech features include speech speed features, volume features and intonation features; Determine the magnitude of feature change based on basic speech features in each audio signal; Based on the basic features of speech and the amplitude of feature changes, the classification model is used to analyze abnormal behavior and obtain the speech behavior abnormality score.

[0016] The present embodiment provides a method for evaluating user willingness for handling financial business. By taking the basic speech features in each audio signal and determining the feature change amplitude based on the basic speech features in each audio signal, the method achieves accurate acquisition of the user's speech change situation. Based on the basic speech features and the feature change amplitude, a classification model is used to perform abnormal behavior analysis to obtain a speech behavior abnormality score, thereby achieving the capture and scoring of the user's abnormal speech behavior.

[0017] In a second aspect, the present invention provides a user willingness evaluation device for financial business handling, the device comprising: An acquisition module is used to acquire audio and video data to be detected, input the audio and video data to be detected into a multimodal fusion model, and obtain multimodal fusion features; A calculation module, used to calculate the proportion of negative emotions and the degree of emotional fluctuation based on multimodal fusion features; A first anomaly detection module is used to extract facial change data based on the audio and video data to be detected, and perform facial behavior anomaly detection based on the facial change data to obtain a facial behavior anomaly score; A second anomaly detection module is used to extract voice change data based on the audio and video data to be detected, and perform voice behavior anomaly detection based on the voice change data to obtain a voice behavior anomaly score; The determination module is used to determine the user willingness evaluation result based on the negative emotion ratio, the degree of emotion fluctuation, the facial behavior abnormality score and the voice behavior abnormality score.

[0018] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the user willingness assessment method for financial business processing according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the user willingness assessment method for handling financial business according to the first aspect or any corresponding embodiment thereof.

[0020] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the user willingness assessment method for handling financial business according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 is a flow chart of a method for evaluating user willingness for handling financial services according to an embodiment of the present invention; Figure 2 is a flow chart of another method for evaluating user willingness for handling financial services according to an embodiment of the present invention; Figure 3 is a schematic diagram of the structure of a multimodal fusion model according to an embodiment of the present invention; Figure 4 is a flow chart of another method for evaluating user willingness for handling financial services according to an embodiment of the present invention; Figure 5 is a schematic diagram of a process for analyzing the user willingness score for handling financial services according to an embodiment of the present invention; Figure 6 is a structural block diagram of a user willingness evaluation device for financial business handling according to an embodiment of the present invention; Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] It should be noted that in the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are all carried out with the user's consent, and are in compliance with relevant laws and regulations and do not violate public order and good morals.

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0025] A financial fraud detection method relates to the field of information security or artificial intelligence, and the method comprises: inputting text data of a transfer transaction to be detected into a pruned financial fraud detection model to obtain the fraud probability of the transfer transaction to be detected; the financial fraud detection model is configured to be pruned in advance by the following operations: inputting text data of historical transfer transactions into the M-layer data processing layer to be pruned; obtaining the initial text vector output by each neuron in the m-th layer data processing layer; processing the output initial text vector based on the scaling factor of each neuron to obtain the transfer transaction text vector of each neuron; updating the scaling factor according to the fraud probability of the historical transfer transaction and the loss between the fraud label. The method uses a text-based fraud detection model to analyze the text information of the historical transfer transaction to predict the fraud probability. There is a lag problem in the way of analyzing customer risks based on historical transaction records. If the transaction has already occurred, the fraud risk cannot be effectively prevented.

[0026] The financial fraud detection method based on graph neural network includes: obtaining user information and user relationship data; constructing a multi-relation fraud graph based on the obtained user information and user relationship data; and detecting the constructed multi-relation fraud graph through a trained graph neural network model, and classifying and judging each node in the multi-relation fraud graph to determine the fraudulent user. In this method, graph neural network is used to implement fraud detection. If the user behavior or relationship changes, the user relationship graph needs to be retrained or adjusted to adapt to the new data distribution. In addition, this method relies on historical data and lacks real-time and flexibility.

[0027] The embodiment of the present invention provides a method for evaluating user willingness for financial business processing, which is a new risk assessment method that combines psychological analysis with artificial intelligence technology. Compared with the static data screening method, the embodiment of the present invention provides a method for dynamically capturing the emotional changes and abnormal behaviors of customers during the business processing process through the fusion processing of multimodal data (such as voice, facial expressions, etc.), and establishing a willingness evaluation calculation system through emotional and behavioral results, which can more accurately discover potential fraud risks; in addition, the method also supports real-time analysis, helping banks to respond quickly at key nodes of customer interaction, effectively reducing losses caused by financial fraud.

[0028] In order to solve the above technical problems, the embodiment of the present invention provides a user willingness assessment method for financial business handling. It should be noted that the user willingness assessment for financial business handling provided by the embodiment of the present invention can be performed by a user willingness assessment device for financial business handling. The user willingness assessment device for financial business handling can be implemented as part or all of an electronic device through software, hardware, or a combination of software and hardware. The electronic device can be a server or a terminal. The server in the embodiment of the present application can be a single server or a server cluster composed of multiple servers. The terminal in the embodiment of the present application can be a smart phone, a personal computer, a tablet computer, a wearable device, an intelligent robot, and other intelligent hardware devices. In the following method embodiments, the execution subject is an electronic device as an example for explanation.

[0029] According to an embodiment of the present invention, an embodiment of a method for evaluating user willingness for handling financial business is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0030] In this embodiment, a method for evaluating user willingness for financial business handling is provided, which can be used in the above-mentioned electronic device. Figure 1 is a flow chart of a method for evaluating user willingness for financial business handling according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps: Step S101, obtaining audio and video data to be detected, inputting the audio and video data to be detected into a multimodal fusion model to obtain multimodal fusion features.

[0031] Specifically, the multimodal fusion model is mainly composed of the fusion of three branches: video space branch, speech branch, and timing branch; the face detection pre-trained model is used in the video space branch to extract features from a single frame, and then introduced into the subsequent 2D convolution block; the speech branch mainly extracts audio signal features and applies 1D convolution layer blocks, each block consists of convolution layer, batch normalization, ReLU activation and maximum pooling; the timing feature branch adopts feature extraction convolution model and timing model to realize timing feature extraction; the modal fusion method uses the dot product attention mechanism to fuse some features of the visual space branch, speech branch and timing branch.

[0032] Step S102, calculating the negative emotion ratio and the degree of emotion fluctuation based on the multimodal fusion features.

[0033] Specifically, by calculating the proportion of negative emotions and the degree of emotional fluctuation, the emotional changes can be quantified.

[0034] Step S103, extracting facial change data based on the audio and video data to be detected, and performing facial behavior abnormality detection based on the facial change data to obtain a facial behavior abnormality score.

[0035] Specifically, the facial behavior abnormality scoring is mainly based on the extraction of facial key points and the analysis of dynamic features, combined with the classification algorithm, to capture and score abnormal behaviors such as changes in facial features, head movements, and blinking frequency.

[0036] Step S104: extracting voice change data based on the audio and video data to be detected, and performing voice behavior anomaly detection based on the voice change data to obtain a voice behavior anomaly score.

[0037] Specifically, the speech behavior anomaly score focuses on the dynamic changes of characteristics such as intonation, volume, and speaking speed. Through frequency domain and time domain analysis, signal processing and classification methods, it determines whether the speech behavior is abnormal and scores it.

[0038] Step S105, determining the user willingness evaluation result based on the negative emotion ratio, the emotion fluctuation degree, the facial behavior abnormality score and the voice behavior abnormality score.

[0039] Specifically, a linear regression model is established through the proportion of negative emotions, the degree of emotional fluctuation, the facial behavior abnormality score, the voice behavior abnormality score and the user willingness score as the user willingness scoring system, and then the user willingness score, that is, the user willingness evaluation result, is calculated through the user willingness scoring system.

[0040] Furthermore, user willingness score The calculation formula is as follows: (1) in, is the intercept, , , , are the weights of the four feature variables, , , , There are four characteristic variables, namely, the proportion of negative emotions, the degree of emotional fluctuations, the score of facial behavior abnormality and the score of voice behavior abnormality.

[0041] Furthermore, the proportion of negative emotions, the degree of emotional fluctuation, the facial behavior abnormality score, the voice behavior abnormality score and the user willingness score are combined into samples. By annotating the sample data to form a data set, a multivariate linear regression model is trained for fitting, and the intercept with the best goodness of fit and the weight of the characteristic variables can be obtained, thereby constructing a user willingness scoring system.

[0042] The present embodiment provides a user willingness evaluation method for financial business handling, which inputs the audio and video data to be detected into a multimodal fusion model to obtain multimodal fusion features, and calculates the negative emotion ratio and the degree of emotion fluctuation based on the multimodal fusion features; extracts facial change data based on the audio and video data to be detected, and performs facial behavior abnormality detection based on the facial change data to obtain a facial behavior abnormality score; extracts voice change data based on the audio and video data to be detected, and performs voice behavior abnormality detection based on the voice change data to obtain a voice behavior abnormality score; determines the user willingness based on the negative emotion ratio, the degree of emotion fluctuation, the facial behavior abnormality score and the voice behavior abnormality score. The system can detect changes in user emotions and abnormal facial and voice behaviors through modal fusion features, dynamically capture changes in user emotions and abnormal behaviors during business handling, and then build a real willingness evaluation calculation system based on the results of emotion and behavior analysis. It can more accurately detect potential fraud risks without the need for historical user relationship data. It supports real-time analysis of customer emotions and behaviors during transactions and calculates willingness. When risks are discovered, it can terminate business handling in advance to protect the rights and interests of both parties, avoid delays in risk detection, and effectively reduce losses caused by financial fraud.

[0043] In this embodiment, a method for evaluating user willingness for financial business handling is provided, which can be used in the above-mentioned electronic device. Figure 2 is a flow chart of a method for evaluating user willingness for financial business handling according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps: Step S201, obtaining audio and video data to be detected, inputting the audio and video data to be detected into a multimodal fusion model to obtain multimodal fusion features.

[0044] Specifically, Figure 3 As shown, the above step S201 includes: Step S2011, using the speech branch to extract speech features from the audio signal in the audio and video data to be detected, to obtain speech branch coding features.

[0045] Specifically, in order to obtain the content and emotional information contained in the audio signal in the audio and video data to be detected, the speech signal is processed and transformed through the following steps to obtain the speech branch coding features: 1) Extract speech signals: Extract speech signals using speech signal extraction tools .

[0046] 2) Obtain Mel-spectrogram features: Convert the speech signal into Mel-spectrogram features (i.e., MFCC spectrum features) to provide rich frequency domain features for subsequent convolution operations.

[0047] 3) Obtain speech feature representation: Apply 1D convolution layer blocks to perform deep feature extraction on the Mel spectrum features, and process the input audio features through 1D convolution operations, that is, the input is the Mel spectrum features of the speech signal , get the audio feature representation , specifically expressed as: (2) in, is the convolution kernel, is bias.

[0048] 4) Obtaining important speech feature representation: Normalizing the speech feature representation to obtain , and then perform RELU (Rectified Linear Unit, linear rectification function) activation and maximum pooling operation on the normalized features, where k is the pooling window size, and the important features of speech are represented , the specific calculation formula is: (3) (4) 5) Obtain N important speech feature representations: Use N 1D convolutional layers to perform the above 3) and 4) to obtain N important speech feature representations. , , ..., .

[0049] 6) Obtain speech branch coding features: Concatenate N important speech feature representations to obtain speech branch coding features .

[0050] (5) Step S2012: extract video features from video frame information in the audio and video data to be detected using video space branches to obtain video space branch coding features.

[0051] Specifically, in order to obtain the image content features and attribute information in the input video frame, the video frame is processed and transformed to obtain the video space branch coding features, and the steps include: 1) Extract video frames: Use the video frame extraction tool to read each frame of the video.

[0052] 2) Extract facial feature vectors: Perform color space conversion on each video frame and resize it to a format suitable for the input of the face recognition pre-trained model. Then, process the input video frame through a convolutional layer to obtain the facial feature vectors. , facial feature vector It can be expressed as: (6) in, Represents a video frame, is the extracted facial feature vector with dimension d.

[0053] 3) Facial deep feature encoding: The facial feature vector is input into the fully connected layer to obtain a high-dimensional vector representation, which is the facial deep feature encoding .

[0054] 4) Extracting complex facial features: Introducing facial deep feature encoding into the subsequent CNN (Convolutional Neural Networks) layer to learn and extract complex features such as facial contours, expressions, skin color, etc. , complex facial features It can be expressed as: (7) in, Encode the facial depth feature of length 2d and transform it into a size of Complex facial features, and are width and height respectively, is the number of channels.

[0055] 5) Extract global feature representation of video frames: Design a 2D convolution block, which consists of a set of convolution layers, activation functions (such as ReLU), pooling layers, etc. The purpose is to enhance the nonlinear ability of complex facial features, extract global information in video frames, and obtain global feature representation of video frames. : (8) 6) Obtaining video spatial branch coding features: Concatenate the global feature representation of the video frame to obtain a comprehensive spatial feature vector as the video spatial branch coding feature. .

[0056] (9) Step S2013, using the timing branch to extract the timing features of the video frame information to obtain the video timing coding features.

[0057] Specifically, in order to obtain dynamic change features such as the direction of the face motion trajectory in the video, the following steps are performed to process and transform the video to obtain the temporal coding features: 1) Obtaining spatial feature representation of video frames: Calculate video frame features through a convolutional neural network model (i.e., convolutional model) to obtain spatial feature representation of video frames : (10) in, is a feature of size [C,H,W], and the video frame is , Represents the time index of the video frame, C is the number of channels, and H and W are the image height and width.

[0058] 2) Obtaining the global features of the video frame space: Perform a global average pooling operation on the video frame space feature representation to compress the video frame space feature into a one-dimensional vector to obtain the global features of the video frame space : (11) in, is a feature vector of length C.

[0059] 3) Obtain the initial temporal feature sequence of the video frame: Calculate the spatial global features of each video frame , forming a video frame initial temporal feature sequence : (12) in, is the number of video frames, For a size of The matrix represents the spatial features of all video frames.

[0060] 4) Calculate and obtain the temporal dependency feature representation of the video frame: Process the initial temporal feature sequence of the video frame through the temporal neural network to obtain the temporal dependency feature representation of the video frame : (13) (14) (15) (16) (17) in, , , Represent the activation values ​​of the input gate, forget gate and output gate respectively, Represents a memory unit, Indicates the hidden state, W and b Represents the learned weight matrix and bias.

[0061] 5) Obtain video temporal coding features: The video frame temporal dependency feature representation sequence is , H is calculated through maximum pooling to obtain the video temporal coding feature , the calculation formula is as follows: (18) (19) in, It is a feature vector representing the entire video, including timing information.

[0062] Step S2014, the speech branch coding features, the video space branch coding features and the video temporal coding features are fused to obtain multimodal fusion features.

[0063] In some optional implementations, the above step S2014 includes: In step a1, the speech branch coding features, video space branch coding features and video temporal coding features are respectively input into the attention network to obtain the attention coefficient of each modal feature.

[0064] Specifically, the attention weight of each modality is calculated through the attention network, and the input of the attention network is , and , output the attention coefficient of each modal feature, where each modal feature is processed using the FC full connection mapping, and the calculation formula of the attention coefficient is as follows: (20) (twenty one) (twenty two) in, represents the attention coefficient of the temporal encoding feature of the video, represents the attention coefficient of the video spatial branch encoding feature, Indicates the attention coefficient of the speech branch encoding features.

[0065] Step a2, using the attention coefficient of each modal feature to perform weighted fusion on the speech branch coding features, video space branch coding features and video temporal coding features to obtain multimodal fusion features.

[0066] Specifically, the attention coefficients of each modal feature are used to weight each modal feature and then add and fuse them to obtain a multimodal fusion feature. The calculation formula of the multimodal fusion feature is as follows: (twenty three) in, Encode the multimodal features after modal fusion.

[0067] Step S202, calculate the negative emotion ratio and the degree of emotion fluctuation based on the multimodal fusion features. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0068] Step S203: extract facial change data based on the audio and video data to be detected, and perform facial behavior abnormality detection based on the facial change data to obtain a facial behavior abnormality score. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0069] Step S204: extract voice change data based on the audio and video data to be detected, and perform voice behavior anomaly detection based on the voice change data to obtain a voice behavior anomaly score. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0070] Step S205, determining the user willingness evaluation result based on the negative emotion ratio, emotion fluctuation degree, facial behavior abnormality score and voice behavior abnormality score. Figure 1 Step S105 of the illustrated embodiment will not be described in detail here.

[0071] The present embodiment provides a method for evaluating user willingness for financial business processing, which utilizes voice branches, video space branches, and timing branches to achieve accurate extraction of multimodal features in audio and video data to be detected, and performs feature fusion of multimodal features, namely, voice branch coding features, video space branch coding features, and video timing coding features, thereby laying a foundation for the subsequent dynamic capture of the user's emotional changes and behavioral abnormalities during the business processing process.

[0072] In this embodiment, a method for evaluating user willingness for financial business handling is provided, which can be used in the above-mentioned electronic device. Figure 4 is a flow chart of a method for evaluating user willingness for financial business handling according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps: Step S401, obtain the audio and video data to be detected, input the audio and video data to be detected into the multimodal fusion model, and obtain the multimodal fusion features. Figure 2 Step S201 of the illustrated embodiment will not be described in detail here.

[0073] Step S402, calculating the negative emotion ratio and the degree of emotion fluctuation based on the multimodal fusion features.

[0074] Specifically, in order to obtain changes in customer emotions, the emotion probability sequence is calculated by multimodal fusion features, and the proportion of negative emotions and the degree of emotion fluctuation are quantified.

[0075] The above step S402 includes: Step S4021, calculating the emotion probability sequence based on the multimodal fusion features.

[0076] Specifically, the emotion probability sequence is the result sequence of the user's emotion changes in the video inferred by the multimodal fusion model; wherein the video frame information of the audio and video data to be detected is segmented and The emotion probability sequence is obtained by inference calculation. The calculation formula is as follows: (twenty four) in, N The number of video segments.

[0077] Step S4022, statistics are collected on the emotion classification results in the emotion probability sequence, and the negative emotion ratio is calculated based on the statistical results.

[0078] Specifically, the emotion classification results at all moments in the emotion probability sequence are counted, and the proportion of negative emotions is calculated. Anger, sadness, anxiety, and tension are set as negative emotions. The calculation formula is as follows: (25) in, is the total length of the emotion probability sequence, is the indicator function, when The value of the maximum probability is 1 when the emotion is negative, otherwise it is 0.

[0079] Step S4023, calculating the standard deviation of the emotion probability based on the emotion probability sequence, and calculating the degree of emotion fluctuation based on the standard deviation of the emotion probability.

[0080] Specifically, the degree of emotional fluctuation is measured by the standard deviation of emotional probability. The larger the standard deviation of emotional probability, the more intense the emotional fluctuation. The calculation formula for the degree of emotional fluctuation is as follows: (26) in, Indicates the degree of emotional fluctuation. The smaller the number, the more stable the emotions. Indicates the maximum probability value of the emotion in the video frame, represents the standard deviation of the sentiment probability, is the frame number.

[0081] Step S403, extracting facial change data based on the audio and video data to be detected, and performing facial behavior abnormality detection based on the facial change data to obtain a facial behavior abnormality score.

[0082] Specifically, in order to obtain the user's facial changes, a classification model is established using the key points of the facial features, head angle and blinking frequency as key features, and the facial behavior abnormality score is calculated.

[0083] The above step S403 includes: Step S4031, extracting the coordinates of the key points of the facial features, the tip of the nose, the chin and the eyes based on the audio and video data to be detected.

[0084] Step S4032, calculating the change in the coordinates of the key points of the facial features based on the coordinates of the key points of the facial features at adjacent moments.

[0085] Specifically, the coordinate changes of the key points of the facial features are quantified, that is, at time t and t +1, i The coordinates of the key points of the facial features change as follows: (27) (28) in, For the i The horizontal coordinates of the key points of facial features change. For the i The total coordinate changes of the key points of facial features, , for t The horizontal and vertical coordinates of the key points of the facial features at the +1 moment, , for tThe horizontal and vertical coordinates of the key points of the facial features at the moment.

[0086] Furthermore, the coordinate changes of the facial features key points for: (29) Step S4033, calculating the relative displacement of the head based on the coordinates of the nose tip key point and the chin key point.

[0087] Specifically, by calculating the changes in the positions of the nose tip and chin, the relative displacement of the head is obtained, thereby monitoring the pitch and rotation of the head.

[0088] Furthermore, assuming that the nose tip key point , chin key point , then the relative displacement of the head The calculation formula is as follows: (30) Step S4034, determining the eye closure degree based on the eye key point coordinates, and determining the blinking frequency based on the eye closure degree.

[0089] Specifically, the eye closure is calculated through the eye key points , when the eyes are closed, its value will decrease. The blink frequency is obtained by calculating the number of blinks in a period of time. The calculation formula is as follows: (31) Among them, 1 is the indicator function, which means that when the closure degree is less than the threshold , then it returns 1, otherwise it returns 0. Indicates a time period.

[0090] Step S4035, based on the coordinate changes of the key points of the facial features, the relative displacement of the head and the blinking frequency, a classification algorithm is used to calculate the facial behavior abnormality score.

[0091] Specifically, the facial features key point coordinate changes, head relative displacement and blinking frequency are used as features, and the facial behavior abnormality score is calculated using a classification algorithm. The calculation formula is as follows: (32) (33) in, is a general two-classification model. Indicates whether it is abnormal, 1 is abnormal, 0 is normal, hour, Randomly score between 80 and 90.

[0092] Step S404: extracting voice change data based on the audio and video data to be detected, and performing voice behavior anomaly detection based on the voice change data to obtain a voice behavior anomaly score.

[0093] Specifically, in order to obtain changes in customer voice, a classification model is established through features such as speaking speed, volume, and intonation, and the voice behavior abnormality score is calculated.

[0094] The above step S404 includes: Step S4041, segment the audio signal in the audio and video data to be detected, and extract basic speech features in each audio signal segment; the basic speech features include speech rate features, volume features and intonation features.

[0095] Specifically, after preprocessing the audio signal, the basic features of speech are extracted, including speech speed features. , volume characteristics and intonation features , the calculation formula of basic speech features is as follows: (34) (35) (36) in, is the number of frames, is the total duration, is the signal amplitude, is the signal base frequency.

[0096] Step S4042, determining the feature change amplitude based on the basic speech features in each audio signal segment.

[0097] Specifically, the maximum and minimum values ​​of the speech rate feature, volume feature, and intonation feature in each audio segment are calculated, and then the feature variation amplitude is calculated based on the maximum and minimum values.

[0098] Step S4043, based on the basic features of the speech and the amplitude of feature changes, the classification model is used to perform abnormal behavior analysis to obtain a speech behavior abnormality score.

[0099] Specifically, the speech rate feature, volume feature, intonation feature and their feature variation amplitude are used as the input features of the classification model, the behavior anomaly is analyzed and the score is calculated to obtain the speech behavior anomaly score. The calculation formula is as follows: (37) (38) in They represent the difference between the maximum and minimum values ​​of the speech rate feature, volume feature, and intonation feature, that is, the feature variation range. is a general two-classification model. Indicates whether it is abnormal, 1 is abnormal, 0 is normal, hour, Randomly score between 40 and 60.

[0100] Step S405: Determine the user willingness evaluation result based on the negative emotion ratio, emotion fluctuation degree, facial behavior abnormality score and voice behavior abnormality score. Figure 2 Step S205 of the illustrated embodiment will not be described in detail here.

[0101] The present embodiment provides a user willingness assessment method for financial business handling. By calculating the emotion probability sequence through multimodal fusion features, the customer's emotion changes can be accurately obtained. The proportion of negative emotions is obtained by statistically analyzing the emotion classification results in the emotion probability sequence. The degree of emotion fluctuation is calculated based on the standard deviation of the emotion probability, and the proportion of negative emotions and the degree of emotion fluctuation are quantified. Secondly, by extracting the coordinates of the key points of the facial features, the key points of the nose tip, the key points of the chin, and the key points of the eyes, the user's facial changes can be accurately obtained. Then, the change amount of the coordinates of the key points of the facial features, the relative displacement of the head, and the blinking frequency are calculated. The facial behavior abnormality score is calculated using the classification algorithm, and the abnormal behaviors such as facial features changes, head movements, and blinking frequency are captured and scored. Finally, by taking the basic voice features in each audio signal and determining the feature change amplitude based on the basic voice features in each audio signal, the user's voice changes can be accurately obtained. Based on the basic voice features and the feature change amplitude, the classification model is used to perform abnormal behavior analysis to obtain the voice behavior abnormality score, and the user's voice abnormal behavior is captured and scored.

[0102] The following is a specific example to illustrate the specific steps of a method for evaluating user willingness for financial business processing.

[0103] Embodiment 1: Collect a video of a financial business with a frame width and height of 1280×720 and 30s. Figure 5 As shown, the steps for analyzing the real willingness to handle financial business include: Step S1: Extracting video space branch coding features: In order to obtain the image content features and attribute information in the input video frame, the video frame is processed and transformed and calculated through the following steps to obtain the video space branch coding features: Step S11: Extract video frames. Use a video frame extraction tool to read each frame of the video.

[0104] Step S12: Extract facial feature vectors. Perform color space conversion on each video frame and resize it to an input format suitable for the face recognition pre-training model. Then, process the input frame through the convolution layer to obtain the facial feature vectors. ; Among them, the color space uses RGB, the size is 224×224, and EfficientFace (a lightweight deep learning network designed for facial expression recognition) is selected as the pre-trained model. The input is processed into the EfficientFace pre-trained model to obtain high-dimensional features with a dimension size of 512.

[0105] Step S13: Facial deep feature encoding. The facial feature vector is input into the fully connected layer to obtain a high-dimensional vector representation; wherein the fully connected layer structure is 512×768, and the dimension size of the facial deep feature encoding obtained by inputting the fully connected layer is 768.

[0106] Step S14: Extract complex facial features. The facial deep feature encoding is introduced into the subsequent CNN layer to learn and extract complex features, such as facial contour, expression, skin color, etc., to obtain complex facial features. The obtained complex facial feature shape is 28×28×512.

[0107] Step S15: Extract global feature representation of video frames. Design a 2D convolution block, which consists of a set of convolution layers, activation functions (such as ReLU), pooling layers, etc. The purpose is to enhance the nonlinear ability of complex facial features, extract global information in video frames, and obtain global feature representation of video frames. , where the shape of the video frame global feature representation is 64×64×512.

[0108] Step S16: Obtaining video spatial branch coding features. Perform feature concatenation on the global feature representation of the video frame, and finally obtain a comprehensive spatial feature vector as the video spatial branch coding feature, wherein the video spatial coding feature representation has a dimension size of 1024.

[0109] Step S2: Extracting speech branch coding features: In order to obtain the content and emotional information contained in the speech signal in the video, the speech signal is processed and transformed through the following steps to obtain the speech branch coding features: Step S21: Extract the speech signal. Extract the speech signal using a speech signal extraction tool. , where the audio signal T=30000 (30 seconds of audio, sampling rate 1000Hz).

[0110] Step S22: Obtain Mel spectrum features. Convert the speech signal into Mel spectrum features to provide rich frequency domain features for subsequent convolution operations; the number of frequencies is 64, the number of time windows is 128, and the shape is 64×128.

[0111] Step S23: Obtain speech feature representation. Apply 1D convolution layer blocks to perform deep feature extraction on the Mel spectrum features. The input audio features are processed through 1D convolution operations. The input is the Mel spectrum features of the speech signal. , get the audio feature representation , apply a 1D convolution block with kernel k=5 and stride 1, and the shape after convolution is 64×124.

[0112] Step S24: Obtain important speech feature representation. Normalize the speech feature representation to obtain , and then perform RELU activation and maximum pooling operation on the normalized features, where k is the pooling window size, to obtain the important features of speech. The dimension size of the important features of speech is 62.

[0113] Step S25: Obtain N important speech feature representations. Using N 1D convolutional layers, perform steps S23 and S24 to obtain N important speech feature representations, where N is 30.

[0114] Step S26: Obtain speech branch coding features. Concatenate the N important speech feature representations to obtain speech branch coding features. The speech branch encoding feature dimension size is 1024.

[0115] Step S3: Extracting video temporal coding features: In order to obtain dynamic change features such as the direction of the face movement trajectory in the video, the following steps are performed to process and transform the video temporal coding features: Step S31: Obtaining the spatial feature representation of the video frame. The video frame features are calculated by the convolutional neural network model to obtain the spatial feature representation of the video frame. Among them, the convolutional neural network uses Resnet50, and the output shape is 2048×7×7.

[0116] Step S32: Obtain the global feature of the video frame space. Perform a global average pooling operation on the video frame space feature representation to compress the video frame space feature into a one-dimensional vector to obtain the global feature of the video frame space. , where the global feature dimension size of the video frame space is 2048.

[0117] Step S33: Obtain the initial temporal feature sequence of the video frame. Calculate the spatial global feature of each video frame , forming a video frame initial temporal feature sequence X.

[0118] Step S34: Calculate and obtain the temporal dependency feature representation of the video frame. The initial temporal feature sequence of the video frame is processed by the temporal neural network to obtain the temporal dependency feature representation of the video frame. This implementation case uses the LSTM time series neural network to extract time series features, and the dimension size of the video frame time series dependency feature representation is 512.

[0119] Step S35: Obtain video temporal coding features. The video frame temporal dependency feature represents a sequence of . H is calculated through maximum pooling to obtain the video temporal coding features , where the video temporal feature representation has a dimension size of 1024.

[0120] Step S4: Obtaining multimodal fusion features: In order to obtain the coding features containing spatial information, voice information and temporal information in the input video, the multimodal fusion features are calculated and obtained through the following steps.

[0121] Step S41: Obtain the attention coefficient of each modality. Design an attention network to calculate the attention weight of each modality. The inputs of the attention network are , , , the output is the attention coefficient of each modality, and each feature is processed using the FC fully connected mapping to calculate the attention coefficient: Step S42: Calculate and obtain multimodal fusion features. Use the calculated attention coefficient to weight the features of each modality and then add and fuse them to obtain a multimodal fusion feature, where the dimension size of the multimodal fusion feature is 512.

[0122] Step S5: Obtaining the proportion of negative emotions and the degree of emotional fluctuation: In order to obtain the emotional changes of customers, the emotional probability sequence is calculated by multimodal fusion features, and the proportion of negative emotions and the degree of emotional fluctuation are quantified: Step S51: Obtain the emotion probability sequence. The inference calculation results in a sequence of emotion probabilities.

[0123] Step S52: Calculate the proportion of negative emotions. Statistically classify the emotion classification results at all moments in the emotion sequence, calculate the proportion of negative emotions, and define anger, sadness, anxiety, and tension as negative emotions. . Among them, the proportion of negative emotions is 20%.

[0124] Step S53: Quantify and calculate the degree of emotional fluctuation. The degree of emotional fluctuation is measured by the standard deviation of emotional probability. The larger the standard deviation, the more intense the emotional fluctuation, where the standard deviation of emotional fluctuation is 4.5.

[0125] Step S6: Obtain facial behavior anomaly score: In order to obtain the customer's facial changes, a classification model is established using facial features, head angles, and blinking frequency as key features, and the anomaly score is calculated: Step S61: Calculate the coordinate changes of the key points of the facial features.

[0126] Step S62: Calculate the relative displacement. By calculating the position changes of the nose tip and chin, the relative displacement of the head is obtained, so as to monitor the pitch and rotation of the head.

[0127] Step S63: Calculate the blink frequency. Calculate the eye closure degree through the eye key points , when the eyes are closed, its value will decrease. By calculating the number of blinks in a period of time, the blink frequency can be obtained. , with a blink frequency of 13 times.

[0128] Step S64: Calculate the facial behavior abnormality score. , relative displacement and blink frequency As a feature, the anomaly score is calculated using the classification algorithm , where facial behavior abnormality score was 40 points.

[0129] Step S7: Calculate the abnormality score of voice behavior: In order to obtain the changes in the customer's voice, a classification model is established based on features such as speech speed, volume, and intonation, and the abnormality score is calculated: Step S71: After preprocessing the original speech signal, basic speech features are extracted, including speech rate feature SR, tone feature V and intonation feature P, where SR=150, V=80, and P=165.

[0130] Step S72: segment the audio and calculate the maximum and minimum values ​​of the speech rate feature, volume feature and intonation feature in each time period to obtain the feature change amplitude.

[0131] Step S73: Use the speech rate feature, volume feature, intonation feature and their feature change amplitude as the classification model input features, analyze the behavior abnormality and calculate the score to obtain the speech behavior abnormality score, wherein the speech behavior abnormality score is 50 points.

[0132] Step S8: Calculate the willingness score through the real willingness scoring system: A linear regression model is established between the negative emotion ratio, emotional fluctuation degree, facial behavior abnormality score, voice behavior abnormality score and other features obtained in steps S5-7 and the real willingness, as the real willingness scoring system. The user willingness score is calculated through the real willingness scoring system, and the user willingness score is 90 points.

[0133] In the above-mentioned embodiment 1, a multimodal fusion model is used to detect changes in customer emotions, and a classification model is constructed to detect abnormal facial and voice behaviors. According to the results of emotion and behavior analysis, a real willingness evaluation calculation system is constructed. The method of comparing and analyzing customer text transaction records is more real-time, and non-verbal clues such as micro-expressions and abnormal voice intonation that are difficult to detect by structured data analysis can be discovered. At the same time, there is no need to rely on customer historical relationship data, and it supports real-time analysis of user emotions and behaviors during transactions and calculation of willingness. When risks are discovered, the business handling is terminated in advance to protect the rights and interests of both parties, and there will be no delay in risk detection.

[0134] In this embodiment, a user willingness evaluation device for financial business handling is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made are not repeated here. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0135] This embodiment provides a user willingness evaluation device for financial business handling, such as Figure 6 As shown, including: The acquisition module 601 is used to acquire the audio and video data to be detected, input the audio and video data to be detected into the multimodal fusion model, and obtain the multimodal fusion features; A calculation module 602 is used to calculate the negative emotion ratio and the degree of emotion fluctuation based on the multimodal fusion features; A first anomaly detection module 603 is used to extract facial change data based on the audio and video data to be detected, and perform facial behavior anomaly detection based on the facial change data to obtain a facial behavior anomaly score; A second anomaly detection module 604 is used to extract voice change data based on the audio and video data to be detected, and perform voice behavior anomaly detection based on the voice change data to obtain a voice behavior anomaly score; The determination module 605 is used to determine the user willingness evaluation result based on the negative emotion ratio, the emotion fluctuation degree, the facial behavior abnormality score and the voice behavior abnormality score.

[0136] In some optional implementations, the acquisition module 601 includes: A speech feature extraction unit, used to extract speech features from the audio signal in the audio and video data to be detected using the speech branch, and obtain speech branch coding features; A video feature extraction unit, used to extract video features from video frame information in the audio and video data to be detected using a video space branch, and obtain a video space branch coding feature; A time sequence feature extraction unit, used to extract time sequence features from video frame information using time sequence branches to obtain video time sequence coding features; The feature fusion unit is used to fuse the speech branch coding features, the video space branch coding features and the video time sequence coding features to obtain multimodal fusion features.

[0137] In some optional implementations, the feature fusion unit includes: A calculation subunit, used to input speech branch coding features, video space branch coding features and video temporal coding features into the attention network respectively, to obtain the attention coefficient of each modal feature; The weighted fusion subunit is used to use the attention coefficient of each modal feature to weightedly fuse the speech branch coding features, video space branch coding features and video temporal coding features to obtain multimodal fusion features.

[0138] In some optional implementations, the calculation module 602 includes: A first calculation unit, used for calculating an emotion probability sequence based on multimodal fusion features; A statistical unit, used to count the emotion classification results in the emotion probability sequence, and calculate the negative emotion ratio based on the statistical results; The second calculation unit is used to calculate the standard deviation of the emotion probability based on the emotion probability sequence, and calculate the degree of emotion fluctuation based on the standard deviation of the emotion probability.

[0139] In some optional implementations, the first anomaly detection module 603 includes: A key point coordinate extraction unit, used to extract the key point coordinates of facial features, nose tip, chin and eyes based on the audio and video data to be detected; A third calculation unit is used to calculate the change amount of the coordinates of the key points of the facial features based on the coordinates of the key points of the facial features at adjacent moments; A fourth calculation unit, used for calculating the relative displacement of the head based on the coordinates of the nose tip key point and the chin key point; A first determining unit, configured to determine the eye closure degree based on the eye key point coordinates, and determine the blinking frequency based on the eye closure degree; The fifth calculation unit is used to calculate the facial behavior abnormality score using a classification algorithm based on the change in coordinates of key points of facial features, the relative displacement of the head and the blinking frequency.

[0140] In some optional implementations, the second anomaly detection module 604 includes: A segmentation unit is used to segment the audio signal in the audio and video data to be detected, and extract the basic speech features in each audio signal; the basic speech features include speech speed features, volume features and intonation features; A second determining unit, configured to determine a feature variation range based on basic speech features in each audio signal segment; The analysis unit is used to perform abnormal behavior analysis based on basic speech features and feature change amplitudes using a classification model to obtain a speech behavior abnormality score.

[0141] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0142] In this embodiment, a user willingness assessment device for financial business processing is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0143] The embodiment of the present invention also provides a computer device having the above Figure 6 A user willingness assessment device for handling financial business is shown.

[0144] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0145] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0146] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0147] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0148] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0149] The computer device also includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0150] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the computer device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator bar, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device may be a touch screen.

[0151] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0152] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0153] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for evaluating user willingness for financial business handling, characterized in that: The method comprises: Acquire audio and video data to be detected, input the audio and video data to be detected into a multimodal fusion model to obtain multimodal fusion features; Calculating the negative emotion ratio and the degree of emotion fluctuation based on the multimodal fusion features; Extracting facial change data based on the audio and video data to be detected, and performing facial behavior abnormality detection based on the facial change data to obtain a facial behavior abnormality score; Extracting voice change data based on the audio and video data to be detected, and performing voice behavior anomaly detection based on the voice change data to obtain a voice behavior anomaly score; The user willingness evaluation result is determined based on the negative emotion ratio, the emotion fluctuation degree, the facial behavior abnormality score and the voice behavior abnormality score.

2. The method according to claim 1, characterized in that The multimodal fusion model includes a speech branch, a video space branch and a time sequence branch; the step of inputting the audio and video data to be detected into the multimodal fusion model to obtain multimodal fusion features includes: Using the speech branch to extract speech features from the audio signal in the audio and video data to be detected, to obtain speech branch coding features; Using the video space branch to extract video features from the video frame information in the audio and video data to be detected, to obtain video space branch coding features; Extracting timing features from the video frame information using the timing branch to obtain video timing coding features; The speech branch coding feature, the video space branch coding feature and the video time sequence coding feature are feature fused to obtain the multimodal fusion feature.

3. The method according to claim 2, characterized in that The step of fusing the speech branch coding feature, the video space branch coding feature, and the video time sequence coding feature to obtain the multimodal fusion feature includes: The speech branch coding feature, the video space branch coding feature and the video temporal coding feature are respectively input into the attention network to obtain the attention coefficient of each modal feature; The speech branch coding features, the video space branch coding features and the video time sequence coding features are weightedly fused respectively using the attention coefficients of the modal features to obtain the multimodal fusion features.

4. The method according to claim 1, characterized in that: The calculating the negative emotion ratio and the degree of emotion fluctuation based on the multimodal fusion features includes: Calculating an emotion probability sequence based on the multimodal fusion features; Performing statistics on the emotion classification results in the emotion probability sequence, and calculating the negative emotion ratio based on the statistical results; The standard deviation of the emotion probability is calculated based on the emotion probability sequence, and the degree of emotion fluctuation is calculated based on the standard deviation of the emotion probability.

5. The method according to claim 1, characterized in that: The extracting facial change data based on the audio and video data to be detected, and performing facial behavior abnormality detection based on the facial change data to obtain a facial behavior abnormality score, includes: Extracting the key point coordinates of the facial features, the tip of the nose, the chin and the eyes based on the audio and video data to be detected; Calculate the change in the coordinates of the key points of the facial features based on the coordinates of the key points of the facial features at adjacent moments; Calculate the relative displacement of the head based on the coordinates of the nose tip key point and the coordinates of the chin key point; Determine eye closure based on the eye key point coordinates, and determine blink frequency based on the eye closure; The facial behavior abnormality score is calculated using a classification algorithm based on the change in the coordinates of the key points of the facial features, the relative displacement of the head and the blinking frequency.

6. The method according to claim 1, characterized in that The extracting voice change data based on the audio and video data to be detected, and performing voice behavior anomaly detection based on the voice change data to obtain a voice behavior anomaly score, includes: The audio signal in the audio and video data to be detected is segmented, and basic speech features are extracted from each audio signal segment; the basic speech features include speech speed features, volume features and intonation features; Determine the feature change amplitude based on the basic speech features in each audio signal; Based on the basic features of the speech and the amplitude of the feature changes, a classification model is used to perform abnormal behavior analysis to obtain the speech behavior abnormality score.

7. A user willingness evaluation device for financial business handling, characterized in that: The device comprises: An acquisition module is used to acquire audio and video data to be detected, and input the audio and video data to be detected into a multimodal fusion model to obtain multimodal fusion features; A calculation module, used for calculating the negative emotion ratio and the degree of emotion fluctuation based on the multimodal fusion features; A first anomaly detection module is used to extract facial change data based on the audio and video data to be detected, and perform facial behavior anomaly detection based on the facial change data to obtain a facial behavior anomaly score; A second anomaly detection module is used to extract voice change data based on the audio and video data to be detected, and perform voice behavior anomaly detection based on the voice change data to obtain a voice behavior anomaly score; A determination module is used to determine a user willingness evaluation result based on the negative emotion ratio, the emotion fluctuation degree, the facial behavior abnormality score and the voice behavior abnormality score.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the user willingness assessment method for financial business processing according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the user willingness assessment method for financial business processing according to any one of claims 1 to 6.

10. A computer program product, characterized in that It comprises computer instructions, and the computer instructions are used to make a computer execute the user willingness evaluation method for financial business processing according to any one of claims 1 to 6.

Images