Multi-mode fraud emotion recognition method and device

By employing a multimodal fraud emotion recognition method, combining adaptive fuzzy clustering and encoding of optical flow motion features and speech features, and utilizing convolutional neural networks and support vector machines, the problem of low fraud recognition accuracy in existing technologies is solved, achieving efficient capture and robust recognition of micro-expressions.

CN118942142AInactive Publication Date: 2024-11-12PUTIAN EASTERN COMM GRP CO LTD

Patent Information

Application Number
CN202411420928.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-12
Publication Date
2024-11-12
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing fraud detection methods have low accuracy in the internet environment, struggle to effectively capture subtle changes and dynamic patterns in micro-expressions, have high computational complexity, and fail to effectively handle robustness in lighting and noisy environments.

Method used

A multimodal fraud emotion recognition method is adopted. By collecting facial video data, optical flow motion features and speech features are extracted, adaptive fuzzy clustering and encoding are performed, and convolutional neural networks and support vector machines are combined to perform fraud emotion recognition. Multimodal features are then fused for analysis.

Benefits of technology

It improves the accuracy and robustness of fraud detection, effectively captures subtle changes in micro-expressions, reduces computational complexity, is suitable for real-time application scenarios, and enhances adaptability to different lighting and noise environments.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942142A_ABST
    Figure CN118942142A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of fraudulent emotion recognition, and discloses a multi-mode fraudulent emotion recognition method and device, and the method comprises the steps: collecting face video data, respectively extracting the face motion information and voice signal flow information in the face video data, and obtaining optical flow motion features and voice features; performing adaptive fuzzy clustering on the optical flow motion features, and performing adaptive fuzzy motion coding on the optical flow motion features after the adaptive fuzzy clustering to obtain optical flow self-fuzzy motion coding features; performing voice recognition based on the voice features to obtain voice coding features; and performing fraud emotion recognition based on the optical flow self-fuzzy motion coding features and the voice coding features to obtain a fraud emotion recognition result. The method can effectively capture and analyze the fine motion change of the face of the user, and improves the accuracy and robustness of fraud recognition.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of fraud emotion recognition, and in particular to a multimodal fraud emotion recognition method and device. Background Art

[0002] With the rapid development of mobile Internet, the characteristics of the Internet's remote environment make it impossible to effectively implement traditional face-to-face audits and risk control measures. Online fraud activities such as telecommunications network fraud and Internet text message fraud have seriously infringed on the property safety of citizens.

[0003] Emotion recognition technology is used to identify fraud, but related fraud identification methods are prone to deviations, which greatly reduces the accuracy of fraud identification. Summary of the invention

[0004] In view of this, the present invention provides a multimodal fraud emotion recognition method and device to solve the problem that fraud recognition in related fraud recognition methods is prone to deviations, which greatly reduces the accuracy of fraud recognition.

[0005] In a first aspect, the present invention provides a multimodal fraud emotion recognition method, the method comprising: Collecting face video data, extracting facial motion information and voice signal flow information from the face video data, and obtaining optical flow motion features and voice features; Adaptive fuzzy clustering is performed on the optical flow motion features, and adaptive fuzzy motion coding is performed on the optical flow motion features after adaptive fuzzy clustering to obtain optical flow self-fuzzy motion coding features; Perform speech recognition based on speech features to obtain speech coding features; Fraud emotion recognition is performed based on optical flow self-blurred motion coding features and speech coding features to obtain fraud emotion recognition results.

[0006] The present embodiment provides a multimodal fraud emotion recognition method, which extracts optical flow motion features and speech features, performs adaptive fuzzy clustering and adaptive fuzzy motion coding on the optical flow motion features, performs speech recognition on the speech features to obtain speech coding features, and then performs fraud emotion recognition based on the optical flow self-fuzzy motion coding features and the speech coding features. The adaptive fuzzy clustering and adaptive fuzzy motion coding greatly enhance the ability to express subtle changes in micro-expressions, and can effectively capture and analyze subtle motion changes in the user's face. In addition, by fusing the optical flow self-fuzzy motion coding features and audio information, multiple modal features are used to perform aggregate analysis on fraud emotions, thereby improving the accuracy and robustness of fraud recognition.

[0007] In an optional implementation, facial motion information and speech signal flow information are extracted from face video data respectively to obtain optical flow motion features and speech features, including: Decode and integrate the face video data to obtain the face three-dimensional data video; Based on the 3D face data video, the optical flow algorithm is used to calculate the optical flow motion features; The speech signal stream information is separated from the three-dimensional face data video, and the speech signal stream information is preprocessed to obtain speech features.

[0008] A multimodal fraud emotion recognition method provided in this embodiment decodes and integrates facial video data, and then uses the decoded and integrated three-dimensional facial data video to calculate optical flow motion features and separate voice signal stream information, thereby achieving accurate extraction of optical flow motion features and voice features, laying the foundation for subsequent multimodal fraud emotion recognition.

[0009] In an optional implementation, adaptive fuzzy clustering is performed on the optical flow motion features, and adaptive fuzzy motion coding is performed on the adaptive fuzzy clustered optical flow motion features to obtain optical flow self-fuzzy motion coding features, including: Clustering the optical flow motion feature data into multiple fuzzy clusters to obtain adaptive fuzzy motion features and the corresponding membership of the adaptive fuzzy motion features; Initialize the fuzzy motion coding matrix; the size of the fuzzy motion coding matrix is ​​the same as the matrix size corresponding to the adaptive fuzzy motion feature; The adaptive fuzzy motion feature is mapped to the fuzzy motion coding matrix, and the adaptive fuzzy motion feature and the membership degree corresponding to the adaptive fuzzy motion feature are multiplied to obtain the optical flow self-fuzzy motion coding feature.

[0010] The present embodiment provides a multimodal fraud emotion recognition method, which distributes optical flow motion features into multiple fuzzy clusters by clustering, and distributes a membership of a fuzzy set to each optical flow motion feature; the adaptively blurred optical flow motion features are mapped into the fuzzy space, and each feature point has a membership belonging to each cluster, which represents the fuzzy distribution of the feature points in different clusters; further, the features are extracted by fuzzy motion coding, which reduces the influence of environmental noise, improves the robustness to different lighting and noise environments in the fraud recognition process, makes full use of the dynamic patterns between consecutive frames, and enhances the sensitivity to short-term changes in micro-expressions; and, through adaptive fuzzy clustering and adaptive fuzzy motion coding, the dimension and computational complexity of the optical flow self-fuzzy motion coding features can be simplified while maintaining the expressive power of the optical flow self-fuzzy motion coding features, which is more suitable for real-time application scenarios and has lower requirements on computing resources. The fuzzy processing technology can effectively process the uncertainty information in the micro-expression changes, and the adaptive fuzzy motion coding can better process the uncertainty caused by the slow changes in micro-expressions, thereby improving the accuracy of fraud recognition.

[0011] In an optional implementation, clustering the optical flow motion feature data into a plurality of fuzzy clusters to obtain adaptive fuzzy motion features and the membership degrees corresponding to the adaptive fuzzy motion features includes: Initialize the membership matrix, and calculate the cluster center of each fuzzy cluster based on the membership matrix and optical flow motion features; Update the membership matrix based on the cluster center of each fuzzy cluster; It is determined whether the updated membership matrix converges, and the membership matrix is ​​iterated based on the determination result until the membership matrix converges, and the iteration is stopped to obtain the adaptive fuzzy motion feature and the membership corresponding to the adaptive fuzzy motion feature.

[0012] The present embodiment provides a multimodal fraud emotion recognition method, which distributes optical flow motion features into multiple fuzzy clusters through clustering, and assigns a fuzzy set membership to each data point, laying a foundation for the subsequent capture of complex dynamic changes in micro-expressions.

[0013] In an optional implementation, speech recognition is performed based on speech features to obtain speech coding features, including: Based on the speech features, the speech spectrum is obtained by using fast Fourier transform; Perform weighted summation on the spectrum energy of each frequency band in the speech spectrum to obtain the logarithmic Mel spectrum; Based on the logarithmic Mel spectrum, the Mel frequency cepstrum coefficients are obtained by discrete cosine transform; The Mel-frequency cepstral coefficients are sorted, and a preset number of Mel-frequency cepstral coefficients are selected as speech coding features based on the sorting result.

[0014] This embodiment provides a multimodal fraud emotion recognition method, which captures the characteristics of speech signals by performing frequency domain conversion on speech features and then performing cepstrum analysis on the speech spectrum. The Mel-frequency cepstrum coefficients can better capture the subtle differences in speech in speech recognition tasks and improve the accuracy and efficiency of speech coding feature recognition.

[0015] In an optional implementation, fraud emotion recognition is performed based on the optical flow self-blur motion coding features and the speech coding features to obtain a fraud emotion recognition result, including: Convert the optical flow self-blur motion coding features and speech coding features into different formats; The optical flow self-blurred motion coding features and speech coding features after format conversion are fused and spliced ​​to obtain multi-modal features; Based on multimodal features, high-order coding features are extracted using pre-trained convolutional neural networks; The high-order encoding features are input into the pre-trained support vector machine classifier to obtain the fraudulent emotion recognition results.

[0016] A multimodal fraud emotion recognition method provided in this embodiment fuses and splices the format-converted optical flow self-blurred motion coding features and speech coding features, extracts high-order coding features through a convolutional neural network, and then classifies them through a support vector machine, thereby achieving accurate prediction of emotion prediction data. At the same time, the accuracy of fraud emotion recognition is improved by utilizing multiple modal information features.

[0017] In a second aspect, the present invention provides a multimodal fraud emotion recognition device, the device comprising: An extraction module is used to collect face video data, extract facial motion information and voice signal flow information in the face video data, and obtain optical flow motion features and voice features; The encoding module is used to perform adaptive fuzzy clustering on the optical flow motion features, and perform adaptive fuzzy motion encoding on the optical flow motion features after adaptive fuzzy clustering to obtain optical flow self-fuzzy motion encoding features; A speech recognition module, used to perform speech recognition based on speech features to obtain speech coding features; The fraud emotion recognition module is used to perform fraud emotion recognition based on the optical flow self-blur motion coding features and the speech coding features to obtain the fraud emotion recognition results.

[0018] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the multimodal fraud emotion recognition method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0019] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the multimodal fraud emotion recognition method of the first aspect or any corresponding embodiment thereof.

[0020] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the multimodal fraud emotion recognition method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0022] Figure 1 is a flowchart of a multimodal fraud emotion recognition method according to an embodiment of the present invention; Figure 2 is a flow chart of another multimodal fraud emotion recognition method according to an embodiment of the present invention; Figure 3 is a flowchart of another multimodal fraud emotion recognition method according to an embodiment of the present invention; Figure 4 is a flowchart of another multimodal fraud emotion recognition method according to an embodiment of the present invention; Figure 5 is a flow chart of a method for identifying fraudulent emotions using face video data according to an embodiment of the present invention; Figure 6 is a structural block diagram of a multimodal fraud emotion recognition device according to an embodiment of the present invention; Figure 7 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0024] The general implementation steps of emotion recognition technology are feature extraction, feature aggregation and emotion classification. Feature extraction usually uses manually designed features of static images (such as LBP, HOG, etc.) or directly uses features calculated by optical flow to capture changes in facial expressions. Although these features can describe local motion and texture information, they cannot fully capture the complex dynamic changes of micro-expressions. Related emotion recognition technologies rely on static features or direct optical flow calculations, and the feature expression ability is limited, making it difficult to fully capture the complex dynamic information of micro-expressions. Feature aggregation For emotion recognition in videos, features of multiple frames are aggregated together, and simple time series models or pooling methods are usually used to describe the dynamic changes of emotions. This method is not enough to capture the subtle and rapid changes of micro-expressions. Emotion classification usually uses machine learning classifiers such as SVM (Support Vector Machine) and KNN (K-NearestNeighbor) to classify the extracted features. Although the above methods perform well in static expression recognition, they are often not ideal when dealing with complex micro-expression recognition.

[0025] In response to the above defects, related emotion recognition technologies directly use optical flow calculations, which have the following disadvantages: 1) Limited feature expression capabilities. Although the features directly obtained by using optical flow calculations can reflect the motion information in the image, these features are often relatively primitive. Unprocessed optical flow features may contain a large amount of redundant information and noise, and cannot fully express the subtle changes in micro-expressions; 2) Insufficient utilization of dynamic information. Although optical flow can capture inter-frame motion, directly using optical flow features may ignore more complex dynamic patterns between consecutive frames, which are very important for micro-expression recognition; 3) High computational complexity. The computational complexity of optical flow calculations themselves is high. Directly using optical flow features for classification may further increase the computational complexity, especially when processing large-scale video data, and the real-time performance is poor; 4) Lack of fuzzy processing. The results of optical flow calculations are deterministic, and the fuzzy characteristics of micro-expressions are not taken into account. Uncertain information caused by the slow changes in micro-expressions cannot be processed.

[0026] The embodiment of the present invention provides a multimodal fraud emotion recognition method, which maps optical flow features to fuzzy space through optical flow self-fuzzy motion coding, and further encodes, and then uses multimodality to perform fraud emotion recognition analysis, which greatly enhances the ability to express subtle changes in micro-expressions. The method can be widely used in remote identity authentication (when customers perform remote identity authentication through video, banks can use multimodal recognition technology to detect customers' emotional changes), loan application evaluation (in the process of online loan application, the borrower's emotional state may reflect his true intentions), anti-money laundering monitoring (the system can analyze customers' micro-expressions to identify customers who show abnormal emotional changes in high-risk activities such as fund transfers and cross-border transactions, thereby providing more accurate risk warnings for anti-money laundering departments), customer service conversation monitoring (the system detects that customers show abnormal emotions (such as anger, confusion or extreme tension) in key conversations, and customer service can take more cautious response measures to prevent potential fraud) and other fraud scenarios.

[0027] According to an embodiment of the present invention, an embodiment of a multimodal fraud emotion recognition method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0028] In this embodiment, a multimodal fraud emotion recognition method is provided, which can be used in server-type devices. Figure 1 is a flow chart of a multimodal fraud emotion recognition method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps: Step S101, collect face video data, extract facial motion information and voice signal flow information in the face video data, and obtain optical flow motion features and voice features.

[0029] Specifically, users can import video data into an image processing or computer vision system through a programming interface or software platform. The image processing or computer vision system first receives raw video data containing a face, i.e., face video data. The face video data can be captured in real time from a camera or loaded from a storage device with related video files.

[0030] Step S102, performing adaptive fuzzy clustering on the optical flow motion features, and performing adaptive fuzzy motion coding on the adaptive fuzzy clustered optical flow motion features to obtain optical flow self-fuzzy motion coding features.

[0031] Step S103: Perform speech recognition based on the speech features to obtain speech coding features.

[0032] Step S104, performing fraudulent emotion recognition based on the optical flow self-blurred motion coding features and the speech coding features to obtain a fraudulent emotion recognition result.

[0033] The present embodiment provides a multimodal fraud emotion recognition method, which extracts optical flow motion features and speech features, performs adaptive fuzzy clustering and adaptive fuzzy motion coding on the optical flow motion features, performs speech recognition on the speech features to obtain speech coding features, and then performs fraud emotion recognition based on the optical flow self-fuzzy motion coding features and the speech coding features. The adaptive fuzzy clustering and adaptive fuzzy motion coding greatly enhance the ability to express subtle changes in micro-expressions, and can effectively capture and analyze subtle motion changes in the user's face. In addition, by fusing the optical flow self-fuzzy motion coding features and audio information, multiple modal features are used to perform aggregate analysis on fraud emotions, thereby improving the accuracy and robustness of fraud recognition.

[0034] In this embodiment, a multimodal fraud emotion recognition method is provided, which can be used in the above-mentioned server-type devices. Figure 2 is a flow chart of a multimodal fraud emotion recognition method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps: Step S201, collect face video data, extract facial motion information and voice signal flow information in the face video data, and obtain optical flow motion features and voice features.

[0035] Specifically, the above step S201 includes: Step S2011, decode and integrate the face video data to obtain a three-dimensional face data video.

[0036] Specifically, a video file in a compressed format (i.e., face video data) is decoded into an original frame sequence through a video decoder, and the decoded video is split into a series of static image frames, each frame being a picture of a face, thereby obtaining a face image frame sequence.

[0037] Furthermore, the facial image frame sequence is integrated into a three-dimensional data structure (usually called 3D volume data), where the third dimension represents the time axis. The size of the facial three-dimensional data video is ,in is the spatial resolution of the video frame, Indicates the height of the image. Indicates the width of the image. Indicates the number of frames of the 3D face data video.

[0038] Furthermore, the color image in the face 3D data video is converted into a grayscale image to reduce the computational complexity; The grayscale image of the face of the frame is represented as ,in and are pixel coordinates.

[0039] Step S2012, based on the three-dimensional face data video, use the optical flow algorithm to calculate the optical flow motion features.

[0040] Specifically, the optical flow algorithm is used to calculate the optical flow field between video frames to obtain the motion vector of each pixel, that is, the optical flow motion feature. The optical flow motion feature contains the direction and amplitude data of the motion. The steps of calculating the optical flow motion feature using the optical flow algorithm include: calculating the time gradient of each pixel in the grayscale image of the face , and spatial gradient and , where the temporal gradient reflects the change of pixels over time, the spatial gradient reflects the change of pixels over space, and the temporal gradient , and spatial gradient and It can be expressed as follows: (1) (2) (3) Furthermore, assuming that the brightness consistency principle is met, the image brightness remains unchanged during motion, according to the time gradient , and spatial gradient and Construct the optical flow equation. The expression of the optical flow equation is as follows: (4) in, and is the pixel displacement to be calculated.

[0041] Furthermore, in a local window, the optical flow equation is solved and the displacement of each pixel is obtained by the least square method; wherein the gradient matrix is ​​constructed according to the above optical flow equation and the time gradient vector , the gradient matrix and the time gradient vector It can be expressed as: (5) (6) Furthermore, the gradient matrix and the time gradient vector Enter the expression , solve the optical flow vector , optical flow vector The calculation formula is as follows: (7) Furthermore, for each consecutive frame, the above calculation is repeated to obtain the optical flow vector field between all frames, and the result is a vector field of size The optical flow feature matrix is ​​, where "2" represents the horizontal displacement and vertical displacement of each pixel; wherein the optical flow vector field is used as the optical flow feature, which includes the motion characteristics of horizontal and vertical displacements.

[0042] Step S2013, separating the voice signal stream information from the three-dimensional face data video, and preprocessing the voice signal stream information to obtain voice features.

[0043] Specifically, the voice signal stream information is separated from the video file, and the voice signal stream information is saved as a wav file (standard digital audio file).

[0044] Furthermore, the audio signal is preprocessed to remove background noise, and the volume of the audio signal is normalized and adjusted to reach a uniform amplitude level, thereby obtaining speech features.

[0045] Step S202, adaptive fuzzy clustering is performed on the optical flow motion features, and adaptive fuzzy motion coding is performed on the adaptive fuzzy clustered optical flow motion features to obtain optical flow self-fuzzy motion coding features. Figure 1 Step S102 of the illustrated embodiment will not be described in detail here.

[0046] Step S203: Perform speech recognition based on the speech features to obtain speech coding features. Figure 1 Step S103 of the illustrated embodiment will not be described in detail here.

[0047] Step S204: perform fraudulent emotion recognition based on the optical flow self-blurred motion coding features and the speech coding features to obtain fraudulent emotion recognition results. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0048] A multimodal fraud emotion recognition method provided in this embodiment decodes and integrates facial video data, and then uses the decoded and integrated three-dimensional facial data video to calculate optical flow motion features and separate voice signal stream information, thereby achieving accurate extraction of optical flow motion features and voice features, laying the foundation for subsequent multimodal fraud emotion recognition.

[0049] In this embodiment, a multimodal fraud emotion recognition method is provided, which can be used in the above-mentioned server-type devices. Figure 3 is a flow chart of a multimodal fraud emotion recognition method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps: Step S301, collect face video data, extract facial motion information and voice signal flow information from the face video data, and obtain optical flow motion features and voice features. Figure 2 Step S201 of the illustrated embodiment will not be described in detail here.

[0050] Step S302, adaptive fuzzy clustering is performed on the optical flow motion features, and adaptive fuzzy motion coding is performed on the adaptive fuzzy clustered optical flow motion features to obtain optical flow self-fuzzy motion coding features.

[0051] Specifically, the above step S302 includes: Step S3021, clustering the optical flow motion feature data into multiple fuzzy clusters to obtain adaptive fuzzy motion features and the membership degrees corresponding to the adaptive fuzzy motion features.

[0052] In some optional implementations, the above step S3012 includes: Step a1, initialize the membership matrix, and calculate the cluster center of each fuzzy cluster according to the membership matrix and the optical flow motion feature.

[0053] Specifically, the number of clustering iterations, fuzzy factors and membership matrix are initialized, and the pixel data points in the grayscale face image are divided according to the number of clustering iterations to obtain multiple fuzzy clusters. The membership in the membership matrix after initialization describes the possibility that the pixel data points in the grayscale face image belong to a fuzzy cluster. The value of the membership is between 0 and 1, indicating the degree of membership of the pixel data point to a cluster. The higher the membership, the more likely the pixel data point is to belong to the cluster. Each pixel data point can belong to multiple fuzzy clusters at the same time. The fuzziness helps to capture the complex dynamic changes in micro-expressions.

[0054] Furthermore, the membership matrix The size is , where “3” represents three clusters.

[0055] Further, The cluster centers of fuzzy clusters The calculation formula is as follows: (8) in, is the number of pixels, For the The optical flow vector of pixels, For the The pixel data point is The membership degree of a cluster, is the fuzzy factor, The value of is usually set to 2.

[0056] Step a2: Update the membership matrix based on the cluster center of each fuzzy cluster.

[0057] Specifically, according to the new cluster center , update the membership matrix , the membership degree in the updated membership matrix The calculation formula is as follows: (9) in, is the current iteration number of clustering, is the number of clustering iterations.

[0058] Step a3, judging whether the updated membership matrix converges, iterating the membership matrix based on the judgment result until the membership matrix converges, stopping the iteration, and obtaining the adaptive fuzzy motion feature and the membership corresponding to the adaptive fuzzy motion feature.

[0059] Specifically, it is determined whether the difference between the membership matrix before and after the update is less than a threshold value. If it has not converged, it returns to step a1 and continues to iterate until the membership matrix converges, and then the iteration is stopped.

[0060] Step S3022, initialize the fuzzy motion coding matrix; the size of the fuzzy motion coding matrix is ​​the same as the matrix size corresponding to the adaptive fuzzy motion feature.

[0061] Specifically, the size of the fuzzy motion coding matrix is ​​the same as the matrix size corresponding to the adaptive fuzzy motion feature, that is, the size of the fuzzy motion coding matrix is ,in is the number of clusters.

[0062] Step S3023, mapping the adaptive fuzzy motion feature to the fuzzy motion coding matrix, and multiplying the adaptive fuzzy motion feature and the membership degree corresponding to the adaptive fuzzy motion feature to obtain the optical flow self-fuzzy motion coding feature.

[0063] Specifically, for each pixel corresponding to the adaptive fuzzy motion feature, the cluster is weighted using its membership degree to obtain the fuzzy motion coding; and Clusters, optical flow self-blurring motion encoding features The calculation formula is: (10) in, and To adapt to the characteristic components of blurred motion features, is a membership matrix composed of the memberships corresponding to the adaptive fuzzy motion features.

[0064] Step S303: Perform speech recognition based on the speech features to obtain speech coding features.

[0065] Specifically, the above step S303 includes: Step S3031, based on the speech features, the speech spectrum is obtained by using fast Fourier transform.

[0066] Specifically, the Fast Fourier Transform (FFT) is used to capture the intensity and distribution of frequency components in speech features, and then the time domain signal corresponding to the speech features is converted into the frequency domain to obtain the speech spectrum.

[0067] Step S3032: weighted summing is performed on the spectrum energy of each frequency band in the speech spectrum to obtain a logarithmic Mel spectrum.

[0068] Specifically, the speech spectrum is mapped to the Mel scale using the ‌triangular window function, and the logarithmic energy of the spectrum on each Mel scale is calculated. That is, the FFT energy of each frequency band is weighted summed through the Mel filter group, and the energy distribution of the speech signal on the Mel frequency is calculated to obtain the output value of each filter, that is, the logarithmic Mel spectrum.

[0069] Step S3033: Based on the logarithmic Mel frequency spectrum, a discrete cosine transform is used to obtain Mel frequency cepstrum coefficients.

[0070] Specifically, the logarithmic Mel spectrum is processed using discrete cosine transform to obtain a set of Mel Frequency Cepstrum Coefficients (MFCC).

[0071] Step S3034: sort the Mel-frequency cepstral coefficients, and select a preset number of Mel-frequency cepstral coefficients as speech coding features based on the sorting result.

[0072] Specifically, the Mel-frequency cepstral coefficients are sorted from large to small, and the first n Mel-frequency cepstral coefficients are selected as speech coding features. The value of n may be 13.

[0073] Step S304: perform fraudulent emotion recognition based on the optical flow self-blurred motion coding features and the speech coding features to obtain fraudulent emotion recognition results. Figure 2 Step S204 of the illustrated embodiment will not be described in detail here.

[0074] The present embodiment provides a multimodal fraud emotion recognition method, which distributes optical flow motion features into multiple fuzzy clusters by clustering, and distributes a membership of a fuzzy set to each optical flow motion feature; the adaptively blurred optical flow motion features are mapped into the fuzzy space, and each feature point has a membership belonging to each cluster, which represents the fuzzy distribution of the feature points in different clusters; further, the features are extracted by fuzzy motion coding, which reduces the influence of environmental noise, improves the robustness to different lighting and noise environments in the fraud recognition process, makes full use of the dynamic patterns between consecutive frames, and enhances the sensitivity to short-term changes in micro-expressions; and, through adaptive fuzzy clustering and adaptive fuzzy motion coding, the dimension and computational complexity of the optical flow self-fuzzy motion coding features can be simplified while maintaining the expressive power of the optical flow self-fuzzy motion coding features, which is more suitable for real-time application scenarios and has lower requirements on computing resources. The fuzzy processing technology can effectively process the uncertainty information in the micro-expression changes, and the adaptive fuzzy motion coding can better process the uncertainty caused by the slow changes in micro-expressions, thereby improving the accuracy of fraud recognition.

[0075] In this embodiment, a multimodal fraud emotion recognition method is provided, which can be used in the above-mentioned server-type devices. Figure 4 is a flow chart of a multimodal fraud emotion recognition method according to an embodiment of the present invention. Figure 4 As shown, the process includes the following steps: Step S401, collect face video data, extract facial motion information and voice signal flow information from the face video data, and obtain optical flow motion features and voice features. Figure 3 Step S301 of the illustrated embodiment will not be described in detail here.

[0076] Step S402, adaptive fuzzy clustering is performed on the optical flow motion features, and adaptive fuzzy motion coding is performed on the optical flow motion features after adaptive fuzzy clustering to obtain optical flow self-fuzzy motion coding features. Figure 3 Step S302 of the illustrated embodiment will not be described in detail here.

[0077] Step S403: Perform speech recognition based on the speech features to obtain speech coding features. Figure 3 Step S303 of the illustrated embodiment will not be described in detail here.

[0078] Step S404, performing fraudulent emotion recognition based on the optical flow self-blurred motion coding features and the speech coding features to obtain a fraudulent emotion recognition result.

[0079] Specifically, the above step S404 includes: Step S4041, converting the optical flow self-blurred motion coding features and the speech coding features into a format.

[0080] Specifically, the optical flow self-blurred motion coding features are converted into a 4-dimensional format: height × width × number of channels × number of samples, and the speech coding features are converted into the same dimensional shape as the blurred motion coding.

[0081] Step S4042, fusing and splicing the optical flow self-blurred motion coding features and speech coding features after format conversion to obtain multi-modal features.

[0082] Step S4043, based on the multimodal features, high-order coding features are extracted using a pre-trained convolutional neural network.

[0083] Specifically, a convolutional neural network is constructed and trained, and then the convolutional neural network is used to extract high-order coding features after blurred motion coding. The high-order coding features are the output results of the fully connected layer in the convolutional neural network.

[0084] Step S4044, input the high-order coding features into a pre-trained support vector machine classifier to obtain fraudulent emotion recognition results.

[0085] Specifically, the historical data of high-order coding features and their corresponding emotion labels are used to train a support vector machine classifier, and the trained support vector machine classifier is used to predict the emotion data to obtain emotion prediction data, and then fraud identification is performed based on the emotion prediction data.

[0086] A multimodal fraud emotion recognition method provided in this embodiment fuses and splices the format-converted optical flow self-blurred motion coding features and speech coding features, extracts high-order coding features through a convolutional neural network, and then classifies them through a support vector machine, thereby achieving accurate prediction of emotion prediction data. At the same time, the accuracy of fraud emotion recognition is improved by utilizing multiple modal information features.

[0087] The specific steps of a multimodal fraud emotion recognition method are described below through a specific embodiment.

[0088] Embodiment 1: like Figure 5 As shown in FIG. 1 , assuming that micro-expressions of a facial video data with a size of 71×81×25 are to be recognized, the specific steps of a multimodal fraud emotion recognition method are as follows: Step S1: Calculate optical flow motion features: Extract facial motion information to provide basic data for subsequent feature processing. Use the optical flow algorithm to calculate the optical flow field between video frames and obtain the motion vector of each pixel. The optical flow feature contains the direction and amplitude of the motion.

[0089] Step S11: Read the face video, the size of which is 71×81×25, which are the video resolution height, resolution width, and video frame number, respectively.

[0090] Step S12: Image grayscale: Convert the color image into a grayscale image to reduce the computational complexity: Assume that the grayscale image of the i-th frame is represented as ,in and are pixel coordinates, and the grayscale image size is 71×81×25.

[0091] Step S13: Calculate image gradient: Calculate the temporal gradient of each pixel , and spatial gradient and The temporal gradient reflects the change of pixels over time, and the spatial gradient reflects the change of pixels over space. The size of each gradient result is 71×81×25. The temporal gradient of each pixel is , and spatial gradient and It is expressed as:

[0092]

[0093]

[0094] Step S14: Establishing an optical flow equation: Taking the middle pixel as an example, the expression of the optical flow equation is as follows:

[0095] Step S15: Solve the optical flow equation: The size of the window is 3×3, solve the optical flow equation, and use the least squares method to obtain the displacement of each pixel:

[0096]

[0097] The gradient matrix and the time gradient vector Enter the expression , solve the optical flow vector , optical flow vector The calculation formula is as follows:

[0098] Therefore, the optical flow vector of this pixel is .

[0099] Step S16: Apply to all frames: For each pair of consecutive frames, repeat the above calculation to obtain the optical flow vector field between all frames. The result is an optical flow feature matrix of size 71×81×24×2, where the last 2 represents the horizontal and vertical displacement of each pixel. The optical flow vector field between all frames is: [[[(-1,0),(-1,0),...,(-1,0)],[(-1,0),(-1,0),...,(-1,0)],...,[(-1,0),(-1,0),...,(-1,0)]],...,[[(-1,0),(-1,0),...,(-1,0)],[(-1,0),(-1,0),...,(-1,0)],...,[(-1,0),(-1,0),...,(-1,0)]]].

[0100] Step S2: Adaptive fuzzy calculation of optical flow features: The optical flow motion features are assigned to multiple fuzzy clusters by clustering, and a membership degree of a fuzzy set is assigned to each data point.

[0101] Step S21: Initialize parameters, number of clusters , fuzzy factor , initialize the membership matrix (random value).

[0102] Step S22: Initialize the membership matrix , membership matrix The size of is 71×81×24×3, where the last 3 represents three clusters; assuming that a certain pixel position The optical flow vector at for , membership matrix .

[0103] Step S23: Calculate cluster centers .

[0104] Step S24: Update the membership matrix , according to the new cluster center , update the membership matrix .

[0105] Step S25: Check convergence, check membership matrix Whether it converges, that is, whether the difference before and after the update is less than 0.00001, if not converged, return to step S23 and continue iterating. The final adaptive fuzzy motion feature size is 71×81×24×2.

[0106] Step S3: Fuzzy motion coding calculation, generating optical flow self-fuzzy motion coding features, enhancing the expressiveness and robustness of the features: the adaptively blurred optical flow motion features are mapped to the fuzzy space, and each feature point will have a membership degree belonging to each cluster, which represents the fuzzy distribution of the feature points in different clusters.

[0107] Step S31: determine the size of the blurred motion coding matrix. The size of the blurred motion coding matrix is ​​the same as the optical flow feature matrix, that is, 71×81×24×3.

[0108] Step S32: Calculate the fuzzy motion code. For each pixel and each frame, use its membership degree to weight the clusters to obtain the fuzzy motion code: The optical flow vector is , the pixel's membership to the first cluster is 0.4. The norm of the optical flow vector is 1, that is ( ), then the optical flow self-blurred motion encoding feature The values ​​are: ; If the pixel's membership to the second cluster is 0.3, then: ; For the third cluster, assuming the membership degree is 0.3, then: ; Finally, the blurred motion coding matrix at this pixel position The result is: .

[0109] Step S4: Speech feature extraction: extract the speech signal stream information in the original micro-expression video and pre-process it.

[0110] Step S41: Separate the audio stream signal from the video file and save the signal as a wav file.

[0111] Step S42: pre-process the audio signal to remove background noise and normalize and adjust the volume of the audio signal to a uniform amplitude level.

[0112] Step S5: Speech coding feature calculation: Use MFCC Mel-frequency cepstral coefficients to convert the speech signal from the time domain to the frequency domain, and perform cepstral analysis on its spectrum to capture the characteristics of the speech signal.

[0113] Step S51: Convert the time domain signal into the frequency domain to obtain the spectrum of the signal, and use the Fast Fourier Transform (FFT) to capture the intensity and distribution of the frequency components in the speech signal.

[0114] Step S52: Perform weighted summation on the FFT energy of each frequency band through the Mel filter bank to obtain the output value of each filter, that is, the Mel spectrum, and calculate the energy distribution of the speech signal on the Mel frequency.

[0115] Step S53: Use discrete cosine transform to process the logarithmic Mel spectrum to obtain a set of Mel frequency cepstral coefficients MFCC, and take the first 13 coefficients as the final speech features.

[0116] Step S6: Emotion recognition, modally fuse the optical flow self-blurring motion coding features with the speech coding features, and apply the classifier to predict micro-expressions. After the optical flow self-blurring motion coding features and the speech coding features are concatenated, feature learning is performed through a convolutional neural network, and then classification is performed through a support vector machine.

[0117] Step S61: Convert the matrix corresponding to the optical flow self-blurred motion coding feature into a 4-dimensional format: 71×81×24×3=1,091,775.

[0118] Step S62: transform the speech coding features into the same dimensional shape as the optical flow self-blurred motion coding features, which is 4D.

[0119] Step S63: construct and train a convolutional neural network to learn the fused multimodal features and use the multimodal features to perform emotion classification.

[0120] Step S64: high-level feature extraction, extracting features of each layer from the video through the trained convolutional neural network. The length of the feature vector obtained is 128 (output of the fully connected layer).

[0121] Step S65: training a support vector machine classifier, using features extracted by the convolutional neural network to train the support vector machine classifier.

[0122] Step S66: emotion recognition, using the trained support vector machine classifier to predict the emotion data.

[0123] In this embodiment, a multimodal fraud emotion recognition device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. As used below, the term "module" may be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0124] This embodiment provides a multi-modal fraud emotion recognition device, such as Figure 6 As shown, including: Extraction module 601, used to collect face video data, extract facial motion information and voice signal flow information in the face video data, and obtain optical flow motion features and voice features; The encoding module 602 is used to perform adaptive fuzzy clustering on the optical flow motion features, and perform adaptive fuzzy motion encoding on the adaptive fuzzy clustered optical flow motion features to obtain optical flow self-fuzzy motion encoding features; A speech recognition module 603 is used to perform speech recognition based on speech features to obtain speech coding features; The fraud emotion recognition module 604 is used to perform fraud emotion recognition based on the optical flow self-blur motion coding features and the speech coding features to obtain a fraud emotion recognition result.

[0125] In some optional implementations, the extraction module 601 includes: A decoding and integration unit, used for decoding and integrating the face video data to obtain a three-dimensional face data video; A calculation unit, used for calculating optical flow motion features using an optical flow algorithm based on the three-dimensional face data video; The separation unit is used to separate the voice signal stream information from the three-dimensional face data video, and pre-process the voice signal stream information to obtain voice features.

[0126] In some optional implementations, the encoding module 602 includes: An allocation unit, used for clustering and allocating the optical flow motion feature data into a plurality of fuzzy clusters, and obtaining adaptive fuzzy motion features and the membership degrees corresponding to the adaptive fuzzy motion features; An initialization unit, used for initializing a fuzzy motion coding matrix; the size of the fuzzy motion coding matrix is ​​the same as the size of the matrix corresponding to the adaptive fuzzy motion feature; The mapping unit is used to map the adaptive fuzzy motion feature to the fuzzy motion coding matrix, and multiply the adaptive fuzzy motion feature and the membership degree corresponding to the adaptive fuzzy motion feature to obtain the optical flow self-fuzzy motion coding feature.

[0127] In some optional embodiments, the allocation unit includes: An initialization subunit is used to initialize the membership matrix and calculate the cluster center of each fuzzy cluster according to the membership matrix and the optical flow motion feature; An updating subunit, used for updating the membership matrix based on the cluster center of each fuzzy cluster; The judgment subunit is used to judge whether the updated membership matrix converges, iterate the membership matrix based on the judgment result until the membership matrix converges, stop the iteration, and obtain the adaptive fuzzy motion feature and the membership corresponding to the adaptive fuzzy motion feature.

[0128] In some optional implementations, the speech recognition module 603 includes: A first transform unit is used to obtain a speech spectrum by using a fast Fourier transform based on speech features; A weighted summing unit is used to perform weighted summing of the spectrum energy of each frequency band in the speech spectrum to obtain a logarithmic Mel spectrum; A second transform unit is used to obtain Mel frequency cepstrum coefficients by discrete cosine transform based on the logarithmic Mel frequency spectrum; The selection unit is used to sort the Mel-frequency cepstral coefficients and select a preset number of Mel-frequency cepstral coefficients as speech coding features based on the sorting result.

[0129] In some optional implementations, the fraud emotion recognition module 604 includes: A conversion unit, used for converting the optical flow self-blur motion coding features and the speech coding features into a format; A splicing unit, used for fusing and splicing the optical flow self-blurred motion coding features and speech coding features after format conversion to obtain multi-modal features; An extraction unit, used for extracting high-order coding features based on multimodal features using a pre-trained convolutional neural network; The prediction unit is used to input the high-order coding features into the pre-trained support vector machine classifier to obtain the fraud emotion recognition result.

[0130] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0131] A multimodal fraud emotion recognition device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0132] The embodiment of the present invention also provides a computer device having the above Figure 6 A multimodal fraud emotion recognition device is shown.

[0133] See also Figure 7 , Figure 7 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 7As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 7 A processor 10 is taken as an example.

[0134] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0135] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0136] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0137] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0138] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0139] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0140] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0141] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A multimodal fraud emotion recognition method, characterized in that: The method comprises: Collecting face video data, respectively extracting facial motion information and voice signal flow information from the face video data, and obtaining optical flow motion features and voice features; Performing adaptive fuzzy clustering on the optical flow motion features, and performing adaptive fuzzy motion coding on the optical flow motion features after adaptive fuzzy clustering to obtain optical flow self-fuzzy motion coding features; Perform speech recognition based on the speech features to obtain speech coding features; Fraudulent emotion recognition is performed based on the optical flow self-blurred motion coding features and the speech coding features to obtain a fraudulent emotion recognition result.

2. The method according to claim 1, characterized in that The extracting facial motion information and speech signal flow information from the face video data to obtain optical flow motion features and speech features includes: Decoding and integrating the face video data to obtain a three-dimensional face data video; Based on the three-dimensional face data video, using an optical flow algorithm to calculate the optical flow motion feature; The speech signal stream information is separated from the three-dimensional face data video, and the speech signal stream information is preprocessed to obtain the speech feature.

3. The method according to claim 1, characterized in that The step of performing adaptive fuzzy clustering on the optical flow motion features and performing adaptive fuzzy motion coding on the optical flow motion features after adaptive fuzzy clustering to obtain optical flow self-fuzzy motion coding features includes: Clustering the optical flow motion feature data into a plurality of fuzzy clusters to obtain adaptive fuzzy motion features and membership degrees corresponding to the adaptive fuzzy motion features; Initializing a fuzzy motion coding matrix; the size of the fuzzy motion coding matrix is ​​the same as the matrix size corresponding to the adaptive fuzzy motion feature; The adaptive fuzzy motion feature is mapped to the fuzzy motion coding matrix, and the adaptive fuzzy motion feature and the membership degree corresponding to the adaptive fuzzy motion feature are multiplied to obtain the optical flow self-fuzzy motion coding feature.

4. The method according to claim 3, characterized in that The clustering and allocating the optical flow motion feature data into a plurality of fuzzy clusters to obtain adaptive fuzzy motion features and the membership degrees corresponding to the adaptive fuzzy motion features includes: Initializing a membership matrix, and calculating a cluster center of each fuzzy cluster according to the membership matrix and the optical flow motion feature; Updating the membership matrix based on the cluster center of each fuzzy cluster; Determine whether the updated membership matrix converges, iterate the membership matrix based on the determination result until the membership matrix converges, stop iterating, and obtain the adaptive fuzzy motion feature and the membership corresponding to the adaptive fuzzy motion feature.

5. The method according to claim 1, characterized in that The performing speech recognition based on the speech feature to obtain speech coding features includes: Based on the speech features, a speech spectrum is obtained by using a fast Fourier transform; Performing weighted summation on the spectrum energy of each frequency band in the speech spectrum to obtain a logarithmic Mel spectrum; Based on the logarithmic Mel spectrum, using discrete cosine transform to obtain Mel frequency cepstrum coefficients; The Mel-frequency cepstral coefficients are sorted, and a preset number of the Mel-frequency cepstral coefficients are selected as the speech coding features based on the sorting result.

6. The method according to claim 1, characterized in that The performing fraudulent emotion recognition based on the optical flow self-blurred motion coding feature and the speech coding feature to obtain the fraudulent emotion recognition result includes: Converting the optical flow self-blurred motion coding feature and the speech coding feature into a format; The optical flow self-blurred motion coding features and speech coding features after format conversion are fused and spliced ​​to obtain multi-modal features; Based on the multimodal features, extracting high-order coding features using a pre-trained convolutional neural network; The high-order coding features are input into a pre-trained support vector machine classifier to obtain the fraudulent emotion recognition result.

7. A multimodal fraud emotion recognition device, characterized in that: The device comprises: An extraction module is used to collect face video data, extract facial motion information and voice signal flow information in the face video data, and obtain optical flow motion features and voice features; An encoding module is used to perform adaptive fuzzy clustering on the optical flow motion features, and perform adaptive fuzzy motion encoding on the optical flow motion features after adaptive fuzzy clustering to obtain optical flow self-fuzzy motion encoding features; A speech recognition module, used for performing speech recognition based on the speech features to obtain speech coding features; The fraud emotion recognition module is used to perform fraud emotion recognition based on the optical flow self-blur motion coding features and the speech coding features to obtain a fraud emotion recognition result.

8. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multimodal fraud emotion recognition method according to any one of claims 1 to 6 by executing the computer instructions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the multimodal fraud emotion recognition method described in any one of claims 1 to 6.

10. A computer program product, characterized in that It comprises computer instructions, and the computer instructions are used to make a computer execute the multimodal fraud emotion recognition method according to any one of claims 1 to 6.

Citation Information

Patent Citations

  • Video data fraud detection method and device, computer equipment and storage medium

    CN110781916A

  • Facial fraud action recognition method based on visual optical flow features

    CN113901916A

  • Virtual reality equipment and multi-mode emotion recognition method

    CN116400802A

  • Human activity identification method based on convolution and recurrent neural network fusion

    CN117133013A

Cited By

  • User willingness evaluation method and device for financial business handling

    CN119417492A

  • Old people non-voluntary transaction detection method and system based on psychological-behavioral reasoning

    CN121526750A