Method, apparatus, and medium for multi-modal information analysis

Abstract

A method and device for multi-modal information analysis based on a joint discriminant network, an electronic device and a medium are provided, which can be applied to the fields of big data technology and artificial intelligence technology. The method comprises: obtaining facial information, audio information and action information; obtaining micro-expression emotional feature expression and a first risk probability; obtaining audio emotional feature expression and a second risk probability; obtaining action feature expression and a third risk probability; obtaining a final risk discriminant score based on the micro-expression emotional feature expression, the first risk probability, the audio emotional feature expression, the second risk probability, the action feature expression, the third risk probability and a pre-trained joint discriminant model; and providing a warning scheme based on the final risk discriminant score.

Description

[0001] statement

[0002] Before acquiring or collecting users' personal information, this invention obtains the user's authorization or consent. The collection, storage, use, processing, transmission, provision, disclosure, and application of users' personal information involved in this invention all comply with relevant laws and regulations, necessary confidentiality measures have been taken, and it does not violate public order and good morals. Technical Field

[0003] This invention relates to the fields of artificial intelligence and big data technologies, and more specifically to a method, apparatus, electronic device, and medium for multimodal information analysis. Background Technology

[0004] An Automated Teller Machine (ATM) is a sophisticated electromechanical device that typically uses magnetic stripe cards or smart cards to provide self-service financial transactions and is an important component of banking services. With the development of electronic finance, ATMs are increasingly widely used in banking financial services.

[0005] Traditional ATM security monitoring systems typically employ an internal and passive response approach. The monitoring equipment in these systems merely stores relevant information and cannot proactively analyze external security risks. Therefore, traditional ATM security monitoring systems lack effective early warning and proactive alert methods for external security threats, making it difficult to guarantee that operators are not subject to external risks such as fraud, coercion, or eavesdropping. Clearly, this lack of a monitoring mechanism increases additional security risks associated with self-service transactions.

[0006] Currently, the application of ATM self-service monitoring information for external security risk analysis is still in its early stages. This involves recording and backing up the user's ATM operations using monitoring equipment. Furthermore, the monitoring records are only used for passive operational security analysis when problems arise or the user disputes their actions. Specifically, the typical steps are: the user contacts staff after experiencing losses; staff retrieves the monitoring records to review the operation and manually check for risks; and if risks are found, the user is informed. Clearly, this method operates after losses have occurred, making it difficult to promptly mitigate the damage already done.

[0007] Furthermore, the existing ATM security inspection systems suffer from inaccurate or imprecise algorithms or an inability to promptly identify abnormal situations, leading to false alarms about normal operations or misjudgments of malicious behavior. This results in legitimate users experiencing unnecessary interference or being denied service. Summary of the Invention

[0008] In view of the above problems, according to a first aspect of the present invention, a multimodal information analysis method based on a joint discriminant network is provided, characterized in that the method includes: acquiring facial information, audio information, and motion information; acquiring micro-expression emotion features based on the facial information and a pre-trained micro-expression emotion analysis model; acquiring a first risk probability based on the micro-expression emotion features; acquiring audio emotion features based on the audio information and a pre-trained audio emotion analysis model; acquiring a second risk probability based on the audio emotion features; acquiring motion features based on the motion information and a pre-trained motion recognition analysis model; acquiring a third risk probability based on the motion features; acquiring a final risk discrimination score based on the micro-expression emotion features, the first risk probability, the audio emotion features, the second risk probability, the motion features, the third risk probability, and a pre-trained joint discriminant model; and providing an early warning scheme based on the final risk discrimination score.

[0009] According to some exemplary embodiments, the acquisition of facial information, audio information, and motion information specifically includes: acquiring facial information based on a first camera, the facial information including first video data of a first person; acquiring audio information based on a first recorder, the audio information including audio data of the first person; and acquiring motion information based on a second camera, the motion information including second video data of a second person.

[0010] According to some exemplary embodiments, the step of obtaining micro-expression emotion features based on the facial information and a pre-trained micro-expression emotion analysis model specifically includes: performing grayscale processing on the first video data to obtain grayscale processed first video data; extracting micro-expression keyframes of the first user based on the grayscale processed first video data; and obtaining micro-expression emotion features based on the micro-expression keyframes and the pre-trained micro-expression emotion analysis model.

[0011] According to some exemplary embodiments, the step of extracting the micro-expression keyframes of the first user specifically includes: using a first frame of a face image with a recognizable face in the grayscale processed first video data as a reference image; using the Euclidean distance method to obtain the i-th Euclidean distance between the i-th frame of the face image with a recognizable face in the grayscale processed first video data and the reference image, where i is greater than 1 and i is a positive integer; obtaining a peak frame based on the i-th Euclidean distance and a peak frame algorithm; and using the peak frame as a keyframe to obtain the micro-expression keyframes of the first user.

[0012] According to some exemplary embodiments, the step of using the Euclidean distance method to obtain the i-th Euclidean distance between the i-th frame face image with a recognizable face in the first grayscale processed video data and the reference image specifically includes: obtaining an origin v and N key feature points, where N is a positive integer; vectorizing the reference image based on the N key feature points to obtain a first vector; vectorizing the i-th frame face image with a recognizable face in the video data of the first person to obtain an i-th vector; calculating the sum of squares of the differences between the first vector and the i-th vector based on the origin v and the N key feature points, taking the square root and absolute value of the sum to obtain the i-th variation difference; and normalizing the i-th variation difference to obtain the i-th Euclidean distance.

[0013] According to some exemplary embodiments, the micro-expression emotion analysis model is trained based on a convolutional neural network.

[0014] According to some exemplary embodiments, obtaining audio emotional feature expressions based on the audio information and a pre-trained audio sentiment analysis model specifically includes: performing audio preprocessing on the audio data of the first person to obtain first audio data; obtaining an audio data feature sequence based on Mel-frequency cepstral coefficients and the first audio data; and obtaining audio emotional feature expressions based on the audio data feature sequence and the pre-trained audio sentiment analysis model.

[0015] According to some exemplary embodiments, the step of obtaining an audio data feature sequence based on Mel-frequency cepstral coefficients and the first audio data specifically includes: pre-emphasizing, framing, and windowing the first audio data to obtain second audio data; obtaining a power spectrum based on the second audio data and the periodogram method; filtering the power spectrum using a Mel-frequency filter bank and calculating the energy value within each filter; taking the logarithm of the energy value within each filter to obtain a logarithmic energy value; performing a discrete cosine transform on the logarithmic energy value to obtain Mel-frequency cepstral coefficient features; and obtaining an audio data feature sequence based on the Mel-frequency cepstral coefficient features.

[0016] According to some exemplary embodiments, the audio sentiment analysis model is trained based on a gated recurrent neural network.

[0017] According to some exemplary embodiments, the action recognition and analysis model pre-trained based on the action information to obtain action feature expression specifically includes: performing grayscale processing on the second video data to obtain grayscale-processed second video data; based on the grayscale-processed second video data, using mean shift algorithm, frame difference method, background difference method, maximum inter-class variance method, human morphology processing, and thinning algorithm to obtain first human skeleton sequence information or second human skeleton sequence information; and based on the first human skeleton sequence information or second human skeleton sequence information and the pre-trained first action recognition and analysis model or second action recognition and analysis model, obtaining first action feature expression.

[0018] According to some exemplary embodiments, obtaining first or second human skeleton sequence information based on the grayscale-processed second video data using a mean-shift algorithm, frame difference method, background subtraction method, maximum inter-class variance method, human morphology processing, and a thinning algorithm specifically includes: obtaining first clustering information based on the grayscale-processed second video data using a mean-shift algorithm; selecting a first or second reference frame based on the first clustering information and obtaining a first foreground using a frame difference method and a background subtraction method; binarizing the first foreground using the maximum inter-class variance method; performing human morphology processing on the binarized first foreground to obtain a first human contour; and thinning the first human contour based on a thinning algorithm to obtain the first human skeleton sequence information.

[0019] According to some exemplary embodiments, the action recognition and analysis model is trained based on a long short-term memory network.

[0020] According to some exemplary embodiments, a first risk probability is obtained based on the micro-expression emotion feature expression and the SoftMax layer; a second risk probability is obtained based on the audio emotion feature expression and the SoftMax layer; and a third risk probability is obtained based on the action feature expression and the SoftMax layer.

[0021] According to some exemplary embodiments, obtaining a final risk discrimination score based on the micro-expression emotion feature expression, the first risk probability, the audio emotion feature expression, the second risk probability, the action feature expression, the third risk probability, and a pre-trained joint discrimination model specifically includes: performing feature fusion using linear connections based on the micro-expression emotion feature expression, the action feature expression, and the audio emotion feature expression to obtain a fused feature expression; obtaining a potential fused feature expression based on the fused feature expression and a fully connected layer network; obtaining a joint risk probability based on the potential fused feature expression and a SoftMax layer; and performing a weighted calculation on the first risk probability, the second risk probability, the third risk probability, and the joint risk probability to obtain the final risk discrimination score.

[0022] According to some exemplary embodiments, the joint discrimination model is trained based on a neural network.

[0023] According to some exemplary embodiments, the early warning scheme includes intervention by security personnel for observation or voice broadcast prompts.

[0024] According to a second aspect of the present invention, a device for multimodal information analysis based on a joint discriminant network is proposed. The device includes: an information acquisition module for acquiring facial information, audio information, and action information; a micro-expression emotion feature acquisition module for acquiring micro-expression emotion features based on the facial information and a pre-trained micro-expression emotion analysis model; a first risk probability acquisition module for acquiring a first risk probability based on the micro-expression emotion features; an audio emotion feature acquisition module for acquiring audio emotion features based on the audio information and a pre-trained audio emotion analysis model; and a second risk probability acquisition module for acquiring audio emotion features based on the audio information and a pre-trained audio emotion analysis model. The system comprises: a frequency-based emotion feature expression module for obtaining a second risk probability; an action feature expression acquisition module for obtaining action feature expressions based on a pre-trained action recognition and analysis model of the action information; a third risk probability acquisition module for obtaining a third risk probability based on the action feature expressions; a final risk discrimination score acquisition module for obtaining a final risk discrimination score based on the micro-expression emotion feature expression, the first risk probability, the audio emotion feature expression, the second risk probability, the action feature expression, the third risk probability, and a pre-trained joint discrimination model; and an early warning scheme provision module for providing an early warning scheme based on the final risk discrimination score.

[0025] According to some exemplary embodiments, the micro-expression emotion feature expression acquisition module includes: a first video data acquisition unit, a micro-expression keyframe extraction module, and a micro-expression emotion feature expression acquisition unit.

[0026] According to some exemplary embodiments, the first video data acquisition unit can be used to perform grayscale processing on the first video data to obtain grayscale processed first video data.

[0027] According to some exemplary embodiments, the micro-expression keyframe extraction module can be used to extract the micro-expression keyframes of the first user based on the grayscale processed first video data.

[0028] According to some exemplary embodiments, the micro-expression emotion feature acquisition unit can be used to acquire micro-expression emotion features based on the micro-expression keyframes and the pre-trained micro-expression emotion analysis model.

[0029] According to some exemplary embodiments, the micro-expression keyframe extraction module includes: a reference image acquisition unit, an i-th Euclidean distance acquisition module, a peak frame acquisition unit, and a micro-expression keyframe acquisition unit.

[0030] According to some exemplary embodiments, the reference image acquisition unit can be used to take a first frame face image with a recognizable face in the grayscale processed first video data as a reference image.

[0031] According to some exemplary embodiments, the i-th Euclidean distance acquisition module can be used to obtain the i-th Euclidean distance between the i-th frame face image with recognizable face in the grayscale processed first video data and the reference image using the Euclidean distance method, where i is greater than 1 and i is a positive integer.

[0032] According to some exemplary embodiments, the peak frame acquisition unit can be used to obtain a peak frame based on the i-th Euclidean distance and the peak frame algorithm.

[0033] According to some exemplary embodiments, the micro-expression keyframe acquisition unit can be used to obtain the micro-expression keyframes of the first user by using the peak frame as a keyframe.

[0034] According to some exemplary embodiments, the i-th Euclidean distance acquisition module includes: a feature point acquisition unit, a vector acquisition unit, and an i-th Euclidean distance acquisition unit.

[0035] According to some exemplary embodiments, the feature point acquisition unit can be used to acquire the origin v and N key feature points, where N is a positive integer.

[0036] According to some exemplary embodiments, the vector acquisition unit can be used to vectorize the reference image based on the N key feature points to obtain a first vector, and to vectorize the i-th frame face image with a recognizable face in the video data of the first person to obtain the i-th vector.

[0037] According to some exemplary embodiments, the i-th Euclidean distance acquisition unit can be used to calculate the sum of squares of the differences between the first vector and the i-th vector based on the origin v and the N key feature points, and then take the square root and absolute value of the sum to obtain the i-th variation difference, and normalize the i-th variation difference to obtain the i-th Euclidean distance.

[0038] According to some exemplary embodiments, the audio emotion feature expression acquisition module includes: a first audio data acquisition unit, an audio data feature sequence acquisition module, and an audio emotion feature expression acquisition unit.

[0039] According to some exemplary embodiments, the first audio data acquisition unit can be used to perform audio preprocessing on the audio data of the first person to obtain first audio data.

[0040] According to some exemplary embodiments, the audio data feature sequence acquisition module can be used to acquire an audio data feature sequence based on Mel-frequency cepstral coefficients and the first audio data.

[0041] According to some exemplary embodiments, the audio emotion feature expression acquisition unit can be used to acquire audio emotion feature expression based on the audio data feature sequence and the pre-trained audio emotion analysis model.

[0042] According to some exemplary embodiments, the audio data feature sequence acquisition module includes: a second audio data acquisition unit, a power spectrum acquisition unit, an energy value acquisition unit, a logarithmic energy value acquisition unit, a Mel-Cepstral coefficient feature acquisition unit, and an audio data feature sequence acquisition unit.

[0043] According to some exemplary embodiments, the second audio data acquisition unit can be used to pre-emphasize, frame, and window the first audio data to obtain the second audio data.

[0044] According to some exemplary embodiments, the power spectrum acquisition unit can be used to acquire the power spectrum based on the second audio data and the periodogram method.

[0045] According to some exemplary embodiments, the energy value acquisition unit can be used to filter the power spectrum using a Mel filter bank and calculate the energy value within each filter.

[0046] According to some exemplary embodiments, the logarithmic energy value acquisition unit can be used to take the logarithm of the energy value in each filter to obtain the logarithmic energy value.

[0047] According to some exemplary embodiments, the Mel-Cepstral Coefficient Feature Acquisition Unit can be used to perform a discrete cosine transform on the logarithmic energy value to obtain Mel-Cepstral Coefficient Features.

[0048] According to some exemplary embodiments, the audio data feature sequence acquisition unit can be used to acquire an audio data feature sequence based on the Mel-Cepstral Coefficients feature.

[0049] According to some exemplary embodiments, the motion feature expression acquisition module includes: a second video data acquisition unit, a human skeleton sequence information acquisition module, and a motion feature expression acquisition unit.

[0050] According to some exemplary embodiments, the second video data acquisition unit can be used to perform grayscale processing on the second video data to obtain grayscale processed second video data.

[0051] According to some exemplary embodiments, the human skeleton sequence information acquisition module can be used to obtain human skeleton sequence information based on the grayscale processed second video data, using mean shift algorithm, frame difference method, background difference method, maximum inter-class variance method, human morphology processing and thinning algorithm.

[0052] According to some exemplary embodiments, the action feature expression acquisition unit can be used to acquire action feature expressions based on the human skeleton sequence information and the pre-trained action recognition analysis model.

[0053] According to some exemplary embodiments, the human skeleton sequence information acquisition module includes a first clustering information acquisition unit, a first foreground acquisition unit, a binarization unit, a human contour acquisition unit, and a human skeleton sequence information acquisition unit.

[0054] According to some exemplary embodiments, the first clustering information acquisition unit acquires first clustering information based on the grayscale processed second video data using a mean shift algorithm.

[0055] According to some exemplary embodiments, the first foreground acquisition unit can be used to select a first reference frame or a second reference frame based on the first clustering information, and acquire the first foreground using the frame difference method and the background difference method.

[0056] According to some exemplary embodiments, the binarization unit can be used to binarize the first foreground using the maximum inter-class variance method.

[0057] According to some exemplary embodiments, the human body contour acquisition unit can be used to perform human body morphology processing on the binarized first foreground to obtain a human body contour.

[0058] According to some exemplary embodiments, the human skeleton sequence information acquisition unit refines the first human outline based on a refinement algorithm to obtain the human skeleton sequence information.

[0059] According to some exemplary embodiments, the final risk discrimination score acquisition module includes: a fusion feature expression acquisition unit, a potential fusion feature expression acquisition unit, a joint risk probability acquisition unit, and a final risk discrimination score acquisition unit.

[0060] According to some exemplary embodiments, the fusion feature expression acquisition unit can be used to perform feature fusion based on the micro-expression emotion expression, the action feature expression and the audio emotion expression, using linear connections to obtain the fusion feature expression.

[0061] According to some exemplary embodiments, the potential fusion feature expression acquisition unit can be used to obtain potential fusion feature expressions based on the fusion feature expressions and the fully connected layer network.

[0062] According to some exemplary embodiments, the joint risk probability acquisition unit can be used to obtain the joint risk probability based on the potential fusion feature expression and the SoftMax layer.

[0063] According to some exemplary embodiments, the final risk discrimination score acquisition unit can be used to perform weighted calculation on the first risk probability, the second risk probability, the third risk probability and the joint risk probability to obtain the final risk discrimination score.

[0064] According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method as described above.

[0065] According to a fourth aspect of the present invention, a computer-readable storage medium is provided having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method described above.

[0066] According to a fifth aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0067] The above one or more embodiments have the following advantages or beneficial effects: The security early warning method based on multimodal information of joint discriminant network provided by the present invention can combine different information sources to more accurately determine whether there are potential security risks or abnormal behaviors, thereby improving the accuracy of early warning; at the same time, traditional ATM early warning methods usually only rely on single-modal data, while the method provided by the present invention can analyze from multiple dimensions, thereby providing more accurate judgment and reducing the possibility of false alarms and misjudgments.

[0068] Furthermore, the security early warning method of the present invention can analyze the usage of ATMs in real time. Once abnormal behavior or potential security risks are detected, it can immediately issue an early warning and take corresponding measures, thereby improving the security of ATM operations and reducing customer property losses. Attached Figure Description

[0069] The above-described features, other objects, and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0070] Figure 1 The illustration shows an application scenario of the method, apparatus, device, and medium for multimodal information analysis based on a joint discriminant network according to embodiments of the present invention.

[0071] Figure 2 A flowchart illustrating a method for multimodal information analysis based on a joint discriminant network according to an embodiment of the present invention is shown.

[0072] Figure 3 A flowchart illustrating a method for acquiring micro-expression emotional features according to an embodiment of the present invention is shown.

[0073] Figure 4 A flowchart illustrating a method for keyframe extraction according to an embodiment of the present invention is shown schematically.

[0074] Figure 5 A flowchart illustrating a method for obtaining the i-th Euclidean distance according to an embodiment of the present invention is shown schematically.

[0075] Figure 6 The diagram illustrates the network structure of the micro-expression emotion analysis model trained based on a convolutional neural network according to an embodiment of the present invention.

[0076] Figure 7 A flowchart illustrating a method for obtaining audio emotional features according to an embodiment of the present invention is shown.

[0077] Figure 8 A flowchart illustrating a method for a sequence of audio data features according to an embodiment of the present invention is shown schematically.

[0078] Figure 9 A flowchart illustrating a method for obtaining action feature representations according to an embodiment of the present invention is shown.

[0079] Figure 10 A flowchart illustrating a method for obtaining human skeleton sequence information according to an embodiment of the present invention is shown.

[0080] Figure 11A flowchart illustrating a method for obtaining a final risk assessment score according to an embodiment of the present invention is shown.

[0081] Figure 12 A schematic block diagram of a device for multimodal information analysis based on a joint discriminant network according to an embodiment of the present invention is shown.

[0082] Figure 13 A block diagram of an electronic device suitable for implementing a multimodal information analysis method based on a joint discriminant network according to an embodiment of the present invention is shown schematically. Detailed Implementation

[0083] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0084] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0085] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0086] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0087] In the technical solution of this invention, the acquisition, storage and application of user personal information all comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0088] First, the technical terms used in this article are explained and clarified as follows.

[0089] Multimodal: Obtaining multiple types of information from data from different sources, such as video and audio.

[0090] Representation learning: learning the meaning of single or group of signal features, transforming raw data into a form that can be effectively developed by machine learning.

[0091] Emotion recognition: By analyzing and processing signal features through algorithms, the behavior of the signal sender can be used to determine the emotional state of the signal sender and the emotion or emotional tendency expressed by the signal sender.

[0092] Keyframes: In video coding, keyframes are video frames that show significant changes in facial expressions or behavioral characteristics. Keyframes are the foundational frames of a video sequence; they are encoded independently, without relying on other frames, providing random access, fault tolerance, and ease of editing.

[0093] Peak frames: Video frames that show the peak of changes in facial expressions or behavioral characteristics.

[0094] Euclidean distance: measures the absolute distance between two points in a multidimensional space. Euclidean distance can calculate the relative positions and relationships between feature points, such as the distance between the two eyes, the distance from the eyes to the nose, etc., thereby improving the accuracy of facial recognition and analysis.

[0095] Key feature points (KNPs) are specific locations or feature points in a facial image that are of significant importance, such as the center of the eyes, the highest point of the eyebrows, the tip of the nose, and the corners of the lips. The position and shape of these KNPs can be used to represent the geometric structure and features of the face, thus enabling face comparison. KNPs are typically obtained through facial landmark detection algorithms. These algorithms use machine learning or deep learning methods to learn the locations of facial feature points based on large amounts of training data. Commonly used landmark detection algorithms include Dlib, OpenCV, and MTCNN.

[0096] Convolutional Neural Network (CNN): A deep neural network that includes convolutional computations. Its main network structure consists of convolutional layers and pooling layers.

[0097] Long Short-Term Memory (LSTM) networks: a type of neural network with the ability to process temporal information. Its main structure consists of an input gate, an output gate, and a forget gate.

[0098] Mean-shift human tracking algorithm: a non-parametric method based on density gradient ascent, which finds the target position through iterative calculation to achieve target tracking.

[0099] Frame difference method: A method to obtain the contour of a moving target by performing a difference operation on two adjacent frames in a video sequence.

[0100] Background subtraction method: a method for obtaining the contour of a moving target by comparing the current frame in a video sequence with a background reference model.

[0101] Background: Unchanging parts of a video or image, typically areas without motion or unrelated to motion.

[0102] Foreground: A moving or important part of a video or image that stands out from the background.

[0103] Otsu's method is a simple and effective image segmentation algorithm that determines the image binarization threshold by maximizing the inter-class variance. It can divide a grayscale image into two or more parts to obtain the foreground and background.

[0104] Morphological methods: Image processing techniques used to analyze and manipulate the shape, structure, and spatial relationships in images. Based on mathematical morphology theory, they alter the shape and features of an image through structuring elements and a series of morphological operations. Common morphological processing methods include dilation, erosion, filtering, and opening / closing operations.

[0105] Thinning algorithms: a general term for a class of methods that calculate the basic shape features of an image and remove redundant data. They include layer-by-layer peeling thinning and distance transform-based thinning. Among them, distance transform-based thinning algorithms determine which pixels to retain during the thinning process by calculating the distance from a pixel to its nearest boundary pixel.

[0106] Mel-scale Frequency Cepstral Coefficients (MFCC): A feature extraction method widely used in speech signal processing and speech recognition. It is designed based on the human auditory system's perception of sound frequencies and extracts cepstral parameters in the Mel-scale frequency domain. The Mel scale describes the nonlinear characteristics of human ear frequencies.

[0107] Gated Recurrent Unit (GRU): A variant of LSTM network with a simpler structure, mainly consisting of update gates and reset gates.

[0108] SoftMax layer: A commonly used output layer type in CNNs, used to transform the model's output into a probability distribution. The input to a SoftMax layer is typically a vector, and the output is a probability distribution, where each element represents the probability of the corresponding class.

[0109] Existing ATMs, due to their passive inspection and delayed early warning characteristics, cannot proactively provide real-time security warnings for external security risks posed by operators, potentially increasing the difficulty of recovering losses caused by such risks. Specifically, most ATMs rely on operators actively reporting problems or anomalies. This passive inspection mechanism leads to a delay in problem detection and resolution, and relying on user feedback may result in time delays and escalation of losses. Some ATMs use log-based inspection and early warning methods, analyzing ATM log records to check for abnormal behavior. However, this method usually requires offline analysis and post-processing, and cannot provide real-time monitoring and early warning, resulting in significant delays. Existing ATMs typically lack real-time monitoring mechanisms, failing to monitor various ATM states and operations in real time. This means that abnormal behavior may be detected late, limiting timely responses to security issues. Furthermore, due to limited human resources, continuous manual monitoring of every ATM is not possible.

[0110] Based on this, embodiments of the present invention provide a method for multimodal information analysis based on a joint discriminant network. The method includes: acquiring facial information, audio information, and action information; acquiring micro-expression emotion features based on the facial information and a pre-trained micro-expression emotion analysis model; acquiring a first risk probability based on the micro-expression emotion features; acquiring audio emotion features based on the audio information and a pre-trained audio emotion analysis model; acquiring a second risk probability based on the audio emotion features; acquiring action feature expressions based on the action information and a pre-trained action recognition analysis model; acquiring a third risk probability based on the action feature expressions; acquiring a final risk discrimination score based on the micro-expression emotion features, the first risk probability, the audio emotion features, the second risk probability, the action feature expressions, the third risk probability, and a pre-trained joint discriminant model; and providing an early warning scheme based on the final risk discrimination score. In the method according to embodiments of the present invention, multiple modal information such as sound, images, and videos around the ATM are collected from the moment the user inserts their card into the ATM, using segmented incremental transmission features. By acquiring multimodal information and analyzing it through a joint discriminant network, real-time early warnings for ATM operations can be provided. At the same time, by using multimodal information for representation learning, suspicious external behaviors and states can be analyzed to provide proactive early warnings and prompts for safe operation.

[0111] It should be noted that the method, apparatus, device, and medium for multimodal information analysis based on joint discriminant networks as defined in this invention can be used in the fields of big data technology and artificial intelligence technology, as well as in the financial field, and can also be used in a variety of fields other than big data technology, artificial intelligence technology, and the financial field. The application fields of the method, apparatus, device, and medium for multimodal information analysis based on joint discriminant networks provided in the embodiments of this invention are not limited.

[0112] In the technical solution of this invention, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information all comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0113] In the technical solution of the present invention, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0114] Figure 1 The illustration shows an application scenario of the method, apparatus, device, and medium for multimodal information analysis based on a joint discriminant network according to embodiments of the present invention.

[0115] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal device 101, network 102, and servers 103 and 104. Network 102 is used as a medium to provide a communication link between terminal device 101 and servers 103 and 104. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0116] Users can use terminal device 101 to interact with servers 103 and 104 via network 102 to receive or send messages, etc.

[0117] Terminal device 101 is an ATM machine. It is understood that multiple ATM machines can also be used as terminal devices.

[0118] Servers 103 and 104 can be used to analyze multimodal information from terminal device 101 and perform security probing, respectively; they can also be servers providing various services, such as a backend management server that supports operations performed by users using terminal device 101 (for example only). The backend management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0119] It should be noted that the multimodal information analysis method based on a joint discriminant network provided in this embodiment of the invention can generally be executed by server 103. Correspondingly, the apparatus for multimodal information analysis based on a joint discriminant network provided in this embodiment of the invention can generally be located in server 103. The multimodal information analysis method based on a joint discriminant network provided in this embodiment of the invention can also be executed by a server or server cluster that is different from server 103 and capable of communicating with terminal device 101 and / or server 103. Correspondingly, the apparatus for multimodal information analysis based on a joint discriminant network provided in this embodiment of the invention can also be located in a server or server cluster that is different from server 103 and capable of communicating with terminal device 101 and / or server 103.

[0120] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0121] Figure 2 A flowchart illustrating a method for multimodal information analysis based on a joint discriminant network according to an embodiment of the present invention is shown.

[0122] like Figure 2 As shown, the method 200 for multimodal information analysis based on a joint discriminant network in this embodiment may include operations S210 to S290.

[0123] During operation S210, facial information, audio information, and motion information are acquired.

[0124] In an embodiment of the present invention, facial information is acquired based on a first camera, the facial information including first video data of a first person; audio information is acquired based on a first recorder, the audio information including audio data of the first person; and motion information is acquired based on a second camera, the motion information including second video data of a second person.

[0125] Specifically, when the first person inserts their card into the ATM, the camera and recording equipment begin transmitting audio and video data. The first camera in front of the ATM captures the first person's facial information, the first recorder in front of the ATM captures the operator's audio information, and the second camera in the self-service area captures the second person's movement information. The specific locations of the equipment can be found in [reference needed]. Figure 1 The settings are shown in the image.

[0126] In embodiments of the present invention, the first person refers to the operator of the ATM, and the second person refers to the person waiting in the self-service area. Acquiring multimodal information, including the operator's facial information, audio information, and the waiting person's action information, can help determine whether the operator's actions involve external risks such as being deceived, coerced, or spied on.

[0127] It should be noted that the acquisition of multimodal information listed here is merely exemplary and is not intended to limit the scope of multimodal information included in the embodiments of the present invention. That is, the multimodal information in the embodiments of the present invention may also include other content. For example, in the case where there is no second person in the self-service room, the facial information and audio information of the first person can be directly obtained for subsequent analysis to avoid the first person being in danger of telephone fraud or telephone threats.

[0128] It should be noted that before obtaining a user's facial, audio, or motion information, prior confirmation or authorization from the user must be obtained. The user's wishes must be respected, and any operation that obtains personal information without the user's permission must be avoided. Specifically, clear text prompts can be displayed on the ATM machine requiring the user to perform a confirmation or authorization operation; alternatively, audio prompts reminding the user of authorization can be played, allowing the user to confirm or authorize through auditory feedback.

[0129] In operation S220, based on the facial information and the pre-trained micro-expression emotion analysis model, the micro-expression emotion features are obtained.

[0130] Figure 3 A flowchart illustrating a method for acquiring micro-expression emotional features according to an embodiment of the present invention is shown.

[0131] In an embodiment of the present invention, the method for obtaining micro-expression emotion features includes operations S310 to S330.

[0132] In operation S310, the first video data is processed into grayscale to obtain grayscale processed first video data.

[0133] In embodiments of the present invention, to simplify image information, reduce computational load, and improve algorithm processing speed, grayscale processing is required before processing image data (including all image and video data in embodiments of the present invention). The grayscale processing process is as follows: converting each frame of color image into a grayscale image by converting the color value of each pixel into a brightness value.

[0134] In operation S320, based on the first video data after grayscale processing, the micro-expression keyframes of the first user are extracted.

[0135] In embodiments of the present invention, facial expression feature extraction is a crucial part of acquiring micro-expression emotional features. Key feature points of facial expression changes are highly correlated with the emotional expression of the first person. Extracting key features that can summarize the changes in the first person's facial expressions can improve the network's recognition performance. In facial expression feature extraction, since videos typically contain multiple consecutive frames, some of which may be redundant or lack facial information, key frames, as representative or important frames in the video sequence, can be used to extract facial images from the video. Therefore, by selecting key frames, the amount of data processed can be reduced, and important facial information can usually be captured.

[0136] In embodiments of the present invention, keyframes are extracted by selecting peak frames. Peak frames constantly monitor changes in key facial feature points and can be used to extract keyframes; furthermore, selecting peak frames can also achieve local focus, thereby reducing the influence of the background on feature extraction.

[0137] Figure 4 A flowchart illustrating a method for keyframe extraction according to an embodiment of the present invention is shown schematically.

[0138] In an embodiment of the present invention, the main process of keyframe extraction includes operations S410 to S440.

[0139] In operation S410, the first frame face image with a recognizable face in the first grayscale processed video data is used as the reference image.

[0140] In operation S420, the Euclidean distance method is used to obtain the i-th Euclidean distance between the i-th frame face image with recognizable face in the first video data after grayscale processing and the reference image, where i is greater than 1 and i is a positive integer.

[0141] It should be noted that the first frame face image and the i-th frame face image in the embodiments of the present invention are images with recognizable faces. Cases without face images or with unrecognizable faces are not considered. i is greater than 1, which means that the i-th frame face image is a subsequent image of the first frame face image.

[0142] Figure 5 A flowchart illustrating a method for obtaining the i-th Euclidean distance according to an embodiment of the present invention is shown schematically.

[0143] In an embodiment of the present invention, the main process of obtaining the i-th Euclidean distance includes operations S510 to S530.

[0144] In operation S510, the origin v and N key feature points are obtained, where N is a positive integer.

[0145] In operation S520, based on the N key feature points, the reference image is vectorized to obtain a first vector, and the i-th frame face image with a recognizable face in the video data of the first person is vectorized to obtain the i-th vector.

[0146] In an embodiment of the present invention, the reference image and the i-th frame face image are both two-dimensional image data extracted from the first video data. The vectorization process can extract feature vectors based on key feature points, and the feature vectors can be represented by coordinates.

[0147] In operation S530, based on the origin v and the N key feature points, the sum of squares of the differences between the first vector and the i-th vector is calculated, and the square root and absolute value are taken to obtain the i-th variation difference. The i-th variation difference is then normalized to obtain the i-th Euclidean distance.

[0148] Specifically, the formula for calculating the i-th variation difference is as follows:

[0149]

[0150] Among them, L i Let be the i-th variation difference. Based on this formula, we can determine the i-th variation difference between the i-th frame face image with N key feature points and the reference image.

[0151] Return to reference Figure 4 In operation S430, the peak frame is obtained based on the i-th Euclidean distance and peak frame algorithm.

[0152] In embodiments of the present invention, the peak frame can be determined based on the following formula:

[0153] Idx = argmax(Z) i (2)

[0154] Among them, Z i Let Idx represent the i-th Euclidean distance obtained by normalizing the i-th variation difference. i The frame corresponding to the element with the largest Euclidean distance value, i.e., Idx, is considered the peak frame.

[0155] In operation S440, the peak frame is used as a key frame to obtain the micro-expression key frame of the first user.

[0156] In embodiments of the present invention, by acquiring the micro-expression keyframes, the amount of data processed can be reduced, and important information about the face can usually be captured.

[0157] Return to reference Figure 3 In operation S330, based on the micro-expression keyframes and the pre-trained micro-expression emotion analysis model, the expression of micro-expression emotion features is obtained.

[0158] In an embodiment of the present invention, the micro-expression emotion analysis model is trained based on a convolutional neural network.

[0159] It should be noted that convolutional neural networks (CNNs) can be used for feature extraction and classification in facial micro-expression recognition in this embodiment of the invention. Since micro-expressions are typically subtle and fleeting, their features are more nuanced compared to traditional facial expressions. CNNs can learn and extract local and global features of an image through multiple convolutional and pooling layers, thereby classifying and recognizing micro-expressions. By training on a large number of samples, CNNs can learn discriminative features of micro-expressions, enabling accurate recognition on new micro-expression samples.

[0160] In an embodiment of the present invention, pre-training a micro-expression sentiment analysis model specifically includes: acquiring a first historical video dataset, including historical facial information of a first person; obtaining historical keyframes based on the first historical video dataset, and preprocessing the historical keyframes to obtain preprocessed historical keyframes, i.e., processing them into a data type that the model can recognize; annotating the historical keyframes to obtain a micro-expression training dataset and a micro-expression test dataset; constructing a convolutional neural network model, including P convolutional blocks and Q fully connected layers, where P and Q are preset values; training the convolutional neural network model based on the micro-expression training dataset to obtain an intermediate micro-expression sentiment analysis model; and evaluating the intermediate micro-expression sentiment analysis model based on the micro-expression test dataset to finally obtain the micro-expression sentiment analysis model.

[0161] Figure 6 The diagram illustrates the network structure of the micro-expression emotion analysis model trained based on a convolutional neural network according to an embodiment of the present invention.

[0162] like Figure 6 As shown, in an embodiment of the present invention, a baseline convolutional neural network is used for emotion recognition. This network consists of 5 convolutional blocks and 3 fully connected layers. Each convolutional block contains 2 consecutive convolutional layers and one max-pooling layer. Each convolutional block uses a 3×3 convolutional kernel and a 2×2 max-pooling layer. The number of channels in each convolutional block is 64, 128, 256, 512, and 512, respectively.

[0163] According to embodiments of the present invention, convolutional neural networks can learn and extract local and global features of images through multiple convolutional layers and pooling layers, thereby classifying and recognizing micro-expressions.

[0164] It should be noted that the process of obtaining micro-expression emotional features through facial information complies with relevant laws and regulations, takes necessary confidentiality measures, and does not violate public order and good morals.

[0165] Return to reference Figure 2 In operation S230, the first risk probability is obtained based on the micro-expression emotional features.

[0166] In an embodiment of the present invention, a first risk probability is obtained based on the micro-expression emotion feature expression and the SoftMax layer.

[0167] In embodiments of the present invention, the SoftMax layer is used to transform the model's output into a probability distribution. Specifically, as shown below... Figure 6 As shown, to add this output layer, a fully connected layer can be added before the SoftMax layer. The output channels of the fully connected layer are 4096, 2048, and N, respectively. The output of this fully connected layer serves as the input to the SoftMax layer, which converts each feature into a corresponding probability value to determine the probability of risk. The SoftMax layer transforms the raw output values ​​into a probability distribution, making the output results more intuitive and easier to apply later.

[0168] In operation S240, based on the audio information and the pre-trained audio sentiment analysis model, the audio sentiment feature expression is obtained.

[0169] Figure 7 A flowchart illustrating a method for obtaining audio emotional features according to an embodiment of the present invention is shown.

[0170] In embodiments of the present invention, the main process of obtaining audio emotional features includes operations S710 to S730.

[0171] In operation S710, the audio data of the first person is preprocessed to obtain the first audio data.

[0172] In embodiments of the present invention, the audio preprocessing operations may include noise reduction, dereverberation, equalization, normalization and segmentation, localization, etc., to improve audio quality, reduce interference and enhance signal characteristics, so as to facilitate subsequent operations.

[0173] In operation S720, an audio data feature sequence is obtained based on Mel-frequency cepstral coefficients and the first audio data.

[0174] Figure 8 A flowchart illustrating a method for a sequence of audio data features according to an embodiment of the present invention is shown schematically.

[0175] In an embodiment of the present invention, the main process of obtaining the audio data feature sequence includes operations S810 to S860.

[0176] In operation S810, the first audio data is pre-emphasized, framed, and windowed to obtain the second audio data.

[0177] It's important to note that pre-emphasis, framing, and windowing are common techniques in digital signal processing, typically used for audio signal processing and analysis. Pre-emphasis is a filtering technique used to increase the amplitude of high-frequency components in an audio signal to reduce signal loss during transmission. This process uses a high-pass filter to attenuate low-frequency components, thereby increasing the amplitude of high-frequency components. Framing is the process of dividing an audio signal into short time segments. This can be done by applying a window function to the audio signal, then dividing the windowed signal into multiple frames. This process usually uses overlap, adding the overlapping portions between two adjacent frames. The purpose of this process is to divide a long signal into short blocks for subsequent processing. Windowing is a signal processing technique typically used to reduce frequency leakage between discrete signals. The windowing process uses a window function to multiply the framed signal by the window function. The windowed signal can then be analyzed and processed using Fourier transform or other frequency domain processing techniques.

[0178] In operation S820, the power spectrum is obtained based on the second audio data and the periodogram method.

[0179] In operation S830, the power spectrum is filtered using a Mel filter bank, and the energy value within each filter is calculated.

[0180] In operation S840, the logarithm of the energy value within each filter is taken to obtain the logarithmic energy value.

[0181] In operation S850, the logarithmic energy value is subjected to discrete cosine transform to obtain the Mel-Cepstral coefficient characteristics.

[0182] In operation S860, an audio data feature sequence is obtained based on the Mel-Cepstral Coefficients feature.

[0183] In embodiments of the present invention, the audio data feature sequence is selected from the Mel-frequency cepstral coefficient features. Specifically, low-order Mel-frequency cepstral coefficients, such as those from order 2 to 13, are generally selected.

[0184] In embodiments of the present invention, the Mel-Cepstral Coefficients simulate the perceptual characteristics of the human ear to the sound spectrum by using Mel filter banks and logarithmic compression; extract the main features of language, the Mel-Cepstral Coefficients mainly capture the resonance features of the language signal, and are relatively insensitive to the speaker's voice features and speech changes; the feature dimension is low, and fewer discrete cosine transform coefficients are selected as Mel-Cepstral Coefficient features, which can greatly reduce the feature dimension and improve computational efficiency while retaining the main information.

[0185] Return to reference Figure 7 In operation S730, based on the audio data feature sequence and the pre-trained audio sentiment analysis model, the audio sentiment feature expression is obtained.

[0186] In an embodiment of the present invention, the audio sentiment analysis model is trained based on a gated recurrent neural network.

[0187] In embodiments of the present invention, the gated recurrent neural network is an improved temporal network proposed to address the gradient problem caused by long-term dependencies in RNNs. Furthermore, compared to long short-term memory networks, it has a simpler structure, reducing structural complexity. A gated recurrent neural network typically includes two structures: a reset gate and an update gate. Its calculation formula is as follows:

[0188] z t =σ(W z x t +U z h t-1 (3)

[0189] r t =σ(W r x t +U r h t-1 (4)

[0190]

[0191]

[0192] Where z is the update gate, r is the reset gate, and h represents the hidden state. It is a candidate hidden state, which is only related to the input x and the hidden state h of the previous time series. t-1 Related; t is the time point, σ is the activation function, · represents element-wise multiplication, W and U are the weight matrices to be learned, and x is the input feature.

[0193] In an embodiment of the present invention, pre-training an audio sentiment analysis model specifically includes: acquiring a historical audio dataset, including historical audio information of a first person; acquiring a historical audio feature sequence based on Mel-frequency cepstral coefficients and the historical audio dataset; labeling the historical audio feature sequence to obtain an audio sentiment training dataset and an audio sentiment test dataset; constructing a gated recurrent neural network model; training the gated recurrent neural network model based on the audio sentiment training dataset to obtain an intermediate audio sentiment analysis model; and evaluating the intermediate audio sentiment analysis model based on the audio sentiment test dataset to finally obtain the audio sentiment analysis model.

[0194] Return to reference Figure 2 In operation S250, a second risk probability is obtained based on the emotional features expressed in the audio.

[0195] In an embodiment of the present invention, a second risk probability is obtained based on the audio emotional feature expression and the SoftMax layer.

[0196] In an embodiment of the present invention, the SoftMax layer is used to convert the output of the audio sentiment analysis model into a probability distribution and determine a second risk probability indicating the presence of risk. The operation of obtaining the second risk probability based on the SoftMax layer is similar to obtaining the first risk probability, and will not be described again here.

[0197] It should be noted that the process of obtaining audio emotional features through audio information complies with relevant laws and regulations, takes necessary confidentiality measures, and does not violate public order and good morals.

[0198] In operation S260, based on the action information and the pre-trained action recognition and analysis model, action feature representation is obtained.

[0199] Figure 9 A flowchart illustrating a method for obtaining action feature representations according to an embodiment of the present invention is shown.

[0200] In embodiments of the present invention, the main process of obtaining action feature representation includes operations S910 to S930.

[0201] In operation S910, the second video data is processed into grayscale to obtain grayscale processed second video data.

[0202] In operation S920, based on the second video data after grayscale processing, human skeleton sequence information is obtained by using mean shift algorithm, frame difference method, background difference method, maximum inter-class variance method, human morphology processing and thinning algorithm.

[0203] Figure 10A flowchart illustrating a method for obtaining human skeleton sequence information according to an embodiment of the present invention is shown.

[0204] In an embodiment of the present invention, the main process for obtaining human skeleton sequence information includes operations S1010 to S1050.

[0205] In operation S1010, based on the second video data after grayscale processing, the first clustering information is obtained using the mean shift algorithm.

[0206] It should be noted that the mean shift algorithm is a method for finding stable statistical local values ​​within the density distribution of a set of data. Assume that for a d-dimensional space R... d For a sample at a point x, the multivariate kernel density estimation formula for the window radius h is as follows:

[0207]

[0208] Where K(x) is the kernel function, and the contour of the kernel function is a function k from [0, ∞) to R, then K(x) = k(||x|| 2 If k(x) is differentiable almost everywhere in [0, ∞), then we can define a function g(x) = -k′(x). This leads to the new kernel function G(x) = λg(||x||). 2 ), where λ is the normalization factor, and the multivariate kernel density estimate of this kernel is:

[0209]

[0210] Therefore, the relevant kernel density gradient estimate can be obtained as follows:

[0211]

[0212] According to gradient estimation, it always points in the direction of maximum density increase.

[0213] In an embodiment of the present invention, the mean shift algorithm comprises the following steps: selecting a search window, calculating the centroid, and setting the window center at the centroid; then, iteratively calculating the centroid and resetting the window center until the window position is fixed. Specifically, the tracking center is positioned on the movement of the second person, eliminating the influence of the first person, and the tracking result is used as a human motion image frame for skeleton information extraction.

[0214] In operation S1020, based on the first clustering information, a first reference frame or a second reference frame is selected, and the first foreground is obtained using the frame difference method and the background difference method.

[0215] In operation S1030, the first foreground is binarized using the Otsu's method.

[0216] In operation S1040, the binarized first foreground is subjected to human morphology processing to obtain the human body outline.

[0217] In operation S1050, based on the thinning algorithm, the first human body contour is thinned to obtain the human skeleton sequence information.

[0218] In embodiments of the present invention, the background, as opposed to the first foreground, refers to the unchanging portion of a video or image. Therefore, for establishing a background image, an image frame where no one is operating can be used as the background frame. For each frame F of the video transmission containing the human body... i (i = 1, 2, ..., t), where t represents the corresponding frame value. Therefore, the difference DW between adjacent frames is calculated using the following formula:

[0219] DW i =|F i (x, y) - F i-1 (x, y)| (10)

[0220] Where (x, y) is the target point.

[0221] In an embodiment of the present invention, a threshold T is set to determine whether a motion change has occurred in the corresponding frame. If DW ≥ T and falls within the tracking window, it indicates that the target frame has moved; otherwise, no motion has occurred. Further, motion changes in adjacent image frames are obtained by binarizing the difference image. For the detection of moving human bodies, embodiments of the present invention calculate the motion by comparing the absolute difference between each frame's grayscale image F and the background grayscale image B, and modify the formula as follows:

[0222] D i (x, y) = |F i (x, y) - B(x, y)| (11)

[0223] Similarly, by defining a threshold T, a binary image is obtained, and the moving human body region is displayed. The formula is as follows:

[0224]

[0225] Here, H represents the range of the tracking window.

[0226] In embodiments of the present invention, the maximum inter-class variance method is used for differential image binarization. After binarization, noise problems such as rough edges, broken edges, and internal bubbles inevitably occur. Morphological processing is a good solution in this case; common morphological processing methods include dilation, erosion, and opening / closing operations. Embodiments of the present invention use median filtering, erosion, and dilation to obtain a relatively smooth and complete human body contour, and then use a thinning algorithm to extract the human skeleton of a second person.

[0227] It should be noted that this invention uses a combination of frame differencing and background subtraction for second-person motion detection. This reduces false detections that can occur when either method is used alone, and improves the coverage and sensitivity of motion detection, including both fast and slow movements, thus enhancing detection capabilities. It also adapts to dynamic backgrounds. The distance transform-based thinning algorithm effectively extracts elongated structures from the original image while maintaining connectivity, and its simple implementation makes the computation relatively efficient.

[0228] Return to reference Figure 9 In operation S930, based on the human skeleton sequence information and the pre-trained action recognition analysis model, action feature expression is obtained.

[0229] In an embodiment of the present invention, the action recognition and analysis model is trained based on a long short-term memory network.

[0230] In embodiments of the present invention, Long Short-Term Memory (LSTM) networks are a special type of convolutional neural network that can learn long-term dependent information and alleviate the gradient vanishing and gradient exploding problems caused by convolutional neural networks. Its main structure consists of an input gate, an output gate, and a forget gate. The computational equation for the LSM network is as follows:

[0231] Γ i =σ(w i [a t-1 x t )+b i (13)

[0232] Γ f =σ(w f [a t-1 x t )+b f (14)

[0233] Γ o =σ(w o [a t-1 x t )+b o (15)

[0234]

[0235]

[0236] a t =Γ o ·tanh(c t (18)

[0237] Among them, Γ i , Γf and Γ o These are the input gate, forget gate, and output gate, respectively. σ is the activation function, a is short-term memory, c is long-term memory, t is the time series number, w is the weight, and b is the bias value. By using the skeleton sequence information as temporal feature input, action risk discrimination and output representation are obtained. Temporal features refer to time-related feature information contained in the time series.

[0238] In an embodiment of the present invention, pre-training the action recognition analysis model specifically includes: acquiring a second historical video dataset, which includes historical action information of a second person; acquiring historical human skeleton sequence information based on the second historical video dataset; acquiring an action recognition training dataset and an action recognition test dataset based on the historical human skeleton sequence information; constructing a long short-term memory network model; training the long short-term memory network model based on the action recognition training dataset to obtain an intermediate action recognition analysis model; and evaluating the intermediate action recognition analysis model based on the action recognition test dataset to finally obtain the action recognition analysis model.

[0239] Return to reference Figure 2 In operation S270, a third risk probability is obtained based on the action feature expression.

[0240] In an embodiment of the present invention, a third risk probability is obtained based on the action feature representation and the SoftMax layer.

[0241] In an embodiment of the present invention, the SoftMax layer is used to convert the output of the action feature analysis model into a probability distribution and determine the probability of a third risk. The operation of obtaining the third risk probability based on the SoftMax layer is similar to that of obtaining the first risk probability, and will not be described again here.

[0242] It should be noted that the process of obtaining action feature expressions through action information complies with the provisions of relevant laws and regulations, takes necessary confidentiality measures, and does not violate public order and good morals.

[0243] In operation S280, based on the micro-expression emotion feature expression, the first risk probability, the audio emotion feature expression, the second risk probability, the action feature expression, the third risk probability, and the pre-trained joint discrimination model, the final risk discrimination score is obtained.

[0244] Figure 11 A flowchart illustrating a method for obtaining a final risk assessment score according to an embodiment of the present invention is shown.

[0245] In an embodiment of the present invention, the main process for obtaining the final risk discrimination score includes operations S1110 to S1140.

[0246] In operation S1110, based on the micro-expression emotion feature expression, the action feature expression, and the audio emotion feature expression, feature fusion is performed using linear connections to obtain fused feature expression.

[0247] In operation S1120, a potential fusion feature representation is obtained based on the fusion feature representation and the fully connected layer network.

[0248] In operation S1130, the joint risk probability is obtained based on the potential fusion feature representation and the SoftMax layer.

[0249] In operation S1140, the first risk probability, the second risk probability, the third risk probability, and the joint risk probability are weighted and calculated to obtain the final risk discrimination score.

[0250] In embodiments of the present invention, facial micro-expression emotion features, action features, and audio emotion features are obtained as inputs, and these three feature expressions are fused.

[0251] In embodiments of the present invention, linear connections can be used for feature fusion, followed by a three-layer fully connected network for further feature fusion to obtain potential feature representations. For each fully connected layer, the number of output channels is K1, K2, and K3, respectively, with the specific number of channels determined based on the dimension of the input fused features. Finally, a SoftMax layer is used to obtain the risk probability discrimination result, which is then weighted and calculated with the first, second, and third risk probabilities of the present invention to obtain the final discrimination result score. The specific calculation formula is as follows:

[0252] f=λf u +η(μ1f i +μ2f j +μ3f k (19)

[0253] λ+η=1 (20)

[0254] μ1+μ2+μ3=1 (21)

[0255] Where u1, u2, and u3 are used to weigh the importance of the first risk probability, the second risk probability, and the third risk probability, respectively, and λ and η represent the joint discrimination result and the common hyperparameters of the first risk probability, the second risk probability, and the third risk probability, respectively. Through weighted calculation, a final risk discrimination score is obtained, which will be used to determine whether the current operator faces external risks.

[0256] In an embodiment of the present invention, the joint discrimination model is trained based on a neural network.

[0257] In an embodiment of the present invention, pre-training a joint discrimination model specifically includes: acquiring historical micro-expression features, historical action features, and historical audio emotion features, wherein the historical micro-expression features, historical action features, and historical audio emotion features are obtained based on a first historical video dataset and a micro-expression emotion analysis model, a historical audio dataset and an audio emotion analysis model, and a second historical video dataset and an action recognition analysis model, respectively, and further obtaining a first historical risk probability, a second historical risk probability, and a third historical risk probability; acquiring a historical training set and a historical test set based on the historical micro-expression features, the historical action features, and the historical audio emotion features, as well as the first historical risk probability, the second historical risk probability, and the third historical risk probability; constructing a discrimination model, training the discrimination model based on the historical training set to obtain an intermediate joint discrimination model; evaluating the intermediate joint discrimination model based on the historical test set to finally obtain the joint discrimination model.

[0258] Return to reference Figure 2 In operation S290, an early warning scheme is provided based on the final risk assessment score.

[0259] In embodiments of the present invention, the early warning scheme includes intervention by security personnel for observation or voice prompts, as well as remote alarms. Specifically, staff will make a judgment based on the final risk assessment score and provide corresponding countermeasures.

[0260] It should be understood that the first video data, audio data, and second video data in the embodiments of the present invention will be cleaned up after a period of time to avoid leakage of user privacy.

[0261] It should be noted that the collection, storage, use, processing, transmission, provision, disclosure, and application of users' personal information all comply with relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0262] According to embodiments of the present invention, a method for multimodal information analysis based on a joint discriminant network is proposed, which can achieve the following technical effects:

[0263] 1. Proactive detection and early warning: When the operator inserts the card into the machine, multimodal information is actively transmitted, and the early warning method begins to proactively analyze external security risks;

[0264] 2. High early warning efficiency: The trained early warning method can predict risky behaviors in a short time, and can provide prediction results within a few seconds, and issue real-time early warnings based on the prediction results;

[0265] 3. Low early warning cost: Compared with the huge losses and costs of manual safety inspections after the fact, this method only requires automatic analysis by the early warning method, saving manpower costs;

[0266] 4. Improve security and reduce customer losses: When the model analysis identifies potential security risks, it provides early warnings, improving the security of ATM operations and potentially reducing customer financial losses.

[0267] Figure 12 A schematic block diagram of a device for multimodal information analysis based on a joint discriminant network according to an embodiment of the present invention is shown.

[0268] like Figure 12 As shown, the device 1200 for multimodal information analysis based on a joint discriminant network according to this embodiment includes an information acquisition module 1210, a micro-expression emotion feature acquisition module 1220, a first risk probability acquisition module 1230, an audio emotion feature acquisition module 1240, a second risk probability acquisition module 1250, an action feature acquisition module 1260, a third risk probability acquisition module 1270, a final risk discrimination score acquisition module 1280, and an early warning scheme provision module 1290.

[0269] The information acquisition module 1210 can be used to acquire facial information, audio information, and motion information. In one embodiment, the information acquisition module 1210 can be used to perform the operation S210 described above, which will not be repeated here.

[0270] The micro-expression emotion feature acquisition module 1220 can be used to acquire micro-expression emotion features based on the facial information and a pre-trained micro-expression emotion analysis model. In one embodiment, the micro-expression emotion feature acquisition module 1220 can be used to perform the operation S220 described above, which will not be repeated here.

[0271] The first risk probability acquisition module 1230 can be used to acquire a first risk probability based on the micro-expression emotion features. In one embodiment, the first risk probability acquisition module 1230 can be used to perform the operation S230 described above, which will not be repeated here.

[0272] The audio emotion feature acquisition module 1240 can be used to acquire audio emotion features based on the audio information and a pre-trained audio emotion analysis model. In one embodiment, the audio emotion feature acquisition module 1240 can be used to perform the operation S240 described above, which will not be repeated here.

[0273] The second risk probability acquisition module 1250 can be used to acquire a second risk probability based on the emotional features expressed in the audio. In one embodiment, the second risk probability acquisition module 1250 can be used to perform the operation S250 described above, which will not be repeated here.

[0274] The action feature expression acquisition module 1260 can be used to acquire action feature expressions based on a pre-trained action recognition and analysis model of the action information. In one embodiment, the action feature expression acquisition module 1260 can be used to perform the operation S260 described above, which will not be repeated here.

[0275] The third risk probability acquisition module 1270 can be used to acquire a third risk probability based on the action feature expression. In one embodiment, the third risk probability acquisition module 1270 can be used to perform the operation S270 described above, which will not be repeated here.

[0276] The final risk discrimination score acquisition module 1280 can be used to obtain a final risk discrimination score based on the micro-expression emotion feature expression, the first risk probability, the audio emotion feature expression, the second risk probability, the action feature expression, the third risk probability, and a pre-trained joint discrimination model. In one embodiment, the final risk discrimination score acquisition module 1280 can be used to perform the operation S280 described above, which will not be repeated here.

[0277] The early warning scheme providing module 1290 can be used to provide an early warning scheme based on the final risk discrimination score. In one embodiment, the early warning scheme providing module 1290 can be used to perform the operation S290 described above, which will not be repeated here.

[0278] According to an embodiment of the present invention, the micro-expression emotion feature acquisition module 1220 includes: a first video data acquisition unit, a micro-expression keyframe extraction module, and a micro-expression emotion feature acquisition unit.

[0279] The first video data acquisition unit can be used to perform grayscale processing on the first video data to obtain grayscale-processed first video data. In one embodiment, the first video data acquisition unit can be used to perform the operation S3 10 described above, which will not be repeated here.

[0280] The micro-expression keyframe extraction module can be used to extract micro-expression keyframes of the first user based on the grayscale processed first video data. In one embodiment, the micro-expression keyframe extraction module can be used to perform the operation S320 described above, which will not be repeated here.

[0281] The micro-expression emotion feature acquisition unit can be used to acquire micro-expression emotion features based on the micro-expression keyframes and the pre-trained micro-expression emotion analysis model. In one embodiment, the micro-expression emotion feature acquisition unit can be used to perform the operation S330 described above, which will not be repeated here.

[0282] According to an embodiment of the present invention, the micro-expression keyframe extraction module includes: a reference image acquisition unit, an i-th Euclidean distance acquisition module, a peak frame acquisition unit, and a micro-expression keyframe acquisition unit.

[0283] The reference image acquisition unit can be used to use a first frame face image with a recognizable face in the grayscale processed first video data as a reference image. In one embodiment, the reference image acquisition unit can be used to perform the operation S410 described above, which will not be repeated here.

[0284] The i-th Euclidean distance acquisition module can be used to obtain the i-th Euclidean distance between the i-th frame face image with recognizable face in the first grayscale processed video data and the reference image using the Euclidean distance method, where i is greater than 1 and i is a positive integer. In one embodiment, the i-th Euclidean distance acquisition module can be used to perform the operation S420 described above, which will not be repeated here.

[0285] The peak frame acquisition unit can be used to obtain a peak frame based on the i-th Euclidean distance and the peak frame algorithm. In one embodiment, the peak frame acquisition unit can be used to perform the operation S430 described above, which will not be repeated here.

[0286] The micro-expression keyframe acquisition unit can be used to obtain the micro-expression keyframes of the first user by using the peak frames as keyframes. In one embodiment, the micro-expression keyframe acquisition unit can be used to perform the operation S440 described above, which will not be repeated here.

[0287] According to an embodiment of the present invention, the i-th Euclidean distance acquisition module includes: a feature point acquisition unit, a vector acquisition unit, and an i-th Euclidean distance acquisition unit.

[0288] The feature point acquisition unit can be used to acquire the origin v and N key feature points, where N is a positive integer. In one embodiment, the feature point acquisition unit can be used to perform the operation S510 described above, which will not be repeated here.

[0289] The vector acquisition unit can be used to vectorize the reference image based on the N key feature points to obtain a first vector, and to vectorize the i-th frame face image with a recognizable face in the video data of the first person to obtain the i-th vector. In one embodiment, the vector acquisition unit can be used to perform the operation S520 described above, which will not be repeated here.

[0290] The i-th Euclidean distance acquisition unit can be used to calculate the sum of squares of the differences between the first vector and the i-th vector based on the origin v and the N key feature points, and then take the square root and absolute value of the sum to obtain the i-th variation difference. The i-th variation difference is then normalized to obtain the i-th Euclidean distance. In one embodiment, the i-th Euclidean distance acquisition unit can be used to perform the operation S530 described above, which will not be repeated here.

[0291] According to an embodiment of the present invention, the audio emotion feature expression acquisition module 1240 includes: a first audio data acquisition unit, an audio data feature sequence acquisition module, and an audio emotion feature expression acquisition unit.

[0292] The first audio data acquisition unit can be used to perform audio preprocessing on the audio data of the first person to obtain first audio data. In one embodiment, the first audio data acquisition unit can be used to perform the operation S710 described above, which will not be repeated here.

[0293] The audio data feature sequence acquisition module can be used to acquire an audio data feature sequence based on Mel-frequency cepstral coefficients and the first audio data. In one embodiment, the audio data feature sequence acquisition module can be used to perform the operation S720 described above, which will not be repeated here.

[0294] The audio emotion feature acquisition unit can be used to acquire audio emotion features based on the audio data feature sequence and the pre-trained audio emotion analysis model. In one embodiment, the audio emotion feature acquisition unit can be used to perform the operation S730 described above, which will not be repeated here.

[0295] According to an embodiment of the present invention, the audio data feature sequence acquisition module includes: a second audio data acquisition unit, a power spectrum acquisition unit, an energy value acquisition unit, a logarithmic energy value acquisition unit, a Mel-Cepstral coefficient feature acquisition unit, and an audio data feature sequence acquisition unit.

[0296] The second audio data acquisition unit can be used to pre-emphasize, frame, and window the first audio data to obtain the second audio data. In one embodiment, the second audio data acquisition unit can be used to perform the operation S810 described above, which will not be repeated here.

[0297] The power spectrum acquisition unit can be used to acquire the power spectrum based on the second audio data and the periodogram method. In one embodiment, the power spectrum acquisition unit can be used to perform the operation S820 described above, which will not be repeated here.

[0298] The energy value acquisition unit can be used to filter the power spectrum using a Mel filter bank and calculate the energy value within each filter. In one embodiment, the energy value acquisition unit can be used to perform the operation S830 described above, which will not be repeated here.

[0299] The logarithmic energy value acquisition unit can be used to take the logarithm of the energy value in each filter to obtain the logarithmic energy value. In one embodiment, the logarithmic energy value acquisition unit can be used to perform the operation S840 described above, which will not be repeated here.

[0300] The Mel-Cepstral Coefficient Feature Acquisition Unit can be used to perform a discrete cosine transform on the logarithmic energy value to obtain Mel-Cepstral Coefficient Features. In one embodiment, the Mel-Cepstral Coefficient Feature Acquisition Unit can be used to perform the operation S850 described above, which will not be repeated here.

[0301] The audio data feature sequence acquisition unit can be used to acquire an audio data feature sequence based on the Mel-frequency cepstral coefficient features. In one embodiment, the audio data feature sequence acquisition unit can be used to perform the operation S860 described above, which will not be repeated here.

[0302] According to an embodiment of the present invention, the motion feature expression acquisition module 1220 includes: a second video data acquisition unit, a human skeleton sequence information acquisition module, and a motion feature expression acquisition unit.

[0303] The second video data acquisition unit can be used to perform grayscale processing on the second video data to obtain grayscale-processed second video data. In one embodiment, the second video data acquisition unit can be used to perform the operation S910 described above, which will not be repeated here.

[0304] The human skeleton sequence information acquisition module can be used to obtain human skeleton sequence information based on the grayscale-processed second video data, using mean shift algorithm, frame difference method, background difference method, maximum inter-class variance method, human morphology processing, and thinning algorithm. In one embodiment, the human skeleton sequence information acquisition module can be used to perform the operation S920 described above, which will not be repeated here.

[0305] The action feature expression acquisition unit can be used to acquire action feature expressions based on the human skeleton sequence information and the pre-trained action recognition analysis model. In one embodiment, the action feature expression acquisition unit can be used to perform the operation S930 described above, which will not be repeated here.

[0306] In an embodiment of the present invention, the human skeleton sequence information acquisition module includes a first clustering information acquisition unit, a first foreground acquisition unit, a binarization unit, a human contour acquisition unit, and a human skeleton sequence information acquisition unit.

[0307] The first clustering information acquisition unit obtains first clustering information based on the grayscale-processed second video data using a mean-shift algorithm. In one embodiment, the first clustering information acquisition unit can be used to perform the operation S1010 described above, which will not be repeated here.

[0308] The first foreground acquisition unit can be used to select a first reference frame or a second reference frame based on the first clustering information, and acquire the first foreground using frame difference and background difference methods. In one embodiment, the first foreground acquisition unit can be used to perform the operation S1020 described above, which will not be repeated here.

[0309] The binarization unit can be used to binarize the first foreground using the maximum inter-class variance method. In one embodiment, the binarization unit can be used to perform the operation S1030 described above, which will not be repeated here.

[0310] The human body contour acquisition unit can be used to perform human morphological processing on the binarized first foreground to obtain a human body contour. In one embodiment, the human body contour acquisition unit can be used to perform the operation S1040 described above, which will not be repeated here.

[0311] The human skeleton sequence information acquisition unit refines the first human contour based on a thinning algorithm to obtain the human skeleton sequence information. In one embodiment, the human skeleton sequence information acquisition unit can be used to perform the operation S1050 described above, which will not be repeated here.

[0312] In an embodiment of the present invention, the final risk discrimination score acquisition module 1280 includes: a fusion feature expression acquisition unit, a potential fusion feature expression acquisition unit, a joint risk probability acquisition unit, and a final risk discrimination score acquisition unit.

[0313] The fusion feature expression acquisition unit can be used to perform feature fusion based on the micro-expression emotion expression, the action feature expression, and the audio emotion expression, using linear connections to obtain a fusion feature expression. In one embodiment, the fusion feature expression acquisition unit can be used to perform the operation S1110 described above, which will not be repeated here.

[0314] The potential fusion feature representation acquisition unit can be used to obtain potential fusion feature representations based on the fusion feature representations and the fully connected layer network. In one embodiment, the potential fusion feature representation acquisition unit can be used to perform the operation S1120 described above, which will not be repeated here.

[0315] The joint risk probability acquisition unit can be used to obtain the joint risk probability based on the potential fusion feature representation and the SoftMax layer. In one embodiment, the joint risk probability acquisition unit can be used to perform the operation S1130 described above, which will not be repeated here.

[0316] The final risk discrimination score acquisition unit can be used to perform a weighted calculation on the first risk probability, the second risk probability, the third risk probability, and the joint risk probability to obtain a final risk discrimination score. In one embodiment, the final risk discrimination score acquisition unit can be used to perform the operation S1140 described above, which will not be repeated here.

[0317] Figure 13 A block diagram of an electronic device suitable for implementing a multimodal information analysis method based on a joint discriminant network according to an embodiment of the present invention is shown schematically.

[0318] like Figure 13 As shown, an electronic device 1300 according to an embodiment of the present invention includes a processor 1301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1302 or a program loaded from a storage portion 1308 into a random access memory (RAM) 1303. The processor 1301 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1301 may also include onboard memory for caching purposes. The processor 1301 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0319] RAM 1303 stores various programs and data required for the operation of electronic device 1300. Processor 1301, ROM 1302, and RAM 1303 are interconnected via bus 1304. Processor 1301 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 1302 and / or RAM 1303. It should be noted that the programs may also be stored in one or more memories other than ROM 1302 and RAM 1303. Processor 1301 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in said one or more memories.

[0320] According to an embodiment of the present invention, the electronic device 1300 may further include an input / output (I / O) interface 1305, which is also connected to the bus 1304. The electronic device 1300 may also include one or more of the following components connected to the I / O interface 1305: an input section 1306 including a keyboard, mouse, etc.; an output section 1307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1308 including a hard disk, etc.; and a communication section 1309 including a network interface card such as a LAN card, modem, etc. The communication section 1309 performs communication processing via a network such as the Internet. A drive 1310 is also connected to the I / O interface 1305 as needed. A removable medium 1311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1310 as needed so that computer programs read from it can be installed into the storage section 1308 as needed.

[0321] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of the present invention.

[0322] According to embodiments of the present invention, a computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of the present invention, a computer-readable storage medium may include ROM 1302 and / or RAM 1303 and / or one or more memories other than ROM 1302 and RAM 1303 described above.

[0323] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of the present invention.

[0324] When the computer program is executed by processor 1 301, it performs the functions defined in the system / apparatus of this invention. According to embodiments of the invention, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0325] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1309, and / or installed from the removable medium 1311. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0326] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1309, and / or installed from the removable medium 1311. When the computer program is executed by the processor 1301, it performs the functions defined in the system of this embodiment of the invention. According to embodiments of the invention, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0327] According to embodiments of the present invention, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0328] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0329] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. A method for multimodal information analysis based on a joint discriminant network, characterized in that, The method includes: Acquire facial information, audio information, and motion information, wherein the facial information includes first video data of a first person, the audio information includes audio data of the first person, and the motion information includes second video data of a second person; Based on the facial information and the pre-trained micro-expression emotion analysis model, micro-expression emotion features are obtained, wherein micro-expression keyframes are obtained based on the Euclidean distance method and the peak frame extraction algorithm, and the micro-expression emotion features are obtained based on the micro-expression keyframes and the micro-expression emotion analysis model. Based on the micro-expression emotional features, a first risk probability is obtained; Based on the audio information and a pre-trained audio sentiment analysis model, audio sentiment features are obtained. Based on the emotional features expressed in the audio, a second risk probability is obtained; Based on the action information and the pre-trained action recognition and analysis model, action feature representation is obtained, wherein the action information is processed using mean shift algorithm, frame difference method, background difference method, maximum inter-class variance method, human morphology processing and thinning algorithm; Based on the expressed action features, the third risk probability is obtained; Based on the micro-expression emotion features, the action features, and the audio emotion features, feature fusion is performed using linear connections to obtain a fused feature expression; based on the fused feature expression and a fully connected layer network, a potential fused feature expression is obtained; based on the potential fused feature expression and a SoftMax layer, a joint risk probability is obtained; the first risk probability, the second risk probability, the third risk probability, and the joint risk probability are weighted and calculated to obtain a final risk discrimination score; and Based on the final risk assessment score, an early warning scheme is provided.

2. The method according to claim 1, characterized in that, The acquisition of facial information, audio information, and motion information specifically includes: Facial information is acquired based on the first camera; Based on the first recorder, acquire audio information; Motion information is acquired using a second camera.

3. The method according to claim 2, characterized in that, The micro-expression emotion analysis model, based on the facial information and a pre-trained micro-expression emotion analysis model, obtains micro-expression emotion features, specifically including: The first video data is processed into grayscale to obtain the grayscale processed first video data. Based on the grayscale processed first video data, extract the micro-expression keyframes of the first person; and Based on the micro-expression keyframes and the pre-trained micro-expression emotion analysis model, micro-expression emotion features are obtained.

4. The method according to claim 3, characterized in that, The extraction of micro-expression keyframes of the first person specifically includes: The first frame of the face image with a recognizable face in the first video data after grayscale processing is used as the reference image; The Euclidean distance method is used to obtain the i-th Euclidean distance between the i-th frame face image with recognizable face in the first grayscale processed video data and the reference image, where i is greater than 1 and i is a positive integer; Based on the i-th Euclidean distance and peak frame extraction algorithm, the peak frame is obtained; and The peak frames are used as keyframes to obtain the micro-expression keyframes of the first person.

5. The method according to claim 4, characterized in that, The step of using the Euclidean distance method to obtain the i-th Euclidean distance between the i-th frame face image with recognizable face in the first grayscale processed video data and the reference image specifically includes: Obtain the origin v and N key feature points, where N is a positive integer; Based on the N key feature points, the reference image is vectorized to obtain a first vector, and the i-th frame face image with a recognizable face in the video data of the first person is vectorized to obtain the i-th vector. Based on the origin v and the N key feature points, calculate the sum of squares of the differences between the first vector and the i-th vector, take the square root and absolute value, and obtain the i-th change difference; and The i-th variation difference is normalized to obtain the i-th Euclidean distance.

6. The method according to claim 3, characterized in that, The micro-expression sentiment analysis model is trained based on a convolutional neural network.

7. The method according to claim 2, characterized in that, The acquisition of audio emotional features based on the audio information and a pre-trained audio sentiment analysis model specifically includes: The audio data of the first person is preprocessed to obtain the first audio data; Based on the Mel-frequency cepstral coefficients and the first audio data, an audio data feature sequence is obtained; and Based on the audio data feature sequence and the pre-trained audio sentiment analysis model, the audio sentiment feature expression is obtained.

8. The method according to claim 7, characterized in that, The step of obtaining the audio data feature sequence based on Mel-frequency cepstral coefficients and the first audio data specifically includes: The first audio data is pre-emphasized, framed, and windowed to obtain the second audio data. Based on the second audio data and the periodogram method, the power spectrum is obtained; The power spectrum is filtered using a Mel filter bank, and the energy value within each filter is calculated. Take the logarithm of the energy value within each filter to obtain the logarithmic energy value; Perform a discrete cosine transform on the logarithmic energy values ​​to obtain the Mel-frequency cepstral coefficient characteristics; and Based on the Mel-Cepstral Coefficients feature, an audio data feature sequence is obtained.

9. The method according to claim 7, characterized in that, The audio sentiment analysis model is trained based on a gated recurrent neural network.

10. The method according to claim 2, characterized in that, The acquisition of action feature representation based on the action information and the pre-trained action recognition and analysis model specifically includes: The second video data is processed into grayscale to obtain grayscale processed second video data. Based on the grayscale-processed second video data, human skeleton sequence information is obtained using mean shift algorithm, frame difference method, background difference method, maximum inter-class variance method, human morphology processing, and thinning algorithm; and Based on the human skeleton sequence information and the pre-trained action recognition and analysis model, action feature representation is obtained.

11. The method according to claim 10, characterized in that, The second video data, after grayscale processing, utilizes mean shift algorithm, frame difference method, background difference method, maximum inter-class variance method, human morphology processing, and thinning algorithm to obtain human skeleton sequence information, specifically including: Based on the grayscale processed second video data, the first clustering information is obtained using the mean shift algorithm; Based on the first clustering information, a first reference frame or a second reference frame is selected, and the first foreground is obtained using the frame difference method and the background difference method. The first foreground is binarized using the Otsu's method; The binarized first foreground is subjected to human morphological processing to obtain a human body outline; and Based on the thinning algorithm, the human body contour is thinned to obtain the human skeleton sequence information.

12. The method according to claim 10, characterized in that, The action recognition and analysis model is trained based on a long short-term memory network.

13. The method according to claim 1, characterized in that, The method further includes: Based on the micro-expression emotion features and the SoftMax layer, the first risk probability is obtained; Based on the aforementioned audio emotional feature expression and the SoftMax layer, a second risk probability is obtained; and Based on the action feature representation and the SoftMax layer, the third risk probability is obtained.

14. The method according to claim 1, characterized in that, The joint discrimination model is trained based on a neural network.

15. The method according to claim 1, characterized in that, The early warning system includes intervention by security personnel for observation or voice broadcast prompts.

16. A monitoring device based on multimodal information analysis using a joint discriminant network, characterized in that, The device includes: The information acquisition module is used to: acquire facial information, audio information, and motion information, wherein the facial information includes first video data of a first person, the audio information includes audio data of the first person, and the motion information includes second video data of a second person; The micro-expression emotion feature acquisition module is used to: acquire micro-expression emotion features based on the facial information and a pre-trained micro-expression emotion analysis model, wherein micro-expression key frames are obtained based on the Euclidean distance method and the peak frame extraction algorithm, and the micro-expression emotion features are acquired based on the micro-expression key frames and the micro-expression emotion analysis model. The first risk probability acquisition module is used to: acquire a first risk probability based on the micro-expression emotional features; The audio emotion feature expression acquisition module is used to: acquire audio emotion feature expressions based on the audio information and a pre-trained audio emotion analysis model; The second risk probability acquisition module is used to: acquire a second risk probability based on the emotional features expressed in the audio. The action feature representation acquisition module is used to: acquire action feature representations based on a pre-trained action recognition and analysis model of the action information; The third risk probability acquisition module is used to: acquire the third risk probability based on the action feature expression; The final risk discrimination score acquisition module is used for: fusing features based on the micro-expression emotion features, the action feature features, and the audio emotion features using linear connections to obtain a fused feature expression; obtaining a potential fused feature expression based on the fused feature expression and a fully connected layer network; obtaining a joint risk probability based on the potential fused feature expression and a SoftMax layer; and performing a weighted calculation on the first risk probability, the second risk probability, the third risk probability, and the joint risk probability to obtain a final risk discrimination score; and The early warning scheme providing module is used to: provide an early warning scheme based on the final risk assessment score.

17. An electronic device comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 15.

18. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 15.

19. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 15.

Images