Management layer psychological state recognition method and system based on multi-modal feature fusion
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEFEI UNIV OF TECH
- Filing Date
- 2022-08-30
- Publication Date
- 2026-06-26
AI Technical Summary
Existing technologies for identifying psychological states based on single-modal information have low accuracy and cannot meet the financial sector's need for precise analysis of the psychological states of management.
A multimodal feature fusion method is adopted, which extracts facial image features, behavioral pose features, and text data features of text interaction interface from video data through CNN convolutional neural network, graph convolutional network and BERT model. After feature fusion using TFN method, psychological state recognition is performed by combining PCA principal component analysis and support vector machine.
It improves the accuracy of psychological state recognition, enabling more accurate identification of the psychological state of management, which helps investors conduct precise analysis and make investment decisions.
Smart Images
Figure CN115457627B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of mental state recognition technology, specifically to a method and system for identifying the mental state of management based on multimodal feature fusion. Background Technology
[0002] The World Health Organization (WHO) defines health as a three-dimensional condition: "health is a state of complete physical, mental and social well-being." Mental state refers to a series of responses an individual makes in response to specific stimuli, manifested in physical or facial movements and coordinated changes in physiological state. Mental state is crucial in daily life and interpersonal interactions; therefore, research on mental state identification has become increasingly important in recent years. In the financial sector, the mental state of management is often correlated with the company's performance. Assessing the mental state of management during earnings call videos can help investors understand their management's psychological state, thereby aiding in their analysis and investment decisions.
[0003] Currently, common mental state recognition technologies mainly analyze and study information from a single modality, such as mental state recognition based on facial expressions, mental state recognition based on physiological signals such as electroencephalograms (EEGs), and mental state recognition based on speech information.
[0004] However, the indicator features extracted from single-modal information do not provide a complete information representation of psychological states. Therefore, the accuracy of psychological state identification based on this method needs to be improved. Summary of the Invention
[0005] (a) Technical problems to be solved
[0006] To address the shortcomings of existing technologies, this invention provides a method and system for identifying the mental state of management based on multimodal feature fusion, which solves the problem of low accuracy in existing mental state identification based on single-modal information.
[0007] (II) Technical Solution
[0008] To achieve the above objectives, the present invention provides the following technical solution:
[0009] Firstly, this invention proposes a method for identifying the mental state of management levels based on multimodal feature fusion, the method comprising:
[0010] Acquire video data of management personnel speaking, text information data from text interaction interfaces, and data on the psychological state classification tags of management personnel;
[0011] Facial image features of the video data are extracted based on CNN convolutional neural network; behavioral pose features of the video data are extracted based on graph convolutional network and temporal convolutional network; and text data features of the text information data of the text interaction interface are obtained based on BERT model.
[0012] The matrix-based TFN method fuses the facial image features, the behavioral pose features, and the text data features to obtain fused features.
[0013] Principal component analysis (PCA) is used to reduce the dimensionality of the fused features to obtain the dimensionality-reduced fused features.
[0014] Based on the dimensionality-reduced and fused features and the management psychological state classification label data, support vector machine is used to identify the management psychological state.
[0015] Preferably, the extraction of facial image features from the video data based on a CNN convolutional neural network includes:
[0016] Cut the video data file into frame images and save them;
[0017] Use Python to call the cv2 library to perform face recognition on the frame images and save the images containing faces;
[0018] Perform operations on images containing human faces, including preprocessing, grayscale conversion, geometric transformation, and image enhancement.
[0019] The facial image features of the face image after the above operations are extracted using a CNN convolutional neural network.
[0020] Preferably, the extraction of behavioral pose features from the video data based on graph convolutional networks and temporal convolutional networks includes:
[0021] Use Openpose to obtain the upper body joint information of the management layer in each frame of the image;
[0022] The joint coordinates in the joint information are normalized.
[0023] The normalized joint coordinates are transformed in spatial and temporal dimensions using graph convolutional networks and temporal convolutional networks to output behavioral pose features.
[0024] Preferably, obtaining the psychological state classification label data of the management includes: after several subjects watch several video clips and text interaction interfaces in the video data, classifying each video clip and text interaction interface into negative and positive dimensions respectively.
[0025] Preferably, the step of using a support vector machine to identify the psychological state of management based on the dimensionality-reduced fused features and the management psychological state classification label data includes:
[0026] A mental state classifier for multimodal data mental state recognition is constructed based on the support vector machine algorithm;
[0027] The dimensionality-reduced and fused features are used as input to the classifier, and the management psychological state classification label data is used as output to train the psychological state classifier.
[0028] The trained mental state classifier is used to identify the mental state of the management team.
[0029] Secondly, this invention also proposes a management-level psychological state recognition system based on multimodal feature fusion, the system comprising:
[0030] The data acquisition module is used to acquire video data of management personnel speaking, text information data from the text interaction interface, and classification label data of management personnel's psychological state.
[0031] The multimodal feature extraction module is used to extract facial image features from the video data based on CNN convolutional neural network, extract behavioral pose features from the video data based on graph convolutional network and temporal convolutional network, and obtain text data features from the text information data of the text interaction interface based on BERT model.
[0032] The multimodal feature fusion module is used to perform feature fusion on the face image features, the behavioral pose features, and the text data features based on the matrix TFN method to obtain fused features;
[0033] The feature dimensionality reduction module is used to reduce the dimensionality of the fused features using principal component analysis (PCA) to obtain the dimensionality-reduced fused features.
[0034] The psychological state recognition module is used to perform management psychological state recognition using a support vector machine based on the dimensionality-reduced fused features and the management psychological state classification label data.
[0035] Preferably, the multimodal feature extraction module extracts facial image features from the video data based on a CNN convolutional neural network, including:
[0036] Cut the video data file into frame images and save them;
[0037] Use Python to call the cv2 library to perform face recognition on the frame images and save the images containing faces;
[0038] Perform operations on images containing human faces, including preprocessing, grayscale conversion, geometric transformation, and image enhancement.
[0039] The facial image features of the face image after the above operations are extracted using a CNN convolutional neural network.
[0040] Preferably, the multimodal feature extraction module extracts behavioral pose features of the video data based on graph convolutional networks and temporal convolutional networks, including:
[0041] Use Openpose to obtain the upper body joint information of the management layer in each frame of the image;
[0042] The joint coordinates in the joint information are normalized.
[0043] The normalized joint coordinates are transformed in spatial and temporal dimensions using graph convolutional networks and temporal convolutional networks to output behavioral pose features.
[0044] Preferably, the data acquisition module obtains the psychological state classification label data of the management team by: after several subjects watch several video clips and text interaction interfaces in the video data, classifying each video clip and text interaction interface into negative and positive dimensions respectively.
[0045] Preferably, the psychological state recognition module performs management psychological state recognition using a support vector machine based on the dimensionality-reduced fused features and the management psychological state classification label data, including:
[0046] A mental state classifier for multimodal data mental state recognition is constructed based on the support vector machine algorithm;
[0047] The dimensionality-reduced and fused features are used as input to the classifier, and the management psychological state classification label data is used as output to train the psychological state classifier.
[0048] The trained mental state classifier is used to identify the mental state of the management team.
[0049] (III) Beneficial Effects
[0050] This invention provides a method and system for identifying the mental state of management levels based on multimodal feature fusion. Compared with existing technologies, it has the following advantages:
[0051] This invention acquires multimodal data, including video data of management personnel speaking and text information data from text interaction interfaces. It then extracts facial image features, behavioral pose features, and text data features from this multimodal data using different feature extraction methods. These features are then fused using a matrix-based TFN method, and the dimensionality of the fused features is reduced using PCA principal component analysis. Finally, a support vector machine-based multimodal data psychological state recognition model is trained using the dimensionality-reduced fused features and pre-acquired management psychological state classification label data. This trained model can then be used to identify the psychological state of management personnel. Compared to existing psychological state recognition technologies, the proposed management psychological state recognition technology offers higher accuracy and can quickly and easily identify the psychological state of management in financial scenarios, thus assisting investors in making accurate analysis and investment decisions. Attached Figure Description
[0052] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0053] Figure 1 This is a flowchart of a management-level psychological state recognition method based on multimodal feature fusion in Embodiment 1 of the present invention;
[0054] Figure 2 This is a schematic diagram of a management psychological state recognition system based on multimodal feature fusion in Embodiment 2 of the present invention. Detailed Implementation
[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are described clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0056] This application provides a method and system for identifying the psychological state of management based on multimodal feature fusion. This solves the problem of low accuracy in existing technologies that identify psychological states based on single-modal information data. It achieves the goal of accurately identifying the psychological state of management, thereby assisting investors in analyzing, judging and making investment decisions.
[0057] The technical solution in this application is to solve the above-mentioned technical problems, and the general idea is as follows:
[0058] To facilitate convenient, rapid, and accurate identification of the psychological state of management personnel in financial scenarios, thereby assisting investors in making precise analyses and investment decisions, the technical solution of this invention first acquires video data of management personnel speaking and text information data from text interaction interfaces. Simultaneously, it obtains management psychological state classification label data through a combination of machine and manual annotation. Then, it extracts facial image features and behavioral pose features from the video data using CNN convolutional neural networks, graph convolutional networks, and temporal convolutional networks, respectively; and obtains text data features from the text interaction interface text information data based on the BERT model. Next, it uses a matrix-based TFN method to fuse the above multimodal features to obtain fused features, and then uses PCA principal component analysis to reduce the dimensionality of the fused features to obtain dimensionality-reduced fused features. Finally, it trains a multimodal data psychological state recognition model based on support vector machines based on the dimensionality-reduced fused features and the management psychological state classification label data. The trained multimodal data psychological state recognition model can then be used to identify the psychological state of management personnel. The management psychological state recognition technology proposed in this invention achieves higher accuracy compared to existing psychological state recognition technologies.
[0059] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.
[0060] Example 1:
[0061] Firstly, this invention proposes a method for identifying the psychological state of management levels based on multimodal feature fusion, see [link to relevant documentation]. Figure 1 The method includes:
[0062] S1. Obtain video data of management personnel speaking, text information data of text interaction interface, and management psychological state classification label data;
[0063] S2. Extract facial image features from the video data based on CNN convolutional neural network; extract behavioral pose features from the video data based on graph convolutional network and temporal convolutional network; and obtain text data features from the text information data of the text interaction interface based on BERT model;
[0064] S3. The matrix-based TFN method is used to fuse the facial image features, the behavioral pose features, and the text data features to obtain the fused features;
[0065] S4. Use PCA (Principal Component Analysis) to reduce the dimensionality of the fused features and obtain the dimensionality-reduced fused features.
[0066] S5. Based on the dimensionality-reduced fused features and the management psychological state classification label data, support vector machine is used to identify the management psychological state.
[0067] As can be seen, this embodiment acquires multimodal data including video data of management personnel speaking and text information data from text interaction interfaces. Based on different feature extraction methods, it extracts facial image features, behavioral posture features, and text data features from the multimodal data. Then, it fuses these features using a matrix-based TFN method and uses PCA principal component analysis to reduce the dimensionality of the fused features. Finally, it uses the dimensionality-reduced fused features and pre-acquired management psychological state classification label data to train a support vector machine-based multimodal data psychological state recognition model. The trained multimodal data psychological state recognition model can then be used to identify the psychological state of management personnel. Compared with existing psychological state recognition technologies, the management psychological state recognition technology proposed in this invention has higher recognition accuracy and can conveniently and quickly identify the psychological state of management in financial scenarios, which is beneficial for assisting investors in making accurate analysis and judgments and investment decisions.
[0068] The following is in conjunction with the appendix Figure 1-2 The following details the implementation process of an embodiment of the present invention, including explanations of the specific steps S1-S5.
[0069] S1. Obtain video data of management personnel speaking, text information data of text interaction interface, and management psychological state classification label data.
[0070] 1) Video Data Acquisition. In this embodiment, the video URLs of earnings briefings are manually copied from the earnings briefing sections of the Shenzhen Stock Exchange or Shanghai Stock Exchange listed companies' video live streaming platforms. A web crawler program written in Python is then used to crawl the videos from each company's link. The crawled video data files from each company's earnings briefing are then trimmed and combined into video clips of each manager's speech, forming a video library.
[0071] One hundred company earnings call videos were selected from the video library. Each company's earnings call video included a video clip of a key executive speaking at the event. The selection criteria for the video clips were: moderate video length, approximately 10 minutes; and clear depiction of the target audience's psychological state.
[0072] 2) Text data collection. Scrape text-based Q&A information from the interactive text interface of the performance briefing section on the video live streaming platform.
[0073] 3) Tag Data Collection. In order to obtain more accurate management psychological state classification tag data, in this embodiment, we obtained the corresponding management psychological state classification tags in the performance briefing video clips through experiments. That is, we obtained the corresponding management psychological state classification tag data in the performance briefing video clips by combining machine learning psychological state annotation and manual annotation.
[0074] First, 20 volunteer students (10 girls and 10 boys) were recruited from within the school to participate in the experiment. All participants were required to have normal vision or corrected vision to normal. During the experiment, the participants were in a comfortable and quiet environment and their attention was relatively focused.
[0075] Then, participants watched 100 video clips and the company's interactive text-based webpage, rating each video on a scale of 1 to 7, categorizing them as negative or positive. Each video clip had a 10-second start-up prompt, was approximately 10 minutes long, and participants had 50 seconds to rate their performance after watching. Based on the participants' ratings, levels less than 3 were considered negative psychological states, and levels greater than 5 were considered positive psychological states. The corresponding psychological state labels were: negative psychological state = -1, positive psychological state = +1.
[0076] S2. Extract facial image features from the video data based on CNN convolutional neural network, extract behavioral pose features from the video data based on graph convolutional network and temporal convolutional network; and obtain text data features from the text information data of the text interaction interface based on BERT model.
[0077] After acquiring multimodal data, including video data of management personnel speaking and text information from the text interaction interface, we performed feature extraction on this multimodal data. Specifically,
[0078] When extracting features from the video data, we extracted facial image features and behavioral posture features of the management during their speeches from the aforementioned video data.
[0079] 1) Facial image feature extraction.
[0080] 1.1) Cut the video file into frames, capturing one frame per second and saving the frame images;
[0081] 1.2) After segmenting the video into images, use Python to call the cv2 library for face recognition, loop through each image, determine the location of the face, and save the image containing the face;
[0082] 1.3) Perform preprocessing on the above images containing human faces, including grayscale conversion, geometric transformation, and image enhancement.
[0083] 1.4) Use a CNN convolutional neural network to extract facial image features, and then output a specific feature space for each image, which is the facial image feature, specifically represented as a facial image feature vector, denoted as vector X.
[0084] 2) Extraction of behavioral posture features.
[0085] 2.1) Input the acquired video clip, use Openpose to analyze the upper body behavior and posture of the management layer in the video clip, and obtain the upper body joint information of the management layer in each frame, including the position coordinates and confidence of the points;
[0086] 2.2) Normalize the coordinates of the key points to maintain a consistent data ratio;
[0087] 2.3) Using graph convolutional networks and temporal convolutional networks, the spatial and temporal dimensions are transformed to output behavioral pose features, specifically in the form of behavioral pose feature vectors, denoted as vector Y.
[0088] 3) Text Data Feature Extraction. For the question-and-answer text information data portion of the performance briefing's interactive interface, since the prediction target is the management's psychological state, only the text content of the management's responses is selected for text analysis. This embodiment uses the BERT natural language processing method based on the Transformer encoder to extract the text data features of this portion of the data. BERT's advantage lies in its ability to capture contextual information and other forms of information, solve the problem of polysemy, and preserve semantic features to the greatest extent. Therefore, a pre-trained BERT model is used as the word vector training model to process the text information data of the interactive interface, ultimately generating 768-dimensional word vectors, denoted as the text data feature vector Z.
[0089] S3. The matrix-based TFN method is used to fuse the facial image features, the behavioral pose features, and the text data features to obtain the fused features.
[0090] After feature extraction, the original data yields three modal representation vectors (X, Y, Z): face image feature vector X, behavioral pose feature vector Y, and text data feature vector Z. Then, a matrix-based TFN method is used to fuse the multimodal data features. TFN is a typical multimodal network that fuses features through matrix operations; it directly fuses the three feature vectors X, Y, and Z from the three modalities to obtain the fused features, represented by vector M.
[0091] S4. Use PCA (Principal Component Analysis) to reduce the dimensionality of the fused features and obtain the dimensionality-reduced fused features.
[0092] After feature fusion in step S3, information redundancy still exists among the acquired multimodal data. In this embodiment, we choose to alleviate this problem using Principal Component Analysis (PCA). PCA is commonly used for dimensionality reduction of high-dimensional data. It uses linear transformation to recombine several variables or indicators into several new, uncorrelated composite variables, and then selects a few composite variables that retain as much of the original information as possible. In other words, PCA is an exploratory statistical analysis method that concentrates information scattered across a group of variables onto a few unrelated composite indicators (principal components). Generally, we select principal components with an information content greater than 85% or eigenvalues greater than 1. Here, we select those with an information content greater than 85%. Specifically, the calculation steps for dimensionality reduction of the fused features using PCA include:
[0093] 1) Standardize the original fused feature data;
[0094] 2) Calculate the correlation coefficient matrix;
[0095] 3) Calculate the eigenvalues λ of the correlation coefficient matrix. i and the corresponding eigenvector α i =(α i1 ,α i2 ...α ip T, i = 1, 2, ..., p;
[0096] Calculate and select the principal components, i.e., F1, F2, ..., F P Among them, the principal component F i =α i1 x1+α i2 x2+...+α ip x p F1, F2, ..., F P The components are uncorrelated and their variances decrease. The first m principal components are selected based on the cumulative variance contribution rate G(m). When the cumulative contribution rate (information content) reaches 85%, it is considered sufficient to reflect the information of the original variables. The corresponding m is the first m principal components extracted, that is, the first m principal components in the fused feature data are used as the dimensionality-reduced fused features.
[0097] S5. Based on the dimensionality-reduced fused features and the management psychological state classification label data, support vector machine is used to identify the management psychological state.
[0098] In this embodiment, a support vector machine (SVM) algorithm is selected to construct a mental state classifier for multimodal data mental state recognition. The previously fused and dimensionality-reduced fused features are used as input, and manually labeled mental state tags are used as output to train this mental state classifier. Specifically,
[0099] The experimental dataset was divided into training and test sets in an 8:2 ratio. Support Vector Machine (SVM) is a machine learning algorithm that essentially transforms the input space into a high-dimensional space through a linear transformation defined by an inner product function, and then finds the optimal classification surface in this high-dimensional space. For nonlinear problems, the problem can be transformed into a linear problem in a high-dimensional space through kernel function transformation. Therefore, this example applies the SVM model to the identification of the psychological state of management in a financial context.
[0100] The basic idea of Support Vector Machines is as follows: For a given training sample set D = {(x1,y1),(x2,y2),...,(x...}... n ,y n )},y i ∈{-1,+1}, i=1,2,...,n. Where, (x i ,y i () represents a sample point, in this embodiment x i Let y represent the characteristic matrix. i Indicates category label.
[0101] The primary goal of the SVM model is to find a hyperplane f(x) = ω with the largest geometric margin. T x + b = 0 partitions the feature space. Here, ω is the normal vector, and b is the displacement term. A set of ω and b uniquely defines a hyperplane, therefore the hyperplane is denoted as (ω, b). Similar to the distance from a point to a line, the distance from a point in the sample space to the hyperplane is: in If the hyperplane can correctly partition the sample data, then there exists (x i ,y i )∈D, when y i =+1, then ω T x i +b>0; when y i =-1, then ω T x i +b<0, that is
[0102]
[0103] The goal of SVM is to maximize the maximum distance between the two classes of sample points and the hyperplane, i.e., to find the values of parameters ω and b that make (1.1) hold true, and to maximize γ. Maximizing the distance is equivalent to maximizing ||ω|| -1 This is equivalent to minimizing ||ω|| 2 ,Right now
[0104]
[0105] st,y i (ω T x i +b)≥1,i=1,2,...m.
[0106] It can be seen that (1.2) is essentially a convex quadratic programming problem. The primal problem can be transformed into its dual problem using Lagrange duality, making the solution simpler. The Lagrangian function of this problem can be written as:
[0107]
[0108] In the formula, α=(α1,α2,...,α m ) T Setting the partial derivatives of L(ω,b,α) with respect to ω and b to 0, we obtain the dual problem of equation (1.2):
[0109]
[0110]
[0111] α i ≥0, i=1,2,...,m.
[0112] Next, by performing a minimax transformation on the objective function of the above equation, we can obtain the dual problem of (1.5) that has the same optimal solution:
[0113]
[0114]
[0115] α i ≥0, i=1,2,...,m.
[0116] Find the solution α in the above formula. * This is the solution to the dual optimization problem.
[0117] For the linearly inseparable problem in this example, to transform it into a linearly separable high-dimensional space, we can map the data to make it separable in a higher-dimensional space. This involves introducing a mapping function. After mapping the data x, the original data becomes... If we express that, then (1.4) becomes
[0118]
[0119] α i ≥0, i=1,2,...,m.
[0120] because The calculation of this is quite difficult, therefore a kernel function is introduced. Commonly used kernel functions include linear kernel functions, polynomial kernel functions, Gaussian kernel functions, and sigmoid kernel functions. Here, we choose the Gaussian kernel function (RBF kernel function).
[0121] Where σ>0 is the bandwidth of the Gaussian kernel.
[0122] Calculate f(x) to obtain the classification result.
[0123]
[0124] When a support vector machine (SVM) algorithm constructs a mental state classifier for multimodal data mental state recognition, and this classifier has been trained a predetermined number of times (or its classification accuracy reaches a predetermined level), it indicates that the classifier has met the requirements for mental state recognition. Subsequently, the trained classifier can be used to identify the mental states of management. Specifically,
[0125] By collecting video data and text information from the text interaction interface during the speeches of management personnel, and inputting this data into the pre-trained psychological state classifier based on the support vector machine algorithm to identify the psychological state of management personnel, the psychological state identification results of management personnel can be obtained. This can be used to assess the psychological state of management when they hold performance briefings. Investors can then analyze and judge based on the psychological state of management during their speeches and make investment decisions.
[0126] This completes the entire process of the present invention, a method for identifying the psychological state of management based on multimodal feature fusion.
[0127] Example 2:
[0128] Secondly, this invention also provides a management-level psychological state recognition system based on multimodal feature fusion, see [link to relevant documentation]. Figure 2 The system includes:
[0129] The data acquisition module is used to acquire video data of management personnel speaking, text information data from the text interaction interface, and classification label data of management personnel's psychological state.
[0130] The multimodal feature extraction module is used to extract facial image features from the video data based on CNN convolutional neural network, extract behavioral pose features from the video data based on graph convolutional network and temporal convolutional network, and obtain text data features from the text information data of the text interaction interface based on BERT model.
[0131] The multimodal feature fusion module is used to perform feature fusion on the face image features, the behavioral pose features, and the text data features based on the matrix TFN method to obtain fused features;
[0132] The feature dimensionality reduction module is used to reduce the dimensionality of the fused features using principal component analysis (PCA) to obtain the dimensionality-reduced fused features.
[0133] The psychological state recognition module is used to perform management psychological state recognition using a support vector machine based on the dimensionality-reduced fused features and the management psychological state classification label data.
[0134] Optionally, the multimodal feature extraction module extracts facial image features from the video data based on a CNN convolutional neural network, including:
[0135] Cut the video data file into frame images and save them;
[0136] Use Python to call the cv2 library to perform face recognition on the frame images and save the images containing faces;
[0137] Perform operations on images containing human faces, including preprocessing, grayscale conversion, geometric transformation, and image enhancement.
[0138] The facial image features of the face image after the above operations are extracted using a CNN convolutional neural network.
[0139] Optionally, the multimodal feature extraction module extracts behavioral pose features of the video data based on graph convolutional networks and temporal convolutional networks, including:
[0140] Use Openpose to obtain the upper body joint information of the management layer in each frame of the image;
[0141] The joint coordinates in the joint information are normalized.
[0142] The normalized joint coordinates are transformed in spatial and temporal dimensions using graph convolutional networks and temporal convolutional networks to output behavioral pose features.
[0143] Optionally, the data acquisition module obtains the psychological state classification label data of the management team by: after several subjects watch several video clips and text interaction interfaces in the video data, classifying each video clip and text interaction interface into negative and positive dimensions respectively.
[0144] Optionally, the psychological state recognition module performs management psychological state recognition using a support vector machine based on the dimensionality-reduced fused features and the management psychological state classification label data, including:
[0145] A mental state classifier for multimodal data mental state recognition is constructed based on the support vector machine algorithm;
[0146] The dimensionality-reduced and fused features are used as input to the classifier, and the management psychological state classification label data is used as output to train the psychological state classifier.
[0147] The trained mental state classifier is used to identify the mental state of the management team.
[0148] It is understood that the management psychological state recognition system based on multimodal feature fusion provided in this embodiment of the invention corresponds to the management psychological state recognition method based on multimodal feature fusion described above. The explanations, examples, and beneficial effects of the relevant content can be referred to the corresponding content in the management psychological state recognition method based on multimodal feature fusion, and will not be repeated here.
[0149] In summary, compared with existing technologies, it has the following beneficial effects:
[0150] 1. This invention acquires multimodal data, including video data and text information from text interaction interfaces during speeches by management personnel. It then extracts facial image features, behavioral pose features, and text data features from this multimodal data using different feature extraction methods. These features are then fused using a matrix-based TFN method, and the dimensionality of the fused features is reduced using PCA principal component analysis. Finally, a support vector machine-based multimodal data psychological state recognition model is trained using the dimensionality-reduced fused features and pre-acquired management psychological state classification label data. This trained model can then be used to identify the psychological state of management personnel. Compared to existing psychological state recognition technologies, this invention's management psychological state recognition technology offers higher accuracy and can quickly and easily identify the psychological state of management in financial scenarios, thus assisting investors in making accurate analyses and investment decisions.
[0151] 2. This invention extracts video data and text information data from the text interaction interface when management speaks, and extracts multimodal information features such as facial image features, behavioral posture features, and text data features based on these data. Compared with the indicator features extracted from single-modal information, it can improve the effectiveness and accuracy of identifying the psychological state of management in the video.
[0152] 3. When obtaining the psychological state classification labels of management, this invention combines machine learning psychological state annotation and manual annotation, making the annotation of video databases more efficient.
[0153] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0154] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for identifying the psychological state of management based on multimodal feature fusion, characterized in that, The method includes: Acquire video data of management personnel speaking, text information data from text interaction interfaces, and data on the psychological state classification tags of management personnel; Facial image features of the video data are extracted based on CNN convolutional neural network; behavioral pose features of the video data are extracted based on graph convolutional network and temporal convolutional network; and text data features of the management response text content in the text information data of the text interaction interface are obtained based on BERT model. The matrix-based TFN method performs feature fusion on the face image features, the behavioral pose features, and the text data features to obtain fused features; Principal component analysis (PCA) is used to reduce the dimensionality of the fused features to obtain the dimensionality-reduced fused features. Based on the dimensionality-reduced fused features and the management psychological state classification label data, a support vector machine is used to identify the management psychological state. The behavioral pose features extracted from the video data based on graph convolutional networks and temporal convolutional networks include: Use Openpose to obtain the upper body joint information of the management layer in each frame of the image; The joint coordinates in the joint information are normalized. The normalized joint coordinates are transformed in spatial and temporal dimensions using graph convolutional networks and temporal convolutional networks to output behavioral pose features.
2. The method as described in claim 1, characterized in that, The facial image features extracted from the video data based on the CNN convolutional neural network include: Cut the video data file into frame images and save them; Use Python to call the cv2 library to perform face recognition on the frame images and save the images containing faces; Perform operations on images containing human faces, including preprocessing, grayscale conversion, geometric transformation, and image enhancement. The facial image features of the face image after the above operations are extracted using a CNN convolutional neural network.
3. The method as described in claim 1, characterized in that, The acquisition of psychological state classification label data for management personnel includes: after several subjects watch several video clips and text interaction interfaces in the video data, each video clip and text interaction interface is classified into negative and positive dimensions.
4. The method as described in claim 1, characterized in that, The step of identifying the psychological state of management personnel using a support vector machine based on the dimensionality-reduced fused features and the management psychological state classification label data includes: A mental state classifier for multimodal data mental state recognition is constructed based on the support vector machine algorithm; The dimensionality-reduced and fused features are used as input to the classifier, and the management psychological state classification label data is used as output to train the psychological state classifier. The trained mental state classifier is used to identify the mental state of the management team.
5. A management-level psychological state recognition system based on multimodal feature fusion, characterized in that, The system includes: The data acquisition module is used to acquire video data of management personnel speaking, text information data from the text interaction interface, and classification label data of management personnel's psychological state. The multimodal feature extraction module is used to extract facial image features from the video data based on CNN convolutional neural network, extract behavioral pose features from the video data based on graph convolutional network and temporal convolutional network, and obtain text data features of the management response text content in the text information data of the text interaction interface based on BERT model; The multimodal feature fusion module is used to perform feature fusion on the face image features, the behavioral pose features, and the text data features based on the matrix TFN method to obtain fused features; The feature dimensionality reduction module is used to reduce the dimensionality of the fused features using principal component analysis (PCA) to obtain the dimensionality-reduced fused features. The psychological state recognition module is used to recognize the psychological state of management based on the dimensionality-reduced fused features and the management psychological state classification label data using a support vector machine. The behavioral pose features extracted from the video data based on graph convolutional networks and temporal convolutional networks include: Use Openpose to obtain the upper body joint information of the management layer in each frame of the image; The joint coordinates in the joint information are normalized. The normalized joint coordinates are transformed in spatial and temporal dimensions using graph convolutional networks and temporal convolutional networks to output behavioral pose features.
6. The system as described in claim 5, characterized in that, The multimodal feature extraction module extracts facial image features from the video data based on a CNN convolutional neural network, including: Cut the video data file into frame images and save them; Use Python to call the cv2 library to perform face recognition on the frame images and save the images containing faces; Perform operations on images containing human faces, including preprocessing, grayscale conversion, geometric transformation, and image enhancement. The facial image features of the face image after the above operations are extracted using a CNN convolutional neural network.
7. The system as described in claim 5, characterized in that, The data acquisition module obtains the psychological state classification label data of the management team by: after several subjects watch several video clips and text interaction interfaces in the video data, classifying each video clip and text interaction interface into negative and positive dimensions respectively.
8. The system as described in claim 5, characterized in that, The psychological state recognition module performs management psychological state recognition using a support vector machine based on the dimensionality-reduced fused features and the management psychological state classification label data, including: A mental state classifier for multimodal data mental state recognition is constructed based on the support vector machine algorithm; The dimensionality-reduced and fused features are used as input to the classifier, and the management psychological state classification label data is used as output to train the psychological state classifier. The trained mental state classifier is used to identify the mental state of the management team.