Multimodal-based emotion recognition and prediction method and system

Abstract

The embodiment of the application discloses a kind of in multimodal emotion recognition, prediction method and system. Method includes: the multimodal signal of human individual is collected as input data;The multimodal signal is input to pre-training, analysis good emotion recognition, prediction model and is identified and predicted, and result is obtained;The result is analyzed and explained by reverse reasoning, and the cause of forming emotion is found, and timely emotional intervention is beneficial to improve physical and mental health. Compared with prior art, the present application can further improve the accuracy of emotion recognition and prediction results, more importantly, it can be intelligently inferred and learned by combining the multimodal data of human individuals, to explain the mechanism of emotion occurrence, improve the reliability and credibility of emotion recognition and prediction results, and remove the obstacles of intelligent methods in practical application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of emotion analysis, in particular to a multi-modal based emotion recognition and prediction method and system. BACKGROUND

[0002] Emotion recognition refers to judging a person's emotional state through various means such as observing facial expressions, body language, voice tone, etc. It is an important technology in many fields such as human-computer interaction, psychology research, market research, etc.

[0003] Emotion prediction is a process of estimating a person's emotional state in the future period of time according to the person's historical emotional data, current environmental factors, event information, etc. It helps to take intervention measures in advance to guide the emotion to develop in a positive direction.

[0004] In the prior art, there are various methods for recognizing and predicting human individual emotions. For example, Chinese Patent No. CN111312292A discloses an emotion recognition method based on speech recognition, which mainly includes: obtaining to-be-recognized speech data; generating a spectrogram corresponding to the to-be-recognized speech data; inputting the spectrogram corresponding to the to-be-recognized speech data into a deep learning based emotion recognition model to obtain an emotion recognition result corresponding to the to-be-recognized speech data. This scheme only performs model training and emotion recognition based on speech data, resulting in relatively single data input to the model, thereby affecting the final recognition accuracy. SUMMARY

[0005] In view of the technical defects of the prior art, the purpose of the embodiments of the present application is to provide a multi-modal based emotion recognition and prediction method and system.

[0006] To achieve the above-mentioned purpose, in a first aspect, the embodiments of the present application provide a multi-modal based emotion recognition and prediction method, comprising:

[0007] Collecting multi-modal signals of a human individual as input data; the input data includes a set of text data input by the human individual on a network terminal within a preset time period, a shaking image of the human individual collected by a terminal device within the preset time period, and a set of physiological signals collected by the terminal within the preset time period;

[0008] Inputting the multi-modal signals into a pre-trained and analyzed emotion recognition and prediction model for recognition and prediction to obtain a result;

[0009] Performing reverse reasoning analysis and interpretation on the result to find the causes of forming emotions.

[0010] Further, as a specific implementation manner of the present application, before collecting the multi-modal signals of the human individual, the method further comprises:

[0011] constructing a multi-modal data sample library;

[0012] constructing a daily life knowledge logic library;

[0013] constructing an emotion recognition and prediction model;

[0014] training the emotion recognition and prediction model based on the daily life knowledge logic library;

[0015] conducting reliability analysis on the trained emotion recognition and prediction model under reverse reasoning analysis and interpretation based on the sample data.

[0016] As a specific implementation mode of the present application, the multi-modal data sample library is constructed in the following manner:

[0017] Multi-modal data is collected through a smart bracelet, a microphone, a camera or a mobile phone APP to form a multi-modal data sample library; the multi-modal data includes physiological data, sound data, image data and text data.

[0018] As a specific implementation mode of the present application, the daily life knowledge logic library is constructed in the following manner:

[0019] The daily life knowledge logic library is constructed by referencing a third-party knowledge graph or extracting from Wikipedia on the Internet.

[0020] As a specific implementation mode of the present application, the reliability analysis on the trained emotion recognition and prediction model under reverse reasoning analysis and interpretation is conducted in the following manner:

[0021] The sample data in the multi-modal data sample library is extracted and tested to analyze the explainable mechanism of the output model, and the mechanism is compared with human knowledge in the daily life knowledge logic library to test the reliability of the mechanism of the model.

[0022] The input data is denoted as P r , the text data set is denoted as C r , the vibration image is denoted as I v，r , and the physiological signal set is denoted as S r .

[0023] The architecture of the emotion recognition and prediction model includes a deep convolutional neural network, a long short memory neural network, a Transformer network and a deep symbolic neural network.

[0024] The deep convolutional neural network is used to process the vibration image data I v，r , and calculate the vibration image semantic encoding related to emotion.

[0025] a long short-term memory neural network for processing the physiological signal S r , computing a physiological semantic encoding related to the emotion;

[0026] a transformer network for processing the text data set C r , computing a text semantic encoding related to the emotion, decoding the text semantic encoding into emotion elements by a decoder, inputting the emotion elements into a deep symbolic neural network, computing inferences, and fusing the semantic encodings to obtain emotion indicator data.

[0027] wherein the neural units of the deep symbolic neural network include a logical computing part and an arithmetic computing part, adopt a closed-loop structure, and the inference result of the deep symbolic neural network can be input again for inference calculation until a stable and unchanged result is obtained, and then output to the transformer network to estimate and predict the emotion state E r , E r+1 , …, E r+n .

[0028] As a specific implementation mode of the present application, the training of the emotion recognition and prediction model is specifically:

[0029] According to the daily life knowledge library, load logical inference knowledge and arithmetic operation knowledge, express the knowledge in the deep symbolic neural network according to the weight of a polynomial and a logical example, and convert the logical inference and arithmetic knowledge in the knowledge base into the weight of the deep symbolic neural network and initialization values;

[0030] generate weights by training and adjusting sample data in the multi-modal data sample library and Finally, the emotion recognition and prediction model is obtained.

[0031] As a specific implementation mode of the present application, the explainable analysis of the trained emotion recognition and prediction model includes a first part and a second part;

[0032] wherein the first part is a main cause analysis of emotion occurrence, specifically:

[0033] Suppose the model of the transformer network is E r =F T (A r |W sT ), wherein A r is an emotion element;

[0034] Suppose a i ∈A rContribution rate Q of E r Contribution rate Q of E i Therefore, it is necessary to calculate by the following calculation formula: E is the expected value, which can be obtained by sample output test statistics, Q i is a vector with the same dimension as E r , representing the value a i Contribution rate of Q i , the larger the value in the vector, the greater the effect of a i on the induced emotion;

[0035] The second part is the generation process of emotional elements, specifically:

[0036] Define the loss function as Loss inv = |A r0 |1+[A r -G ∞ (A r0 |W La )]

[0037] Where G 2 (A r0 |W La ) = G(G(A r0 |W La )), … G ∞ = G(G(G……)) is the reasoning iteration, and represents the number of iterations to reach a stable A r ;

[0038] The reverse reasoning is obtained by solving s.t.G ∞ (A r |W La ) = A r

[0039] The optimization problem can obtain the initial emotional element After obtaining the initial element A r0 , the key inducement to induce the emotional result can be obtained.

[0040] In a second aspect, the embodiments of the present application also provide a multi-modal based emotion recognition and prediction system, comprising a processor, an input device, an output device and a memory, the processor, the input device, the output device and the memory are connected with each other, wherein the memory is used to store a computer program, the computer program comprises program instructions, the processor is configured to call the program instructions, and execute the method of the first aspect.

[0041] The embodiment of the present application is implemented, and a model is established by analyzing the related data of a human individual to identify and predict emotions, obtain results, and perform reverse reasoning analysis and interpretation on the results to find the causes of forming emotions and timely perform emotional intervention to improve physical and mental health. Compared with the prior art, the present application can further improve the accuracy of emotional identification and prediction results, and more importantly, can perform intelligible intelligent reasoning learning by combining the multi-modal data of a human individual to explain the emotional occurrence mechanism, improve the reliability and credibility of the emotional identification and prediction results, and remove the obstacles of intelligent methods in practical applications. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced as follows.

[0043] Figure 1 is a flowchart of the multi-modal based emotion identification and prediction method provided by the embodiment of the present application;

[0044] Figure 2 is a network structure block diagram;

[0045] Figure 3 is an example diagram of a deep symbolic neural network and a logic part;

[0046] Figure 4 is an example diagram of a deep symbolic neural network and an arithmetic part;

[0047] Figure 5 is a structure diagram of the multi-modal based emotion identification and prediction system provided by the embodiment of the present application. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0049] It should be understood that, when used in the present specification and the appended claims, the terms "comprise" and "include" indicate the existence of described features, integers, steps, operations, elements, and / or components, but do not exclude one or more other features, integers, steps, operations, elements, components, and / or sets thereof.

[0050] Please refer to Figure 1 is a multi-modal based emotion identification and prediction method provided by the embodiment of the present application, which comprises the following steps:

[0051] S1, construct a multi-modal data sample library.

[0052] In specific implementation, multi-modal data is collected, and physiological data, sound data, image data, and text data are obtained through smart bands, microphones, cameras, mobile phone APPs, etc., to form a multi-modal data sample library.

[0053] S2, construct a daily life knowledge logic library.

[0054] In specific implementation, the daily life knowledge logic library is constructed by referencing third-party knowledge graphs or extracting from online Wikipedia. It should be noted that the inventors believe that the daily life knowledge logic is necessary because the changes in emotions may be related to events and elements in daily life.

[0055] S3, construct an emotion recognition and prediction model, train it, and analyze its reliability.

[0056] Before introducing the architecture, construction, training, and reliability analysis of the model, the following explanations are given for data collection and model input and output:

[0057] (I) Input data for human individual recognition and prediction model:

[0058] where T is the length of the time series record, and r is the time series number. The input data P r is constructed as follows:

[0059] P r = (t r , C r , I v，r , S r )

[0060] where t r represents the time of the time series number r, and represents a fixed length of time, such as 14:00 to 15:00 on September 17, 2024; C r is the text data set input by the human individual on the network terminal such as the mobile phone APP or social network terminal within this period of time, I v，r is the vibration image of the human individual collected by the terminal device within this period of time, and S r is the physiological signal set such as electrocardiogram collected by the terminal device within this period of time.

[0061] (II) Intermediate output process data for human individual recognition and prediction model:

[0062] P rIdentify specific emotion-forming events, elements, etc. related to the knowledge base, collectively referred to as emotion-forming elements, hereinafter referred to as emotion elements, write the elements as A r =(a1, a2, …, a m , b1, …, b n ), where m represents the total amount of events, elements related to the expression in the knowledge base, where a i ∈[0, 1], depicting the probability of the existence of emotion logic elements (events, elements), b j ∈[0, 1], indicating the measure of emotion elements (events, elements).

[0063] The events contained in the emotion elements include daily life events such as promotion, salary increase, death of relatives, etc. The elements contained in the emotion elements include some physiological, psychological, and social elements such as increased heart rate, feeling sad, friends visiting, etc. These elements and events can be obtained through physiological signals, text language, vibration images, etc.

[0064] (Three) Output data of human individual recognition and prediction model:

[0065] Through P r and element A r , the identified emotion index E r =(e1, …, e 10 ), where e1, …, e 10 are ten emotion indexes, including: aggression, stress, tension, suspicion, balance, depression, charm, vitality, self-discipline, neuroticism, happiness, and each category has ten grade indexes, represented by continuous numerical values 0-1, [0, 0.1] represents grade 1, (0.1, 0.2] represents grade 2, …, (0.9, 1] represents grade 10. Based on this data description, the model also outputs a prediction set of future emotion indexes where n represents the predicted future time step, i.e. the emotion of the future Tn length.

[0066] In this embodiment, the overall structure of the emotion recognition and prediction model is shown in Figure 2 The model is composed of a neural network, which includes: a deep convolutional (CNN) neural network, a long short-term memory (LSTM) neural network, a Transformer network, and a deep symbolic neural network. The deep convolutional neural network is responsible for processing vibration image data I v，r , calculating the vibration image semantic encoding related to the emotion, the LSTM neural network is responsible for processing physiological signals S r, calculate the physiological semantic code related to emotion; the Transformer network (attention mechanism encoding) is responsible for processing text data, calculating the text semantic code related to emotion; and through the decoder, the codes are decoded into emotional elements, and the emotional elements are input into the deep symbolic neural network, the reasoning is calculated, and the semantic codes are fused and decoded to obtain emotional index data. Among them, the physiological elements and semantic elements are high-dimensional vector embedding expressions.

[0067] It should be noted that in Figure 2 , the unmarked encoder and decoder are all fully connected neural networks FNN (emotional element decoding).

[0068] The deep symbolic neural network of the present application will be described below. The neural unit calculation of the deep symbolic neural network is defined as two parts: logical calculation and arithmetic calculation.

[0069] (1) For symbolic logical calculation, the structure is as shown in Figure 3

[0070] Assume that x∈[0, 1] and y∈[0, 1] are the preconditions, and z∈[0, 1] is the calculation result. The logical operation includes AND, OR, and NOT, and any logic can be constructed from the three combinations. The calculation formula of the arithmetic logic operation is as follows:

[0071] (1) Find the complement z=1-x,

[0072] (2) Find the AND ∧: z=x·y, and in the case of multi-element operation, z=Π i x i

[0073] (3) Find the OR ∨: z=x+y-x·y, and in the case of multi-element operation, z=1-Π i (1-x i )

[0074] The symbolic operation unit function is formed by Figure 3 structure: wherein vec(.) is the matrix vectorization operation.

[0075] where x∈[0, 1] p is the input, z∈[0, 1] q is the intermediate logical process result, q is the logical capacity size, p is the input dimension, W L ∈{-1, 0, 1} q×p is the weight matrix.

[0076] Through multi-layer combination, a symbolic neural network

[0077] ​

[0078] where Logical inference of network under network weights Constraint regularization:

[0079] Sparse regularization of network weights

[0080] Numerical constraint of network weights Regularization of construction:

[0081]

[0082] (ii) For the arithmetic operation part, such as Figure 4

[0083] Using a similar structure, a polynomial neural network whose single layer network contains 1st order, 1.5th order input combination terms is: where W a is the weight. By stacking this single layer network, the arithmetic part of the symbolic neural network is obtained, which is

[0084]

[0085] From the above structure of the neural network, terms with the highest even order of 2s and the highest odd order of 3+2(s-2) can be constructed. Among them s is a given parameter, the larger the parameter, the higher the polynomial order that can be expressed. The constraint for the arithmetic part of the network is the row group sparse constraint: where w Lji is the weight vector of the i-th row of the j-th layer of the network.

[0086] Therefore, the structure of the deep symbolic neural network is:

[0087]

[0088] Let x=(a1, …, a m ), y=(b1, …, b n ), that is, the input is the emotional element. It should be noted that the symbolic neural network inference calculation uses a closed loop structure, that is, the inference result can also be used as the input of the symbolic neural network to perform inference calculation again until a stable and unchanged result is obtained, and then output to the Transformer decoder to estimate and predict the emotional state E r , E r+1 , …, E r+n .

[0089] Further, the model training and reverse inference analysis and explanation method is explained as follows: ​

[0090] First, according to the daily knowledge base, load the logical reasoning knowledge and arithmetic operation knowledge, express the weight of the polynomial and Figure 3 The logical example in the knowledge base converts the logical reasoning and arithmetic knowledge in the knowledge base into symbolic neural network weights And Initialize the numerical value. Then further adjust the generated symbolic network by training the sample data, and finally obtain the emotion recognition prediction model. The specific training process is as follows:

[0091] Suppose at time r, the data P r The corresponding emotion is E r , and the emotion state in the future 10T needs to be predicted, and the recognition model is constructed as

[0092]

[0093] Where is the weight of the symbolic neural network, and W s is the weight of other networks in the model. Then the following least square loss function for training is constructed plus the weight penalty of the symbolic neural network.

[0094]

[0095] The training sample is input into the model, and the network weight parameter W s is adjusted to minimize the Loss (such as using the stochastic gradient descent method), and the finally applicable network model is trained. After training, some mechanism processes about the emotional elements and emotional output can be obtained by checking Through analysis and model testing of W s , the contribution rate of the factors can be obtained, so as to analyze which factors are the main factors inducing emotions.

[0096] The explainable analysis of the model can be divided into two parts. The first part is the main cause analysis of the occurrence of emotions, that is, by analyzing the relationship between the output E r and the emotional elements A r , it can be found out which events and factors are the main factors inducing emotions. The second part is the generation process of the emotional elements, that is, through the reverse process of symbolic reasoning calculation, the generation process of the emotional elements (events and factors) is obtained, so as to have an intuitive logical explanation of the induction of emotions.

[0097] For the analysis of the first part, suppose the model of the Transformer decoder is E r = F T (A r | W sT ), where Assume that a is to be analyzed i ∈ A r The contribution rate Q r of a to E i is calculated by the following formula: Where E is the expectation, which can be obtained by the sample output test statistics. Q i is a vector with the same dimension as E r , which represents the contribution rate of a i to Q i . The larger the value in the vector, the greater the effect of a i on the induction of emotions.

[0098] For the symbolic calculation part, it can be obtained by back calculation, that is, calculating A r0 = G -1 (A r | W La ), where A r0 is the initial emotional element, which is the minimum initial emotional element set, that is, the key emotional element, and A r is the stable result of the symbolic neural network calculation iteration. Here, the back reasoning calculation model G -1 is constructed. The loss function is defined as:

[0099] Loss inv = |A r0 | 1 + [A r - G ∞ (A r0 | W La )],

[0100] Where G 2 (A r0 | W La ) = G(G(A r0 | W La ))…G ∞ = G(G(G…)) is the reasoning iteration, and ∞ represents the number of iterations to reach a stable A r . Back reasoning can be obtained by solving the

[0101]

[0102] optimization problem, that is, the initial emotional element After obtaining the initial element A r0 , the key inducement of the induced emotional result can be obtained.

[0103] S4, collect multi-modal signals of human individuals as input data.

[0104] The input data includes a text data set input by a human individual at a network terminal within a preset time period, a shaking image of the human individual collected by a terminal device within the preset time period, and a physiological signal set collected by the terminal within the preset time period.

[0105] S5, inputting the multi-modal signals into a pre-trained, analyzed emotion recognition and prediction model for recognition and prediction to obtain a result.

[0106] S6, performing reverse reasoning analysis and interpretation on the result to find the cause of forming the emotion.

[0107] As can be known from the above description, the present application identifies and predicts the emotion of a human individual based on multi-modal data, proposes an interpretable deep symbolic neural network perception reasoning prediction framework method, and establishes an intelligent recognition prediction model through learning of multi-modal data such as shaking images, natural language, and physiological signals. The model identifies multi-modal signals as personal emotional elements, and takes the same as input, combines symbolic neural network reasoning calculation of a daily emotional cause knowledge model, constructs a deep symbolic neural network framework, and forms an interpretable reasoning network.

[0108] The present application not only further improves the accuracy of emotion recognition and prediction results, but more importantly, can intelligently reason and learn in an interpretable manner by combining multi-modal data of a human individual to explain the emotion occurrence mechanism, improve the reliability and credibility of emotion recognition and prediction results, and remove obstacles in practical application of intelligent methods.

[0109] In addition, the method can accurately find the cause of forming the emotion through reverse reasoning combined with feature contribution analysis based on the proposed deep symbolic neural network architecture. Unlike the input feature association and contribution analysis adopted by most interpretable neural networks, the method proposed in the present application not only analyzes the input feature contribution, but also accurately identifies the key cause of emotion based on the reverse reasoning method of the symbolic neural network, thereby further improving the reliability and credibility of the intelligent model method and enhancing its application value.

[0110] It should be emphasized that the deep symbolic neural network proposed in the present application is not limited to the emotion recognition and prediction method, but can be used in all interpretable, reliable, and credible neural network methods.

[0111] Based on the same inventive concept, the present application also provides an emotion recognition and prediction system based on multi-modal data. As Figure 5As shown, the system can include one or more processors 101, one or more input devices 102, one or more output devices 103, and a memory 104, which are connected with each other through a bus 105. The memory 104 is configured to store a computer program, the computer program including program instructions, and the processor 101 is configured to invoke the program instructions to execute the methods in the method embodiment part.

[0112] It should be understood that, in the embodiments of the present application, the processor 101 can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0113] The input device 102 can include a keyboard, etc., and the output device 103 can include a display (LCD, etc.), a speaker, etc.

[0114] The memory 104 can include read-only memory and random access memory, and provide instructions and data for the processor 101. A portion of the memory 104 can also include non-volatile random access memory. For example, the memory 104 can also store device type information.

[0115] In a specific implementation, the processor 101, the input device 102, and the output device 103 described in the embodiments of the present application can perform the implementation described in the embodiments of the emotion recognition and prediction method based on multi-modal provided by the embodiments of the present application, and will not be described here.

[0116] It should be noted that the specific workflow of the present embodiment can refer to the method embodiment part described above, and will not be described here.

[0117] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical range disclosed by the present application, and these modifications or replacements should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A multi-modal based emotion recognition, prediction method, characterized in that, The method comprises the following steps: Collecting multi-modal signals of a human individual as input data; the input data comprises a set of text data input by the human individual on a network terminal within a preset time period, a shaking image of the human individual collected by a terminal device within a preset time period, and a set of physiological signals collected by the terminal within a preset time period; Inputting the multi-modal signals into a pre-trained and analyzed emotion recognition and prediction model for recognition and prediction to obtain a result; Performing reverse reasoning analysis and interpretation on the result to find the cause of the emotion; The explainable analysis of the trained emotion recognition and prediction model comprises a first part and a second part; The first part is the main cause analysis of emotion occurrence, specifically: Assume that the model of the Transformer network is where ; A r is an emotional element; E r is an emotional indicator; Assuming we want to analyze For E r Contribution rate Q i Then it needs to be calculated using the following formula: ， E is the expectation, which can be obtained by the sample output test statistics, Q i is a vector, the dimension is the same as E r , which represents the numerical value a i The contribution rate of Q i , the larger the value in the vector, the greater the effect on the emotional induction; a i ; , which depicts the probability of the existence of emotional logic elements; The second part is the generation process of emotional elements, specifically: The loss function is defined as , wherein is the number of times the iteration reaches a stable A r for the reasoning iteration, ∞ indicates that the iteration reaches a stable A Reverse reasoning by solving The optimization problem is to obtain the initial emotional elements , Obtain initial element A r0 After that, the key triggers to induce emotional outcomes can be identified.

2. The method of claim 1, wherein, Before collecting the multi-modal signals of the human individual, the method further comprises the following steps: Building a multi-modal data sample library; Building a daily life knowledge logic library; Building an emotion recognition and prediction model; Training the emotion recognition and prediction model based on the daily life knowledge logic library; Performing reliability analysis on the trained emotion recognition and prediction model under reverse reasoning analysis and interpretation based on sample data.

3. The method of claim 2, wherein, Building a multi-modal data sample library specifically comprises: Collecting multi-modal data through a smart bracelet, a microphone, a camera, or a mobile phone APP to form a multi-modal data sample library; the multi-modal data includes physiological data, sound data, image data, and text data.

4. The method of claim 2, wherein, Building a daily life knowledge logic library specifically comprises: Building a daily life knowledge logic library by referencing a third-party knowledge graph or extracting from Wikipedia on the Internet.

5. The method of claim 2, wherein, Performing reliability analysis on the trained emotion recognition and prediction model under reverse reasoning analysis and interpretation specifically comprises: Extracting and testing sample data in the multi-modal data sample library, analyzing the explainable mechanism of the output model, and comparing it with human knowledge in the daily life knowledge logic library to test the reliability of the mechanism of the model.

6. The method of claim 2, wherein, The input data is denoted by P r , a set of text data is denoted by C r , a shaking image is denoted by I v，r , and a set of physiological signals is denoted by S r ; The architecture of the emotion recognition and prediction model comprises a deep convolutional neural network, a long short-term memory neural network, a Transformer network, and a deep symbolic neural network; a deep convolutional neural network for processing seismic image data I v，r computing a seismic image semantic encoding related to the emotion therein; a long short-term memory neural network for processing the physiological signal S r computing a physiological semantic encoding related to the emotion therein; A transformer network for processing a set of text data C r , calculate the text semantic encoding related to the emotion, decode the text semantic encoding into emotion elements through the decoder, input the emotion elements into the deep symbolic neural network, calculate the inference, and fuse and decode the semantic encoding to obtain the emotion index data.

7. The method of claim 6, wherein, The neural unit of the deep symbolic neural network comprises a logical calculation part and an arithmetic calculation part, adopts a closed loop structure, and the inference result of the deep symbolic neural network can be input again for inference calculation until a stable and unchanged result is obtained, and then output to the Transformer network for estimation and prediction of the emotion state E r ,E r+1 ,…,E r+n .

8. The method of claim 6, wherein, Training the emotion recognition and prediction model specifically comprises: According to the daily life knowledge library, load logical reasoning knowledge and arithmetic operation knowledge, express and depth symbol neural network in the weight of polynomial logical example, convert the logical reasoning and arithmetic knowledge in the knowledge base into the weight of the depth symbol neural network and the initial value of ; Training, adjusting and generating weights by sample data in a multi-modal data sample library and to obtain an emotion recognition prediction model.

9. A multi-modal based emotion recognition, prediction system, characterized in that, A computer device comprising a processor, an input device, an output device, and a memory, wherein the processor, the input device, the output device, and the memory are connected to each other, the memory is used to store a computer program, the computer program comprises program instructions, the processor is configured to invoke the program instructions, and the method according to any one of claims 1-8 is executed.

Images