Emotion recognition and prediction method and system based on multiple modes

Abstract

The embodiment of the invention discloses an emotion recognition and prediction method and system based on multiple modes. The method comprises the following steps: collecting a multi-modal signal of a human individual as input data; inputting the multi-modal signal into a pre-trained and analyzed emotion recognition and prediction model for recognition and prediction to obtain a result; and reverse reasoning analysis and explanation are carried out on the result, inducements for forming emotions are found, and emotion intervention is carried out in time, so that the physical and psychological health is improved. Compared with the prior art, the precision of emotion recognition and prediction results can be further improved, more importantly, explanatable intelligent reasoning learning can be performed by combining multi-modal data of human individuals so as to explain the emotion generation mechanism therein, the reliability and credibility of the emotion recognition and prediction results are improved, and the emotion recognition and prediction accuracy is improved. And obstacles of the intelligent method in practical application are removed.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion analysis, and in particular to a multimodal-based emotion recognition and prediction method and a system thereof. Background Art

[0002] Emotion recognition refers to judging a person's current emotional state through various means, such as observing facial expressions, body language, voice intonation, etc. It is an important technology in many fields such as human-computer interaction, psychology research, and market research.

[0003] Emotion prediction is the process of estimating an individual's emotional state in the future based on their historical emotional data, current environmental factors, event information, etc. It helps to take intervention measures in advance to guide emotions in a positive direction.

[0004] In the prior art, there are many methods for identifying and predicting human individual emotions. For example, the Chinese patent with publication number CN111312292A discloses an emotion recognition method based on speech recognition, which mainly includes: obtaining speech data to be recognized; generating a spectrogram corresponding to the speech data to be recognized; inputting the spectrogram corresponding to the speech data to be recognized into an emotion recognition model based on deep learning to obtain the emotion recognition result corresponding to the speech data to be recognized. This solution only performs model training and emotion recognition based on speech data, resulting in relatively simple data input to the model, which affects the final recognition accuracy. Summary of the invention

[0005] In view of the technical deficiencies of the prior art, an object of the embodiments of the present invention is to provide a multimodal emotion recognition and prediction method and system thereof.

[0006] To achieve the above objectives, in a first aspect, an embodiment of the present invention provides a multimodal emotion recognition and prediction method, comprising:

[0007] Collecting multimodal signals of human individuals as input data; the input data includes a text data set input by the human individual in a network terminal within a preset time period, a vibration image of the human individual collected by the terminal device within a preset time period, and a physiological signal set collected by the terminal within a preset time period;

[0008] Inputting the multimodal signal into a pre-trained and analyzed emotion recognition and prediction model for recognition and prediction to obtain a result;

[0009] Conduct reverse reasoning analysis and interpretation on the results to find out the causes of the emotions.

[0010] Furthermore, as a specific implementation of the present application, before collecting the multimodal signal of the human individual, the method further includes:

[0011] Build a multimodal data sample library;

[0012] Build a logical library of daily life knowledge;

[0013] Build emotion recognition and prediction models;

[0014] Training the emotion recognition and prediction model based on the daily life knowledge logic library;

[0015] Based on the sample data, the trained emotion recognition and prediction model is subjected to reliability analysis under reverse reasoning analysis interpretation.

[0016] As a specific implementation of this application, building a multimodal data sample library is specifically as follows:

[0017] Multimodal data is collected through smart bracelets, microphones, cameras or mobile phone APPs to form a multimodal data sample library; the multimodal data includes physiological data, sound data, image data and text data.

[0018] As a specific implementation of this application, constructing a daily life logic knowledge base is specifically as follows:

[0019] Build a logical library of daily life knowledge by referencing third-party knowledge graphs or extracting online Wikipedia.

[0020] As a specific implementation of the present application, the trained emotion recognition and prediction model is subjected to reliability analysis under reverse reasoning analysis and interpretation, specifically:

[0021] By extracting and testing sample data in the multimodal data sample library, analyzing the output model's explanatory mechanism, and comparing it with the human knowledge in the daily life knowledge logic library, the reliability of the model's mechanism is verified.

[0022] Wherein, the input data is denoted as P r , the text data set is denoted as C r , the vibration image is recorded as I v，r , the physiological signal set is denoted as S r ;

[0023] The architecture of the emotion recognition and prediction model includes a deep convolutional neural network, a long short-term memory neural network, a Transformer network, and a deep symbolic neural network;

[0024] Deep convolutional neural network for processing vibration image data I v，r , calculate the semantic coding of the vibration image related to emotions;

[0025] Long short-term memory neural network for processing physiological signals S r , calculate the physiological semantic codes related to emotions;

[0026] Transformer network, used to process text data set C r , calculate the text semantic coding related to emotions, decode the text semantic coding into emotional elements through a decoder, input the emotional elements into a deep symbolic neural network, calculate reasoning, and fuse and decode the semantic coding to obtain emotional indicator data.

[0027] The neural unit of the deep symbolic neural network includes a logic calculation part and an arithmetic calculation part, and adopts a closed-loop structure. The reasoning result of the deep symbolic neural network can be used as input to perform reasoning calculation again until a stable and unchanged result is obtained, and then output to the Transformer network to estimate and predict the emotional state E. r , E r+1 , …, E r+n .

[0028] As a specific implementation of the present application, the emotion recognition and prediction model is trained as follows:

[0029] According to the daily life knowledge logic library, the logical reasoning knowledge and arithmetic operation knowledge are loaded, and the logical reasoning and arithmetic knowledge in the knowledge library are converted into the weights of the symbolic neural network according to the weight expression of polynomials and the logic examples in the deep symbolic neural network. and The initialization value of

[0030] Through the sample data in the multimodal data sample library, the generated weights are adjusted and Finally, the emotion recognition prediction model is obtained.

[0031] As a specific implementation of the present application, the interpretable analysis of the trained emotion recognition and prediction model includes a first part and a second part;

[0032] The first part is an analysis of the main causes of emotions, specifically:

[0033] Assume that the model of the Transformer network is E r =F T (A r |W sT ),in A r For emotional elements;

[0034] Suppose we want to analyze a i ∈A rFor E r Contribution rate Q i , you need to calculate it by the following formula: E is the expectation, which can be obtained by testing the sample output statistics, Q i is a vector with the same dimension as E r Same, indicating the value a i Q i The contribution rate of a. The larger the value in the vector, the greater the contribution rate of a. i The greater the effect on emotion induction;

[0035] The second part is the generation process of emotional elements, specifically:

[0036] Define the loss function as Loss inv =|A r0 |1+[A r -G ∞ (A r0 |W La )]

[0037] Among them G 2 (A r0 |W La )=G(G(A r0 |W La )),…G ∞ =G(G(G……)) is the inference iteration, ∞ means the iteration reaches a stable A r The number of times;

[0038] Reverse reasoning by solving sG ∞ (A r |W La )=A r

[0039] Optimize the problem to get the initial emotional elements Get the initial element A r0 Then you can find the key triggers that trigger emotional outcomes.

[0040] In the second aspect, an embodiment of the present invention further provides a multimodal emotion recognition and prediction system, comprising a processor, an input device, an output device and a memory, wherein the processor, input device, output device and memory are interconnected, wherein the memory is used to store a computer program, and the computer program includes program instructions, and the processor is configured to call the program instructions to execute the method of the first aspect above.

[0041] By implementing the embodiments of the present invention, by analyzing the relevant data of human individuals, a model is established to identify and predict emotions, and results are obtained. The results are analyzed and interpreted by reverse reasoning, and the causes of the emotions are found, so that timely emotional intervention is beneficial to improve physical and mental health. Compared with the prior art, the present invention can not only further improve the accuracy of emotion recognition and prediction results, but more importantly, it can combine the multimodal data of human individuals to perform interpretable intelligent reasoning and learning to explain the mechanism of emotion generation, improve the reliability and credibility of emotion recognition and prediction results, and remove the obstacles of intelligent methods in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the drawings required for use in the specific implementation or the description of the prior art are briefly introduced below.

[0043] Figure 1 is a flow chart of a multimodal emotion recognition and prediction method provided by an embodiment of the present invention;

[0044] Figure 2 It is a network structure diagram;

[0045] Figure 3 This is an example diagram of the deep symbolic neural network and the logic part;

[0046] Figure 4 This is an example diagram of the deep symbolic neural network and arithmetic part;

[0047] Figure 5 It is a structural diagram of a multimodal emotion recognition and prediction system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0049] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0050] Please refer to Figure 1 , the multimodal emotion recognition and prediction method provided by the embodiment of the present invention includes the following steps:

[0051] S1, build a multimodal data sample library.

[0052] In the specific implementation, multimodal data collection is carried out to obtain physiological data, sound data, image data, and text data through smart bracelets, microphones, cameras, mobile phone apps, etc. to form a multimodal data sample library.

[0053] S2, build a logical library of daily life knowledge.

[0054] In specific implementation, a daily life knowledge logic library is constructed by referencing a third-party knowledge graph or extracting Wikipedia on the Internet. It should be noted that the inventor believes that daily life knowledge logic is necessary because changes in emotions may be related to events and elements in daily life.

[0055] S3, builds emotion recognition and prediction models, and performs training and reliability analysis on them.

[0056] Before introducing the model architecture, construction, training, and reliability analysis, the following description is given of data collection and model input and output:

[0057] (I) Regarding human individual identification and prediction model input data:

[0058] Where T is the length of the recorded time series, r is the time series number, and constructs the input data P r for:

[0059] P r =(t r , C r , I v，r , S r )

[0060] The timestamp t r (represents the time with time series number r, representing a period of fixed length, such as 14:00 to 15:00 on September 17, 2024); C r It is a collection of text data entered by human individuals on network terminals such as mobile phone APP or social network terminals during this period. v，r is the vibration image of this human individual collected by the terminal device during this period of time, S r It is a collection of physiological signals such as electrocardiogram collected by the terminal device during this period.

[0061] (II) Intermediate output process data of human individual recognition and prediction models:

[0062] By P rIdentify specific emotion-forming events and elements related to the knowledge base, which are collectively referred to as elements that induce emotion formation, and are referred to as emotion elements in the following. r =(a1, a2, ..., a m , b1,…,b n ), where m represents the total number of events and elements related to the expression in the knowledge base, and a i ∈[0, 1], describing the probability of the existence of emotional logic elements (events, elements), b j ∈[0, 1], represents the measurement of emotional elements (events, factors).

[0063] The events included in the emotional elements include daily life events, such as promotion, salary increase, death of relatives, etc. The elements included in the emotional elements include some physiological, psychological, and social elements, such as increased heart rate, feeling sad, friends visiting, etc. These elements and events can be obtained through physiological signals, text language, vibration images and other channels.

[0064] (III) Output data of human individual recognition and prediction models:

[0065] By P r and element A r , the identified emotion indicator E r =(e1, ····, e 10 ), where e1, ..., e 10 There are 10 emotional indicators, including: aggressiveness, stress, tension, suspicion, balance, depression, charm, vitality, self-discipline, neuroticism, and happiness. Each category has ten level indexes, represented by continuous values ​​0 to 1, [0, 0.1] represents level 1, (0.1, 0.2] represents level 2, ..., (0.9, 1] represents level 10. Based on this data description, the model also outputs the predicted set results of future emotional indicators. Where n represents the predicted future time step, that is, the emotion of the future time length Tn.

[0066] In this embodiment, the overall structure of the emotion recognition and prediction model is as follows: Figure 2 The model is composed of neural networks, which include deep convolutional neural network (CNN), long short-term memory (LSTM) neural network, Transformer network and deep symbolic neural network. The deep convolutional neural network is responsible for processing vibration image data. v，r , calculate the semantic encoding of the vibration image related to emotions, and the LSTM neural network is responsible for processing the physiological signal S r, calculate the physiological semantic codes related to emotions; the Transformer network (attention mechanism coding) is responsible for processing text data and calculating the text semantic codes related to emotions; and through the decoder, these codes are decoded into emotional elements, and the emotional elements are input into the deep symbolic neural network for calculation and reasoning, and the semantic codes are fused and decoded to obtain emotional index data. Among them, physiological elements and semantic elements are both high-dimensional vector embedding expressions.

[0067] It should be noted that in Figure 2 In the figure, the unspecified encoders and decoders are all fully connected neural networks FNN (emotional element decoding).

[0068] The following focuses on describing the deep symbolic neural network of the present application, whose neural unit calculation is defined as two parts: logical calculation and arithmetic calculation.

[0069] (I) For symbolic logic calculation, the structure is as follows Figure 3 Shown

[0070] Assume that x∈[0,1], y∈[0,1] are the preconditions, z∈[0,1], and z is the calculation result. Logical operations include AND, OR, and NOT. Any logic can be constructed by combining these three operations. The calculation formula for arithmetic logic operations is as follows:

[0071] (1) Seeking non- z = 1-x,

[0072] (2) Solving for ∧: z = x·y, or z = Π for multivariate operations i x i

[0073] (3) Find or ∨: z = x + yx y, in the case of multivariate operations, z = 1 - Π i (1-x i )

[0074] Depend on Figure 3 Structure forming symbolic operation unit function: in vec(.) is a matrix vectorization operation.

[0075] where x∈[0,1] p is the input, z∈[0,1] q is the result of the intermediate logic process, q is the logic capacity, is the given parameter, p is the input dimension, W L ∈{-1, 0, 1} q×p is the weight matrix.

[0076] Through multi-layer compounding, we can get a symbolic neural network

[0077]

[0078] in Network weights under logical reasoning of the network The constraint regularity is:

[0079] Sparse regularization of network weights

[0080] Numerical constraints on network weights That is, construct the regularization:

[0081]

[0082] (ii) For the arithmetic operation part, Figure 4 Shown

[0083] Using a similar structure, the polynomial neural network, whose single-layer network contains 1st order and 1.5th order input combinations are: in W a is the weight. By stacking this single-layer network, we can get the arithmetic part of the symbolic neural network, which is

[0084]

[0085] The neural network structure above can construct items with the highest even order being 2s, and the highest odd order being 3+2(s-2). s is a given parameter. The larger the value, the higher the polynomial order that can be expressed. The constraint on the arithmetic part of the network is the row group sparse constraint: where w Lji is the weight vector of the i-th row in the j-th layer of the network.

[0086] Therefore, the structure of a deep symbolic neural network is:

[0087]

[0088] Let x = (a1, ..., a m ), y=(b1,…,b n ), that is, the input is the emotional element. It should be noted that the symbolic neural network reasoning calculation adopts a closed-loop structure, that is, the reasoning result can be used as the input of the symbolic neural network for reasoning calculation again until a stable and unchanged result is obtained, and then output to the Transformer decoder to estimate and predict the emotional state E r , E r+1 , …, E r+n .

[0089] Furthermore, the model training and reverse reasoning analysis interpretation methods are explained as follows:

[0090] First, according to the daily knowledge base, logical reasoning knowledge and arithmetic operation knowledge are loaded, and the weight expression and Figure 3 The logic example in the above example converts the logical reasoning and arithmetic knowledge in the knowledge base into symbolic neural network weights. and Initialize the value. Then further adjust the generated symbol network through sample data training, and finally obtain the emotion recognition prediction model. The specific training process is as follows:

[0091] Assume that at time r, data P r The corresponding emotion is E r , we need to predict the emotional state of the next 10T, and the recognition model we built is

[0092]

[0093] in is the symbolic neural network weight, W s is the weight of other networks in the model. Then we need to construct the least squares loss function for training plus the weight penalty of the symbolic neural network.

[0094]

[0095] Input the training samples into the model and adjust the network weight parameters W s Minimize Loss (such as using stochastic gradient descent method), train to get the final applicable network model, and check after training. We can get some mechanism processes about emotional elements and emotional output. s By conducting analysis and model testing, we can obtain the contribution rate of factors and analyze which factors are the main factors inducing emotions.

[0096] The interpretable analysis of the model can be divided into two parts. The first part is the analysis of the main causes of emotions, that is, by analyzing the output E r With emotional elements A r The relationship between them can help us find out which events and elements are the main factors that induce emotions. The second part is the generation process of emotional elements, that is, through the reverse process of symbolic reasoning calculation, we can know the generation process of emotional elements (events, elements), so as to have an intuitive logical explanation for the induction of emotions.

[0097] For the analysis in the first part, assume that the model of the Transformer decoder is E r =F T (A r |W sT ),in Suppose we want to analyze a i ∈A r For E r Contribution rate Q i . Then you need to calculate it using the following formula: Where E is the expectation, which can be obtained by testing the sample output statistics. i is a vector with the same dimension as E r Same, indicating the value a i Q i The contribution rate of a. The larger the value in the vector, the greater the contribution rate of a. i The greater the effect on emotion induction.

[0098] For the symbol calculation part, it can be obtained by reverse calculation, that is, calculating A r0 =G -1 (A r |W La ), where A r0 is the initial emotional element, is the minimum set of initial emotional elements, i.e., the key emotional element, A r It is the stable result of symbolic neural network calculation iteration. Here, the reverse reasoning calculation model G is constructed. -1 . The loss function is defined as:

[0099] Loss inv =|A r0 |1+[A r -G ∞ (A r0 |W La )],

[0100] Among them G 2 (A r0 |W La )=G(G(A r0 |W La )),…G ∞ =G(G(G……)) is the inference iteration, ∞ means the iteration reaches a stable A r The reverse reasoning is solved by

[0101]

[0102] Optimize the problem to get the initial emotional elements Get the initial element A r0 Then you can find the key triggers that trigger emotional outcomes.

[0103] S4, collects multimodal signals of human individuals as input data.

[0104] The input data includes a set of text data input by a human individual in a network terminal within a preset time period, a set of vibration images of a human individual collected by a terminal device within a preset time period, and a set of physiological signals collected by a terminal within a preset time period.

[0105] S5, input the multimodal signal into the pre-trained and analyzed emotion recognition and prediction model for recognition and prediction to obtain the result.

[0106] S6, conduct reverse reasoning analysis and interpretation of the results to find the causes of the emotions.

[0107] From the above description, we can know that the present invention proposes an interpretable deep symbolic neural network perception reasoning prediction framework method based on multimodal data recognition and prediction of human individual emotions, and establishes an intelligent recognition prediction model through learning multimodal data such as vibration images, natural language, and physiological signals. The model recognizes multimodal signals as personal emotional elements, and uses them as input, combined with the symbolic neural network reasoning calculation of the daily emotional inducement knowledge model, to construct a deep symbolic neural network framework and form an interpretable reasoning network.

[0108] The present invention can not only further improve the accuracy of emotion recognition and prediction results, but more importantly, it can perform explainable intelligent reasoning and learning by combining multimodal data of human individuals to explain the mechanism of emotion generation, improve the reliability and credibility of emotion recognition and prediction results, and remove obstacles in the practical application of intelligent methods.

[0109] In addition, the method can accurately find the causes of emotions by combining reverse reasoning with feature contribution analysis through the proposed deep symbolic neural network architecture. Unlike the input feature association and contribution analysis used by most interpretable neural networks, the method proposed in this invention not only analyzes the input feature contribution, but also accurately identifies the key causes of emotions in an interpretable way based on the reverse reasoning method of the symbolic neural network, thus further improving the reliability and credibility of the intelligent model method, thereby enhancing its application value.

[0110] It should be emphasized that the deep symbolic neural network proposed in the present invention is not limited to the prediction method of emotion recognition, but can be used for all interpretable, credible and reliable neural network methods.

[0111] Based on the same inventive concept, the embodiment of the present invention also provides a multimodal emotion recognition and prediction system. Figure 5As shown, the system may include: one or more processors 101, one or more input devices 102, one or more output devices 103 and a memory 104, wherein the processors 101, the input devices 102, the output devices 103 and the memory 104 are interconnected via a bus 105. The memory 104 is used to store a computer program, wherein the computer program includes program instructions, and the processor 101 is configured to call the program instructions to execute the method of the above method embodiment.

[0112] It should be understood that in the embodiment of the present invention, the processor 101 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0113] The input device 102 may include a keyboard, etc., and the output device 103 may include a display (LCD, etc.), a speaker, etc.

[0114] The memory 104 may include a read-only memory and a random access memory, and provides instructions and data to the processor 101. A portion of the memory 104 may also include a non-volatile random access memory. For example, the memory 104 may also store information on the device type.

[0115] In a specific implementation, the processor 101, input device 102, and output device 103 described in the embodiment of the present invention can execute the implementation method described in the embodiment of the multimodal emotion recognition and prediction method provided in the embodiment of the present invention, which will not be repeated here.

[0116] It should be noted that the specific working process of this embodiment can be found in the aforementioned method embodiment part, which will not be described in detail here.

[0117] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A multimodal emotion recognition and prediction method, characterized in that: include: Collecting multimodal signals of human individuals as input data; the input data includes a text data set input by the human individual in a network terminal within a preset time period, a vibration image of the human individual collected by the terminal device within a preset time period, and a physiological signal set collected by the terminal within a preset time period; Inputting the multimodal signal into a pre-trained and analyzed emotion recognition and prediction model for recognition and prediction to obtain a result; Conduct reverse reasoning analysis and interpretation on the results to find out the causes of the emotions.

2. The method according to claim 1, characterized in that Before collecting the multimodal signals of the human individual, the method further includes: Build a multimodal data sample library; Build a logical library of daily life knowledge; Build emotion recognition and prediction models; Training the emotion recognition and prediction model based on the daily life knowledge logic library; Based on the sample data, the trained emotion recognition and prediction model is subjected to reliability analysis under reverse reasoning analysis interpretation.

3. The method according to claim 2, characterized in that The specific steps of building a multimodal data sample library are as follows: Multimodal data is collected through smart bracelets, microphones, cameras or mobile phone APPs to form a multimodal data sample library; the multimodal data includes physiological data, sound data, image data and text data.

4. The method according to claim 2, characterized in that The specific steps to build a knowledge base of daily life logic are: Build a logical library of daily life knowledge by referencing third-party knowledge graphs or extracting online Wikipedia.

5. The method according to claim 2, characterized in that The trained emotion recognition and prediction model is subjected to reliability analysis under reverse reasoning analysis and interpretation, specifically: By extracting and testing sample data in the multimodal data sample library, analyzing the output model's explanatory mechanism, and comparing it with the human knowledge in the daily life knowledge logic library, the reliability of the model's mechanism is verified.

6. The method according to claim 2, characterized in that The input data is denoted as P r , the text data set is denoted as C r , the vibration image is recorded as I v,r , the physiological signal set is denoted as S r ; The architecture of the emotion recognition and prediction model includes a deep convolutional neural network, a long short-term memory neural network, a Transformer network, and a deep symbolic neural network; Deep convolutional neural network for processing vibration image data I v,r , calculate the semantic coding of the vibration image related to emotions; Long short-term memory neural network for processing physiological signals S r , calculate the physiological semantic codes related to emotions; Transformer network, used to process text data set C r , calculate the text semantic coding related to emotions, decode the text semantic coding into emotional elements through a decoder, input the emotional elements into a deep symbolic neural network, calculate reasoning, and fuse and decode the semantic coding to obtain emotional indicator data.

7. The method according to claim 6, characterized in that The neural unit of the deep symbolic neural network includes a logical calculation part and an arithmetic calculation part, and adopts a closed-loop structure. The reasoning result of the deep symbolic neural network can be used as input to perform reasoning calculation again until a stable and unchanged result is obtained, and then output to the Transformer network to estimate and predict the emotional state E r ,E r+1 ,…,E r+n .

8. The method according to claim 6, characterized in that The emotion recognition and prediction model is trained specifically as follows: According to the daily life knowledge logic library, the logical reasoning knowledge and arithmetic operation knowledge are loaded, and the logical reasoning and arithmetic knowledge in the knowledge library are converted into the weights of the symbolic neural network according to the weight expression of polynomials and the logic examples in the deep symbolic neural network. and The initialization value of Through the sample data in the multimodal data sample library, the generated weights are adjusted and Finally, the emotion recognition prediction model is obtained.

9. The method according to claim 2, characterized in that The interpretable analysis of the trained emotion recognition and prediction model includes a first part and a second part; The first part is an analysis of the main causes of emotions, specifically: Assume that the model of the Transformer network is E r =F T (A r |W sT ),in A r For emotional elements; Suppose we want to analyze a i ∈A r For E r Contribution rate Q i , you need to calculate it by the following formula: E is the expectation, which can be obtained by testing the sample output statistics, Q i is a vector with the same dimension as E r Same, indicating the value a i Q i The contribution rate of a. The larger the value in the vector, the greater the contribution rate of a. i The greater the effect on emotion induction; The second part is the generation process of emotional elements, specifically: Define the loss function as Loss inv =|A r0 |1+[a r -G ∞ (A r0 |W La )] Among them G 2 (A r0 |W La )=G(G(A r0 |W La )),…G ∞ =G(G(G……)) is the inference iteration, ∞ means the iteration reaches a stable A r The number of times; Reverse reasoning by solving Optimize the problem to get the initial emotional elements Get the initial element A r0 Then you can find the key triggers that trigger emotional outcomes.

10. A multimodal emotion recognition and prediction system, characterized in that: The method comprises a processor, an input device, an output device and a memory, wherein the processor, the input device, the output device and the memory are interconnected, wherein the memory is used to store a computer program, the computer program comprises program instructions, and the processor is configured to call the program instructions to execute the method according to any one of claims 1 to 9.

Images