Emotion recognition method and system based on evidence theory multi-modal data decision fusion

Abstract

The invention provides an emotion recognition method and system based on evidence theory multi-modal data decision fusion. According to the method, two cluster centers of each piece of data are clustered through a PCM clustering algorithm, and the two cluster centers are used as an upper bound and a lower bound to construct an interval number model, so that the effect of the interval number model in a noise environment is improved. And a recognition framework is constructed, and the support degree of data features is calculated according to the interval length, so that the accuracy of the generated BPA during classification is improved. The BPA of each sample is calculated by using the similarity and the feature support degree of the test sample and the interval number model, and then evidence combination is carried out through a Dempster combination rule. Secondly, the conflict between the evidences is judged by using a Jousseme distance, then the similarity between the evidences is calculated to construct a similarity adjacent matrix, then the relationship strength between the evidences is defined by using the sum of the similarities, the weight of the conflict evidences is calculated and modified by integrating the direct relationship and the indirect relationship, and the original evidence body is subjected to weighted correction through the overall weight; and finally, combining through a Dempster synthesis formula to obtain a final result.

Description

Technical Field

[0001] The present invention belongs to the technical field of information fusion, and in particular relates to a multimodal data decision fusion method and system based on evidence theory. Background Art

[0002] Information fusion involves multiple disciplines and is one of the key technologies that has received widespread attention and has great influence in recent years. It refers to the integration and processing of information from different sources and in different forms to extract more comprehensive, accurate and useful information. However, in the real environment, various interferences and uncertainties are everywhere, which makes the data detected by sensors have great uncertainty, which has a huge impact on the accuracy of information fusion.

[0003] At the same time, before applying the theory of evidence, determining BPA is also a complex process. It not only requires evaluating the reliability and weight of the evidence, but also determining the measure of uncertainty for each piece of evidence. The determined BPA needs to be tested and adjusted. In most cases, it is necessary to rely on expert opinions, and there is currently no unified and reasonable method for determining BPA.

[0004] DS evidence theory (Dempster-Shafer Evidence Theory) is a mathematical theory used to deal with incomplete and uncertain information. After being extended and applied to the field of data fusion, it has unique advantages in how to reasonably and effectively fuse multi-source uncertain information.

[0005] Therefore, the present invention designs a multimodal data decision fusion method based on evidence theory to solve the problem of determining BPA and the fusion between multiple modalities, and applies it to the field of emotion recognition. Summary of the invention

[0006] In view of the above research status and existing problems, the present invention provides a multimodal data decision fusion method based on evidence theory to solve the basic probability assignment (BPA) problem existing in the application of evidence theory and the fusion paradox problem caused by high data conflict.

[0007] Aiming at the basic probability assignment (BPA) problem when applying evidence theory, the present invention adopts the PCM clustering algorithm to obtain the upper and lower bounds of the interval number to construct a single proposition interval number model, and constructs a compound proposition interval number model through interval intersection. The support of data features is calculated using interval length, and combined with interval similarity to generate BPA.

[0008] Aiming at the fusion paradox problem caused by high data conflict, the present invention uses the Jousselme distance formula to calculate the evidence distance, then constructs the evidence similarity matrix, calculates the evidence weight by comprehensively considering the direct and indirect relationship of the evidence, and applies the Dempster combination rule to combine the evidence after weighted correction of the original evidence.

[0009] For the above problems and solutions thereof, the detailed technical solution of the present invention is: a multimodal data decision fusion method based on evidence theory, the steps are as follows:

[0010] First, assume that the data set is m data with c types and n features, Class = {1,2,…,c}, Feature = {1,2,…,n}. In emotion recognition, Class can be defined as positive emotion, negative emotion, and no emotion, etc.; Feature is the feature used to identify emotions, such as eyes, mouth, etc. in facial recognition; in audio recognition, it can be the frequency of sound, etc. Assuming c = 3, n = 4, then in the BPA generation of evidence theory, the recognition framework is Θ = {{C1}, {C2}, {C3}, {C1,C2}, {C1,C3}, {C2,C3}, {C1,C2,C3}}. Among them, a set with only one element is a single proposition, such as {C1}; a set with more than one element is a compound proposition, such as {C1,C2} or {C1,C2,C3}.

[0011] S1. Dataset division: Split the original emotion recognition dataset into training data and test data according to a certain ratio from each type of each feature;

[0012] S2. Use PCM clustering to obtain an interval number model: Use the PCM clustering algorithm to cluster two clusters on the training set of each class of each feature, obtain two cluster centers and determine the interval number model of a single proposition; after determining the interval number model of a single proposition, use the interval number intersection formula to calculate the interval number intersection of each element in the compound proposition to obtain the interval number model of the compound proposition, and then combine the single proposition interval model to obtain a complete interval number model;

[0013] S3. Calculate the similarity between the test sample and the model: In the test data set, combine the interval number model obtained in the previous step, use the interval number distance formula to calculate the distance between the test sample and the model, and then calculate the similarity between the test sample and the model based on the distance;

[0014] S4. Calculate feature support: According to the different support of different features for classification, introduce and calculate feature support;

[0015] S5, BPA generation: In the recognition framework, the product of interval similarity and feature support is normalized to generate the BPA of each attribute, i.e., the evidence source;

[0016] S6. Measure evidence conflict and construct similarity matrix: After BPA is generated, the distance between evidence is calculated to quantify the degree of conflict. The larger the distance value, the higher the conflict. Then, the similarity relationship between evidence is obtained based on the evidence distance to construct the similarity matrix.

[0017] S7, calculating the overall weight of the modified evidence: calculating the direct relationship weight of the modified evidence and the indirect relationship coefficient and weight of the modified evidence, and then integrating the direct relationship modification weight and the indirect relationship modification weight to obtain the overall weight;

[0018] S8. Fusion of weighted modified evidence sources: Modify the evidence sources by weighted averaging and fuse the modified evidence sources. Finally, use the Dempster synthesis rule to combine the modified evidence sources in pairs to obtain the BPA of each type. Select the type with the largest BPA as the recognition result.

[0019] Furthermore, in step S1, it is assumed that the original emotion recognition dataset is m data with c types and n features, Class = {1,2,…,c}, Feature = {1,2,…,n}; a set element with only one type is a single proposition, and a set element with more than one type is a compound proposition; the recognition framework in step S5 is a set consisting of single propositions and compound propositions.

[0020] Furthermore, the specific implementation method of clustering the interval number model in S2 is as follows:

[0021] S21, use the PCM algorithm to cluster two clusters on the training set of each category of each feature, and obtain two cluster centers P cn1 ,P cn2 Similarly, two cluster centers can be obtained for other classes and features respectively. The cluster centers are converted into a single proposition interval number model by formula (1):

[0022] IN cn =[min(P cn1 ,P cn2 ),max(P cn1 ,P cn2 )]=[P cn-min ,P cn-max ] (1)

[0023] Among them, IN cn represents the number of intervals of the cth category under the nth feature; P cn-min Indicates P cn1 ,P cn2 The smaller one, P cn-max It means the larger one;

[0024] S22, after determining the interval number of a single proposition, use the interval number intersection formula (2) to calculate the interval number intersection of two elements in the compound proposition. For the compound proposition {C1, C2}, its interval number is formula (3), where IN 1,21 represents the intersection of the interval number models of proposition C1 and proposition C2 in the first feature data; max(·) is a function that takes the larger of the two numbers. When a1>b2 or b1>a2, Get the interval number of each compound proposition and determine the interval number model of the compound proposition; similarly, the interval number model of other compound propositions can be calculated;

[0025] A∩B=[max(a1,b1),max(a2,b2)] (2)

[0026] IN 1,21 =[max(P 11-min ,P 21-min ),max(P 11-max ,P 21-max )] (3)

[0027] S23, combining the interval number model of single proposition and the interval number model of compound proposition to obtain a complete interval number model.

[0028] Furthermore, the specific implementation method of calculating the similarity between the test sample and the model in S3 is as follows:

[0029] In the test data set, select a test sample (a1, a2, ..., a k ), k represents the number of features of the sample, and then each number corresponds to a special interval number under a feature, and the test instance interval is converted into A o =([a1,a1],[a2,a2],…,[a k ,a k ]), i = 1, ..., k, use the interval number distance formula (4) to calculate the distance between the test sample and the model, as shown in formula (5); IN cn-max and IN cn-min The interval model IN represents the nth feature of the cth proposition respectively. cn The upper and lower bounds of are:

[0030]

[0031] Then use the interval number similarity formula (6) to calculate the similarity between the test sample and the model, as shown in formula (7):

[0032]

[0033] Among them, A = [a1, a2] and B = [b1, b2] are two interval numbers, and the support coefficient α is a constant.

[0034] Furthermore, in step S4, the sum of the lengths of the intervals of each proposition under each feature is calculated using formula (8): n , where N Θ represents the number of propositions of the recognition frame Θ;

[0035]

[0036] IN cn-max and IN cn-min The interval model IN represents the nth feature of the cth proposition respectively. cn The upper and lower bounds of ; then, calculate its feature support, as shown in formula (9):

[0037]

[0038] Among them, Sup n Indicates the support of the nth feature; F n Indicates the number of features.

[0039] Furthermore, the specific implementation method of BPA generation in S5 is as follows:

[0040] In S3 and S4, the interval similarity and feature support have been calculated, and their product is normalized to generate the BPA of each attribute, as shown in formula (10):

[0041]

[0042] Among them, m(X cn ) represents the generated BPA quality function, X cn The cth proposition representing the nth feature, Sup n represents the support of the nth feature; S(A i ,IN cn ) represents the similarity between the test sample and the model.

[0043] Furthermore, the specific implementation method of constructing the similarity matrix in S6 is as follows:

[0044] After BPA is generated, the distance between evidences is calculated using the Jousselme distance formula (11) to quantify the degree of conflict. The larger the distance value, the higher the conflict. Based on the evidence distance, the similarity calculation formula (12) is used to obtain the similarity relationship between evidences and construct an evidence similarity adjacency matrix, which is defined as follows:

[0045]

[0046]

[0047] Sim(m i ,m j )=1-d ij (m i ,m j ) (13)

[0048] in, and represents two row vectors of evidence, D is 2 N ×2 N N refers to the number of elements in the recognition frame Θ, D(A,B) represents the ratio of the number of elements in the intersection and union of sets A and B, that is, their cardinality ratio; |·| represents the cardinality of the set, A,B∈2 Θ , for a given recognition frame Θ, its power set is the set consisting of all subsets of Θ, denoted as 2 Θ .

[0049] Furthermore, the specific implementation method of calculating the overall weight of the modification evidence in S7 is as follows:

[0050] S71, calculate the direct relationship weight of the modified evidence: the relationship strength of the evidence is the sum of the similarities between the evidence body and all other evidence bodies, as shown in formula (14). The relationship strength is standardized to obtain the direct relationship weight between the evidence, as shown in formula (15): where N i represents the other evidence after removing evidence i, RS i The strength of the relationship for the evidence;

[0051]

[0052] S72, then calculate the indirect relationship coefficient and weight of the modified evidence: first, by removing a specific body of evidence from the original evidence set, a new evidence set with a new relationship network is formed; then, the relationship strength between the evidence bodies in the original evidence set and the new evidence set is compared, where the new evidence set refers to the data set after the specific body of evidence is removed; the indirect relationship is quantified using the relationship coefficient calculation formula, as shown in formula (16); where RS j is to remove m i Front m j The strength of the relationship, is to remove m i After m j strength of relationship;

[0053]

[0054] Then through the indirect relationship coefficient α of the evidence i Calculate the indirect relationship weight, as shown in formula (17):

[0055]

[0056] S73, calculate the overall weight of the modification evidence: integrate the direct relationship modification weight and the indirect relationship modification weight to obtain the overall weight, as shown in formula (18):

[0057]

[0058] Furthermore, the specific implementation method of fusing the weighted modified evidence sources in S8 is as follows:

[0059] The weighted average evidence is obtained by weight distribution of formula (19), and then the weighted modified evidence m′ of the original evidence source is obtained by formula (20): i ;Finally, the Dempster synthesis formula is used on the modified evidence source to obtain the BPA of each category, and the category with the largest BPA is selected as the identification result;

[0060]

[0061] The present invention also provides an emotion recognition system based on evidence theory multimodal data decision fusion, comprising:

[0062] A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute an emotion recognition method based on evidence theory multimodal data decision fusion as described in the above technical solution.

[0063] Compared with the prior art, the present invention has the following advantages:

[0064] (1) Before applying the theory of evidence, the basic probability assignment problem of each piece of information is determined, and the defects of existing methods in processing noisy data are addressed. The PCM clustering and interval number model methods are used to generate BPA to solve the problem of BPA generation in noisy environments in the real world. The present invention clusters two cluster centers of each data through the PCM clustering algorithm, and uses them as upper and lower bounds to construct an interval number model, thereby improving the effect of the interval number model in a noisy environment. A recognition framework is constructed to calculate the support of data features based on the interval length, thereby improving the accuracy of the generated BPA during classification. The BPA of each sample is calculated using the similarity and feature support of the test sample and the interval number model, and then the evidence is combined using the Dempster combination rule.

[0065] (2) When applying evidence theory, the high conflict problem between multi-source data and the paradoxical problem that may be caused by the fusion of high-conflict data are targeted. The present invention uses an improved decision fusion method of evidence theory. The present invention intends to use the Jouselme distance to judge the conflict between evidences, and then calculate the similarity between evidences to construct a similarity adjacency matrix, and then use the sum of similarities to define the strength of the relationship between evidences, and calculate the weight of modifying the conflicting evidence by combining direct and indirect relationships. The original evidence body is weighted and modified through the overall weight, and finally the final result is obtained through the Dempster synthesis formula combination. The paradoxical situation when the evidence theory fuses high-conflict data is eliminated to solve the problem of high-conflict data fusion, thereby improving the accuracy of information fusion decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0067] Figure 1 It is a flow chart of a method according to an embodiment of the present invention.

[0068] Figure 2 It is a flow chart of BPA generation in an embodiment of the present invention.

[0069] Figure 3 It is a flow chart of the decision fusion method for improving evidence theory in an embodiment of the present invention.

[0070] Figure 4 It is a model diagram of PCM clustering and interval number in an embodiment of the present invention. DETAILED DESCRIPTION

[0071] The present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0072] In order to solve the basic probability assignment (BPA) problem and the fusion paradox caused by high data conflict when applying evidence theory, this embodiment proposes an improved multimodal data decision fusion method based on evidence theory. Figure 1 The flowchart shown specifically includes the following steps:

[0073] S1. Dataset division: Split the original dataset into training data and test data in a certain proportion from each class of each feature according to the label.

[0074] S2. Use PCM clustering to obtain an interval number model: Use the PCM clustering algorithm to cluster two clusters on the training set of each class of each feature, obtain two cluster centers and determine the interval number model of a single proposition. After determining the interval number model of a single proposition, use the interval number intersection formula to calculate the interval number intersection of each element in the compound proposition to obtain the interval number model of the compound proposition. Then combine the single proposition interval model to obtain a complete interval number model.

[0075] S3. Calculate the similarity between the test sample and the model: In the test data set, combined with the interval number model obtained in the previous step, use the interval number distance formula to calculate the distance between the test sample and the model, and then use the distance to calculate the similarity between the test sample and the model.

[0076] S4. Calculate feature support: Considering that different features have different supports for classification, feature support is introduced and calculated.

[0077] S5. BPA generation: In the recognition framework, the product of interval similarity and feature support is normalized to generate the BPA of each attribute, i.e., evidence (source).

[0078] S6. Measure evidence conflict and construct similarity matrix: After BPA is generated, the distance between evidence is calculated to quantify the degree of conflict. The larger the distance value, the higher the conflict. Then, based on the evidence distance, the similarity relationship between evidence is obtained to construct the similarity matrix.

[0079] S7. Calculate the overall weight of the modified evidence: Calculate the direct relationship weight of the modified evidence and the indirect relationship coefficient and weight of the modified evidence. Then integrate the direct relationship modification weight and the indirect relationship modification weight to obtain the overall weight.

[0080] S8. Fusion of weighted modified evidence sources: Modify the evidence sources by weighted average and fuse the modified evidence sources. Finally, the modified evidence sources are combined in pairs using the Dempster synthesis rule to obtain the BPA of each type, and the type with the largest BPA is selected as the recognition result.

[0081] The following is an introduction to each module.

[0082] (1) BPA generation module

[0083] The BPA generation module is to pave the way for the subsequent application of the improved evidence theory decision fusion method. Its main task is to quantify the relationship between evidence and uncertainty, which affects the accuracy and effectiveness of subsequent information fusion and provides a solid foundation for the subsequent application of evidence theory. Figure 2 shown.

[0084] In existing studies, the determination of BPA has always been a complex process, which requires not only evaluating the reliability and weight of evidence, but also determining the measure of uncertainty for each piece of evidence. The determined BPA needs to be tested and adjusted, and in most cases, it needs to rely on expert opinions.

[0085] This embodiment adopts the method of PCM clustering and interval number construction model, such as Figure 4 Assume that the data set is m data with c types and n features, Class = {1, 2, ..., c}, Feature = {1, 2, ..., n}.

[0086] In this embodiment, multimodal emotion recognition is used as the research field, and Class is defined as sad emotion, neutral emotion, happy emotion, etc.; Feature is the feature used to identify emotions, for example, in facial recognition, it can be eyes, mouth, etc.; in audio recognition, it can be sound frequency, etc.; in text recognition, it can be text semantics, etc.

[0087] In this embodiment, taking the facial recognition modality as an example, let c = 3, n = 4, C1, C2, C3 are sad emotions, neutral emotions and happy emotions respectively; n1, n2, n3, n4 are eyes, mouth, eyebrows and nose respectively. Then in the BPA generation of evidence theory, the recognition framework is Θ = {{C1}, {C2}, {V3}, {C1, C2}, {C1, C3}, {C2, C3}, {C1, C2, C3}}. Among them, the set with only one element is a single proposition, such as {C1}; the set with more than one element is a compound proposition, such as {C1, C2} or {F1, C2, C3}.

[0088] After feature extraction, the dataset is divided into training and test data according to a certain proportion from the sadness, neutrality and happiness emotions in the features of eyes, mouth, eyebrows and nose. For example, 50% of the data from Class1 of Feature1 is taken as the training set of the first class of the first feature, and finally combined into an overall dataset. The training set is used to build the model, and the test set is used to evaluate the accuracy of the proposed algorithm.

[0089] Using the PCM algorithm, the eye and mouth features are clustered into two clusters in terms of sadness, and two cluster centers P cn1 ,P cn2 Similarly, two cluster centers can be obtained for other classes and features respectively. The cluster centers are converted into a single proposition interval number model using formula (1):

[0090] IM cn =[min(P cn1 ,P cn2 ),max(P cn1 ,Pcn2 )]=[P cn-min ,P cn-max ] (1)

[0091] Among them, IN cn represents the number of intervals of the cth category under the nth feature; P cn-min Indicates P cn1 ,P cn2 The smaller one, P cn-max It indicates the larger one.

[0092] After determining the interval number of a single proposition, the interval number intersection formula (2) is used to calculate the interval number intersection of two elements in a compound proposition. Taking {C1, C2} as an example, its interval number is formula (3), where IN 1,21 represents the intersection of the interval number model of sadness and neutral emotions in the eye feature data; max(·) is a function that takes the larger of the two numbers. When a1>b2 or b1>a2, Similarly, the interval number models of other complex propositions can be calculated.

[0093] A∩B=[max(a1,b1),max(a2,b2)] (2)

[0094] IN 1,21 =[max(P 11-min ,P 21-min ),max(P 11-max ,P 21-max )] (3)

[0095] The complete interval number model is obtained by combining the single proposition interval number model and the compound proposition interval number model.

[0096] In the test data set, a test sample (5.1, 3.5, 1.4, 0.2) with the category of sadness is selected as an example, and then each sample corresponds to a special interval number under a feature (specially with the same upper and lower bounds). The test instance interval is converted into A i =([5.1,5.1],[3.5,3.5],[1.4,1.4],[0.2,0.2]), and use the interval number distance formula (4) to calculate the distance between the test sample and the model, as shown in formula (5). Taking the test sample interval [5.1,5.1] of eye features as an example, assuming the interval model [4.716,5.333] of proposition {C1}, calculate the distance between the test sample and the model.

[0097]

[0098] Then, the similarity between the test sample and the model is calculated using the interval number similarity formula (6), as shown in formula (7). According to expert judgment, the support coefficient α of this embodiment is 5. Wherein, A = [a1, a2] and B = [b1, b2] are two interval numbers.

[0099]

[0100] In the previous step, the interval number model of each feature in the recognition framework has been calculated. Considering the different support of different features for classification, feature support is introduced. The interval length of the interval number reflects the uncertainty of the interval number. The longer the interval length, the higher the uncertainty of the information. The higher the uncertainty, the lower the support of the feature for classification, and vice versa. Therefore, to calculate the support of the feature, the sum of the interval number lengths of each proposition under each feature should be calculated using formula (8) L n , where N Θ Represents the number of propositions of the recognition frame Θ.

[0101]

[0102] Then, the feature support is calculated as shown in formula (9). n Indicates the support of the nth feature; F n Indicates the number of features.

[0103]

[0104] In the first two steps, the interval similarity and feature support have been calculated, and their products are normalized to generate the BPA of each attribute, where m(X cn ) represents the generated BPA quality function, X cn The cth proposition of the nth feature is represented by formula (10):

[0105]

[0106] (2) Improved decision fusion method based on evidence theory

[0107] This part is to solve the problem that when there is high conflict in the data, if the high conflict of the data is not considered, the fusion result may violate common sense if it is directly fused, thereby improving the accuracy of information fusion decision-making. Figure 3 shown.

[0108] In this embodiment, it is assumed that there are four modalities m1-m4, namely text modality, facial modality, audio modality, and gesture modality. In the previous module, this embodiment has obtained the BPA of one of the modalities in each category. Similarly, the BPA of other modalities in each category can also be calculated. For the convenience of understanding and calculation, it is assumed that the recognition framework is {C1, C2, C3}, and the BPA of the four modalities m1-m4 for emotion recognition of the same data is as follows.

[0109]

[0110] Then, the distance between evidences is calculated using the Jousselme distance formula (12) to quantify the degree of conflict. The larger the distance value, the higher the conflict. Based on the evidence distance, the similarity calculation formula (13) is used to obtain the similarity relationship between evidences and construct an evidence similarity adjacency matrix:

[0111]

[0112] Sim(m i ,m j )=1-d ij (m i ,m j ) (13)

[0113] in, and represents two row vectors of evidence, D is 2 N ×2 N , N refers to the number of elements in the recognition frame Θ, D(A,B) represents the ratio of the number of elements in the intersection and union of sets A and B, that is, their cardinality ratio. Sets A and B here refer to specific focal elements. Sets A and B are the basic elements that constitute the proof body. The proof body is composed of all such focal elements and their corresponding BPAs; |·| represents the cardinality of the set, A,B∈2 Θ For a given recognition frame Θ, its power set is the set consisting of all subsets of Θ, denoted as 2 Θ .

[0114]

[0115] Next, the direct relationship weight of the modified evidence is calculated. The relationship strength of the evidence is the sum of the similarities between the evidence body and all other evidence bodies, as shown in formula (14). The direct relationship weight between the evidence is obtained by normalizing the relationship strength, as shown in formula (15). i represents the other evidence after removing evidence i, RS i The strength of the relationship for the evidence.

[0116]

[0117] RS2=0.9981; RS3=2.0801; RS4=2.0822.

[0118]

[0119] Calculate the indirect relationship coefficient and weight of the modified evidence. First, by removing a specific body of evidence from the original evidence set, a new evidence set with a new relationship network is formed. Then, the relationship strength between the evidence bodies in the original evidence set (when it contains specific evidence) and the new evidence set (when it does not contain specific evidence) is compared. This difference provides clues about the indirect relationship. In order to quantify this indirect relationship, a specific relationship coefficient calculation formula is used, such as formula (16). Among them, RS j is to remove m i Front m j The strength of the relationship, is to remove m i After m j strength of relationship.

[0120]

[0121] α1=0.6250; α2=1.0006; a3=0.6400; α4=0.6393.

[0122] Then through the indirect relationship coefficient α of the evidence i Calculate the indirect relationship weight as shown in formula (17).

[0123]

[0124] Calculate the overall weight of the modification evidence. Integrate the direct relationship modification weight and the indirect relationship modification weight to obtain the overall weight, as shown in formula (18):

[0125]

[0126]

[0127] The weighted average evidence is obtained by weight distribution of formula (19), and then the weighted modified evidence m′ of the original evidence source is obtained by formula (20): i .

[0128]

[0129] m AVE (I) = 0.6132; m AVE (II) = 0.1801; m AVE (III) = 0.2068.

[0130]

[0131] m′1(C1)=0.6629m′1(C2)=0.1588m′1(C3)=0.1588

[0132] m′2(C1)=0.4904m′2(C2)=0.2441m′2(C3)=0.2655

[0133] m′3(C1)=0.6363m′3(C2)=0.1454m′3(C3)=0.2183

[0134] m′4(C1)=0.6630m′4(C2)=0.1801m′4(C3)=0.1649

[0135] Finally, the Dempster synthesis formula is used on the modified evidence source to obtain the BPA of each class. The class with the largest BPA is selected as the recognition result. In this embodiment, the recognition result can be C1: sad emotion.

[0136] m(C1)=0.9809; m(C2)=0.0069; m(C3)=0.0122.

[0137] The present invention also provides an emotion recognition system based on evidence theory multimodal data decision fusion, comprising:

[0138] A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute an emotion recognition method based on evidence theory multimodal data decision fusion as described in the above technical solution.

[0139] The contents of the above method embodiments are all applicable to the present system embodiments. The functions specifically implemented by the present system embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0140] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. An emotion recognition method based on multimodal data decision fusion based on evidence theory, characterized in that: The steps include: S1. Dataset division: Split the original emotion recognition dataset into training data and test data according to a certain ratio from each type of each feature; S2. Use PCM clustering to obtain an interval number model: Use the PCM clustering algorithm to cluster two clusters on the training set of each class of each feature, obtain two cluster centers and determine the interval number model of a single proposition; after determining the interval number model of a single proposition, use the interval number intersection formula to calculate the interval number intersection of each element in the compound proposition to obtain the interval number model of the compound proposition, and then combine the single proposition interval model to obtain a complete interval number model; S3. Calculate the similarity between the test sample and the model: In the test data set, combine the interval number model obtained in the previous step, use the interval number distance formula to calculate the distance between the test sample and the model, and then calculate the similarity between the test sample and the model based on the distance; S4. Calculate feature support: According to the different support of different features for classification, introduce and calculate feature support; S5, BPA generation: In the recognition framework, the product of interval similarity and feature support is normalized to generate the BPA of each modal attribute, i.e., the source of evidence; S6. Measure evidence conflict and construct similarity matrix: After BPA is generated, the distance between evidence is calculated to quantify the degree of conflict. The larger the distance value, the higher the conflict. Then, the similarity relationship between evidence is obtained based on the evidence distance to construct the similarity matrix. S7, calculating the overall weight of the modified evidence: calculating the direct relationship weight of the modified evidence and the indirect relationship coefficient and weight of the modified evidence, and then integrating the direct relationship modification weight and the indirect relationship modification weight to obtain the overall weight; S8. Fusion of weighted modified evidence sources: Modify the evidence sources by weighted averaging and fuse the modified evidence sources. Finally, use the Dempster synthesis rule to combine the modified evidence sources in pairs to obtain the BPA of each type. Select the type with the largest BPA as the recognition result.

2. The emotion recognition method based on evidence theory multimodal data decision fusion as claimed in claim 1, characterized in that: In step S1, it is assumed that the original emotion recognition dataset is m data with c types and n features, Class = {1,2,…,c}, Feature = {1,2,…,n}; a set element with only one type is a single proposition, and a set element with more than one type is a compound proposition; the recognition framework in step S5 is a set consisting of single propositions and compound propositions.

3. The emotion recognition method based on evidence theory multimodal data decision fusion as claimed in claim 2, characterized in that: The specific implementation method of clustering interval number model in S2 is as follows: S21, use the PCM algorithm to cluster two clusters on the training set of each category of each feature, and obtain two cluster centers P cn1 ,P cn2 Similarly, two cluster centers are obtained for other classes and features respectively. The cluster centers are converted into a single proposition interval number model by formula (1): IN cn =[min(P cn1 ,P cn2 ),max(P cn1 ,P cn2 )]=[P cn-min ,P cn-max ] (1) Among them, IN cn represents the number of intervals of the cth category under the nth feature; P cn-min Indicates P cn1 ,P cn2 The smaller one, P cn-max It means the larger one; S22, after determining the interval number of a single proposition, use the interval number intersection formula (2) to calculate the interval number intersection of two elements in the compound proposition. For the compound proposition {C1, C2}, its interval number is formula (3), where IN 1,21 represents the intersection of the interval number models of proposition C1 and proposition C2 in the first feature data; max(·) is a function that takes the larger of the two numbers. When a1>b2 or b1>a2, Get the interval number of each compound proposition and determine the interval number model of the compound proposition; similarly, calculate and get the interval number model of other compound propositions; A∩B=[max(a1,b1),max(a2,b2)] (2) IN 1,21 =[max(P 11-min ,P 21-min ),max(P 11-max ,P 21-max )] (3) S23, combining the interval number model of single proposition and the interval number model of compound proposition to obtain a complete interval number model.

4. The emotion recognition method based on evidence theory multimodal data decision fusion as claimed in claim 1, characterized in that: The specific implementation method for calculating the similarity between the test sample and the model in S3 is as follows: In the test data set, select a test sample (a1, a2, ..., a k ), k represents the number of features of the sample, and then each number corresponds to a special interval number under a feature, and the test instance interval is converted into A i =([a1,a1],[a2,a2],…,[a k ,a k ]), i = 1, ..., k, use the interval number distance formula (4) to calculate the distance between the test sample and the model, as shown in formula (5); IN cn-max and IN cn-min The interval model IN represents the nth feature of the cth proposition respectively. cn The upper and lower bounds of are: Then use the interval number similarity formula (6) to calculate the similarity between the test sample and the model, as shown in formula (7): Among them, A = [a1, a2] and B = [b1, b2] are two interval numbers, and the support coefficient α is a constant.

5. The emotion recognition method based on evidence theory multimodal data decision fusion as claimed in claim 1, characterized in that: In step S4, the sum of the lengths of the intervals of each proposition under each feature is calculated using formula (8): n , where N Θ represents the number of propositions of the recognition frame Θ; IN cn-max and IN cn-min The interval model IN represents the nth feature of the cth proposition respectively. cn The upper and lower bounds of ; then, calculate its feature support, As shown in formula (9): Among them, Sup n Indicates the support of the nth feature; F n Indicates the number of features.

6. The emotion recognition method based on evidence theory multimodal data decision fusion as claimed in claim 1, characterized in that: The specific implementation method of BPA generation in S5 is as follows: In S3 and S4, the interval similarity and feature support have been calculated, and their product is normalized to generate the BPA of each attribute, as shown in formula (10): Among them, m(X cn ) represents the generated BPA quality function, X cn The cth proposition representing the nth feature, Sup n represents the support of the nth feature; S(A i ,IN cn ) represents the similarity between the test sample and the model.

7. The emotion recognition method based on evidence theory multimodal data decision fusion as claimed in claim 1, characterized in that: The specific implementation method of constructing the similarity matrix in S6 is as follows: After BPA is generated, the distance between evidences is calculated using the Jousselme distance formula (11) to quantify the degree of conflict. The larger the distance value, the higher the conflict. Based on the evidence distance, the similarity calculation formula (12) is used to obtain the similarity relationship between evidences and construct an evidence similarity adjacency matrix, which is defined as follows: Sim (m i ,m j )=1-d ij (m i ,m j ) (13) in, and represents two row vectors of evidence, D is 2 N ×2 N N refers to the number of elements in the recognition frame Θ, D(A,B) represents the ratio of the number of elements in the intersection and union of sets A and B, that is, their cardinality ratio; |·| represents the cardinality of the set, A,B∈2 Θ , for a given recognition frame Θ, its power set is the set consisting of all subsets of Θ, denoted as 2 Θ .

8. The emotion recognition method based on evidence theory multimodal data decision fusion as claimed in claim 1, characterized in that: The specific implementation method for calculating the overall weight of modification evidence in S7 is as follows: S71, calculate the direct relationship weight of the modified evidence: the relationship strength of the evidence is the sum of the similarities between the evidence body and all other evidence bodies, as shown in formula (14). The relationship strength is standardized to obtain the direct relationship weight between the evidence, as shown in formula (15): where N i represents the other evidence after removing evidence i, RS i The strength of the relationship for the evidence; S72, then calculate the indirect relationship coefficient and weight of the modified evidence: first, by removing a specific body of evidence from the original evidence set, a new evidence set with a new relationship network is formed; then, the relationship strength between the evidence bodies in the original evidence set and the new evidence set is compared, where the new evidence set refers to the data set after the specific body of evidence is removed; the indirect relationship is quantified using the relationship coefficient calculation formula, as shown in formula (16); where RS j is to remove m i Front m j The strength of the relationship, is to remove m i After m j strength of relationship; Then through the indirect relationship coefficient α of the evidence i Calculate the indirect relationship weight, as shown in formula (17): S73, calculate the overall weight of the modification evidence: integrate the direct relationship modification weight and the indirect relationship modification weight to obtain the overall weight, as shown in formula (18):

9. The emotion recognition method based on evidence theory multimodal data decision fusion as claimed in claim 1, characterized in that: The specific implementation method of fusing the weighted modified evidence sources in S8 is as follows: The weighted average evidence is obtained by weight distribution of formula (19), and then the weighted modified evidence m′ of the original evidence source is obtained by formula (20): i ;Finally, the Dempster synthesis formula is used on the modified evidence source to obtain the BPA of each category, and the category with the largest BPA is selected as the identification result; 10. An emotion recognition system based on multimodal data decision fusion based on evidence theory, characterized in that: include: A processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute an emotion recognition method based on evidence theory multimodal data decision fusion as described in any one of claims 1 to 9.

Images