Emotion type intelligent recognition method, device and equipment and storage medium

Abstract

The application relates to the technical field of emotion recognition, and provides an emotion type intelligent recognition method, device, equipment and storage medium, which can be used for recognizing the emotion type of a user when a customer service answers a question of the user in a financial field, wherein the method comprises the following steps: converting multi-modal data into N emotion signals, and matching a plurality of emotion types corresponding to each emotion signal; calculating the frequency of each emotion type in a database to obtain emotion frequency; calculating the weight of each emotion signal in the corresponding i emotion types to obtain i recognition degree weights; multiplying the i recognition degree weights by the corresponding emotion frequencies respectively to obtain i emotion scores; summing the emotion scores of the same emotion type in the i emotion scores to obtain M emotion type scores corresponding to M emotion types; and screening j target emotion types according to the M emotion type scores. The emotion frequency and the recognition degree weight can be adjusted to each other, and more accurate emotion scores are obtained.

Description

Technical Field

[0001] This application relates to the field of emotion recognition technology, such as methods, devices, equipment, and storage media for intelligent recognition of emotion types. Background Technology

[0002] With the rapid development of biometric technology, emotion recognition is increasingly being applied across numerous industries. For example, many users consult customer service via phone or video call regarding financial products and / or financial apps. By recognizing the user's emotions, customer service representatives can answer questions about financial products and / or apps, improving user satisfaction. Current emotion recognition methods primarily identify different emotions by matching facial expressions with corresponding emotions. Specifically, various facial expressions, facial features, and / or body movements corresponding to different emotions in video consultations are used as data. The emotions corresponding to these data are labeled, and the labeled data is used to train the emotion recognition model. However, because the user's environment during communication with customer service is complex, emotion recognition based on single-modal data suffers from low matching degrees between emotion signals and emotion categories, resulting in low accuracy. Summary of the Invention

[0003] This application provides an intelligent emotion type recognition method, device, equipment, and storage medium, aiming to solve the problem that emotion recognition based on single-modal data has a low matching degree between emotion signals and emotion categories, resulting in low accuracy of emotion recognition.

[0004] To solve the above problems, this application adopts the following technical solution:

[0005] This article provides an intelligent method for emotion type recognition, including:

[0006] Acquire multimodal data, convert the multimodal data into N emotion signals, and match multiple emotion types corresponding to each emotion signal, where N≥1;

[0007] Calculate the frequency of each emotion type in the database to obtain the emotion frequency;

[0008] Calculate the weight of each emotion signal in the corresponding i emotion types to obtain i recognition weights, where i≥1;

[0009] Multiply each of the i identification weights by its corresponding emotion frequency to obtain i emotion scores;

[0010] Summing the emotion scores of the same emotion type among the i emotion scores yields M emotion type scores corresponding to the M emotion types, where M≥1;

[0011] Based on the M emotion type scores, select j target emotion types, where 1≤j≤M.

[0012] Preferably, the step of selecting j target emotion types based on the M emotion type scores includes:

[0013] The M emotion type scores are sorted in descending order to obtain the emotion type sequence.

[0014] Filter out the first to the jth emotion type scores in the emotion type sequence to obtain j target emotion type scores;

[0015] The j target emotion type scores are defined as the j target emotion types.

[0016] Preferably, the step of calculating the weight of each emotion signal in the corresponding i emotion types to obtain i recognition weights includes:

[0017] Calculate the number of times each emotion signal appears in the i emotion types to obtain the number of times each of the i signals appears;

[0018] Add the occurrence counts of the i signals together to obtain the total number of signal occurrences;

[0019] Calculate the ratio of the number of occurrences of the first signal to the number of occurrences of the i-th signal to the total number of occurrences of the signal to obtain the i-th recognition weight.

[0020] Preferably, calculating the frequency of each emotion type in the database to obtain the emotion frequency includes:

[0021] The number of times each emotion type appears in the database is counted to obtain the emotion count.

[0022] The total number of times each of the aforementioned emotion types appears in the database is counted to obtain the total number of emotion occurrences.

[0023] The ratio of the number of emotional episodes to the total number of emotional episodes is taken as the emotional frequency.

[0024] Preferably, the matching of multiple emotion types corresponding to each emotion signal includes:

[0025] Retrieve all the aforementioned emotion types from the database;

[0026] The emotional types that have appeared with the emotional signals are statistically analyzed to obtain multiple emotional types.

[0027] Preferably, the step of summing the emotion scores of the same emotion type among the i emotion scores to obtain M emotion type scores corresponding to the M emotion types includes:

[0028] Count the number of emotion types corresponding to i emotion scores to obtain M emotion types;

[0029] The summation of all the emotion type scores corresponding to each of the M emotion types is obtained to get the M emotion type scores.

[0030] Preferably, the multimodal data includes any combination of two-dimensional face images, three-dimensional human body images, and speech audio.

[0031] This application also provides an intelligent emotion type recognition device, including:

[0032] An emotion type matching module is used to acquire multimodal data, convert the multimodal data into N emotion signals, and match multiple emotion types corresponding to each emotion signal, where N≥1;

[0033] The emotion frequency calculation module is used to calculate the frequency of each emotion type in the database to obtain the emotion frequency.

[0034] The identification weight calculation module is used to calculate the weight of each emotion signal in the corresponding i emotion types to obtain i identification weights, where i≥1;

[0035] The emotion score calculation module is used to multiply the i identification weights by the corresponding emotion frequencies to obtain i emotion scores.

[0036] The emotion type rating calculation module is used to sum the emotion ratings of the same emotion type among the i emotion ratings to obtain M emotion type ratings corresponding to M emotion types, where M≥1;

[0037] The target emotion type filtering module is used to filter j target emotion types based on M emotion type scores, where 1≤j≤M.

[0038] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the emotional type intelligent recognition method described in any of the above claims.

[0039] This application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the emotional type intelligent recognition method described in any of the above claims.

[0040] The intelligent emotion type recognition method of this application includes: acquiring multimodal data; converting the multimodal data into N emotion signals; matching each emotion signal with multiple emotion types, where N≥1; calculating the frequency of each emotion type in the database to obtain the emotion frequency; calculating the weight of each emotion signal in the corresponding i emotion types to obtain i recognition weights, where i≥1; multiplying the i recognition weights by their corresponding emotion frequencies to obtain i emotion scores; summing the emotion scores of the same emotion type among the i emotion scores to obtain M emotion type scores corresponding to M emotion types, where M≥1; and selecting j target emotion types based on the M emotion type scores, where 1≤j≤M. The N emotion signals obtained from multimodal data can comprehensively reflect the user's emotions. The more frequently an emotion signal appears, the lower the recognition weight corresponding to the emotion signal, since multiple emotion types may share the same emotion signal. The emotion score is calculated by using recognition weights and emotion frequency, allowing the emotion frequency (reflecting the number of times an emotion occurs) and the recognition weight (reflecting the recognition of emotion) to mutually adjust each other, resulting in a more accurate emotion score. The target emotion type selected based on emotion scores is more accurate. Attached Figure Description

[0041] Figure 1 This is a flowchart illustrating an embodiment of an intelligent emotion type recognition method.

[0042] Figure 2 This is a flowchart illustrating the process of selecting j target emotion types based on M emotion type scores, as an example.

[0043] Figure 3 This is a flowchart illustrating the process of calculating the weight of each emotion signal in the corresponding i emotion types, as an example.

[0044] Figure 4 This is a flowchart illustrating the process of calculating the frequency of each emotion type in a database, as shown in one embodiment.

[0045] Figure 5 This is a schematic diagram illustrating the process of matching multiple emotion types corresponding to each emotion signal in one embodiment.

[0046] Figure 6 This is a schematic block diagram of the structure of an intelligent emotion type recognition device according to one embodiment;

[0047] Figure 7 This is a schematic block diagram of the structure of a computer device according to one embodiment.

[0048] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0050] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of this application means the presence of features, integers, steps, operations, elements, units, cells, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, units, cells, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless couplings. The term “and / or” as used herein includes all or any of the units and all combinations thereof of one or more associated listed items.

[0051] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0052] Reference Figure 1 This is a flowchart illustrating the intelligent emotion type recognition method proposed in this application. This method can be used to identify user emotions during user consultations in the financial sector. Specifically, it acquires multimodal data from video or telephone consultations regarding financial products and / or financial apps, converts this data into emotion signals, calculates the emotion frequency of the corresponding emotion type, calculates the recognition weight of the emotion signal within the emotion type, calculates multiple emotion types based on the recognition weight and emotion frequency, selects the target emotion type from these multiple emotion types, recommends the target emotion type to the user, and judges the accuracy of the selected emotion type based on the user's choice, thereby improving user satisfaction during the process of providing consultation services for financial products and / or financial apps.

[0053] The intelligent emotion type recognition method includes the following steps S1-S6:

[0054] S1: Acquire multimodal data, convert the multimodal data into N emotion signals, and match multiple emotion types corresponding to each emotion signal, where N≥1.

[0055] The multimodal data includes any combination of two-dimensional face images, three-dimensional human body images, and speech audio. This application embodiment uses multimodal data including two-dimensional face images, three-dimensional human body images, and speech audio as an example.

[0056] For a two-dimensional face image, by extracting the set of key points from the two-dimensional face image, and performing expression analysis based on the set of key points, N1 emotion signals corresponding to the two-dimensional face image are obtained.

[0057] For a 3D human body image, by extracting the human body posture from the 3D human body image, emotion analysis is performed on the human body posture to obtain N2 emotion signals corresponding to the 3D human body image.

[0058] For speech audio, Mel frequency cepstral coefficients are obtained by performing Mel frequency analysis on the speech audio. The Mel frequency cepstral coefficients and speech audio are then input into a trained audio analysis model to obtain N3 emotion signals corresponding to the speech audio.

[0059] The relationship between the above N1 emotional signals, N2 emotional signals, N3 emotional signals and N emotional signals is N1+N2+N3=N.

[0060] The matching of multiple emotion types corresponding to each emotion signal includes:

[0061] Retrieve all the aforementioned emotion types from the database;

[0062] The emotional types that have appeared with the emotional signals are statistically analyzed to obtain multiple emotional types.

[0063] S2: Calculate the frequency of each emotion type in the database to obtain the emotion frequency.

[0064] The number of times each emotion type appears in the database is counted to obtain the emotion count.

[0065] The total number of times each of the aforementioned emotion types appears in the database is counted to obtain the total number of emotion occurrences.

[0066] The ratio of the number of emotional episodes to the total number of emotional episodes is taken as the emotional frequency.

[0067] The more frequently an emotion type appears in the database, the higher the frequency of the corresponding emotion. Conversely, the fewer frequently an emotion type appears in the database, the lower the frequency of the corresponding emotion.

[0068] S3: Calculate the weight of each emotion signal in the corresponding i emotion types to obtain i recognition weights, where i≥1.

[0069] Calculate the number of times each emotion signal appears in the i emotion types to obtain the number of times each of the i signals appears;

[0070] Add the occurrence counts of the i signals together to obtain the total number of signal occurrences;

[0071] Calculate the ratio of the number of occurrences of the first signal to the number of occurrences of the i-th signal to the total number of occurrences of the signal to obtain the i-th recognition weight.

[0072] Since the same emotional signal appears in all i emotion types, the more times the emotional signal appears in the i emotion types, the more difficult it is to distinguish which emotion type the emotional signal belongs to, and the lower the recognition weight of the emotional signal for identifying the emotion type.

[0073] Discrimination weights can modulate the frequency of emotions.

[0074] S4: Multiply the i identification weights by the corresponding emotion frequencies to obtain i emotion scores.

[0075] The fewer emotion types that have appeared as emotional signals, the higher the frequency of the emotional signals in the database, the higher the emotion score corresponding to the emotional signals, and the higher the accuracy of the emotion type judgment.

[0076] S5: Summate the emotion scores of the same emotion type among the i emotion scores to obtain M emotion type scores corresponding to M emotion types, where M≥1.

[0077] Count the number of emotion types corresponding to i emotion scores to obtain M emotion types;

[0078] The summation of all the emotion type scores corresponding to each of the M emotion types is obtained to get the M emotion type scores.

[0079] Since the acquired multimodal data includes multiple emotional signals, and different emotional signals may correspond to the same emotional type, it is necessary to calculate the emotional score corresponding to each emotional signal, and sum all emotional scores belonging to the same type to obtain the emotional type score for each emotional type. The higher the emotional type score, the more closely the emotional type corresponding to the emotional type matches the user's actual emotional type during the consultation of financial products and / or financial apps.

[0080] S6: Select j target emotion types based on M emotion type scores, where 1≤j≤M.

[0081] The M emotion type scores are sorted in descending order to obtain the emotion type sequence.

[0082] Filter out the first to the jth emotion type scores in the emotion type sequence to obtain j target emotion type scores;

[0083] The j target emotion type scores are defined as the j target emotion types.

[0084] Preferably, j is set to 3.

[0085] After recommending j target emotion types to the user, the user is prompted to select their true emotion type from these j target emotion types. The system receives the user's feedback on their true emotion type. If the true emotion type is one or more of the j target emotion types, the emotion recognition effect is considered good. If the true emotion type is not any of the j target emotion types, the emotion recognition effect is considered poor. The database is then optimized based on the user's feedback on their true emotion type and the currently acquired multimodal data.

[0086] An emotion score is calculated by combining recognition weights and emotion frequency. This allows the emotion frequency (reflecting the number of times an emotion occurs) and the recognition weights (reflecting the recognizability of an emotion) to mutually adjust each other, resulting in a more accurate emotion score. The target emotion type selected based on the emotion score is also more accurate.

[0087] The intelligent emotion type recognition method of this application includes acquiring multimodal data, converting the multimodal data into N emotion signals, matching each emotion signal with multiple emotion types, where N≥1; calculating the frequency of each emotion type in the database to obtain the emotion frequency; calculating the weight of each emotion signal in the corresponding i emotion types to obtain i recognition weights, where i≥1; multiplying the i recognition weights by the corresponding emotion frequencies to obtain i emotion scores; summing the emotion scores of the same emotion type among the i emotion scores to obtain M emotion type scores corresponding to M emotion types, where M≥1; and selecting j target emotion types based on the M emotion type scores, where 1≤j≤M. The N emotion signals obtained from multimodal data can comprehensively reflect the user's emotions. The more frequently an emotion signal appears, the lower the recognition weight corresponding to the emotion signal, since multiple emotion types may share the same emotion signal. The emotion score is calculated by using recognition weights and emotion frequency, allowing the emotion frequency (reflecting the number of times an emotion occurs) and the recognition weight (reflecting the recognition of emotion) to mutually adjust each other, resulting in a more accurate emotion score. The target emotion type selected based on emotion scores is more accurate.

[0088] In one embodiment, refer to Figure 2 Step S6, which selects j target emotion types based on M emotion type scores, includes the following steps S61-S63:

[0089] S61: Sort the M emotion type scores in descending order to obtain the emotion type sequence.

[0090] For M emotion type ratings S1, S2, ..., S M Sort the emotions in descending order; the smaller the number in the emotion type sequence, the higher the corresponding emotion type score.

[0091] S62: Select the first to the jth emotion type scores from the emotion type sequence to obtain j target emotion type scores.

[0092] The first emotion type received the highest score, indicating that the corresponding emotion type best matched the user's actual emotion type.

[0093] Preferably, j is set to 3.

[0094] S63: Take the j target emotion type scores corresponding to the j emotion types as the j target emotion types.

[0095] The j target emotion types have a high degree of matching with the user's actual emotion type.

[0096] As described above, selecting j target emotion types based on M emotion type scores involves sorting the M emotion type scores in descending order to obtain an emotion type sequence. The first to the jth emotion type scores in this sequence are then selected to obtain j target emotion type scores. The j emotion types corresponding to these j target emotion type scores are then taken as the j target emotion types. These j target emotion types show a high degree of matching with the user's actual emotion type.

[0097] In one embodiment, refer to Figure 3 The step S3, which calculates the weight of each emotion signal in the corresponding i emotion types to obtain i recognition weights, further includes the following steps S31-S33:

[0098] S31: Calculate the number of times each of the emotional signals appears in the i emotional types to obtain the number of times the i signals appear.

[0099] For example, the first emotion signal D1 exists in a total of i emotion types E1, E2, ..., E iAs seen in the example, the first emotional signal D1 appears 3 times in the first emotional type E1, and 4 times in the second emotional type E2. The first emotional signal D1 appears in the i-th emotional type E... i It appears 8 times.

[0100] S32: Add the occurrence counts of the i signals together to obtain the total number of signal occurrences.

[0101] For example, the number of occurrences of the first emotion signal D1 in the i emotion types is added together to get a total of 50 occurrences of the signal.

[0102] S33: Calculate the ratio of the number of occurrences of the first signal to the number of occurrences of the i-th signal to the total number of occurrences of the signal to obtain the i-th recognition weight.

[0103] For example, the first signal corresponding to the first emotion signal D1 appears 3 times, and the corresponding first recognition weight is 0.06; the second signal corresponding to the first emotion signal D1 appears 4 times, and the corresponding second recognition weight is 0.08; the third signal corresponding to the first emotion signal D1 appears 8 times, and the corresponding third recognition weight is 0.16.

[0104] Since the same emotional signal appears in all i emotion types, the more times the emotional signal appears in the i emotion types, the more difficult it is to distinguish which emotion type the emotional signal belongs to, and the lower the recognition weight of the emotional signal for identifying the emotion type.

[0105] Discrimination weights can modulate the frequency of emotions.

[0106] As described above, the weight of each emotional signal in the corresponding i emotion types is calculated to obtain i identification weights. This includes calculating the number of times each emotional signal appears in the i emotion types to obtain i signal occurrence counts. The occurrence counts of the i signals are summed to obtain the total number of signal occurrences. The ratio of the occurrence counts of the first to the i-th signal to the total number of signal occurrences is calculated to obtain the i identification weights. Since the same emotional signal can appear in all i emotion types, the more frequently an emotional signal appears in the i emotion types, the more difficult it is to distinguish which emotion type it belongs to, and the lower the identification weight of the emotional signal for identifying the emotion type. The identification weights can adjust the emotional frequency.

[0107] In one embodiment, refer to Figure 4 The step S2, which calculates the frequency of each emotion type in the database to obtain the emotion frequency, includes the following steps S21-S23:

[0108] S21: Count the number of times each emotion type appears in the database to obtain the emotion count.

[0109] For example, if the first emotion type E1 appears 30 times in the database, then the number of times the first emotion type E1 appears is 30. If the second emotion type E2 appears 10 times in the database, then the number of times the second emotion type E2 appears is 10.

[0110] S22: Count the total number of times all the aforementioned emotion types appear in the database to obtain the total number of emotion occurrences.

[0111] For example, if the total number of emotion types is k, k≥M, and the total number of occurrences of k emotion types in the database is 1300, then the total number of occurrences of the emotion is 1300.

[0112] S23: The ratio of the number of emotional episodes to the total number of emotional episodes is taken as the emotional frequency.

[0113] For example, the ratio of the number of times the first emotion type E1 appears in the database to the total number of emotions is approximately 2.3%, and the ratio of the number of times the second emotion type E2 appears in the database to the total number of emotions is approximately 0.77%.

[0114] The more frequently an emotion type appears in the database, the higher the frequency of the corresponding emotion. Conversely, the fewer frequently an emotion type appears in the database, the lower the frequency of the corresponding emotion.

[0115] As mentioned above, calculating the frequency of each emotion type in the database yields the emotion frequency. This involves counting the number of times each emotion type appears in the database, resulting in an emotion count. The total number of times all emotion types appear in the database is then counted, resulting in the total emotion count. The ratio of the emotion count to the total emotion count is used as the emotion frequency. The more times an emotion type appears in the database, the higher its corresponding emotion frequency. Conversely, the fewer times an emotion type appears in the database, the lower its corresponding emotion frequency.

[0116] In one embodiment, refer to Figure 5 The step S1, which involves acquiring multimodal data, converting the multimodal data into N emotion signals, and matching each emotion signal with multiple emotion types, wherein N≥1, includes the following steps S11-S14:

[0117] S11: Acquire multimodal data.

[0118] The multimodal data includes any combination of two-dimensional face images, three-dimensional human body images, and speech audio.

[0119] Two-dimensional facial images can be captured by a camera or mobile phone lens, while three-dimensional human body images can be captured by a 3D imaging device. 3D human body images can also be obtained by combining multiple two-dimensional facial images and a 3D human body model. Voice audio can be obtained by recording the content of telephone conversations between users and customer service representatives.

[0120] S12: Convert the multimodal data into N emotion signals.

[0121] For a two-dimensional face image, by extracting the set of key points from the two-dimensional face image, and performing expression analysis based on the set of key points, N1 emotion signals corresponding to the two-dimensional face image are obtained.

[0122] For a 3D human body image, by extracting the human body posture from the 3D human body image, emotion analysis is performed on the human body posture to obtain N2 emotion signals corresponding to the 3D human body image.

[0123] For speech audio, Mel frequency cepstral coefficients are obtained by performing Mel frequency analysis on the speech audio. The Mel frequency cepstral coefficients and speech audio are then input into a trained audio analysis model to obtain N3 emotion signals corresponding to the speech audio.

[0124] The relationship between the above N1 emotional signals, N2 emotional signals, N3 emotional signals and N emotional signals is N1+N2+N3=N.

[0125] S13: Retrieve all the emotion types from the database.

[0126] The emotion types in the database include happiness, surprise, astonishment, confusion, anger, and disappointment.

[0127] S14: Statistically analyze the emotion types that have appeared with the emotional signal to obtain multiple emotion types.

[0128] For example, if the first emotional signal D1 appears in both the happy and surprised emotional types, then the multiple emotional types corresponding to the first emotional signal D1 are happy and surprised.

[0129] Emotion types are used to calculate emotion frequency and recognition weights.

[0130] As described above, multimodal data is acquired, converted into N emotion signals, and matched with multiple emotion types corresponding to each emotion signal, where N ≥ 1. This process includes acquiring multimodal data and converting it into N emotion signals. All emotion types are retrieved from the database, and the total number of emotion types that have appeared with each emotion signal is counted, resulting in multiple emotion types. These emotion types are used to calculate emotion frequency and recognition weights.

[0131] In one embodiment, step S5, which involves summing the emotion scores of the same emotion type among the i emotion scores to obtain M emotion type scores corresponding to M emotion types, where M≥1, includes the following steps S51-S52:

[0132] S51: Count the number of emotion types corresponding to the i emotion scores to obtain M emotion types.

[0133] An emotion rating may appear in one emotion type, an emotion rating may appear in multiple emotion types, and multiple emotion ratings may appear in one emotion type. Therefore, M may be greater than i, M may be equal to i, and M may be less than i.

[0134] S52: Sum all the emotion type scores corresponding to each of the M emotion types to obtain the M emotion type scores.

[0135] For example, the first emotion type E1 corresponds to two emotion type scores: the first emotion signal corresponds to an emotion type score of 0.69%, and the second emotion signal corresponds to an emotion type score of 1.65%. Therefore, the emotion type score corresponding to the first emotion type E1 is 2.34%.

[0136] Since the acquired multimodal data includes multiple emotional signals, and different emotional signals may correspond to the same emotional type, it is necessary to calculate the emotional score corresponding to each emotional signal, and sum all emotional scores belonging to the same type to obtain the emotional type score for each emotional type. The higher the emotional type score, the more closely the emotional type corresponding to the emotional type matches the user's actual emotional type during the consultation of financial products and / or financial apps.

[0137] As described above, the emotional scores of the same emotional type among the i emotional scores are summed to obtain M emotional type scores corresponding to M emotional types, where M≥1. This includes counting the number of emotional types corresponding to the i emotional scores to obtain M emotional types. The emotional type scores corresponding to each of the M emotional types are then summed to obtain M emotional type scores. Since the acquired multimodal data includes multiple emotional signals, and different emotional signals may correspond to the same emotional type, it is necessary to calculate the emotional score corresponding to each emotional signal and sum all emotional scores belonging to the same type to obtain the emotional type score for each emotional type.

[0138] Reference Figure 6 This is a schematic block diagram of an intelligent emotion type recognition device proposed in this application. The device includes:

[0139] The emotion type matching module 10 is used to acquire multimodal data, convert the multimodal data into N emotion signals, and match multiple emotion types corresponding to each emotion signal, where N≥1;

[0140] The emotion frequency calculation module 20 is used to calculate the frequency of each emotion type in the database to obtain the emotion frequency.

[0141] The identification weight calculation module 30 is used to calculate the weight of each emotion signal in the corresponding i emotion types to obtain i identification weights, where i≥1;

[0142] The emotion scoring calculation module 40 is used to multiply the i identification weights by the corresponding emotion frequencies to obtain i emotion scores.

[0143] The emotion type rating calculation module 50 is used to sum the emotion ratings of the same emotion type among the i emotion ratings to obtain M emotion type ratings corresponding to M emotion types, where M≥1;

[0144] The target emotion type filtering module 60 is used to filter j target emotion types based on M emotion type scores, where 1≤j≤M.

[0145] The aforementioned intelligent emotion type recognition device is used to implement the intelligent emotion type recognition method.

[0146] In one embodiment, the target emotion type screening module 60 further includes:

[0147] A descending sorting unit is used to sort the M emotion type scores in descending order to obtain an emotion type sequence;

[0148] An emotion type rating filtering unit is used to filter out the first to the jth emotion type ratings in the emotion type sequence to obtain j target emotion type ratings;

[0149] The target emotion type definition unit is used to define the j emotion types corresponding to the j target emotion type scores as the j target emotion types.

[0150] In one embodiment, the discrimination weight calculation module 30 further includes:

[0151] The signal occurrence count calculation unit is used to calculate the number of times each of the emotional signals appears in the i emotional types, and obtain the occurrence counts of the i signals;

[0152] The signal occurrence total count calculation unit is used to add the occurrence counts of the i signals to obtain the total number of signal occurrences;

[0153] The identification weight calculation unit is used to calculate the ratio of the number of times the first signal appears to the number of times the i-th signal appears to the total number of times the signal appears, so as to obtain the i-th identification weight.

[0154] In one embodiment, the emotion frequency calculation module 20 further includes:

[0155] An emotion frequency counting unit is used to count the number of times each emotion type appears in the database to obtain the emotion frequency.

[0156] The total number of emotion occurrences unit is used to count the total number of times all the emotion types appear in the database to obtain the total number of emotion occurrences.

[0157] An emotion frequency definition unit is used to define the emotion frequency as the ratio of the number of emotions to the total number of emotions.

[0158] In one embodiment, the emotion type matching module 10 further includes:

[0159] An emotion type retrieval unit is used to retrieve all the emotion types in the database;

[0160] The total number of emotion types is counted by the unit used to count the emotion types that have appeared with the emotion signal, thereby obtaining multiple emotion types.

[0161] In one embodiment, the emotion type rating calculation module 50 further includes:

[0162] An emotion type quantity counting unit is used to count the number of emotion types corresponding to i emotion scores, thereby obtaining M emotion types;

[0163] The emotion type rating summation unit is used to sum all the emotion type ratings corresponding to each of the M emotion types to obtain M emotion type ratings.

[0164] Reference Figure 7 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 7As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor is designed to provide computing and control capabilities. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used to store emotion scores, etc. The network interface of the computer device is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements an intelligent emotion type recognition method. The aforementioned intelligent emotion type recognition method includes:

[0165] Acquire multimodal data, convert the multimodal data into N emotion signals, and match multiple emotion types corresponding to each emotion signal, where N≥1;

[0166] Calculate the frequency of each emotion type in the database to obtain the emotion frequency;

[0167] Calculate the weight of each emotion signal in the corresponding i emotion types to obtain i recognition weights, where i≥1;

[0168] Multiply each of the i identification weights by its corresponding emotion frequency to obtain i emotion scores;

[0169] Summing the emotion scores of the same emotion type among the i emotion scores yields M emotion type scores corresponding to the M emotion types, where M≥1;

[0170] Based on the M emotion type scores, select j target emotion types, where 1≤j≤M.

[0171] In one embodiment, the step of filtering j target emotion types based on M emotion type scores includes:

[0172] The M emotion type scores are sorted in descending order to obtain the emotion type sequence.

[0173] Filter out the first to the jth emotion type scores in the emotion type sequence to obtain j target emotion type scores;

[0174] The j target emotion type scores are defined as the j target emotion types.

[0175] In one embodiment, calculating the weight of each emotion signal in the corresponding i emotion types to obtain i recognition weights includes:

[0176] Calculate the number of times each emotion signal appears in the i emotion types to obtain the number of times each of the i signals appears;

[0177] Add the occurrence counts of the i signals together to obtain the total number of signal occurrences;

[0178] Calculate the ratio of the number of occurrences of the first signal to the number of occurrences of the i-th signal to the total number of occurrences of the signal to obtain the i-th recognition weight.

[0179] In one embodiment, calculating the frequency of each emotion type in the database to obtain the emotion frequency includes:

[0180] The number of times each emotion type appears in the database is counted to obtain the emotion count.

[0181] The total number of times each of the aforementioned emotion types appears in the database is counted to obtain the total number of emotion occurrences.

[0182] The ratio of the number of emotional episodes to the total number of emotional episodes is taken as the emotional frequency.

[0183] In one embodiment, matching multiple emotion types corresponding to each of the emotion signals includes:

[0184] Retrieve all the aforementioned emotion types from the database;

[0185] The emotional types that have appeared with the emotional signals are statistically analyzed to obtain multiple emotional types.

[0186] In one embodiment, summing the emotion scores of the same emotion type among the i emotion scores to obtain M emotion type scores corresponding to the M emotion types includes:

[0187] Count the number of emotion types corresponding to i emotion scores to obtain M emotion types;

[0188] The summation of all the emotion type scores corresponding to each of the M emotion types is obtained to get the M emotion type scores.

[0189] In one embodiment, the multimodal data includes any combination of two-dimensional face images, three-dimensional human body images, and audio recordings.

[0190] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.

[0191] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an intelligent emotion type recognition method. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.

[0192] The above-mentioned intelligent emotion type recognition methods include:

[0193] Acquire multimodal data, convert the multimodal data into N emotion signals, and match multiple emotion types corresponding to each emotion signal, where N≥1;

[0194] Calculate the frequency of each emotion type in the database to obtain the emotion frequency;

[0195] Calculate the weight of each emotion signal in the corresponding i emotion types to obtain i recognition weights, where i≥1;

[0196] Multiply each of the i identification weights by its corresponding emotion frequency to obtain i emotion scores;

[0197] Summing the emotion scores of the same emotion type among the i emotion scores yields M emotion type scores corresponding to the M emotion types, where M≥1;

[0198] Based on the M emotion type scores, select j target emotion types, where 1≤j≤M.

[0199] In one embodiment, the step of filtering j target emotion types based on M emotion type scores includes:

[0200] The M emotion type scores are sorted in descending order to obtain the emotion type sequence.

[0201] Filter out the first to the jth emotion type scores in the emotion type sequence to obtain j target emotion type scores;

[0202] The j target emotion type scores are defined as the j target emotion types.

[0203] In one embodiment, calculating the weight of each emotion signal in the corresponding i emotion types to obtain i recognition weights includes:

[0204] Calculate the number of times each emotion signal appears in the i emotion types to obtain the number of times each of the i signals appears;

[0205] Add the occurrence counts of the i signals together to obtain the total number of signal occurrences;

[0206] Calculate the ratio of the number of occurrences of the first signal to the number of occurrences of the i-th signal to the total number of occurrences of the signal to obtain the i-th recognition weight.

[0207] In one embodiment, calculating the frequency of each emotion type in the database to obtain the emotion frequency includes:

[0208] The number of times each emotion type appears in the database is counted to obtain the emotion count.

[0209] The total number of times each of the aforementioned emotion types appears in the database is counted to obtain the total number of emotion occurrences.

[0210] The ratio of the number of emotional episodes to the total number of emotional episodes is taken as the emotional frequency.

[0211] In one embodiment, matching multiple emotion types corresponding to each of the emotion signals includes:

[0212] Retrieve all the aforementioned emotion types from the database;

[0213] The emotional types that have appeared with the emotional signals are statistically analyzed to obtain multiple emotional types.

[0214] In one embodiment, summing the emotion scores of the same emotion type among the i emotion scores to obtain M emotion type scores corresponding to the M emotion types includes:

[0215] Count the number of emotion types corresponding to i emotion scores to obtain M emotion types;

[0216] The summation of all the emotion type scores corresponding to each of the M emotion types is obtained to get the M emotion type scores.

[0217] In one embodiment, the multimodal data includes any combination of two-dimensional face images, three-dimensional human body images, and audio recordings.

[0218] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0219] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.

[0220] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for intelligent recognition of emotion types, characterized in that, include: Acquire multimodal data, convert the multimodal data into N emotion signals, and match multiple emotion types corresponding to each emotion signal, where N≥1; Calculate the frequency of each emotion type in the database to obtain the emotion frequency; Calculate the weight of each emotion signal in the corresponding i emotion types to obtain the i recognition weights corresponding to the emotion signal, where i≥1, and the emotion types corresponding to different emotion signals can be repeated; Multiply the i identification weights corresponding to each emotion signal by the corresponding emotion frequency to obtain the i emotion scores corresponding to the emotion signal. Sum the emotion scores corresponding to all emotion signals to form an emotion score set. Summ the emotion scores of the same emotion type in the emotion score set to obtain M emotion type scores corresponding to M emotion types, where M≥1. Based on the M emotion type scores, select j target emotion types, where 1≤j≤M.

2. The intelligent emotion type recognition method according to claim 1, characterized in that, The step of selecting j target emotion types based on M emotion type scores includes: The M emotion type scores are sorted in descending order to obtain the emotion type sequence. Filter out the first to the jth emotion type scores in the emotion type sequence to obtain j target emotion type scores; The j target emotion type scores are defined as the j target emotion types.

3. The intelligent emotion type recognition method according to claim 1, characterized in that, The calculation of the weight of each emotion signal in the corresponding i emotion types to obtain i recognition weights includes: Calculate the number of times each emotion signal appears in the i emotion types to obtain the number of times each of the i signals appears; Add the occurrence counts of the i signals together to obtain the total number of signal occurrences; Calculate the ratio of the number of occurrences of the first signal to the number of occurrences of the i-th signal to the total number of occurrences of the signal to obtain the i-th recognition weight.

4. The intelligent emotion type recognition method according to claim 1, characterized in that, The calculation of the frequency of each emotion type in the database to obtain the emotion frequency includes: The number of times each emotion type appears in the database is counted to obtain the emotion count. The total number of times each of the aforementioned emotion types appears in the database is counted to obtain the total number of emotion occurrences. The ratio of the number of emotional episodes to the total number of emotional episodes is taken as the emotional frequency.

5. The intelligent emotion type recognition method according to claim 1, characterized in that, The matching of multiple emotion types corresponding to each emotion signal includes: Retrieve all the aforementioned emotion types from the database; The emotional types that have appeared with the emotional signals are statistically analyzed to obtain multiple emotional types.

6. The intelligent emotion type recognition method according to claim 1, characterized in that, The summation of the emotion scores of the same emotion type among the i emotion scores to obtain M emotion type scores corresponding to M emotion types includes: Count the number of emotion types corresponding to i emotion scores to obtain M emotion types; The summation of all the emotion type scores corresponding to each of the M emotion types is obtained to get the M emotion type scores.

7. The intelligent emotion type recognition method according to claim 1, characterized in that, The multimodal data includes any combination of two-dimensional face images, three-dimensional human body images, and speech audio.

8. An intelligent emotion type recognition device, characterized in that, include: An emotion type matching module is used to acquire multimodal data, convert the multimodal data into N emotion signals, and match multiple emotion types corresponding to each emotion signal, where N≥1; The emotion frequency calculation module is used to calculate the frequency of each emotion type in the database to obtain the emotion frequency. The identification weight calculation module is used to calculate the weight of each emotion signal in the corresponding i emotion types, and obtain the i identification weights corresponding to the emotion signal, where i≥1, and the emotion types corresponding to different emotion signals can be repeated; The emotion score calculation module is used to multiply the i recognition weights corresponding to each emotion signal by the corresponding emotion frequency to obtain the i emotion scores corresponding to the emotion signal. The emotion type rating calculation module is used to summarize the emotion ratings corresponding to all emotion signals to form an emotion rating set, and to sum the emotion ratings of the same emotion type in the emotion rating set to obtain M emotion type ratings corresponding to M emotion types, where M≥1; The target emotion type filtering module is used to filter j target emotion types based on M emotion type scores, where 1≤j≤M.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the emotional type intelligent recognition method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the emotional type intelligent recognition method according to any one of claims 1 to 7.

Images