An asynchronous multi-modal emotion recognition method, device, equipment and medium

Abstract

The present application relates to the technical field of emotion recognition, in particular to an asynchronous multi-modal emotion recognition method, device, equipment and medium. The present application respectively carries out pulse coding on each modality biological data of the user, obtains the pulse sequence of each modality biological data, applies cross-modal synchronous neurons to each pulse sequence based on the firing time of pulse coding, obtains the emotion feature of the user, and finally applies an emotion classifier to the emotion feature to obtain the emotion classification result. Since the pulse sequence formed by the pulse coding of the present application can record the time when the biological data changes, and the cross-modal synchronous neurons are applied to the pulse sequence according to the firing time of the pulse coding to analyze the emotion feature, the combination of the pulse coding and the cross-modal synchronous neurons of the present application can identify the time when each multi-modal biological data changes, and identify the emotion feature of the user according to the time sequence before and after the change of the multi-modal biological data, so as to improve the accuracy of emotion recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of emotion recognition, in particular to an asynchronous multi-modal emotion recognition method, device, equipment and medium. BACKGROUND

[0002] The emotion recognition technology is applied to an intelligent system to realize human-computer interaction, and the user emotion can be recognized through multi-modal biological data of the user. The prior art fuses multi-modal biological data in a static splicing or attention weighting manner, and then recognizes the user emotion based on the fused biological data. However, the static splicing or attention weighting manner ignores the asynchrony of the multi-modal biological data, that is, the order of the multi-modal biological data, and the same biological data represents different emotions due to different orders, that is, the asynchrony of the multi-modal biological data affects the emotion recognition result, and the prior art ignores the asynchrony of the multi-modal biological data.

[0003] In summary, the prior art reduces the accuracy of emotion recognition.

[0004] Therefore, the prior art still needs to be improved and enhanced. SUMMARY

[0005] To solve the above technical problems, the present application provides an asynchronous multi-modal emotion recognition method, device, equipment and medium, which solves the problem of reducing the accuracy of emotion recognition in the prior art.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides an asynchronous multi-modal emotion recognition method, comprising:

[0008] Obtaining each modality biological data related to emotion of a user, the each modality biological data being asynchronous data, respectively performing pulse coding on the each modality biological data to obtain a pulse sequence of each modality biological data;

[0009] Applying a cross-modal synchronous neuron to each pulse sequence based on the firing time of the pulse coding on the pulse sequence to obtain an emotion feature of the user;

[0010] Applying an emotion classifier to the emotion feature to obtain an emotion classification result of the user.

[0011] In an implementation manner, respectively performing pulse coding on the each modality biological data to obtain a pulse sequence of each modality biological data comprises:

[0012] Determining a face image, a hand image and a voice signal in the each modality biological data;

[0013] determine a variation strength of the voice signal, and determine the pulse time interval of the pulse coding according to the variation strength, and construct a voice pulse sequence of the voice signal according to the pulse time interval corresponding to the voice signal;

[0014] extract a key point on the hand image, and determine a gesture posture presented by the key point, and determine a gesture semantic category corresponding to the gesture posture;

[0015] obtain a hand pulse sequence of the hand image according to the gesture semantic category;

[0016] extract a facial feature related to an expression from the face image;

[0017] determine a pulse time interval of the pulse coding of the face image according to the pulse time interval of the voice pulse sequence and the pulse time interval of the hand pulse sequence, and construct a face pulse sequence of the face image according to the pulse time interval corresponding to the face image and the facial feature.

[0018] In an implementation manner, the determination of the variation strength of the voice signal and the determination of the pulse time interval of the pulse coding according to the variation strength comprises:

[0019] extract a instantaneous frequency and an instantaneous amplitude of the voice signal, and determine a power differential of the instantaneous frequency with respect to time and an amplitude differential of the instantaneous amplitude with respect to time;

[0020] obtain the variation strength of the voice signal according to the power differential and the amplitude differential;

[0021] apply a natural exponential function to the variation strength to obtain the pulse time interval, the natural exponential function being a decreasing function.

[0022] In an implementation manner, the obtaining of the hand pulse sequence of the hand image according to the gesture semantic category comprises:

[0023] determine a pulse coding template matched with the gesture semantic category;

[0024] obtain the hand pulse sequence of the hand image according to the pulse coding template.

[0025] In an implementation manner, the obtaining of the emotion feature of the user comprises:

[0026] extracting each same-group pulse code from each of the pulse sequences, each of the same-group pulse codes containing pulse codes of the biological data of each modality at respective firing times, a time difference between the respective firing times being less than a threshold value;

[0027] determining a respective emotion type corresponding to each of the same-group pulse codes;

[0028] determining a respective cross-modality synchronous neuron corresponding to each of the same-group pulse codes according to the emotion type corresponding to each of the same-group pulse codes, and applying the respective cross-modality synchronous neuron to each of the same-group pulse codes respectively, to obtain the emotion feature of the user corresponding to each of the same-group pulse codes respectively.

[0029] In an implementation manner, the emotion classifier is applied to the emotion features to obtain an emotion classification result of the user, including:

[0030] obtaining a weight of the emotion feature corresponding to each of the same-group pulse codes respectively according to the time difference;

[0031] applying the emotion classifier to each of the emotion features and the weight corresponding to each of the emotion features respectively to obtain the emotion classification result of the user.

[0032] In an implementation manner, the synaptic weight updating manner of the cross-modality synchronous neuron includes:

[0033] applying an asymmetric time correlation synaptic plasticity mechanism to the time difference to obtain a front-back synaptic weight change amount of the cross-modality synchronous neuron;

[0034] updating the weight between the front-back synapses according to the front-back synaptic weight change amount.

[0035] In a second aspect, an embodiment of the present application further provides an asynchronous multi-modal emotion recognition device, wherein the device includes the following components:

[0036] a pulse coding module, configured to obtain biological data of each modality of a user related to emotion, the biological data of each modality being asynchronous data, and perform pulse coding on the biological data of each modality respectively to obtain a pulse sequence of each of the biological data of each modality;

[0037] a feature recognition module, configured to apply a cross-modality synchronous neuron to each of the pulse sequences based on the firing time of the pulse coding on each of the pulse sequences, to obtain an emotion feature of the user;

[0038] an emotion classification module, configured to apply an emotion classifier to the emotion feature to obtain an emotion classification result of the user.

[0039] In a third aspect, the embodiments of the present application further provide a terminal device, wherein the terminal device comprises a memory, a processor, and an asynchronous multi-modal emotion recognition program stored in the memory and executable on the processor, and the processor implements the steps of the asynchronous multi-modal emotion recognition method when executing the asynchronous multi-modal emotion recognition program.

[0040] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, wherein the computer readable storage medium stores an asynchronous multi-modal emotion recognition program, and the processor implements the steps of the asynchronous multi-modal emotion recognition method when executing the asynchronous multi-modal emotion recognition program.

[0041] Beneficial effects: The present application firstly performs pulse coding on each modality biological data of the user respectively to obtain a pulse sequence of each modality biological data, then applies a cross-modal synchronous neuron to each pulse sequence based on the firing time of the pulse coding to obtain an emotion feature of the user, and finally applies an emotion classifier to the emotion feature to obtain an emotion classification result of the user. Since the pulse sequence formed by the pulse coding of the present application can record the time when the biological data changes, and the cross-modal synchronous neuron is applied to the pulse sequence based on the firing time of the pulse coding to analyze the emotion feature, the pulse coding and the cross-modal synchronous neuron of the present application are combined to identify the time when each modality biological data changes, and to identify the emotion feature of the user according to the time sequence before and after the change of the multi-modal biological data, and the time sequence before and after the change is closely related to the emotion, so the present application can improve the accuracy of emotion recognition. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is a whole flowchart of the present application;

[0043] Figure 2 is an emotion feature extraction flowchart in the embodiments of the present application;

[0044] Figure 3 is a flowchart of emotion classification using an emotion classifier in the embodiments of the present application;

[0045] Figure 4 is a structure diagram of an asynchronous multi-modal emotion recognition device provided by the present application;

[0046] Figure 5 is an internal structure principle block diagram of a terminal device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the present application will be described clearly and completely in the following combined with the embodiments and the accompanying drawings. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts belong to the protection scope of the present application.

[0048] It is found through research that the emotion recognition technology is applied to the intelligent system to realize human-computer interaction, and the user emotion can be recognized through the multi-modal biological data of the user. The prior art fuses the multi-modal biological data in a static splicing or attention weighting manner, and then recognizes the user emotion based on the fused biological data. The static splicing or attention weighting manner ignores the asynchrony of the multi-modal biological data, that is, the sequence of the multi-modal biological data, and the same biological data represents different emotions due to the different sequence of occurrence, that is, the asynchrony of the multi-modal biological data affects the emotion recognition result, and the prior art ignores the asynchrony of the multi-modal biological data, thereby reducing the accuracy of emotion recognition.

[0049] To solve the above technical problems, the present application provides an asynchronous multi-modal emotion recognition method, device, equipment and medium, which solves the problem of reducing the accuracy of emotion recognition in the prior art. In specific implementation, first, the pulse coding is performed on each modal biological data to obtain the pulse sequence of each modal biological data; then, the cross-modal synchronous neuron is applied to each pulse sequence to obtain the emotion feature of the user; and finally, the emotion classifier is applied to the emotion feature to obtain the emotion classification result of the user.

[0050] For example, the modal biological data includes a face image, a voice signal and a hand image, the voice signal is composed of voice amplitudes at different times, when the voice amplitude at a certain time changes greatly, the voice amplitude at the time is coded (the coding is pulse coding), and the voice amplitudes at other times which change greatly are coded in the same way, and the pulse coding corresponding to multiple times is sequentially sorted to form a pulse sequence, that is, a voice pulse sequence.

[0051] The key points on the hand image are extracted and coded to obtain a hand pulse sequence, and the pulse coding on the hand pulse sequence represents the change information of the key points; the feature points on the face image are extracted and coded to obtain a face pulse sequence, and the pulse coding on the face pulse sequence represents the change information of the feature points.

[0052] A pulse code is extracted from a voice pulse sequence, a hand pulse sequence, and a face pulse sequence respectively, and the difference between the firing times of the three pulse codes (the firing time is the time when the biological data changes) is less than a threshold value. The three pulse codes are input to the cross-modal synchronous neuron, and the cross-modal synchronous neuron identifies the emotional features of the user based on the three pulse codes and the order of the firing times of the three pulse codes. Finally, the emotional features are input to the emotion classifier to obtain the emotional classification result of the user. Since the order of the firing times of the three pulse codes represents the time when the biological data of the three modalities changes, and the time when the biological data of the three modalities changes is associated with the emotional features, the cross-modal synchronous neuron of the present application can accurately identify the emotional features.

[0053] The asynchronous multi-modal emotion recognition method of the embodiment can be applied to a terminal device, which can be a terminal product with image and voice processing functions, such as a human-computer interaction system, etc. In the embodiment, as shown in Figure 1 the asynchronous multi-modal emotion recognition method specifically includes the following steps:

[0054] S100, obtaining user emotional related modal biological data, the modal biological data is asynchronous data, and the modal biological data is respectively pulse coded to obtain the pulse sequence of each modal biological data;

[0055] S200, applying a cross-modal synchronous neuron to each pulse sequence based on the firing time of the pulse code on each pulse sequence to obtain the emotional features of the user;

[0056] S300, applying an emotion classifier to the emotional features to obtain the emotional classification result of the user.

[0057] The emotional classification result based on steps S100, S200, and S300 is applied to a human-computer dialogue system to develop an animation short video for improving the user's mood according to the emotional classification result of the user.

[0058] In the embodiment, the pulse coding of the modal biological data in step S100 to obtain the pulse sequence of each modal biological data includes the following specific steps S101-S109 as shown in Figure 2 .

[0059] S101, determining the face image, hand image, and voice signal in the modal biological data.

[0060] The face image, the hand image and the voice signal constitute multi-modal biological data of the user, and the multi-modal biological data is used for emotion recognition of the user, so that the error recognition of emotion by single modal data can be avoided. Moreover, the face image, the hand image and the voice signal are related to each other, that is, the change of emotion is simultaneously related to the face, the hand and the voice, and the correlation analysis of the three can improve the emotion recognition accuracy.

[0061] S102, extracting the instantaneous frequency and the instantaneous amplitude of the voice signal, and determining the power differential of the instantaneous frequency with respect to time and the amplitude differential of the instantaneous amplitude with respect to time.

[0062] The voice signal is decomposed into a plurality of IMF components, and each IMF component represents a local oscillation of the voice signal in a different frequency band. The IMF is an intrinsic mode function.

[0063] The Hilbert-Huang transform or wavelet analysis is used for each IMF component to obtain the instantaneous frequency and the instantaneous amplitude of the voice signal on the IMF component. The above Hilbert-Huang transform is HHT, which is Hilbert-Huang Transform.

[0064] S103, obtaining the change intensity of the voice signal according to the power differential and the amplitude differential.

[0065] The power differential is represented by , and the amplitude differential is represented by , wherein represents a differential symbol.

[0066] ; (1)

[0067] In the formula, represents the weight of the power differential, represents the weight of the amplitude differential, and , represents the change intensity at the current time .

[0068] S104, applying a natural exponential function to the change intensity to obtain a pulse time interval, the natural exponential function being a decreasing function, and constructing a voice pulse sequence of the voice signal according to the pulse time interval corresponding to the voice signal.

[0069] ​The pulse time interval is the interval between the time of emission of two pulse codes, denoted by The next pulse code represents the time interval between the current time The next pulse code.

[0070] ; (2)

[0071] In the formula, The maximum emission interval of the pulse code, It can be 200 milliseconds, that is, the maximum delay of 200 milliseconds of the smooth speech signal to emit the next pulse code. The sensitivity coefficient of That is, Used to control The degree of influence on the emission frequency of the pulse code. The base of the natural logarithm, Forms a natural exponential function, because Is greater than zero, It is a decreasing function. The decreasing function is The greater, The smaller, so as to reduce the emission time interval of the pulse code for the frequently changing speech signal, so as to capture more information from the frequently changing speech signal for subsequent emotion recognition.

[0072] From formula (2), it can be seen that the time interval is related to the change intensity of the speech signal, that is, the time interval will change with the speech mutation to form sparse pulse codes and dense pulse codes, that is, the pulse codes on the pulse sequence are sparse and staggered.

[0073] If the current time Is the Emission time, and Denotes the Emission time, then the Emission time And The relationship is as follows:

[0074] ; (3)

[0075] Because the current time Is the Emission time , it is necessary to change In formula (2) to , Represents the change intensity of the speech signal at the emission time .

[0076] S105, extract the key points on the hand image, determine the gesture posture presented by the key points, and determine the gesture semantic category corresponding to the gesture posture.

[0077] The MediaPipe Hands algorithm is used on the hand image to extract 2D or 3D key points, the hand image includes 21 key points, according to the relative positions of each key point to each other, the gesture posture presented by the hand is judged, and then the gesture semantic category is judged according to the gesture posture.

[0078] The gesture posture includes hand stretching, chest hugging, and fist clenching, and the gesture semantic category includes doubt, defense, anger, affirmation, rejection, and distrust. For example, the semantic category of the hand stretching gesture posture is doubt, and the semantic category of the chest hugging gesture posture is defense or distrust.

[0079] S106, determine the pulse coding template matched with the gesture semantic category.

[0080] A correspondence table of gesture semantic category and pulse coding template is established in advance, for example, when the gesture semantic category is doubt, the pulse coding template is [0, 10, 20], 0, 10, and 20 respectively represent the time of generating pulse coding representing doubt, that is, there is a pulse coding at the 0th millisecond, the 10th millisecond, and the 20th millisecond respectively, and three pulse codings with an interval of 10 milliseconds represent the gesture semantic category of doubt.

[0081] When the gesture semantic category is defense, the pulse coding template is [0, 3, 5, 10] plus 30 milliseconds of zero potential, 0, 3, 5, and 10 respectively represent the time of generating pulse coding representing defense, that is, there is a pulse coding at the 0th millisecond, the 3rd millisecond, the 5th millisecond, and the 10th millisecond respectively, and the pulse coding represents a potential greater than zero potential.

[0082] When the gesture semantic category is anger, the pulse coding template is [0, 2, 4, 6, 8, 10], 0, 2, 4, 6, 8, and 10 respectively represent the time of generating pulse coding representing anger, that is, there is a pulse coding at the 0th millisecond, the 2nd millisecond, the 4th millisecond, the 6th millisecond, the 8th millisecond, and the 10th millisecond respectively.

[0083] When the gesture semantic category is affirmation, the pulse coding template is [0, 30], 0 and 30 respectively represent the time of generating pulse coding representing affirmation, that is, there is a pulse coding at the 0th millisecond and the 30th millisecond respectively.

[0084] Pulse coding templates are used to represent the semantic categories of gestures. Subsequently, the hand pulse sequence formed by the pulse coding templates is input into the cross-modal synchronous neuron. The cross-modal synchronous neuron can recognize the gesture posture and thus the user's emotion based on the hand pulse sequence.

[0085] S107, Based on the pulse coding template, obtain the hand pulse sequence of the hand image.

[0086] In other words, by arranging the pulse coding templates corresponding to each gesture semantic category in the order in which they appear, the hand pulse sequence is obtained.

[0087] S108, Extract facial features related to facial expressions from the facial image.

[0088] CNN (Convolutional Neural Network) is applied to face images to extract facial features.

[0089] S109, based on the pulse time interval of the speech pulse sequence and the pulse time interval of the hand pulse sequence, determine the pulse time interval of the pulse code of the face image, and construct the face pulse sequence of the face image based on the pulse time interval corresponding to the face image and the face features.

[0090] The pulse intervals in a face pulse sequence are longer than those in a speech pulse sequence, and also longer than those in a hand pulse sequence. The pulse codes in a face pulse sequence represent the user's facial features at the specific pulse moments.

[0091] Instead of encoding the entire biological data, pulse sequences, which are discontinuous, non-uniform signals, are used to represent changes in biological data. Using pulse sequences to represent biological data can compress redundant information that is irrelevant to emotion recognition while retaining important information, thereby saving computational resources for subsequent cross-modal synchronization of neurons and improving the sensitivity of recognizing sudden emotions (including anger and fright).

[0092] In this embodiment, as Figure 3 As shown, step S200 includes the following specific steps S201, S202, and S203:

[0093] S201, extract each group of pulse codes from each of the pulse sequences, each group of pulse codes containing the pulse codes of each modality of biological data at its respective release time, and the time difference between the release times is less than a threshold.

[0094] That is, a plurality of same-group pulse codes are extracted from the voice pulse sequence, the hand pulse sequence, and the face pulse sequence, each same-group pulse code including one pulse code in the voice pulse sequence, one pulse code in the hand pulse sequence, and one pulse code in the face pulse sequence. A time difference between the emission times of any two pulse codes in the three pulse codes (denoted as a threshold value of 50 milliseconds.

[0095] S202, determining an emotion type corresponding to each same-group pulse code.

[0096] The emotion type includes happy, lost, peaceful, and irritable. A correspondence between the pulse codes and the emotion types is established in advance, so that the corresponding emotion type can be obtained according to the three pulse codes in the same-group pulse codes.

[0097] S203, determining a cross-modal synchrony neuron corresponding to each same-group pulse code according to the emotion type corresponding to each same-group pulse code, and applying the cross-modal synchrony neuron to each same-group pulse code to obtain an emotion feature of the user corresponding to each same-group pulse code.

[0098] The cross-modal synchrony neuron is CMSN, and the CMSN is a trained neuron. Different cross-modal synchrony neurons are trained using pulse codes corresponding to different emotion types, so that the cross-modal synchrony neurons learn emotion features corresponding to the pulse codes of different emotion types.

[0099] A same-group pulse code is input into the cross-modal synchrony neuron, and the cross-modal synchrony neuron outputs an emotion feature, which includes the stability of the emotion.

[0100] For example, if the emotion type corresponding to the same-group pulse code is happy, the emotion feature output by the cross-modal synchrony neuron is whether the emotion of the user is stable.

[0101] In this embodiment, step S300 includes the following specific steps: obtaining a weight of the emotion feature corresponding to each same-group pulse code according to the time difference (i.e., the above-mentioned ), and applying an emotion classifier to each emotion feature and the weight corresponding to each emotion feature to obtain an emotion classification result of the user.

[0102] The weight of the emotion feature is denoted as The calculation formula of the weight of the emotion feature is as follows:​

[0103] (4)

[0104] In the formula, Represents emotional characteristics, For cross-modal synchronization neurons, pulse encoding based on three modalities is used to generate signals from three modalities. Choose one from the middle In other words, cross-modal synchronous neurons not only output the aforementioned emotional features but also output... .

[0105] The emotional features output by each cross-modal synchronized neuron, and the weights of those emotional features. The data is input into an emotion classifier, which outputs emotion classification results, including positive, negative, neutral, and ambivalent emotions. The emotion classifier also outputs which modality of biological data played a dominant role in the emotion classification result. The emotion classifier itself is an existing technology.

[0106] Another embodiment provides a method for updating synaptic weights of cross-modal synchronous neurons, including: updating the time difference (i.e., the one described above). The asymmetric time-dependent synaptic plasticity mechanism is applied to obtain the change in the weights of the anterior and posterior synapses of the cross-modal synchronous neuron; the weights between the anterior and posterior synapses are updated based on the change in the weights of the anterior and posterior synapses.

[0107] The asymmetric time-dependent synaptic plasticity mechanism, also known as A-STDP, or Affective-biased Asymmetric STDP, uses formula (5) to calculate the change in weights between synapses and updates the weights based on this change.

[0108] (5)

[0109] In the formula, This represents the change in weight between the anterior and posterior synapses. Representing emotion type (using) The emotion bias factor (representing emotion type), when When the represented emotion type is peaceful, The value is 1. Represents the enhancement factor under positive time difference. This represents the enhancement factor under negative time difference. Represents a positive time constant (that is) (Time constant corresponding to a value greater than zero) Represents a negative time constant (i.e.) (The time constant corresponding to a value less than zero).

[0110] The embodiment can also calculate according to formula (6) :

[0111] ; (6)

[0112] In the formula, represents an emotional bias factor of one of the emotion types, is an exponential decay type function about , and is a constant related to the emotion type.

[0113] The embodiment also provides an asynchronous multi-modal emotion recognition device, as shown in Figure 4 , the device comprises the following components:

[0114] A pulse coding module 01 is configured to acquire user emotion-related multi-modal biological data, the multi-modal biological data is asynchronous data, and the multi-modal biological data is respectively pulse coded to obtain respective pulse sequences of the multi-modal biological data.

[0115] A feature recognition module 02 is configured to apply a cross-modal synchronous neuron to each pulse sequence based on the firing time of the pulse coding on the pulse sequence to obtain emotion features of the user.

[0116] An emotion classification module 03 is configured to apply an emotion classifier to the emotion features to obtain an emotion classification result of the user.

[0117] Based on the above embodiment, the application further provides a terminal device, and a principle block diagram thereof can be as shown in Figure 5 . The terminal device comprises a processor, a memory, a network interface, and a display screen connected through a system bus. The processor of the terminal device is configured to provide computing and control capabilities. The memory of the terminal device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal device is configured to communicate with external terminals through network connection. The computer program is executed by the processor to implement an asynchronous multi-modal emotion recognition method. The display screen of the terminal device can be a liquid crystal display screen or an electronic ink display screen.

[0118] Those skilled in the art can understand that Figure 5The principle block diagram shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the terminal device to which the scheme of the present application is applied. The specific terminal device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0119] In one embodiment, a terminal device is provided, which includes a memory, a processor, and an asynchronous multi-modal emotion recognition program stored in the memory and executable on the processor. When the processor executes the asynchronous multi-modal emotion recognition program, the following operation instructions are implemented.

[0120] Obtaining each modality biological data of a user related to emotion, the each modality biological data being asynchronous data, pulse coding the each modality biological data respectively to obtain a pulse sequence of the each modality biological data;

[0121] Applying a cross-modal synchronous neuron to each of the pulse sequences based on the firing time of the pulse coding on the each of the pulse sequences to obtain an emotion feature of the user;

[0122] Applying an emotion classifier to the emotion feature to obtain an emotion classification result of the user.

[0123] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0124] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An asynchronous multimodal emotion recognition method, characterized in that, include: Acquire user-related modal biometric data, which is asynchronous data. Perform pulse coding on each modal biometric data to obtain pulse sequences for each modal biometric data. The modal biometric data includes facial images, hand images, and voice signals. Based on the release time of the pulse codes on each of the pulse sequences, a cross-modal synchronization neuron is applied to each of the pulse sequences to obtain the user's emotional features. The cross-modal synchronization neuron is used to identify the user's emotional features based on the three pulse codes and the order of their release times. The three pulse codes are pulse codes extracted from the speech pulse sequence, hand pulse sequence, and face pulse sequence, respectively. An emotion classifier is applied to the emotion features to obtain the emotion classification result for the user. Based on the firing time of the pulse codes on each of the pulse sequences, cross-modal synchronous neurons are applied to each of the pulse sequences to obtain the user's emotional characteristics, including: Extract each group of pulse codes from each of the pulse sequences, each group of pulse codes containing the pulse codes of each modality of biological data at its respective release time, and the time difference between the respective release times being less than a threshold; Determine the emotion type corresponding to each of the pulse codes in the same group; Based on the emotion type corresponding to each of the same group of pulse codes, the cross-modal synchronization neurons corresponding to each of the same group of pulse codes are determined, and the cross-modal synchronization neurons are applied to each of the same group of pulse codes to obtain the emotional features of the user corresponding to each of the same group of pulse codes. Applying an emotion classifier to the emotion features yields the user's emotion classification result, including: Based on the time difference, the weights of the emotional features corresponding to each of the pulse codes in the same group are obtained; An emotion classifier is applied to each of the stated emotion features and their respective weights to obtain the user's emotion classification result.

2. The asynchronous multimodal emotion recognition method as described in claim 1, characterized in that, Pulse coding is performed on the biological data of each modality to obtain the pulse sequence of each biological data modality, including: The intensity of the change in the speech signal is determined, and the pulse time interval of the pulse code is determined based on the intensity of the change, and a speech pulse sequence of the speech signal is constructed based on the pulse time interval corresponding to the speech signal; Extract key points from the hand image, determine the gesture posture presented by the key points, and determine the gesture semantic category corresponding to the gesture posture; Based on the gesture semantic category, the hand pulse sequence of the hand image is obtained; Extract facial features related to facial expressions from the facial image; Based on the pulse time interval of the speech pulse sequence and the pulse time interval of the hand pulse sequence, the pulse time interval of the pulse code of the face image is determined, and based on the pulse time interval corresponding to the face image and the face features, the face pulse sequence of the face image is constructed.

3. The asynchronous multimodal emotion recognition method as described in claim 2, characterized in that, Determining the intensity of changes in the speech signal and determining the pulse time interval of the pulse code based on the intensity of changes includes: Extract the instantaneous frequency and instantaneous amplitude of the speech signal, and determine the power derivative of the instantaneous frequency with respect to time and the amplitude derivative of the instantaneous amplitude with respect to time; The intensity of the change in the speech signal is obtained based on the power derivative and the amplitude derivative. The pulse time interval is obtained by applying the natural exponential function to the intensity of the change, wherein the natural exponential function is a decreasing function.

4. The asynchronous multimodal emotion recognition method as described in claim 2, characterized in that, Based on the gesture semantic category, the hand pulse sequence of the hand image is obtained, including: Determine a pulse coding template that matches the gesture semantic category; Based on the pulse coding template, the hand pulse sequence of the hand image is obtained.

5. The asynchronous multimodal emotion recognition method as described in claim 1, characterized in that, The synaptic weight update method of the cross-modal synchronous neuron includes: By applying the asymmetric time-dependent synaptic plasticity mechanism to the time difference, the change in synaptic weights before and after the cross-modal synchronous neuron is obtained. The weights between the anterior and posterior synapses are updated based on the changes in their weights.

6. An asynchronous multimodal emotion recognition device, characterized in that, The device comprises the following components: The pulse coding module is used to acquire user-related modal biodata of various emotions. The modal biodata is asynchronous. Pulse coding is performed on each modal biodata to obtain the pulse sequence of each modal biodata. The modal biodata includes face images, hand images, and voice signals. The feature recognition module is used to apply cross-modal synchronization neurons to each of the pulse sequences based on the release time of the pulse codes on each pulse sequence to obtain the user's emotional features. The cross-modal synchronization neurons are used to recognize the user's emotional features based on the three pulse codes and the order of their release times. The three pulse codes are pulse codes extracted from the speech pulse sequence, hand pulse sequence, and face pulse sequence, respectively. The emotion classification module is used to apply an emotion classifier to the emotion features to obtain the emotion classification result of the user. Based on the firing time of the pulse codes on each of the pulse sequences, cross-modal synchronous neurons are applied to each of the pulse sequences to obtain the user's emotional characteristics, including: Extract each group of pulse codes from each of the pulse sequences, each group of pulse codes containing the pulse codes of each modality of biological data at its respective release time, and the time difference between the respective release times being less than a threshold; Determine the emotion type corresponding to each of the pulse codes in the same group; Based on the emotion type corresponding to each of the same group of pulse codes, the cross-modal synchronization neurons corresponding to each of the same group of pulse codes are determined, and the cross-modal synchronization neurons are applied to each of the same group of pulse codes to obtain the emotional features of the user corresponding to each of the same group of pulse codes. Applying an emotion classifier to the emotion features yields the user's emotion classification result, including: Based on the time difference, the weights of the emotional features corresponding to each of the pulse codes in the same group are obtained; An emotion classifier is applied to each of the stated emotion features and their respective weights to obtain the user's emotion classification result.

7. A terminal device, characterized in that, The terminal device includes a memory, a processor, and an asynchronous multimodal emotion recognition program stored in the memory and executable on the processor. When the processor executes the asynchronous multimodal emotion recognition program, it implements the steps of the asynchronous multimodal emotion recognition method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an asynchronous multimodal emotion recognition program, which, when executed by a processor, implements the steps of the asynchronous multimodal emotion recognition method as described in any one of claims 1-5.

Images