A system for real-time analysis of conversation emotions based on speech and physiological features

Abstract

The present invention provides a system for real-time analysis of conversational emotions based on speech and physiological characteristics. The system includes an audio acquisition module, a physiological characteristic acquisition module, and an emotion analysis module, each electrically connected to the audio acquisition module and the physiological characteristic acquisition module. The audio acquisition module is used to acquire speech audio data during a user's conversation and extract speech features from it. The physiological characteristic acquisition module is used to acquire physiological images of the user during the conversation using an optical detector and extract physiological features from them, including facial and body movement feature data. The emotion analysis module is used to temporally align speech features with physiological features to obtain emotion feature data, classify and identify the user's real-time emotions based on the emotion feature data, and output the identified emotion type and level. The present invention achieves more accurate real-time emotion analysis based on speech and physiological features.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular to a system for real-time analysis of conversation emotions based on speech and physiological characteristics. Background Art

[0002] Emotion recognition is a research area in artificial intelligence and machine learning, aiming to enable computers to identify, understand, and respond to human emotional states. Existing emotion analysis systems primarily rely on physiological feature analysis or semantic analysis of conversational speech. Physiological features primarily include facial expressions and body movements. These analysis methods require pre-training the analysis model using real-world data from the general public. However, the subjects being analyzed may have individual characteristics in their conversational style and physiological characteristics. When these characteristics differ from the general public, emotion analysis results obtained solely based on any of these methods are prone to errors.

[0003] Therefore, how to organically combine the semantic analysis of speech and the analysis of physiological characteristics to obtain more accurate emotion analysis results is a technical problem that urgently needs to be solved. Summary of the Invention

[0004] In order to at least solve the technical problems existing in the above-mentioned background technology, the present invention provides a system, electronic device, computer storage medium and computer program product for real-time analysis of conversation emotions based on speech and physiological characteristics.

[0005] The present invention provides a system for real-time analysis of conversation emotions based on speech and physiological characteristics. The system includes an audio acquisition module, a physiological characteristic acquisition module, and an emotion analysis module. The emotion analysis module is electrically connected to the audio acquisition module and the physiological characteristic acquisition module respectively.

[0006] The audio acquisition module is used to collect speech audio data during the user's conversation and extract speech features from it;

[0007] The physiological feature acquisition module is used to collect physiological images of the user during the conversation through an optical detector and extract physiological features therefrom, wherein the physiological features include facial feature data and body movement feature data;

[0008] The emotion analysis module is used to perform time alignment on the speech features and the physiological features to obtain emotion feature data, classify and identify the user's real-time emotions based on the emotion feature data, and output the identified emotion type and level.

[0009] In some embodiments, the audio collection module is used to collect speech audio data during user conversations and extract speech features therefrom, including:

[0010] The audio collection module collects first speech audio data during the user conversation and extracts a first speech feature therefrom;

[0011] Inputting the first speech feature into an emotion fluctuation model to obtain an emotion fluctuation evaluation value; if the emotion fluctuation evaluation value is higher than a fluctuation threshold, retaining the first speech feature and continuing to extract a second speech feature from the second speech audio data during the user conversation;

[0012] The first speech feature and the second speech feature are used as the speech features.

[0013] In some embodiments, the physiological feature acquisition module is configured to acquire physiological images of the user during the conversation using an optical detector and extract physiological features therefrom, including:

[0014] The physiological feature acquisition module acquires physiological images of the user during the conversation through an optical detector, and recognizes and obtains facial images, body images, and limb images from the physiological images;

[0015] Extracting facial dynamic features from the facial image to obtain facial feature data;

[0016] The body dynamic features and the limb dynamic features are extracted from the body image and the limb image respectively, and the two are fused to obtain the limb movement feature data.

[0017] In some embodiments, the emotion analysis module is configured to perform time registration on the speech feature and the physiological feature, including:

[0018] Aligning the speech features and the physiological features according to the acquisition time, and then extracting target speech features and target physiological features from the speech features and the physiological features respectively using a preset time frame;

[0019] The time frame is determined based on the first speech feature.

[0020] In some embodiments, the time frame is determined as follows:

[0021] Acquiring historical speech features and historical physiological features involving multiple users, calculating a time deviation value between each of the historical physiological features and the historical speech features, and fitting all of the time deviation values ​​to obtain a time deviation equivalent value;

[0022] Determine the initial duration of the time frame according to the time deviation equivalent value;

[0023] Inputting the first speech feature into an emotion recognition model, and the emotion recognition model predicting a number of preliminary emotion types and corresponding matching values ​​based on the first speech feature;

[0024] If any of the matching values ​​is higher than the matching threshold, the first coefficient is multiplied by the initial duration to obtain a first target duration; if all of the matching values ​​are lower than the matching threshold, the average of the matching values ​​is calculated, a second coefficient is determined based on the average, and the second coefficient is multiplied by the initial duration to obtain a second target duration; wherein the first coefficient is smaller than the second coefficient;

[0025] The first target duration or the second target duration is determined as the duration of the time frame.

[0026] In some embodiments, extracting target speech features and target physiological features from the speech features and the physiological features using a preset time frame includes:

[0027] The target physiological feature is extracted from the physiological feature using a time frame corresponding to the initial duration, and the target speech feature is extracted from the speech feature using a time frame corresponding to the first target duration or the second target duration.

[0028] In some embodiments, the emotion analysis module classifies and identifies the user's real-time emotions based on the emotion feature data, and outputs the identified emotion type and level, including:

[0029] The emotion feature data is input into the emotion recognition model, and the emotion recognition model outputs the emotion type and level of the user, that is, completing the classification and recognition of the user's real-time emotion;

[0030] The emotion recognition model is implemented based on Transformer.

[0031] The present invention also provides an electronic device, which is applied to a system for real-time analysis of conversation emotions based on speech and physiological characteristics as described in any of the above items; the electronic device includes: a memory storing executable program code; a processor coupled to the memory; the processor can call the executable program code stored in the memory.

[0032] The present invention also provides a computer storage medium, which is applied to a system for real-time analysis of conversation emotions based on speech and physiological characteristics as described in any of the above items; a computer program is stored on the storage medium, and the computer program can be called and executed by a processor.

[0033] The present invention also provides a computer program product, which is applied to any of the aforementioned systems for real-time analysis of conversation emotions based on speech and physiological features; the computer program product includes a computer program that can be called and executed by a processor.

[0034] The beneficial effects of the present invention are:

[0035] The solution of the present invention combines both linguistic and physiological emotional features to analyze user emotions, resulting in more accurate and reliable analysis results. Furthermore, the present invention takes this practical situation into account by temporally aligning speech and physiological features, allowing them to associate speech and physiological features that belong to the same emotion. This allows for the acquisition of minimal emotional feature data containing non-emotional features, effectively improving the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0037] Figure 1 This is a schematic diagram of the structure of a system for real-time analysis of conversation emotions based on speech and physiological features disclosed in an embodiment of the present invention.

[0038] Figure 2 It is a structural diagram of an electronic device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0040] The terms used in the examples of this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a," "the," and "the" used in the examples of this application and the appended claims are also intended to include plural forms, and unless the context clearly indicates otherwise, "a plurality" generally includes at least two.

[0041] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0042] It should be understood that although the terms "first," "second," "third," etc. may be used in the embodiments of this application to describe "...," these "..." should not be limited to these terms. These terms are merely used to distinguish "...." For example, "first..." could also be referred to as "second...", and similarly, "second..." could also be referred to as "first..." without departing from the scope of the embodiments of this application.

[0043] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0044] It should also be noted that the terms "include," "comprises," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a product or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such product or system. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the product or system comprising the element.

[0045] See Figure 1 As shown, an embodiment of the present invention discloses a system for real-time analysis of conversation emotions based on speech and physiological characteristics, the system comprising an audio acquisition module, a physiological characteristic acquisition module, and an emotion analysis module, wherein the emotion analysis module is electrically connected to the audio acquisition module and the physiological characteristic acquisition module respectively;

[0046] The audio acquisition module is used to collect speech audio data during the user's conversation and extract speech features from it;

[0047] The physiological feature acquisition module is used to collect physiological images of the user during the conversation through an optical detector and extract physiological features therefrom, wherein the physiological features include facial feature data and body movement feature data;

[0048] The emotion analysis module is used to perform time alignment on the speech features and the physiological features to obtain emotion feature data, classify and identify the user's real-time emotions based on the emotion feature data, and output the identified emotion type and level.

[0049] The present invention constructs an audio acquisition module and a physiological feature acquisition module, each used to extract the user's linguistic and physiological characteristics during a conversation. The emotion analysis module then performs temporal registration on these feature data, thereby obtaining emotional feature data that can be used to comprehensively characterize the user's emotions. Finally, this emotional feature data is used to identify the emotions expressed by the user during the conversation, that is, to obtain the corresponding emotion type and level. The solution of the present invention combines emotional features from both linguistic and physiological characteristics to analyze the user's emotions, resulting in more accurate and reliable analysis results.

[0050] At the same time, for the same emotion, there is a time difference between the verbal expression and the physiological expression of different users. The present invention takes this actual situation into consideration and, by performing time registration on the verbal features and the physiological features, associates the verbal features and the physiological features belonging to the same emotion, thereby obtaining as little emotional feature data as possible containing non-emotional features, which can effectively improve the accuracy of emotion recognition.

[0051] Emotion types include positive emotions, neutral emotions, and negative emotions. Positive emotions can include several different levels, such as excitement and happiness. Negative emotions can also include several different levels, such as worry and sadness. Neutral emotions can also include several different levels. The present invention does not limit the specific distribution of emotion types and their levels.

[0052] In some embodiments, the audio collection module is used to collect speech audio data during user conversations and extract speech features therefrom, including:

[0053] The audio collection module collects first speech audio data during the user conversation and extracts a first speech feature therefrom;

[0054] Inputting the first speech feature into an emotion fluctuation model to obtain an emotion fluctuation evaluation value; if the emotion fluctuation evaluation value is higher than a fluctuation threshold, retaining the first speech feature and continuing to extract a second speech feature from the second speech audio data during the user conversation;

[0055] The first speech feature and the second speech feature are used as the speech features.

[0056] In this embodiment, during a user conversation, the audio acquisition module (including a microphone) collects the user's speech audio in real time, and uses its own emotion fluctuation model to predict whether the user has emotional fluctuations in speech, and obtains a corresponding emotion fluctuation evaluation value. When the emotion fluctuation evaluation value is higher than the fluctuation threshold, it indicates that the user's emotions have a tendency to change compared to the identified emotion type and level in the previous cycle. At this time, on the one hand, the extracted first speech feature is retained, and the second speech feature is further extracted from the subsequent speech audio, and the first speech feature and the second speech feature are used as the final speech feature. It can be seen that the speech features finally obtained by the present invention include both the speech features in the early stage of the user's emotion change and the speech features during the period of the user's emotion change. These features contain more feature data that can characterize the emotion type and level, and the subsequent emotion recognition based on them will be more accurate.

[0057] The emotion fluctuation model of the present invention can be based on a finite state machine structure. The finite state machine can describe the transition process of the user's emotional state. By analyzing the state sequence in the first speech feature, the probability of the user's emotional state transitioning to another new emotional state can be estimated, and this probability is the emotion fluctuation evaluation value.

[0058] The speech features in the present invention primarily include pitch, volume, speaking rate, intonation, formants, fundamental frequency, harmonic structure, non-periodic characteristics, and vibrato. Pitch changes can reflect emotional tension or relaxation, volume is generally correlated with emotional intensity, speaking speed can reflect emotions such as excitement, nervousness, or hesitation, intonation can convey emotional changes, formant changes can affect the emotional tone of a voice, fundamental frequency stability or changes can reflect emotional stability or fluctuation, harmonic components and their ratios can influence emotional perception, non-periodic sound components such as breathing and swallowing can provide additional information about emotional state, and the amplitude and frequency of vibrato can provide clues to emotional tension or excitement. At least one of the above-mentioned speech features can be selected based on the general situation of the subject to be analyzed, with multiple combinations preferably used for combined analysis.

[0059] In some embodiments, the physiological feature acquisition module is configured to acquire physiological images of the user during the conversation using an optical detector and extract physiological features therefrom, including:

[0060] The physiological feature acquisition module acquires physiological images of the user during the conversation through an optical detector, and recognizes and obtains facial images, body images, and limb images from the physiological images;

[0061] Extracting facial dynamic features from the facial image to obtain facial feature data;

[0062] The body dynamic features and the limb dynamic features are extracted from the body image and the limb image respectively, and the two are fused to obtain the limb movement feature data.

[0063] In this embodiment, the physiological feature acquisition module is equipped with an optical detector (various types of cameras) that captures several high-definition images of the user during a conversation. These images primarily include the user's upper body, from which facial, torso, and limb region images are identified and segmented. Specified dynamic feature extraction is performed on the facial region images to obtain facial feature data. Furthermore, specified dynamic feature extraction is performed on the torso and limb region images to obtain dynamic features of the torso and limbs (i.e., arms and hands). By matching the limb dynamic features with the torso dynamic features, feature data representing the user's body movements during the conversation is obtained. In some cases, body movements cannot accurately represent emotions and require fusion with torso movements. This fusion process removes features of body movements that are not relevant to emotions.

[0064] Among them, facial features include the position and shape of eyebrows, the degree of eye opening, wrinkles at the corners of the eyes, the shape of the lips, the opening and closing of the mouth, the lift of the cheeks, the shape of the chin and jaw, the tension of the facial muscles, micro-expressions, etc.

[0065] In some embodiments, the emotion analysis module is configured to perform time registration on the speech feature and the physiological feature, including:

[0066] Aligning the speech features and the physiological features according to the acquisition time, and then extracting target speech features and target physiological features from the speech features and the physiological features respectively using a preset time frame;

[0067] The time frame is determined based on the first speech feature.

[0068] In this embodiment, after extracting the user's speech and physiological features during a conversation, they are first aligned in time based on the moment of acquisition. Then, using a time frame, target speech and physiological features belonging to the same time period are captured. These features constitute the features associated with the user's real-time emotion. In other words, the changes reflected in the target speech and physiological features are influenced by the user's real-time emotion.

[0069] Among them, the speech features and physiological features are aligned and registered according to the collection time, and the starting time of the two can be aligned, that is, the time corresponding to the earliest data of the speech features and physiological features is the same or highly close.

[0070] In some embodiments, the time frame is determined as follows:

[0071] Acquiring historical speech features and historical physiological features involving multiple users, calculating a time deviation value between each of the historical physiological features and the historical speech features, and fitting all of the time deviation values ​​to obtain a time deviation equivalent value;

[0072] Determine the initial duration of the time frame according to the time deviation equivalent value;

[0073] Inputting the first speech feature into an emotion recognition model, and the emotion recognition model predicting a number of preliminary emotion types and corresponding matching values ​​based on the first speech feature;

[0074] If any of the matching values ​​is higher than the matching threshold, the first coefficient is multiplied by the initial duration to obtain a first target duration; if all of the matching values ​​are lower than the matching threshold, the average of the matching values ​​is calculated, a second coefficient is determined based on the average, and the second coefficient is multiplied by the initial duration to obtain a second target duration; wherein the first coefficient is smaller than the second coefficient;

[0075] The first target duration or the second target duration is determined as the duration of the time frame.

[0076] In this embodiment, after aligning the start times of speech and physiological features, it is necessary to determine their end times. Actual testing has found that during a conversation, when a user expresses a new emotion, their physiological features generally appear before their speech features, and the degree of this "before" varies from user to user. Therefore, when capturing both speech and physiological features, it is necessary to collect more of the speech features.

[0077] To address this, the present invention first collects emotional big data from multiple users, including corresponding paired historical speech and physiological features. It then calculates the time offset of these historical speech and physiological features, and then calculates the time offset equivalent of all time offset values. The time offset equivalent value can be an average or median. This time offset equivalent value is added to the base duration of the time frame to obtain the initial duration of the time frame.

[0078] The aforementioned initial duration represents the general public, but the individual needs of the specific user must still be considered. To this end, the present invention further utilizes a constructed emotion recognition model to pre-identify the emotion of the aforementioned first speech feature. The emotion recognition model essentially calculates the similarity between the first speech feature and multiple pre-stored emotion types and their levels, thereby deriving several preliminary emotion types and corresponding matching values.

[0079] If any matching value is above the matching threshold (e.g., corresponding to multiple levels of the same emotion type), the user's current real-time emotion is highly consistent with the general public's emotional characteristics. If all matching values ​​are below the matching threshold, the user's current real-time emotion is significantly inconsistent with the general public's emotional characteristics. For the former, the present invention uses a smaller first coefficient to appropriately expand the initial duration obtained above, i.e., slightly more speech features are collected when intercepting the target speech features. For the latter, a larger second coefficient is used to further expand the initial duration obtained above, i.e., more speech features are collected when intercepting the target speech features.

[0080] In some embodiments, extracting target speech features and target physiological features from the speech features and the physiological features using a preset time frame includes:

[0081] The target physiological feature is extracted from the physiological feature using a time frame corresponding to the initial duration, and the target speech feature is extracted from the speech feature using a time frame corresponding to the first target duration or the second target duration.

[0082] In this embodiment, the aforementioned embodiment can be to intercept the target speech features and target physiological features according to the same time frame. However, as mentioned above, physiological features are more direct in representing emotions and have higher accuracy, so the target physiological features can be intercepted according to the initial duration that represents the general situation of the public. Compared with physiological features, speech features have weaker direct representation of emotions and are generally more delayed in expression timing. Therefore, the present invention uses a larger time frame corresponding to the first target duration or the second target duration to intercept the target speech features from the speech features. Obviously, the target speech feature data has a larger time span than the target physiological feature data, and the corresponding data volume is also larger.

[0083] In some embodiments, the emotion analysis module classifies and identifies the user's real-time emotions based on the emotion feature data, and outputs the identified emotion type and level, including:

[0084] The emotion feature data is input into the emotion recognition model, and the emotion recognition model outputs the emotion type and level of the user, that is, completing the classification and recognition of the user's real-time emotion;

[0085] The emotion recognition model is implemented based on Transformer.

[0086] In this embodiment, the present invention has previously used the emotion recognition model to pre-identify the emotion of the first speech feature based on the language elements, and in the final emotion recognition, a comprehensive analysis of the speech elements and body movement elements is performed to obtain a more accurate emotion recognition result.

[0087] Furthermore, traditional RNN models cannot perform parallel computations, making them incompatible with the present invention's approach of analyzing emotion based on both speech and body language elements, and integrating the results of each analysis. Furthermore, the self-attention mechanism in the Transformer can produce a more interpretable model. Therefore, the present invention prefers to use the Transformer to implement the emotion recognition model. The model construction and training process will not be elaborated upon in this article.

[0088] Among them, the emotion recognition model can first predict the user's preliminary emotion type and level based on the speech feature matrix in the emotion feature data, and then verify the confidence of the aforementioned preliminary emotion type and level based on the physiological feature matrix in the emotion feature data, thereby realizing the correction of the preliminary emotion type and level, and obtaining the final more accurate emotion type and level.

[0089] See Figure 2 As shown, the present invention also discloses an electronic device, which is applied to a system for real-time analysis of conversation emotions based on speech and physiological characteristics as described in any of the above items; the electronic device includes: a memory storing executable program code; a processor coupled to the memory; the processor can call the executable program code stored in the memory.

[0090] The present invention also discloses a computer storage medium, which is applied to a system for real-time analysis of conversation emotions based on speech and physiological characteristics as described in any of the above items; a computer program is stored on the storage medium, and the computer program can be called and executed by a processor.

[0091] The present invention also discloses a computer program product, which is applied to any of the aforementioned systems for real-time analysis of conversation emotions based on speech and physiological characteristics; the computer program product includes a computer program that can be called and executed by a processor.

[0092] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0093] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that the present application can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, an information push server, or a network device, etc.) to execute the methods described in various embodiments of the present application or certain parts of the embodiments.

[0094] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0095] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0096] It should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation method can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0097] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A system for real-time analysis of conversation emotions based on speech and physiological features, characterized by: The system includes an audio acquisition module, a physiological characteristics acquisition module, and an emotion analysis module, wherein the emotion analysis module is electrically connected to the audio acquisition module and the physiological characteristics acquisition module respectively; The audio acquisition module is used to collect speech audio data during the user's conversation and extract speech features from it; The physiological feature acquisition module is used to collect physiological images of the user during the conversation through an optical detector and extract physiological features therefrom, wherein the physiological features include facial feature data and body movement feature data; The emotion analysis module is configured to perform time registration on the speech features and the physiological features to obtain emotion feature data, classify and identify the user's real-time emotions based on the emotion feature data, and output the identified emotion type and level; The audio acquisition module is configured to acquire speech audio data during a user conversation and extract speech features therefrom, including: the audio acquisition module acquiring first speech audio data during the user conversation and extracting a first speech feature therefrom; inputting the first speech feature into an emotion fluctuation model to obtain an emotion fluctuation evaluation value; if the emotion fluctuation evaluation value is higher than a fluctuation threshold, retaining the first speech feature and continuing to extract a second speech feature from second speech audio data during the user conversation; and using the first speech feature and the second speech feature as the speech feature; The emotion analysis module is configured to perform time registration on the speech feature and the physiological feature, including: aligning the speech feature and the physiological feature according to the acquisition time, and then extracting target speech features and target physiological features from the speech feature and the physiological feature respectively using a preset time frame; wherein the time frame is determined based on the first speech feature; The time frame is determined as follows: historical speech features and historical physiological features involving multiple users are obtained, the time deviation value of each historical physiological feature and the historical speech feature is calculated, and all the time deviation values ​​are fitted to obtain a time deviation equivalent value; the initial duration of the time frame is determined based on the time deviation equivalent value; the first speech feature is input into the emotion recognition model, and the emotion recognition model predicts a number of preliminary emotion types and corresponding matching values ​​based on this; if any of the matching values ​​is higher than the matching threshold, the first coefficient is multiplied by the initial duration to obtain a first target duration; if all the matching values ​​are lower than the matching threshold, the average value of the matching values ​​is calculated, the second coefficient is determined based on the average value, and the second coefficient is multiplied by the initial duration to obtain a second target duration; wherein, the first coefficient is less than the second coefficient; the first target duration or the second target duration is determined as the duration of the time frame.

2. The system for real-time analysis of conversation emotions based on speech and physiological features according to claim 1, characterized in that: The physiological feature acquisition module is used to collect physiological images of the user during the conversation process through an optical detector and extract physiological features therefrom, including: The physiological feature acquisition module acquires physiological images of the user during the conversation through an optical detector, and recognizes and obtains facial images, body images, and limb images from the physiological images; Extracting facial dynamic features from the facial image to obtain facial feature data; The body dynamic features and the limb dynamic features are extracted from the body image and the limb image respectively, and the two are fused to obtain the limb movement feature data.

3. The system for real-time analysis of conversation emotions based on speech and physiological features according to claim 1, characterized in that: The method of using a preset time frame to extract target speech features and target physiological features from the speech features and the physiological features respectively includes: The target physiological feature is extracted from the physiological feature using a time frame corresponding to the initial duration, and the target speech feature is extracted from the speech feature using a time frame corresponding to the first target duration or the second target duration.

4. The system for real-time analysis of conversation emotions based on speech and physiological features according to claim 3, characterized in that: The emotion analysis module classifies and identifies the user's real-time emotions based on the emotion feature data, and outputs the identified emotion type and level, including: The emotion feature data is input into the emotion recognition model, and the emotion recognition model outputs the emotion type and level of the user, that is, completing the classification and recognition of the user's real-time emotion; The emotion recognition model is implemented based on Transformer.

5. An electronic device, characterized in that: A system for real-time analysis of conversation emotions based on speech and physiological characteristics, as described in any one of claims 1-4; the electronic device comprises: a memory storing executable program code; a processor coupled to the memory; the processor being capable of calling the executable program code stored in the memory.

6. A computer storage medium, characterized in that: A system for real-time analysis of conversation emotions based on speech and physiological characteristics, as described in any one of claims 1-4; a computer program is stored on the storage medium, and the computer program can be called and executed by a processor.

7. A computer program product, characterized in that: A system for real-time analysis of conversation emotions based on speech and physiological features as described in any one of claims 1-4; the computer program product includes a computer program that can be called and executed by a processor.

Images