System for carrying out real-time analysis on dialogue emotion based on speech and physiological features

Abstract

The invention provides a system for analyzing dialogue emotion in real time based on speech and physiological features. The system comprises an audio acquisition module, a physiological feature acquisition module and an emotion analysis module, wherein the emotion analysis module is electrically connected with the audio acquisition module and the physiological feature acquisition module; the audio acquisition module is used for acquiring speech audio data in a user dialogue process and extracting speech features from the speech audio data; the physiological feature collection module is used for collecting physiological images in the dialogue process of the user through an optical detector and extracting physiological features from the physiological images, and the physiological features comprise facial feature data and limb movement feature data; and the emotion analysis module is used for carrying out time registration on the speech features and the physiological features to obtain emotion feature data, carrying out classification recognition on real-time emotions of the user according to the emotion feature data, and outputting emotion types and grades obtained through recognition. According to the method, more accurate real-time emotion analysis is obtained based on the speech features and the physiological features.

Description

Technical Field

[0001] The present invention relates to the technical field of emotion recognition, and in particular to a system for real-time analysis of conversation emotions based on speech and physiological characteristics. Background Art

[0002] Emotion recognition is a research direction in the field of artificial intelligence and machine learning, which aims to enable computers to recognize, understand and respond to human emotional states. Existing emotion analysis systems are mainly implemented through physiological feature analysis or semantic analysis of conversational speech. Physiological features mainly include facial expressions and body movements. This analysis method requires pre-training the analysis model with actual measured data from the public. However, the analyzed object may have some personalized characteristics in terms of conversation style and physiological representation. When these characteristics are different from the general situation of the public, the emotion analysis results obtained by relying solely on any of the above methods are prone to errors.

[0003] Therefore, how to organically combine the semantic analysis of speech and the analysis of physiological characteristics to obtain more accurate emotion analysis results is a technical problem that urgently needs to be solved. Summary of the invention

[0004] In order to at least solve the technical problems existing in the above-mentioned background technology, the present invention provides a system, electronic device, computer storage medium and computer program product for real-time analysis of conversation emotions based on speech and physiological characteristics.

[0005] The present invention provides a system for real-time analysis of conversation emotions based on speech and physiological characteristics, the system comprising an audio acquisition module, a physiological characteristic acquisition module, and an emotion analysis module, the emotion analysis module being electrically connected to the audio acquisition module and the physiological characteristic acquisition module respectively; The audio collection module is used to collect speech audio data during the user's conversation and extract speech features from it; The physiological feature acquisition module is used to collect physiological images of the user during the conversation process through an optical detector, and extract physiological features therefrom, wherein the physiological features include facial feature data and body movement feature data; The emotion analysis module is used to perform time alignment on the speech features and the physiological features to obtain emotion feature data, classify and identify the user's real-time emotions based on the emotion feature data, and output the identified emotion type and level.

[0006] In some embodiments, the audio collection module is used to collect speech audio data during the user conversation process and extract speech features therefrom, including: The audio collection module collects first speech audio data during the user conversation and extracts a first speech feature therefrom; Inputting the first speech feature into an emotion fluctuation model to obtain an emotion fluctuation evaluation value; if the emotion fluctuation evaluation value is higher than a fluctuation threshold, retaining the first speech feature, and continuing to extract a second speech feature from the second speech audio data during the user conversation; The first speech feature and the second speech feature are used as the speech features.

[0007] In some embodiments, the physiological feature acquisition module is used to acquire physiological images of the user during the conversation process through an optical detector and extract physiological features therefrom, including: The physiological feature acquisition module acquires physiological images of the user during the conversation process through an optical detector, and recognizes and obtains facial images, body images and limb images from the physiological images; Extracting dynamic facial features from the facial image to obtain facial feature data; The body dynamic features and the limb dynamic features are extracted from the body image and the limb image respectively, and the two are fused to obtain the limb movement feature data.

[0008] In some embodiments, the emotion analysis module is used to perform time registration on the speech feature and the physiological feature, including: Aligning and registering the speech feature and the physiological feature according to the acquisition time, and then using a preset time frame to respectively extract a target speech feature and a target physiological feature from the speech feature and the physiological feature; The time frame is determined based on the first speech feature.

[0009] In some embodiments, the time frame is determined as follows: Acquire historical speech features and historical physiological features involving multiple users, calculate time deviation values ​​of each of the historical physiological features and the historical speech features, and fit all the time deviation values ​​to obtain time deviation equivalent values; Determine the initial duration of the time frame according to the time deviation equivalent value; Inputting the first speech feature into an emotion recognition model, and the emotion recognition model predicting a number of preliminary emotion types and corresponding matching values ​​based on the first speech feature; If any of the matching values ​​is higher than the matching threshold, the first coefficient is multiplied by the initial duration to obtain a first target duration; if all the matching values ​​are lower than the matching threshold, the average of the matching values ​​is calculated, a second coefficient is determined based on the average, and the second coefficient is multiplied by the initial duration to obtain a second target duration; wherein the first coefficient is less than the second coefficient; The first target duration or the second target duration is determined as the duration of the time frame.

[0010] In some embodiments, the step of using a preset time frame to extract a target speech feature and a target physiological feature from the speech feature and the physiological feature, respectively, comprises: The target physiological feature is extracted from the physiological feature using a time frame corresponding to the initial duration, and the target speech feature is extracted from the speech feature using a time frame corresponding to the first target duration or the second target duration.

[0011] In some embodiments, the emotion analysis module classifies and identifies the user's real-time emotions according to the emotion feature data, and outputs the identified emotion type and level, including: The emotion feature data is input into the emotion recognition model, and the emotion recognition model outputs the emotion type and level of the user, that is, completes the classification and recognition of the real-time emotion of the user; Among them, the emotion recognition model is implemented based on Transformer.

[0012] The present invention also provides an electronic device, which is applied to a system for real-time analysis of conversation emotions based on speech and physiological characteristics as described in any of the above items; the electronic device includes: a memory storing executable program code; a processor coupled to the memory; the processor can call the executable program code stored in the memory.

[0013] The present invention also provides a computer storage medium, which is applied to a system for real-time analysis of conversation emotions based on speech and physiological characteristics as described in any of the above items; a computer program is stored on the storage medium, and the computer program can be called and executed by a processor.

[0014] The present invention also provides a computer program product, which is applied to a system for real-time analysis of conversation emotions based on speech and physiological features as described in any of the above items; the computer program product includes a computer program that can be called and executed by a processor.

[0015] The beneficial effects of the present invention are: The solution of the present invention combines the emotion features of both language features and physiological features to analyze the user's emotions, and the obtained analysis results are more accurate and reliable. At the same time, the present invention takes this actual situation into consideration, and by performing time registration on the speech features and physiological features, the speech features and physiological features belonging to the same emotion are associated and attributed, thereby obtaining as little emotion feature data containing non-emotion features as possible, which can effectively improve the accuracy of emotion recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 It is a structural diagram of a system for real-time analysis of conversation emotions based on speech and physiological characteristics disclosed in an embodiment of the present invention.

[0018] Figure 2 It is a structural schematic diagram of an electronic device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0021] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0022] It should be understood that although the terms first, second, third, etc. may be used to describe ... in the embodiments of the present application, these ... should not be limited to these terms. These terms are only used to distinguish .... For example, without departing from the scope of the embodiments of the present application, the first ... may also be referred to as the second ..., and similarly, the second ... may also be referred to as the first ....

[0023] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0024] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such a product or system. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the product or system including the elements.

[0025] See also Figure 1 As shown, an embodiment of the present invention discloses a system for real-time analysis of conversation emotions based on speech and physiological characteristics, the system comprising an audio acquisition module, a physiological characteristic acquisition module, and an emotion analysis module, the emotion analysis module being electrically connected to the audio acquisition module and the physiological characteristic acquisition module respectively; The audio collection module is used to collect speech audio data during the user's conversation and extract speech features from it; The physiological feature acquisition module is used to collect physiological images of the user during the conversation process through an optical detector, and extract physiological features therefrom, wherein the physiological features include facial feature data and body movement feature data; The emotion analysis module is used to perform time alignment on the speech features and the physiological features to obtain emotion feature data, classify and identify the user's real-time emotions based on the emotion feature data, and output the identified emotion type and level.

[0026] The present invention constructs an audio acquisition module and a physiological feature acquisition module, which are respectively used to extract the language features and physiological features of the user during the conversation. The emotion analysis module then performs time registration on the feature data of the two, thereby obtaining emotion feature data that can be used to comprehensively characterize the user's emotions. Finally, the emotion feature data is used to identify the emotions expressed by the user during the conversation, that is, to obtain the corresponding emotion type and level. The solution of the present invention combines the emotion features of both language features and physiological features to analyze the user's emotions, and the obtained analysis results are more accurate and reliable.

[0027] At the same time, for the same emotion, there is a time difference between the verbal expression and the physiological expression of different users. The present invention takes this actual situation into consideration and, by performing time alignment on the verbal features and the physiological features, associates and attributes the verbal features and the physiological features belonging to the same emotion, thereby obtaining as little emotional feature data containing non-emotional features as possible, which can effectively improve the accuracy of emotion recognition.

[0028] Emotion types include positive emotions, neutral emotions, and negative emotions. Positive emotions may include several different levels such as excitement and happiness, negative emotions may include several different levels such as worry and sadness, and neutral emotions may also include several different levels. The present invention does not limit the specific distribution of emotion types and their levels.

[0029] In some embodiments, the audio collection module is used to collect speech audio data during the user conversation process and extract speech features therefrom, including: The audio collection module collects first speech audio data during the user conversation and extracts a first speech feature therefrom; Inputting the first speech feature into an emotion fluctuation model to obtain an emotion fluctuation evaluation value; if the emotion fluctuation evaluation value is higher than a fluctuation threshold, retaining the first speech feature, and continuing to extract a second speech feature from the second speech audio data during the user conversation; The first speech feature and the second speech feature are used as the speech features.

[0030] In this embodiment, during the user conversation, the audio acquisition module (including a microphone) collects the user's speech audio in real time, and predicts whether the user has verbal emotional fluctuations through its own emotional fluctuation model, and obtains the corresponding emotional fluctuation evaluation value. When the emotional fluctuation evaluation value is higher than the fluctuation threshold, it means that the user's emotions have a tendency to change compared to the identified emotional type and level in the previous cycle. At this time, on the one hand, the extracted first speech feature is retained, and the second speech feature is continuously extracted from the subsequent speech audio, and the first speech feature and the second speech feature are used as the final speech feature. It can be seen that the speech features finally obtained by the present invention include both the speech features in the early stage when the user's emotions change, and the speech features during the period when the user's emotions change. These features contain more feature data that can characterize the emotion type and its level, and the subsequent emotion recognition based on them will be more accurate.

[0031] The emotion fluctuation model of the present invention can be based on a finite state machine structure. The finite state machine can describe the transition process of the user's emotional state. By analyzing the state sequence in the first speech feature, the probability of the user's emotional state transitioning to another new emotional state can be estimated, and this probability is the emotion fluctuation evaluation value.

[0032] The speech features in the present invention mainly include pitch, volume, speaking speed, intonation, formant, fundamental frequency of sound, harmonic structure of sound, non-periodic characteristics of sound, vibrato of sound, etc. Among them, the change of pitch can reflect the tense or relaxed state of emotion, the size of volume is usually related to the intensity of emotion, the speed of speaking can reflect emotions such as excitement, tension or hesitation, the ups and downs of intonation can convey the change of emotion, the change of formant can affect the emotional color of sound, the stability or change of fundamental frequency can reflect the stability or fluctuation of emotion, the harmonic components and their proportions can affect the perception of emotion, non-periodic sound components, such as breathing sound, swallowing sound, etc., can provide additional information of emotional state, and the amplitude and frequency of vibrato can provide clues of tense or excited emotion. At least one of the above-mentioned types of speech features can be selected according to the general situation of the object to be analyzed, and it is preferred to use multiple types for combined analysis.

[0033] In some embodiments, the physiological feature acquisition module is used to acquire physiological images of the user during the conversation process through an optical detector and extract physiological features therefrom, including: The physiological feature acquisition module acquires physiological images of the user during the conversation process through an optical detector, and recognizes and obtains facial images, body images and limb images from the physiological images; Extracting dynamic facial features from the facial image to obtain facial feature data; The body dynamic features and the limb dynamic features are extracted from the body image and the limb image respectively, and the two are fused to obtain the limb movement feature data.

[0034] In this embodiment, the physiological feature acquisition module is equipped with an optical detector (various types of cameras), which takes several high-definition images of the user during the conversation. The high-definition images mainly include the upper body area of ​​the user, from which the user's facial area image, torso area image and limb area image are identified and segmented. The facial area image is subjected to a specified dynamic feature extraction to obtain facial feature data; and the torso area image and limb area image are subjected to a specified dynamic feature extraction to obtain the dynamic features of the torso and the dynamic features of the limbs (i.e., arms and hands). The dynamic features of the limbs are matched with the dynamic features of the torso to obtain the feature data of the limb movements that can characterize the user during the conversation. In some cases, limb movements cannot accurately represent emotions, but only after the limb movements and torso movements are fused, the limb movement features that are not related to emotions are deleted during the fusion process.

[0035] Among them, facial features include the position and shape of eyebrows, degree of eye opening, wrinkles at the corners of the eyes, shape of lips, mouth opening and closing, cheek lift, shape of chin and jaw, facial muscle tension, micro-expressions, etc.

[0036] In some embodiments, the emotion analysis module is used to perform time registration on the speech feature and the physiological feature, including: Aligning and registering the speech feature and the physiological feature according to the acquisition time, and then using a preset time frame to respectively extract a target speech feature and a target physiological feature from the speech feature and the physiological feature; The time frame is determined based on the first speech feature.

[0037] In this embodiment, after extracting the speech features and physiological features of the user during the conversation, the two are first aligned in time according to the acquisition time, and then the target speech features and target physiological features belonging to the same time period are respectively intercepted using the time frame, and these features constitute the features related to the user's real-time emotions. In other words, the changes reflected by the target speech features and target physiological features are affected by the user's real-time emotions.

[0038] Among them, the speech features and physiological features are aligned and registered according to the collection time, and the starting time of the two can be aligned, that is, the time corresponding to the earliest data of the speech features and the physiological features is the same or highly close.

[0039] In some embodiments, the time frame is determined as follows: Acquire historical speech features and historical physiological features involving multiple users, calculate time deviation values ​​of each of the historical physiological features and the historical speech features, and fit all the time deviation values ​​to obtain time deviation equivalent values; Determine the initial duration of the time frame according to the time deviation equivalent value; Inputting the first speech feature into an emotion recognition model, and the emotion recognition model predicting a number of preliminary emotion types and corresponding matching values ​​based on the first speech feature; If any of the matching values ​​is higher than the matching threshold, the first coefficient is multiplied by the initial duration to obtain a first target duration; if all the matching values ​​are lower than the matching threshold, the average of the matching values ​​is calculated, a second coefficient is determined based on the average, and the second coefficient is multiplied by the initial duration to obtain a second target duration; wherein the first coefficient is less than the second coefficient; The first target duration or the second target duration is determined as the duration of the time frame.

[0040] In this embodiment, after aligning the start time of the speech feature and the physiological feature, it is necessary to determine the end time of the two. In actual tests, it is found that during a conversation, when a user has a new emotion, his physiological features generally appear earlier than the speech features, and the degree of "earlier" is different for different users. Therefore, when intercepting speech features and physiological features, it is necessary to collect more speech features.

[0041] To this end, the present invention first collects emotional big data of multiple users, including corresponding paired historical speech features and historical physiological features, calculates the time deviation of these historical speech features and historical physiological features, and then calculates the time deviation equivalent value of all time deviation values, which can be an average value or a median value. The time deviation equivalent value is added to the basic duration of the time frame to obtain the initial duration of the time frame.

[0042] The above initial duration represents the general situation of the public, and the personalized situation of the specific user still needs to be considered. In this regard, the present invention further uses the constructed emotion recognition model to pre-recognize the emotion of the above-mentioned first speech feature. The emotion recognition model essentially calculates the similarity between the first speech feature and multiple pre-stored emotion types and their levels, so as to obtain several preliminary emotion types and corresponding matching values.

[0043] If any matching value is higher than the matching threshold (for example, corresponding to multiple levels of the same emotion type), it means that the user's real-time emotion at this time is very consistent with the general emotional characteristics of the public, and when all matching values ​​are lower than the matching threshold, it means that the user's real-time emotion at this time is obviously inconsistent with the general emotional characteristics of the public. For the former, the present invention uses a smaller first coefficient to appropriately expand the initial duration obtained above, that is, to collect slightly more speech features when intercepting the target speech features; and for the latter, a larger second coefficient is used to expand the initial duration obtained above to a greater extent, that is, to collect more speech features when intercepting the target speech features.

[0044] In some embodiments, the step of using a preset time frame to extract a target speech feature and a target physiological feature from the speech feature and the physiological feature, respectively, comprises: The target physiological feature is extracted from the physiological feature using a time frame corresponding to the initial duration, and the target speech feature is extracted from the speech feature using a time frame corresponding to the first target duration or the second target duration.

[0045] In this embodiment, the aforementioned embodiment can be to intercept the target speech features and target physiological features according to the same time frame. However, as mentioned above, physiological features are more direct and more accurate in representing emotions, so the target physiological features can be intercepted according to the initial duration that represents the general situation of the public. Compared with physiological features, speech features have weaker direct representation of emotions and are generally more delayed in expression timing. Therefore, the present invention uses a larger time frame corresponding to the first target duration or the second target duration to intercept the target speech features from the speech features. Obviously, the target speech feature data has a larger time span than the target physiological feature data, and the corresponding data volume is also larger.

[0046] In some embodiments, the emotion analysis module classifies and identifies the user's real-time emotions according to the emotion feature data, and outputs the identified emotion type and level, including: The emotion feature data is input into the emotion recognition model, and the emotion recognition model outputs the emotion type and level of the user, that is, completes the classification and recognition of the real-time emotion of the user; Among them, the emotion recognition model is implemented based on Transformer.

[0047] In this embodiment, the present invention has previously used the emotion recognition model to pre-identify the emotion of the first speech feature based on the language elements, and in the final emotion recognition, a comprehensive analysis of the speech elements and body movement elements is performed to obtain a more accurate emotion recognition result.

[0048] In addition, the traditional RNN model cannot be calculated in parallel, which is incompatible with the solution of the present invention that performs emotion analysis based on two lines, speech elements and body movement elements, and then integrates the respective analysis results; moreover, the self-attention mechanism in Transformer can produce a more interpretable model. Therefore, the present invention preferably uses Transformer to implement the emotion recognition model. The present invention will not go into details about the construction and training process of the model.

[0049] Among them, the emotion recognition model can first predict the user's preliminary emotion type and level based on the speech feature matrix in the emotion feature data, and then verify the confidence of the aforementioned preliminary emotion type and level based on the physiological feature matrix in the emotion feature data, thereby realizing the correction of the preliminary emotion type and level, and obtaining the final more accurate emotion type and level.

[0050] See also Figure 2 As shown, the present invention also discloses an electronic device, which is applied to a system for real-time analysis of conversation emotions based on speech and physiological characteristics as described in any of the above items; the electronic device includes: a memory storing executable program code; a processor coupled to the memory; the processor can call the executable program code stored in the memory.

[0051] The present invention also discloses a computer storage medium, which is applied to a system for real-time analysis of conversation emotions based on speech and physiological characteristics as described in any of the above items; a computer program is stored on the storage medium, and the computer program can be called and executed by a processor.

[0052] The present invention also discloses a computer program product, which is applied to any of the aforementioned systems for real-time analysis of conversation emotions based on speech and physiological features; the computer program product includes a computer program that can be called and executed by a processor.

[0053] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0054] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, an information push server, or a network device, etc.) to execute the methods described in the various implementation methods of the present application or certain parts of the implementation methods.

[0055] The device implementation described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present implementation scheme. Those of ordinary skill in the art may understand and implement it without creative work.

[0056] The present application may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0057] It should be understood that although this specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each implementation mode may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

[0058] The series of detailed descriptions listed above are only specific descriptions of feasible implementation methods of the present invention. They are not intended to limit the scope of protection of the present invention. Any equivalent implementation methods or changes that do not deviate from the technical spirit of the present invention should be included in the scope of protection of the present invention.

Claims

1. A system for real-time analysis of conversation emotions based on speech and physiological features, characterized by: The system comprises an audio acquisition module, a physiological characteristic acquisition module, and an emotion analysis module, wherein the emotion analysis module is electrically connected to the audio acquisition module and the physiological characteristic acquisition module respectively; The audio collection module is used to collect speech audio data during the user's conversation and extract speech features from it; The physiological feature acquisition module is used to collect physiological images of the user during the conversation process through an optical detector, and extract physiological features therefrom, wherein the physiological features include facial feature data and body movement feature data; The emotion analysis module is used to perform time alignment on the speech features and the physiological features to obtain emotion feature data, classify and identify the user's real-time emotions based on the emotion feature data, and output the identified emotion type and level.

2. The system for real-time analysis of conversation emotions based on speech and physiological features according to claim 1, characterized in that: The audio collection module is used to collect speech audio data during the user conversation process and extract speech features therefrom, including: The audio collection module collects first speech audio data during the user conversation and extracts a first speech feature therefrom; Inputting the first speech feature into an emotion fluctuation model to obtain an emotion fluctuation evaluation value; if the emotion fluctuation evaluation value is higher than a fluctuation threshold, retaining the first speech feature, and continuing to extract a second speech feature from the second speech audio data during the user conversation; The first speech feature and the second speech feature are used as the speech features.

3. The system for real-time analysis of conversation emotions based on speech and physiological features according to claim 1, characterized in that: The physiological feature acquisition module is used to acquire physiological images of the user during the conversation process through an optical detector and extract physiological features therefrom, including: The physiological feature acquisition module acquires physiological images of the user during the conversation process through an optical detector, and recognizes and obtains facial images, body images and limb images from the physiological images; Extracting dynamic facial features from the facial image to obtain facial feature data; The body dynamic features and the limb dynamic features are extracted from the body image and the limb image respectively, and the two are fused to obtain the limb movement feature data.

4. The system for real-time analysis of conversation emotions based on speech and physiological features according to claim 2, characterized in that: The emotion analysis module is used to perform time registration on the speech feature and the physiological feature, including: Aligning and registering the speech feature and the physiological feature according to the acquisition time, and then using a preset time frame to respectively extract a target speech feature and a target physiological feature from the speech feature and the physiological feature; The time frame is determined based on the first speech feature.

5. The system for real-time analysis of conversation emotions based on speech and physiological features according to claim 4, characterized in that: The time frame is determined as follows: Acquire historical speech features and historical physiological features involving multiple users, calculate time deviation values ​​of each of the historical physiological features and the historical speech features, and fit all the time deviation values ​​to obtain time deviation equivalent values; Determine the initial duration of the time frame according to the time deviation equivalent value; Inputting the first speech feature into an emotion recognition model, and the emotion recognition model predicting a number of preliminary emotion types and corresponding matching values ​​based on the first speech feature; If any of the matching values ​​is higher than the matching threshold, the first coefficient is multiplied by the initial duration to obtain a first target duration; if all the matching values ​​are lower than the matching threshold, the average of the matching values ​​is calculated, a second coefficient is determined based on the average, and the second coefficient is multiplied by the initial duration to obtain a second target duration; wherein the first coefficient is less than the second coefficient; The first target duration or the second target duration is determined as the duration of the time frame.

6. The system for real-time analysis of conversation emotions based on speech and physiological features according to claim 5, characterized in that: The method of using a preset time frame to respectively extract a target speech feature and a target physiological feature from the speech feature and the physiological feature includes: The target physiological feature is extracted from the physiological feature using a time frame corresponding to the initial duration, and the target speech feature is extracted from the speech feature using a time frame corresponding to the first target duration or the second target duration.

7. The system for real-time analysis of conversation emotions based on speech and physiological characteristics according to claim 5, characterized in that: The emotion analysis module classifies and identifies the user's real-time emotions according to the emotion feature data, and outputs the identified emotion type and level, including: The emotion feature data is input into the emotion recognition model, and the emotion recognition model outputs the emotion type and level of the user, that is, completes the classification and recognition of the real-time emotion of the user; Among them, the emotion recognition model is implemented based on Transformer.

8. An electronic device, characterized in that: A system for real-time analysis of conversation emotions based on speech and physiological characteristics as described in any one of claims 1-7; the electronic device comprises: a memory storing executable program code; a processor coupled to the memory; the processor can call the executable program code stored in the memory.

9. A computer storage medium, characterized in that: A system for real-time analysis of conversation emotions based on speech and physiological characteristics as described in any one of claims 1-7; a computer program is stored on the storage medium, and the computer program can be called and executed by a processor.

10. A computer program product, characterized in that: A system for real-time analysis of conversation emotions based on speech and physiological features as described in any one of claims 1-7; the computer program product includes a computer program that can be called and executed by a processor.

Images