Human intelligence-based group real-time state evaluation method and apparatus, and electronic device

Abstract

The embodiment of the invention provides a group real-time state evaluation method and device based on human intelligence and electronic equipment, and the method comprises the steps: obtaining voice signals of at least two users through the electronic equipment, obtaining the human body related data of each user, and carrying out the real-time state evaluation of the group based on the human intelligence; the speech signals of the at least two users can be separated to obtain the speech signal of each user, and then the state of each user is evaluated according to the human body related data of each user and the speech signal of each user, so that the speech signals of the at least two users can be evaluated in a multi-user speech environment. According to the technical scheme, the real-time state of each user is evaluated according to the voice signal and the human body related data of each user, and when the identity of the user is recognized, recognition is performed by combining the voice signal and the human body related data of the user, so that the accuracy of identity recognition in a multi-user speaking environment can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the fields of electronic digital data processing, bioinformatics technology, human factors intelligence, and human factors and ergonomics technology, and in particular to a method, device, and electronic device for real-time group status assessment based on human factors intelligence. Background Art

[0002] Voiceprint recognition technology uses sound input to extract individual voice features and confirm the identities of different speakers. As the name suggests, multi-person voiceprint recognition refers to technology that can simultaneously identify the voice features of multiple speakers and distinguish between different speakers.

[0003] In an environment where multiple people are talking, background noise and / or echo may affect the accuracy of voiceprint recognition. Therefore, it is necessary to provide a solution to improve the accuracy of identity recognition in an environment where multiple people are talking. Summary of the invention

[0004] The embodiments of the present application provide a method, device and electronic device for real-time status assessment of a group based on human factor intelligence. The embodiments of the present application also provide a computer-readable storage medium to enable the status of each user to be assessed based on the voice signal and body-related data of each user in a multi-user speaking environment, and to improve the accuracy of identity recognition in a multi-user speaking environment.

[0005] In a first aspect, an embodiment of the present application provides a method for real-time status assessment of a group based on human factors intelligence, which obtains voice signals of at least two users and human body-related data of each user; wherein the voice signals of the at least two users are collected by a sound collection device, and the human body-related data of each user is collected by a human body-related data collection device worn by each user; the voice signals of the at least two users are separated to obtain the voice signal of each user; and the status of each user is assessed based on the human body-related data of each user and the voice signal of each user.

[0006] In the above-mentioned real-time group status assessment method based on human factors intelligence, after the electronic device obtains the voice signals of at least two users and the human body-related data of each user, the voice signals of the above-mentioned at least two users can be separated to obtain the voice signal of each user, and then the status of each user is evaluated based on the human body-related data of each user and the voice signal of each user. Therefore, in a multi-user speaking environment, the real-time status of each user can be evaluated based on the voice signal and human body-related data of each user, and when identifying the identity of the user, the user's voice signal and human body-related data are combined for identification, thereby improving the accuracy of identity recognition in a multi-user speaking environment.

[0007] In one possible implementation, evaluating the status of each user based on the body-related data of each user and the voice signal of each user includes: authenticating the identity of each user based on the body-related data of each user and the voice signal of each user, and evaluating the emotional state and stress fatigue level of each user.

[0008] In one possible implementation, authenticating each user based on the body-related data of each user and the voice signal of each user includes: obtaining a first feature related to the identity based on the body-related data of each user; and obtaining a second feature related to the voiceprint based on the voice signal of each user; and authenticating the user based on the first feature and the second feature.

[0009] In one possible implementation, authenticating the user according to the first feature and the second feature includes: fusing the first feature with the second feature to obtain a third feature; and authenticating each user according to the third feature.

[0010] In one possible implementation, authenticating the user according to the first feature and the second feature includes: identifying the identity of each user according to the first feature to obtain a first identity identification result; and identifying the identity of each user according to the second feature to obtain a second identity identification result; and obtaining an identity authentication result for each user according to the first identity identification result and the second identity identification result.

[0011] In one possible implementation, evaluating the emotional state of each user based on the body-related data of each user and the voice signal of each user includes: obtaining a fourth feature related to emotion based on the body-related data of each user; and obtaining a fifth feature related to emotion based on the voice signal of each user; and evaluating the emotional state of each user based on the fourth feature and the fifth feature.

[0012] In one possible implementation, after evaluating the emotional state of each user according to the fourth feature and the fifth feature, it also includes: performing group emotion analysis according to the evaluation results of the emotional state of each user to obtain the evaluation results of the emotional states of the at least two users.

[0013] In one possible implementation, evaluating the stress fatigue level of each user based on the body-related data of each user and the voice signal of each user includes: obtaining a sixth feature related to stress fatigue based on the body-related data of each user; and obtaining a seventh feature related to stress fatigue based on the voice signal of each user; and evaluating the stress fatigue level of each user based on the sixth feature and the seventh feature.

[0014] In one possible implementation, after evaluating the stress fatigue level of each user according to the sixth feature and the seventh feature, it also includes: performing group fatigue stress analysis based on the evaluation results of the stress fatigue level of each user to obtain the evaluation results of the stress fatigue levels of the at least two users.

[0015] In one possible implementation, the human body-related data includes one or a combination of the following: skin electrodermal temperature data, pulse data, blood pressure data, blood oxygen data, electrocardiogram data, electromyography data, muscle oxygen data, respiratory data, biomechanical data, near-infrared brain imaging data, electroencephalogram data, transcranial stimulation data, heart rate variability data, heart rate data, and eye movement data.

[0016] In a second aspect, an embodiment of the present application provides an electronic device, comprising: one or more processors; a memory; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to perform the method provided in the first aspect.

[0017] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a computer program, which, when executed by a computer, is used to execute the method provided in the first aspect.

[0019] In one possible design, the program in the fourth aspect may be stored in whole or in part on a storage medium packaged together with the processor, or may be stored in whole or in part on a memory not packaged together with the processor.

[0020] In a fifth aspect, an embodiment of the present application provides a multi-user status evaluation device, which can be set in an electronic device, and the multi-user status evaluation device includes: an acquisition module, used to acquire voice signals of at least two users, and acquire human body-related data of each user; wherein the voice signals of the at least two users are acquired by a sound acquisition device, and the human body-related data of each user is acquired by a human body-related data acquisition device worn by each user; a separation module, used to separate the voice signals of the at least two users to obtain the voice signal of each user; and an evaluation module, used to evaluate the status of each user based on the human body-related data of each user and the voice signal of each user.

[0021] In one possible implementation, the evaluation module is specifically used to authenticate each user based on the body-related data of each user and the voice signal of each user, and to evaluate the emotional state and stress fatigue level of each user. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application;

[0023] Figure 2 A flowchart of a group real-time status assessment method based on human factor intelligence provided by an embodiment of the present application;

[0024] Figure 3 A flowchart of a group real-time status assessment method based on human intelligence provided by another embodiment of the present application;

[0025] Figure 4 A flowchart of a group real-time status assessment method based on human intelligence provided in yet another embodiment of the present application;

[0026] Figure 5 A flowchart of a group real-time status assessment method based on human intelligence provided in yet another embodiment of the present application;

[0027] Figure 6 A flowchart of a group real-time status assessment method based on human intelligence provided in yet another embodiment of the present application;

[0028] Figure 7 A schematic structural diagram of an electronic device provided in another embodiment of the present application. DETAILED DESCRIPTION

[0029] The terms used in the implementation section of this application are only used to explain the specific embodiments of this application and are not intended to limit this application.

[0030] The group real-time status assessment method based on human intelligence provided in the embodiment of the present application can be applied to electronic devices, wherein the electronic devices can be personal computers (personal computers, PCs), smart phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) devices, laptop computers or ultra-mobile personal computers (ultra-mobile personal computers, UMPCs) and other devices; or, the electronic devices can also be servers. The embodiment of the present application does not impose any restrictions on the specific type of electronic devices.

[0031] For example, Figure 1 A schematic diagram of the structure of an electronic device provided by an embodiment of the present application is shown as follows: Figure 1 As shown, the electronic device 100 may include one or more processors 110; a memory 130; multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory 130, and the one or more computer programs include instructions. When the above instructions are executed by the electronic device 100, the electronic device 100 can execute the real-time group status assessment method based on human intelligence provided in an embodiment of the present application.

[0032] In addition, the electronic device 100 may further include a communication interface 120. The processor 110, the communication interface 120 and the memory 130 may communicate with each other through an internal connection path to transfer control and / or data signals.

[0033] The processor 110 and the memory 130 may be combined into a processing device, or more commonly, they are independent components, and the processor 110 is used to execute program codes stored in the memory 130. In specific implementation, the memory 130 may also be integrated into the processor 110, or may be independent of the processor 110.

[0034] In addition, in order to make the functions of the electronic device 100 more complete, the electronic device 100 may further include one or more of an input unit 160 and a display unit 170 .

[0035] Optionally, the electronic device 100 may further include a power supply 150 for providing power to various devices or circuits in the electronic device 100 .

[0036] It should be understood that Figure 1The processor 110 in the electronic device 100 shown may be a system on chip SOC, and the processor 110 may include a central processing unit (CPU), and may further include other types of processors, such as a graphics processing unit (GPU) and the like.

[0037] The following is a description of the real-time group status assessment method based on human intelligence provided in an embodiment of the present application.

[0038] Figure 2 A flowchart of a group real-time status assessment method based on human intelligence is provided in one embodiment of the present application. The group real-time status assessment method based on human intelligence can be executed by the electronic device 100.

[0039] like Figure 2 As shown, the above-mentioned group real-time status assessment method based on human intelligence may include:

[0040] Step 201: Acquire speech signals of at least two users, and acquire body-related data of each user.

[0041] Among them, the voice signals of the at least two users are collected by a sound collection device, and the body-related data of each user is collected by a body-related data collection device worn by each user.

[0042] Specifically, the sound collection device may be a device that can realize the sound collection function, such as a microphone. In specific implementation, one or more microphones may be used to collect the voice signals of the at least two users, and then the electronic device 100 obtains the voice signals collected by the one or more microphones.

[0043] The above-mentioned human body-related data may include one or a combination of the following: skin electrodermal temperature data, pulse data, blood pressure data, blood oxygen data, electrocardiogram data, electromyography data, muscle oxygen data, respiratory data, biomechanical data, near-infrared brain imaging data, electroencephalogram data, transcranial stimulation data, heart rate variability data, heart rate data, and eye movement data.

[0044] Step 202: Separate the speech signals of the at least two users to obtain a speech signal of each user.

[0045] Specifically, the voice signals of the at least two users may be separated by using one or a combination of the following methods:

[0046] 1) Blind source separation (BSS): Use algorithms to separate overlapping voice signals of multiple users. Common algorithms include independent component analysis (ICA) and non-negative matrix factorization (NMF). Blind source separation technology can extract each individual's voiceprint signal from a mixed signal without knowing the location or content of the sound source.

[0047] 2) Adaptive beam forming: By using a microphone array and sound source localization technology to point the beam, the sound signal in a specific direction is enhanced and interference from other directions is suppressed. This method is often used in multi-person conference scenarios and can improve the recognition accuracy of specific target voiceprints.

[0048] 3) Deep learning model: Use a deep neural network to separate the voice signals of at least two users. The deep learning model can learn a more accurate separation model through training data in a complex acoustic environment.

[0049] Step 203: Evaluate the status of each user according to the body-related data of each user and the voice signal of each user.

[0050] Specifically, based on the human body data of each user and the voice signal of each user, evaluating the state of each user may include: authenticating each user based on the human body data of each user and the voice signal of each user, and evaluating the emotional state and stress fatigue level of each user. In addition, it should be noted that, in this embodiment, evaluating the state of each user may be evaluating the real-time state of each user.

[0051] When authenticating each user, it is necessary to ensure the reliability of multi-user authentication. The following three aspects need to be considered:

[0052] 1) Voice signal quality control: In a scenario where multiple users are speaking, background noise or echo will affect the accuracy of recognition. Therefore, noise reduction algorithms and echo cancellation techniques can be used to improve the quality of voice signals.

[0053] 2) Multi-channel data fusion: By using multiple microphones to obtain sound signals from different angles, data fusion is performed to enhance the robustness of the voiceprint recognition system. This can effectively improve the accuracy of voiceprint recognition when audio data is missing or the signal is unclear.

[0054] 3) Continuous recognition based on conversation features: Scenarios where multiple users are talking are often accompanied by continuous conversations. By extracting and analyzing the frequency, speaking speed, volume and other features of each user in the conversation, each user is dynamically authenticated to ensure the stability of recognition.

[0055] In addition, the following three aspects are also included when evaluating each user's emotional state and stress fatigue level:

[0056] 1) Speech emotion recognition: By analyzing the voice characteristics such as pitch, speaking speed, energy, etc., the user's emotional state, such as anger, happiness, sadness and / or fatigue, can be identified. Using emotion recognition algorithms, the emotions of each person in the group can be evaluated, thereby analyzing group dynamics.

[0057] 2) Stress and fatigue detection: Combine changes in voice features, such as frequency band changes and / or pitch jitter of the audio signal, to infer the user's stress or fatigue level. Stress and fatigue detection is very important in scenarios such as safety monitoring or driver monitoring.

[0058] 3) Multi-person voice interaction analysis: Evaluate the group interaction status through indicators such as the alternation frequency of voice signals, overlapping voices and / or response time to help identify emotional states such as tension, conflict or collaboration that may exist in the group.

[0059] In the above-mentioned method for real-time status assessment of a group based on human factors intelligence, after the electronic device 100 obtains the voice signals of at least two users and the human body-related data of each user, the voice signals of the above-mentioned at least two users can be separated to obtain the voice signal of each user, and then the status of each user can be assessed based on the human body-related data of each user and the voice signal of each user. In this way, in a multi-user speaking environment, the real-time status of each user can be assessed based on the voice signal and human body-related data of each user, and when identifying the identity of the user, the user's voice signal and human body-related data are combined for identification, thereby improving the accuracy of identity recognition in a multi-user speaking environment.

[0060] Figure 3 A flowchart of a group real-time status assessment method based on human factor intelligence is provided in another embodiment of the present application. This embodiment is also executed by the electronic device 100. This embodiment is for the present application Figure 2 In the group real-time status assessment method based on human factor intelligence provided in the illustrated embodiment, a method for authenticating the identity of each user is introduced.

[0061] like Figure 3 As shown, according to the human body related data of each user and the voice signal of each user, identity authentication of each user may include:

[0062] Step 301, obtaining a first feature related to identity based on the body-related data of each user; and obtaining a second feature related to voiceprint based on the voice signal of each user.

[0063] Step 302: Authenticate the user based on the first feature and the second feature.

[0064] In one implementation, authenticating a user based on the first feature and the second feature may be performed by: fusing the first feature with the second feature to obtain a third feature; and then authenticating each user based on the third feature.

[0065] In an example of this implementation, the third feature obtained by fusion can be matched with the saved identity feature, and if there is an identity feature matching the third feature among the saved identity features, the user's identity authentication result can be obtained according to the identity feature matching the third feature. The above identity feature can be a feature obtained by fusion of the user's body-related features and the user's voice features.

[0066] That is to say, the user's body-related data and voice signal can be obtained in advance, the body-related features can be obtained according to the user's body-related data, the user's voice features can be obtained according to the user's voice signal, and then the user's body-related features and voice features can be merged to obtain the user's identity features and save them. Later, when the user needs to be authenticated, the third feature can be matched with the saved identity feature to obtain the user's identity authentication result.

[0067] Of course, it is understandable that the feature fusion method used to obtain the identity feature in advance is the same as the feature fusion method used to obtain the third feature; and the pre-acquired human body-related data is also the same as the human body-related data used to obtain the first feature. For example, the pre-acquired user's human body-related data includes: skin galvanic skin temperature data, eye movement data and EEG data. Subsequently, in a multi-user speaking environment, the human body-related data collection device collects human body-related data for each user, including: skin galvanic skin temperature data, eye movement data and EEG data.

[0068] In another example of this implementation, the third feature obtained by fusion can be input into a pre-trained machine learning model to perform feature matching and identity authentication to obtain the user's identity authentication result.

[0069] In another implementation, the user identity authentication according to the first feature and the second feature may be: according to the first feature, the identity of each user is identified to obtain a first identity identification result; and according to the second feature, the identity of each user is identified to obtain a second identity identification result. Then, according to the first identity identification result and the second identity identification result, the identity authentication result of each user is obtained.

[0070] In this implementation, the identity of each user is identified based on the first feature, which is a technology that uses the physiological characteristics or activity patterns of the human body to perform identity authentication. Specifically, after the human body-related data acquisition device collects the human body-related data of each user, the collected human body-related data can be filtered, denoised and / or normalized to remove noise and interference and improve data quality. Then, feature parameters that can characterize individual differences are extracted from the processed data, such as waveform features, spectrum features and / or time domain features, etc. The above-extracted features are the first features.

[0071] Next, the first feature is input into a pre-trained machine learning model to perform feature matching and identity authentication to obtain a first identity recognition result.

[0072] In addition, the identity of each user is identified according to the second feature, that is, voiceprint recognition is performed according to the second feature. Specifically, after obtaining the voice signal of each user, the voice signal of each user can be firstly subjected to voice processing and preprocessing, for example: performing processing steps such as denoising, audio segmentation and / or feature standardization on the above-mentioned voice signal to improve the quality of the voice signal. Then, the sound feature of the processed signal is extracted to obtain the acoustic feature of voiceprint recognition (i.e., the second feature), which can be mel frequency cepstral coefficients (MFCC), linear prediction cepstral coefficients (LPCC) or spectral features, etc. Finally, the second feature is identified using a machine learning model to obtain a second identity recognition result.

[0073] In this implementation, the identity authentication result of each user can be obtained based on the first identity recognition result and the second identity recognition result: weighted sum of the first identity recognition result and the second identity recognition result, and the identity authentication result of each user can be obtained based on the result of the weighted sum. In specific implementation, the weight of the first identity recognition result and the weight of the second identity recognition result can be set according to system performance and / or implementation requirements. This embodiment does not limit the size of the weight of the first identity recognition result and the weight of the second identity recognition result. For example, if the first identity recognition result is more important, the weight of the first identity recognition result can be set to be greater than the weight of the second identity recognition result; if the second identity recognition result is more important, the weight of the second identity recognition result can be set to be greater than the weight of the first identity recognition result.

[0074] The machine learning model mentioned in the above description of this embodiment can be one of the following models or a combination: classification model, support vector machine (SVM), hidden Markov model (HMM), convolutional neural network (CNN), recurrent neural network (RNN), deep belief network (DBN); of course, the machine learning model can also be other models, which is not limited in this embodiment.

[0075] Figure 4 A flowchart of a group real-time status assessment method based on human factor intelligence is provided in another embodiment of the present application. This embodiment is also executed by the electronic device 100. This embodiment is for the present application Figure 2 In the real-time group status assessment method based on human factor intelligence provided in the illustrated embodiment, a method for assessing the emotional state of each user is introduced.

[0076] like Figure 4 As shown, based on the body-related data of each user and the voice signal of each user, evaluating the emotional state of each user may include:

[0077] Step 401, obtaining a fourth feature related to emotion based on the body-related data of each user; and obtaining a fifth feature related to emotion based on the voice signal of each user.

[0078] As described above, human body related data may include skin electrical skin temperature data, EEG data, heart rate variability data, eye movement data and / or electromyography data, etc. Taking heart rate variability data as an example, heart rate variability can distinguish the activities of sympathetic and parasympathetic nerves, thereby inferring the user's emotional state. For example, high heart rate variability is usually associated with relaxation and positive emotions, while low heart rate variability is associated with stress and anxiety. Therefore, after obtaining the human body related data of each user, the human body related data may be first filtered, denoised and / or normalized to remove noise and interference and improve data quality, and then feature parameters that can characterize the emotional state are extracted from the processed data to obtain the fourth feature. For example, the fourth feature may be a frequency feature, a time domain feature and / or a nonlinear feature.

[0079] A speech signal refers to the sound waveform produced by human pronunciation, which contains rich information, such as pitch, sound intensity, speaking speed, etc. By analyzing and comparing the user's speech features, the user's current emotional state can be inferred. Specifically, after obtaining each user's speech signal, the speech signal can be filtered, denoised and / or normalized to remove noise and interference and improve the quality of the speech signal. Then, feature parameters that can characterize the emotional state are extracted from the processed speech signal to obtain the fifth feature, which may include acoustic features, language features and / or non-language features.

[0080] Step 402: Evaluate the emotional state of each user based on the fourth and fifth characteristics.

[0081] Specifically, in one implementation, the emotional state of each user is evaluated based on the fourth and fifth features as follows: the fourth feature is merged with the fifth feature to obtain an eighth feature; and then, the emotional state of each user is evaluated based on the eighth feature.

[0082] In one example, the eighth feature obtained by fusion can be input into a pre-trained machine learning model for feature matching and emotional state inference to obtain the emotional state of each user.

[0083] In another implementation, according to the fourth and fifth features, evaluating the emotional state of each user may be: according to the fourth feature, evaluating the emotional state of each user to obtain a first emotion evaluation result; and according to the fifth feature, evaluating the emotional state of each user to obtain a second emotion evaluation result. Then, according to the first emotion evaluation result and the second emotion evaluation result, an emotion evaluation result of each user is obtained.

[0084] In this implementation, the emotional state of the user is evaluated according to the fourth feature and the fifth feature respectively, and then the emotional evaluation result of each user is obtained according to the emotional evaluation results respectively obtained. Specifically, according to the first emotional evaluation result and the second emotional evaluation result, the emotional evaluation result of each user can be obtained by: weighted summing the first emotional evaluation result and the second emotional evaluation result, and obtaining the emotional evaluation result of each user according to the result of the weighted summation. In specific implementation, the weight of the first emotional evaluation result and the weight of the second emotional evaluation result can be set according to system performance and / or implementation requirements. This embodiment does not limit the size of the weight of the first emotional evaluation result and the weight of the second emotional evaluation result. For example, if the first emotional evaluation result is more valued, the weight of the first emotional evaluation result can be set to be greater than the weight of the second emotional evaluation result; if the second emotional evaluation result is more valued, the weight of the second emotional evaluation result can be set to be greater than the weight of the first emotional evaluation result.

[0085] In this embodiment, after obtaining the evaluation result of the emotional state of each user, group emotion analysis can be performed based on the evaluation result of the emotional state of each user to obtain the evaluation results of the emotional state of at least two users. In other words, the evaluation result of the real-time emotional state of the current group can be given based on the evaluation result of the emotional state of each user.

[0086] Figure 5 A flowchart of a group real-time status assessment method based on human factor intelligence is provided in another embodiment of the present application. This embodiment is also executed by the electronic device 100. This embodiment is for the present application Figure 2 In the group real-time status assessment method based on human factor intelligence provided in the illustrated embodiment, a method for assessing the stress fatigue level of each user is introduced.

[0087] like Figure 5 As shown, based on the human body related data of each user and the voice signal of each user, evaluating the stress fatigue level of each user may include:

[0088] Step 501, obtaining a sixth feature related to stress fatigue based on the body-related data of each user; and obtaining a seventh feature related to stress fatigue based on the voice signal of each user.

[0089] As described above, human body related data may include skin electrical skin temperature data, EEG data, heart rate variability data, eye movement data and / or electromyography data, etc. Taking heart rate variability data as an example, heart rate variability measures the change in heartbeat intervals, reflecting the activity level of the autonomic nervous system. Heart rate variability is one of the important indicators for assessing stress levels. For example, high heart rate variability is usually associated with a relaxed state, indicating that the parasympathetic nervous system is more active; low heart rate variability is associated with a stress state, indicating that the sympathetic nervous system is more active. Therefore, after obtaining the human body related data of each user, the human body related data may be first filtered, denoised and / or normalized to remove noise and interference and improve data quality, and then feature parameters that can characterize the stress state are extracted from the processed data to obtain the sixth feature. For example, the sixth feature may include the time domain and frequency domain features of heart rate variability, the peak and mean of skin electrical skin temperature data, the frequency band power of respiratory frequency and / or EEG data, etc.

[0090] A speech signal refers to the sound waveform produced by human pronunciation, which contains rich information, such as pitch, sound intensity, speaking speed, etc. By analyzing and comparing the user's speech features, the user's current stress fatigue level can be inferred. Specifically, after obtaining each user's speech signal, the speech signal can be filtered, denoised and / or normalized to remove noise and interference and improve the quality of the speech signal. Then, feature parameters that can characterize the stress state are extracted from the processed speech signal to obtain the seventh feature, which may include acoustic features, language features and / or non-language features.

[0091] Step 502: Evaluate the stress fatigue level of each user based on the sixth feature and the seventh feature.

[0092] Specifically, in one implementation, based on the sixth and seventh features, the stress fatigue level of each user is evaluated by: fusing the sixth feature with the seventh feature to obtain a ninth feature; and then, based on the ninth feature, evaluating the stress fatigue level of each user.

[0093] In one example, the ninth feature obtained by fusion can be input into a pre-trained machine learning model for feature matching and fatigue stress level inference to obtain the stress fatigue level of each user.

[0094] In another implementation, according to the sixth and seventh features, evaluating the stress fatigue level of each user may be: according to the sixth feature, evaluating the stress fatigue level of each user to obtain a first stress fatigue level evaluation result; and according to the seventh feature, evaluating the stress fatigue level of each user to obtain a second stress fatigue level evaluation result. Then, according to the first stress fatigue level evaluation result and the second stress fatigue level evaluation result, the stress fatigue level evaluation result of each user is obtained.

[0095] In this implementation, the stress fatigue level of the user is evaluated according to the sixth feature and the seventh feature respectively, and then the stress fatigue level evaluation result of each user is obtained according to the stress fatigue level evaluation results obtained respectively. Specifically, according to the first stress fatigue level evaluation result and the second stress fatigue level evaluation result, the stress fatigue level evaluation result of each user can be obtained by weighted summing the first stress fatigue level evaluation result and the second stress fatigue level evaluation result, and according to the result of the weighted summing, the stress fatigue level evaluation result of each user is obtained. In specific implementation, the weight of the first stress fatigue level evaluation result and the weight of the second stress fatigue level evaluation result can be set according to system performance and / or implementation requirements. This embodiment does not limit the weight of the first stress fatigue level evaluation result and the weight of the second stress fatigue level evaluation result. For example, if the first stress fatigue level evaluation result is more important, the weight of the first stress fatigue level evaluation result can be set to be greater than the weight of the second stress fatigue level evaluation result; if the second stress fatigue level evaluation result is more important, the weight of the second stress fatigue level evaluation result can be set to be greater than the weight of the first stress fatigue level evaluation result.

[0096] In this embodiment, after obtaining the evaluation result of the stress fatigue level of each user, a group fatigue stress analysis can be performed based on the evaluation result of the stress fatigue level of each user to obtain the evaluation results of the stress fatigue levels of at least two users. In other words, the evaluation result of the real-time stress fatigue level of the current group can be given based on the evaluation result of the stress fatigue level of each user.

[0097] As described above, the embodiment of the present application provides a real-time group status assessment method based on human factor intelligence, which can authenticate the user's identity and assess the user's emotional state and stress fatigue level based on voice signals and human body related data.

[0098] Figure 6 A flowchart of a group real-time status assessment method based on human intelligence is provided in another embodiment of the present application, such as Figure 6 As shown, it may include:

[0099] Step 601: Acquire speech signals of at least two users and body-related data of each user.

[0100] Step 602: Separate the speech signals of at least two users to obtain a speech signal of each user.

[0101] Step 603 , performing identity authentication, emotional state assessment, and fatigue stress level assessment on each user based on the body-related data of each user and the voice signal of each user.

[0102] Step 604: Perform group emotion analysis based on the evaluation result of the emotional state of each user to obtain the evaluation result of the emotional state of the current group.

[0103] Step 605: Perform group fatigue stress analysis based on the evaluation result of the fatigue stress level of each user to obtain the evaluation result of the fatigue stress level of the current group.

[0104] Among them, step 604 and step 605 can be executed successively or in parallel. This embodiment does not limit the execution order of step 604 and step 605.

[0105] It is to be understood that some or all of the steps or operations in the above embodiments are merely examples, and the present application embodiments may also perform other operations or variations of various operations. In addition, the various steps may be performed in different orders presented in the above embodiments, and it is possible that not all of the operations in the above embodiments need to be performed.

[0106] It is understandable that, in order to realize the above functions, the electronic device includes hardware and / or software modules corresponding to the execution of each function. In combination with the algorithm steps of each example described in the embodiments disclosed in this application, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments, but such implementation should not be considered to exceed the scope of this application.

[0107] This embodiment can divide the electronic device into functional modules according to the above method embodiment. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one module. The above integrated module can be implemented in the form of hardware. It should be noted that the division of modules in this embodiment is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0108] Figure 7A schematic diagram of the structure of an electronic device provided in another embodiment of the present application, in which each functional module is divided according to each function, Figure 7 A possible schematic diagram of the composition of the electronic device 700 involved in the above embodiment is shown. The electronic device 700 may include a multi-user status evaluation device.

[0109] like Figure 7 As shown, the electronic device 700 may include: an acquisition module 701, a separation module 702 and an evaluation module 703;

[0110] The acquisition module 701 is used to acquire voice signals of at least two users and to acquire body-related data of each user; wherein the voice signals of the at least two users are acquired by a sound acquisition device, and the body-related data of each user is acquired by a body-related data acquisition device worn by each user;

[0111] A separation module 702, configured to separate the voice signals of the at least two users to obtain a voice signal of each user;

[0112] The evaluation module 703 is used to evaluate the status of each user according to the body-related data of each user and the voice signal of each user.

[0113] In this embodiment, the evaluation module 703 is specifically used to authenticate each user according to the body-related data of each user and the voice signal of each user, and to evaluate the emotional state and stress fatigue level of each user.

[0114] In one implementation, the evaluation module 703 is specifically used to obtain a first feature related to the identity based on the body-related data of each user; and to obtain a second feature related to the voiceprint based on the voice signal of each user; and to authenticate the user based on the first feature and the second feature.

[0115] In some examples, the evaluation module 703 is specifically used to merge the first feature with the second feature to obtain a third feature; and authenticate each user based on the third feature.

[0116] In other examples, the evaluation module 703 is specifically used to identify the identity of each user based on a first feature to obtain a first identity identification result; and to identify the identity of each user based on a second feature to obtain a second identity identification result; and to obtain an identity authentication result for each user based on the first identity identification result and the second identity identification result.

[0117] In another implementation, the evaluation module 703 is specifically used to obtain a fourth feature related to emotion based on the body-related data of each user; and to obtain a fifth feature related to emotion based on the voice signal of each user; and to evaluate the emotional state of each user based on the fourth feature and the fifth feature.

[0118] The evaluation module 703 is further configured to, after evaluating the emotional state of each user, perform group emotion analysis according to the evaluation result of the emotional state of each user, and obtain the evaluation results of the emotional states of at least two users.

[0119] In another implementation, the evaluation module 703 is specifically used to obtain a sixth feature related to stress fatigue based on the body-related data of each user; and to obtain a seventh feature related to stress fatigue based on the voice signal of each user; and to evaluate the stress fatigue level of each user based on the sixth feature and the seventh feature.

[0120] The evaluation module 703 is further used to, after evaluating the stress fatigue level of each user, perform group fatigue stress analysis according to the evaluation result of the stress fatigue level of each user, so as to obtain the evaluation results of the stress fatigue levels of at least two users.

[0121] All relevant contents of each step involved in the embodiment of the method of the present application can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0122] The electronic device 700 provided in this embodiment is used to execute the present application Figure 2 to Figure 5 The human intelligence-based group real-time status assessment method provided by the illustrated embodiment can therefore achieve the same effect as the above-mentioned method.

[0123] It should be understood that the electronic device 700 may correspond to Figure 1 The electronic device 100 shown in FIG. The functions of the acquisition module 701, the separation module 702 and the evaluation module 703 can be Figure 1 The processor 110 in the electronic device 100 is shown implemented.

[0124] In the case of adopting an integrated module, the electronic device 700 may include a processing module, a storage module, and a communication module.

[0125] The processing module can be used to control and manage the actions of the electronic device 700, for example, it can be used to support the electronic device 700 to execute the steps executed by the acquisition module 701, the separation module 702 and the evaluation module 703. The storage module can be used to support the electronic device 700 to store program codes and data, etc. The communication module can be used to support the communication between the electronic device 700 and other devices.

[0126] Among them, the processing module can be a processor or a controller, which can implement or execute various exemplary logic boxes, modules and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, and so on. The storage module can be a memory. The communication module can specifically be a device that interacts with other electronic devices, such as a radio frequency circuit, a Bluetooth chip and / or a Wi-Fi chip.

[0127] In one embodiment, when the processing module is a processor and the storage module is a memory, the electronic device 700 involved in this embodiment can be a Figure 1 Device of the structure shown.

[0128] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the present application. Figure 2 to Figure 5 The method provided by the illustrated embodiment.

[0129] The present application also provides a computer program product, which includes a computer program, which, when executed on a computer, enables the computer to execute the present application. Figure 2 to Figure 5 The method provided by the illustrated embodiment.

[0130] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, c can be single or multiple.

[0131] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0133] In several embodiments provided in the present application, any function can be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as an independent product. Based on this understanding, the technical solution of the present application can be embodied in the form of a software product in essence or in other words, the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0134] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for real-time group status assessment based on human intelligence, characterized in that: include: Acquire voice signals of at least two users, and acquire body-related data of each user; wherein the voice signals of the at least two users are acquired by a sound acquisition device, and the body-related data of each user is acquired by a body-related data acquisition device worn by each user; Separating the speech signals of the at least two users to obtain a speech signal of each user; The state of each user is evaluated according to the human body related data of each user and the voice signal of each user.

2. The method according to claim 1, characterized in that The evaluating the status of each user according to the human body related data of each user and the voice signal of each user comprises: According to the human body related data of each user and the voice signal of each user, the identity of each user is authenticated, and the emotional state and stress fatigue level of each user are evaluated.

3. The method according to claim 2, characterized in that The performing identity authentication on each user according to the human body related data of each user and the voice signal of each user comprises: According to the human body related data of each user, a first feature related to the identity is obtained; and according to the voice signal of each user, a second feature related to the voiceprint is obtained; The user is authenticated according to the first feature and the second feature.

4. The method according to claim 3, characterized in that The performing identity authentication on the user according to the first feature and the second feature includes: Merging the first feature with the second feature to obtain a third feature; According to the third feature, identity authentication is performed on each user.

5. The method according to claim 3, characterized in that: The performing identity authentication on the user according to the first feature and the second feature includes: According to the first feature, the identity of each user is identified to obtain a first identity identification result; and according to the second feature, the identity of each user is identified to obtain a second identity identification result; An identity authentication result of each user is obtained according to the first identity recognition result and the second identity recognition result.

6. The method according to claim 2, characterized in that The evaluating the emotional state of each user according to the human body related data of each user and the voice signal of each user comprises: Obtaining a fourth feature related to emotion according to the human body related data of each user; and obtaining a fifth feature related to emotion according to the voice signal of each user; The emotional state of each user is evaluated based on the fourth feature and the fifth feature.

7. The method according to claim 6, characterized in that After evaluating the emotional state of each user according to the fourth feature and the fifth feature, the method further includes: According to the evaluation result of the emotional state of each user, group emotion analysis is performed to obtain the evaluation results of the emotional states of the at least two users.

8. The method according to claim 2, characterized in that: The evaluating the stress fatigue level of each user according to the human body related data of each user and the voice signal of each user comprises: According to the human body related data of each user, a sixth feature related to stress fatigue is obtained; and according to the voice signal of each user, a seventh feature related to stress fatigue is obtained; Based on the sixth and seventh features, the stress fatigue level of each user is evaluated.

9. The method according to claim 8, characterized in that After evaluating the stress fatigue level of each user according to the sixth feature and the seventh feature, the method further includes: According to the evaluation result of the stress fatigue level of each user, a group fatigue stress analysis is performed to obtain the evaluation results of the stress fatigue levels of the at least two users.

10. The method according to any one of claims 1 to 9, characterized in that: The human body related data includes one or a combination of the following: skin electrodermal temperature data, pulse data, blood pressure data, blood oxygen data, electrocardiogram data, electromyography data, muscle oxygen data, respiratory data, biomechanical data, near-infrared brain imaging data, electroencephalogram data, transcranial stimulation data, heart rate variability data, heart rate data, and eye movement data.

11. An electronic device, characterized in that: include: one or more processors; Memory; Multiple applications; and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions, which, when executed by the electronic device, enable the electronic device to perform the method as described in any one of claims 1-10.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed on a computer, enables the computer to execute the method according to any one of claims 1 to 10.

13. A multi-user status evaluation device, characterized in that: include: An acquisition module, used to acquire voice signals of at least two users, and to acquire body-related data of each user; wherein the voice signals of the at least two users are acquired by a sound acquisition device, and the body-related data of each user is acquired by a body-related data acquisition device worn by each user; A separation module, used to separate the voice signals of the at least two users to obtain a voice signal of each user; The evaluation module is used to evaluate the status of each user according to the body-related data of each user and the voice signal of each user.

14. The device according to claim 13, characterized in that The evaluation module is specifically used to authenticate each user based on the body-related data of each user and the voice signal of each user, and to evaluate the emotional state and stress fatigue level of each user.

Images