Personnel state analysis method, system and equipment based on human intelligence, and medium

Abstract

The invention provides a human factor intelligence-based personnel state analysis method, system and device, and a medium. The method comprises the steps of obtaining a sound signal acquired by a sound acquisition device; performing signal separation based on the sound signals to obtain a first type of sound signals, a second type of sound signals and a third type of sound signals; performing voiceprint recognition based on the first type of sound signals to obtain identity information of the target person; performing sentiment analysis based on the first type of sound signals and the second type of sound signals to obtain sentiment information of the target person; performing sound analysis based on the third type of sound signals to obtain environment sound information; and performing personnel state analysis based on the identity information of the target personnel, the emotion information of the target personnel and the environmental sound information. According to the method provided by the invention, personnel state analysis is performed by integrating the identity information, the emotion information and the environment sound information of the personnel, so that the personnel state information obtained through analysis is more comprehensive, and meanwhile, the accuracy of an emotion analysis result is improved.

Description

Technical Field

[0001] The present application relates to the field of sound analysis technology, and in particular to a method, system, device and medium for analyzing personnel status based on human factor intelligence. Background Art

[0002] There is a wealth of information hidden in sound. When vision, touch, smell, etc. are not enough to support people to complete their work, sound analysis and detection can play a huge role. At the same time, because sound signals are non-contact, it avoids the difficulty of collecting information due to lack of contact. Sound analysis and detection systems have received more and more attention and have played a huge role in the fields of medical care, manufacturing, transportation, security, warehousing, construction, etc. However, when analyzing the status of people through sound analysis technology, there are problems such as the status information of the people obtained by analysis is single and the accuracy of the results of people's emotional analysis is low. Summary of the invention

[0003] The present application provides a personnel status analysis method, system, device and medium based on human factors intelligence, which helps to conduct personnel status analysis by integrating personnel identity information, emotional information and ambient sound information, making the personnel status information obtained by analysis more comprehensive, while improving the accuracy of emotional analysis results.

[0004] In a first aspect, the present application provides a method for analyzing personnel status based on human factor intelligence, comprising:

[0005] Acquire a sound signal collected by a sound collection device;

[0006] Based on the sound signal, signal separation is performed to obtain a first type of sound signal, a second type of sound signal and a third type of sound signal, wherein the first type of sound signal is a speech signal of the target person, the second type of sound signal is a non-verbal sound signal emitted by the target person, and the third type of sound signal is a sound signal heard by the target person;

[0007] Perform voiceprint recognition based on the first type of sound signal to obtain the identity information of the target person;

[0008] Performing sentiment analysis based on the first type of sound signal and the second type of sound signal to obtain the target person's sentiment information;

[0009] Perform sound analysis based on the third type of sound signal to obtain environmental sound information;

[0010] Personnel status analysis is performed based on the target person’s identity information, emotional information, and ambient sound information.

[0011] In one possible implementation, voiceprint recognition is performed based on the first type of sound signal to obtain the identity information of the target person, including:

[0012] Extracting speech features based on the first type of sound signal;

[0013] Compare the speech feature with multiple trained voiceprint models to determine a target voiceprint model, where the target voiceprint model is used to represent the voiceprint model that is most similar to the speech feature among the multiple trained voiceprint models;

[0014] Determine the identity information of the target person based on the target voiceprint model.

[0015] One possible implementation method is to compare the speech features with multiple trained voiceprint models, including:

[0016] The matching algorithm is used to calculate the similarity between the speech features and multiple trained voiceprint models, and the target voiceprint model is the voiceprint model with the highest similarity.

[0017] In one possible implementation, emotion analysis is performed based on the first type of sound signal and the second type of sound signal to obtain emotion information of the target person, including:

[0018] Using speech recognition technology to convert the first type of sound signal into a text signal and extract text features;

[0019] Extracting sound features based on the second type of sound signal;

[0020] The text features and sound features are input into the trained sentiment classification model for recognition to obtain the target person’s sentiment information.

[0021] In one possible implementation, sound analysis is performed based on the third type of sound signal to obtain ambient sound information, including:

[0022] Extracting audio features based on the third type of sound signal;

[0023] Based on the audio features, the third type of sound signal is subjected to audio classification, audio segmentation and audio event detection to obtain the ambient sound information.

[0024] In one possible implementation, the personnel status analysis is performed based on the identity information, emotional information, and ambient sound information of the target person, including:

[0025] The identity information, emotional information and ambient sound information of the target person are quantified and fused to obtain fusion features;

[0026] The fused features are input into the trained personnel status analysis model to obtain the status of the target personnel.

[0027] In one possible implementation, before performing voiceprint recognition based on the first type of sound signal, the method further includes:

[0028] The first type of sound signal is subjected to noise reduction and signal compensation.

[0029] In one possible implementation, the second type of sound signal includes at least crying, laughing, coughing, groaning, and applause; and the third type of sound signal includes at least environmental sound.

[0030] In a second aspect, the present application provides a personnel status analysis system based on human factor intelligence, comprising:

[0031] An acquisition module is used to acquire the sound signal collected by the sound collection device;

[0032] A signal separation module is used to perform signal separation based on the sound signal to obtain a first type of sound signal, a second type of sound signal and a third type of sound signal, wherein the first type of sound signal is a speech signal of a target person, the second type of sound signal is a non-verbal sound signal emitted by the target person, and the third type of sound signal is a sound signal heard by the target person;

[0033] A voiceprint recognition module, used to perform voiceprint recognition based on the first type of sound signal to obtain the identity information of the target person;

[0034] The emotion analysis module is used to perform emotion analysis based on the first type of sound signal and the second type of sound signal to obtain the emotion information of the target person;

[0035] A sound analysis module, used for performing sound analysis based on the third type of sound signal to obtain ambient sound information;

[0036] The personnel status analysis module is used to analyze the personnel status based on the identity information of the target personnel, the emotional information of the target personnel, and the ambient sound information.

[0037] In a third aspect, the present application provides an electronic device comprising: a processor and a memory, the memory being used to store a computer program; the processor being used to run the computer program to implement the personnel status analysis method based on human factor intelligence as described in the first aspect.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer implements the personnel status analysis method based on human intelligence as described in the first aspect.

[0039] Compared with the prior art, this application has at least the following technical effects:

[0040] The present application provides a personnel status analysis method based on human factors intelligence, which separates the acquired sound information to obtain the voice signal of the target person, the non-verbal sound signal emitted by the target person and the sound signal heard by the target person, performs voiceprint recognition based on the voice signal of the target person to obtain the identity information of the target person, performs emotion analysis based on the voice signal of the target person and the non-verbal sound signal emitted by the target person to obtain the emotion information of the target person, performs sound analysis based on the sound signal heard by the target person to obtain ambient sound information, and performs personnel status analysis by integrating the identity information, emotion information and ambient sound information of the person, so that the personnel status information obtained by the analysis includes both the identity information of the person and the emotion information of the person, making the status information more comprehensive, and at the same time combines the voice signal and the sound signal related to the person (specifically referring to the non-verbal sound signal emitted by the target person) to perform emotion analysis, so that the result of the emotion analysis is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of a personnel status analysis method based on human factor intelligence provided in an embodiment of the present application;

[0042] Figure 2 A schematic diagram of the voiceprint recognition process provided in the embodiment of the present application;

[0043] Figure 3 A schematic diagram of the sentiment analysis process provided in the embodiment of the present application;

[0044] Figure 4 A schematic diagram of the sound analysis process provided in the embodiment of the present application;

[0045] Figure 5 A schematic diagram of the structure of a personnel status analysis system based on human factor intelligence provided in an embodiment of the present application;

[0046] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] In the embodiments of the present application, unless otherwise specified, the character " / " indicates that the objects before and after the association are in an or relationship. For example, A / B can represent A or B. "And / or" describes the association relationship of the associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone.

[0048] It should be pointed out that the words "first", "second", etc. involved in the embodiments of the present application are only used to distinguish the description purpose, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated, nor can they be understood as indicating or implying order.

[0049] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. In addition, "at least one of the following" or similar expressions refers to any combination of these items, which may include any combination of single items or plural items. For example, at least one of A, B, or C may represent: A, B, C, A and B, A and C, B and C, or A, B and C. Among them, each of A, B, and C may be an element itself, or a set containing one or more elements.

[0050] In the embodiments of the present application, "exemplary", "in some embodiments", "in another embodiment", etc. are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0051] In the embodiments of the present application, "of", "corresponding", and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings to be expressed are consistent. In the embodiments of the present application, communication and transmission can sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings to be expressed are consistent. For example, transmission can include sending and / or receiving, which can be a noun or a verb.

[0052] The equal to involved in the embodiments of the present application can be used in conjunction with greater than, and is applicable to the technical solution adopted when greater than, and can also be used in conjunction with less than, and is applicable to the technical solution adopted when less than. It should be noted that when equal to is used in conjunction with greater than, it cannot be used in conjunction with less than; when equal to is used in conjunction with less than, it cannot be used in conjunction with greater than.

[0053] In the related art, the personnel status analysis method includes:

[0054] (1) Methods based on voiceprint recognition

[0055] The specific steps include sound collection, data preprocessing, voiceprint recognition and state processing, among which sound collection is to obtain audio through sound collection devices such as microphones and input it into the analysis system; the data preprocessing stage is mainly to reduce the noise and extract features of sound signals; voiceprint recognition is mainly to determine the label of the sound file, input the preprocessed feature vector into the constructed voiceprint recognition model, and determine the label of the sound file; state processing is to determine the state based on the result, that is, to determine the state of the object based on the audio.

[0056] (2) Methods based on deep speech sentiment analysis

[0057] Specifically, it performs deep speech sentiment analysis on the input speech to capture human emotions, such as tension, anxiety, relaxation, etc., from the speech signal.

[0058] However, these two methods have shortcomings in analyzing personnel status: the status information obtained by the voiceprint recognition method is mainly the personnel's ID (Identity) authentication information, which is single in content and not comprehensive enough; the method based on deep speech emotion analysis only focuses on the voice signal, and the status information obtained does not contain personnel information.

[0059] Based on the above problems, an embodiment of the present application proposes a personnel status analysis method based on human factors intelligence, which helps to integrate personnel identity information, emotional information and ambient sound information to perform personnel status analysis, making the analyzed personnel status information more comprehensive, while improving the accuracy of the emotional analysis results.

[0060] Figure 1 The flowchart of the personnel status analysis method based on human factor intelligence provided in the embodiment of the present application specifically includes the following steps:

[0061] Step S11, obtaining a sound signal collected by a sound collection device.

[0062] Specifically, a sound acquisition device is used to collect sound signals. Any sound acquisition device supported by a sound card can collect sound signals, and the final sound digital signal is obtained through operations such as analog-to-digital conversion and filtering.

[0063] Step S12: performing signal separation based on the sound signal to obtain a first type of sound signal, a second type of sound signal and a third type of sound signal.

[0064] Specifically, the blind source separation technology is used on the sound signal to separate the three types of sound signals of interest, that is, the sound signal is divided into the first type of sound signal, the second type of sound signal and the third type of sound signal. The main analysis methods of the blind source separation (BSS) technology include principal component analysis (PCA) and independent component analysis (ICA).

[0065] In the present application, the first type of sound signal is the voice signal of the target person, the second type of sound signal is the non-verbal sound signal emitted by the target person, and the third type of sound signal is the sound signal heard by the target person. Among them, the voice signal refers to the sound emitted by the target person during language communication, which is used to communicate and transmit information. The non-verbal sound signal emitted by the target person refers to other sounds emitted by the target person other than language communication, which are used to express emotions or physical conditions, including crying, laughing, coughing, groaning, applause, sighing, shouting, etc. The sound signals heard by the target person include environmental sounds (such as traffic noise, natural sounds, music, animal calls, etc.), other people's voices and non-verbal sounds, etc., which are used to provide environmental information.

[0066] Step S13: performing voiceprint recognition based on the first type of sound signal to obtain the identity information of the target person.

[0067] like Figure 2 As shown, Figure 2 The flowchart of voiceprint recognition provided by the embodiment of the present application is shown in step S13, which specifically includes the following steps:

[0068] Step S131: extracting speech features based on the first type of sound signal.

[0069] In step 1, useful feature information is extracted from the first type of sound signal, that is, speech features are extracted, such as MFCC (Mel-Frequency Cepstral Coefficients), PLP (Perceptual Linear Predictive, linear prediction coding), etc. These features can reflect the language sound characteristics of the speaker.

[0070] Step S132, comparing the speech feature with a plurality of trained voiceprint models to determine a target voiceprint model. Comparing the speech feature with a plurality of trained voiceprint models to determine a target voiceprint model.

[0071] In this step, a large amount of known speech data is used to train multiple voiceprint models, each of which represents the speech features of a different speaker. Common voiceprint models include GMM-UBM (Gaussian Mixture Model-Universal Background Model), i-vector (Identity-Vector, fixed-length vector representation) and PLDA (Probabilistic Linear Discriminant Analysis). The speech features extracted from the target person's speech signal are compared with multiple trained voiceprint models to find the most similar voiceprint model, that is, to confirm the target voiceprint model. The target voiceprint model is used to characterize the voiceprint model that is most similar to the speech features among multiple trained voiceprint models.

[0072] Optionally, a matching algorithm can be used to calculate the similarity between the speech feature and multiple trained voiceprint models, and the voiceprint model with the highest similarity is found as the target voiceprint model. Common matching algorithms include Euclidean distance, cosine similarity, and deep neural network.

[0073] Step S133: determining the identity information of the target person based on the target voiceprint model.

[0074] Specifically, the speaker identity corresponding to the target voiceprint model is the speaker identity of the input voice signal, thereby determining the identity information of the target person, such as the ID information of the target person.

[0075] In an optional embodiment, before step S13, that is, before voiceprint recognition is performed based on the first type of sound signal, the first type of sound signal (i.e., speech signal) can also be subjected to noise reduction and signal compensation to increase the signal-to-noise ratio of the speech signal and compensate for signal distortion. For example, the frequency components in the noise spectrum can be removed by using filtering technology to reduce the impact of noise and improve the signal-to-noise ratio. Common filtering methods include low-pass filtering, high-pass filtering, band-pass filtering, etc., or the use of wavelet decomposition methods for speech noise reduction. The human voice signal after noise reduction is subjected to speech estimation and compensation to make the sound sound clear and complete.

[0076] Step S14, performing emotion analysis based on the first type of sound signal and the second type of sound signal to obtain emotion information of the target person.

[0077] like Figure 3 As shown, Figure 3 The flowchart of sentiment analysis provided in the embodiment of the present application, step S14 specifically includes the following steps:

[0078] Step S141: Use speech recognition technology to convert the first type of sound signal into a text signal and extract text features.

[0079] Specifically, use speech recognition technology to convert the first type of sound signal (i.e., speech signal) into a text signal, and extract text features from the text signal. Specifically, text features can be extracted through word segmentation, stop word removal, stemming, etc. Among them, word segmentation refers to splitting text into words or phrases, stop word removal refers to removing common but low-information words such as "de" (of), "shi" (is), etc., and stemming refers to reducing words to their basic form. Through word segmentation, stop word removal, stemming, etc., it helps to simplify the text signal and extract more meaningful vocabulary from it, that is, extract text features. Optionally, text features include TF-IDF (Term Frequency-Inverse Document Frequency), word embedding, etc.

[0080] Step S142: Extract sound features based on the second type of sound signal.

[0081] Specifically, extract sound features from the second type of sound signal, including loudness (the intensity or volume of the sound), fundamental frequency (the pitch of the sound), timbre (the quality or characteristic of the sound), etc. These features can be used to analyze the emotional state of the speaker.

[0082] Step S143: Input the text features and the sound features into a trained emotion classification model for recognition to obtain the emotional information of the target person.

[0083] Specifically, use a large amount of known speech data and non-speech data to train an emotion classification model. This model can be constructed and trained based on rules (such as specific grammar or patterns), statistical models (such as Hidden Markov Models), or deep learning (such as Convolutional Neural Networks, Recurrent Neural Networks), etc., to achieve the recognition of a person's emotional information from speech and text, such as anger, satisfaction, anxiety, nervousness, relaxation, etc.

[0084] Input the text features and the sound features into a trained emotion classification model for recognition. The model outputs a probability distribution indicating the likelihood of the input features belonging to each emotion category. Based on this probability distribution, the most likely emotion category can be determined, and thus the emotional information of the target person can be obtained.

[0085] This application combines the target person's voice signal and the non-verbal sound signal emitted by the target person for emotion analysis, and integrates the text features extracted from the voice signal and the sound features extracted from the non-verbal sound signal to help improve the accuracy of emotion recognition. For example, through the text content of the target person's voice and the loudness of the laughter, it can be determined that the target person's emotional state is happy at this time, and the degree of happiness can also be determined.

[0086] Step S15: Perform sound analysis based on the third type of sound signal to obtain ambient sound information.

[0087] As shown in 4, Figure 4 The flowchart of the sound analysis provided in the embodiment of the present application is shown in step S15, wherein step S15 specifically includes the following steps:

[0088] Step S151: extracting audio features based on the third type of sound signal.

[0089] Specifically, audio features are extracted from the sound signals heard by the target person (such as environmental sounds, etc.), and the audio features include MFCC, chromaticity diagram, fundamental frequency, energy, etc.

[0090] Step S152: performing audio classification, audio segmentation, and audio event detection on the third type of sound signal based on the audio feature to obtain the ambient sound information.

[0091] In this step, an audio classifier (or classification model) can be used to classify the sound signals heard by the target person. The audio classifier is trained using audio data of known categories so that it can recognize and classify new audio signals. Furthermore, it is also possible to upload sound segments marked as misrecognition and corresponding category labels, and upload the currently used classification model at the same time, use the new sound data and the original training data, adjust the parameters of the classification model, and then download the trained new model to replace the original model to ensure the accuracy and reliability of the model.

[0092] Optionally, the third type of sound signal can be segmented using a supervised or unsupervised method, wherein the supervised method refers to segmenting different parts of the audio (e.g., speech, music, silence, etc.) using labeled data, and the unsupervised method refers to segmenting different parts of the audio using unlabeled data.

[0093] Audio event detection includes detecting audio events and silent segments. Audio event detection refers to identifying specific events in audio signals, such as speech, music clips, environmental noise, etc. Silence segment detection refers to identifying silent or low-energy segments in audio signals.

[0094] By performing audio classification, audio segmentation and audio event detection on the third type of sound signal, environmental sound information can be obtained. The environmental sound information includes the type of sound, the intensity of the sound, etc., so as to obtain information about the environmental status.

[0095] Step S16, performing personnel status analysis based on the target person's identity information, the target person's emotional information, and the ambient sound information.

[0096] In this step, the identity information, emotional information, and ambient sound information of the target person are quantified and fused to obtain fused features; the fused features are input into a trained personnel status analysis model to obtain the status of the target person.

[0097] Among them, feature quantization refers to converting various types of information (such as the identity information, emotional information, and ambient sound information of the target person) into quantifiable numerical or vector forms. For example, emotional information can be converted into emotional scores through a sentiment analysis model, and ambient sound information can be converted into feature vectors through audio feature extraction. Feature fusion refers to combining these quantified information to form a comprehensive feature representation (i.e., obtaining fused features). Feature fusion can be performed at the feature level (for example, splicing different feature vectors together) or at the decision level (for example, using weighted average or other fusion strategies). The fused feature representation is a feature set that integrates multiple types of information, which can more comprehensively describe the state of the target person.

[0098] Next, the fusion features are input into the trained personnel status analysis model to obtain the status of the target personnel. The personnel status analysis model is trained using a large amount of training data, so that it can obtain differential representations of different states from a large amount of training data, that is, it can capture the differences between different states and identify and distinguish the feature representations of different personnel states.

[0099] The present application performs signal separation on the acquired sound information to obtain the voice signal of the target person, the non-verbal sound signal emitted by the target person and the sound signal heard by the target person, performs voiceprint recognition based on the voice signal of the target person to obtain the identity information of the target person, performs emotion analysis based on the voice signal of the target person and the non-verbal sound signal emitted by the target person to obtain the emotion information of the target person, performs sound analysis based on the sound signal heard by the target person to obtain ambient sound information, performs personnel status analysis based on the identity information, emotion information and ambient sound information of the person, so that the personnel status information obtained by analysis includes both the identity information of the person and the emotion information of the person, making the status information more comprehensive, and simultaneously performs emotion analysis in combination with the voice signal and the sound signal related to the person (specifically referring to the non-verbal sound signal emitted by the target person), so that the result of the emotion analysis is more accurate.

[0100] In practical applications, taking security monitoring scenarios as an example, in a security monitoring system, the system can comprehensively analyze the ID authentication information (identity confirmation), emotional information (such as tension or relaxation) and ambient sound information (such as whether there is abnormal noise) of people in the monitoring area to determine the status information of the person and the security status of the area. For example, if an unauthorized person is monitored to enter, and the person's emotional state is excited, and the sound of glass breaking is detected in the ambient sound, the system may trigger an alarm.

[0101] Furthermore, the present application integrates the identity information, emotional information and ambient sound information of the personnel to perform personnel status analysis, and can also enable the personnel status analysis system to adjust the emotional analysis results according to the ambient sound information. For example, if there is a very harsh mechanical noise in the ambient sound, it may cause negative emotions (anxiety, disgust, etc.) in people, so the proportion of positive emotions in the emotional analysis results should not be too high. If people hear pleasant bird calls, etc., it may cause positive emotions (such as happiness, excitement, etc.), so the negative emotions in the emotional analysis results should not be too high. The accuracy and reliability of the emotional analysis results can be reflected through the ambient sound information, and the emotional analysis results can be further adjusted.

[0102] Based on the same idea, the embodiment of the present application also provides a personnel status analysis system based on human factor intelligence. Figure 5 A schematic diagram of the structure of a personnel status analysis system based on human factor intelligence provided in an embodiment of the present application, such as Figure 5 As shown, the personnel status analysis system 50 based on human factor intelligence may include:

[0103] An acquisition module 51 is used to acquire a sound signal collected by a sound collection device;

[0104] A signal separation module 52 is used to perform signal separation based on the sound signal to obtain a first type of sound signal, a second type of sound signal and a third type of sound signal, wherein the first type of sound signal is a speech signal of a target person, the second type of sound signal is a non-language sound signal emitted by the target person, and the third type of sound signal is a sound signal heard by the target person;

[0105] A voiceprint recognition module 53 is used to perform voiceprint recognition based on the first type of sound signal to obtain the identity information of the target person;

[0106] The emotion analysis module 54 is used to perform emotion analysis based on the first type of sound signal and the second type of sound signal to obtain the emotion information of the target person;

[0107] A sound analysis module 55 is used to perform sound analysis based on the third type of sound signal to obtain ambient sound information;

[0108] The personnel status analysis module 56 is used to perform personnel status analysis based on the identity information of the target personnel, the emotional information of the target personnel, and the ambient sound information.

[0109] In one possible implementation, the voiceprint recognition module 53 may also be used to:

[0110] Extracting speech features based on the first type of sound signal;

[0111] Compare the speech feature with multiple trained voiceprint models to determine a target voiceprint model, where the target voiceprint model is used to represent the voiceprint model that is most similar to the speech feature among the multiple trained voiceprint models;

[0112] Determine the identity information of the target person based on the target voiceprint model.

[0113] In one possible implementation, the speech feature is compared with a plurality of trained voiceprint models, including: using a matching algorithm to calculate the similarity between the speech feature and the plurality of trained voiceprint models respectively, and the target voiceprint model is the voiceprint model with the highest similarity.

[0114] In one possible implementation, the sentiment analysis module 54 may also be used to:

[0115] Using speech recognition technology to convert the first type of sound signal into a text signal and extract text features;

[0116] Extracting sound features based on the second type of sound signal;

[0117] The text features and sound features are input into the trained sentiment classification model for recognition to obtain the target person’s sentiment information.

[0118] In one possible implementation, the sound analysis module 55 may also be used to:

[0119] Extracting audio features based on the third type of sound signal;

[0120] Based on the audio features, the third type of sound signal is subjected to audio classification, audio segmentation and audio event detection to obtain the ambient sound information.

[0121] In one possible implementation, the personnel status analysis module 56 may also be used to:

[0122] The identity information, emotional information and ambient sound information of the target person are quantified and fused to obtain fusion features;

[0123] The fused features are input into the trained personnel status analysis model to obtain the status of the target personnel.

[0124] In one possible implementation, before performing voiceprint recognition based on the first type of sound signal, the method further includes:

[0125] The first type of sound signal is subjected to noise reduction and signal compensation.

[0126] In one possible implementation, the second type of sound signal includes at least crying, laughing, coughing, groaning, and applause; and the third type of sound signal includes at least environmental sound.

[0127] Figure 5 The human factors intelligence-based personnel status analysis system 50 provided in the illustrated embodiment can be used to execute the technical solution of the method embodiment shown in the present application. Its implementation principle and technical effects can be further referred to the relevant description in the method embodiment.

[0128] It should be understood that the above Figure 5 The division of the modules of the personnel status analysis system 50 based on human intelligence shown is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. For example, the signal separation module can be a separately established processing element, or it can be integrated in a chip of an electronic device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in a processor element or instructions in the form of software.

[0129] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (DSP), or one or more field programmable gate arrays (FPGA). For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0130] In the above embodiments, the processor involved may include, for example, a CPU, a DSP, a microcontroller or a digital signal processor, and may also include a GPU, an embedded neural network processor (Neural-network Process Units; hereinafter referred to as: NPU) and an image signal processor (Image Signal Processing; hereinafter referred to as: ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASIC, or one or more integrated circuits for controlling the execution of the program of the technical solution of the present application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

[0131] An embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method provided by the embodiment shown in the present application.

[0132] Combine the following Figure 6 The exemplary electronic device provided in the embodiments of the present application is further introduced. Figure 6 A schematic structural diagram of an electronic device 6000 is shown.

[0133] The above-mentioned electronic device 6000 may include: at least one processor; and at least one memory communicatively connected to the above-mentioned processor, wherein: the above-mentioned memory stores program instructions that can be executed by the above-mentioned processor, and the processor calls the above-mentioned program instructions to execute the personnel status analysis method based on human factors intelligence provided in the embodiment shown in this application.

[0134] Figure 6 A block diagram of an exemplary electronic device 6000 suitable for implementing the embodiments of the present application is shown. Figure 6 The electronic device 6000 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application. The embodiments of the present application do not limit the type of electronic device, and the electronic device can be a desktop computer, a tablet computer, a notebook computer, a palm computer, an ultra-mobile personal computer (UMPC) or a netbook, etc.

[0135] like Figure 6 As shown, the electronic device 6000 is in the form of a general computing device. The components of the electronic device 6000 may include, but are not limited to: one or more processors 6010, a memory 6020, a communication bus 6040 connecting different system components (including the memory 6020 and the processor 6010), and a communication interface 6030.

[0136] The communication bus 6040 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus and Peripheral Component Interconnection (PCI) bus.

[0137] The electronic device 6000 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device, including volatile and non-volatile media, removable and non-removable media.

[0138] The memory 6020 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. The electronic device may further include other removable / non-removable, volatile / non-volatile computer system storage media. Figure 6 Not shown, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a compact disc read only memory (CD-ROM), a digital versatile disc read only memory (DVD-ROM), or other optical media) may be provided. In these cases, each drive may be connected to the communication bus 6040 via one or more data medium interfaces. The memory 6020 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present application.

[0139] A program / utility having a set (at least one) of program modules may be stored in the memory 6020, such program modules including but not limited to an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. The program modules generally perform the functions and / or methods of the embodiments described herein.

[0140] The electronic device 6000 may also communicate with one or more external devices (e.g., keyboard, pointing device, display, etc.), one or more devices that enable a user to interact with the electronic device, and / or any device that enables the electronic device to communicate with one or more other computing devices (e.g., network card, modem, etc.). Such communication may be performed through the communication interface 6030. In addition, the electronic device 6000 may also communicate with the network adapter ( Figure 6 The network adapter can communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the communication bus 6040. It should be understood that although Figure 6 Not shown, other hardware and / or software modules may be used in conjunction with the electronic device 6000, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, disk arrays (Redundant Arrays of Independent Drives; hereinafter referred to as: RAID) systems, tape drives, and data backup storage systems.

[0141] The processor 6010 executes various functional applications and data processing by running the programs stored in the memory 6020, such as implementing the method provided in the embodiment of the present application.

[0142] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 6000. In other embodiments of the present application, the electronic device 6000 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0143] In the above embodiments, the processor involved may include, for example, a CPU, a DSP, a microcontroller or a digital signal processor, and may also include a GPU, an embedded neural network processor (Neural-network Process Units; hereinafter referred to as: NPU) and an image signal processor (Image Signal Processing; hereinafter referred to as: ISP). The processor may also include necessary hardware accelerators or logic processing hardware circuits, such as ASIC, or one or more integrated circuits for controlling the execution of the program of the technical solution of the present application. In addition, the processor may have the function of operating one or more software programs, and the software programs may be stored in a storage medium.

[0144] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems, systems and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0146] In several embodiments provided in the present application, if any function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory; hereinafter referred to as: ROM), random access memory (Random Access Memory; hereinafter referred to as: RAM), disk or optical disk, and other media that can store program codes.

[0147] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A personnel status analysis method based on human factor intelligence, characterized in that: The method comprises: Acquire a sound signal collected by a sound collection device; Perform signal separation based on the sound signal to obtain a first type of sound signal, a second type of sound signal and a third type of sound signal, wherein the first type of sound signal is a speech signal of a target person, the second type of sound signal is a non-language sound signal emitted by the target person, and the third type of sound signal is a sound signal heard by the target person; Performing voiceprint recognition based on the first type of sound signal to obtain identity information of the target person; Performing emotion analysis based on the first type of sound signal and the second type of sound signal to obtain emotion information of the target person; Performing sound analysis based on the third type of sound signal to obtain ambient sound information; Personnel status analysis is performed based on the identity information of the target person, the emotional information of the target person, and the ambient sound information.

2. The method according to claim 1, characterized in that The performing voiceprint recognition based on the first type of sound signal to obtain the identity information of the target person includes: Extracting speech features based on the first type of sound signal; Comparing the speech feature with a plurality of trained voiceprint models to determine a target voiceprint model, wherein the target voiceprint model is used to represent a voiceprint model that is most similar to the speech feature among the plurality of trained voiceprint models; The identity information of the target person is determined based on the target voiceprint model.

3. The method according to claim 2, characterized in that The comparing the speech feature with a plurality of trained voiceprint models includes: The similarities between the speech feature and the multiple trained voiceprint models are calculated respectively by using a matching algorithm, and the target voiceprint model is the voiceprint model with the highest similarity.

4. The method according to claim 1, characterized in that: The performing emotion analysis based on the first type of sound signal and the second type of sound signal to obtain the emotion information of the target person includes: Using speech recognition technology to convert the first type of sound signal into a text signal and extract text features; extracting sound features based on the second type of sound signal; The text features and the sound features are input into a trained emotion classification model for identification to obtain the emotion information of the target person.

5. The method according to claim 1, characterized in that The performing sound analysis based on the third type of sound signal to obtain ambient sound information includes: Extracting audio features based on the third type of sound signal; Based on the audio features, the third type of sound signal is subjected to audio classification, audio segmentation and audio event detection to obtain the ambient sound information.

6. The method according to any one of claims 1 to 5, characterized in that: The personnel status analysis based on the identity information of the target person, the emotional information of the target person, and the ambient sound information includes: Quantifying and fusing the identity information of the target person, the emotional information of the target person, and the ambient sound information to obtain fused features; The fused features are input into a trained personnel status analysis model to obtain the status of the target person.

7. The method according to claim 1, characterized in that Before performing voiceprint recognition based on the first type of sound signal, the method further includes: Noise reduction and signal compensation are performed on the first type of sound signal.

8. The method according to any one of claims 1 to 7, characterized in that: The second type of sound signal includes at least crying, laughing, coughing, groaning, and applause; the third type of sound signal includes at least environmental sound.

9. A personnel status analysis system based on human factor intelligence, characterized in that: The system comprises: An acquisition module is used to acquire the sound signal collected by the sound collection device; A signal separation module is used to perform signal separation based on the sound signal to obtain a first type of sound signal, a second type of sound signal and a third type of sound signal, wherein the first type of sound signal is a speech signal of a target person, the second type of sound signal is a non-language sound signal emitted by the target person, and the third type of sound signal is a sound signal heard by the target person; A voiceprint recognition module, used to perform voiceprint recognition based on the first type of sound signal to obtain the identity information of the target person; An emotion analysis module, used for performing emotion analysis based on the first type of sound signal and the second type of sound signal to obtain emotion information of the target person; A sound analysis module, used for performing sound analysis based on the third type of sound signal to obtain ambient sound information; The personnel status analysis module is used to perform personnel status analysis based on the identity information of the target person, the emotional information of the target person, and the ambient sound information.

10. An electronic device, characterized in that: include: A processor and a memory, the memory being used to store a computer program; the processor being used to run the computer program to implement the personnel status analysis method based on human factor intelligence as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed on a computer, the personnel status analysis method based on human factor intelligence as described in any one of claims 1 to 8 is implemented.

Images