Intelligent psychological state analysis method and system, electronic equipment and product

Abstract

The invention belongs to the technical field of emotion recognition, and aims to provide a psychological state intelligent analysis method and system, electronic equipment and a product. Multi-modal monitoring data such as facial emotion features, posture emotion features and sound emotion features can be fused, psychological state intelligent analysis and early warning are performed based on the multi-modal monitoring data, and emotion recognition precision is enhanced by combining diversified feature information of emotions; meanwhile, the trend of the psychological state is accurately recognized through the prediction model, more comprehensive psychological health management and early warning support are provided for key personnel of an enterprise, then the early intervention ability of the abnormal psychological state of the key personnel is improved, intervention of a management layer in the early stage of the psychological problem of the designated personnel is facilitated, and the management efficiency is improved. Therefore, healthy development of enterprises and employees is guaranteed.

Description

Technical Field

[0001] The present invention belongs to the technical field of emotion recognition, and specifically relates to a psychological state intelligent analysis method, system, electronic equipment and product. Background Art

[0002] In corporate management, the psychological state (such as stress, anxiety, and depression) of key personnel (hereinafter referred to as "key personnel") has a profound impact on work performance, team atmosphere, and even the long-term development of the company. At present, in order to achieve effective monitoring of the psychological state of key personnel, facial emotion recognition technology is gradually being applied to corporate management to analyze employees' emotional changes and provide them with healthy management support for their psychological state. Existing facial emotion recognition technology is mainly based on methods such as image processing and deep learning. By analyzing facial emotional features, it judges the emotional state (such as happiness, anger, or sadness, etc.), and then infers the emotional trend of employees. However, in the process of using the existing technology, the inventor found that there are at least the following problems in the existing technology: First, the existing technology is insufficient in terms of emotion recognition accuracy. Especially in complex work scenarios, the effect of emotion recognition is easily affected by multiple factors such as lighting, subtle changes in expression and occlusion. At the same time, data sources that rely solely on facial expressions are not sufficient to reflect the true emotional state of employees, resulting in a large deviation between the emotion recognition results and the actual situation, and low emotion recognition accuracy. Secondly, the existing technology mainly performs simple recognition of emotional states, but lacks dynamic correlation analysis between emotional states and long-term psychological states, making it difficult to fully reflect the changes in the psychological states of key personnel. In addition, since the instantaneous changes in emotional states and the cumulative changes in psychological states are not linear, it is impossible to directly deduce the psychological state through simple emotion recognition, resulting in low accuracy in the prediction of psychological states by existing technologies. Finally, most of the existing technologies currently rely on historical data for static analysis of emotional states, lack a forward-looking early warning mechanism, and cannot provide early warning support before the psychological state changes significantly, making it difficult to achieve early identification and intervention of employee psychological problems. Summary of the invention

[0003] The present invention aims to solve the above technical problems at least to a certain extent. The present invention provides a method, system, electronic device and product for intelligent analysis of psychological state.

[0004] In order to achieve the above object, the present invention adopts the following technical solutions: In a first aspect, the present invention provides a method for intelligent analysis of mental states, comprising: Collect multimodal monitoring data of designated personnel in real time, and perform feature extraction processing on the multimodal monitoring data to obtain multimodal feature information; Performing weighted fusion processing on the multimodal feature information to obtain the current fused emotional features of the designated person, and obtaining the current emotional label of the designated person according to the current fused emotional features; Obtain all historical emotion tags of a specified historical period before the current moment, and obtain emotion tag time series data based on all historical emotion tags and the current emotion tag; Inputting the emotion tag time series data into a preset emotion and psychological state mapping model, so as to obtain a predicted value of the psychological state of the designated person based on the emotion and psychological state mapping model; According to the psychological state prediction value, the psychological state abnormality detection is performed on the designated person, and when the psychological state prediction value is abnormal, the psychological state abnormality warning is issued to the designated person.

[0005] In a possible design, based on a preset multi-channel convolutional neural network, feature extraction processing is performed on the multimodal monitoring data, weighted fusion processing is performed on the multimodal feature information, and the current emotion label of the designated person is obtained according to the current fused emotion feature; the multi-channel convolutional neural network includes an input layer, a multi-channel convolutional layer, a feature fusion layer, a fully connected layer and an output layer connected in sequence, feature extraction processing is performed on the multimodal monitoring data based on the multi-channel convolutional neural network to obtain multimodal feature information, weighted fusion processing is performed on the multimodal feature information to obtain the current fused emotion feature of the designated person, and the current emotion label of the designated person is obtained according to the current fused emotion feature, including: Receiving the multimodal monitoring data based on the input layer, and inputting the multimodal monitoring data into a plurality of convolution channels in the multi-channel convolution layer in a one-to-one correspondence; Based on the multiple convolution channels in the multi-channel convolution layer, each modality monitoring data is independently processed to obtain multimodal feature information; Based on the feature fusion layer, the multimodal feature information is weightedly fused to obtain the current fused emotional feature of the designated person; Classifying the current fused emotion feature based on the fully connected layer to obtain a current emotion label of the designated person; The current emotion label is output based on the output layer.

[0006] In a possible design, the multimodal monitoring data includes facial image data, voice data and body data. Correspondingly, the multimodal feature information includes facial emotion features extracted from the facial image data, voice emotion features extracted from the voice data and body emotion features extracted from the body data; correspondingly, the current fused emotion features of the designated person are: F emotion = oh 1. F face + oh 2. F voice + oh 3. F posture ; In the formula, F face is the facial emotion feature, F voice is the voice emotion feature, F posture is the body emotion feature, oh 1. oh 2 and oh 3 is a preset weight corresponding to the multimodal feature information; The current emotion label is: .

[0007] In one possible design, the emotion and psychological state mapping model uses a long short-term memory network.

[0008] In a possible design, when obtaining the predicted value of the psychological state of the designated person based on the emotion and psychological state mapping model, the method further includes: Calculating the attention weight of each emotion tag in the emotion tag time series data through an attention mechanism; Using the attention weights of each emotion tag, weighted averaging is performed on the emotion tag time series data to obtain a global emotion feature representation; The global emotion feature representation is input into the fully connected layer in the emotion and psychological state mapping model to obtain the psychological state prediction value of the designated person.

[0009] In one possible design, the psychological state prediction value is: ; In the formula, W p and b p are respectively the preset weights and biases of the fully connected layer in the emotion and psychological state mapping model; is the preset activation function; H att is the global emotional feature representation.

[0010] In a second aspect, the present invention provides a mental state intelligent analysis system, which is used to implement a mental state intelligent analysis method as described in any one of the above; the mental state intelligent analysis system comprises: A feature information acquisition module is used to collect multimodal monitoring data of a designated person in real time, and perform feature extraction processing on the multimodal monitoring data to obtain multimodal feature information; An emotion label acquisition module, which is in communication with the feature information acquisition module and is used to perform weighted fusion processing on the multimodal feature information to obtain the current fused emotion feature of the designated person, and obtain the current emotion label of the designated person according to the current fused emotion feature; A psychological state prediction module is connected to the emotion label acquisition module for obtaining all historical emotion labels of a specified historical period before the current moment, and obtaining emotion label time series data according to all historical emotion labels and the current emotion label; and is also used to input the emotion label time series data into a preset emotion and psychological state mapping model, so as to obtain the psychological state prediction value of the designated person based on the emotion and psychological state mapping model; The abnormal warning module is communicatively connected with the psychological state prediction module, and is used to detect the abnormal psychological state of the designated person according to the psychological state prediction value, and to issue an abnormal psychological state warning to the designated person when the psychological state prediction value is abnormal.

[0011] In a third aspect, the present invention provides an electronic device, comprising: a memory for storing computer program instructions; and, A processor is used to execute the computer program instructions to complete the operation of a psychological state intelligent analysis method as described in any one of the above.

[0012] In a fourth aspect, the present invention provides a computer program product, comprising a computer program or instructions, which, when executed by a computer, implements a method for intelligent analysis of a psychological state as described in any one of the above.

[0013] In a fifth aspect, the present invention provides a computer-readable storage product having instructions stored thereon. When the instructions are run on a computer, a method for intelligent analysis of a psychological state as described in any one of the above is executed.

[0014] The beneficial effects of the present invention are: The present invention discloses a method, system, electronic device and product for intelligent analysis of psychological state, which can improve the accuracy of emotion recognition and the accuracy of psychological state assessment, and can also realize early warning of abnormal psychological state of personnel. The present invention can fuse multimodal monitoring data such as facial emotion features, body emotion features and voice emotion features, and perform intelligent analysis and early warning of psychological state based on multimodal monitoring data, and enhance the accuracy of emotion recognition by combining diverse characteristic information of emotions; at the same time, the present invention can accurately identify the trend of psychological state through a prediction model, provide more comprehensive mental health management and early warning support for key personnel of the enterprise, and thus improve the ability to intervene in advance for abnormal psychological state of key personnel, which helps the management to intervene in the early stage of psychological problems of designated personnel, thereby ensuring the healthy development of enterprises and employees.

[0015] Other beneficial effects of the present invention will be further described in the specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a flow chart of the method for intelligent analysis of psychological state in Example 1; Figure 2 is a module block diagram of the mental state intelligent analysis system in Example 2; Figure 3 This is a module block diagram of the electronic device in Example 3. DETAILED DESCRIPTION

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in combination with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0018] Embodiment 1: This embodiment discloses a method for intelligent analysis of mental states, which can be executed by, but is not limited to, a computer device or a virtual machine with certain computing resources, such as a personal computer, a smart phone, a personal digital assistant, or a wearable device, or by a virtual machine.

[0019] like Figure 1 As shown, a method for intelligent analysis of mental state may include but is not limited to the following steps: S1. Collect multimodal monitoring data of designated persons in real time, and perform feature extraction processing on the multimodal monitoring data to obtain multimodal feature information; specifically, in this embodiment, the multimodal monitoring data includes facial image data of the designated person captured by a camera, voice data of the designated person recorded by a microphone, and body data of the designated person obtained by a gesture recognition sensor, and correspondingly, the multimodal feature information includes facial emotion features extracted from the facial image data, voice emotion features such as tone and intonation extracted from the voice data that can characterize the user's emotions, and body emotion features extracted from the body data. In order to improve the extraction accuracy and efficiency of feature information, during the implementation of this embodiment, feature extraction is performed on data of different modalities through data channels of different modalities to obtain multimodal feature information.

[0020] S2. Perform weighted fusion processing on the multimodal feature information to obtain the current fused emotional features of the designated person, and obtain the current emotional label of the designated person based on the current fused emotional features.

[0021] In step S1 and step S2, based on the preset multi-channel convolutional neural network (MCNN), feature extraction processing is performed on the multimodal monitoring data, weighted fusion processing is performed on the multimodal feature information, and the current emotion label of the designated person is obtained according to the current fused emotion feature; the fused emotion feature is generated by the multi-channel convolutional neural network, which can make the emotion recognition result more comprehensive and accurate. The multi-channel convolutional neural network includes an input layer, a multi-channel convolutional layer, a feature fusion layer, a fully connected layer and an output layer connected in sequence. Based on the multi-channel convolutional neural network, feature extraction processing is performed on the multimodal monitoring data to obtain multimodal feature information, weighted fusion processing is performed on the multimodal feature information to obtain the current fused emotion feature of the designated person, and the current emotion label of the designated person is obtained according to the current fused emotion feature, including: A1. receiving the multimodal monitoring data based on the input layer, and inputting the multimodal monitoring data into the multiple convolution channels in the multi-channel convolution layer in a one-to-one correspondence; A2. Based on the multiple convolution channels in the multi-channel convolution layer, each modality monitoring data is independently processed to obtain multimodal feature information; specifically, in this embodiment, each convolution channel in the multi-channel convolution layer uses a number of convolution layers and pooling layers to capture local features in the corresponding modality monitoring data, and then the emotion feature information under different modalities can be obtained based on the multi-channel convolution layer, that is, multimodal feature information.

[0022] A3. Based on the feature fusion layer, the multimodal feature information is weightedly fused to obtain the current fused emotional features of the designated person; A4. classifying the current fused emotional features based on the fully connected layer to obtain the current emotional label of the designated person; A5. Output the current emotion label based on the output layer.

[0023] It should be noted that the multi-channel convolutional neural network is a deep learning model structure for processing multimodal data. The network designs independent convolution channels for data of different modalities, extracts feature information of each modality, and performs weighted fusion of these multimodal features in subsequent layers. It can generate more representative, accurate and stable emotion labels, providing basic data support for subsequent psychological state analysis.

[0024] Specifically, in step S1, the multimodal monitoring data includes facial image data, voice data and body data, and correspondingly, the multimodal feature information includes facial emotion features extracted from the facial image data, voice emotion features extracted from the voice data, and body emotion features extracted from the body data; correspondingly, in step S2, the current fused emotion features of the designated person are: F emotion = oh 1. F face + oh 2. F voice + oh 3. F posture ; In the formula, F face is the facial emotion feature, F voice is the voice emotion feature, F posture is the body emotion feature, oh 1. oh 2 and oh 3 is a preset weight corresponding to the multimodal feature information; it should be noted that, in this embodiment, the weight corresponding to the multimodal feature information is adjusted according to the contribution of different modalities to emotion recognition to ensure that the fused features are more representative. During the implementation process, the multi-channel convolutional neural network automatically learns the most weighted of each modality to ensure that the final fused emotion feature results are more reliable.

[0025] The current emotion label is: .

[0026] In this embodiment, the emotion label is represented by a numerical value, which can facilitate subsequent psychological state analysis and prediction.

[0027] As an example, suppose the multimodal feature information of a specified person is: facial emotion features F face =0.7, voice emotion characteristics F voice =0.4, body emotion characteristics F posture =0.6, if the weight oh 1=0.5, oh 2=0.3, oh 3=0.2, then the calculated current fused emotion feature is F emotion =0.5·0.7+0.3·0.4+0.2·0.6=0.59, according to the preset emotion classification standard, F emotion >0.5, the emotion is judged to be “happy” and the corresponding emotion label is “1”.

[0028] S3. Obtain all historical emotion tags of a specified historical period before the current moment, and obtain the emotion tag time series data based on all historical emotion tags and the current emotion tag; specifically, in step S3, the emotion tag time series data is: E=[e t-n+1 ,……,e t-1 , e t ], n is a natural number greater than 1, e t is the current emotion label at the current time t, e t-1 is the emotion label of the previous collection time before the current time t, ..., e t-n+1 is the emotion label of n-1 collection moments before the current moment t.

[0029] S4. Inputting the emotion tag time series data into a preset emotion and psychological state mapping model, so as to obtain a predicted value of the psychological state of the designated person based on the emotion and psychological state mapping model.

[0030] In step S4, the emotion and psychological state mapping model adopts a long short-term memory network (LSTM). It should be noted that the LSTM gradually processes the emotion label time series data in the time dimension through multiple gated units, captures the change pattern of the emotion label over time, identifies the trend of emotion change, and generates the psychological state prediction value of the designated person.

[0031] During the implementation of this embodiment, since the long short-term memory network has the ability to memorize long-term series information, it can capture the cumulative effect of emotional changes over time. Compared with traditional static analysis, this embodiment can understand the continuous changes of emotional states over time and more accurately predict the psychological states accumulated over a long period of time, such as stress and anxiety, which is suitable for realizing real-time dynamic warnings in the daily emotional monitoring of key personnel; at the same time, through the long short-term memory network, the model can better utilize the contextual information of the emotional label time series data, associate the emotional changes at the current moment with the previous emotional state, reduce the misjudgment of a single emotional fluctuation, and thus improve the accuracy of the judgment of the overall psychological state. In addition, the long short-term memory network can handle the nonlinear relationship between emotional changes and psychological states, and is suitable for complex emotional patterns (such as gradually accumulated stress or sudden emotional fluctuations), so that this embodiment is not only sensitive to linear trends, but also can reflect the implicit features in the emotional sequence, thereby capturing the potential impact of different emotional changes on psychological states.

[0032] In an implementation of obtaining the predicted value of the psychological state of the designated person, the emotion and psychological state mapping model processes the emotion tag time series data through a series of gated units, captures the long-term dependencies in the emotion tag time series data, and outputs the hidden state and cell state of each time step. Correspondingly, the steps of obtaining the predicted value of the psychological state of the designated person based on the emotion and psychological state mapping model in S4 are as follows: S401. Initialize hidden state h0 and cell state c0: The initial state of the emotion and psychological state mapping model when starting to process the emotion label time series data is usually initialized to a zero vector, indicating that the model has no memory information at the beginning.

[0033] S402. At each time step, the forget gate is used to control whether to retain the cell state at the previous moment, and the input gate and the candidate cell state are used to determine whether to update the current cell state. When the cell state needs to be updated, the forget gate and the input gate are combined to update the cell state, and the output gate and the hidden state are used to generate the hidden state of the current time step, so as to realize the cyclic processing of each emotion label in the emotion label time series data, and update the hidden state and the cell state in turn.

[0034] S403. After the emotion and psychological state mapping model processes all time steps, the final hidden state is obtained. h T , and can map the hidden state to the psychological state prediction value through a fully connected layer, and output the psychological state prediction value.

[0035] Specifically, the psychological state prediction value is: P = Wp + h T + b p ; In the formula, W p and b p are respectively the preset weights and biases of the fully connected layer in the emotion and psychological state mapping model, which are obtained by optimizing the model based on the preset training data through the back propagation algorithm; h T It is the final hidden state of the emotion and psychological state mapping model, which is obtained according to all emotion tags in the emotion tag time series data and contains comprehensive information of all emotion tags in the entire emotion tag time series data.

[0036] The above scheme is still insufficient in dealing with complex emotion change patterns and long-term emotion fluctuation prediction. Emotional states have long-term dependencies and nonlinear changes in time series, making it difficult to capture key emotion fluctuations and their core impact on psychological states. Therefore, in another implementation of obtaining the psychological state prediction value of the designated person, in order to cope with the prediction of multiple emotion evolution patterns and long-term emotion fluctuations, this embodiment further makes the following improvements: when obtaining the psychological state prediction value of the designated person based on the emotion and psychological state mapping model, the method also includes: The attention weight of each emotion tag in the emotion tag time series data is calculated by the attention mechanism; it should be noted that, in this embodiment, the importance of each emotion tag in the emotion tag time series data can be calculated by the attention mechanism, and the key emotion tags in the emotion tag time series data are given a higher weight, thereby obtaining the attention weight of each emotion tag. The attention weight of any emotion tag is used to reflect the degree of influence of any emotion tag on the current psychological state of the designated person. Specifically, in this embodiment, the calculation formula of the attention weight of the current emotion tag in the emotion tag time series data is as follows: ; In the formula, W a It is a trainable preset weight matrix for emotion feature mapping; h j For the j The hidden state of the emotion label, T is the total number of emotion tags in the emotion tag time series data, and T is a natural number greater than 1; h t is the hidden state of the current emotion label.

[0037] The attention weights of each emotion tag are used to perform weighted average processing on the emotion tag time series data to obtain a global emotion feature representation for subsequent psychological state prediction; specifically, the global emotion feature representation is: ; The global emotional feature representation is input into the fully connected layer in the emotion and psychological state mapping model to obtain the psychological state prediction value of the designated person. At this time, the psychological state prediction value is: ; In the formula, W p and b p are respectively the preset weights and biases of the fully connected layer in the emotion and psychological state mapping model; It is a preset activation function, such as Sigmoid or ReLU, which is used to map the output to a preset range of the psychological state, such as mapping it to the range of [0, 1].

[0038] Based on this, this embodiment can effectively capture the impact of dynamic changes in emotions on psychological states through deep learning and attention mechanisms on the basis of multimodal emotion recognition, thereby enriching the correlation analysis between emotions and psychological states.

[0039] S5. Perform psychological state abnormality detection on the designated person according to the psychological state prediction value, and issue a psychological state abnormality warning to the designated person when the psychological state prediction value is abnormal.

[0040] This embodiment realizes intelligent analysis of the psychological state of key personnel based on emotion recognition, which can improve the accuracy of emotion recognition and the accuracy of psychological state assessment, and can also realize early warning of abnormal psychological state of personnel. Specifically, in the implementation process of this embodiment, multimodal monitoring data of designated personnel are collected in real time, and feature extraction processing is performed on the multimodal monitoring data to obtain multimodal feature information; then, weighted fusion processing is performed on the multimodal feature information to obtain the current fused emotional features of the designated personnel, and the current emotional label of the designated personnel is obtained according to the current fused emotional features; then, all historical emotional labels of the designated historical period before the current moment are obtained, and the emotional label time series data is obtained according to all historical emotional labels and the current emotional label; then, the emotional label time series data is input into a preset emotion and psychological state mapping model, so as to obtain the psychological state prediction value of the designated personnel based on the emotion and psychological state mapping model; finally, according to the psychological state prediction value, the psychological state abnormality detection is performed on the designated personnel, and when the psychological state prediction value is abnormal, the psychological state abnormality warning is given to the designated personnel. Based on this, this embodiment can integrate multimodal monitoring data such as facial emotion features, body emotion features and voice emotion features, and perform intelligent analysis and early warning of psychological state based on the multimodal monitoring data, and enhance the accuracy of emotion recognition by combining the diverse characteristic information of emotions; at the same time, this embodiment can accurately identify the trend of psychological state through the prediction model, provide more comprehensive mental health management and early warning support for key personnel of the enterprise, and thus improve the ability to intervene in the abnormal psychological state of key personnel in advance, which helps the management to intervene in the early stages of psychological problems of designated personnel, thereby ensuring the healthy development of the enterprise and employees.

[0041] Embodiment 2: This embodiment discloses a psychological state intelligent analysis system for implementing the psychological state intelligent analysis method in embodiment 1; Figure 2 As shown, the mental state intelligent analysis system includes: A feature information acquisition module is used to collect multimodal monitoring data of a designated person in real time, and perform feature extraction processing on the multimodal monitoring data to obtain multimodal feature information; An emotion label acquisition module, which is in communication with the feature information acquisition module and is used to perform weighted fusion processing on the multimodal feature information to obtain the current fused emotion feature of the designated person, and obtain the current emotion label of the designated person according to the current fused emotion feature; A psychological state prediction module is connected to the emotion label acquisition module for obtaining all historical emotion labels of a specified historical period before the current moment, and obtaining emotion label time series data according to all historical emotion labels and the current emotion label; and is also used to input the emotion label time series data into a preset emotion and psychological state mapping model, so as to obtain the psychological state prediction value of the designated person based on the emotion and psychological state mapping model; The abnormal warning module is communicatively connected with the psychological state prediction module, and is used to detect the abnormal psychological state of the designated person according to the psychological state prediction value, and to issue an abnormal psychological state warning to the designated person when the psychological state prediction value is abnormal.

[0042] It should be noted that the working process, working details and technical effects of the intelligent analysis system for mental state provided in this embodiment 2 can be found in embodiment 1 and will not be described in detail here.

[0043] Embodiment 3: Based on Embodiment 1 or 2, this embodiment discloses an electronic device, which may be a smart phone, a tablet computer, a laptop computer, or a desktop computer. The electronic device may be referred to as a user terminal, a portable terminal, a desktop terminal, etc. Figure 3 As shown, the electronic equipment includes: a memory for storing computer program instructions; and, A processor is used to execute the computer program instructions to complete the operation of a psychological state intelligent analysis method as described in any one of Example 1.

[0044] Specifically, the processor 301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen.

[0045] The memory 302 may include one or more computer-readable storage media, which may be non-transitory. The memory 302 may also include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 302 is used to store at least one instruction, which is used to be executed by the processor 301 to implement the mental state intelligent analysis method provided in Example 1 of the present application.

[0046] In some embodiments, the terminal may further optionally include: a communication interface 303 and at least one peripheral device. The processor 301, the memory 302 and the communication interface 303 may be connected via a bus or a signal line. Each peripheral device may be connected to the communication interface 303 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 304, a display screen 305 and a power supply 306.

[0047] The communication interface 303 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 301 and the memory 302. In some embodiments, the processor 301, the memory 302, and the communication interface 303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 301, the memory 302, and the communication interface 303 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0048] The radio frequency circuit 304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 304 communicates with a communication network and other communication devices through electromagnetic signals.

[0049] The display screen 305 is used to display a UI (User Interface). The UI may include any combination of graphics, text, icons, and videos.

[0050] The power supply 306 is used to supply power to various components in the electronic device.

[0051] Embodiment 4: Based on any one of Embodiments 1 to 3, this embodiment discloses a computer program product, including a computer program or instructions, which, when executed by a computer, implements a method for intelligent analysis of mental states as described in any one of Embodiments 1. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0052] Embodiment 5: Based on any one of Embodiments 1 to 3, this embodiment discloses a computer-readable storage product, on which instructions are stored, and when the instructions are run on a computer, a method for intelligent analysis of psychological states as described in any one of Embodiments 1 is executed. The computer-readable storage product refers to a carrier for storing data, which may include but is not limited to computer-readable storage media such as floppy disks, optical disks, hard disks, flash memory, USB flash disks, and / or memory sticks, and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0053] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent analysis of psychological state, characterized in that: include: Collect multimodal monitoring data of designated personnel in real time, and perform feature extraction processing on the multimodal monitoring data to obtain multimodal feature information; Performing weighted fusion processing on the multimodal feature information to obtain the current fused emotional features of the designated person, and obtaining the current emotional label of the designated person according to the current fused emotional features; Obtain all historical emotion tags of a specified historical period before the current moment, and obtain emotion tag time series data based on all historical emotion tags and the current emotion tag; Inputting the emotion tag time series data into a preset emotion and psychological state mapping model, so as to obtain a predicted value of the psychological state of the designated person based on the emotion and psychological state mapping model; According to the psychological state prediction value, the psychological state abnormality detection is performed on the designated person, and when the psychological state prediction value is abnormal, the psychological state abnormality warning is issued to the designated person.

2. A method for intelligent analysis of psychological state according to claim 1, characterized in that: Based on a preset multi-channel convolutional neural network, feature extraction processing is performed on the multimodal monitoring data, weighted fusion processing is performed on the multimodal feature information, and the current emotion label of the designated person is obtained according to the current fused emotion feature; the multi-channel convolutional neural network includes an input layer, a multi-channel convolutional layer, a feature fusion layer, a fully connected layer and an output layer connected in sequence, feature extraction processing is performed on the multimodal monitoring data based on the multi-channel convolutional neural network to obtain multimodal feature information, weighted fusion processing is performed on the multimodal feature information to obtain the current fused emotion feature of the designated person, and the current emotion label of the designated person is obtained according to the current fused emotion feature, including: Receiving the multimodal monitoring data based on the input layer, and inputting the multimodal monitoring data into a plurality of convolution channels in the multi-channel convolution layer in a one-to-one correspondence; Based on the multiple convolution channels in the multi-channel convolution layer, each modality monitoring data is independently processed to obtain multimodal feature information; Based on the feature fusion layer, the multimodal feature information is weightedly fused to obtain the current fused emotional feature of the designated person; Classifying the current fused emotion feature based on the fully connected layer to obtain a current emotion label of the designated person; The current emotion label is output based on the output layer.

3. A method for intelligent analysis of psychological state according to claim 2, characterized in that: The multimodal monitoring data includes facial image data, voice data and body data. Correspondingly, the multimodal feature information includes facial emotion features extracted from the facial image data, voice emotion features extracted from the voice data and body emotion features extracted from the body data. Correspondingly, the current fused emotion features of the designated person are: F emotion = ω 1· F face + ω 2· F voice + ω 3· F posture ； In the formula, F face is the facial emotion feature, F voice is the voice emotion feature, F posture is the body emotion feature, ω 1. ω 2 and ω 3 is a preset weight corresponding to the multimodal feature information; The current emotion label is: 。 4. A method for intelligent analysis of psychological state according to claim 1, characterized in that: The emotion and psychological state mapping model adopts a long short-term memory network.

5. A method for intelligent analysis of psychological state according to claim 4, characterized in that: When obtaining the predicted value of the psychological state of the designated person based on the emotion and psychological state mapping model, the method further includes: Calculating the attention weight of each emotion tag in the emotion tag time series data through an attention mechanism; Using the attention weights of each emotion tag, weighted averaging is performed on the emotion tag time series data to obtain a global emotion feature representation; The global emotion feature representation is input into the fully connected layer in the emotion and psychological state mapping model to obtain the psychological state prediction value of the designated person.

6. A method for intelligent analysis of psychological state according to claim 5, characterized in that: The psychological state prediction value is: ； In the formula, W p and b p are respectively the preset weights and biases of the fully connected layer in the emotion and psychological state mapping model; is the preset activation function; H att is the global emotional feature representation.

7. A psychological state intelligent analysis system, characterized in that: Used to implement a psychological state intelligent analysis method as claimed in any one of claims 1 to 6; The mental state intelligent analysis system comprises: A feature information acquisition module is used to collect multimodal monitoring data of a designated person in real time, and perform feature extraction processing on the multimodal monitoring data to obtain multimodal feature information; An emotion label acquisition module, which is in communication with the feature information acquisition module and is used to perform weighted fusion processing on the multimodal feature information to obtain the current fused emotion feature of the designated person, and obtain the current emotion label of the designated person according to the current fused emotion feature; A psychological state prediction module is connected to the emotion label acquisition module for obtaining all historical emotion labels of a specified historical period before the current moment, and obtaining emotion label time series data according to all historical emotion labels and the current emotion label; and is also used to input the emotion label time series data into a preset emotion and psychological state mapping model, so as to obtain the psychological state prediction value of the designated person based on the emotion and psychological state mapping model; The abnormal warning module is communicatively connected with the psychological state prediction module, and is used to detect the abnormal psychological state of the designated person according to the psychological state prediction value, and to issue an abnormal psychological state warning to the designated person when the psychological state prediction value is abnormal.

8. An electronic device, characterized in that: include: a memory for storing computer program instructions; as well as, A processor is used to execute the computer program instructions to complete the operation of a psychological state intelligent analysis method as described in any one of claims 1 to 6.

9. A computer program product comprising a computer program or instructions, characterized in that When the computer program or the instruction is executed by a computer, the computer program or the instruction implements a method for intelligent analysis of psychological state as described in any one of claims 1 to 6.

10. A computer readable storage product, characterized in that: The computer-readable storage product stores instructions, and when the instructions are run on a computer, a method for intelligent analysis of psychological states as described in any one of claims 1 to 6 is executed.

Images