A dynamic abnormal emotion early warning monitoring system based on the Internet of Things
By combining the cross-modal features of face images and heart rate data, deep learning technology is used for emotion recognition, which solves the problem of low accuracy of single modal data recognition, realizes the comprehensive and accurate emotion monitoring, and improves the efficiency of emotion management .
Patent Information
- Application Number
- CN202410607397.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-16
- Publication Date
- 2024-11-15
- Estimated Expiration
- 2044-05-16
AI Technical Summary
The existing abnormal emotions monitoring system mainly relies on single-modal data for emotion recognition, which cannot fully reflect the individual's emotional state, and is easily affected by noise and interference, resulting in low accuracy of emotion recognition.
Using artificial intelligence technology based on deep learning, the face image and heart rate data of the monitored object are monitored and analyzed, and the multi-scale semantic features of the face image and the timing dynamic change mode of the heart rate data is extracted, and the cross-modal joint features are carried out. Emotion category recognition and generate warning prompts.
It realizes comprehensive, accurate and real-time monitoring and early warning of individual emotional states, and improves the efficiency and quality of emotional monitoring management.
Smart Images

Figure CN118319307B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of intelligent monitoring technology, and more specifically, to a dynamic abnormal emotion early warning monitoring system based on the Internet of Things. Background Art
[0002] In modern society, emotion management is crucial to the mental health of individuals and the harmony of interpersonal relationships. Traditional emotion monitoring methods often rely on manual observation and questionnaire surveys, which are not only inefficient but also easily affected by subjective factors. With the rapid development of Internet of Things technology, intelligent abnormal emotion monitoring systems have emerged, which can monitor and analyze the emotional state of individuals in real time, and issue warnings in time when abnormal emotions are found, providing effective emotion management support for individuals. At present, emotion monitoring technology is widely used in schools, hospitals, enterprises and military environments to support mental health monitoring and intervention.
[0003] However, existing abnormal emotion monitoring systems mainly rely on single-modal data for emotion recognition, such as voice, text, or facial expressions. This single-modal data often cannot fully reflect the emotional state of an individual and is easily affected by noise and interference, resulting in low accuracy in emotion recognition. Therefore, an optimized dynamic abnormal emotion early warning monitoring system based on the Internet of Things is expected. Summary of the invention
[0004] In order to solve the above technical problems, the present application is proposed. The embodiment of the present application provides a dynamic abnormal emotion early warning monitoring system based on the Internet of Things, which uses artificial intelligence technology based on deep learning to monitor and analyze the facial images and heart rate data of the monitored object, extract the multi-scale semantic features of the facial images, and the temporal dynamic change pattern of the heart rate data, and intelligently identify the emotional category of the monitored object based on the cross-modal joint features between the two, thereby generating early warning prompts for the abnormal emotions of the monitored object. In this way, comprehensive, accurate and real-time monitoring and early warning of the individual's emotional state can be achieved, effectively improving the efficiency and quality of emotion monitoring management.
[0005] Accordingly, according to one aspect of the present application, a dynamic abnormal emotion early warning monitoring system based on the Internet of Things is provided, which includes:
[0006] A physiological data acquisition module, used to acquire a facial image of a monitored object acquired by a camera, and a time series of heart rate values of the monitored object acquired by a wearable device;
[0007] An Internet of Things communication transmission module, used to transmit the time series of the face image and the heart rate value to a background monitoring server via an Internet of Things communication network;
[0008] A multi-scale semantic feature extraction module for face status, used to perform multi-scale semantic feature mining on the face image in the background monitoring server to obtain a multi-scale semantic feature vector for the face status;
[0009] A heart rate time series correlation coding module is used to perform time series correlation coding on the time series of the heart rate value in the background monitoring server to obtain a heart rate time series correlation implicit feature vector;
[0010] The cross-modal joint feature analysis module is used to determine whether to generate a warning prompt signal in the background monitoring server based on the cross-modal joint features of the multi-scale semantic feature vector of the facial state and the heart rate time series associated implicit feature vector.
[0011] In the above-mentioned dynamic abnormal emotion early warning monitoring system based on the Internet of Things, the multi-scale semantic feature extraction module of the facial state includes: a facial adaptive feature extraction unit, which is used to pass the facial image through a facial adaptive feature extractor based on the GoogLeNet model to obtain a facial state feature map; a multi-scale semantic encoding unit, which is used to input the facial state feature map into the multi-scale semantic feature extractor to obtain the multi-scale semantic feature vector of the facial state.
[0012] In the above-mentioned dynamic abnormal emotion early warning monitoring system based on the Internet of Things, the multi-scale semantic encoding unit includes: a global semantic feature extraction subunit, used to extract the global semantic features of the facial state feature map to obtain a facial state global semantic feature vector; a local semantic feature extraction subunit, used to extract the local semantic features of the facial state feature map to obtain a facial state local semantic feature vector; a multi-scale semantic feature fusion subunit, used to fuse the facial state global semantic feature vector and the facial state local semantic feature vector to obtain the facial state multi-scale semantic feature vector.
[0013] In the above-mentioned dynamic abnormal emotion early warning monitoring system based on the Internet of Things, the global semantic feature extraction subunit is used to: upsample the facial state feature map to obtain an upsampled facial state feature map; perform global mean pooling processing on the upsampled facial state feature map to obtain an upsampled facial state semantic channel feature vector; perform point convolution encoding on the upsampled facial state semantic channel feature vector to obtain the facial state global semantic feature vector.
[0014] In the above-mentioned dynamic abnormal emotion early warning monitoring system based on the Internet of Things, the local semantic feature extraction subunit is used to: upsample the facial state feature map to obtain an upsampled facial state feature map; perform point convolution encoding on the upsampled facial state feature map to obtain a channel-modulated upsampled facial state feature map; perform two-dimensional convolution encoding on the channel-modulated upsampled facial state feature map to obtain the facial state local semantic feature vector.
[0015] In the above-mentioned dynamic abnormal emotion early warning monitoring system based on the Internet of Things, the heart rate time series association encoding module is used to: input the time series of the heart rate value into a heart rate time series encoder based on a bidirectional long short-term memory neural network model to obtain the heart rate time series association implicit feature vector.
[0016] In the above-mentioned dynamic abnormal emotion early warning monitoring system based on the Internet of Things, the cross-modal joint feature analysis module includes: a cross-modal joint encoding unit, which is used to perform cross-modal joint encoding on the multi-scale semantic feature vector of the facial state and the heart rate time series associated implicit feature vector to obtain a heart rate-face image multi-modal implicit association feature matrix; an emotion category recognition unit, which is used to pass the heart rate-face image multi-modal implicit association feature matrix through a classifier-based emotion recognizer to obtain an emotion recognition result, and the emotion recognition result is used to represent the emotion category label; an abnormal emotion early warning unit, which is used to generate the early warning prompt signal in response to the emotion recognition result belonging to the abnormal emotion class.
[0017] In the above-mentioned dynamic abnormal emotion warning monitoring system based on the Internet of Things, the cross-modal joint encoding unit is used to: input the multi-scale semantic feature vector of the facial state and the heart rate time series associated implicit feature vector into the cross-modal feature joint encoder based on the Clip model to obtain the heart rate-face image multi-modal implicit association feature matrix.
[0018] In the above-mentioned dynamic abnormal emotion early warning monitoring system based on the Internet of Things, it also includes a training module for training the face adaptive feature extractor based on the GoogLeNet model, the multi-scale semantic feature extractor, the heart rate time series encoder based on the bidirectional long short-term memory neural network model, the cross-modal feature joint encoder based on the Clip model and the classifier-based emotion recognizer.
[0019] In the above-mentioned dynamic abnormal emotion early warning monitoring system based on the Internet of Things, the training module includes: a training data acquisition unit, which is used to acquire training data, wherein the training data includes a training face image of the monitored object collected by a camera and a time series of training heart rate values of the monitored object collected by a wearable device, as well as the true value of the emotion category label; a training data transmission unit, which is used to transmit the training face image and the time series of training heart rate values to a background monitoring server through an Internet of Things communication network; a training face image feature extraction unit, which is used to extract the training face image through the GoogLeNet model-based a face adaptive feature extractor to obtain a training face state feature map; a training multi-scale semantic feature extraction unit for inputting the training face state feature map into the multi-scale semantic feature extractor to obtain a training face state multi-scale semantic feature vector; a training heart rate value time series encoding unit for inputting the time series of the training heart rate value into the heart rate time series encoder based on the bidirectional long short-term memory neural network model to obtain a training heart rate time series associated implicit feature vector; a training data joint encoding unit for inputting the training face state multi-scale semantic feature vector and the training heart rate time series associated implicit feature vector into the encoder based on the Cl The present invention relates to a cross-modal feature joint encoder of the Cl ip model to obtain a training heart rate-face image multi-modal implicit correlation feature matrix; a classification loss unit, used to pass the training heart rate-face image multi-modal implicit correlation feature matrix through the classifier-based emotion recognizer to obtain a class probability value and a classification loss function value; a multi-distribution noise loss unit, used to calculate a multi-distribution noise loss function value based on the class probability value; a model training unit, used to train the face adaptive feature extractor based on the GoogLeNet model, the multi-scale semantic feature extractor, the heart rate time series encoder based on the bidirectional long short-term memory neural network model, the cross-modal feature joint encoder based on the Cl ip model and the classifier-based emotion recognizer with the weighted sum of the classification loss function value and the multi-distribution noise loss function value as the loss function value.
[0020] Compared with the prior art, the dynamic abnormal emotion early warning monitoring system based on the Internet of Things provided by this application uses artificial intelligence technology based on deep learning to monitor and analyze the facial images and heart rate data of the monitored object, extract the multi-scale semantic features of the facial images, and the temporal dynamic change pattern of the heart rate data, and intelligently identify the emotional category of the monitored object based on the cross-modal joint features between the two, thereby generating early warning prompts for the abnormal emotions of the monitored object. In this way, comprehensive, accurate, and real-time monitoring and early warning of individual emotional states can be achieved, effectively improving the efficiency and quality of emotional monitoring management. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.
[0022] Figure 1 It is a block diagram of a dynamic abnormal emotion early warning monitoring system based on the Internet of Things according to an embodiment of the present application.
[0023] Figure 2 Schematic diagram of the architecture of a dynamic abnormal emotion early warning monitoring system based on the Internet of Things according to an embodiment of the present application.
[0024] Figure 3 It is a block diagram of a multi-scale semantic feature extraction module for facial state in a dynamic abnormal emotion early warning monitoring system based on the Internet of Things according to an embodiment of the present application.
[0025] Figure 4 It is a block diagram of a multi-scale semantic coding unit in a dynamic abnormal emotion early warning monitoring system based on the Internet of Things according to an embodiment of the present application.
[0026] Figure 5 It is a block diagram of a cross-modal joint feature analysis module in a dynamic abnormal emotion early warning monitoring system based on the Internet of Things according to an embodiment of the present application.
[0027] Figure 6 It is a block diagram of a training module in the dynamic abnormal emotion early warning monitoring system based on the Internet of Things according to an embodiment of the present application. DETAILED DESCRIPTION
[0028] Below, the embodiments of the present application will be described in more detail in conjunction with the accompanying drawings, and the above and other purposes, features and advantages of the present application will become more apparent. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the example embodiments described herein.
[0029] As mentioned in the background technology above, the existing abnormal emotion monitoring system mainly relies on single-modal data for emotion recognition, such as only through voice, text or facial expressions. Such single-modal data often cannot fully reflect the emotional state of an individual, and is easily affected by noise and interference, resulting in low accuracy of emotion recognition. In response to the above technical problems, the technical concept of the present application is to use artificial intelligence technology based on deep learning to monitor and analyze the facial images and heart rate data of the monitored object, extract the multi-scale semantic features of the facial image, and the time series dynamic change pattern of the heart rate data, and intelligently identify the emotional category of the monitored object based on the cross-modal joint features between the two, so as to generate early warning prompts for the abnormal emotions of the monitored object. In this way, comprehensive, accurate and real-time monitoring and early warning of the individual's emotional state can be achieved, effectively improving the efficiency and quality of emotion monitoring management.
[0030] Figure 1 It is a block diagram of a dynamic abnormal emotion early warning monitoring system based on the Internet of Things according to an embodiment of the present application. Figure 2 FIG. 1 is a schematic diagram of the architecture of a dynamic abnormal emotion early warning monitoring system based on the Internet of Things according to an embodiment of the present application. Figure 1 and Figure 2 As shown, according to the embodiment of the present application, the dynamic abnormal emotion early warning monitoring system 100 based on the Internet of Things includes: a physiological data acquisition module 110, which is used to obtain a facial image of a monitored object collected by a camera, and a time series of a heart rate value of the monitored object collected by a wearable device; an Internet of Things communication transmission module 120, which is used to transmit the facial image and the time series of the heart rate value to a background monitoring server through an Internet of Things communication network; a facial state multi-scale semantic feature extraction module 130, which is used to perform multi-scale semantic feature mining on the facial image in the background monitoring server to obtain a facial state multi-scale semantic feature vector; a heart rate time series association coding module 140, which is used to perform time series association coding on the time series of the heart rate value in the background monitoring server to obtain a heart rate time series association implicit feature vector; a cross-modal joint feature analysis module 150, which is used to determine whether to generate an early warning prompt signal based on the cross-modal joint features of the facial state multi-scale semantic feature vector and the heart rate time series association implicit feature vector in the background monitoring server.
[0031] In the above-mentioned dynamic abnormal emotion early warning monitoring system 100 based on the Internet of Things, the physiological data acquisition module 110 is used to obtain the facial image of the monitored object collected by the camera, and the time series of the heart rate value of the monitored object collected by the wearable device. It should be understood that the facial image and the heart rate data represent different aspects of emotional expression. Specifically, the facial image provides visual clues, such as expressions, eyes, muscle movements, etc., which are important components of emotional expression. The heart rate data provides physiological clues and reflects the impact of emotional changes on the autonomic nervous system. However, facial images and heart rate data each have advantages and limitations in emotion recognition. Among them, facial images are easily disturbed by external factors such as lighting, angles, and occlusions, while heart rate data may be affected by individual differences, physical activity levels, and other factors. Based on this, in the technical solution of the present application, by collecting the facial image of the monitored object and the time series of the heart rate value, the facial image data and the heart rate data of two different modalities are combined, and the two complement each other, thereby reducing the limitations of a single data source and improving the accuracy of emotion recognition.
[0032] In the above-mentioned dynamic abnormal emotion early warning monitoring system 100 based on the Internet of Things, the Internet of Things communication transmission module 120 is used to transmit the time series of the face image and the heart rate value to the background monitoring server through the Internet of Things communication network. It should be understood that, considering the security, reliability and real-time requirements of data transmission, in the technical solution of the present application, the Internet of Things communication network is used as a channel for data transmission, and the time series of the face image and the heart rate value are transmitted to the background monitoring server for centralized processing. Among them, the Internet of Things communication network has a wide coverage range and high-speed data transmission capability, and can transmit the face image and heart rate data to the background monitoring server safely, reliably and in real time for emotion type recognition. The background monitoring server is a large server with remote monitoring and data processing functions, equipped with powerful computing power and storage resources, and can centrally process and analyze a large number of data streams from multiple cameras and wearable devices, so that administrators or relevant personnel can view and analyze data by remotely accessing the server, realizing remote monitoring and management across regions and time.
[0033] In the above-mentioned dynamic abnormal emotion early warning monitoring system 100 based on the Internet of Things, the face state multi-scale semantic feature extraction module 130 is used to perform multi-scale semantic feature mining on the face image in the background monitoring server to obtain a face state multi-scale semantic feature vector. Figure 3 FIG. 1 is a block diagram of a multi-scale semantic feature extraction module for facial state in a dynamic abnormal emotion early warning monitoring system based on the Internet of Things according to an embodiment of the present application. Figure 3As shown, the face state multi-scale semantic feature extraction module 130 includes: a face adaptive feature extraction unit 131, used to pass the face image through a face adaptive feature extractor based on the GoogLeNet model to obtain a face state feature map; a multi-scale semantic encoding unit 132, used to input the face state feature map into the multi-scale semantic feature extractor to obtain the face state multi-scale semantic feature vector.
[0034] Specifically, the face adaptive feature extraction unit 131 is used to pass the face image through a face adaptive feature extractor based on the GoogLeNet model to obtain a face state feature map. It should be understood that in order to extract effective features that can reflect the emotional state from the face image, in the technical solution of the present application, a face adaptive feature extractor based on the GoogLeNet model is used to process the face image. Specifically, the GoogLeNet model is an efficient deep learning network structure with powerful feature learning capabilities. Through training and optimization, the face adaptive feature extractor based on the GoogLeNet model can learn the key features related to emotional expression in the face image, such as facial contours, eyes, facial muscle movements, etc., thereby providing important basic data support for the emotional state recognition of the monitored object.
[0035] Specifically, the multi-scale semantic encoding unit 132 is used to input the facial state feature map into a multi-scale semantic feature extractor to obtain the facial state multi-scale semantic feature vector. It should be understood that, considering that in the facial image, the expression characteristics of emotions are usually distributed at different scales and levels, such as subtle facial muscle movements, overall facial contours and expressions, etc. Therefore, in order to more fully extract the semantic features of the facial image, thereby more comprehensively reflecting the emotional state of the monitored object, in the technical solution of the present application, a multi-scale semantic feature extractor is further used to process the facial state feature map. Among them, the multi-scale semantic feature extractor adopts convolution and pooling operations at different scales to extract the global semantic features and local semantic features of the face state feature map, and mines the overall facial contour and expression state of the face, as well as the local scale muscle movement, deformation of the eyes and mouth, etc., and comprehensively represents the macroscopic features of the emotional state of the monitored object and the subtle changes in emotional expression by fusing the global semantic features and local semantic features of the face state feature map, so as to improve the understanding of the emotional state of the monitored object, thereby providing richer and more accurate information basis for subsequent emotion recognition.
[0036] Figure 4 FIG. 1 is a block diagram of a multi-scale semantic coding unit in a dynamic abnormal emotion early warning monitoring system based on the Internet of Things according to an embodiment of the present application. Figure 4As shown, the multi-scale semantic encoding unit 132 includes: a global semantic feature extraction subunit 1321, used to extract the global semantic features of the face state feature map to obtain a face state global semantic feature vector; a local semantic feature extraction subunit 1322, used to extract the local semantic features of the face state feature map to obtain a face state local semantic feature vector; a multi-scale semantic feature fusion subunit 1323, used to fuse the face state global semantic feature vector and the face state local semantic feature vector to obtain the face state multi-scale semantic feature vector.
[0037] More specifically, the global semantic feature extraction subunit 1321 is used to: upsample the face state feature map to obtain an upsampled face state feature map; perform global mean pooling on the upsampled face state feature map to obtain an upsampled face state semantic channel feature vector; perform point convolution encoding on the upsampled face state semantic channel feature vector to obtain the face state global semantic feature vector.
[0038] More specifically, the local semantic feature extraction subunit 1322 is used to: upsample the facial state feature map to obtain an upsampled facial state feature map; perform point convolution encoding on the upsampled facial state feature map to obtain a channel-modulated upsampled facial state feature map; perform two-dimensional convolution encoding on the channel-modulated upsampled facial state feature map to obtain the facial state local semantic feature vector.
[0039] In the above-mentioned dynamic abnormal emotion early warning monitoring system 100 based on the Internet of Things, the heart rate time series association coding module 140 is used to perform time series association coding on the time series of the heart rate value in the background monitoring server to obtain the heart rate time series association implicit feature vector. In a specific example of the present application, the encoding method of performing time series association coding on the time series of the heart rate value to obtain the heart rate time series association implicit feature vector is to input the time series of the heart rate value into a heart rate time series encoder based on a bidirectional long short-term memory neural network model to obtain the heart rate time series association implicit feature vector. It should be understood that considering that the heart rate data is time series data and has time series association, that is, the current heart rate value is often affected by the heart rate value in the previous period of time. Therefore, in order to effectively extract the time series dynamic change pattern in the heart rate data, in the technical solution of the present application, a heart rate time series encoder based on a bidirectional long short-term memory neural network model is used to process the time series of the heart rate value. Specifically, the bidirectional long short-term memory neural network model has a powerful time series modeling capability. It introduces gating mechanisms (such as forget gate, input gate and output gate) to learn and memorize the time series dependencies in the time series of the heart rate values, and extracts the time series correlation implicit features of the heart rate data, thereby reflecting the changing trends and fluctuation patterns of the emotional state of the monitored object, providing an important physiological signal basis for emotion recognition.
[0040] In the above-mentioned dynamic abnormal emotion early warning monitoring system 100 based on the Internet of Things, the cross-modal joint feature analysis module 150 is used to determine whether to generate an early warning prompt signal based on the cross-modal joint features of the multi-scale semantic feature vector of the face state and the heart rate time series associated implicit feature vector in the background monitoring server. Figure 5 FIG. 1 is a block diagram of a cross-modal joint feature analysis module in a dynamic abnormal emotion early warning monitoring system based on the Internet of Things according to an embodiment of the present application. Figure 5 As shown, the cross-modal joint feature analysis module 150 includes: a cross-modal joint encoding unit 151, which is used to perform cross-modal joint encoding on the multi-scale semantic feature vector of the facial state and the heart rate time series associated implicit feature vector to obtain a heart rate-face image multi-modal implicit associated feature matrix; an emotion category recognition unit 152, which is used to pass the heart rate-face image multi-modal implicit associated feature matrix through a classifier-based emotion recognizer to obtain an emotion recognition result, and the emotion recognition result is used to represent the emotion category label; an abnormal emotion warning unit 153, which is used to generate the warning prompt signal in response to the emotion recognition result belonging to the abnormal emotion class.
[0041] Specifically, the cross-modal joint encoding unit 151 is used to perform cross-modal joint encoding on the multi-scale semantic feature vector of the face state and the implicit feature vector associated with the heart rate time series to obtain the heart rate-face image multi-modal implicit association feature matrix. It should be understood that by cross-modally combining the multi-scale semantic feature vector of the face state and the implicit feature vector associated with the heart rate time series, the information of two different modes of face image and heart rate data can be comprehensively utilized to overcome the limitations of single modality data and improve the accuracy and robustness of emotion recognition. In a specific example of the present application, the encoding method for cross-modal joint encoding of the multi-scale semantic feature vector of the face state and the implicit feature vector associated with the heart rate time series is to input the multi-scale semantic feature vector of the face state and the implicit feature vector associated with the heart rate time series into a cross-modal feature joint encoder based on the Clip model to obtain the heart rate-face image multi-modal implicit association feature matrix. Among them, the cross-modal feature joint encoder based on the Clip model is a key component for realizing the deep fusion of face image and heart rate data. Specifically, the Clip model, with its powerful cross-modal learning ability, maps the multi-scale semantic feature vector of the facial state and the heart rate time series associated implicit feature vector to the same feature space, so that the two are correlated and complementary to each other in the feature space, effectively capturing the intrinsic connections and correlations between data of different modalities, thereby fully considering visual clues such as expressions, eye contact, and muscle movements in facial images, as well as physiological clues such as the impact of emotional changes reflected in heart rate data on the autonomic nervous system, generating multimodal implicit associated features with rich emotional expression information, overcoming the limitations of a single data source in emotion recognition, and improving the accuracy of emotion recognition.
[0042] Specifically, the emotion category recognition unit 152 is used to pass the heart rate-face image multimodal implicit correlation feature matrix through a classifier-based emotion recognizer to obtain an emotion recognition result, and the emotion recognition result is used to represent the emotion category label. It should be understood that the classifier-based emotion recognizer is trained with a large amount of labeled data and can learn the mapping relationship from input features to emotion categories. In the technical solution of the present application, by inputting the heart rate-face image multimodal implicit correlation feature matrix into the classifier-based emotion recognizer, the discriminant ability of the classifier can be used to perform feature learning and classification mapping on the heart rate-face image multimodal implicit correlation feature matrix, thereby outputting corresponding emotion category labels, such as happiness, sadness, anger, surprise, fear, etc., to achieve intuitive expression of the emotional state of the monitored object.
[0043] Specifically, the abnormal emotion warning unit 153 is used to generate the warning prompt signal in response to the emotion recognition result belonging to the abnormal emotion class. That is, the emotion recognition result is divided into a normal emotion class and an abnormal emotion class. When the emotion recognition result belongs to the abnormal emotion class, a corresponding warning prompt is automatically generated so that the administrator or relevant personnel can take emotion management measures in time to ensure the mental health and emotional stability of the monitored object.
[0044] It should be understood that before using the above neural network model, it needs to be trained. That is to say, in the dynamic abnormal emotion early warning monitoring system based on the Internet of Things of the present application, it also includes a training module for training the face adaptive feature extractor based on the GoogLeNet model, the multi-scale semantic feature extractor, the heart rate time series encoder based on the bidirectional long short-term memory neural network model, the cross-modal feature joint encoder based on the Clip model and the emotion recognizer based on the classifier.
[0045] Figure 6 FIG. 1 is a block diagram of a training module in a dynamic abnormal emotion early warning monitoring system based on the Internet of Things according to an embodiment of the present application. Figure 6As shown, the training module 200 includes: a training data acquisition unit 210, which is used to acquire training data, wherein the training data includes a training face image of a monitored object collected by a camera and a time series of training heart rate values of the monitored object collected by a wearable device, as well as a true value of an emotion category label; a training data transmission unit 220, which is used to transmit the training face image and the time series of training heart rate values to a background monitoring server via an Internet of Things communication network; a training face image feature extraction unit 230, which is used to extract the training face image through the face adaptive feature extraction unit based on the GoogLeNet model. extractor to obtain a training face state feature map; a training multi-scale semantic feature extraction unit 240, used to input the training face state feature map into the multi-scale semantic feature extractor to obtain a training face state multi-scale semantic feature vector; a training heart rate value time series encoding unit 250, used to input the time series of the training heart rate value into the heart rate time series encoder based on the bidirectional long short-term memory neural network model to obtain a training heart rate time series associated implicit feature vector; a training data joint encoding unit 260, used to input the training face state multi-scale semantic feature vector and the training heart rate time series associated implicit feature vector into the C based l ip model's cross-modal feature joint encoder is used to obtain a training heart rate-face image multi-modal implicit correlation feature matrix; a classification loss unit 270 is used to pass the training heart rate-face image multi-modal implicit correlation feature matrix through the classifier-based emotion recognizer to obtain a class probability value and a classification loss function value; a multi-distribution noise loss unit 280 is used to calculate a multi-distribution noise loss function value based on the class probability value; a model training unit 290 is used to train the GoogLeNet model-based face adaptive feature extractor, the multi-scale semantic feature extractor, the heart rate time series encoder based on the bidirectional long short-term memory neural network model, the C l ip model-based cross-modal feature joint encoder and the classifier-based emotion recognizer using the weighted sum of the classification loss function value and the multi-distribution noise loss function value as the loss function value.
[0046] In a preferred example of the present application, when updating the model parameters through gradient back propagation based on the classification loss function value, a multi-distribution noise loss function is further introduced, and the calculation of the multi-distribution noise loss function includes the following steps: obtaining the mean of the probability values corresponding to each emotion category label when passing the training heart rate-face image multimodal implicit association feature matrix through the classifier-based emotion recognizer as the class probability value of the classifier, and calculating the absolute value of the difference between the class probability value and one to obtain the probability difference; determining a feature set composed of all eigenvalues of the training heart rate-face image multimodal implicit association feature matrix The method comprises the following steps: determining a first product of each eigenvalue in the training heart rate-face image multimodal implicit association feature matrix and the class probability value, and a second product of the probability difference value and the feature mean value; summing the squares of the difference between the first product and the second product corresponding to each eigenvalue in the training heart rate-face image multimodal implicit association feature matrix and dividing the sum by the scale of the training heart rate-face image multimodal implicit association feature matrix to obtain a division result; and subtracting the product of the sum of the squares of the class probability value and the probability difference value and the feature variance from the division result to obtain a multi-distribution noise loss function value.
[0047] Here, for the temporal correlation features of the training heart rate values and the multi-scale image semantic features of the training face images in the cross-modal feature correlation domain, the correlation differences between the distributions of different modal features along their respective dimensions lead to the out-of-distribution noise of the cross-modal correlation feature representation set, resulting in the class distribution noise of the training heart rate-face image multimodal implicit correlation feature matrix as a whole, thereby affecting the accuracy of the classification results. The above-mentioned multi-distribution noise loss function sets the approximate variance expectation characteristics of the out-of-distribution noise of the feature set of the training heart rate-face image multimodal implicit correlation feature matrix relative to the Bernoulli sampling instance of the feature set based on the class probability distribution characteristics expressed by the class probability values and probability differences of the training heart rate-face image multimodal implicit correlation feature matrix through the classifier.
[0048] Therefore, by using the multi-distribution noise loss function to measure the distribution feature noise based on the class probability distribution characteristics expressed by the class probability value and the probability difference, the statistical noise distribution can be used. Based on the probabilistic cross-random selection characteristics of the expected variance for each eigenvalue of the training heart rate-face image multimodal implicit association feature matrix in the classification process, the possible sampling noise characteristics can be degraded, thereby achieving class distribution noise reduction of self-supervised learning in the process of updating model parameters to improve the accuracy of the classification results.
[0049] In summary, the dynamic abnormal emotion early warning monitoring system based on the Internet of Things according to the embodiment of the present application is explained, which uses artificial intelligence technology based on deep learning to monitor and analyze the facial images and heart rate data of the monitored object, extracts the multi-scale semantic features of the facial images, and the temporal dynamic change pattern of the heart rate data, and intelligently identifies the emotional category of the monitored object based on the cross-modal joint features between the two, thereby generating early warning prompts for the abnormal emotions of the monitored object. In this way, comprehensive, accurate, and real-time monitoring and early warning of individual emotional states can be achieved, effectively improving the efficiency and quality of emotional monitoring management.
[0050] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details of the above embodiments are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.
[0051] In the above embodiments, the description of each embodiment has its own emphasis. For the parts that are not described or recorded in detail in a certain embodiment, please refer to the relevant description of other embodiments. In the several embodiments provided by the present invention, it should be understood that the disclosed system can be implemented in other ways. For example, the system embodiment described above is only schematic. For example, the module division is only a logical function division, and there may be other division methods in actual implementation. The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment.
[0052] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0053] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference to a figure in a claim should not be considered as limiting the claim to which it relates.
[0054] In addition, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The multiple units stated in the system claims can also be implemented by one unit through software or hardware.
[0055] Finally, it should be noted that the above description has been given for the purpose of illustration and description. In addition, the above embodiments are only used to illustrate the technical solution of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.
Claims
1. A dynamic abnormal emotion early warning monitoring system based on the Internet of Things, It is characterized in that include: A physiological data acquisition module, used to acquire a facial image of a monitored object acquired by a camera, and a time series of heart rate values of the monitored object acquired by a wearable device; An Internet of Things communication transmission module, used to transmit the time series of the face image and the heart rate value to a background monitoring server via an Internet of Things communication network; A multi-scale semantic feature extraction module for face status, used to perform multi-scale semantic feature mining on the face image in the background monitoring server to obtain a multi-scale semantic feature vector for the face status; A heart rate time series correlation coding module is used to perform time series correlation coding on the time series of the heart rate value in the background monitoring server to obtain a heart rate time series correlation implicit feature vector; A cross-modal joint feature analysis module, used to determine whether to generate a warning prompt signal on the background monitoring server based on the cross-modal joint features of the multi-scale semantic feature vector of the face state and the heart rate time series associated implicit feature vector; It also includes training modules for training a face adaptive feature extractor based on the GoogLeNet model, a multi-scale semantic feature extractor, a heart rate time series encoder based on a bidirectional long short-term memory neural network model, a cross-modal feature joint encoder based on the Clip model, and a classifier-based emotion recognizer; Wherein, the training module includes: A training data acquisition unit, used to acquire training data, wherein the training data includes a training face image of a monitored object collected by a camera and a time series of training heart rate values of the monitored object collected by a wearable device, and a true value of an emotion category label; A training data transmission unit, used for transmitting the time series of the training face image and the training heart rate value to a background monitoring server through an Internet of Things communication network; A training face image feature extraction unit, used for passing the training face image through the face adaptive feature extractor based on the GoogLeNet model to obtain a training face state feature map; A training multi-scale semantic feature extraction unit, used for inputting the training face state feature map into the multi-scale semantic feature extractor to obtain a training face state multi-scale semantic feature vector; A training heart rate value time series encoding unit, used for inputting the time series of the training heart rate value into the heart rate time series encoder based on the bidirectional long short-term memory neural network model to obtain an implicit feature vector associated with the training heart rate time series; A training data joint encoding unit, used for inputting the training face state multi-scale semantic feature vector and the training heart rate time series associated implicit feature vector into the Clip model-based cross-modal feature joint encoder to obtain a training heart rate-face image multi-modal implicit associated feature matrix; A classification loss unit, used for passing the training heart rate-face image multimodal implicit correlation feature matrix through the classifier-based emotion recognizer to obtain a class probability value and a classification loss function value; A multi-distribution noise loss unit, used to calculate a multi-distribution noise loss function value based on the class probability value; A model training unit, used for training the face adaptive feature extractor based on the GoogLeNet model, the multi-scale semantic feature extractor, the heart rate time series encoder based on the bidirectional long short-term memory neural network model, the cross-modal feature joint encoder based on the Clip model, and the classifier-based emotion recognizer by taking the weighted sum of the classification loss function value and the multi-distribution noise loss function value as the loss function value; The calculation of the multi-distribution noise loss function includes the following steps: obtaining the mean of the probability values corresponding to each emotion category label when the training heart rate-face image multimodal implicit association feature matrix passes through the classifier-based emotion recognizer as the class probability value of the classifier, and calculating the absolute value of the difference between the class probability value and one to obtain the probability difference; determining the feature mean and feature variance of the feature set composed of all eigenvalues of the training heart rate-face image multimodal implicit association feature matrix; determining .... The method comprises the following steps: calculating a first product of each eigenvalue in the training heart rate-face image multimodal implicit association feature matrix and the class probability value, and a second product of the probability difference value and the feature mean value; summing the squares of the difference between the first product and the second product corresponding to each eigenvalue in the training heart rate-face image multimodal implicit association feature matrix and dividing the sum by the scale of the training heart rate-face image multimodal implicit association feature matrix to obtain a division result; and subtracting the product of the sum of the squares of the class probability value and the probability difference value and the feature variance from the division result to obtain a multi-distribution noise loss function value.
2. According to claim 1, the dynamic abnormal emotion early warning monitoring system based on the Internet of Things, It is characterized in that The face state multi-scale semantic feature extraction module comprises: A face adaptive feature extraction unit, used for passing the face image through a face adaptive feature extractor based on a GoogLeNet model to obtain a face state feature map; The multi-scale semantic encoding unit is used to input the face state feature map into the multi-scale semantic feature extractor to obtain the face state multi-scale semantic feature vector.
3. According to claim 2, the dynamic abnormal emotion early warning monitoring system based on the Internet of Things, It is characterized in that The multi-scale semantic encoding unit comprises: A global semantic feature extraction subunit, used for extracting the global semantic features of the face state feature map to obtain a face state global semantic feature vector; A local semantic feature extraction subunit, used for extracting local semantic features of the face state feature map to obtain a face state local semantic feature vector; The multi-scale semantic feature fusion subunit is used to fuse the face state global semantic feature vector and the face state local semantic feature vector to obtain the face state multi-scale semantic feature vector.
4. According to claim 3, the dynamic abnormal emotion early warning monitoring system based on the Internet of Things, It is characterized in that The global semantic feature extraction subunit is used to: Upsampling the face state feature map to obtain an upsampled face state feature map; Performing global mean pooling processing on the upsampled face state feature map to obtain an upsampled face state semantic channel feature vector; Point convolution encoding is performed on the upsampled face state semantic channel feature vector to obtain the face state global semantic feature vector.
5. According to claim 4, the dynamic abnormal emotion early warning monitoring system based on the Internet of Things, It is characterized in that The local semantic feature extraction subunit is used to: Upsampling the face state feature map to obtain an upsampled face state feature map; Performing point convolution encoding on the upsampled face state feature map to obtain a channel modulated upsampled face state feature map; Perform two-dimensional convolution coding on the channel-modulated upsampled face state feature map to obtain the face state local semantic feature vector.
6. According to claim 5, the dynamic abnormal emotion early warning monitoring system based on the Internet of Things, It is characterized in that The heart rate timing association coding module is used to: The time series of the heart rate values is input into a heart rate time series encoder based on a bidirectional long short-term memory neural network model to obtain the heart rate time series associated implicit feature vector.
7. According to claim 6, the dynamic abnormal emotion early warning monitoring system based on the Internet of Things, It is characterized in that The cross-modal joint feature analysis module includes: A cross-modal joint encoding unit, used for performing cross-modal joint encoding on the multi-scale semantic feature vector of the face state and the heart rate time series associated implicit feature vector to obtain a heart rate-face image multi-modal implicit associated feature matrix; An emotion category recognition unit, used for passing the heart rate-face image multimodal implicit correlation feature matrix through a classifier-based emotion recognizer to obtain an emotion recognition result, wherein the emotion recognition result is used to represent an emotion category label; The abnormal emotion warning unit is used to generate the warning prompt signal in response to the emotion recognition result belonging to the abnormal emotion class.
8. According to claim 7, the dynamic abnormal emotion early warning monitoring system based on the Internet of Things, It is characterized in that The cross-modal joint encoding unit is used to: The multi-scale semantic feature vector of the facial state and the heart rate temporal correlation implicit feature vector are input into a cross-modal feature joint encoder based on the Clip model to obtain the heart rate-face image multi-modal implicit correlation feature matrix.
