Personnel emotion intelligent monitoring system

Abstract

The invention provides a personnel emotion intelligent monitoring system. The personnel emotion intelligent monitoring system comprises an acquisition module, an image preprocessing module, a multi-modal feature fusion module and an emotional state visualization module. According to different adaptability of different wavebands to ambient light, by analyzing wide line features in images of different wavebands, key areas of the face are accurately positioned, important space information is provided for emotion recognition, meanwhile, image data of different wavebands are preprocessed, the wavebands with qualified quality and rich information are screened out, and the emotion recognition accuracy is improved. And the accuracy of subsequent emotion recognition of the system is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of visual recognition, and in particular to an intelligent monitoring system for human emotions. Background Art

[0002] In the current regulatory field, emotion monitoring is crucial to ensuring the mental health of personnel and improving work efficiency. For example, in actual applications, when people face high-pressure situations, their facial expressions and physiological indicators will undergo subtle but critical changes.

[0003] However, existing emotion monitoring technologies are often not sensitive and accurate enough when dealing with complex emotion changes, especially in real-time monitoring and multimodal data processing. For example, patent document CN114707530A discloses a dual-modal emotion recognition method and system based on multi-source signals and neural networks, and proposes to realize emotion recognition by inputting continuous picture frames of the eyes and mouth into the model, which reduces the amount of model calculation while retaining the main emotional information of the face, solving the problem that contact-type acquisition is prone to cause discomfort to the human body and has poor robustness.

[0004] However, from the technical solution of the patent document, the PPG signal detected by it is greatly affected by the change of the light in the detection environment. Specifically, it uses a detrending method based on L1 trend filtering and a phase tracking algorithm to reduce the impact of sudden changes in light intensity on the PPG signal. However, under extreme light conditions, this method still cannot completely eliminate the problem of signal distortion. ① Under direct sunlight or backlight conditions, the camera equipment may capture images with overexposure or loss of facial details. At this time, the PPG signal will be distorted due to the strong reflection of light or the loss of facial features; ② In an environment with rapid changes in light, such as entering indoors from outdoors, or under strobe lights, the images captured by the camera are usually accompanied by flickering or changes in brightness, which will affect the stability of the PPG signal; ③ In an environment with multiple light sources, such as a shopping mall or a staff duty office, light of different directions and intensities will produce complex shadows and reflections on the face of the staff, interfering with the quality of the PPG signal. In summary, the above-mentioned method means that in an environment with complex or drastic changes in light conditions, the accuracy of emotion recognition will be affected, which will lead to insufficient and inaccurate monitoring of emotional states.

[0005] Based on this, the present invention proposes a method of pre-processing image data of different bands by utilizing the different adaptability of different bands to ambient light, and screening out bands with qualified quality and rich information through image quality evaluation and image enhancement, thereby improving the accuracy of subsequent emotion recognition. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention aims to provide an intelligent monitoring system for people's emotions, which utilizes the different adaptability of different bands to ambient light, analyzes the broad line features (such as facial contours, eyes, nose, etc.) in images of different bands, accurately locates the key areas of the face, and provides important spatial information for emotion recognition. At the same time, the image data of different bands are preprocessed to screen out bands with qualified quality and rich information, thereby improving the accuracy of subsequent emotion recognition. This is to solve the problems raised in the above background technology.

[0007] In order to achieve the above object, the present invention is implemented through the following technical solutions: a personnel emotion intelligent monitoring system, comprising Acquisition module, image preprocessing module, multimodal feature fusion module and emotional state visualization module, among which, The acquisition module samples the video based on the time scale rolling, and uses the set frame difference threshold T scene To compare the differences between consecutive sampled video frames, identify changes in the sampling scene, and select frames near the change point as key individual frame data of static image data representing the facial expression of the target person; and synchronize the preset monitoring equipment, establish a time scale sequence, and collect the physiological signal data of the target person at different time steps S; The image preprocessing module includes an image quality evaluation unit connected to the acquisition module, which is used to evaluate the quality of the acquired static image data to obtain static image data with qualified quality, including the steps of: S1-1. Construct a four-band static image dataset based on the collected static image data, and set the pixel value constraint range of each band in the four-band static image dataset to 0 to 255. Select a local area containing 3 rows and 3 columns, with a total of 9 pixels, from each acquired static image data, and record it as [A1, A2, A3; B1, B2, B3; C1, C2, C3], so as to capture the local changes of the static image data, and only extract the pixel values ​​of this area; combine the extracted pixel values ​​into a four-band static image dataset, with each pixel value as a 4-dimensional feature vector; S1-2, respectively setting a recognizability index D for each band in the four-band static image data set, and sorting the bands 1 to 4, selecting candidate feature bands, and further evaluating the clarity, contrast, and noise of the acquired static image data to achieve the best recognition effect; S1-3, according to the brightness information of the candidate characteristic band, select the pixel point with the best brightness, and determine the band with the highest recognizability by comparing the maximum brightness pixel position of the grayscale images of different bands on the vertical coordinate, generate a channel image with clear visual effect, and obtain static image data of qualified quality; The multimodal feature fusion module is connected to the image preprocessing module, and analyzes the temporal and spatial characteristics of the facial expressions of the target person by creating a multimodal temporal information fusion model, fusing the sampled video and physiological signal data, so as to identify and understand the emotional changes of the target person on different time scales; The emotional state visualization module is connected to the multimodal feature fusion module and is used to visualize the emotional changes of the target person on different time scales predicted by the multimodal feature fusion module to the monitoring personnel or the system for decision analysis.

[0008] Compared with the prior art, the present invention has the following beneficial effects: In order to solve the problem in the prior art that the accuracy of emotion recognition is affected in an environment with complex or drastic light conditions, the present invention firstly utilizes the different adaptability of different bands to ambient light, such as utilizing the characteristics that near-infrared band images can still provide clear images under low light conditions, while visible light band images are greatly affected by ambient light, and pre-processes the four-band image data through the set image quality evaluation unit and image enhancement unit, screens out bands with qualified quality and rich information in the early stage, and enhances the images of these bands to improve the accuracy of subsequent emotion recognition; secondly, using the pixel values ​​corresponding to each band in the image-enhanced four-band static image data set as input, using the pre-trained I3D network to extract spatiotemporal features of key frames in the video, so that the present invention can capture dynamic changes in the video and provide time sequence information for emotion recognition; thirdly, by constructing a multimodal feature fusion module and combining video and physiological signal data, the present invention can analyze the emotional changes of the target object in real time, provide a more comprehensive emotional state analysis, and thus provide more accurate emotional monitoring support for industry regulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The disclosure of the present invention is described with reference to the accompanying drawings. It should be understood that the drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In the drawings, the same reference numerals are used to refer to the same components. Among them: Figure 1 A schematic diagram of the structure of an emotional intelligence monitoring system proposed in one embodiment of the present invention; Figure 2 It is a schematic diagram of enhancing the image of the wide line feature when the matching degree between the wide line feature of the channel image and the dark wide line area is the lowest according to an embodiment of the present invention; Figure 3 A schematic diagram of the process of the method for intelligently monitoring human emotions proposed in one embodiment of the present invention; Figure 4The figure is a schematic diagram of the arrangement of pixel values ​​in a 3*3 region when performing statistical measurement of quality indicators of each band according to an embodiment of the present invention. DETAILED DESCRIPTION

[0010] It is easy to understand that according to the technical solution of the present invention, without changing the essential spirit of the present invention, a person skilled in the art can propose a variety of interchangeable structural modes and implementation modes. Therefore, the following specific implementation modes and drawings are only exemplary descriptions of the technical solution of the present invention, and should not be regarded as the whole of the present invention or as a limitation or restriction to the technical solution of the present invention.

[0011] The present invention is further described in detail below with reference to the accompanying drawings, but is not intended to limit the present invention.

[0012] As an understanding of the technical concept and implementation principle of the present invention, the means of monitoring personnel emotions in the prior art are: a detrending method based on L1 trend filtering and a phase tracking algorithm are adopted, but when the control personnel are on duty at night, if the indoor light is insufficient, the useful signal intensity in the PPG signal becomes less significant relative to the background noise, thereby reducing the signal-to-noise ratio, making it difficult for the system to accurately capture the facial blood flow signal of the control personnel, thereby affecting the accuracy of emotion recognition; for example, under strong lighting or backlight conditions, the camera sensor will perform dynamic range compression on high-brightness areas to avoid overexposure, which leads to the loss of facial details of the control personnel, especially in areas with high reflectivity. This compression will reduce the dynamic range of the PPG signal, causing the PPG signal extracted from the video to be distorted, making it difficult for the emotion recognition system to extract accurate PPG signals from the video; for example, in an environment with multiple light sources, such as a shopping mall or a duty office for personnel, light of different directions and intensities will produce complex shadows and reflections on the faces of personnel, interfering with the quality of the PPG signal.

[0013] To this end, the present invention proposes an intelligent monitoring system for personnel emotions, the purpose of which is to capture subtle changes in facial expressions from multiple angles by combining image data in four different bands: near infrared, shortwave infrared, visible red area, and visible blue area. At the same time, an image quality evaluation unit S101 is constructed to perform quality evaluation on the collected images to ensure that only high-quality image data is used for subsequent emotion analysis, thereby improving the accuracy of emotion recognition and facilitating a more comprehensive understanding and analysis of emotional states.

[0014] When implementing it, Figure 1 As shown, as an embodiment of the present invention, a human emotion intelligent monitoring system is proposed, comprising: Acquisition module S0: includes image data acquisition unit S001 and physiological signal data acquisition unit S002; The image data acquisition unit S001 is used to collect facial expression and physiological signal data of personnel. The specific steps include: S0-1, extracting key individual frame data from the sampled video based on time scale rolling as static image data representing the facial expression of the target person; wherein, The key single frame data is extracted by setting the frame difference threshold T scene Compare the differences between consecutive sampled video frames, denoted as (1), Where ξ(φ) represents the difference between the φth frame of the sampled video frame and the previous frame; They are represented as the pixel values ​​at the coordinates (x, y) of the φth frame and the φ-1th frame respectively; in specific implementation, If the difference ξ(φ) between the upper and lower frames of the sampled video exceeds the frame difference threshold T scene , it is considered that the scene of the current target person’s facial expression has changed. Once a scene change is detected, the video frame in the area of ​​the scene change point is selected as the key individual frame data, and the frame difference threshold T scene Set according to video content and application requirements.

[0015] It can be understood that in the actual sampled video, key individual frame data is extracted by setting a frame difference threshold. Specifically, by comparing the differences between continuously sampled video frames, when the difference between the upper and lower frames exceeds the set frame difference threshold, it is considered that the facial expression scene of the person has changed. In this case, the video frames near the scene change point are selected as key individual frame data. This extraction method can capture important moments in the video, thereby more accurately monitoring and analyzing the emotional state of the person.

[0016] The physiological signal data acquisition unit S002 is used to collect physiological signal data. The specific steps include: S0-2, preset monitoring equipment, and establish a time scale sequence to obtain a data set of physiological signals of the target person corresponding to each time step S in the time scale sequence; the specific steps include: S0-21. Based on the heart rate monitoring device, obtain the target person's continuous heart rate data H R : (2), In the formula, H Ravg It is expressed as the average heart rate of the target person monitored in time step S under the time scale sequence; Based on the skin conductance sensor, the skin conductance data S representing the target person's emotional arousal level is obtained. C :S Cmax =max(S C1 ,S C2 ,...,SCS )(3) In the formula, S Cmax It is expressed as the maximum skin conductance value of the target person monitored within the time step S under the time scale sequence.

[0017] In one embodiment of the present invention, the proposed emotional intelligence monitoring system further includes an image preprocessing module S1: including an image quality evaluation unit S101 and an image enhancement unit S201 respectively connected to the acquisition module S0. It can be understood that the image quality evaluation unit S101 is used to evaluate the quality of the acquired static image data to obtain static image data of qualified quality. Figure 4 As shown, the specific steps include: S1-1. Based on the acquired static image data, a four-band static image dataset is constructed. It can be understood that the purpose of using four-band images is to take advantage of the advantages of different band images (different bands have different adaptability to ambient light) to capture and analyze subtle changes in the facial expressions of the target person, so as to perform emotion recognition more accurately.

[0018] S1-11. The band represented by the near-infrared static image data representing the facial expression of the target person in the custom four-band static image data set is band 1; the band represented by the short-wave infrared static image data is band 2; the band represented by the visible light red area static image data is band 3; the band represented by the visible light blue area static image data is band 4; and the pixel value constraint range of each band is set to 0 to 255; S1-12. Select a local area consisting of 3 rows and 3 columns, with a total of 9 pixels, [A1, A2, A3; B1, B2, B3; C1, C2, C3] from each static image data obtained to capture local changes in the static image data, and only extract the pixel values ​​of this area; and the purpose of selecting the local area consisting of 3 rows and 3 columns, with a total of 9 pixels is that in facial expression analysis, some local areas (such as eyes and mouth) contain more information about emotional states, so selecting these areas can provide more accurate emotion analysis.

[0019] Example: The pixel values ​​of band 1 represented by the near-infrared static image data are: ,

[0020] The pixel values ​​of band 2 represented by the shortwave infrared static image data are: ,

[0021] The pixel values ​​of band 3 represented by the static image data of the visible light red region are: ,

[0022] The pixel values ​​of band 4 represented by the static image data in the visible light blue region are: ,

[0023] S1-13, combining the extracted pixel values ​​into a four-band static image data set, where each pixel value is a 4-dimensional feature vector; for example, when the pixel point at position (1,1) in band 1 is selected, the four-band static image data set is: [120, 35, 85, 70]; S1-14, looping steps S1-12 and S1-13 to construct a complete four-band static image dataset; S1-2, set a recognizability index D for each band in the four-band static image data set, sort bands 1-4, select high-value bands as candidate feature bands to achieve the best recognition effect, and evaluate the clarity, contrast and noise of the acquired static image data. The high-value band is the band with the highest recognizability. In specific implementation, S1-21, according to the formula (4), based on the calculated variances of band 1, band 2, band 3, and band 4 , respectively calculate the recognizability index D corresponding to band 1, band 2, band 3, and band 4 to perform statistical measurement on the quality index of each band; where σ i 2 It is expressed as the variance of the i-th band, i∈(1,2,3,4): 0.01 is the minimum positive number used to ensure calculation stability; It should be noted that the variance calculation steps for Band 1, Band 2, Band 3, and Band 4 are as follows: first calculate the average of all pixel values ​​in each band, and then divide the sum of the squares of the differences between each pixel value and the average by the total number of pixels minus one to obtain the sample variance.

[0024] For example, when the pixel point at position (1,1) in band 1 is selected, the static image data set of the four bands is: [120, 35, 85, 70], Then the average pixel value of band 1 is calculated as (120+35+85+70) / 4=77.5. Then its variance is: .

[0025] It should be noted that the above variance value is only the pixel value of a single pixel point in band 1 (1,1), and is not the variance calculation of the actual four-band static image dataset. In actual application, the four-band static image dataset will have a large number of pixels, and the variance calculation needs to be based on all these pixel values; In the embodiment of the present invention, in order to facilitate the calculation and understanding, it is assumed that Take 0.06, 0.18, 0.30, and 0.45 respectively; .

[0026] S1-22, candidate feature band selection: D4<D3<D2<D1, it is considered that band 4 has the highest recognizability and is selected as the candidate feature band, indicating that it may contain the most useful information, followed by band 3, band 2 and band 1; S1-3. Based on the obtained candidate feature bands, call the brightness information representing the facial expression of the target person, select the pixel points with the best brightness within the constraint range of 0 to 255 pixel values, and further determine the band with the highest recognizability by comparing the maximum brightness pixel positions of grayscale images of different bands on the vertical coordinate. Finally, generate a channel image with clear visual effects as static image data of qualified quality through the image brightness information method, wherein the calculation formula for the maximum brightness pixel position of grayscale images of different bands on the vertical coordinate is: (5), Where, i∈(1,2,3,4), I ave represents the grayscale image brightness of the static image data of the i-th band; σi represents the standard deviation of the grayscale image of the static image data of the i-th band; x, y represent the horizontal and vertical coordinates of the grayscale image of the static image data of the i-th band; I ave ≤128 means the pixel value of each band is obtained with the best brightness within the constraint range of 0 to 255; max (X) represents the maximum brightness pixel position of brightness y=1,2,.,M on the vertical coordinate of the grayscale image of the static image data of the i-th band; I max The smaller (X) is, the channel image in the acquired candidate feature band is considered to have the highest recognizability.

[0027] As an embodiment of the present invention, the image enhancement unit S201 is used to perform color image enhancement on static image data of qualified quality to improve image quality. The specific steps include: S2-1, using the Retinex algorithm to process each color channel of the obtained quality-qualified static image data, and calculate the initial input quality-qualified static image data CS (x, y) at the maximum brightness pixel position I max The pixel value of (X) is convolved with the Gaussian convolution kernel GS(x,y) to obtain a static image with qualified output quality after being enhanced by the Retinex algorithm. Data, where the Gaussian convolution kernel GS(x,y) is determined by the standard deviation σi and the adjustment coefficient ε to ensure the normalization of the probability density function; S2-2, using isotropic nonlinear filtering technology, by comparing and counting the maximum brightness pixel position Imax The similarity between the brightness of the central pixel at (X) and the surrounding pixels is used to enhance the wide line features in the static image data of qualified quality. The wide line features are such as the collected facial contours of the person, including eyes, nose, ears and eyebrows, which further improve the image quality and achieve the final image enhancement effect. The specific steps for calculating the similarity are: S2-21, respectively presetting the maximum brightness pixel position I max The pixel value coordinates (α0, β0) of (X) and the maximum brightness pixel position I max The coordinates (α, β) of the surrounding pixels at (X) and the brightness contrast threshold τ are used to evaluate the surrounding pixels and the maximum brightness pixel position I max (X) contrast difference between pixels; S2-22, for each surrounding pixel point coordinate (α, β), calculate its relative position to the maximum brightness pixel position I max The brightness difference of the pixel value of (X) at coordinates (α0, β0) ,in, like τ≤brightness contrast threshold τ, then the coordinates of each surrounding pixel point (α, β) are considered to meet the contrast threshold condition; that is, its influence on the filter response η(α0, β0) = 1. Conversely, if there is a brightness change of at least one pair of pixels, τ>brightness contrast threshold τ, then calculate η(α0,β0) to quantify the impact of this significant brightness change on the filter response; S2-23, continue to judge based on S2-22: Based on the obtained quality qualified static image data, if there is at least one pair of surrounding pixel coordinates (α, β), and at this coordinate, the brightness value f(α, β) of the surrounding pixel coordinates (α, β) is equal to the maximum brightness pixel position I max If the absolute difference between the brightness values ​​f(α0, β0) at the pixel value coordinates (α0, β0) of (X) is greater than a preset brightness contrast threshold τ, it indicates that the brightness change in the area is significant, and its impact on the filter response needs to be further calculated. The purpose is to reduce the amount of calculation. In practical applications, by setting the contrast threshold condition, the scope of analysis can be limited, and only the areas with high contrast can be focused on, thereby reducing unnecessary calculations and improving processing speed; (6), Where ∑∑ represents the sum of the coordinates (α, β) of all surrounding pixel points in each color channel of the qualified static image data; η(α0, β0) is a metric value, which represents the degree of matching of a certain wide line feature in the qualified static image data. The larger the η value, the higher the matching degree of the wide line feature.

[0028] Example: Figure 2As shown in the figure, the dotted line represents the dark wide line where the channel image where the visual effect is not clear when capturing the facial expression of the target person in the continuously sampled video frames (such as Figure 2 Middle Region II), Figure 2 The middle area I is the optimal brightness area where the channel image with clear visual effect is located when collecting the facial expression of the target person. It can be understood that when the static image data in the optimal brightness area of ​​area I is collected for the first time, the maximum brightness pixel position I max The surrounding pixel values ​​at (X) and the maximum brightness pixel position I max The central pixel values ​​at (X) are close, so the η value is large, indicating that the channel image with clear visual effect has a higher matching degree of wide line features; on the contrary, when the channel image where the target person’s facial expression is collected is in area II, the visual effect of this channel image is not clear. At this time, the maximum brightness pixel position I max When the central pixel value at (X) is within the dark wide line region, the η value is the smallest, which indicates that the matching degree between the wide line feature of the channel image and the dark wide line region is the lowest. Therefore, the wide line feature in the image is enhanced in the above manner.

[0029] In one embodiment of the present invention, the proposed emotional intelligence monitoring system further includes: The wide line feature extraction module S3, whose input end is connected to the image enhancement unit S201 and whose output end is connected to the multimodal feature fusion module S4, is used to extract the wide line features representing the facial expression of the target person using the pre-trained I3D network to obtain the spatiotemporal feature sequence F of the key individual frame data in the sampled video; the specific steps include: S3-1, taking the pixel value corresponding to each band in the four-band static image data set after color image enhancement as input; Example: If the above four-band static image data set is used, the pixel values ​​of band 1 represented by the near-infrared static image data are [120, 130, 135; 125, 140, 130; 115, 120, 125]; the pixel values ​​of band 2 represented by the short-wave infrared static image data are [35, 50, 50; 35, 40, 35; 30, 35, 40]; the pixel values ​​of band 3 represented by the visible red region static image data are [85, 105, 110; 95, 100, 95; 90, 95, 100]; the pixel values ​​of band 4 represented by the visible blue region static image data are: [70, 75, 80; 65, 70, 65; 60, 65, 70]; S3-2, using the width, height and time length of the frame in the key individual frame data in the sampled video as three dimensions, establishing a three-dimensional tensor required to be input by the I3D network, and at the same time, converting the frame formats of the width, height and time length of the frame in the key individual frame data in the sampled video into the input format of the I3D network; Example: Based on step S1-12, it should be noted that the expected input of the I3D network is a video clip with a specific width, height and time dimension, that is, assuming that the video clip acceptable to the I3D network is: width w=32 pixels, height h=32 pixels, time length t=16 frames, then, since each static image data obtained is a local area containing 3 rows and 3 columns, a total of 9 pixels, it is necessary to adjust each frame to 32*32 pixels required by the I3D network through an interpolation method; next, it is necessary to construct a three-dimensional tensor with a time dimension that the I3D network needs to input. In specific implementation, a local area containing 3 rows and 3 columns, a total of 9 pixels, is selected from each band as a single frame, recorded as [120, 130,..., 125, ...], and this frame is repeated to construct a video clip with 16 frames, recorded as ; It should be noted that each band has an existing frame sequence. Next, we need to construct a tensor, that is, fill the entire 32*32 pixel frame by copying the local area of ​​the above 9 pixels. At the same time, the entire input tensor is constructed by copying the frame 16 times. Finally, the constructed 16-frame video clip is formatted into a three-dimensional tensor, and the final input tensor dimension will be (16, 32, 32, 4), where 4 represents the number of bands of the four-band static image dataset.

[0030] S3-3, using the pre-trained I3D network to perform forward propagation on the formatted key individual frame data, extracting spatiotemporal features, and forming a spatiotemporal feature sequence F. In specific implementation, the I3D network can gradually extract and compress the features of the video frame through a series of 3D convolutional layers and pooling layers; Example: First, input the data and use the final input tensor dimension (16, 32, 32, 4) as the input size required by the I3D network. Second, build and configure the 3D convolution layer: convolution kernel size (3,3,3), stride (1,1,1), number of convolution kernels 32 output channels; third, pass the input data through the convolution layer and apply a nonlinear activation function (such as ReLU) to achieve forward propagation. Finally, connect the 3D pooling layer to perform maximum pooling, reduce the spatial dimension of the features, reduce the amount of calculation, and output new spatiotemporal features to form a spatiotemporal feature sequence F, where the spatiotemporal feature sequence F is a multidimensional array.

[0031] In one embodiment of the present invention, the proposed emotional intelligence monitoring system also includes a multimodal feature fusion module S4, which is connected to the image preprocessing module S1, and based on the created multimodal temporal information fusion model, fuses the video and physiological signal data, and analyzes the temporal and spatial characteristics of the facial expression of the target person in real time to identify and understand the emotional changes of the target person on different time scales, so as to understand and analyze the target object more comprehensively and accurately. The specific steps include: S4-1. Fuse spatiotemporal feature data at different time granularities to build a multimodal temporal information fusion model containing multiple modalities (such as video and physiological signals) to reflect the impact of changes in emotional state over time; S4-11, based on step S0-2 and step S3-3, extract the feature vectors A=[A1, A2, A3, ..., A] of the key individual frame data of the static image data representing the facial expression of the target person, respectively. m ]; and the feature vector B=[B1,B2,B3,...,B n ], m and n represent the dimensions of feature vectors A and B respectively; S4-12. Use linear regression algorithm to train and construct a multimodal time series information fusion model: y=a×[A,B]+b(7) In the formula, a represents the weight vector of dimension (m+n); b represents the bias term; [A,B] represents the fusion feature, [A,B]=[A1,A2,A3,...,A m ,B1,B2,B3,...,B n ]; Example: Assume that the number of samples is N=100; the dimension of the feature vector A of the key individual frame data is m=999; the dimension of the feature vector B of the physiological signal data set is n=80; then the dimension of the fusion feature is a=m+n=1079; At this time, the example steps of using the linear regression algorithm to train and build a multimodal time series information fusion model are: First, initialize the weights a=[a1,a2,a3,...,a 1079 ] and bias term b=0. It can be understood that in machine learning and deep learning, bias term b is usually used together with weight a to parameterize the model. The calculation of bias term b depends on the algorithm and optimization method used. It is generally calculated by minimizing the loss function (mean square error). The bias term is initialized to 0. Secondly, the optimization iteration is performed. For each sample N i , calculate its predicted value =a×[A,B] Ni +b, based on minimizing the loss function, i.e., the mean square error, calculates its loss rate: ,yNi It is represented as the actual value calculated based on 100 samples. In the above calculation, y Ni is a continuous value; Again, the weight a is updated after one iteration to obtain the new fusion feature [A1, B1] = [A 11 ,A 12 ,...,A 1999 ,B 11 ,B 12 ,...,B 180 ]; It should be noted that the aforementioned [A1, B1] is a single long vector formed by sequentially connecting the key individual frame data features of the static image data representing the facial expression of the target person and the features of the target person's physiological signal data set; this vector contains all the features used to predict the target variable y Ni The characteristic information of the target variable y is N=100. Ni It is known that represents the specific value predicted by the fusion feature [A, B]. In the specific actual calculation, since the present invention uses a linear regression algorithm to train a multimodal time series information fusion model, y Ni It is a continuous value. In the training process of the multimodal time series information fusion model, new fusion features [A1, B1] need to be used to predict y Ni , and adjust the model parameters through an optimization algorithm to minimize the difference between the predicted and actual values.

[0032] In one embodiment of the present invention, the proposed emotional intelligence monitoring system also includes The emotional state visualization module S5 is connected to the multimodal feature fusion module S4, and is used to visualize the emotional change state of the target person on different time scales predicted by the multimodal feature fusion module to the monitoring personnel or the system for decision analysis.

[0033] As a second aspect of the present invention, a method for intelligently monitoring human emotions is proposed, such as Figure 3 As shown, the steps include: (1) Preprocessing and feature extraction of facial expressions and physiological signal data of target persons: For video data representing facial expressions of target persons, key frames are extracted from the video and features are extracted using an I3D network; for data representing physiological signals of target persons, heart rate and skin conductance data are collected; I3D network feature extraction: Use the pre-trained I3D network to forward propagate the key frames and extract spatiotemporal features. It should be noted that in this step, the key frame data representing the facial expressions of the target person has been adjusted to the input size required by the I3D network, that is, (16, 32, 32, 4); Generate spatiotemporal feature sequence F: Based on the I3D network, output a multi-dimensional spatiotemporal feature sequence F that represents the facial expression of the target person. It can be understood that this spatiotemporal feature sequence F contains the spatiotemporal features of the video frames of the continuously sampled facial expressions of the target person. In this step, the feature dimension output by the last convolutional layer of the I3D network is (16, 32, 32,999); Multimodal feature fusion: For the video data features representing the facial expressions of the target person, the feature vector extracted from the I3D network has a dimension of 999; for the data features representing the physiological signals of the target person, the heart rate and skin conductance data are converted into numerical feature vectors. Suppose the target person's continuous average heart rate data H is used. Ravg and maximum skin conductance S Cmax As a feature, the dimension is 80; The new feature vector after fusing video and physiological signal data is constructed by converting the feature vectors of key individual frame data [A1, A2, A3, ..., A 999 ] and the feature vectors of the physiological signal dataset [B1,B2,B3,...,B 80 ] are connected in series to form the final fusion feature vector F eig =[A1,...,A 999， B1,...,B 80 ], let [A1,A2,A3,...,A 999 ]=[0.15,0.42,...,0.56],[B1,B2,B3,...,B 80 ]=[100,101,...,1,...], the feature vector B of the physiological signal data set is set according to the actual measurement values ​​at different points; A multimodal temporal information fusion model is created using a linear regression model to fuse video and physiological signal data, analyze the temporal and spatial characteristics of the target person's facial expression in real time, and predict and output a score that represents the target person's emotional change state at different time scales: the final fusion feature vector F eig =[A1,...,A 999， B1,...,B 80 ], as the input vector of the multimodal time series information fusion model, randomly set the initial weight a=[a1,a2,a3,...,a 1079 ] and the bias term b=0, then the predicted value =a1×A1+a2×A2+a 999 ×A 999 +a 1000 ×B1+...+a 1079 ×B 80 , decision analysis can be performed based on the category labels that reflect the target person's emotional state, which are preset by the monitoring personnel or the system: (8), For example, the predicted value =0.8, it means that the target person is in a tense state. The above ω is the continuous emotional state value predicted by the multimodal time series information fusion model, and Cω is the emotional state category label determined according to the ω value. It should be noted that in the specific implementation, ω can be any value, and Cω will be mapped to a predefined emotional state category label according to the above decision analysis rules.

[0034] The technical scope of the present invention is not limited to the contents in the above description. Those skilled in the art can make various deformations and modifications to the above embodiments without departing from the technical idea of ​​the present invention, and these deformations and modifications should all fall within the protection scope of the present invention.

Claims

1. A personnel emotion intelligent monitoring system, characterized by: include Acquisition module (S0), image preprocessing module (S1), multimodal feature fusion module (S4) and emotional state visualization module (S5), wherein: The acquisition module (S0) samples the video based on the time scale rolling, and uses the set frame difference threshold T scene To compare the differences between consecutive sampled video frames, identify changes in the sampled scenes, and select frames near the change points as key individual frame data of static image data representing the facial expressions of the target person; and synchronize the preset monitoring equipment, establish a time scale sequence, and collect the physiological signal data of the target person at different time steps S; The image preprocessing module (S1) includes an image quality evaluation unit (S101) connected to the acquisition module (S0), which is used to evaluate the quality of the acquired static image data to obtain static image data of qualified quality, including the steps of: S1-1. Construct a four-band static image dataset based on the collected static image data, and set the pixel value constraint range of each band in the four-band static image dataset to 0 to 255. Select a local area containing 3 rows and 3 columns, with a total of 9 pixels, from each acquired static image data, and record it as [A1, A2, A3; B1, B2, B3; C1, C2, C3], so as to capture the local changes of the static image data, and only extract the pixel values ​​of this area; combine the extracted pixel values ​​into a four-band static image dataset, with each pixel value as a 4-dimensional feature vector; S1-2, respectively setting a recognizability index D for each band in the four-band static image data set, and sorting the bands 1 to 4, selecting candidate feature bands, and further evaluating the clarity, contrast, and noise of the acquired static image data to achieve the best recognition effect; S1-3, according to the brightness information of the candidate characteristic band, select the pixel point with the best brightness, and determine the band with the highest recognizability by comparing the maximum brightness pixel position of the grayscale images of different bands on the vertical coordinate, generate a channel image with clear visual effect, and obtain static image data of qualified quality; The multimodal feature fusion module (S4) is connected to the image preprocessing module (S1), and analyzes the temporal and spatial characteristics of the facial expressions of the target person by creating a multimodal temporal information fusion model, fusing the sampled video and physiological signal data, so as to identify and understand the emotional changes of the target person at different time scales; The emotional state visualization module (S5) is connected to the multimodal feature fusion module (S4) and is used to visualize the emotional changes of the target person on different time scales predicted by the multimodal feature fusion module to the monitoring personnel or the system for decision analysis.

2. The intelligent human emotion monitoring system according to claim 1, characterized in that: The image preprocessing module (S1) further includes an image enhancement unit (S201) for performing color image enhancement on the static image data of qualified quality to further improve the image quality. The specific steps include: S2-1, using the Retinex algorithm to process each color channel of the obtained quality-qualified static image data respectively, and calculating the maximum brightness pixel position I of the initially input quality-qualified static image data CS (x, y). max The pixel value of (X) is convolved with the Gaussian convolution kernel GS(x,y) to obtain the output quality static image data enhanced by the Retinex algorithm. , wherein the Gaussian convolution kernel GS(x,y) is determined by the standard deviation σi and the adjustment coefficient ε to ensure the normalization of the probability density function; S2-2, using isotropic nonlinear filtering technology, by comparing and counting the maximum brightness pixel position I max The similarity between the brightness of the central pixel at (X) and the surrounding pixels enhances the wide line features in the quality qualified static image data.

3. The intelligent human emotion monitoring system according to claim 2, characterized in that: The specific steps of calculating the similarity include: S2-21, respectively presetting the maximum brightness pixel position I max The pixel value coordinates (α0, β0) of (X) and the maximum brightness pixel position I max The coordinates (α, β) of the surrounding pixels at (X) and the brightness contrast threshold τ are used to evaluate the relationship between the surrounding pixels and the maximum brightness pixel position I max (X) contrast difference between pixels; S2-22, for each of the surrounding pixel coordinates (α, β), calculate its relative position to the maximum brightness pixel position I max The brightness difference of the pixel value of (X) at coordinates (α0, β0) ,in, like T≤brightness contrast threshold τ, then it is considered that each of the surrounding pixel coordinates (α, β) meets the contrast threshold condition; continue to execute S2-23; S2-23, according to the obtained quality qualified static image data, if there is at least one pair of surrounding pixel coordinates (α, β), and at this coordinate, the brightness value f(α, β) of the surrounding pixel coordinates (α, β) is equal to the maximum brightness pixel position I max The absolute difference between the brightness values ​​f(α0, β0) at the pixel value coordinates (α0, β0) of (X) is greater than a preset brightness contrast threshold τ, and its impact on the filter response needs to be further calculated: , where ∑∑ represents the sum of the coordinates (α, β) of all surrounding pixel points in each color channel of the qualified static image data; η(α0, β0) is a metric value, which represents the degree of matching of a certain wide line feature in the qualified static image data. The larger the η value, the higher the matching degree of the wide line feature.

4. The intelligent human emotion monitoring system according to claim 2, characterized in that: The emotional intelligence monitoring system further includes a broad line feature extraction module (S3), The input end of the wide line feature extraction module (S3) is connected to the image enhancement unit (S201), and the output end is connected to the multimodal feature fusion module (S4), wherein the wide line feature extraction module (S3) is used to use a pre-trained I3D network to extract wide line features from quality-qualified static image data representing facial expressions of target persons, and obtain a spatiotemporal feature sequence F of key individual frame data in the sampled video.

5. The intelligent human emotion monitoring system according to claim 4, characterized in that: The broad line feature extraction module (S3) extracts broad line features to obtain a spatiotemporal feature sequence F, including the following specific steps: S3-1, taking the pixel value corresponding to each band in the four-band static image data set after color image enhancement as input; S3-2, using the width, height and time length of the frame in the key individual frame data in the sampled video as three dimensions, establish a three-dimensional tensor required to be input to the I3D network, and at the same time, The frame formats of the width, height and time length of the frame in the key individual frame data in the sampled video are respectively converted into the input format of the I3D network: a local area containing 3 rows and 3 columns, and a total of 9 pixels, is selected from each band as a single frame; S3-3. Use the pre-trained I3D network to perform forward propagation on the formatted key individual frame data to extract spatiotemporal features and form a spatiotemporal feature sequence F: Use a series of 3D convolutional layers and pooling layers to gradually extract and compress the features of the video frames to form a spatiotemporal feature sequence F, wherein the spatiotemporal feature sequence F is a multidimensional array.

6. The intelligent human emotion monitoring system according to claim 1, characterized in that: The specific steps to create a multimodal time series information fusion model include: S4-11, extract the feature vectors A=[A1, A2, A3, ..., A] of the key individual frame data of the static image data representing the facial expression of the target person respectively. m ] and the feature vector B=[B1,B2,B3,...,B n ], m and n represent the dimensions of feature vectors A and B respectively; S4-12. Use the linear regression algorithm to train and construct a multimodal time series information fusion model: y=a×[A,B]+b, where a represents the weight vector of dimension (m+n); b represents the bias term; [A,B] represents the fusion feature, [A,B]=[A1,A2,A3,...,A m ,B1,B2,B3,...,B n ].

7. The intelligent human emotion monitoring system according to claim 6, characterized in that: The calculation of the bias term b is performed by minimizing the loss function, and the bias term is initialized to 0; The specific steps of using the linear regression algorithm to train and build a multimodal time series information fusion model include: First, initialize the weights a=[a1,a2,a3,...,a m+n ] and bias term b=0; Secondly, the optimization iteration is performed. For each sample N i , calculate its predicted value , and calculate its loss rate based on minimizing the loss function: ,y Ni Represented as the actual value calculated based on N samples. In the above calculation, y Ni is a continuous value; Finally, the weight a is updated after one iteration to obtain the new fusion feature [A1, B1] = [A 11 ,A 12 ,...,A 1m ,B 11 ,B 12 ,...,B 1n ], wherein [A1, B1] is a single long vector formed by sequentially connecting the key individual frame data features of the static image data representing the facial expression of the target person and the features of the target person's physiological signal data set, which contains all the features used to predict the target variable y Ni feature information.

8. The intelligent human emotion monitoring system according to claim 1, characterized in that: The four-band static image data set includes near-infrared static image data, short-wave infrared static image data, visible light red region static image data, and visible light blue region static image data; The band represented by the custom near-infrared static image data is band 1; the band represented by the short-wave infrared static image data is band 2, the band represented by the visible light red zone static image data is band 3; the band represented by the visible light blue zone static image data is band 4, and the variance of each band is obtained after independent calculation of each band, and then the variance of band 1, band 2, band 3, and band 4 is obtained. , according to the formula , calculate the recognizability index D corresponding to each band respectively, so as to perform statistical measurement on the quality index of each band; where σ i 2 Expressed as the variance of the i-th band, i∈(1,2,3,4): 0.01 is the minimum positive number used to ensure calculation stability.

9. The intelligent human emotion monitoring system according to claim 1, characterized in that: Compare the maximum brightness pixel position of grayscale images of different bands on the vertical coordinate to determine the band with the highest recognizability. The specific calculation method is as follows: , Where, i∈(1,2,3,4), I ave Represented as the grayscale image brightness of the static image data of the i-th band; σ i It is represented as the standard deviation of the grayscale image of the static image data of the i-th band; I ave ≤128 means the pixel value of each band is obtained with the best brightness within the constraint range of 0 to 255; max (X) represents the maximum brightness pixel position of brightness y=1,2,.,M on the vertical coordinate of the grayscale image of the static image data of the i-th band; I max The smaller (X) is, the channel image in the acquired candidate feature band is considered to have the highest recognizability.

10. The intelligent human emotion monitoring system according to claim 1, characterized in that: The acquisition module (S0) includes an image data acquisition unit (S001) and a physiological signal data acquisition unit (S002), wherein: The image data acquisition unit (S001) selects frames near the change point as key individual frame data, specifically in the following manner: According to the frame difference threshold T scene Compare the differences between consecutive sampled video frames, denoted as , where ξ(φ) represents the difference between the φth frame of the sampled video frame and the previous frame; Represented as the pixel values ​​of the φth frame and the φ-1th frame at the coordinates (x, y) respectively; if the difference ξ(φ) between the upper and lower frames of the sampled video exceeds the frame difference threshold T scene ,but It is considered that the scene of the current target person’s facial expression has changed. Once the scene change is detected, the video frame in the area of ​​the scene change point is selected as the key single frame data, and the frame difference threshold T scene Set according to video content and application requirements; The physiological signal data acquisition unit (S002) collects heart rate data H R and skin conductance data S C The physiological signal data is as follows: Based on the heart rate monitoring device, obtain the target person's continuous heart rate data H R : , In the formula, H Ravg It is expressed as the average heart rate of the target person monitored in time step S under the time scale sequence; Based on the skin conductance sensor, the skin conductance data S representing the target person's emotional arousal level is obtained. C :S Cmax =max(S C1 ,S C2 ,...,S CS ), where S Cmax It is expressed as the maximum skin conductance value of the target person monitored within the time step S under the time scale sequence.

Images