Emotion prediction method and system based on human intelligence, edge computing device and medium

Abstract

The invention provides an emotion prediction method and system based on human intelligence, an edge computing device and a medium, and the method comprises the steps: obtaining original video stream data, extracting an image of a key frame from the original video stream data according to a preset frame rate, and carrying out the preprocessing of the extracted image of each key frame; wherein the image of the key frame comprises a plurality of face images; performing emotion recognition on the preprocessed image of each key frame, recognizing the emotion category of each face in the image of the key frame, and counting the occurrence frequency of different emotion categories in the image of the current key frame; according to the time sequence of the key frames, the occurrence frequency of each emotion category in the images of all the key frames is integrated and counted in a time sequence data mode, and a comprehensive group emotion prediction result is weighted and calculated according to a preset weight.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition technology, and in particular to an emotion prediction method, system, edge computing device and medium based on human intelligence. Background Art

[0002] Emotion recognition is extremely important in today's society. It is not only related to the progress of human-computer interaction, but also shows great application potential in many fields. This technology can be used to improve human-computer interaction, psychology, medicine, customer service, etc. Summary of the invention

[0003] Embodiments of the present invention provide an emotion prediction method, system, edge computing device, and medium based on human intelligence.

[0004] One aspect of the present invention provides an emotion prediction method based on human intelligence, which includes the following steps: obtaining original video stream data, extracting key frame images from the original video stream data according to a preset frame rate, and preprocessing each extracted key frame image; wherein the key frame image contains multiple face images; performing emotion recognition on each preprocessed key frame image, identifying the emotion category of each face in the key frame image, and counting the frequency of occurrence of different emotion categories in the current key frame image; integrating and counting the frequency of occurrence of each emotion category in all key frame images in the form of time series data according to the time sequence of the key frames, and weighted calculating the comprehensive group emotion prediction result according to preset weights.

[0005] In some embodiments of the present invention, preprocessing the extracted image of each key frame includes: performing size normalization on the extracted image of each key frame to adjust the resolution, pixel size or data specifications of the image to a preset range; and performing lighting correction on the size-normalized image.

[0006] In some embodiments of the present invention, emotion recognition is performed on the preprocessed image of each key frame, including: performing multi-scale perception on the preprocessed image of each key frame to capture image features at different scales to obtain multi-scale perceived image features; down-sampling the multi-scale perceived image features to obtain a first output result, a second output result, and a third output result; predicting emotion categories for the first output result, the second output result, and the third output result, respectively, and determining a final emotion recognition result according to preset weights based on the predicted emotion categories.

[0007] In some embodiments of the present invention, the multi-scale perceptual image features are downsampled respectively to obtain a first output result, a second output result and a third output result, including: the original image is subjected to spatial depth decoupling downsampling processing to obtain an initial feature map; the initial feature map is subjected to multi-head query attention downsampling processing to obtain a first feature map; the first feature map is subjected to differential fusion convolution processing to obtain a second feature map; the second feature map is subjected to spatial depth decoupling downsampling processing to obtain a third feature map; the third feature map is subjected to differential fusion convolution layer processing to obtain a fourth feature map; the fourth feature map is subjected to spatial depth decoupling downsampling processing to obtain a fifth feature map; the fifth feature map is subjected to differential fusion convolution processing to obtain a sixth feature map; the sixth feature map is subjected to multi-head query attention processing to obtain a seventh feature map; the seventh feature map is subjected to multi-head query attention downsampling processing to obtain an eighth feature map; wherein the fourth feature map is used as the first output result, the seventh feature map is used as the second output result, and the eighth feature map is used as the third output result.

[0008] In some embodiments of the present invention, the frequency of occurrence of each emotion category in the images of all key frames is integrated and counted in the form of time series data according to the timing of key frames, including: each key frame image has a time point label, the corresponding emotion category recognition results have the same time point label, and the emotion category recognition results are arranged in a time series using the time point labels; a data structure is constructed to organize and store all emotion category recognition results in time series.

[0009] In some embodiments of the present invention, a comprehensive group emotion prediction result is weighted and calculated according to preset weights, including: for each emotion category, taking the frequency of the emotion category appearing in all emotion category recognition results multiplied by the preset weight of the emotion category as the importance parameter of the corresponding emotion category; calculating the proportion of each emotion category in the group by combining the importance parameters of each emotion category; eliminating emotion categories whose proportion is lower than a preset threshold, and generating a group emotion prediction result including the remaining emotion categories and the proportion of the emotion categories in the group.

[0010] In some embodiments of the present invention, the method also includes: obtaining subjective emotion labels collected from each person in the group through a subjective scale or questionnaire; learning the correlation between time series data and the collected subjective emotion labels through a deep learning model to optimize the preset weights as a training target of the deep learning model and adjust the preset weights.

[0011] Corresponding to the above method, the present invention also provides an emotion prediction system based on human intelligence, comprising: a video stream preprocessing module, used to extract key frame images from the acquired original video stream data according to a preset frame rate, and preprocess each extracted key frame image; a lightweight emotion recognition model, used to perform emotion recognition on each preprocessed key frame image, and obtain a corresponding emotion category recognition result respectively; a group emotion prediction module, used to integrate all emotion category recognition results into time series data according to the time sequence of key frames, count the frequency of each emotion category in all emotion category recognition results through the time series data, and calculate the comprehensive group emotion prediction result by weighting according to preset weights.

[0012] Corresponding to the above method, the present invention also provides an edge computing device, including a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the edge computing device implements the steps of the method described in any one of the above embodiments.

[0013] Corresponding to the above method, the present invention also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the steps of the method described in any one of the above embodiments are implemented.

[0014] The emotion prediction method based on human intelligence proposed in the present invention is conducive to real-time and lightweight group emotion recognition through emotion recognition models, and a standardized group emotion atmosphere measurement method and detection framework are proposed, which can effectively fill the gap in the existing technology for real-time analysis and recognition of group emotions, and realize real-time and accurate group emotion analysis and recognition.

[0015] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be achieved and obtained by the structures specifically indicated in the specification and the accompanying drawings.

[0016] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention. The components in the drawings are not drawn to scale, but are only for illustrating the principles of the present invention. In order to facilitate the illustration and description of some parts of the present invention, the corresponding parts in the drawings may be enlarged, that is, they may become larger relative to other components in the exemplary device actually manufactured according to the present invention. In the drawings:

[0018] Figure 1 This is a flow chart of a lightweight emotion recognition method in one embodiment of the present invention.

[0019] Figure 2 Schematic diagram of the structure of a group emotion prediction system in one embodiment of the present invention.

[0020] Figure 3 Schematic diagram of the structure of a lightweight emotion recognition model in one embodiment of the present invention.

[0021] Figure 4 This is a workflow diagram of a group emotion prediction module in one embodiment of the present invention.

[0022] Figure 5 A schematic diagram of the structure of the computer equipment included in the system. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0024] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0025] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, elements, steps or components, but does not exclude the presence or addition of one or more other features, elements, steps or components.

[0026] It should also be noted that, unless otherwise specified, the term “connection” herein may refer not only to a direct connection but also to an indirect connection involving an intermediate.

[0027] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar components, or the same or similar steps.

[0028] Figure 1This is a flow chart of a lightweight emotion recognition method in an embodiment of the present invention. The method comprises the following steps:

[0029] Step S110: acquiring original video stream data, extracting key frame images from the original video stream data at a preset frame rate, and preprocessing each extracted key frame image; wherein the key frame images include a plurality of face images.

[0030] In the specific implementation process, the processing flow of the video stream preprocessing module is as follows: (1) Extract key frames from the video stream at a certain frame rate to avoid data redundancy while ensuring that the dynamic changes of emotions can be captured; (2) Perform size normalization processing on each frame of the image, and uniformly adjust images of different resolutions to a standard size suitable for model input to facilitate subsequent feature extraction and analysis; (3) Perform illumination correction on each frame of the image to reduce the image quality differences caused by different lighting environments (such as strong light, dark light, side light, etc.), so that the emotional characteristics of the face can be presented more clearly.

[0031] Step S120: performing emotion recognition on the pre-processed image of each key frame, identifying the emotion category of each face in the key frame image, and counting the frequency of occurrence of different emotion categories in the current key frame image.

[0032] In the specific implementation process, the lightweight emotion recognition model belongs to the prediction model, and the predicted emotion categories include 7 categories: happiness, sadness, anger, surprise, disgust, fear and neutral. This implementation is only an example and is not a specific limitation. In step S120, the areas where different faces are located can be first identified, and all face images contained in the key frame image can be cut out, and an emotion category expression is generated for different face images based on the emotion recognition model. On the other hand, face area recognition in an image can also be realized based on the emotion recognition model.

[0033] Step S130: Integrate and count the frequency of each emotion category in the images of all key frames in the time sequence of the key frames by means of time series data, and calculate the comprehensive group emotion prediction result by weighting according to the preset weights.

[0034] In some embodiments of the present invention, all emotion category recognition results can be integrated into time series data according to the time sequence of key frames, and the frequency of each emotion category in all emotion category recognition results can be counted through the time series data, and the comprehensive group emotion prediction result can be calculated according to the preset weights. Figure 4In the array structure of the data structure 1 shown, the vertical number [9, 0, 0, 0, 0, 0, 1] obtained by identifying and counting the image of a single key frame in step S120, and the frequency of occurrence of different emotion categories in the images of all key frames can be stored in the form of data structure 1.

[0035] In the specific implementation process, the group emotion prediction result can be in the form of an array or matrix containing all emotion categories, or a report of the remaining emotion proportion after removing the emotion proportions below the preset threshold. For example, Figure 4 In the embodiment shown, the output report reports the group's emotions as happy or "happy: neutral = 45*w 高兴 :15w 中性 ". In another specific embodiment, the group emotion prediction result can be equivalent to the group emotion prediction result, and based on the preset one-to-one association relationship between emotions, a binary group consisting of "emotion-emotion" is constructed, such as "smile-happy", "frown-frustrated", etc. This is not a specific limitation here, but only an example.

[0036] The emotion prediction method based on human intelligence proposed in the present invention is conducive to lightweight real-time group emotion recognition, and a standardized group emotion atmosphere measurement method and detection framework are proposed, which can effectively fill the gap in the existing technology for real-time analysis and recognition of group emotions, and realize real-time and accurate group emotion analysis and recognition.

[0037] In some embodiments of the present invention, the preprocessing of the image of each extracted key frame in step S110 includes: (1) performing size normalization on the image of each extracted key frame to adjust the resolution, pixel size or data specifications of the image to a preset range; and (2) performing illumination correction on the size normalized image.

[0038] By adopting the embodiment of the invention, a series of preprocessing operations can be performed on the input raw video stream data, aiming to improve the accuracy and efficiency of subsequent emotion recognition and group emotion prediction.

[0039] In some embodiments of the present invention, the lightweight emotion recognition model will perform feature extraction on the input image of the extracted key frame, and the lightweight of the image and the lightweight of emotion recognition can be achieved through the various layers listed below. The lightweight emotion recognition model mainly includes: extended bottleneck residual layer, extended bottleneck residual downsampling layer, spatial depth decoupled convolution layer, spatial depth decoupled downsampling layer, multi-head query attention layer, multi-head query attention downsampling layer, differential fusion convolution layer, multi-scale perception layer, sequential fusion convolution layer and emotion recognition layer.

[0040] Furthermore, in some specific embodiments, emotion recognition is performed on the preprocessed image of each key frame, including: processing the outputs of the sequential fusion convolution layers respectively through the spatial depth decoupled convolution layers to obtain a first prediction head, a second prediction head and a third prediction head, and using the first prediction head, the second prediction head and the third prediction head to predict an emotion category corresponding to the image of the key frame.

[0041] In some other embodiments of the present invention, emotion recognition is performed on the preprocessed image of each key frame, including: (1) performing multi-scale perception on the preprocessed image of each key frame to capture image features at different scales and obtain multi-scale perceived image features; (2) downsampling the multi-scale perceived image features to obtain a first output result, a second output result, and a third output result; (3) predicting emotion categories of the first output result, the second output result, and the third output result, respectively, and determining a final emotion recognition result based on the predicted emotion categories according to preset weights.

[0042] Further, the multi-scale perceptual image features are downsampled respectively to obtain a first output result, a second output result and a third output result, including: (1) the original image is subjected to spatial depth decoupling downsampling processing to obtain an initial feature map; (2) the initial feature map is subjected to multi-head query attention downsampling processing to obtain a first feature map; (3) the first feature map is subjected to differential fusion convolution processing to obtain a second feature map; (4) the second feature map is subjected to spatial depth decoupling downsampling processing to obtain a third feature map; (5) the third feature map is subjected to differential fusion convolution processing (6) performing spatial depth decoupling and downsampling processing on the fourth feature map to obtain a fifth feature map; (7) performing differential fusion convolution processing on the fifth feature map to obtain a sixth feature map; (8) performing multi-head query attention processing on the sixth feature map to obtain a seventh feature map; (9) performing multi-head query attention downsampling processing on the seventh feature map to obtain an eighth feature map; wherein the fourth feature map is used as the first output result, the seventh feature map is used as the second output result, and the eighth feature map is used as the third output result.

[0043] The use of the above-mentioned lightweight emotion recognition model is conducive to lightweight processing of the extracted key frame images. The lightweight emotion recognition model itself is also relatively small in magnitude and can be deployed on edge computing devices or edge computing terminals of the edge network.

[0044] In the specific implementation process, the lightweight emotion recognition model identifies the different facial areas contained in the image of each key frame of the input, performs emotion recognition on different facial areas respectively, and obtains an emotion category for each recognized facial area. The frequency of occurrence of different emotion categories contained in the image is counted, and each face is counted as one.

[0045] Figure 3 FIG. 1 is a schematic diagram of the structure of a lightweight emotion recognition model in an embodiment of the present invention. The following introduces the processing and functions of each level included in the lightweight emotion recognition model.

[0046] The calculation process of the first layer, the expanded bottleneck residual layer 101, is as follows: (1) For a certain feature, the size is set to H2×W2×C2, C2 is the number of input channels, and the feature is recorded as U1; (2) U1 is subjected to a 1×1 convolution operation to obtain feature U2, and the size of the 1×1 convolution is 1×1×C2; the number of 1×1 convolutions is E2, E2 is the number of expanded channels, the activation function is SELU, and the size of U2 is H2×W2×E2; (3) U2 is subjected to a depthwise separable convolution to obtain U3; the depthwise separable convolution performs a convolution operation on each channel of U2 separately, and the activation function is SELU, and finally The calculation structures of all channels are merged; the size of U3 is H2×W2×E2; (4) U3 is subjected to a 1×1 convolution operation to obtain feature U4, and the size of the 1×1 convolution is 1×1×E2; the number of 1×1 convolutions is F2, which is a linear activation, that is, no activation function is used, so the output dimension of U4 is H2×W2×F2, F2 is the number of output channels, C2 and F2 can be equal or unequal; (5) When C2 and F2 are equal, a residual connection operation is performed, that is, U1 and U4 are added to obtain output U5; (6) When C2 and F2 are unequal, U4 is output.

[0047] The bottleneck residual layer is a structure introduced in the deep residual network (ResNet) type, which is used to reduce the computational complexity and enhance the feature extraction capability.

[0048] This solution can reduce the amount of calculation and parameters through deep separable convolution, making the algorithm adaptable to low-computing power devices such as edge devices. By referring to the efficient detection ideas of the YOLO model, it is conducive to quickly extracting emotional information from images and realizing real-time emotion recognition. It can overcome the problem that traditional convolutional neural network algorithms cannot perform real-time recognition on low-computing power devices.

[0049] The calculation process of the second layer, the extended bottleneck residual downsampling layer 102, is as follows: (1) For a certain feature, the size is set to H3×W3×C3, C3 is the number of input channels, and the feature is recorded as E0; (2) Use depthwise separable convolution to downsample E0, the depthwise separable convolution performs convolution operation on each channel of E0 separately, the convolution complement step is set to 2, and no activation function is set. Finally, the calculation structures of all channels are merged to obtain a feature E1 with a size of (H3 / 2)×(W3 / 2)×C3; (3) E1 is subjected to a 1×1 convolution operation to obtain a feature E2, and the size of the 1×1 convolution is 1×1×C3; the number of the 1×1 convolution is E3, E3 is the number of extended channels, E3 is an integer multiple of 8, the activation function is SELU, and the size of E2 is (H3 / 2)×(W3 / 2)×E 3; (4) E2 is subjected to depthwise separable convolution to obtain E3; the depthwise separable convolution performs convolution operation on each channel of E2 separately, the activation function is SELU, and finally the calculation structures of all channels are merged; the size of E3 is (H3 / 2)×(W3 / 2)×E3; (5) E3 is subjected to 1×1 convolution operation to obtain feature E4, the size of 1×1 convolution is 1×1×E3; the number of 1×1 convolutions is F3, no activation function is used here, so the output dimension of U4 is (H3 / 2)×(W3 / 2)×F3, F3 is the number of output channels, C3 and F3 can be equal or unequal; (6) when C3 and F3 are equal, a residual connection operation is performed, that is, E1 and E4 are added to obtain output E5; (7) when C3 and F3 are unequal, E4 is output.

[0050] In one embodiment of the present invention, the 1×1 convolution included in the extended bottleneck residual layer can be used to reduce and increase dimensionality, and the depthwise separable convolution included in the extended bottleneck residual downsampling layer can be used to downsample the output of the extended bottleneck residual layer.

[0051] The calculation process of the third layer, the spatial depth decoupled convolution layer 103, is as follows: (1) For a certain feature, the size is set to H1×W1×C1, and the feature is recorded as I1; (2) I1 is subjected to a 1×1 convolution operation to obtain feature I2, and the size of the 1×1 convolution is 1×1×C1; the number of the 1×1 convolutions is C2, C2 is the number of output channels, and the size of I2 is H1×W1×C2; (3) I2 is subjected to a depthwise separable convolution to obtain I3; the depthwise separable convolution The product performs convolution operation on each channel of I2 separately, and finally merges the calculation structures of all channels; the size of I3 is H1×W1×C2; (4) I3 obtains feature I4 after 1×1 convolution operation, and the size of 1×1 convolution is 1×1×C2; the number of 1×1 convolution is C3, C3 is the number of output channels, and the size of I4 is H1×W1×C3; (5) I4 obtains the final output result I4 after batch normalization and activation function calculation. Among them, batch normalization refers to the technical means of standardizing the output of the middle layer of the neural network so that its distribution is stable near the standard normal distribution with a mean of 0 and a variance of 1.

[0052] The spatial-depth decoupling convolution layer (SDDConvolution) aims to decouple spatial features and depth features to better capture local and global information in the image.

[0053] In one embodiment of the present invention, a spatial-depth decoupling convolution layer may be used to decouple spatial features from depth features, and a spatial-depth decoupling downsampling layer may be used to perform additional downsampling processing on input features.

[0054] The calculation process of the fourth layer, the spatial depth decoupling downsampling layer 104 is as follows: (1) For a certain feature, the size is set to H1×W1×C1, and the feature is recorded as I1; (2) After the I1 is subjected to a 1×1 convolution operation, the feature I2 is obtained, and the size of the 1×1 convolution is 1×1×C1; the number of the 1×1 convolutions is C2, C2 is the number of output channels, and the size of I2 is H1×W1×C2; (3) After the I2 is subjected to a depthwise separable convolution, I3 is obtained; the depthwise separable convolution operates on each channel of I2 separately A convolution operation is performed with a convolution step of 2, and finally the calculation structures of all channels are merged; the size of I3 is (H1 / 2)×(W1 / 2)×C2; (4) I3 is subjected to a 1×1 convolution operation to obtain feature I4, and the size of the 1×1 convolution is 1×1×C2; the number of 1×1 convolutions is C3, C3 is the number of output channels, and the size of I4 is (H1 / 2)×(W1 / 2)×C3; (5) I4 is subjected to batch normalization and activation function calculation to obtain the final output result I4. The difference between the spatial depth decoupled downsampling layer and the spatial depth decoupled convolution layer is whether downsampling is performed.

[0055] The calculation process of the fifth layer, the multi-head query attention layer 105 is as follows: (1) For a certain feature, the size is set to H4×W4×C4, C4 is the number of input channels, and the feature is recorded as R0; (2) Generate query features: R0 is subjected to a 1×1 convolution operation to obtain feature R1, and the size of the 1×1 convolution is 1×1×C4; the number of 1×1 convolutions is E4, E4 is the number of expansion channels, E4 is the product of the multi-head number nh and the key value dimension kd, and no activation function is used. The size of R1 is H4×W4×E4; (3) Generate key features: R0 is subjected to a 1×1 convolution operation to obtain feature R1, and the size of the 1×1 convolution is 1×1×C4; the number of 1×1 convolutions is E4, E4 is the number of expansion channels, and E4 is the product of the multi-head number nh and the key value dimension kd. No activation function is used, and the size of R1 is H4×W4×E4; After a 1×1 convolution operation, feature R2 is obtained, and the size of the 1×1 convolution is 1×1×E4; the number of the 1×1 convolutions is the key value dimension kd, and no activation function is used. The size of R2 is H4×W4×kd; (4) Generate value features: R0 is subjected to a 1×1 convolution operation to obtain feature R3, and the size of the 1×1 convolution is 1×1×kd; the number of the 1×1 convolutions is the key value dimension kd, and no activation function is used. The size of R2 is H4×W4×kd; (5) Calculate the attention score based on the query feature and the key feature, and the calculation formula is: In the above formula, R1 is the query feature, R2 is the key feature, and nh is the number of multi-heads; (6) Calculate the correlation, the calculation formula is: RE = AS * R3; (7) Linear activation: RE obtains the feature R4 after a 1×1 convolution operation, and the size of the 1×1 convolution is 1×1×E4; the number of the 1×1 convolution is C4, no activation function is used, and the size of R4 is H4×W4×C4; (8) To speed up training and inference, set nh and C4 equal, and set kd to 1.

[0056] The sixth layer, the multi-query attention downsampling layer 106, differs from the multi-query attention layer in whether an average pooling layer with a step size of 2 is used at the beginning.

[0057] The processing of multi-head query attention layer and multi-head query attention downsampling layer can be used to capture the long-range dependencies between input features.

[0058] The Multi-Head Attention Layer is a key component in the Transformer architecture. The main purpose of introducing the multi-head attention mechanism is to enhance the lightweight emotion recognition model's ability to capture complex relationships, enhance the model's computational stability, and ensure the stability of emotion recognition results.

[0059] By integrating the Transformer's self-attention mechanism, this solution can capture the long-range dependencies between emotional features and improve the accuracy of emotion recognition. It can achieve accurate recognition in different postures or lighting conditions, ensuring the reliability of the algorithm in a variety of scenarios.

[0060] The seventh layer, the calculation process of the differential fusion convolution layer 107 is as follows: (1) For a certain feature layer, the feature layer is recorded as P1; (2) P1 is calculated by the extended bottleneck residual layer to obtain P2; (3) P2 is calculated by the extended bottleneck residual layer to obtain P3; (4) P1 and P3 are processed by the channel merging layer to obtain P4. Among them, the input feature size and output feature size of the differential fusion convolution layer have the same dimension. The biggest difference from the C3 structure in YOLO is that the differential fusion convolution layer is all based on the extended bottleneck residual layer. While greatly reducing the number of parameters, it increases the learning ability between model channels, better integrates deep and shallow feature information, effectively combines shallow detail features and deep semantic features, improves the expression ability and generalization performance of the model, and can also alleviate the gradient disappearance problem. The differential fusion convolution layer combines the differential operation and the convolution operation in the convolution layer, uses the differential operation to capture the change information in the image or signal, and uses the convolution operation to extract the local features in the image.

[0061] The calculation process of the eighth layer, the multi-scale perception layer 108 is as follows: (1) Assume that the size of a certain original image T is H×W×3; respectively represent the height, width and number of channels of the image; (2) The original image is processed by spatial depth decoupling downsampling to obtain a feature map U0, and the size of U0 is (H / 2)×(W / 2)×64; (3) U0 is processed by the multi-head query attention downsampling layer to obtain a feature map U1, and the size of U1 is (H / 4)×(W / 4)×128; (4) The feature map U1 is processed by the differential fusion convolution layer to obtain a feature map U2, and the size of U2 is (H / 4)×(W / 4)×128; (5) U2 is processed by spatial depth decoupling downsampling to obtain a feature map U3, and the size of U3 is (H / 8)×(W / 8)×256; (6) The feature map U3 is processed by the differential fusion convolution layer to obtain the feature map U4, and the size of U4 is (H / 8)×(W / 8)×256; (7) The feature map U4 is processed by spatial depth decoupling and downsampling to obtain the feature map U5, and the size of U5 is (H / 16)×(W / 16)×512; (8) The feature map U5 is processed by the differential fusion convolution layer to obtain the feature map U6, and the size of U6 is (H / 16)×(W / 16)×512; (9) The feature map U6 is processed by the multi-head query attention layer to obtain the feature map U7, and the size of U7 is (H / 16)×(W / 16)×512; (10) The feature map U7 is processed by the multi-head query attention downsampling to obtain the feature map U8, and the size of U8 is (H / 32)×(W / 32)×1024. Among them, the outputs of the multi-scale perception layer are U4, U7 and U8. Unlike other YOLO target detection algorithms, the multi-scale perception layer directly processes the original input and uses different receptive fields to extract features from the original image at the very beginning of target detection, which is conducive to improving the performance of the model, merging features of different scales through channels, and integrating this information to enhance the perception ability of the model. The multi-scale perception layer can capture image features at different scales, thereby improving the generalization ability and robustness of the lightweight emotion recognition model.

[0062] The ninth layer, the calculation of the sequential fusion convolution layer 109 is consistent with the neck part of YOLOV5, wherein all parts involving downsampling are replaced by spatial depth decoupling downsampling layers, and all parts involving C3 are replaced by differential fusion convolution layers. The outputs of the sequential fusion convolution layer are recorded as T10 (i.e., the first output result), T11 (i.e., the second output result) and T12 (i.e., the third output result); the sizes of T10, T11 and T12 are: (H / 8)×(W / 8)×256, (H / 16)×(W / 16)×512 and (H / 32)×(W / 32)×1024, respectively.

[0063] The calculation process of the tenth layer, the emotion recognition layer 110, is as follows: (1) T10 is processed by the spatial depth decoupling convolution layer in sequence to obtain the first prediction head: the size of the first prediction head is (H / 8)×(W / 8)×g; (2) T11 is processed by the spatial depth decoupling convolution layer in sequence to obtain the second prediction head: the size of the second prediction head is (H / 16)×(W / 16)×g; (3) T12 is processed by the spatial depth decoupling convolution layer in sequence to obtain the third prediction head: the size of the third prediction head is (H / 32)×(W / 32)×g. Among them, the size of g is 36, which is specifically explained as (5+7)×3, where the 5 in (5+7)×3 represents the position (x, y) of the center point of the target frame, the width and height of the target frame, and the conditional probability of the target, and the 7 in (5+7)×3 represents the prediction category, including 7 categories of happiness, sadness, anger, surprise, disgust, fear and neutrality, and 3 means that in the prediction process, each grid has 3 prediction boxes.

[0064] In the specific implementation process, during the training of the lightweight emotion recognition model, the training set contains preprocessed images and annotated emotion category recognition result labels. The prediction ability of the emotion recognition layer can be trained through a variety of training methods. The input image is processed from the first layer to the ninth layer to extract image features, and the emotion recognition layer is trained to predict the emotion category recognition result label based on the extracted image features.

[0065] In other embodiments of the present invention, the MobileNet model can also be used to extract image features, and the machine learning SVM model, Google FaceNet model, DeepFace model or YOLO algorithm can be used as an emotion recognition model to realize the recognition of facial emotions or emotions.

[0066] In some embodiments of the present invention, in step S130, the frequency of occurrence of each emotion category in the images of all key frames is integrated and counted in the form of time series data according to the time sequence of the key frames, including: (1) the image of each key frame has a time point label, and the corresponding emotion category recognition result has the same time point label, and the emotion category recognition results are arranged in a time series using the time point label; (2) a data structure is constructed to organize and store all emotion category recognition results according to the time series.

[0067] By adopting the embodiment of the invention, all emotion category recognition results can be integrated in the form of time series data, so as to comprehensively analyze the emotion distribution in the group.

[0068] In some embodiments of the present invention, a comprehensive group emotion prediction result is weighted and calculated according to preset weights, including: (1) for each emotion category, the frequency of the emotion category appearing in all emotion category recognition results multiplied by the preset weight of the emotion category is used as the importance parameter of the corresponding emotion category; (2) the proportion of each emotion category in the group is calculated by combining the importance parameters of each emotion category; (3) the emotion categories whose proportion is lower than a preset threshold are eliminated, and a group emotion prediction result including the remaining emotion categories and the proportion of the emotion categories in the group is generated.

[0069] By adopting this embodiment of the invention, the importance ratio of different emotion categories can be calculated in a weighted manner. Different emotion categories are associated with different emotions (in this scheme, emotion categories and emotion categories can be used to refer to the same, that is, emotion categories directly correspond to emotions one by one), and emotion categories below the threshold are eliminated, thereby removing interference items and ensuring that the final group emotion recognition results are concise and representative.

[0070] In some embodiments of the present invention, the method further includes: (1) obtaining subjective emotion labels collected from each person in the group through a subjective scale or questionnaire; (2) learning the correlation between time series data and the collected subjective emotion labels through a deep learning model to optimize the preset weights as a training target of the deep learning model and adjust the preset weights.

[0071] By adopting this embodiment of the invention, the distribution of weights can be optimized through subjective data, thereby improving the fit between the group emotion recognition results calculated based on preset weights and the actual emotion distribution state of the group, making the recognition results of this solution more accurate.

[0072] In some embodiments of the present invention, in step S130, the operation process of the group emotion prediction module is as follows: (1) emotion frequency statistics, for each frame of image processed by the lightweight emotion recognition model, the emotion classification of all instances in the output image is performed, and the frequency of each emotion is counted; (2) time series integration, because the video stream is composed of multiple frames of images in chronological order, in order to more comprehensively grasp the changes in group emotions over time, it is necessary to integrate the emotion frequency data obtained from the statistics of each frame of image according to the time series, construct a time series data structure, and add the emotion frequency counted for each frame of image in turn to form a complete time series data reflecting the changes in the emotion frequency over time in the entire video stream. In this way, the frequency fluctuations of various emotions in the group at different times can be captured, providing richer dynamic information for subsequent emotional state analysis. (3) Weight allocation and feature extraction: Considering that the influence of different emotions on the emotional atmosphere of the group may vary, a corresponding weight is assigned to each emotion category. Based on the assigned weight and the time series data of emotion frequency, the weighted total emotion frequency and the proportional relationship of different emotion frequencies are calculated; (4) Emotional atmosphere measurement and prediction: The statistical results in step (3) are used to measure and predict the emotional atmosphere of the group.

[0073] Figure 4 The following is a workflow diagram of the group emotion prediction module in one embodiment of the present invention. First, the emotion categories corresponding to each key frame are arranged in time series and recorded in data structure 1. Data structure 1 can be an array or a linked list. Then, the frequency of occurrence of each emotion category in data structure 1 is counted. Then, for each emotion category, the product of the preset weight and the frequency of occurrence is calculated respectively, and the final result below the preset threshold can be eliminated. For example, the "neutral (which can be commonly understood as bland, without emotion)" emotion category can be eliminated, and the output group emotion recognition result is "happy" or "happy 90%, neutral 10%"... and other such expressions containing group emotion prediction results.

[0074] By adopting this embodiment of the invention, the present invention proposes a standardized group emotional atmosphere measurement method and detection framework, which can make up for the shortcomings of traditional methods in this technical field, can provide quantitative indicators for group emotions, and help further develop research related to group behavior and emotional atmosphere in psychology.

[0075] Figure 2 Schematic diagram of the structure of a group emotion prediction system in one embodiment of the present invention.

[0076] The emotion prediction system based on human intelligence structurally includes: a video stream preprocessing module 100, a lightweight emotion recognition model 200, and a group emotion prediction module 300. The video stream preprocessing module 100 is used to extract key frame images from the acquired original video stream data according to a preset frame rate, and preprocess each extracted key frame image; the lightweight emotion recognition model 200 is used to perform emotion recognition on each preprocessed key frame image, and obtain a corresponding emotion category recognition result respectively; the group emotion prediction module 300 is used to integrate all emotion category recognition results into time series data according to the time sequence of key frames, count the frequency of each emotion category in all emotion category recognition results through time series data, and calculate the comprehensive group emotion prediction result by weighting according to preset weights.

[0077] The video stream preprocessing module 100 is the starting part of the entire algorithm flow. Its function is to perform a series of preprocessing operations on the input raw video stream data, aiming to improve the accuracy and efficiency of subsequent emotion recognition and group emotion prediction.

[0078] The lightweight emotion recognition model 200 is the core part of the entire algorithm. It is built based on the multi-query attention mechanism and has the characteristics of being lightweight, which enables it to run efficiently in environments with limited computing power such as edge devices. The overall architecture of the model adopts deep separable convolution (borrowing the core idea of ​​MobileNet). By splitting the traditional convolution operation into deep convolution and point-by-point convolution, the number of parameters and calculations are greatly reduced, while retaining the ability to extract key features of facial emotions. On this basis, the multi-query attention mechanism is integrated to give the model a stronger feature capture capability. It can simultaneously focus on the correlation between different positions and different features in the facial emotion image from multiple different perspectives. It is not limited to local features, but can effectively capture long-distance dependencies, thereby more accurately extracting the deep-level features of emotions.

[0079] The group emotion prediction module 300 sorts the facial emotion recognition results corresponding to the key frames according to the time sequence of the key frames in the original video stream data, and counts the emotion frequency, constructs a time series data structure, and adds the emotion frequency counted for each frame image in sequence to form a complete time series data reflecting the change of emotion frequency over time in the entire video stream, so as to capture the frequency fluctuation of various emotions in the group at different times, and provide richer dynamic information for subsequent emotional state analysis. Preset weights for different emotions, weighted calculation and further analysis of the group emotional atmosphere.

[0080] In summary, the key points and advantages of the emotion prediction method, system, edge computing device and medium based on human intelligence proposed in the present invention are:

[0081] (1) It is conducive to lightweight real-time group emotion recognition. A standardized group emotion atmosphere measurement method and detection framework are proposed, which can effectively fill the gap in the existing technology for real-time analysis and recognition of group emotions and realize real-time and accurate group emotion analysis and recognition.

[0082] (2) It is able to perform a series of preprocessing operations on the input raw video stream data, aiming to improve the accuracy and efficiency of subsequent emotion recognition and group emotion prediction.

[0083] (3) It is possible to integrate all emotion category recognition results in the form of time series data, thereby comprehensively analyzing the emotion distribution in the group.

[0084] (4) The importance of different emotion categories can be calculated by weighted calculation. Different emotion categories are associated with different emotions, and emotion categories below the threshold are eliminated, thereby removing interference items and ensuring that the final group emotion recognition results are concise and representative.

[0085] (5) The distribution of weights can be optimized through subjective data, thereby improving the fit between the group emotion recognition results calculated based on preset weights and the actual emotional distribution state of the group, making the recognition results of this scheme more accurate.

[0086] (6) A standardized group emotional atmosphere measurement method and detection framework were proposed, which can make up for the shortcomings of traditional methods in this technical field, provide quantitative indicators for group emotions, and help further develop research on group behavior and emotional atmosphere in psychology.

[0087] Corresponding to the above method, the present invention also provides an edge computing device, which includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, the processor is used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the edge computing device implements the steps of the method described above.

[0088] Figure 5 A schematic diagram of the structure of a computer device included in an edge computing device.

[0089] See also Figure 5 The computer device 00 includes: a processor 01, a memory 02, and a computer program stored in the memory 02 and executable on the processor 01. When the processor 01 executes the computer program, the human factor data server access control method provided in the above method embodiment is implemented.

[0090] Among them, the processor 01 is connected to the memory 02, such as through a bus 03. The processor 01 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in combination with the contents disclosed in this application. The processor 01 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of DSP and microprocessors, etc. The bus 03 may include a path to transmit information between the above components. The bus 03 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 130 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The memory 02 is used to store a computer program corresponding to the human factor data server access control method of the above embodiment of the present application, and the computer program is controlled and executed by the processor 01. The processor 01 is used to execute the computer program stored in the memory 02 to implement the content shown in the above method embodiment.

[0091] Corresponding to the above method, the present invention also provides a computer-readable storage medium, on which a computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the steps of the method described in any one of the above embodiments are implemented. The computer-readable storage medium can be a tangible storage medium, such as a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a floppy disk, a hard disk, a removable storage disk, a CD-ROM, or any other form of storage medium known in the technical field.

[0092] Corresponding to the above method, the present invention also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any one of the above embodiments when executed by a processor.

[0093] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0094] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0095] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0096] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for predicting emotions based on human intelligence, characterized in that: The method comprises: Acquire original video stream data, extract key frame images from the original video stream data according to a preset frame rate, and pre-process each extracted key frame image; wherein the key frame image includes multiple face images; Perform emotion recognition on each key frame image that has been preprocessed, identify the emotion category of each face in the key frame image, and count the frequency of different emotion categories appearing in the current key frame image; According to the time sequence of key frames, the frequency of each emotion category in the images of all key frames is integrated and counted through time series data, and the comprehensive group emotion prediction result is calculated according to the preset weights.

2. The method according to claim 1, characterized in that The preprocessing of each key frame image extracted includes: Performing size normalization processing on the image of each extracted key frame, adjusting the resolution, pixel size or data specification of the image to a preset range; Perform lighting correction on the size-normalized image.

3. The method according to claim 1, characterized in that Emotion recognition is performed on each key frame of the preprocessed image, including: Perform multi-scale perception on the pre-processed image of each key frame to capture image features at different scales and obtain multi-scale perception image features; Down-sampling the multi-scale perception image features respectively to obtain a first output result, a second output result and a third output result; The emotion categories predicted for the first output result, the second output result, and the third output result are respectively determined, and a final emotion recognition result is determined according to preset weights based on the predicted emotion categories.

4. The method according to claim 3, characterized in that Downsampling the multi-scale perception image features respectively to obtain a first output result, a second output result and a third output result, including: The original image is processed by spatial depth decoupling and downsampling to obtain the initial feature map; Performing multi-query attention downsampling processing on the initial feature map to obtain a first feature map; Performing differential fusion convolution processing on the first feature map to obtain a second feature map; Performing spatial depth decoupling and downsampling processing on the second feature map to obtain a third feature map; Performing differential fusion convolution layer processing on the third feature map to obtain a fourth feature map; Performing spatial depth decoupling down-sampling processing on the fourth feature map to obtain a fifth feature map; Performing differential fusion convolution processing on the fifth feature map to obtain a sixth feature map; Performing multi-query attention processing on the sixth feature map to obtain a seventh feature map; Performing multi-query attention downsampling processing on the seventh feature map to obtain an eighth feature map; Among them, the fourth characteristic map is used as the first output result, the seventh characteristic map is used as the second output result, and the eighth characteristic map is used as the third output result.

5. The method according to claim 1, characterized in that: According to the time sequence of key frames, the frequency of each emotion category in all key frame images is integrated and counted through time series data, including: The image of each key frame has a time point label, and the corresponding emotion category recognition result has the same time point label. The time point label is used to arrange the emotion category recognition results into a time series; Construct a data structure to organize and store all emotion category recognition results in time series.

6. The method according to claim 1, characterized in that The comprehensive group sentiment prediction results are calculated according to the preset weights, including: For each emotion category, the frequency of the emotion category appearing in all emotion category recognition results multiplied by the preset weight of the emotion category is used as the importance parameter of the corresponding emotion category; The importance parameters of each emotion category are combined to calculate the proportion of each emotion category in the group; Emotion categories whose proportion is lower than a preset threshold are eliminated, and a group emotion prediction result including the remaining emotion categories and the proportion of the emotion categories in the group is generated.

7. The method according to claim 1, characterized in that The method further comprises: Obtain subjective emotion labels collected from each person in the group through subjective scales or questionnaires; The correlation between the time series data and the collected subjective emotion labels is learned through a deep learning model to optimize the preset weights as a training target of the deep learning model and adjust the preset weights.

8. An emotion prediction system based on human intelligence, characterized in that: Include: The video stream preprocessing module is used to extract key frame images from the acquired original video stream data according to a preset frame rate, and preprocess each extracted key frame image; A lightweight emotion recognition model is used to perform emotion recognition on each key frame image after preprocessing, identify the emotion category of each face in the key frame image, and count the frequency of different emotion categories appearing in the current key frame image; The group emotion prediction module is used to integrate and count the frequency of each emotion category in the images of all key frames in the form of time series data according to the time sequence of key frames, and calculate the comprehensive group emotion prediction result according to the preset weights.

9. An edge computing device comprising a processor, a memory, and a computer program / instruction stored in the memory, characterized in that: The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the edge computing device implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method as claimed in any one of claims 1 to 7 are implemented.

Images