Learning emotion adaptive adjustment method and system based on micro-expression analysis

Abstract

The invention discloses a learning emotion self-adaptive adjustment method and system based on micro-expression analysis, and belongs to the technical field of learning machines and micro-expression analys.The method comprises the steps that firstly, a face video stream of a learner is obtained in real time and divided into a plurality of time sliding windows; secondly, performing face region detection to obtain a face region image, and selecting a plurality of ROI regions from the face region image; then, calculating an optical flow vector matrix of the ROI in each frame, and calculating a facial local motion matrix of the ROI; calculating a single-frame micro-expression space feature vector of each frame, and forming a micro-expression space-time fusion matrix: finally, performing structured phase extraction on the micro-expression, and mapping real-time learning emotion of a learner, so as to perform real-time adjustment on learning parameters, and completing adaptive adjustment of the learning emotion. Through recognition of micro expressions and structured phase extraction, the learning machine can sense the emotional state of the learner in real time and make adjustment in time, and the learning efficiency of the learner is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of learning machines and micro-expression analysis, and in particular relates to a method and system for adaptively regulating learning emotions based on micro-expression analysis. Background Art

[0002] In today's era of booming digital education, online learning systems based on machine learning should be a powerful tool for learners to efficiently acquire knowledge. However, current systems of this type generally suffer from a significant shortcoming: the lack of real-time perception of learners' emotions.

[0003] In the dynamic learning process, a learner's emotional state has a crucial impact on their learning outcomes. Positive and joyful emotions help boost learning motivation and focus, while negative and irritable emotions can lead to low learning efficiency and even aversion to learning. However, existing feedback-based adjustment mechanisms suffer from significant lags. Adjustments based on learners' behavioral performance or performance feedback often occur only after their emotions have already significantly impacted the learning process. By then, negative emotions may have already caused irreversible learning losses.

[0004] While macro-expression-based emotion recognition methods can capture learners' emotional changes to a certain extent, they lack sensitivity to low-intensity emotions in learning scenarios. Emotions in learning scenarios are often subtle and complex, and learners may not make the same exaggerated facial expressions and gestures they do in daily life. Macro-expression recognition technology struggles to accurately capture these subtle emotional changes, thus failing to provide timely and effective emotional feedback to the learning system.

[0005] Document CN112257799A proposes a method for determining learners' attention through eye tracking. Although this method can reflect the learner's concentration to a certain extent, attention and emotion are not completely equivalent. It does not involve the key dimension of emotion and cannot fully understand the learner's learning status. Document US2021000492A1 uses speech analysis to identify emotions, but in actual learning environments, the surrounding environmental noise will seriously interfere with the speech signal, resulting in a significant decrease in the accuracy of emotion recognition, making it difficult to meet actual needs.

[0006] In contrast, micro-expressions have the unique advantages of being highly authentic and difficult to hide. They can truly reflect the learner's inner emotional changes without the learner being aware of it.

[0007] Therefore, there is an urgent need to provide a learning emotion adaptive regulation method and system based on micro-expression analysis that can realize micro-expression recognition and can adjust learning content in real time to achieve learners' learning emotion adaptive regulation. Summary of the Invention

[0008] The purpose of the present invention is to provide a method for adaptively regulating learning emotions based on micro-expression analysis to solve the above-mentioned problems existing in the prior art.

[0009] In order to achieve the above object, the present invention adopts the following technical solutions:

[0010] In a first aspect, the present invention provides a method for adaptively regulating learning emotions based on micro-expression analysis, comprising:

[0011] Acquire a learner's facial video stream in real time, and divide the learner's facial video stream into multiple time sliding windows;

[0012] Performing facial region detection on the first frame of each time sliding window to obtain the facial region of the first frame of each time sliding window, and performing facial cropping on the remaining frames in each time sliding window based on the facial region of the first frame to obtain a facial region image, and selecting multiple ROI regions from the facial region image;

[0013] In each time sliding window, a dense optical flow field of the ROI region in each frame is calculated to obtain an optical flow vector matrix of each ROI region, and based on the optical flow vector matrix of each ROI region, a facial local motion matrix of each ROI region is calculated in each frame;

[0014] In each time sliding window, based on the facial local motion matrix of each ROI region, the single-frame micro-expression spatial feature vector of each frame is calculated respectively, and the single-frame micro-expression spatial feature vectors of each frame are arranged sequentially to form the micro-expression spatiotemporal fusion matrix:

[0015] Based on the micro-expression spatiotemporal fusion matrix, structured phase extraction is performed on micro-expressions to obtain effective micro-expressions that conform to the structured phase pattern of micro-expressions. According to the effective micro-expressions in each ROI area, the learner's real-time learning emotions are mapped out. The original learning parameters are adjusted in real time according to the learner's real-time learning emotions to complete the adaptive adjustment of the learner's learning emotions.

[0016] In one possible design, a learner's facial video stream is acquired in real time and divided into multiple time sliding windows, including:

[0017] The learning machine camera captures the pre-learner’s facial video stream in real time;

[0018] Obtaining a preset separation time, and detecting whether there is an invalid segment in the pre-learner's facial video stream in which the learner's facial separation time exceeds the preset separation time;

[0019] If yes, the invalid segments are removed to form a learner facial video stream; if no, the pre-learner facial video stream is used as the learner facial video stream;

[0020] Obtaining a preset window size and a preset overlapping interval to segment the learner's facial video stream according to the preset window size and the preset overlapping interval to form a plurality of original time sliding windows, wherein the coverage rate of the preset overlapping interval to the preset window size is not less than 60%;

[0021] Detecting whether there is a broken frame in each of the original time sliding windows;

[0022] If so, copy the next frame of the broken frame and fill it into the broken frame, and check again whether there is a broken frame in each of the original time sliding windows until each of the original time sliding windows is in a full window state, and use each of the original time sliding windows in the full window state as the time sliding window. If not, use the original time sliding window as the time sliding window.

[0023] In one possible design, facial region detection is performed on the first frame of each time sliding window to obtain the facial region of the first frame of each time sliding window. Based on the facial region of the first frame, facial regions are cropped on the remaining frames in each time sliding window to obtain a facial region image. Multiple ROI regions are selected from the facial region image, including:

[0024] Obtaining the first frame image of each time sliding window, performing facial region detection on the first frame image of each time sliding window, so as to obtain the coordinate value of the upper left corner point of the learner's face, the learner's facial height, and the learner's facial width from the first frame image of each time sliding window;

[0025] Generate the first frame facial region of each time sliding window according to the coordinate value of the upper left corner point of the learner's face, the learner's face height, and the learner's face width obtained in the first frame image of each time sliding window;

[0026] Based on the facial region of the first frame of each time sliding window, the remaining frames in each time sliding window are respectively subjected to corresponding facial cropping to obtain facial region images of each frame;

[0027] Obtaining a preset facial key point model, defining a plurality of facial key points for the facial region image of each frame based on the preset facial key point model, and obtaining coordinate values ​​of each facial key point;

[0028] Obtaining a preset facial motion unit, and selecting a plurality of facial motion feature points from each of the facial key points based on a correspondence between the preset facial motion unit and each of the facial key points;

[0029] Using each of the selected facial motion feature points as a center point, a plurality of ROI regions are formed in the facial region of the first frame of each time sliding window, wherein the ROI region is a rectangular ROI region, and each ROI region includes at least one of the facial key points;

[0030] For each time sliding window, the ROI area in the facial area of ​​the first frame is used as the ROI area of ​​the facial area images of the remaining frames in this time sliding window, and the coordinate values ​​of each pixel point in each ROI area are obtained to form the ROI area coordinates.

[0031] In one possible design, in each time sliding window, the dense optical flow field of the ROI region in each frame is calculated to obtain the optical flow vector matrix of each ROI region, including:

[0032] Using the multi-scale pyramid optical flow algorithm, each frame is calculated frame by frame in each time sliding window to obtain the full-frame dense optical flow field between each pair of adjacent frames;

[0033] According to the coordinates of the ROI region, the dense optical flow field of each ROI region is selected from the dense optical flow field of the entire frame between each pair of adjacent frames;

[0034] From the dense optical flow field of each ROI area, the displacement vector of each pixel point in each ROI area is obtained as the optical flow vector to form the optical flow vector matrix of each ROI area;

[0035] Accordingly, based on the optical flow vector matrix of each ROI region, calculating the ROI region facial local motion matrix of each ROI region in each frame includes:

[0036] From each of the ROI regions, the ROI region corresponding to the learner's nose is selected as the relative rigid body ROI region;

[0037] According to the relative rigid body ROI area, extracting the optical flow vector matrix corresponding to the relative rigid body ROI area from the optical flow vector matrices of each ROI area as the relative rigid body ROI area optical flow vector matrix, and using the optical flow vector matrices of the remaining ROI areas as the non-rigid body ROI area optical flow vector matrices;

[0038] For the optical flow vector matrix of the relative rigid body ROI region, calculating the average direction vector and the average modulus of all optical flow vectors therein, and calculating a synthetic global motion vector based on the average direction vector and the average modulus;

[0039] The synthesized global motion vector is subtracted from each optical flow vector in each non-rigid ROI region optical flow vector matrix to obtain the ROI region facial local motion vector, and the ROI region facial local motion matrix of each ROI region is formed based on the ROI region facial local motion vector.

[0040] In one possible design, in each time sliding window, based on the facial local motion matrix of each ROI region, the single-frame micro-expression spatial feature vector of each frame is calculated, including:

[0041] Obtaining a motion direction interval, dividing the directions of the ROI region facial local motion vectors in the ROI region facial local motion matrix of each ROI region in each frame of the time sliding window, and storing each ROI region facial local motion vector into a corresponding motion direction interval according to the division result, wherein the ROI region facial local motion vectors are each element in the ROI region facial local motion matrix;

[0042] Based on the local facial motion vectors of each ROI region stored in the corresponding motion direction interval, for each ROI region, respectively, calculating the sum of the vector moduli of the local facial motion vectors of each ROI region within the corresponding motion direction interval, and using this sum as the direction modulus value of the motion direction interval;

[0043] Comparing the directional modulus values ​​of each motion direction interval to obtain a maximum value, and selecting the motion direction interval corresponding to the maximum value as the main motion direction interval;

[0044] For each ROI region, calculating the main motion direction average direction vector and the main motion direction average modulus of all ROI region facial local motion vectors corresponding to the main motion direction interval of each ROI region, and calculating the main motion direction feature vector of each ROI region based on the main motion direction average direction vector and the main motion direction average modulus;

[0045] The main motion direction feature vectors of each ROI area are spliced ​​to form a single-frame micro-expression spatial feature vector;

[0046] Accordingly, the single-frame micro-expression spatial feature vectors of each frame are sequentially arranged to form a micro-expression spatiotemporal fusion matrix, which includes:

[0047] Obtain the number of frames and the timestamp of each frame in each time sliding window, and establish an initialization time series matrix based on the number of frames in each time sliding window, wherein the initialization time series matrix is ​​an empty matrix, and the number of rows of the matrix is ​​determined by the number of frames in each time sliding window;

[0048] According to the timestamps of each frame, the single-frame micro-expression spatial feature vectors of each frame are arranged in sequence, and the single-frame micro-expression spatial feature vectors of each frame after the arrangement are filled into the initialized time series matrix row by row to form a micro-expression spatiotemporal fusion matrix, wherein any row of the micro-expression spatiotemporal fusion matrix represents the single-frame micro-expression spatial feature vector of any frame in the corresponding time sequence, and any column of the micro-expression spatiotemporal fusion matrix represents the temporal change trajectory of a single ROI area in different frames.

[0049] In one possible design, based on the micro-expression spatiotemporal fusion matrix, structured phase extraction is performed on micro-expressions to obtain valid micro-expressions that conform to the micro-expression structured phase pattern, including:

[0050] Based on the micro-expression spatiotemporal fusion matrix, the temporal change trajectory of each ROI region is obtained, and the temporal change trajectory of each ROI region is subjected to temporal trajectory filtering and smoothing noise reduction processing to obtain a smooth temporal change trajectory of each ROI region;

[0051] For the smooth temporal change trajectory of each ROI area, calculate the global maximum value of the trajectory within the smooth temporal change trajectory of each ROI area;

[0052] Based on the global maximum value of the trajectory corresponding to each ROI area, a micro-expression peak threshold is set. In the smooth time-series change trajectory of each ROI area, the frames in each ROI area that reach or exceed the micro-expression peak threshold are selected as the peak frames of the corresponding ROI area;

[0053] The peak frames in each ROI region are clustered and merged to form the candidate micro-expression regions of each ROI region;

[0054] From the candidate micro-expression regions of each ROI region, count the number of positive angle frames and the number of negative angle frames, and calculate the difference between the positive angle frame number and the negative angle frame number to obtain the micro-expression phase difference of each ROI region, wherein the positive angle frame number is used to represent the frame number corresponding to the start of the micro-expression, and the negative angle frame number is used to represent the frame number corresponding to the release of the micro-expression;

[0055] Obtain a valid micro-expression phase difference threshold, and for each ROI region, determine whether its micro-expression phase difference does not exceed the valid micro-expression phase difference threshold. If so, the candidate micro-expression region of this ROI region is considered to be a valid micro-expression; if not, the candidate micro-expression region of this ROI region is considered to be an invalid micro-expression.

[0056] In one possible design, the learner's real-time learning emotions are mapped based on the effective micro-expressions in each ROI area, and the original learning parameters are adjusted in real time based on the learner's real-time learning emotions to achieve adaptive adjustment of the learner's learning emotions, including:

[0057] In each time sliding window, the regional dominance coefficient of each ROI area is calculated;

[0058] Based on the regional dominance coefficient of each ROI area and combined with the effective micro-expressions of each ROI area, learning emotion mapping is performed to obtain preliminary learning emotions;

[0059] Counting the number of occurrences of the preliminary learned emotions in a continuous time sliding window, and performing secondary mapping on the preliminary learned emotions to obtain deep learned emotions;

[0060] The deep learning emotion is used as the learner's real-time learning emotion, and the learning parameters of the learning machine are adjusted according to the learner's real-time learning emotion to complete adaptive adjustment of the learner's learning emotion, wherein the learning parameters include learning time, question difficulty, rest time, answering interaction mode and answering interface elements.

[0061] In a second aspect, the present invention provides a learning emotion adaptive regulation system based on micro-expression analysis, comprising:

[0062] A video acquisition unit, configured to acquire a learner's facial video stream in real time and divide the learner's facial video stream into a plurality of time sliding windows;

[0063] a face detection unit, configured to perform face region detection at the first frame of each time sliding window to obtain a face region of the first frame of each time sliding window, and perform face cropping on the remaining frames in each time sliding window based on the face region of the first frame to obtain a face region image, and select multiple ROI regions from the face region image;

[0064] an optical flow matrix calculation unit, configured to calculate a dense optical flow field of the ROI region in each frame in each time sliding window, obtain an optical flow vector matrix of each ROI region, and calculate an ROI region facial local motion matrix of each ROI region in each frame based on the optical flow vector matrix of each ROI region;

[0065] The feature matrix calculation unit is used to calculate the single-frame micro-expression spatial feature vector of each frame based on the ROI area facial local motion matrix of each ROI area in each time sliding window, and sequentially arrange the single-frame micro-expression spatial feature vectors of each frame to form a micro-expression spatiotemporal fusion matrix:

[0066] The adaptive adjustment unit is used to perform structured phase extraction of micro-expressions based on the micro-expression spatiotemporal fusion matrix to obtain effective micro-expressions that conform to the structured phase pattern of micro-expressions, and map the learning emotion classification according to the effective micro-expressions in each ROI area. The original learning parameters are adjusted in real time through the learning emotion classification to complete the adaptive adjustment of the learner's learning emotions.

[0067] In a third aspect, the present invention provides an electronic device comprising a memory, a processor and a transceiver communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the method for learning adaptive emotion regulation based on micro-expression analysis as described in the first aspect or any possible design of the first aspect.

[0068] In a fourth aspect, the present invention provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed on a computer, the method for learning adaptive emotion regulation based on micro-expression analysis described in the first aspect or any possible design of the first aspect is executed.

[0069] In a fifth aspect, the present invention provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the method for learning adaptive emotion regulation based on micro-expression analysis as described in the first aspect or any possible design of the first aspect.

[0070] Beneficial effect: The present invention provides a method and system for adaptively regulating learning emotions based on micro-expression analysis. First, a learner's facial video stream is acquired in real time, and the learner's facial video stream is divided into multiple time sliding windows. Second, facial region detection is performed at the first frame of each time sliding window to obtain the first frame facial region of each time sliding window, and facial cropping is performed on the remaining frames in each time sliding window based on the first frame facial region to obtain a facial region image, and multiple ROI regions are selected from the facial region image. Then, in each time sliding window, a dense optical flow field of the ROI region in each frame is calculated to obtain an optical flow vector matrix of each ROI region, and based on the optical flow vector matrix of each ROI region, The ROI area facial local motion matrix of each ROI area is calculated in each frame; then, in each time sliding window, the single-frame micro-expression spatial feature vector of each frame is calculated based on the ROI area facial local motion matrix of each ROI area, and the single-frame micro-expression spatial feature vectors of each frame are arranged in sequence to form a micro-expression spatiotemporal fusion matrix: finally, based on the micro-expression spatiotemporal fusion matrix, structured phase extraction is performed on the micro-expression to obtain effective micro-expressions that conform to the structured phase pattern of the micro-expression, and the learner's real-time learning emotions are mapped according to the effective micro-expressions of each ROI area, and the original learning parameters are adjusted in real time according to the learner's real-time learning emotions to complete the adaptive adjustment of the learner's learning emotions. Facial videos are segmented through a sliding window, local optical flow features of multiple ROI areas are extracted within the window, and a spatiotemporal fusion matrix is ​​constructed. Micro-expressions are used for structured phase extraction to map real-time learning emotions, and learning parameters such as learning content difficulty, learning duration, and interaction mode are adjusted based on real-time learning emotions. Combining physiological data with a reinforcement learning optimization system enables the learning machine to perceive the learner's emotional state in real time and make timely adjustments, significantly improving the learner's learning efficiency and learning effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A schematic flow chart of the steps of a method for learning adaptive emotion regulation based on micro-expression analysis provided by an embodiment of the present invention;

[0072] Figure 2 A schematic diagram of the functional structure of a learning emotion adaptive regulation system based on micro-expression analysis provided by an embodiment of the present invention;

[0073] Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the present invention will be briefly introduced below in conjunction with the drawings and the description of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work. It should be noted that the description of these embodiments is used to help understand the present invention, but does not constitute a limitation of the present invention.

[0075] It should be understood that although the terms "first," "second," etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element can be referred to as a second element, and similarly, a second element can be referred to as a first element without departing from the scope of the exemplary embodiments of the present invention.

[0076] It should be understood that the term "and / or" that may appear in this document is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may indicate three situations: A exists alone, B exists alone, and A and B exist at the same time. The term " / and" that may appear in this document describes another type of association object relationship, indicating that two relationships may exist. For example, A / and B may indicate two situations: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the previous and subsequent associated objects are in an "or" relationship.

[0077] Example:

[0078] like Figure 1 As shown, this embodiment provides a method for adaptively regulating learning emotions based on micro-expression analysis, which may include but is not limited to the following steps:

[0079] S1 real-time acquisition of learner facial video stream, and the learner facial video stream is divided into multiple time sliding windows;

[0080] In one possible implementation, in step S1, a learner's facial video stream is acquired in real time, and the learner's facial video stream is divided into multiple time sliding windows, which can be decomposed into, but not limited to, the following steps S11-S16, including:

[0081] S11. Capture the pre-learner’s facial video stream in real time through the learning machine camera;

[0082] S12 obtains the preset separation time, the pre-learner facial video stream, detects whether there is an invalid segment where the learner's face is separated from the video for a duration exceeding the preset separation time;

[0083] S13. If so, the invalid segments are removed to form a learner facial video stream. If not, the pre-learner facial video stream is used as the learner facial video stream.

[0084] S14. Obtaining a preset window size and a preset overlapping interval, and dividing the learner's facial video stream according to the preset window size and the preset overlapping interval to form a plurality of original time sliding windows, wherein the coverage rate of the preset overlapping interval to the preset window size is not less than 60%;

[0085] S15. Detecting whether there is a broken frame in each of the original time sliding windows;

[0086] S16. If so, copy the next frame of the broken frame and fill it into the broken frame, and check again whether there is a broken frame in each of the original time sliding windows until each of the original time sliding windows is in a full window state, and use each of the original time sliding windows in the full window state as the time sliding window. If not, use the original time sliding window as the time sliding window.

[0087] It should be noted that in actual implementation, the preset window size can be set to 0.5 seconds, the preset overlap interval can be set to 0.3 seconds, and the preset sliding step size is 0.2 seconds. This is because a learner's micro-expression may span multiple windows. In the worst case, both windows only contain half of the micro-expression. Therefore, the preset overlap interval covers at least 60% of the preset window size to avoid this problem.

[0088] S2. Performing facial region detection at the first frame of each time sliding window to obtain the facial region of the first frame of each time sliding window, and performing facial cropping on the remaining frames in each time sliding window based on the facial region of the first frame to obtain a facial region image, and selecting multiple ROI regions from the facial region image;

[0089] In one possible implementation, in step S2, facial region detection is performed on the first frame of each time sliding window to obtain the facial region of the first frame of each time sliding window, and facial cropping is performed on the remaining frames in each time sliding window based on the facial region of the first frame to obtain a facial region image. Multiple ROI regions are selected from the facial region image, which can be decomposed into, but not limited to, the following steps S21-S27, including:

[0090] S21. Obtain the first frame image of each time sliding window, perform facial region detection on the first frame image of each time sliding window, and obtain the coordinate value of the upper left corner point of the learner's face, the learner's face height, and the learner's face width from the first frame image of each time sliding window;

[0091] S22. Generate the facial region of the first frame of each time sliding window based on the coordinates of the upper left corner of the learner's face, the learner's face height, and the learner's face width obtained in the first frame image of each time sliding window;

[0092] S23. Based on the facial region of the first frame of each time sliding window, the remaining frames in each time sliding window are respectively subjected to corresponding facial cropping to obtain facial region images of each frame;

[0093] S24 obtains a preset facial key point model, defines a plurality of facial key points for each frame of the facial region image based on the preset facial key point model, and obtains the coordinate value of each facial key point;

[0094] S25. Obtaining a preset facial motion unit, and selecting a plurality of facial motion feature points from each of the facial key points according to the correspondence between the preset facial motion unit and each of the facial key points;

[0095] S26. Using each of the selected facial motion feature points as a center point, a plurality of ROI regions are formed in the facial region of the first frame of each time sliding window, wherein the ROI region is a rectangular ROI region, and each ROI region includes at least one of the facial key points;

[0096] S27. For each time sliding window, the ROI region in the facial region of the first frame is used as the ROI region of the facial region images of the remaining frames in this time sliding window, and the coordinate value of each pixel point in each ROI region is obtained to form the ROI region coordinates.

[0097] It should be noted that in actual implementation, the preset facial key point model can be built based on Dlib to form a 68-point model. Through this 68-point model, 68 key points can be defined for the facial area image of each frame. These key points cover parts such as eyebrows, eyes, nose, mouth and facial contours. In order to improve the efficiency of micro-expression recognition, this embodiment selects multiple facial motion feature points from each of the facial key points according to the facial position that will change significantly when micro-expressions appear, and determines at least 8 ROI areas. The shape of each ROI area is rectangular, and their center point is determined by the selected facial motion feature point. Each ROI area contains a large amount of local information. By using this local information, the robustness of micro-expression detection in this embodiment can be improved.

[0098] The pre-defined facial movement units are called AUs (Action Units). AUs describe the degree of movement of different parts of a learner's facial expressions, with each AU representing a change in a specific part of the facial expression. These units can be analyzed and recognized independently, and AUs can be used to describe different aspects of facial expressions, such as the degree of eye openness, eyebrow raise, and mouth opening. By corresponding AUs with various facial key points, any expression can be quantitatively represented, including overall expressions such as happiness, sadness, and surprise.

[0099] S3. In each time sliding window, a dense optical flow field of the ROI region in each frame is calculated to obtain an optical flow vector matrix of each ROI region, and based on the optical flow vector matrix of each ROI region, a facial local motion matrix of each ROI region is calculated in each frame;

[0100] In one possible implementation, in step S3, in each time sliding window, a dense optical flow field of the ROI region in each frame is calculated to obtain an optical flow vector matrix of each ROI region, which can be decomposed into, but not limited to, the following steps S31-S33, including:

[0101] S31. Using a multi-scale pyramid optical flow algorithm, calculate each frame one by one in each time sliding window to obtain a full-frame dense optical flow field between each pair of adjacent frames;

[0102] S32. Selecting the dense optical flow field of each ROI region from the dense optical flow field of the entire frame between each pair of adjacent frames according to the ROI region coordinates;

[0103] S33. Obtain the displacement vector of each pixel in each ROI region from the dense optical flow field of each ROI region as an optical flow vector to form an optical flow vector matrix for each ROI region;

[0104] Accordingly, in step S3, based on the optical flow vector matrix of each ROI region, the ROI region facial local motion matrix of each ROI region is calculated in each frame, which can be decomposed into, but not limited to, the following steps S34-S37, including:

[0105] S34. From each of the ROI regions, select the ROI region corresponding to the learner's nose as the relative rigid body ROI region;

[0106] S35. According to the relative rigid body ROI region, extract the optical flow vector matrix corresponding to the relative rigid body ROI region from the optical flow vector matrices of each ROI region as the optical flow vector matrix of the relative rigid body ROI region, and use the optical flow vector matrices of the remaining ROI regions as the optical flow vector matrices of the non-rigid body ROI region;

[0107] S36. Calculate the average direction vector and average modulus of all optical flow vectors in the optical flow vector matrix relative to the rigid body ROI region, and calculate a synthetic global motion vector based on the average direction vector and the average modulus;

[0108] S37. Subtract the synthesized global motion vector from each optical flow vector in the optical flow vector matrix of each non-rigid ROI region to obtain the ROI region facial local motion vector, and form the ROI region facial local motion matrix of each ROI region based on the ROI region facial local motion vector.

[0109] It should be noted that, according to the actual implementation situation, this embodiment can divide the learner's facial area into at least four parts, including the left eyebrow part, the right eyebrow part, the nose part and the mouth part. Among them, the left eyebrow part and the right eyebrow part can be represented by 2 ROI areas respectively, the nose part is represented by 1 ROI area, and the mouth part is represented by 3 ROI areas. In order to reduce the impact of head movement on micro-expression detection, this embodiment selects the ROI area corresponding to the learner's nose part as the relative rigid body ROI area, and estimates the overall head movement (represented by synthesizing the global motion vector) by calculating the average direction vector and average modulus of all optical flow vectors in the relative rigid body ROI area to ensure the validity of the local facial motion matrix of the ROI area.

[0110] S4. In each time sliding window, based on the facial local motion matrix of each ROI region, calculate the single-frame micro-expression spatial feature vector of each frame, and sequentially arrange the single-frame micro-expression spatial feature vectors of each frame to form a micro-expression spatiotemporal fusion matrix:

[0111] In one possible implementation, in step S4, in each time sliding window, based on the ROI region facial local motion matrix of each ROI region, a single-frame micro-expression spatial feature vector of each frame is calculated, which can be decomposed into, but not limited to, the following steps S41-S45, including:

[0112] S41. Obtaining motion direction intervals. In each frame of the temporal sliding window, dividing the directions of the ROI region facial local motion vectors in the ROI region facial local motion matrix for each ROI region, and storing each ROI region facial local motion vector into a corresponding motion direction interval based on the division result, wherein the ROI region facial local motion vectors are each element in the ROI region facial local motion matrix;

[0113] S42. Based on the local facial motion vectors of each ROI region stored in the corresponding motion direction interval, for each ROI region, calculate the sum of the vector moduli of the local facial motion vectors of each ROI region within the corresponding motion direction interval, and use this sum as the direction modulus value of the motion direction interval;

[0114] S43. Compare the directional modulus values ​​of each motion direction interval to obtain the maximum value, and select the motion direction interval corresponding to the maximum value as the main motion direction interval;

[0115] S44. For each ROI region, calculate the average main motion direction vector and the average main motion direction modulus of the facial local motion vectors of all ROI regions corresponding to the main motion direction interval of each ROI region, and calculate the main motion direction feature vector of each ROI region based on the average main motion direction vector and the average main motion direction modulus;

[0116] S45. Feature concatenation of the main motion direction feature vectors of each ROI region to form a single-frame micro-expression spatial feature vector;

[0117] Accordingly, in step S4, the single-frame micro-expression spatial feature vectors of each frame are sequentially arranged to form a micro-expression spatiotemporal fusion matrix, which can be decomposed into, but not limited to, the following steps S46-S47 including:

[0118] S46. Obtain the number of frames and the timestamp of each frame in each time sliding window, and establish an initialization time series matrix based on the number of frames in each time sliding window, wherein the initialization time series matrix is ​​an empty matrix, and the number of rows of the matrix is ​​determined by the number of frames in each time sliding window;

[0119] S47. Arrange the single-frame micro-expression spatial feature vectors of each frame in sequence according to the timestamp of each frame, and fill the single-frame micro-expression spatial feature vectors of each frame after the arrangement into the initialized time series matrix row by row to form a micro-expression spatiotemporal fusion matrix, wherein any row of the micro-expression spatiotemporal fusion matrix represents the single-frame micro-expression spatial feature vector of any frame in the corresponding time sequence, and any column of the micro-expression spatiotemporal fusion matrix represents the temporal change trajectory of a single ROI area in different frames.

[0120] It should be noted that this embodiment divides the direction of the ROI area facial local motion vector in the ROI area facial local motion matrix of each ROI area, divides the direction of 2π (-π, π) into 4 motion direction intervals, and assigns a number to each motion direction interval to represent its direction. For each pixel in each ROI area, the direction angle of the pixel optical flow is first calculated to determine the motion direction interval to which it belongs; then the modulus of the optical flow is calculated and added to the corresponding motion direction interval; the motion direction interval with the largest direction modulus value is selected as the main motion direction interval, and all optical flow vectors in the main motion direction interval are vector-accumulated to obtain the facial local motion vectors of each ROI area in the main motion direction, and the facial local motion vectors of each ROI area in the main motion direction are projected from the Cartesian coordinate system to the polar coordinate system for feature extraction to obtain the main motion direction feature vectors of each ROI area. In order to fuse the spatial features of the micro-expression video, this embodiment preferably performs feature splicing on the main motion direction feature vectors extracted from each non-rigid ROI area excluding the nose part to form a single-frame micro-expression spatial feature vector.

[0121] The micro-expression spatiotemporal fusion matrix in this embodiment contains the spatiotemporal information of micro-expressions. Each row in the micro-expression spatiotemporal fusion matrix represents the movement of multiple ROI areas on the face. When a micro-expression occurs, multiple related AU units in the learner's facial area are activated, and the rows of the micro-expression spatiotemporal fusion matrix reflect the momentum distribution of the learner's facial space. Each column of the micro-expression spatiotemporal fusion matrix represents the temporal change trajectory of a single ROI area in different frames (that is, the change of each ROI area over time). The columns of the micro-expression spatiotemporal fusion matrix reflect the distribution of fixed areas in the time domain. Therefore, using the micro-expression spatiotemporal fusion matrix to realize micro-expression detection significantly improves the accuracy of micro-expression detection compared to using only a single dimension for detection.

[0122] S5. Based on the micro-expression spatiotemporal fusion matrix, perform structured phase extraction on micro-expressions to obtain effective micro-expressions that conform to the structured phase pattern of micro-expressions. According to the effective micro-expressions in each ROI area, map the learner's real-time learning emotions. The original learning parameters are adjusted in real time based on the learner's real-time learning emotions to complete the adaptive adjustment of the learner's learning emotions.

[0123] In one possible implementation, in step S5, based on the micro-expression spatiotemporal fusion matrix, structured phase extraction is performed on the micro-expression to obtain a valid micro-expression that conforms to the micro-expression structured phase pattern. This can be decomposed into, but is not limited to, the following steps S51-S56, including:

[0124] S51. Based on the micro-expression spatiotemporal fusion matrix, the temporal change trajectory of each ROI region is obtained, and the temporal change trajectory of each ROI region is filtered and smoothed to obtain a smooth temporal change trajectory of each ROI region;

[0125] S52. For each ROI region's smooth temporal change trajectory, calculate the global maximum value of the trajectory within the smooth temporal change trajectory of each ROI region;

[0126] S53. Based on the global maximum value of the trajectory corresponding to each ROI region, a micro-expression peak threshold is set. In the smooth temporal variation trajectory of each ROI region, frames that reach or exceed the micro-expression peak threshold in each ROI region are selected as the peak frames of the corresponding ROI region;

[0127] S54. Perform peak cluster merging on the peak frames in each ROI region to form a candidate micro-expression region for each ROI region;

[0128] S55. Count the number of positive angle frames and negative angle frames within the candidate micro-expression regions of each ROI region, and calculate the difference between the positive angle frames and the negative angle frames to obtain the micro-expression phase difference of each ROI region, wherein the positive angle frames are used to represent the frame number corresponding to the start of the micro-expression, and the negative angle frames are used to represent the frame number corresponding to the release of the micro-expression;

[0129] S56. Obtain a valid micro-expression phase difference threshold, and for each ROI region, determine whether its micro-expression phase difference does not exceed the valid micro-expression phase difference threshold. If so, the candidate micro-expression region of this ROI region is considered to be a valid micro-expression; if not, the candidate micro-expression region of this ROI region is considered to be an invalid micro-expression.

[0130] In one possible implementation, in step S5, the learner's real-time learning emotions are mapped based on the effective micro-expressions in each ROI area, and the original learning parameters are adjusted in real time based on the learner's real-time learning emotions to achieve adaptive adjustment of the learner's learning emotions. This can be decomposed into, but is not limited to, the following steps S57-S59, including:

[0131] S57. Calculate the regional dominance coefficient of each ROI region in each time sliding window;

[0132] S58. Based on the regional dominance coefficient of each ROI region and combined with the effective micro-expressions of each ROI region, learning emotion mapping is performed to obtain preliminary learning emotions;

[0133] S59. Counting the number of occurrences of the preliminary learning emotions in the continuous time sliding window, and performing secondary mapping on the preliminary learning emotions to obtain deep learning emotions;

[0134] S510. Use the deep learning emotion as the learner's real-time learning emotion, and adjust the learning parameters of the learning machine according to the learner's real-time learning emotion to complete adaptive adjustment of the learner's learning emotion, wherein the learning parameters include learning time, question difficulty, rest time, answering interaction mode and answering interface elements.

[0135] It should be noted that micro-expression movement has a unique structured phase pattern. Specifically, micro-expression movement can be decomposed into three movement phases: onset phase (Onset Phase), top frame phase (Apex Phase) and release phase (Offset Phase). Among them, the onset phase is used to indicate facial muscle contraction, and the optical flow angle moves forward. The apex phase is used to indicate the peak intensity of facial movement. The release phase is used to indicate facial muscle relaxation, and the optical flow angle moves in the opposite direction. The onset phase and the release phase form a symmetrical or approximately symmetrical phase structure on the motion trajectory.

[0136] Therefore, first, the Savitzky-Golay filter can be used to perform temporal trajectory filtering and smoothing noise reduction on the temporal change trajectory of each ROI area. Secondly, based on the global maximum value of the trajectory of each ROI area, the micro-expression peak threshold is set to locate the candidate peak and determine the candidate micro-expression area. Then, the symmetry of the candidate micro-expression area is verified according to the effective micro-expression phase difference threshold (set according to the symmetrical phase structure), and finally the effective micro-expression is obtained.

[0137] The effective micro-expressions are converted into computable emotion indices through the regional dominance coefficients of each ROI area. The learner's emotions are preliminarily mapped based on the regional dominance coefficients of each ROI area and the effective micro-expressions of each ROI area to determine the preliminary learning emotions, such as confusion, boredom, and excitement. The number of occurrences of the preliminary learning emotions in a continuous time sliding window is then counted, and the preliminary learning emotions are secondary mapped to obtain deep learning emotions, such as persistent confusion, intermittent confusion, and instantaneous confusion.

[0138] Based on deep learning emotions, real-time adjustments to learning parameters (study time, question difficulty, rest time, interactive mode for answering questions, and elements of the answering interface) must conform to existing cognitive load theory. For example, if a learner's real-time learning emotion is detected as persistent confusion, the learning machine will appropriately reduce study time, lower question difficulty, increase rest time, issue voice prompts, and render key prompt words on the answering interface. This real-time learning adjustment can effectively maintain learners' learning interest and relieve learning pressure, thereby achieving adaptive adjustment of learners' learning emotions.

[0139] In addition, the learning emotion adaptive regulation method based on micro-expression analysis provided in this embodiment can also be used to achieve dual closed-loop control, namely: learner emotion detection (feedforward loop) and learner learning effect feedback (feedback loop). The two together drive learners to perform learning emotion adaptive regulation, which can not only effectively reduce learners' learning pressure, but also greatly improve learners' learning efficiency in learning.

[0140] like Figure 2 As shown, the second aspect of this embodiment provides a hardware system for implementing the method for learning and adaptively regulating emotions based on micro-expression analysis described in the first aspect of the embodiment, including:

[0141] A video acquisition unit, configured to acquire a learner's facial video stream in real time and divide the learner's facial video stream into a plurality of time sliding windows;

[0142] a face detection unit, configured to perform face region detection at the first frame of each time sliding window to obtain a face region of the first frame of each time sliding window, and perform face cropping on the remaining frames in each time sliding window based on the face region of the first frame to obtain a face region image, and select multiple ROI regions from the face region image;

[0143] an optical flow matrix calculation unit, configured to calculate a dense optical flow field of the ROI region in each frame in each time sliding window, obtain an optical flow vector matrix of each ROI region, and calculate an ROI region facial local motion matrix of each ROI region in each frame based on the optical flow vector matrix of each ROI region;

[0144] The feature matrix calculation unit is used to calculate the single-frame micro-expression spatial feature vector of each frame based on the ROI area facial local motion matrix of each ROI area in each time sliding window, and sequentially arrange the single-frame micro-expression spatial feature vectors of each frame to form a micro-expression spatiotemporal fusion matrix:

[0145] The adaptive adjustment unit is used to perform structured phase extraction of micro-expressions based on the micro-expression spatiotemporal fusion matrix to obtain effective micro-expressions that conform to the structured phase pattern of micro-expressions, and map the learning emotion classification according to the effective micro-expressions in each ROI area. The original learning parameters are adjusted in real time through the learning emotion classification to complete the adaptive adjustment of the learner's learning emotions.

[0146] The working process, working details and technical effects of the system provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0147] like Figure 3 As shown, the third aspect of this embodiment provides an electronic device, comprising: a memory, a processor and a transceiver that are communicatively connected in sequence, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the learning emotion adaptive regulation method based on micro-expression analysis as described in the first aspect of the embodiment.

[0148] For example, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in first-out memory (FIFO), and / or first-in last-out memory (FILO); specifically, the processor may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. The processor may be implemented in at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Furthermore, the processor may include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); and the coprocessor is a low-power processor for processing data in a standby state.

[0149] In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. For example, the processor may be, but is not limited to, a microprocessor of the STM32F105 series, a reduced instruction set computer (RISC) microprocessor, an X86 architecture processor, or a processor with an integrated embedded neural network processing unit (NPU); the transceiver may be, but is not limited to, a wireless fidelity (WIFI) wireless transceiver, a Bluetooth wireless transceiver, a general packet radio service technology (GPRS) wireless transceiver, a ZigBee protocol (a low-power local area network protocol based on the IEEE802.15.4 standard, ZigBee) wireless transceiver, a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. In addition, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0150] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0151] The fourth aspect of this embodiment provides a storage medium that stores instructions for the method for learning adaptive emotion regulation based on micro-expression analysis as described in the first aspect of the embodiment, that is, the storage medium stores instructions, and when the instructions are run on a computer, the method for learning adaptive emotion regulation based on micro-expression analysis as described in the first aspect of the embodiment is executed.

[0152] The storage medium refers to a carrier for storing data, which may include but is not limited to a floppy disk, an optical disk, a hard disk, a flash memory, a USB flash drive and / or a memory stick, and the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0153] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment and will not be described in detail here.

[0154] A fifth aspect of this embodiment provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the method for learning adaptive emotion regulation based on micro-expression analysis as described in the first aspect of the embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0155] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

[0156] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

Claims

1. A learning emotion adaptive regulation method based on micro-expression analysis, characterized in that: include: Acquire a learner's facial video stream in real time, and divide the learner's facial video stream into multiple time sliding windows; Performing facial region detection on the first frame of each time sliding window to obtain the facial region of the first frame of each time sliding window, and performing facial cropping on the remaining frames in each time sliding window based on the facial region of the first frame to obtain a facial region image, and selecting multiple ROI regions from the facial region image; In each time sliding window, a dense optical flow field of the ROI region in each frame is calculated to obtain an optical flow vector matrix of each ROI region, and based on the optical flow vector matrix of each ROI region, a facial local motion matrix of each ROI region is calculated in each frame; In each time sliding window, based on the facial local motion matrix of each ROI region, the single-frame micro-expression spatial feature vector of each frame is calculated respectively, and the single-frame micro-expression spatial feature vectors of each frame are arranged sequentially to form the micro-expression spatiotemporal fusion matrix: Based on the micro-expression spatiotemporal fusion matrix, structured phase extraction is performed on micro-expressions to obtain effective micro-expressions that conform to the structured phase pattern of micro-expressions. According to the effective micro-expressions in each ROI area, the learner's real-time learning emotions are mapped out. The original learning parameters are adjusted in real time according to the learner's real-time learning emotions to complete the adaptive adjustment of the learner's learning emotions.

2. The method for adaptively regulating learning emotions based on micro-expression analysis according to claim 1, characterized in that: Acquire a learner's facial video stream in real time, and segment the learner's facial video stream into multiple time sliding windows, including: The learning machine camera captures the pre-learner’s facial video stream in real time; Obtaining a preset separation time, and detecting whether there is an invalid segment in the pre-learner's facial video stream in which the learner's facial separation time exceeds the preset separation time; If yes, the invalid segments are removed to form a learner facial video stream; if no, the pre-learner facial video stream is used as the learner facial video stream; Obtaining a preset window size and a preset overlapping interval to segment the learner's facial video stream according to the preset window size and the preset overlapping interval to form a plurality of original time sliding windows, wherein the coverage rate of the preset overlapping interval to the preset window size is not less than 60%; Detecting whether there is a broken frame in each of the original time sliding windows; If so, copy the next frame of the broken frame and fill it into the broken frame, and check again whether there is a broken frame in each of the original time sliding windows until each of the original time sliding windows is in a full window state, and use each of the original time sliding windows in the full window state as the time sliding window. If not, use the original time sliding window as the time sliding window.

3. The method for adaptively regulating learning emotions based on micro-expression analysis according to claim 1, characterized in that: Perform facial region detection on the first frame of each time sliding window to obtain the facial region of the first frame of each time sliding window, and perform facial cropping on the remaining frames in each time sliding window based on the facial region of the first frame to obtain a facial region image. Select multiple ROI regions from the facial region image, including: Obtaining the first frame image of each time sliding window, performing facial region detection on the first frame image of each time sliding window, so as to obtain the coordinate value of the upper left corner point of the learner's face, the learner's facial height, and the learner's facial width from the first frame image of each time sliding window; Generate the first frame facial region of each time sliding window according to the coordinate value of the upper left corner point of the learner's face, the learner's face height, and the learner's face width obtained in the first frame image of each time sliding window; Based on the facial region of the first frame of each time sliding window, the remaining frames in each time sliding window are respectively subjected to corresponding facial cropping to obtain facial region images of each frame; Obtaining a preset facial key point model, defining a plurality of facial key points for the facial region image of each frame based on the preset facial key point model, and obtaining coordinate values ​​of each facial key point; Obtaining a preset facial motion unit, and selecting a plurality of facial motion feature points from each of the facial key points based on a correspondence between the preset facial motion unit and each of the facial key points; Using each of the selected facial motion feature points as a center point, a plurality of ROI regions are formed in the facial region of the first frame of each time sliding window, wherein the ROI region is a rectangular ROI region, and each ROI region includes at least one of the facial key points; For each time sliding window, the ROI area in the facial area of ​​the first frame is used as the ROI area of ​​the facial area images of the remaining frames in this time sliding window, and the coordinate values ​​of each pixel point in each ROI area are obtained to form the ROI area coordinates.

4. The method for adaptively regulating learning emotions based on micro-expression analysis according to claim 3, characterized in that: In each time sliding window, the dense optical flow field of the ROI area in each frame is calculated to obtain the optical flow vector matrix of each ROI area, including: Using the multi-scale pyramid optical flow algorithm, each frame is calculated frame by frame in each time sliding window to obtain the full-frame dense optical flow field between each pair of adjacent frames; According to the coordinates of the ROI region, the dense optical flow field of each ROI region is selected from the dense optical flow field of the entire frame between each pair of adjacent frames; From the dense optical flow field of each ROI area, the displacement vector of each pixel point in each ROI area is obtained as the optical flow vector to form the optical flow vector matrix of each ROI area; Accordingly, based on the optical flow vector matrix of each ROI region, calculating the ROI region facial local motion matrix of each ROI region in each frame includes: From each of the ROI regions, the ROI region corresponding to the learner's nose is selected as the relative rigid body ROI region; According to the relative rigid body ROI area, extracting the optical flow vector matrix corresponding to the relative rigid body ROI area from the optical flow vector matrices of each ROI area as the relative rigid body ROI area optical flow vector matrix, and using the optical flow vector matrices of the remaining ROI areas as the non-rigid body ROI area optical flow vector matrices; For the optical flow vector matrix of the relative rigid body ROI region, calculating the average direction vector and the average modulus of all optical flow vectors therein, and calculating a synthetic global motion vector based on the average direction vector and the average modulus; The synthesized global motion vector is subtracted from each optical flow vector in each non-rigid ROI region optical flow vector matrix to obtain the ROI region facial local motion vector, and the ROI region facial local motion matrix of each ROI region is formed based on the ROI region facial local motion vector.

5. The method for adaptively regulating learning emotions based on micro-expression analysis according to claim 1, characterized in that: In each time sliding window, based on the facial local motion matrix of each ROI region, the single-frame micro-expression spatial feature vector of each frame is calculated, including: Obtaining a motion direction interval, dividing the directions of the ROI region facial local motion vectors in the ROI region facial local motion matrix of each ROI region in each frame of the time sliding window, and storing each ROI region facial local motion vector into a corresponding motion direction interval according to the division result, wherein the ROI region facial local motion vectors are each element in the ROI region facial local motion matrix; Based on the local facial motion vectors of each ROI region stored in the corresponding motion direction interval, for each ROI region, respectively, calculating the sum of the vector moduli of the local facial motion vectors of each ROI region within the corresponding motion direction interval, and using this sum as the direction modulus value of the motion direction interval; Comparing the directional modulus values ​​of each motion direction interval to obtain a maximum value, and selecting the motion direction interval corresponding to the maximum value as the main motion direction interval; For each ROI region, calculating the main motion direction average direction vector and the main motion direction average modulus of all ROI region facial local motion vectors corresponding to the main motion direction interval of each ROI region, and calculating the main motion direction feature vector of each ROI region based on the main motion direction average direction vector and the main motion direction average modulus; The main motion direction feature vectors of each ROI area are spliced ​​to form a single-frame micro-expression spatial feature vector; Accordingly, the single-frame micro-expression spatial feature vectors of each frame are sequentially arranged to form a micro-expression spatiotemporal fusion matrix, which includes: Obtain the number of frames and the timestamp of each frame in each time sliding window, and establish an initialization time series matrix based on the number of frames in each time sliding window, wherein the initialization time series matrix is ​​an empty matrix, and the number of rows of the matrix is ​​determined by the number of frames in each time sliding window; According to the timestamps of each frame, the single-frame micro-expression spatial feature vectors of each frame are arranged in sequence, and the single-frame micro-expression spatial feature vectors of each frame after the arrangement are filled into the initialized time series matrix row by row to form a micro-expression spatiotemporal fusion matrix, wherein any row of the micro-expression spatiotemporal fusion matrix represents the single-frame micro-expression spatial feature vector of any frame in the corresponding time sequence, and any column of the micro-expression spatiotemporal fusion matrix represents the temporal change trajectory of a single ROI area in different frames.

6. The method for adaptively regulating learning emotions based on micro-expression analysis according to claim 1, characterized in that: Based on the micro-expression spatiotemporal fusion matrix, we extract the structured phase of micro-expressions and obtain effective micro-expressions that conform to the structured phase pattern of micro-expressions, including: Based on the micro-expression spatiotemporal fusion matrix, the temporal change trajectory of each ROI region is obtained, and the temporal change trajectory of each ROI region is subjected to temporal trajectory filtering and smoothing noise reduction processing to obtain a smooth temporal change trajectory of each ROI region; For the smooth temporal change trajectory of each ROI area, calculate the global maximum value of the trajectory within the smooth temporal change trajectory of each ROI area; Based on the global maximum value of the trajectory corresponding to each ROI area, a micro-expression peak threshold is set. In the smooth time-series change trajectory of each ROI area, the frames in each ROI area that reach or exceed the micro-expression peak threshold are selected as the peak frames of the corresponding ROI area; The peak frames in each ROI region are clustered and merged to form the candidate micro-expression regions of each ROI region; From the candidate micro-expression regions of each ROI region, count the number of positive angle frames and the number of negative angle frames, and calculate the difference between the positive angle frame number and the negative angle frame number to obtain the micro-expression phase difference of each ROI region, wherein the positive angle frame number is used to represent the frame number corresponding to the start of the micro-expression, and the negative angle frame number is used to represent the frame number corresponding to the release of the micro-expression; Obtain a valid micro-expression phase difference threshold, and for each ROI region, determine whether its micro-expression phase difference does not exceed the valid micro-expression phase difference threshold. If so, the candidate micro-expression region of this ROI region is considered to be a valid micro-expression; if not, the candidate micro-expression region of this ROI region is considered to be an invalid micro-expression.

7. The method for adaptively regulating learning emotions based on micro-expression analysis according to claim 1, characterized in that: Based on the effective micro-expressions in each ROI area, the learner's real-time learning emotions are mapped out. The original learning parameters are adjusted in real time based on the learner's real-time learning emotions to achieve adaptive adjustment of the learner's learning emotions, including: In each time sliding window, the regional dominance coefficient of each ROI area is calculated; Based on the regional dominance coefficient of each ROI area and combined with the effective micro-expressions of each ROI area, learning emotion mapping is performed to obtain preliminary learning emotions; Counting the number of occurrences of the preliminary learned emotions in a continuous time sliding window, and performing secondary mapping on the preliminary learned emotions to obtain deep learned emotions; The deep learning emotion is used as the learner's real-time learning emotion, and the learning parameters of the learning machine are adjusted according to the learner's real-time learning emotion to complete adaptive adjustment of the learner's learning emotion, wherein the learning parameters include learning time, question difficulty, rest time, answering interaction mode and answering interface elements.

8. A learning emotion adaptive regulation system based on micro-expression analysis, characterized in that: include: A video acquisition unit, configured to acquire a learner's facial video stream in real time and divide the learner's facial video stream into a plurality of time sliding windows; a face detection unit, configured to perform face region detection at the first frame of each time sliding window to obtain a face region of the first frame of each time sliding window, and perform face cropping on the remaining frames in each time sliding window based on the face region of the first frame to obtain a face region image, and select multiple ROI regions from the face region image; an optical flow matrix calculation unit, configured to calculate a dense optical flow field of the ROI region in each frame in each time sliding window, obtain an optical flow vector matrix of each ROI region, and calculate an ROI region facial local motion matrix of each ROI region in each frame based on the optical flow vector matrix of each ROI region; The feature matrix calculation unit is used to calculate the single-frame micro-expression spatial feature vector of each frame based on the ROI area facial local motion matrix of each ROI area in each time sliding window, and sequentially arrange the single-frame micro-expression spatial feature vectors of each frame to form a micro-expression spatiotemporal fusion matrix: The adaptive adjustment unit is used to perform structured phase extraction of micro-expressions based on the micro-expression spatiotemporal fusion matrix to obtain effective micro-expressions that conform to the structured phase pattern of micro-expressions, and map the learning emotion classification according to the effective micro-expressions in each ROI area. The original learning parameters are adjusted in real time through the learning emotion classification to complete the adaptive adjustment of the learner's learning emotions.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a transceiver which are communicatively connected in sequence, wherein the memory is used to store a computer program, the transceiver is used to send and receive messages, and the processor is used to read the computer program and execute the learning emotion adaptive regulation method based on micro-expression analysis as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program or instructions, characterized in that When executed by a computer, the computer program or the instruction implements the learning emotion adaptive regulation method based on micro-expression analysis as described in any one of claims 1 to 7.

Images