Emotion and behavior-based learning tutoring monitoring method and system, and medium

Abstract

The invention discloses a learning tutoring monitoring method and system based on emotion and behavior, and a medium, and the method comprises the steps: obtaining emotion recognition data of a user, and analyzing the emotion state of the user through an emotion recognition model, the emotion recognition data comprising face data and voice data; behavior data of the user are obtained, behavior postures of the user are analyzed through the posture recognition model, and the behavior data comprise behavior images; learning tutoring monitoring is carried out in combination with the emotional state and the behavior posture, when the emotional state and / or the behavior posture meet a preset reminding requirement, an emotional behavior adjustment prompt is generated based on an emotional adjustment mechanism, and the preset reminding requirement comprises that the emotional state is in an abnormal emotion state or the behavior posture has an abnormal behavior. The emotional state and behavior behavior of the parents can be recognized in real time when the parents tutorize the students to do homework, so that improper behaviors are prevented, and the parents are helped to keep proper emotional management in the child tutoring process.

Description

Technical Field

[0001] The present invention relates to the field of smart education technology, and in particular to a learning guidance monitoring method, system and medium based on emotion and behavior. Background Art

[0002] In the process of family education, parents' emotional management when helping their children with homework has an important impact on their children's learning attitude and family atmosphere. However, existing technical means are still imperfect in parental emotional management, lacking real-time monitoring and feedback mechanisms, which may lead to inappropriate behaviors when parents are emotional, such as inappropriate language and corporal punishment, which cannot be effectively prevented and intervened.

[0003] At present, most of the emotion management tools on the market rely on parents' self-control and simple timed reminders, lacking real-time feedback on parents' actual emotions and behaviors during the coaching process, and failing to provide personalized emotion management support. At the same time, existing technologies also make it difficult to identify parents' emotional states in real time and conduct appropriate interventions. Summary of the invention

[0004] The present invention provides a learning tutoring monitoring method, system and medium based on emotion and behavior, which can identify the emotional state and behavioral actions of parents in real time when parents tutor students' homework, so as to prevent inappropriate behavior and help parents maintain appropriate emotional management in the process of tutoring their children.

[0005] In order to solve the above technical problems, the first aspect of the present invention discloses a learning guidance monitoring method based on emotion and behavior, the method comprising:

[0006] Acquire emotion recognition data of the user, and analyze the user's emotional state through an emotion recognition model, wherein the emotion recognition data includes facial data and voice data;

[0007] Acquire user behavior data and analyze the user's behavior posture through a posture recognition model, wherein the behavior data includes a behavior image;

[0008] The emotional state and behavioral posture are combined to perform learning guidance monitoring. When the emotional state and / or behavioral posture meet the preset reminder requirements, an emotional behavior adjustment prompt is generated based on the emotional adjustment mechanism. The preset reminder requirements include that the emotional state is in an abnormal emotion or the behavioral posture has abnormal behavior.

[0009] In some embodiments, the positions of several key points in the behavior data are extracted, and the key points include nose, eyes, ears, shoulders, elbows, wrists, hips, knees and ankles;

[0010] Extracting features of the key points, converting them into feature maps, and calculating the confidence of the key points;

[0011] Generating a posture graph according to the key point positions and the confidence levels;

[0012] The user's current behavior posture is analyzed according to the posture graph.

[0013] In some embodiments, the abnormal behavior includes preset gestures and abnormal postures; the abnormal emotions include anger, alertness, danger, or warning.

[0014] In some embodiments, the emotion behavior adjustment prompt includes a prompt light and a voice announcement; generating the emotion behavior adjustment prompt according to the emotion adjustment mechanism includes:

[0015] Select a corresponding warning light color according to the emotional state; the warning light colors include a first color, a second color, a third color and a fourth color;

[0016] The corresponding voice broadcast content is selected according to the behavior posture, and the voice broadcast content includes a first prompt content and a second prompt content.

[0017] In some implementations, analyzing the user's emotional state using the emotion recognition model includes:

[0018] Preprocessing the facial data and the voice data, and extracting facial emotion features of the facial data and language emotion features of the voice data;

[0019] Concatenate the facial emotion features and the language emotion features to generate a comprehensive emotion feature vector;

[0020] The comprehensive emotional feature vector is input into the Transformer module and the fully connected layer for processing, and the output of the fully connected layer is converted into the probability distribution of multiple emotional states through the softmax activation function;

[0021] The emotional state of the user is determined according to the probability distribution.

[0022] In some embodiments, the preprocessing includes noise reduction, enhancement, and normalization.

[0023] In some embodiments, it further comprises:

[0024] Obtaining user historical data, including historical emotional states, historical behavioral postures, and historical emotional behavior adjustment prompts;

[0025] Generate time series data based on the historical data, and analyze the time series data to generate emotion management suggestions.

[0026] In some embodiments, the posture recognition model uses the PoseNet network, and the loss function is

[0027]

[0028] Wherein, x is the real position vector of the shooting point of the behavior data, represents the position vector of the shooting point of the behavior data predicted by the posture recognition model; q represents the real direction of the behavior data, represents the direction predicted by the posture recognition model; β is a weight parameter used to balance the position error and direction error.

[0029] According to a second aspect of the present invention, a learning guidance monitoring system based on emotion and behavior is disclosed, comprising:

[0030] An emotion recognition module acquires the user's emotion recognition data and analyzes the user's emotional state through an emotion recognition model, wherein the emotion recognition data includes facial data and voice data;

[0031] A behavior posture monitoring module, which obtains the user's behavior data and analyzes the user's behavior posture through a posture recognition model, wherein the behavior data includes a behavior image;

[0032] The learning guidance monitoring module combines the emotional state and behavioral posture to perform learning guidance monitoring. When the emotional state and / or behavioral posture meet the preset reminder requirements, an emotional behavior adjustment prompt is generated based on the emotional adjustment mechanism. The preset reminder requirements include that the emotional state is in an abnormal emotion or the behavioral posture has abnormal behavior.

[0033] In a third aspect, a computer storage medium is disclosed, characterized in that a computer program is stored thereon, and when the computer program is executed by a processor, the steps of any one of the above-mentioned emotion and behavior-based learning tutoring monitoring methods are implemented.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention provides a learning tutoring monitoring method, system, and medium based on emotion and behavior, which can obtain the user's emotional state and behavioral posture in real time and accurately identify the user's current emotional state and behavioral posture. Compared with traditional methods, it helps to timely identify negative emotions and inappropriate behaviors during the learning tutoring process, and helps parents maintain proper emotional management in the process of tutoring their children. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of the process provided by the present invention;

[0037] Figure 2 A schematic diagram of the process of step S1 provided by the present invention;

[0038] Figure 3This is a schematic flow chart of step S2 provided by the present invention. DETAILED DESCRIPTION

[0039] For better understanding and implementation, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0040] The terms "including" and "having" and any variations thereof in the embodiments of the present invention are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or modules is not necessarily limited to those steps or modules explicitly listed, but may include other steps or modules not explicitly listed or inherent to these processes, methods, products or apparatus.

[0041] The embodiment of the present invention discloses a learning tutoring monitoring method based on emotion and behavior, which can identify the emotional state and behavioral actions of parents in real time when parents tutor students' homework, so as to prevent inappropriate behavior and help parents maintain appropriate emotional management in the process of tutoring their children.

[0042] Specifically, the method comprises the following steps:

[0043] Step S1, obtaining the user's emotion recognition data, and analyzing the user's emotional state through an emotion recognition model, wherein the emotion recognition data includes facial data and voice data.

[0044] In this application, users can be parents, students, or teachers. Different students can study and tutor each other, parents or teachers can tutor students, or a single student can study on his own. There is no limitation in this application.

[0045] When the user is receiving learning guidance, the user's emotion recognition data is collected through a camera device. The emotion recognition data includes facial data and voice data. The voice data includes intonation, volume, and speech speed, etc., which further accurately analyzes the user's emotional state.

[0046] Further, such as Figure 2 As shown, analyzing the user's emotional state through the emotion recognition model includes:

[0047] Step S11, pre-processing the facial data and the voice data, and extracting facial emotion features of the facial data and language emotion features of the voice data.

[0048] Preprocess the facial data and voice data collected by the camera, including noise reduction, image enhancement and standardization, to ensure the quality of the input data and improve the clarity of the voice features. Input the preprocessed facial image into a deep neural network, such as ResNet50, to extract the facial emotion features of the facial data, including emotion-related facial muscle movements, eye and mouth shape changes, etc. Input the preprocessed voice data into a voice feature extraction model, such as YAMNet, to extract voice emotion features, including voice pitch, volume, frequency and other information.

[0049] Step S12: concatenate the facial emotion features and the language emotion features, input the concatenated facial emotion features and the language emotion features into the Transformer module and the fully connected layer for processing, and convert them into probability distributions of multiple emotional states through the softmax activation function.

[0050] The facial emotion features and the language emotion features are connected in series to form a comprehensive emotion feature vector. By integrating facial emotion features and language emotion features, the emotion recognition model can take advantage of the complementarity of facial data and voice data to obtain a more complete emotional expression.

[0051] The fused comprehensive emotional feature vector is input into the Transformer model for processing. The Transformer model can capture the complex relationship between features through the self-attention mechanism. For example, the association between facial expressions and voice features helps to identify more subtle emotional changes. Under the action of multi-head attention and feed-forward neural network, the Transformer module generates more representative emotional feature representations. The output of the Transformer is passed to the fully connected layer to further integrate and compress the comprehensive emotional feature vector. The output of the fully connected layer maps the emotional features to a probability distribution through the Softmax activation function. Each emotional category corresponds to a probability, indicating the possibility that the user belongs to the emotional category. Emotional categories can include happiness, anger, sadness, surprise, fear, disgust, alertness, warning or danger, etc. The classification of emotional categories and the corresponding probabilities can be adjusted according to the actual application scenarios, which are not limited in this application.

[0052] Step S13: determining the user's emotional state according to the probability distribution.

[0053] According to the output probability of the Softmax activation function, the emotion category with the highest probability is selected as the user's current emotional state. The user's specific emotional state is identified, such as happiness, anger, sadness, surprise, fear, disgust or neutral, and the emotional state is used for emotional feedback and management in the learning tutoring process.

[0054] Step S2: Obtain the user's behavior data and analyze the user's behavior posture through a posture recognition model.

[0055] The behavior data includes behavior images or behavior videos. The behavior videos or behavior data are collected by a camera. The camera captures the user's body language to analyze the user's behavior posture. A posture recognition model based on the PoseNet deep network is used for posture recognition model. PoseNet uses GoogleLeNet as the backbone network, removes the softmax activation function, sets 4 output branches, including 3 position components for position estimation and 4 direction components for direction estimation, and adds a fully connected layer of 2048 neurons in the output layer to better adapt to the task of position and direction estimation. It also uses multi-scale feature fusion technology to extract richer posture information through feature maps of different scales to improve the accuracy and robustness of posture recognition.

[0056] Specifically, Figure 3 As shown, the user's behavior posture is predicted through the posture recognition model, including:

[0057] Step S21, extracting the positions of several key points in the behavior data; in the present application, the key points include 0 nose, 1 left eye, 2 right eye, 3 left ear, 4 right ear, 5 left shoulder, 6 right shoulder, 7 left elbow, 8 right elbow, 9 left wrist, 10 right wrist, 11 left hip, 12 right hip, 13 left knee, 14 right knee, 15 left ankle, 16 right ankle. The posture of the human body is described by the above key points. For each key point, the PoseNet network generates a heat map, in which each pixel value indicates the possibility of the position being the key point.

[0058] In this application, a 2D coordinate heat map normalized by sigmoid with a size of n×22×22×17 is used to represent the heat map of each key point.

[0059] To improve the accuracy of key point positioning, PoseNet generates an offset. The offset is used to fine-tune the predicted key point positions in the heat map to make them more accurate. The offset of each key point represents the correction value from the preliminary position predicted by the heat map to the actual position. PoseNet preliminarily locates the position of the key point by detecting the local maximum of each heat map. The size of the coordinate offset is n×22×22×34. The size of the forward joint connection score is n×22×22×32, and the size of the reverse joint connection score is n×22×22×32, which indicates the connection strength between the joints.

[0060] Step S22: extract features from the key points, convert them into the feature graph, and calculate the confidence of the key points.

[0061] Monitor the user's behavior in real time, arrange the continuous key point data in time series, and form posture change data over a period of time. Extract features such as relative position, angle change, and movement trajectory between key points to capture subtle information about posture changes. For example, calculate the angle between the elbow and shoulder, and the knee and ankle to identify the pattern of hand or leg movements. In addition, the relative distance and speed between key points can also be calculated to further reflect the intensity and direction of the user's movements.

[0062] The confidence value is the value of the maximum pixel value of the key point in the heat map, which indicates the prediction accuracy of the key point location. The higher the value, the more confident the model is in predicting the point. The confidence of each key point is calculated to indicate the confidence of the model in the existence of the key point. The confidence is usually between 0 and 1. The higher the value, the more certain the model is about the location of the key point.

[0063] At the same time, PoseNet uses the non-maximum suppression (NMS) method to remove overlapping key point predictions. It sorts each point according to its confidence, and retains the key points with the highest built-in confidence in the local area in turn, suppressing other points with low confidence, ensuring that only one optimal prediction is retained for each key point. Non-maximum suppression helps eliminate overlapping detections and improve the accuracy of key point detection.

[0064] Step S23: generating a posture graph according to the key point positions and the confidence levels, and analyzing the user's current behavior posture according to the posture graph.

[0065] After obtaining the precise positions and connection relationships of all key points, a complete posture graph is generated. The posture graph displays the user's current posture by connecting the skeleton structure of each key point. The user's behavior posture is analyzed according to the posture graph to determine the user's specific actions or behaviors, such as standing, sitting, waving, etc., and a posture graph is generated for behavior monitoring and feedback.

[0066] In this application, the posture recognition model uses the PoseNet network, and the loss function is

[0067]

[0068] Wherein, x is the real position vector of the shooting point of the behavior data, represents the position vector of the shooting point of the behavior data predicted by the posture recognition model; q represents the real direction of the behavior data, represents the direction predicted by the posture recognition model; β is a weight parameter used to balance the position error and direction error.

[0069] Step S3, combining the emotional state and behavioral posture to perform learning guidance monitoring, and when the emotional state and / or behavioral posture meet the preset reminder requirements, generating an emotional behavior adjustment prompt based on the emotional adjustment mechanism, and the preset reminder requirements include that the emotional state is in an abnormal emotion or the behavioral posture has abnormal behavior.

[0070] After the gesture recognition model recognizes the user's behavior posture, it identifies abnormal behavior. For example, preset gestures include hitting with hands, frequent gestures, corporal punishment and other abnormal gestures. Abnormal emotions include anger, vigilance, danger, warning, etc. When there is abnormal behavior or abnormal emotion, it is considered that the preset reminder requirements are met.

[0071] Generate emotional behavior adjustment prompts based on the emotional adjustment mechanism. Emotional behavior adjustment prompts include prompt lights and voice broadcasts; the preset reminder requirements include: the emotional state is in an abnormal state, or the behavior posture has abnormal behavior; the abnormal emotions include anger, alertness, danger, and warning.

[0072] Generate emotional behavior adjustment prompts based on emotional adjustment mechanisms, including:

[0073] The corresponding prompt light color is selected according to the emotional state; the prompt light colors include a first color, a second color, a third color and a fourth color.

[0074] The corresponding voice broadcast content is selected according to the behavior posture, and the voice broadcast content includes a first prompt content and a second prompt content.

[0075] Exemplarily, the correspondence between preset colors and emotions is indicated by the color of the prompt light. The colors include blue, green, yellow, and red. Blue represents calmness or relaxation, green represents comfort or safety, yellow represents alertness or attention, and red represents warning or danger. The color of the prompt light is automatically adjusted according to the emotional state of the user, such as the parent. For example, if the parent is nervous or angry (red), the light may change to yellow or blue to help the parent relax.

[0076] What’s more, the voice announcement content can be automatically adjusted according to the parent’s behavior. For example, if the parent is about to implement corporal punishment, the announcement will say “Please pay attention to your behavior. We recommend that you handle the current situation in a calm manner.”

[0077] When parents are helping their children with their homework, the voice input device detects that the parents' facial expressions are tense and the voice tone is rising. After analysis, it is considered that the parents are in a bad mood and the following measures will be taken:

[0078] Color adjustment of alert lights: Adjust the smart lights in your home from white to blue or green to create a more relaxing environment.

[0079] Adjustment of voice announcement content: Voice reminder played through smart speakers: "You seem a little nervous, try to take a deep breath, and we can help your child better understand this problem together."

[0080] Furthermore, the method further comprises:

[0081] Obtain user historical data, including historical emotional state, historical behavior posture, and historical emotional behavior adjustment prompts. Historical data is formed by recording and storing the user's emotional state and behavior posture. The historical emotional state includes the user's emotional state in the past period of time, and the historical behavior posture records the user's different behavior postures and behavior patterns. At the same time, the emotional behavior adjustment prompts provided to the user correspond to the user's emotional state and behavior posture one by one, helping to understand the user's reaction to the prompt, and the user to know his or her own emotional state or behavior posture.

[0082] Time series data is generated based on the historical data, and emotion management suggestions are generated by analyzing the time series data. The time series data can show the changing trend of the user's emotions and behavioral postures, as well as the impact of the emotion behavior adjustment prompts on the user's emotional state and behavioral postures. By analyzing the time series data, personalized emotion management suggestions are generated, which helps to improve the user's emotion regulation and behavior control.

[0083] Exemplary emotion regulation skills include:

[0084] 1. Keep an emotional diary: Encourage parents to record the specific circumstances of each time they feel frustrated or angry, including the time, triggers, and the child’s behavior. This helps parents identify patterns and find clues to solving problems.

[0085] 2. Deep breathing and relaxation training: When parents feel angry, they are advised to take a few deep breaths or leave the situation temporarily to help themselves calm down.

[0086] 3. Positive communication: Parents can learn how to communicate with their children in a more positive way, such as asking more questions and solving problems.

[0087] 4. Seek support: It is recommended that parents exchange experiences with other parents or seek professional psychological counseling to obtain more emotional support and advice.

[0088] Based on the same inventive concept, the present application provides a learning guidance monitoring system based on emotion and behavior, including:

[0089] An emotion recognition module acquires the user's emotion recognition data and analyzes the user's emotional state through an emotion recognition model, wherein the emotion recognition data includes facial data and voice data;

[0090] A behavior posture monitoring module, which obtains the user's behavior data and analyzes the user's behavior posture through a posture recognition model, wherein the behavior data includes a behavior image;

[0091] The learning guidance monitoring module combines the emotional state and behavioral posture to perform learning guidance monitoring. When the emotional state and / or behavioral posture meet the preset reminder requirements, an emotional behavior adjustment prompt is generated based on the emotional adjustment mechanism. The preset reminder requirements include that the emotional state is in an abnormal emotion or the behavioral posture has abnormal behavior.

[0092] The processing method of this system can refer to the description of the above method, which will not be repeated here.

[0093] The present invention provides a learning tutoring monitoring method, system, and medium based on emotion and behavior, which can obtain the user's emotional state and behavioral posture in real time and accurately identify the user's current emotional state and behavioral posture. Compared with traditional methods, it helps to timely identify negative emotions and inappropriate behaviors during the learning tutoring process, and helps parents maintain proper emotional management in the process of tutoring their children.

[0094] The present invention also provides a device, which may include: a memory storing executable program code;

[0095] a processor coupled to the memory;

[0096] A transceiver used to communicate with other devices or communication networks and to receive or send network messages;

[0097] A bus used to connect memory, processors, and transceivers for internal communication.

[0098] The transceiver receives messages transmitted over the network and passes them to the processor via the bus. The processor calls the executable program code stored in the memory through the bus for processing, and passes the processing results to the transceiver via the bus for sending, thereby realizing the method provided in the embodiment of the present application.

[0099] An embodiment of the present application also provides a non-transitory machine-readable storage medium, on which an executable program is stored. When the executable program is executed by a processor, the processor executes the method provided in the above embodiment.

[0100] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the described method.

[0101] An embodiment of the present invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to enable a computer to execute the described method.

[0102] The embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without paying creative labor.

[0103] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0104] Finally, it should be noted that the embodiments disclosed in the present invention are only preferred embodiments of the present invention, which are only used to illustrate the technical solutions of the present invention rather than to limit them. Although the present invention has been described in detail with reference to the aforementioned embodiments, it should be understood by those skilled in the art that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features therein may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A learning guidance monitoring method based on emotion and behavior, characterized in that: The method comprises: Acquire emotion recognition data of the user, and analyze the user's emotional state through an emotion recognition model, wherein the emotion recognition data includes facial data and voice data; Acquire user behavior data and analyze the user's behavior posture through a posture recognition model, wherein the behavior data includes a behavior image; The emotional state and behavioral posture are combined to perform learning guidance monitoring. When the emotional state and / or behavioral posture meet the preset reminder requirements, an emotional behavior adjustment prompt is generated based on the emotional adjustment mechanism. The preset reminder requirements include that the emotional state is in an abnormal emotion or the behavioral posture has abnormal behavior.

2. The method for monitoring learning guidance based on emotion and behavior according to claim 1, characterized in that: Predict the user's behavior posture through the posture recognition model, including: Extracting positions of a plurality of key points in the behavior data, wherein the key points include a nose, eyes, ears, shoulders, elbows, wrists, hips, knees and ankles; Extracting features of the key points, converting them into feature maps, and calculating the confidence of the key points; Generating a posture graph according to the key point positions and the confidence levels; The user's current behavior posture is analyzed according to the posture graph.

3. The learning guidance monitoring method based on emotion and behavior according to claim 1 is characterized in that: The abnormal behaviors include preset gestures and abnormal postures; the abnormal emotions include anger, alertness, danger or warning.

4. The method for monitoring learning guidance based on emotion and behavior according to claim 3, characterized in that: Emotional behavior adjustment prompts include prompt lights and voice broadcasts; Generate emotional behavior adjustment prompts based on emotional adjustment mechanisms, including: Select a corresponding warning light color according to the emotional state; the warning light colors include a first color, a second color, a third color and a fourth color; The corresponding voice broadcast content is selected according to the behavior posture, and the voice broadcast content includes a first prompt content and a second prompt content.

5. A learning guidance monitoring method based on emotion and behavior according to any one of claims 1 to 4, characterized in that: The emotion recognition data includes facial data and voice data, and analyzing the user's emotional state through the emotion recognition model includes: Preprocessing the facial data and the voice data, and extracting facial emotion features of the facial data and language emotion features of the voice data; Concatenate the facial emotion features and the language emotion features to generate a comprehensive emotion feature vector; The comprehensive emotional feature vector is input into the Transformer module and the fully connected layer for processing, and the output of the fully connected layer is converted into the probability distribution of multiple emotional states through the softmax activation function; The emotional state of the user is determined according to the probability distribution.

6. The method for monitoring learning guidance based on emotion and behavior according to claim 5, characterized in that: The preprocessing includes noise reduction, enhancement and normalization.

7. The method for monitoring learning guidance based on emotion and behavior according to claim 6, characterized in that: Also includes: Obtaining user historical data, including historical emotional states, historical behavioral postures, and historical emotional behavior adjustment prompts; Generate time series data based on the historical data, and analyze the time series data to generate emotion management suggestions.

8. The method for monitoring learning guidance based on emotion and behavior according to claim 2, characterized in that: The posture recognition model adopts the PoseNet network, and the loss function is Wherein, x is the real position vector of the shooting point of the behavior data, represents the position vector of the shooting point of the behavior data predicted by the posture recognition model; q represents the real direction of the behavior data, represents the direction predicted by the posture recognition model; β is a weight parameter used to balance the position error and direction error.

9. A learning guidance monitoring system based on emotion and behavior, characterized in that: include: An emotion recognition module acquires the user's emotion recognition data and analyzes the user's emotional state through an emotion recognition model, wherein the emotion recognition data includes facial data and voice data; A behavior posture monitoring module, which obtains the user's behavior data and analyzes the user's behavior posture through a posture recognition model, wherein the behavior data includes a behavior image; The learning guidance monitoring module combines the emotional state and behavioral posture to perform learning guidance monitoring. When the emotional state and / or behavioral posture meet the preset reminder requirements, an emotional behavior adjustment prompt is generated based on the emotional adjustment mechanism. The preset reminder requirements include that the emotional state is in an abnormal emotion or the behavioral posture has abnormal behavior.

10. A computer storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of a learning guidance monitoring method based on emotion and behavior as claimed in any one of claims 1 to 8 are implemented.

Images