Emotion monitoring method, device and equipment and computer storage medium
By analyzing students' expressions and behavior data in classroom and intercourse videos, monitoring students' emotional states, solving the problem of difficulty in timely detection of depression in the existing technology, and achieving accurate monitoring of students' emotions and preventing depression.
Patent Information
- Application Number
- CN202310631349.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-30
- Publication Date
- 2024-12-03
AI Technical Summary
The existing technology is difficult to detect and prevent students' depression in a timely manner, especially among students who are ignored by teachers and parents, the symptoms of depression are hidden and difficult to detect in a timely manner.
By obtaining classroom videos and inter-leisure videos, analyzing students' expression data and behavior data, confirming the emotional values corresponding to expressions and behaviors, calculating individual and group emotional values, and then monitoring students' emotional states to improve the accuracy of emotional recognition.
It effectively improves the accuracy of monitoring students' abnormal emotions, promptly detects potential depression tendencies, reminds teachers and parents to pay attention, and plays a role in preventing depression.
Smart Images

Figure CN119068295A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to emotion monitoring methods, devices, equipment and computer storage media. Background Art
[0002] At present, depression is mostly diagnosed by teachers, doctors, etc., but most of them are not discovered in time. First, it is difficult for teachers to pay attention to the psychological state of each student, especially quiet and withdrawn children, who are more likely to be ignored by teachers. Another reason is that in the family environment, depressed teenagers will behave in the same way as normal teenagers, making it difficult to detect them in time.
[0003] Therefore, it is urgent to analyze students' emotions at school and detect abnormal emotions, so as to promptly discover students' potential depression tendencies during the education process, alert teachers and parents, and play a role in preventing depression. Summary of the invention
[0004] The main purpose of this specification is to provide an emotion monitoring method, device, equipment and computer storage medium, aiming to improve the accuracy of emotion recognition. The technical solution is as follows:
[0005] In a first aspect, an embodiment of the present specification provides an emotion monitoring method, comprising:
[0006] Get the classroom videos and break videos corresponding to the object collection;
[0007] Based on the classroom video and the break video, acquiring expression data and behavior data of a target object, wherein the target object is any object in the object set;
[0008] Confirming a first emotion value corresponding to the expression data and a second emotion value corresponding to the behavior data;
[0009] Based on the first emotion value and the second emotion value, confirming the individual emotion value of the target object;
[0010] Determining a group emotion value of the set of objects based on the individual emotion value of each of the target objects;
[0011] The emotion monitoring result of the target object is confirmed based on the individual emotion value and the group emotion value.
[0012] In a second aspect, the embodiments of this specification provide an emotion monitoring device, including:
[0013] An acquisition module is used to acquire classroom videos and break videos corresponding to the object set;
[0014] An expression and behavior recognition module, used for acquiring expression data and behavior data of a target object based on the classroom video and the break video, wherein the target object is any object in the object set;
[0015] A value confirmation module, used to confirm a first emotion value corresponding to the expression data and a second emotion value corresponding to the behavior data;
[0016] An individual emotion confirmation module, configured to confirm the individual emotion value of the target object based on the first emotion value and the second emotion value;
[0017] A group emotion confirmation module, used to confirm the group emotion value of the object set based on the individual emotion value of each target object;
[0018] A monitoring module is used to confirm the emotion monitoring result of the target object based on the individual emotion value and the group emotion value.
[0019] In a third aspect, an embodiment of the present specification provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the above method when executed by the processor.
[0020] In a fourth aspect, an embodiment of the present specification provides a computer storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.
[0021] In the embodiments of this specification, by obtaining classroom videos and break videos corresponding to the object set, based on the classroom videos and break videos, the expression data and behavior data of the target object are obtained, the first emotion value corresponding to the expression data and the second emotion value corresponding to the behavior data are confirmed, the individual emotion value of the target object is confirmed based on the first emotion value and the second emotion value, the group emotion value of the object set is confirmed based on the individual emotion value of each target object, and the emotion monitoring result of the target object is confirmed based on the individual emotion value and the group emotion value. By introducing break videos and combining them with classroom videos for expression and behavior analysis, richer state data can be obtained, and then individuals and groups can be compared, abnormal emotions can be better monitored, and the accuracy of emotion recognition can be effectively improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0022] In order to more clearly illustrate the embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0023] Figure 1 is an example schematic diagram of an emotion monitoring method provided in an embodiment of this specification;
[0024] Figure 2 It is a flowchart of an emotion monitoring method provided in an embodiment of this specification;
[0025] Figure 3 This is a schematic diagram of an example of an emotion monitoring method provided in an embodiment of this specification;
[0026] Figure 4 A flowchart of an emotion monitoring method provided in an embodiment of this specification;
[0027] Figure 5 This is a schematic diagram of an example of an emotion monitoring method provided in an embodiment of this specification;
[0028] Figure 6 is an example schematic diagram of an emotion monitoring method provided in an embodiment of this specification;
[0029] Figure 7 A flowchart of an emotion monitoring method provided in an embodiment of this specification;
[0030] Figure 8 It is a flowchart of an emotion monitoring method provided in an embodiment of this specification;
[0031] Fig. 9 This is a schematic diagram of an example of an emotion monitoring method provided in an embodiment of this specification;
[0032] Fig.10 This is a schematic diagram of the structure of an emotion monitoring device provided in the embodiment of this specification.
[0033] Fig.11 It is a structural schematic diagram of an electronic device provided in an embodiment of this specification. DETAILED DESCRIPTION
[0034] The following will be combined with the drawings in the embodiments of this specification to clearly and completely describe the technical solutions in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this specification.
[0035] The emotion monitoring device can be a terminal device such as a mobile phone, a computer, a tablet computer or a camera, or it can be a module in the terminal device for implementing the emotion monitoring method. The emotion monitoring device can obtain classroom videos and break videos corresponding to the object set, and based on the classroom videos and break videos, obtain the expression data and behavior data of the target object, confirm the first emotion value corresponding to the expression data and the second emotion value corresponding to the behavior data, confirm the individual emotion value of the target object based on the first emotion value and the second emotion value, confirm the group emotion value of the object set based on the individual emotion value of each target object, and confirm the emotion monitoring result of the target object based on the individual emotion value and the group emotion value.
[0036] Please also see Figure 1 , an example schematic diagram of an emotion monitoring method is provided for an embodiment of this specification. The emotion monitoring device can be a terminal device, which obtains break videos and classroom videos captured by a camera, and analyzes the break videos and classroom videos to obtain the expression data and behavior data of the target object, confirms the individual emotion value of the target object according to the expression data and behavior data of the target object, confirms the group emotion value of the object set to which the target object belongs, and confirms the emotion monitoring result of the target object based on the group emotion value and the individual emotion value.
[0037] The emotion monitoring method provided in this specification is described in detail below in conjunction with specific embodiments.
[0038] See also Figure 2 , is a flow chart of an emotion monitoring method provided in the embodiment of this specification. Figure 2 As shown, the method in the embodiment of this specification may include the following steps S101-S106.
[0039] S101, obtaining classroom videos and break videos corresponding to the object set;
[0040] Specifically, the object set can be a set of multiple objects divided by class, or a set composed of specific objects, such as focusing on student sets, group sets, etc. The object set includes multiple target objects, and the monitored objects can be flexibly selected according to actual conditions. In a feasible implementation, the classroom video and break video corresponding to the object set can be obtained through the acquisition device. Classroom video refers to the video during class, and break video refers to the video after class. The video during class is mainly aimed at video collection in the class. Break video can collect videos in the class, and can also collect videos outside the class, such as videos in the corridor. Among them, when obtaining break videos and classroom videos, you can collect full-time videos in school, and then divide them according to the class schedule to obtain break videos and classroom videos. The acquisition device can be a variety of terminal devices such as a camera with a camera function, a smart TV, a smart blackboard, and smart glasses. The acquisition device can be used for emotion monitoring, and the video collected by the acquisition device can also be uploaded to the emotion monitoring device for emotion monitoring. The acquisition device can be one or more to ensure that the classroom video and break video of the object set as complete as possible are obtained. You can collect break videos and classroom videos on a daily basis to monitor emotions every day, or collect break videos and classroom videos according to preset collection time to monitor students' emotional state over a period of time.
[0041] S102, based on the classroom video and the break video, acquiring expression data and behavior data of a target object, wherein the target object is any object in the object set;
[0042] Specifically, the target object in the object set can be identified based on the classroom video and the break video, and the expression data and behavior data of the target object can be obtained. Among them, the identity information of each object in the video, such as face information, seat information, name plate, etc., can be obtained based on the classroom video and the break video, so as to confirm the identity of each target object in the object set, and then the expression data and behavior data of any target object in the object set are obtained accordingly. In a feasible implementation, by pre-training the behavior recognition model and the expression recognition model, the actions and expressions of each target object in the classroom video and the break video are recognized, and after model reasoning, the behavior of the target object and the corresponding behavior classification confidence are output to obtain the behavior data, and the expression of the target object and the expression classification confidence are obtained to obtain the expression data. In another feasible implementation, by pre-training the action recognition model, the break video and the classroom video are input into the action recognition model, the action of the target object is identified, and the behavior of the target object is confirmed according to the action to obtain the behavior data. It should be noted that the expressions and actions that can be recognized by the target object can be multiple, and there is no specific limitation in the embodiments disclosed in this disclosure. The expression type and action type required for emotion monitoring can be confirmed in advance, and then the required expression type and action type can be identified to obtain expression data and action data.
[0043] In a feasible implementation, the identifiable behavior types may include: interaction, relaxation, isolation, irritability, participation, compliance and escape. It is understandable that, since the scheme of the embodiment of this specification is mainly used in campus to analyze students' mental health, the performance of students in school is divided into classroom performance and inter-class performance in this application scenario. If students often show active participation and interact with classmates and teachers in the classroom, and communicate with classmates during the class break, then these students are mostly lively and cheerful, and have excellent learning. But if students often show evasive behavior in the classroom, and they are alone and do not communicate with classmates during the class break, then these students are more likely to have psychological problems. Therefore, the behavior types of students are divided into two categories: classroom behavior and inter-class behavior, and the behavior in the classroom is divided into four types of interaction, participation, compliance and escape according to the psychological performance of students, and the behavior during the class break is divided into four types of interaction, relaxation, isolation and irritability.
[0044] In a feasible implementation, the types of expressions that can be identified may include: happy, focused, relaxed, neutral, sad, disgusted, and angry. Although humans can make many kinds of expressions, expressions such as fear, surprise, and contempt rarely appear in classrooms and learning scenarios, while focus, boredom, and happiness are the main emotions expressed by students. By combining mental health monitoring and depression recognition scenarios with students' performance in school, the types of expressions are divided into the above types.
[0045] See also Figure 3 , Figure 3 The embodiment of this specification provides an example schematic diagram of an emotion monitoring method. Figure 3 In the definition of student expression and behavior labels, expressions are defined as happy, focused, relaxed, neutral, sad, disgusted and angry, behaviors are defined as classroom behavior and break behavior, classroom behavior is divided into four types: interaction, participation, compliance and escape, and break behavior is divided into four types: interaction, relaxation, isolation and irritability. Furthermore, when the target object's actions are raising hands and discussing, the corresponding behavior is interaction; when the target object's behavior is nodding, looking up, and turning books, the corresponding behavior is participation; when the target object's behavior is lowering the head, yawning, and being in a daze, the corresponding behavior is compliance; when the target object's behavior is playing with mobile phones, looking out the window, and sleeping, the corresponding behavior is escape; when the target object's behavior is communication and group chatting, the corresponding behavior is interaction; when the target object's behavior is getting up, moving limbs, and looking into the distance, the corresponding behavior is relaxation; when the target object's behavior is covering the face and lying on the table, the corresponding behavior is isolation; when the target object's behavior is conflict, the corresponding behavior is irritability.
[0046] It should be noted that the expression types and action types shown above can be changed or further refined according to actual needs, and are not specifically limited.
[0047] S103, confirming a first emotion value corresponding to the expression data and a second emotion value corresponding to the behavior data;
[0048] Specifically, after the expression data and behavior data of the target object are identified, the corresponding emotional values are assigned to the expression data and the behavior data, and the first emotional value corresponding to the expression data and the second emotional value corresponding to the behavior data are obtained. It can be understood that the corresponding emotional values can be confirmed for different expressions and actions to characterize the different degrees of the expressed emotions. For example, the corresponding emotional labels can be confirmed through the behavior data and expression data, and then the specific expressions and behaviors of the students can be associated with the emotions, and different emotional values can be assigned according to different emotional labels, or the probability of expression and behavior can be confirmed according to the behavior data and expression data of the target object, and the emotional value can be confirmed according to the probability of expression and the frequency of behavior. It should be noted that the first emotional value and the second emotional value can include emotional values corresponding to different emotional labels, that is, the first emotional value and the second emotional value can be one or more.
[0049] S104, confirming the individual emotion value of the target object based on the first emotion value and the second emotion value;
[0050] Specifically, the individual emotion value of the target object is determined according to the first emotion value confirmed by the expression data and the second emotion value confirmed by the behavior data. Exemplarily, the individual emotion value of the target object can be obtained by adding the first emotion value and the second emotion value. For example, if the first emotion value has an emotion value a1 corresponding to happiness and the second emotion value has an emotion value a2 corresponding to happiness, then the total emotion value of happiness obtained by adding a1+a2 can be used as the individual emotion value of the target object.
[0051] S105, confirming the group emotion value of the set of objects based on the individual emotion value of each of the target objects;
[0052] Specifically, after obtaining the individual emotion value corresponding to a target object, the individual emotion values of all target objects in the object set can be obtained, and the group emotion value can be obtained by counting the individual emotion values of all target objects. The group emotion value can refer to the number of people distributed at each emotion value, or it can be an average emotion value, and the statistical method can be selected according to the actual situation.
[0053] S106: confirming the emotion monitoring result of the target object based on the individual emotion value and the group emotion value.
[0054] Specifically, after obtaining the individual emotion value and the group emotion value, the individual is associated with the group for comparison, so as to more accurately monitor the emotion of the target object. It can be understood that the emotion monitoring result of the target object can be confirmed according to the level of the individual emotion value and the group emotion value. The emotion monitoring result can include normal and abnormal results. When the individual emotion value of the target object is at a relatively low score, it means that the emotion of the target object is relatively negative, but this is not an accurate judgment. If the individual emotion value of the target object is close to the group emotion value, it means that the entire group is currently in a relatively negative state, which may not be a problem of a single individual, then the emotion of the target object is consistent with the group emotion state; if the individual emotion value of the target object deviates greatly from the group emotion value, such as the group emotion value is at a relatively high score, representing a positive state, but the individual emotion value of the target object is at a relatively low score, compared with the group in a negative state, indicating that the emotion of the target object in the current class is abnormal, and the student needs more attention and active support.
[0055] In the embodiments of this specification, by obtaining the classroom videos and break videos corresponding to the object set, and analyzing the break videos and classroom videos, the expression data and behavior data of the target object are obtained, and the individual emotion value of the target object is confirmed according to the expression data and behavior data of the target object, and the group emotion value of the object set to which the target object belongs is confirmed, and the emotion monitoring result of the target object is confirmed based on the group emotion value and the individual emotion value. By analyzing the behavior and expression in the classroom and break to observe emotions, the emotion tracking of each student is realized, and by comparing the individual and the group, abnormal emotions can be better monitored.
[0056] See also Figure 4 , which is a detailed flow chart of an emotion monitoring method provided in the embodiment of this specification. Figure 4 As shown, the method in the embodiment of this specification may include the following steps S201-S208.
[0057] S201, confirming a first emotion label corresponding to the expression data;
[0058] Specifically, after the expression data is obtained, the first emotion label corresponding to the expression is confirmed, that is, the emotion of the target object represented by the expression is confirmed according to the expression data. In a feasible implementation, the student's emotional value is divided into two dimensions: Valence (valence)-Arousal (arousal). Valence refers to the positivity of the emotion, or the interest / rejection of things, and arousal refers to the intensity of emotional expression. The valence dimension ranges from negative to positive, and the arousal dimension ranges from calm to excited. According to different valence-arousal levels, the student's emotions are defined as twelve types: vitality, dominance, happiness, concentration, friendliness, relaxation, decadence, obedience, sadness, disgust, rejection and irritability. Please refer to Figure 5 , Figure 5 An example schematic diagram of an emotion monitoring method is provided for the embodiment of this specification. Emotions are divided into four quadrants according to different degrees of valence-arousal. Each student's emotion will fall into any position in the four quadrants according to its degree, such as Figure 5 Point 1 corresponds to students with cheerful and generous personalities, and a positive mentality and personality; while point 2 corresponds to students with irritable and disgusted personalities, and rejects the outside world; point 3 corresponds to students who are more likely to be decadent and withdrawn, so we need to pay more attention to and actively support students in the second and third quadrants. Among them, the different emotional levels can be represented according to the individual emotional values of the target object in different emotions.
[0059] It should be noted that the emotion labels defined above are only examples, and the emotion labels can be defined according to actual needs.
[0060] S202, obtaining an expression recognition probability value of the expression data, and confirming a first emotion value of the first emotion tag based on the expression recognition probability value;
[0061] Specifically, the expression data may include the recognized expression categories and the expression recognition probability value corresponding to each expression. It is understandable that when performing expression recognition, the probability that the target object's current expression is the recognized expression type can be obtained. If the probability is higher, it proves that the probability of the target object being of the expression type is higher, and correspondingly, the probability that the target object is in the emotion corresponding to the expression is higher. Therefore, the first emotion value of the first emotion label is confirmed according to the expression recognition probability value.
[0062] S203, confirming a second emotion label corresponding to the behavior data;
[0063] Correspondingly, corresponding emotion tags can also be identified for the behavior data. For example, a second emotion tag corresponding to each behavior in the behavior data can be selected from the emotion tags, and one or more second emotion tags can be identified according to the different behaviors identified. Different behaviors can correspond to the same emotion or different emotions, which is not specifically limited.
[0064] See also Figure 6 , Figure 6 An example schematic diagram of an emotion monitoring method is provided for the embodiment of this specification. Figure 6 The figure shows the relationship between students’ emotions and facial expressions. Figure 6There are 12 types of emotional labels in it: vitality, dominance, happiness, concentration, friendliness, relaxation, decadence, obedience, sadness, disgust, rejection and irritability. If the target object's expression is happy or the target object's behavior is class interaction, the emotional label is happiness; if the target object's expression is concentration or the target object's behavior is class interaction, the emotional label is concentration; if the target object's behavior is class / class interaction, the emotional label is vitality; if the target object's expression is relaxed or the target object's behavior is class interaction / relaxation and class interaction / participation, the emotional label is relaxation; if the target object's behavior is class / class interaction, the emotional label is dominance; if the target object's behavior is class interaction / relaxation or class interaction / participation, the emotional label is Friendly; if the target object's expression is sad or the target object's behavior is isolated during class, the emotion label is sad; if the target object's expression is disgusted or the target object's behavior is avoiding class, the emotion label is disgust; if the target object's behavior is obedient / avoiding in class or isolated during class, the emotion label is decadent; if the target object's expression is irritable or the target object's behavior is irritable during class, the emotion label is irritable; if the target object's behavior is obedient in class / isolated during class, the emotion label is obedient; if the target object's behavior is avoiding class or is isolated / irritable during class, the emotion label is rejection.
[0065] S204: Acquire a behavior recognition probability value of the behavior data, and confirm a second emotion value of the second emotion tag based on the behavior recognition probability value.
[0066] Specifically, the behavior data may include the identified behavior and the behavior recognition probability value for the behavior. It is understandable that if the probability value of identifying that the target object performs a certain behavior is higher, it means that the target object is more likely to be in the emotion corresponding to the behavior. Therefore, according to the behavior recognition probability value corresponding to the behavior, the emotion value of the emotion tag corresponding to the behavior can be confirmed.
[0067] S205, respectively confirming the emotion attributes of the first emotion label and the second emotion label to obtain a first emotion attribute corresponding to the first emotion label and a second emotion attribute corresponding to the second emotion label, wherein the emotion attributes include a positive attribute and a negative attribute;
[0068] Specifically, the emotion attribute may include positive attributes and negative attributes. In a feasible implementation, the emotion tag has positive attributes or negative attributes, such as: happiness, concentration, vitality, relaxation, dominance, and friendliness are positive attributes, and sadness, disgust, decadence, irritability, obedience, and rejection are negative attributes. According to the predefined association relationship between the emotion tag and the emotion attribute, the emotion attributes corresponding to the first emotion tag and the second emotion tag can be confirmed.
[0069] S206, calculating a first attribute emotion value corresponding to the first emotion value based on the first emotion attribute;
[0070] S207, calculating a second attribute emotion value corresponding to the second emotion value based on the second emotion attribute;
[0071] Specifically, when the emotional attribute is a negative attribute, the emotional value is confirmed as a negative value, and when the emotional attribute is a positive attribute, the emotional value is confirmed as a positive value. It can be understood that the first emotional value and the second emotional value confirmed based on the expression data and the behavior data are positive values by default, and the emotional attribute can be further confirmed, and the first attribute emotional value corresponding to the first emotional value is calculated based on the first emotional attribute, and the second attribute emotional value corresponding to the second emotional value is calculated based on the second emotional attribute. For example, the first emotional label is irritable, and the corresponding first emotional value is 3. The first emotional label is a negative attribute, and the first emotional value is processed to obtain a first attribute emotional value of -3.
[0072] S208: confirming the individual emotion value of the target object based on the first attribute emotion value and the second attribute emotion value.
[0073] Specifically, the total emotion value corresponding to each emotion label of the target object can be obtained according to the obtained first attribute emotion value and the second attribute emotion value as the individual emotion value. For example, the first attribute emotion value includes happiness 1 and disgust -2, and the second attribute emotion includes vitality 2, happiness 1, and rejection -2, then the individual emotion values are happiness 2, vitality 2, rejection -2, and disgust -2. Optionally, the emotion labels are generated into emotion pairs according to positive attributes and negative attributes. For example, six emotion pairs such as happiness-sadness, concentration-disgust, vitality-decadence, relaxation-irritability, dominance-submission, and friendliness-rejection can be generated, and the specific score value on each emotion pair is generated according to the first attribute emotion value and the second attribute emotion value as the individual emotion value.
[0074] In the embodiment of the present specification, by confirming the first emotion tag corresponding to the expression data, confirming the first emotion value of the first emotion tag according to the expression recognition probability value, confirming the second emotion tag corresponding to the behavior data, and confirming the second emotion value of the second emotion tag according to the behavior recognition probability value, the expression and behavior are unified with the emotion, so that the emotion can be quantified, which is conducive to the observation of the emotion. And according to different emotion attributes, the emotion tag is assigned a score, and the emotion value is quantified from different attribute dimensions of positive and negative, so as to help discover the abnormal situation of the target object's emotion.
[0075] See also Figure 7 , is a flow chart of an emotion monitoring method provided in the embodiment of this specification. Figure 7 As shown, the method in the embodiment of this specification may include the following steps S301-S303.
[0076] S301, confirming the individual emotional state of the target object based on the individual emotional value, where the emotional state includes a normal state and an abnormal state;
[0077] Specifically, the individual emotion value is compared with the normal emotion value range. If the individual emotion value does not belong to the normal emotion value range, the individual emotion state is abnormal. If the individual emotion value belongs to the normal emotion value range, the individual emotion state is normal. The normal emotion value range can be set according to actual conditions. For example, the score range of the individual emotion value is from -4 to 4, greater than 0 is a positive state, and less than 0 is a negative state. The normal emotion value range can be within -0.5 to 4.
[0078] S302, confirming the group emotional state of the object set based on the group emotional value;
[0079] Specifically, since the group emotion value is obtained based on the individual emotion values of multiple target objects, the group emotion value may be a range, such as the score range of 2 to 4 on the emotion label happiness, and the corresponding number of people at different scores can also be obtained; it can also be a statistical value, such as the average value of the emotion value on each emotion label, for example, the average value on the emotion label happiness is 3. When the group emotion value is a range, the score with the largest number of people can be confirmed to determine whether the score is within the normal emotion value range. When the group emotion value is an average value, it can be confirmed whether the average value is within the normal emotion value range, and then the group emotion state can be obtained. And according to the degree of deviation of the group emotion value from the normal emotion value range, the degree of the group emotion state can be confirmed. For example, if the average value of the group emotion value is a negative number, the group emotion state is abnormal, and the average value is -4, indicating that the abnormal degree of the group emotion state is serious. The normal emotion value ranges for individual emotion values and group emotion values can be different or the same, and are not specifically limited.
[0080] S303: If the individual emotional state is different from the group emotional state, or the deviation between the individual emotional state and the group emotional state is greater than a preset threshold, confirming that the emotion monitoring result of the target object is abnormal.
[0081] Specifically, when the individual emotional state is different from the group emotional state, the emotion monitoring result of the target object is determined to be abnormal. For example, if the individual emotional state of the target object is negative and the group emotional state is positive, the emotion monitoring result is abnormal. When the deviation between the individual emotional state and the group emotional state is greater than a preset threshold, the emotion monitoring result of the target object is abnormal. For example, if the individual emotional state is very negative and the group emotional state is slightly negative, the emotion monitoring result of the target object is abnormal. It is understandable that in the school scenario, individuals are in a class environment for a long time. If the emotional state of the individual is very different from that of the class, the individual may be at risk of being isolated or depressed, and there is a potential possibility of having mental health problems.
[0082] Further, in one embodiment, the confirming the individual emotional state of the target object based on the individual emotional value includes:
[0083] S401, if the individual emotion value is lower than a first preset reference value, confirming that the individual emotion state of the target object is abnormal;
[0084] Specifically, when judging the individual emotional state, it can be confirmed according to the first preset reference value. If the individual emotional value is lower than the first preset reference value, it is an abnormal state. The first preset reference value can be set according to the experience obtained from the usual observation of the student's performance, for example, it can be selected as 0.
[0085] S402, if the individual emotion value is higher than or equal to a first preset reference value, confirming that the individual emotion state of the target object is normal;
[0086] Similarly, when the individual emotion value is higher than or equal to the first preset reference value, it is a normal state.
[0087] Further, in one embodiment, the confirming the group emotional state of the object set based on the group emotional value includes:
[0088] S403, if the group emotion value is lower than a second preset reference value, confirming that the group emotion state of the object set is an abnormal state;
[0089] Specifically, when judging the group emotional state, it can be confirmed according to the second preset reference value. If the group emotional value is lower than the second preset reference value, it is an abnormal state.
[0090] S404: If the group emotion value is higher than or equal to a second preset reference value, confirming that the group emotion state of the object set is a normal state.
[0091] Correspondingly, if the group emotion value is higher than or equal to the second preset reference value, it is a normal state.
[0092] The first preset reference value for an individual and the second preset reference value for a group may be set to be the same or different, and may be set according to actual conditions.
[0093] See also Figure 8 , is a flow chart of an emotion monitoring method provided in the embodiment of this specification. Figure 8 As shown, the method in the embodiment of this specification may include the following steps S501-S504.
[0094] S501, obtaining the individual historical emotion value of the target object in a historical time period;
[0095] S502, obtaining the group historical emotion value of the object set in the historical time period;
[0096] Specifically, when confirming the emotion monitoring result of the target object based on the individual emotion value and the group emotion value, the individual historical emotion value of the target object in the historical time period and the group historical emotion value of the object set in the historical time period can also be obtained. The individual historical emotion value is a plurality of individual emotion values of the student in the historical time period (such as the past month). For example, if the statistical period of the individual emotion value is day, then the individual historical emotion value is the individual emotion value of each day in the past month. The group historical emotion value is a plurality of historical emotion values of the object set in the historical time period. Among them, the statistical period of the group emotion value of the object set is the same as that of the individual emotion value.
[0097] S503, comparing the individual emotion value with the individual historical emotion value, the group emotion value and the group historical emotion value respectively to obtain a comparison result;
[0098] Specifically, for the comparison between the individual emotion value and the group emotion value, please refer to step S106 in the above embodiment, which will not be repeated here. For the comparison between the individual emotion value and the individual historical emotion value, the individual emotion value can be compared with one or some individual historical emotion values in the individual historical emotion value. It can be understood that the target object's weekly schedule will not change within a certain period of time. For example, the individual emotion value of the current target object is obtained by statistics on Wednesday, then the individual historical emotion value of the past Wednesday can be obtained from the individual historical emotion value for comparison, and the deviation between the individual emotion value and the individual historical emotion value can be determined to obtain a comparison result. Similarly, for the comparison between the individual emotion value and the group historical emotion value, the group historical emotion value can also be obtained according to a specific date or period for comparison to obtain a comparison result. The comparison result can be the deviation value between the individual emotion value and the individual historical emotion value, the group emotion value, and the group historical emotion value, or the degree of deviation.
[0099] In one embodiment, the comparing the individual emotion value with the individual historical emotion value, the group emotion value and the group historical emotion value respectively to obtain a comparison result includes:
[0100] S601, obtaining first distribution data of the individual historical emotion value, second distribution data of the group emotion value, and third distribution data of the group historical emotion value;
[0101] Specifically, based on multiple individual historical emotion values, their first distribution data on each emotion label can be confirmed. Based on the group emotion value, their second distribution data on each emotion label can be confirmed. Based on the group historical emotion value, their third distribution data on each emotion label can be confirmed. Exemplarily, the numerical distribution of multiple individuals on each emotion pair in a historical time period (such as the past month) can be obtained, specifically represented by a box plot; based on the group emotion value of the object set, the emotion value distribution of the object set on each emotion pair is obtained as the second distribution data; based on the group historical emotion value, the third distribution data can be obtained by counting the average value of the object set on each emotion pair in the historical time period (such as the past month). Please refer to Fig. 9 , Fig. 9 An example schematic diagram of an emotion monitoring method is provided for the embodiments of this specification, which is divided into six groups of emotion pairs. A corresponding score is obtained for each emotion pair based on the statistics of expression data and behavior data, with a total of 6 scores, each score corresponding to an emotion pair, and the specific score range is from -4 to 4. Fig. 9 The figure shows the numerical distribution data of the group's historical emotion value, individual historical emotion value, group's emotion value on the day (group emotion value obtained with a statistical period of one day), and individual's emotion value on the day (individual emotion value obtained with a statistical period of one day) on the predefined emotion pairs. For the group's emotion value on the day, the darker the color, the more target objects are in the emotion value range, and more objects in the object set are happy rather than sad on the day.
[0102] S602, comparing the individual emotion value with the first distribution data to obtain a first comparison result;
[0103] Specifically, the individual emotion value is compared with the first distribution data, and the specific value of the individual emotion value on each emotion pair is compared with the first distribution data of the individual historical emotion value on each emotion pair. For example, if the first emotion value has never appeared in the first distribution data and does not belong to the range of the first distribution data, the first comparison result can be obtained that the individual emotion value is higher or lower than the first distribution range. The comparison result can include the comparison result corresponding to the emotion value corresponding to each emotion label in the individual emotion value.
[0104] S603, comparing the individual emotion value with the second distribution data to obtain a second comparison result;
[0105] S604, comparing the individual emotion value with the third distribution data to obtain a third comparison result;
[0106] Specifically, the second distribution data of the individual emotion value and the group emotion value, and the individual emotion value and the third distribution data can also be compared in the above manner, which will not be elaborated herein.
[0107] S504: confirming the emotion monitoring result of the target object based on the comparison result.
[0108] Specifically, when the comparison result shows that the individual emotion value is distributed in a value with less distribution or no distribution in the first distribution data or the second distribution data or the third distribution data, the emotion of the target object may be abnormal, which may be abnormally excited or abnormally negative. Exemplarily, it can be confirmed whether the first comparison result, the second comparison result, and the third comparison result meet the abnormal state conditions to obtain the abnormal state judgment result. The abnormal state conditions can be set according to actual conditions. The main focus of this application is on abnormally negative emotions. It can be confirmed based on the comparison results that the emotion attribute is a negative attribute, and the individual emotion value whose emotion value deviates greatly from each distribution data has confirmed the emotion monitoring result of the target object. Please continue to refer to Fig. 9 , through Fig. 9 Analyzing the target object in the group, it can be found that the target object is often in negative emotions such as decadence, submissiveness, and rejection, and is often in a negative state compared to the group. The emotional monitoring result of the target object is abnormal and needs more attention and active support.
[0109] In the embodiments of the present specification, the individual emotional state of the target object is confirmed based on the individual emotional value, and the group emotional state of the object set is confirmed based on the group emotional value. If the individual emotional state is different from the group emotional state, or the deviation between the individual emotional state and the group emotional state is greater than a preset threshold, the emotional monitoring result of the target object is confirmed to be abnormal, thereby realizing the judgment of the emotional state of the target object. In addition, by obtaining the individual historical emotional value of the target object in the historical time period and the group historical emotional value of the object set in the historical time period, and comparing the individual emotional value with the individual historical emotional value and the group historical emotional value, it is possible to more clearly discover the emotional fluctuations of the individual and the abnormal emotions, thereby improving the accuracy of emotional monitoring of the target object.
[0110] The following will be combined with the attached Fig.10 , the emotion monitoring device provided in the embodiment of this specification is introduced in detail. It should be noted that the attached Fig.10The emotion monitoring device in the embodiment of the present invention is used to execute the present invention Figure 2-Figure 9 For the convenience of explanation, only the part related to the embodiment of this specification is shown. For the specific technical details not disclosed, please refer to this specification. Figure 2-Figure 9 The embodiment shown.
[0111] See also Fig.10 , which shows a schematic diagram of the structure of an emotion monitoring device provided by an exemplary embodiment of the present specification. The emotion monitoring device can be implemented as all or part of the device through software, hardware or a combination of both. The device 1 includes an acquisition module 11, an expression behavior recognition module 12, a numerical confirmation module 13, an individual emotion confirmation module 14, a group emotion confirmation module 15, and a monitoring module 16.
[0112] An acquisition module 11 is used to acquire classroom videos and break videos corresponding to the object set;
[0113] An expression and behavior recognition module 12, for acquiring expression data and behavior data of a target object based on the classroom video and the break video, wherein the target object is any object in the object set;
[0114] A value confirmation module 13, used to confirm the first emotion value corresponding to the expression data and the second emotion value corresponding to the behavior data;
[0115] An individual emotion confirmation module 14, configured to confirm the individual emotion value of the target object based on the first emotion value and the second emotion value;
[0116] A group emotion confirmation module 15, configured to confirm the group emotion value of the target object set based on the individual emotion value of each target object;
[0117] The monitoring module 16 is used to confirm the emotion monitoring result of the target object based on the individual emotion value and the group emotion value.
[0118] Optionally, the numerical value confirmation module 13 is specifically used to confirm the first emotion label corresponding to the expression data;
[0119] Acquire an expression recognition probability value of the expression data, and confirm a first emotion value of the first emotion tag based on the expression recognition probability value;
[0120] confirming a second emotion label corresponding to the behavior data;
[0121] A behavior recognition probability value of the behavior data is obtained, and a second emotion value of the second emotion tag is confirmed based on the behavior recognition probability value.
[0122] Optionally, the individual emotion confirmation module 14 is specifically used to confirm the emotion attributes of the first emotion label and the second emotion label respectively, so as to obtain a first emotion attribute corresponding to the first emotion label and a second emotion attribute corresponding to the second emotion label, wherein the emotion attributes include positive attributes and negative attributes;
[0123] Calculating a first attribute emotion value corresponding to the first emotion value based on the first emotion attribute;
[0124] Calculating a second attribute emotion value corresponding to the second emotion value based on the second emotion attribute;
[0125] Based on the first attribute emotion value and the second attribute emotion value, an individual emotion value of the target object is confirmed.
[0126] Optionally, the monitoring module 16 is specifically used to confirm the individual emotional state of the target object based on the individual emotional value, and the emotional state includes a normal state and an abnormal state;
[0127] confirming a group emotional state of the set of objects based on the group emotional value;
[0128] If the individual emotional state is different from the group emotional state, or the deviation between the individual emotional state and the group emotional state is greater than a preset threshold, the emotion monitoring result of the target object is confirmed to be abnormal.
[0129] Optionally, the monitoring module 16 is specifically configured to confirm that the individual emotional state of the target object is an abnormal state if the individual emotional value is lower than a first preset reference value;
[0130] If the individual emotion value is higher than or equal to the first preset reference value, confirming that the individual emotion state of the target object is normal;
[0131] If the group emotion value is lower than a second preset reference value, confirming that the group emotion state of the object set is an abnormal state;
[0132] If the group emotion value is higher than or equal to a second preset reference value, it is confirmed that the group emotion state of the object set is normal.
[0133] Optionally, the monitoring module 16 is specifically used to obtain the individual historical emotion value of the target object in a historical time period;
[0134] Obtaining the group historical emotion value of the object set within the historical time period;
[0135] Comparing the individual emotion value with the individual historical emotion value, the group emotion value and the group historical emotion value respectively to obtain a comparison result;
[0136] The emotion monitoring result of the target object is confirmed based on the comparison result.
[0137] Optionally, the monitoring module 16 is specifically used to obtain the first distribution data of the individual historical emotion value, the second distribution data of the group emotion value, and the third distribution data of the group historical emotion value;
[0138] Comparing the individual emotion value with the first distribution data to obtain a first comparison result;
[0139] Comparing the individual emotion value with the second distribution data to obtain a second comparison result;
[0140] The individual emotion value is compared with the third distribution data to obtain a third comparison result.
[0141] It should be noted that when the emotion monitoring device provided in the above embodiment executes the emotion monitoring method, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the emotion monitoring device provided in the above embodiment and the emotion monitoring method embodiment belong to the same concept, and the implementation process thereof is detailed in the method embodiment, which will not be repeated here.
[0142] The serial numbers of the embodiments of the present specification are for description only and do not represent the advantages and disadvantages of the embodiments. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0143] The present specification also provides a computer storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the above-mentioned Figure 1-Figure 9 The specific execution process of the emotion monitoring method in the embodiment shown can be found in Figure 1-Figure 9 The specific description of the illustrated embodiment will not be repeated here.
[0144] Please refer to Fig.11, which shows a schematic diagram of the structure of an electronic device provided by an exemplary embodiment of this specification. The electronic device in this specification may include one or more of the following components: a processor 110, a memory 120, an input device 130, an output device 140, and a bus 150. The processor 110, the memory 120, the input device 130, and the output device 140 may be connected via the bus 150.
[0145] The processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions and processes data of the terminal 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 110 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes the operating system, user pages, and applications; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.
[0146] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 120 includes a non-transitory computer-readable medium (Non-Transitory Computer-Readable Storage Medium). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The operating system may be an Android system, including a system deeply developed based on the Android system, an IOS system developed by Apple, including a system deeply developed based on the IOS system or other systems.
[0147] The memory 120 can be divided into an operating system space and a user space. The operating system runs in the operating system space, and native and third-party applications run in the user space. In order to ensure that different third-party applications can achieve good operating results, the operating system allocates corresponding system resources to different third-party applications. However, the requirements for system resources in different application scenarios in the same third-party application are also different. For example, in the local resource loading scenario, the third-party application has higher requirements for disk reading speed; in the animation rendering scenario, the third-party application has higher requirements for GPU performance. The operating system and third-party applications are independent of each other, and the operating system often cannot perceive the current application scenario of the third-party application in a timely manner, resulting in the operating system being unable to perform targeted system resource adaptation according to the specific application scenario of the third-party application.
[0148] In order for the operating system to distinguish the specific application scenarios of third-party applications, it is necessary to open up data communication between third-party applications and the operating system so that the operating system can obtain the current scenario information of third-party applications at any time, and then perform targeted system resource adaptation based on the current scenario.
[0149] The input device 130 is used to receive input commands or data, and includes but is not limited to a keyboard, a mouse, a camera, a microphone, or a touch device. The output device 140 is used to output commands or data, and includes but is not limited to a display device and a speaker. In one example, the input device 130 and the output device 140 can be combined, and the input device 130 and the output device 140 are touch screen displays.
[0150] The touch display screen can be designed as a full screen, a curved screen or a special-shaped screen. The touch display screen can also be designed as a combination of a full screen and a curved screen, or a combination of a special-shaped screen and a curved screen, which is not limited in the embodiments of this specification.
[0151] In addition, those skilled in the art can understand that the structure of the electronic device shown in the above drawings does not constitute a limitation on the electronic device, and the electronic device may include more or fewer components than shown, or combine certain components, or arrange the components differently. For example, the electronic device also includes components such as a radio frequency circuit, an input unit, a sensor, an audio circuit, a WiFi module, a power supply, and a Bluetooth module, which will not be described in detail here.
[0152] exist Fig.11 In the electronic device shown, the processor 110 may be used to call a computer application stored in the memory 120 and specifically perform the following operations:
[0153] Get the classroom videos and break videos corresponding to the object collection;
[0154] Based on the classroom video and the break video, acquiring expression data and behavior data of a target object, wherein the target object is any object in the object set;
[0155] Confirming a first emotion value corresponding to the expression data and a second emotion value corresponding to the behavior data;
[0156] Based on the first emotion value and the second emotion value, confirming the individual emotion value of the target object;
[0157] Determining a group emotion value of the set of objects based on the individual emotion value of each of the target objects;
[0158] The emotion monitoring result of the target object is confirmed based on the individual emotion value and the group emotion value.
[0159] In one embodiment, when the processor 110 confirms the first emotion value corresponding to the expression data and the second emotion value corresponding to the behavior data, the processor 110 specifically performs the following operations:
[0160] Confirming a first emotion label corresponding to the expression data;
[0161] Acquire an expression recognition probability value of the expression data, and confirm a first emotion value of the first emotion tag based on the expression recognition probability value;
[0162] confirming a second emotion label corresponding to the behavior data;
[0163] A behavior recognition probability value of the behavior data is obtained, and a second emotion value of the second emotion tag is confirmed based on the behavior recognition probability value.
[0164] In one embodiment, when the processor 110 confirms the individual emotion value of the target object based on the first emotion value and the second emotion value, the processor 110 specifically performs the following operations:
[0165] Respectively confirming the emotion attributes of the first emotion label and the second emotion label to obtain a first emotion attribute corresponding to the first emotion label and a second emotion attribute corresponding to the second emotion label, wherein the emotion attributes include a positive attribute and a negative attribute;
[0166] Calculating a first attribute emotion value corresponding to the first emotion value based on the first emotion attribute;
[0167] Calculating a second attribute emotion value corresponding to the second emotion value based on the second emotion attribute;
[0168] Based on the first attribute emotion value and the second attribute emotion value, an individual emotion value of the target object is confirmed.
[0169] In one embodiment, when the processor 110 confirms the emotion monitoring result of the target object based on the individual emotion value and the group emotion value, the processor 110 specifically performs the following operations:
[0170] confirming the individual emotional state of the target object based on the individual emotional value, wherein the emotional state includes a normal state and an abnormal state;
[0171] confirming a group emotional state of the set of objects based on the group emotional value;
[0172] If the individual emotional state is different from the group emotional state, or the deviation between the individual emotional state and the group emotional state is greater than a preset threshold, the emotion monitoring result of the target object is confirmed to be abnormal.
[0173] In one embodiment, when the processor 110 performs the step of confirming the individual emotional state of the target object based on the individual emotional value, the processor 110 specifically performs the following operations:
[0174] If the individual emotion value is lower than a first preset reference value, confirming that the individual emotion state of the target object is abnormal;
[0175] If the individual emotion value is higher than or equal to the first preset reference value, it is confirmed that the individual emotion state of the target object is normal.
[0176] In one embodiment, when the processor 110 executes the step of confirming the group emotional state of the object set based on the group emotional value, the processor 110 specifically performs the following operations:
[0177] If the group emotion value is lower than a second preset reference value, confirming that the group emotion state of the object set is an abnormal state;
[0178] If the group emotion value is higher than or equal to a second preset reference value, it is confirmed that the group emotion state of the object set is normal.
[0179] In one embodiment, when the processor 110 confirms the emotion monitoring result of the target object based on the individual emotion value and the group emotion value, the processor 110 specifically performs the following operations:
[0180] Obtaining the individual historical emotion value of the target object within a historical time period;
[0181] Obtaining the group historical emotion value of the object set within the historical time period;
[0182] Comparing the individual emotion value with the individual historical emotion value, the group emotion value and the group historical emotion value respectively to obtain a comparison result;
[0183] The emotion monitoring result of the target object is confirmed based on the comparison result.
[0184] In one embodiment, when the processor 110 compares the individual emotion value with the individual historical emotion value, the group emotion value, and the group historical emotion value to obtain a comparison result, the processor 110 specifically performs the following operations:
[0185] Acquire first distribution data of the individual historical emotion values, second distribution data of the group emotion values, and third distribution data of the group historical emotion values;
[0186] Comparing the individual emotion value with the first distribution data to obtain a first comparison result;
[0187] Comparing the individual emotion value with the second distribution data to obtain a second comparison result;
[0188] The individual emotion value is compared with the third distribution data to obtain a third comparison result.
[0189] In the embodiment of the present specification, by obtaining the classroom video and the break video corresponding to the object set, based on the classroom video and the break video, the expression data and behavior data of the target object are obtained, the first emotion value corresponding to the expression data and the second emotion value corresponding to the behavior data are confirmed, based on the first emotion value and the second emotion value, the individual emotion value of the target object is confirmed, based on the individual emotion value of each target object, the group emotion value of the object set is confirmed, and the emotion monitoring result of the target object is confirmed based on the individual emotion value and the group emotion value. By introducing the break video and combining the classroom video for expression and behavior analysis, richer state data can be obtained, and then the individual and the group are compared, so that abnormal emotions can be better monitored, and the accuracy of emotion recognition can be effectively improved. By confirming the first emotion label corresponding to the expression data, confirming the first emotion value of the first emotion label according to the expression recognition probability value, confirming the second emotion label corresponding to the behavior data, and confirming the second emotion value of the second emotion label according to the behavior recognition probability value, the expression and behavior are unified with the emotion, so that the emotion can be quantified, which is conducive to the observation of the emotion. And according to different emotion attributes, the emotion label is assigned a score, and the emotion value is quantified from different attribute dimensions of positive and negative, so as to help discover the abnormal situation of the target object's emotion. By confirming the individual emotional state of the target object based on the individual emotional value, and confirming the group emotional state of the object set based on the group emotional value, if the individual emotional state is different from the group emotional state, or the deviation between the individual emotional state and the group emotional state is greater than a preset threshold, the emotional monitoring result of the target object is confirmed to be abnormal, thereby realizing the judgment of the emotional state of the target object. In addition, by obtaining the individual historical emotional value of the target object in the historical time period and the group historical emotional value of the object set in the historical time period, and comparing the individual emotional value with the individual historical emotional value and the group historical emotional value, it is possible to more clearly discover the emotional fluctuations of the individual, discover abnormal emotions, and improve the accuracy of emotional monitoring of the target object.
[0190] In addition, an embodiment of the present specification provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor of an electronic device, the processor can at least implement the above-mentioned Figures 1 to 9 The emotion monitoring method provided in the illustrated embodiment.
[0191] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0192] The above disclosure is only the preferred embodiment of this specification, which certainly cannot be used to limit the scope of rights of this specification. Therefore, equivalent changes made according to the claims of this specification are still within the scope covered by this specification.
Claims
1. A method for monitoring emotions, characterized in that: The method comprises: Get the classroom videos and break videos corresponding to the object collection; Based on the classroom video and the break video, acquiring expression data and behavior data of a target object, wherein the target object is any object in the object set; Confirming a first emotion value corresponding to the expression data and a second emotion value corresponding to the behavior data; Based on the first emotion value and the second emotion value, confirming the individual emotion value of the target object; Determining a group emotion value of the set of objects based on the individual emotion value of each of the target objects; The emotion monitoring result of the target object is confirmed based on the individual emotion value and the group emotion value.
2. The method according to claim 1, characterized in that The confirming the first emotion value corresponding to the expression data and the second emotion value corresponding to the behavior data includes: Confirming a first emotion label corresponding to the expression data; Acquire an expression recognition probability value of the expression data, and confirm a first emotion value of the first emotion tag based on the expression recognition probability value; confirming a second emotion label corresponding to the behavior data; A behavior recognition probability value of the behavior data is obtained, and a second emotion value of the second emotion tag is confirmed based on the behavior recognition probability value.
3. The method according to claim 2, characterized in that The step of confirming the individual emotion value of the target object based on the first emotion value and the second emotion value includes: Respectively confirming the emotion attributes of the first emotion label and the second emotion label to obtain a first emotion attribute corresponding to the first emotion label and a second emotion attribute corresponding to the second emotion label, wherein the emotion attributes include a positive attribute and a negative attribute; Calculating a first attribute emotion value corresponding to the first emotion value based on the first emotion attribute; Calculating a second attribute emotion value corresponding to the second emotion value based on the second emotion attribute; Based on the first attribute emotion value and the second attribute emotion value, an individual emotion value of the target object is confirmed.
4. The method according to claim 1, characterized in that The confirming the emotion monitoring result of the target object based on the individual emotion value and the group emotion value includes: confirming the individual emotional state of the target object based on the individual emotional value, wherein the emotional state includes a normal state and an abnormal state; confirming a group emotional state of the set of objects based on the group emotional value; If the individual emotional state is different from the group emotional state, or the deviation between the individual emotional state and the group emotional state is greater than a preset threshold, the emotion monitoring result of the target object is confirmed to be abnormal.
5. The method according to claim 4, characterized in that The confirming the individual emotional state of the target object based on the individual emotional value includes: If the individual emotion value is lower than a first preset reference value, confirming that the individual emotion state of the target object is abnormal; If the individual emotion value is higher than or equal to the first preset reference value, confirming that the individual emotion state of the target object is normal; The confirming the group emotional state of the object set based on the group emotional value includes: If the group emotion value is lower than a second preset reference value, confirming that the group emotion state of the object set is an abnormal state; If the group emotion value is higher than or equal to a second preset reference value, it is confirmed that the group emotion state of the object set is normal.
6. The method according to claim 1, characterized in that The confirming the emotion monitoring result of the target object based on the individual emotion value and the group emotion value includes: Obtaining the individual historical emotion value of the target object within a historical time period; Obtaining the group historical emotion value of the object set within the historical time period; Comparing the individual emotion value with the individual historical emotion value, the group emotion value and the group historical emotion value respectively to obtain a comparison result; The emotion monitoring result of the target object is confirmed based on the comparison result.
7. The method according to claim 6, characterized in that The comparison result includes a first comparison result, a second comparison result and a third comparison result; The comparing the individual emotion value with the individual historical emotion value, the group emotion value and the group historical emotion value respectively to obtain a comparison result includes: Acquire first distribution data of the individual historical emotion values, second distribution data of the group emotion values, and third distribution data of the group historical emotion values; Comparing the individual emotion value with the first distribution data to obtain a first comparison result; Comparing the individual emotion value with the second distribution data to obtain a second comparison result; The individual emotion value is compared with the third distribution data to obtain a third comparison result.
8. An emotion monitoring device, characterized in that: The device comprises: An acquisition module is used to acquire classroom videos and break videos corresponding to the object set; An expression and behavior recognition module, used for acquiring expression data and behavior data of a target object based on the classroom video and the break video, wherein the target object is any object in the object set; A value confirmation module, used to confirm a first emotion value corresponding to the expression data and a second emotion value corresponding to the behavior data; An individual emotion confirmation module, configured to confirm the individual emotion value of the target object based on the first emotion value and the second emotion value; A group emotion confirmation module, used to confirm the group emotion value of the object set based on the individual emotion value of each target object; A monitoring module is used to confirm the emotion monitoring result of the target object based on the individual emotion value and the group emotion value.
9. An electronic device, characterized in that: include: Processor and memory; The memory stores a computer program, wherein the computer program is suitable for being loaded by the processor and executing the steps of the method according to any one of claims 1 to 7.
10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Group emotion-based prediction method, device and system, server and terminal
CN108876018A
Method and system for detecting psychological abnormalities based on multi-channel collaboration
CN110464367A
Problem behavior detection and early warning system
CN113869415A
Group expression and emotion recognition method and system based on space-time consistency
CN114694230A
Question and answer emotion evaluation method and device based on AU (action unit) and micro-expression
CN115512418A
Cited By
Audio and video fusion control method and device based on AI
CN120234761A
