A learning emotion recognition and intervention method, system, device, and storage medium

Abstract

The present application belongs to the technical field of emotion recognition based on image recognition, and more particularly relates to a learning emotion recognition and intervention method, device, equipment and storage medium. Whether the to-be-recognized object is in the existing recognized object is judged to distinguish between the newly-added to-be-recognized object and the original to-be-recognized object, and different recognition logics and recognition methods are used for the two kinds of objects. For the original to-be-recognized object, if the eye movement curve data is sufficient, the corresponding relationship between the eye movement curve of the original to-be-recognized object and the learning state can be established, and better emotion recognition of the learning state of the individual can be performed; if the eye movement curve data is not sufficient, emotion recognition of the learning state is performed according to the corresponding relationship between the eye movement curve of the public and the learning state.

Description

Technical Field

[0001] This invention belongs to the field of emotion recognition technology based on image recognition, and more specifically, relates to a method, system, device, and storage medium for learning emotion recognition and intervention. Background Technology

[0002] Because online teaching can overcome geographical limitations, enabling more people to access more learning resources, it is increasingly favored by educational institutions and students.

[0003] However, its shortcomings are also quite prominent. For example, instructors cannot obtain the learning status of students in real time, and cannot guarantee the teaching effect. Therefore, if a method for emotion recognition of students' learning status in online teaching can be developed, and a method that can intervene in real time based on the recognition results can be developed, the above problems can be solved, and it may even bring more competitiveness to online teaching. Summary of the Invention

[0004] This invention provides a method for emotion recognition of the learning state of learners, and a method for real-time intervention based on the recognition results.

[0005] This invention is obtained through the following steps:

[0006] A learning-based emotion recognition and intervention method includes the following steps:

[0007] S1: Obtain the facial image of the object to be identified. After face recognition, match it with the facial data of the object stored in the database. If the match is successful, execute S2; if the match is unsuccessful, execute S3.

[0008] S2: Obtain the historical emotion recognition time of the successfully matched recognition objects in the database. If the acquisition fails, execute S3. If the acquisition is successful, determine whether the difference between the historical emotion recognition time and the current time is greater than the first threshold. If it is greater than the first threshold, execute S3. If it is not greater than the first threshold, obtain the eye movement curve of the object to be recognized within the recognition time. Based on the correspondence between the historical eye movement curve and the learning state, obtain the learning state as the result of this emotion recognition and output it.

[0009] S3: Based on the facial image of the object to be identified, draw the eye movement curve of the object to be identified, obtain the eye movement curve of the object to be identified within the recognition time, and obtain the learning state as the result of this emotion recognition based on the correspondence between the general eye movement curve and the learning state, and output it.

[0010] In step S3, the eye-tracking curve of the object to be identified is drawn based on the facial image of the object to be identified, specifically including:

[0011] Several facial images of the subjects to be identified are collected, and the pupil positions in the facial images are located. Based on the correspondence between pupil position and time, an eye movement curve of the subject to be identified is plotted with time on the horizontal axis and pupil position on the vertical axis.

[0012] If the result of this emotion recognition is a distracted state, a prompt message or a prompt sound effect is sent to the learning device of the object to be recognized to intervene in the learning state of the object to be recognized.

[0013] It is preferable to use a neural network to classify eye movement curves and obtain the correspondence between eye movement curves and learning states per unit time.

[0014] Preferably, if the number of eye movement curves accumulated by the subject to be identified within a historical period reaches the second threshold, then the historical eye movement curves are formed.

[0015] Preferably, if the number of eye movement curves accumulated by the subject to be identified within a historical period does not reach the second threshold, then the set of eye movement curves accumulated by all subjects to be identified within a historical period that do not reach the second threshold is defined as the general eye movement curve.

[0016] Preferably, a prompt message or a prompt sound effect is sent to the learning device of the object to be recognized. Specifically, a symbol is displayed on the display screen of the object to be recognized or a voice prompt is played on the learning device of the object to be recognized.

[0017] A learning emotion recognition and intervention system, comprising:

[0018] Matching module: used to acquire the facial image of the object to be identified, and after face recognition, match it with the facial data of the object to be identified stored in the database;

[0019] Individual Emotion Recognition Module: Obtain the historical emotion recognition time of successfully matched recognition objects in the database. If successful, determine whether the difference between the historical emotion recognition time and the current time is greater than the first threshold. Obtain the eye movement curve of the object to be recognized within the recognition time. Based on the correspondence between the historical eye movement curve and the learning state, obtain the learning state as the result of this emotion recognition and output it.

[0020] The general emotion recognition module: Based on the facial image of the object to be recognized, the eye movement curve of the object to be recognized is drawn, the eye movement curve of the object to be recognized during the recognition time is obtained, and the learning state is obtained as the result of this emotion recognition based on the correspondence between the general eye movement curve and the learning state, and then output.

[0021] A learning emotion recognition and intervention device includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the learning emotion recognition and intervention method.

[0022] A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the learning emotion recognition and intervention method.

[0023] The beneficial effects of this invention are:

[0024] By determining whether the object to be identified is among the existing objects, the newly added object to be identified and the original object to be identified are distinguished, and different identification logic and identification methods are used for the two types of objects.

[0025] For the original subject to be identified, if the eye movement curve data is rich enough, the correspondence between the original subject's own eye movement curve and learning state can be established, which can provide better emotion recognition of the learning state for individuals; if the eye movement curve data is not rich enough, the emotion recognition of the learning state can be performed based on the correspondence between the general public's eye movement curve and learning state.

[0026] For newly added subjects to be identified, since there is no historical data, we can only first identify the emotion of the learning state based on the correspondence between the general public's eye movement curve and the learning state. If the eye movement curve of the subject to be identified reaches the set second threshold, we can establish the correspondence between the subject's eye movement curve and the learning state in a timely manner and carry out targeted individual learning state emotion identification. Detailed Implementation

[0027] The technical solution of this application will be described in detail below with reference to specific embodiments.

[0028] A learning-based emotion recognition and intervention method includes the following steps:

[0029] S1: Obtain the facial image of the object to be identified. After face recognition, match it with the facial data of the object stored in the database. If the match is successful, execute S2; if the match is unsuccessful, execute S3.

[0030] S2: Obtain the historical emotion recognition time of the successfully matched recognition objects in the database. If the acquisition fails, execute S3. If the acquisition is successful, determine whether the difference between the historical emotion recognition time and the current time is greater than the first threshold. If it is greater than the first threshold, execute S3. If it is not greater than the first threshold, obtain the eye movement curve of the object to be recognized within the recognition time. Based on the correspondence between the historical eye movement curve and the learning state, obtain the learning state as the result of this emotion recognition and output it.

[0031] S3: Based on the facial image of the object to be identified, draw the eye movement curve of the object to be identified, obtain the eye movement curve of the object to be identified within the recognition time, and obtain the learning state as the result of this emotion recognition based on the correspondence between the general eye movement curve and the learning state, and output it.

[0032] S1: Obtain the facial image of the person to be identified. After face recognition, match it with the facial data of the person to be identified stored in the database. If the match is successful, it means that the person to be identified is not a new person and belongs to the existing person to be identified, and proceed to S2; if the match is unsuccessful, it means that the person to be identified is a new person and proceed to S3. The specific operations include the following.

[0033] By determining whether the object to be identified is already in the existing list of objects, we can distinguish between newly added objects and existing objects, and apply different identification logic and methods to the two types of objects.

[0034] For the original subject to be identified, if the eye movement curve data is rich enough, a correspondence between the original subject's own eye movement curve and learning state can be established, which can provide better emotion recognition of the learning state for individuals; if the eye movement curve data is not rich enough, emotion recognition of the learning state can be performed based on the correspondence between the general public's eye movement curve and learning state.

[0035] For newly added subjects to be identified, since there is no historical data, we can only first identify the emotion of the learning state based on the correspondence between the general public's eye movement curve and the learning state. If the eye movement curve of the subject to be identified reaches the set second threshold, we can establish the correspondence between the subject's eye movement curve and the learning state in a timely manner and carry out targeted individual learning state emotion identification.

[0036] If the last recognition time of an existing object to be identified exceeds the first threshold, it means that the data of this object has not been updated for too long. Considering that the mental state or habits of the object to be identified may change over time, this type of object needs to be processed as a newly added object to be identified, and the data of this object to be identified needs to be updated.

[0037] S2: Obtain the historical sentiment recognition time of successfully matched objects in the database. If the acquisition fails, it means that the object to be identified has not undergone sentiment recognition training and should be processed as a newly added object. Execute S3. If the acquisition is successful, it means that the object to be identified has undergone sentiment recognition training. To determine whether it can be processed as an original object, it is necessary to check whether the difference between the historical sentiment recognition time and the current time is greater than the first threshold. If it is greater than the first threshold, it means that the data of the object has not been updated for too long. Considering that the mental state or habits of the object may change over time, this type of object needs to be processed as a newly added object. Execute S3. If it is not greater than the first threshold, the data of the object has been updated recently. This type of object can be processed as an original object. The processing involves obtaining the eye movement curve of the object during the recognition time and obtaining the learning state as the result of this sentiment recognition based on the correspondence between the historical eye movement curve and the learning state. Output the result.

[0038] Neural networks are used to classify eye-tracking curves, obtaining the correspondence between eye-tracking curves and learning states within a unit recognition time. Eye-tracking curves are generated by acquiring several facial images of the subject to be recognized, locating the pupil positions in each image, and plotting the eye-tracking curve of the subject with time on the horizontal axis and pupil position on the vertical axis based on the correspondence between pupil position and time.

[0039] The neural network classifies eye movement curves and, based on the position and duration of the pupil within the recognition time, can correspond to learning states such as listening, distracted, and half-listening / half-distracted. Specific classification categories can be set.

[0040] For example, one classification rule is:

[0041] If the pupil position remains unchanged throughout the recognition period, it can be categorized as a state of inattentiveness. If the frequency of changes in pupil position and duration is moderate, neither too fast nor too slow, it can be categorized as a state of attentiveness. If the frequency of changes in pupil position and duration is too fast or too slow, it can be categorized as a state of partial attentiveness and partial inattentiveness.

[0042] If the emotion recognition result indicates a distracted state, a prompt message or sound effect is sent to the learning device of the target user to intervene in the user's learning state. Specifically, sending a prompt message or triggering a sound effect on the target user's learning device involves displaying a symbol on the target user's screen or playing a voice reminder on the target user's learning device.

[0043] If the number of eye movement curves accumulated by the subject over a historical period reaches the second threshold, a correspondence between the subject's own eye movement curve and learning state can be established, enabling better emotional recognition of the individual's learning state and forming the subject's historical eye movement curve.

[0044] This historical time also has a time period requirement, and cannot be considered to start from the time when the first identification record was generated. Considering that the mental state or habits of the identification object may change over time, a corresponding first time threshold will also be set for this historical time.

[0045] S3: Based on the facial image of the object to be identified, draw the eye movement curve of the object to be identified, obtain the eye movement curve of the object to be identified within the recognition time, and obtain the learning state as the result of this emotion recognition based on the correspondence between the general eye movement curve and the learning state, and output it.

[0046] If the emotion recognition result indicates a distracted state, a prompt message or sound effect is sent to the learning device of the target user to intervene in the user's learning state. Specifically, sending a prompt message or triggering a sound effect on the target user's learning device involves displaying a symbol on the target user's screen or playing a voice reminder on the target user's learning device.

[0047] Based on the facial image of the object to be identified, draw the eye movement curve of the object to be identified, specifically including:

[0048] Several facial images of the subjects to be identified are collected, and the pupil positions in the facial images are located. Based on the correspondence between pupil position and time, an eye movement curve of the subject to be identified is plotted with time on the horizontal axis and pupil position on the vertical axis.

[0049] Preferably, if the number of eye-tracking curves accumulated by the individual to be identified within a historical time period does not reach the second threshold, then the set of eye-tracking curves accumulated by all individuals to be identified within a historical time period that do not reach the second threshold is defined as the general eye-tracking curves. Because the individual's eye-tracking curve data is not rich enough, the classification result may be inaccurate. Therefore, the correspondence between general eye-tracking curves and learning states can be used first to obtain the learning state as the result of this emotion recognition. If the individual's eye-tracking data accumulates to a certain amount within the first time threshold, a correspondence between the individual's eye-tracking curves and learning states can be established, and the individual's learning state can be obtained as the result of this emotion recognition, allowing for targeted individual learning state emotion recognition.

[0050] A learning emotion recognition and intervention system, comprising:

[0051] Matching module: used to acquire the facial image of the object to be identified, and after face recognition, match it with the facial data of the object to be identified stored in the database;

[0052] Individual Emotion Recognition Module: Obtain the historical emotion recognition time of successfully matched recognition objects in the database. If successful, determine whether the difference between the historical emotion recognition time and the current time is greater than the first threshold. Obtain the eye movement curve of the object to be recognized within the recognition time. Based on the correspondence between the historical eye movement curve and the learning state, obtain the learning state as the result of this emotion recognition and output it.

[0053] The general emotion recognition module: Based on the facial image of the object to be recognized, the eye movement curve of the object to be recognized is drawn, the eye movement curve of the object to be recognized during the recognition time is obtained, and the learning state is obtained as the result of this emotion recognition based on the correspondence between the general eye movement curve and the learning state, and then output.

[0054] The aforementioned learning emotion recognition and intervention system also includes:

[0055] Eye movement curve plotting module: Collect several facial images of several objects to be identified, locate the pupil position in each facial image, and plot the eye movement curve of the object to be identified with time on the horizontal axis and pupil position on the vertical axis based on the correspondence between pupil position and time.

[0056] Intervention Module: If the result of this emotion recognition is a distracted state, a prompt message or a prompt sound effect will be sent to the learning device of the subject to be recognized to intervene in the learning state of the subject. Specifically, this involves displaying a symbol on the display screen of the subject to be recognized or playing a voice prompt on the learning device of the subject to be recognized.

[0057] Classification module: Uses a neural network to classify eye movement curves and obtain the correspondence between eye movement curves and learning states per unit time.

[0058] Threshold judgment and grouping module: If the number of eye movement curves accumulated by the subject to be identified in the historical time reaches the second threshold, then the historical eye movement curves are formed; if the number of eye movement curves accumulated by the subject to be identified in the historical time does not reach the second threshold, then the set of eye movement curves accumulated by all subjects to be identified in the historical time that do not reach the second threshold is defined as the general eye movement curves.

[0059] A learning emotion recognition and intervention device includes a processor and a memory, wherein the processor executes a computer program stored in the memory to implement the learning emotion recognition and intervention method.

[0060] A computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the learning emotion recognition and intervention method.

Claims

1. A learning emotion recognition and intervention method, characterized by, The method comprises the following steps: S1: obtaining a face image of a to-be-identified object, performing face recognition, and matching the face image with face data of an identified object stored in a database, if the matching is successful, performing S2, and if the matching is not successful, performing S3; S2: obtaining historical emotion recognition time of the identified object matched successfully in the database, if the obtaining is not successful, performing S3, if the obtaining is successful, judging whether a difference between the historical emotion recognition time and current time is greater than a first threshold value, if the difference is greater than the first threshold value, performing S3, if the difference is not greater than the first threshold value, obtaining an eye movement curve in a recognition time of the to-be-identified object, and obtaining a learning state as a result of this time emotion recognition according to a corresponding relationship between the historical eye movement curve and the learning state, and outputting the learning state; if a number of eye movement curves accumulated by the to-be-identified object in a historical time reaches a second threshold value, the historical eye movement curve is formed, if the number of eye movement curves accumulated by the to-be-identified object in the historical time does not reach the second threshold value, all eye movement curves accumulated by the to-be-identified object in the historical time that do not reach the second threshold value are defined as a public eye movement curve; S3: obtaining an eye movement curve in a recognition time of the to-be-identified object according to a face image of the to-be-identified object, and obtaining a learning state as a result of this time emotion recognition according to a corresponding relationship between the public eye movement curve and the learning state, and outputting the learning state.

2. The learning affect recognition and intervention method of claim 1, wherein, In S3, the eye movement curve of the to-be-identified object is drawn according to the face image of the to-be-identified object, and specifically comprises: a plurality of face images of a plurality of to-be-identified objects are collected, positions of pupils in the face images are located respectively, an eye movement curve of the to-be-identified object is drawn with time as a horizontal coordinate and the position of the pupil as a vertical coordinate according to a corresponding relationship between the position of the pupil and the time.

3. The learning affect recognition and intervention method of claim 1, wherein, If the result of this time emotion recognition is a daydreaming state, a prompt information or a prompt sound effect is sent to a learning device of the to-be-identified object, and the learning state of the to-be-identified object is intervened.

4. The learning emotion recognition and intervention method according to claim 2, characterized in that, a neural network is used to classify the eye movement curve, and a corresponding relationship between the eye movement curve in a unit time and the learning state is obtained.

5. The learning affect recognition and intervention method of claim 3, wherein, The prompt information or the prompt sound effect is sent to the learning device of the to-be-identified object, and specifically, a symbol is displayed on a display screen of the to-be-identified object or a voice prompt is played on the learning device of the to-be-identified object.

6. A learning emotion recognition and intervention system characterized by, comprise: a matching module, configured to obtain a face image of a to-be-identified object, perform face recognition, and match the face image with face data of an identified object stored in a database; an individual emotion recognition module, configured to obtain historical emotion recognition time of the identified object matched successfully in the database, if the obtaining is successful, judge whether a difference between the historical emotion recognition time and current time is greater than a first threshold value, if the difference is not greater than the first threshold value, obtain an eye movement curve in a recognition time of the to-be-identified object, and obtain a learning state as a result of this time emotion recognition according to a corresponding relationship between the historical eye movement curve and the learning state, and output the learning state; If the number of eye movement curves accumulated by the to-be-identified object himself in the historical time reaches a second threshold, the historical eye movement curves are constituted, and if the number of eye movement curves accumulated by the to-be-identified object himself in the historical time does not reach the second threshold, all eye movement curves accumulated by the to-be-identified objects in the historical time that do not reach the second threshold are defined as a public eye movement curve; The public emotion recognition module: according to the face image of the to-be-identified object, the eye movement curve of the to-be-identified object in the recognition time is obtained, the learning state is obtained as the result of this time emotion recognition according to the corresponding relationship between the public eye movement curve and the learning state, and is output. 7.A learning emotion recognition and intervention device, characterized in that, A computer program product comprising a processor and a memory, wherein the processor implements the learning emotion recognition and intervention method according to any one of claims 1-5 when executing the computer program stored in the memory.

8. A computer-readable storage medium, characterized in that, A computer program product comprising a processor and a memory, wherein the processor implements the learning emotion recognition and intervention method according to any one of claims 1-5 when executing the computer program stored in the memory.