Emotion recognition ability adaptive training method and device based on eye movement feedback

Abstract

The application discloses an emotion recognition ability adaptive training method and device based on eye movement feedback. The method comprises the following steps: acquiring visual stimulation materials of the current training, including virtual character expression animations of at least two different emotion types; showing the virtual character expression animations of different emotion types to a trained object in a random order, and issuing an instruction to the trained object to distinguish the emotion category corresponding to each virtual character expression animation; acquiring eye movement trajectory data of the training in real time, and acquiring a distinguishing result of each virtual character expression animation of each emotion type; obtaining a similarity of each virtual character expression animation of each emotion type according to the eye movement trajectory data and the distinguishing result corresponding to each virtual character expression animation of each emotion type; and adjusting an expression intensity parameter of the virtual character expression animation of each emotion type according to the similarity of each emotion type, and using the visual stimulation materials as visual stimulation materials of the next round of training, so as to adaptively adjust the intensity of the visual stimulation materials.

Description

Technical Field

[0001] This invention belongs to the field of computer graphics technology, specifically, it relates to an adaptive training method and device for emotion recognition ability based on eye-tracking feedback, a computer-readable storage medium, and a computer device. Background Technology

[0002] Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by varying degrees of impairment in social communication and interaction, as well as stereotyped behaviors and restricted interests. These symptoms appear in early childhood and limit or impair daily functioning. Due to the high disability rate of ASD, 58%–78% of adults with autism experience varying degrees of difficulty in self-care, education, employment, and peer relationships. Difficulty in understanding others' emotions is considered a core characteristic of autism and the most important cause of impaired social communication and interaction. Studies have shown that autistic children are less accurate than typically developing children in recognizing facial expressions in photos, cartoons, and video clips. This is because autistic children cannot effectively focus their attention on the key areas of facial expressions in conversations, thus affecting their ability to recognize emotions.

[0003] To address the issue of impaired emotion recognition in individuals with autism, scholars in related fields have conducted a series of explorations in theoretical research and rehabilitation methods. However, regarding the stimulus materials used in training, existing technologies lack strong correlation with facial expressions and movements, and cannot adaptively change according to the training progress of autistic patients. For example, simply using fixed pictures of real people, videos, or certain real-life scenarios to directly train autistic patients' emotion recognition abilities makes it difficult for them to gain a basic understanding of facial expressions and movements from these stimulus materials, thus failing to enhance their meta-learning abilities in emotion perception. Summary of the Invention

[0004] (I) The technical problem to be solved by the present invention

[0005] The technical problem solved by this invention is: how to adaptively select visual stimulus materials of appropriate intensity according to the training situation of the trainee during the training process, so as to enhance the emotional perception meta-learning ability of the trainee.

[0006] (II) Technical Solution Adopted in this Invention

[0007] An adaptive training method for emotion recognition ability based on eye-tracking feedback, the method comprising:

[0008] Obtain visual stimulus materials for this round of training, wherein the visual stimulus materials include virtual character facial expression animations of at least two different emotion types;

[0009] The virtual character facial animations of different emotion types are displayed to the training subjects in a random order, and when the virtual character facial animation of each emotion type is displayed, an instruction is issued to the training subjects to judge the emotion category corresponding to each virtual character facial animation.

[0010] Real-time acquisition of eye-tracking data of the trainee when observing virtual character facial expressions of different emotion types, and acquisition of the trainee's discrimination results for each type of virtual character facial expression.

[0011] The similarity of virtual character facial expressions for each emotion type is obtained based on the eye-tracking data and discrimination results.

[0012] Adjust the expression intensity parameters of the virtual character facial animations for each emotion type based on the similarity of each emotion type, and use the various virtual character facial animations with adjusted expression intensity parameters as visual stimulus materials for the next round of training. Repeat the above steps until the training completion conditions are met.

[0013] Preferably, the judgment result includes correct judgment result and / or incorrect judgment result. When there is an incorrect judgment result, after obtaining the judgment result of the training object for each virtual character facial expression animation, the method further includes:

[0014] The training object is assisted by using the virtual character facial expression animation corresponding to the error discrimination result, so that the eye movement trajectory data of the training object when observing the virtual character facial expression animation reaches the threshold condition.

[0015] Preferably, when there are erroneous judgment results, the method for obtaining the similarity of virtual character facial expressions for each emotion type based on the eye-tracking data and judgment results includes:

[0016] Set the similarity of the virtual character's facial expression animation corresponding to the incorrect judgment result to zero. A similarity of zero means that the facial expression intensity parameter of the virtual character's facial expression animation remains unchanged.

[0017] Preferably, when a correct judgment result exists, the method for obtaining the similarity of virtual character facial animations for each emotion type based on the eye-tracking data and the judgment result includes:

[0018] Based on the eye movement trajectory data of the virtual character's facial expression animation corresponding to the correct judgment result, extract the single-person gaze trajectory features and the single-person gaze trajectory path length;

[0019] The editing distance is obtained based on the individual gaze trajectory characteristics and the pre-acquired normal person gaze trajectory trend characteristics;

[0020] The similarity of virtual character facial animations for each emotion type is calculated based on the edit distance and the path length of the single-person gaze trajectory.

[0021] Preferably, the method of adjusting the expression intensity parameters of virtual character facial animations for each emotion type based on the similarity of each emotion type, and using the various virtual character facial animations with adjusted expression intensity parameters as visual stimulus material for the next round of training includes:

[0022] The expression intensity parameter is determined based on a preset conversion relationship between similarity and expression intensity parameter, wherein the expression intensity parameter includes expression density coefficient and expression duration;

[0023] The facial control point coordinates are obtained based on the expression density coefficient, and the new coordinates of the virtual character's facial deformation points are obtained using the Dirichlet free deformation algorithm.

[0024] The virtual character's facial expression animation is synthesized by adjusting the expression intensity parameters based on the new coordinate values ​​of the points to be deformed on the virtual character's face and the duration of the expression.

[0025] Preferably, after obtaining the similarity of the virtual character facial expressions for each emotion type, the method further includes:

[0026] Determine whether the similarity of the virtual character's facial expression animation for each emotion type is greater than or equal to a predetermined value;

[0027] If so, the virtual character facial animation with a similarity greater than or equal to a predetermined value will be removed from the visual stimulus material in the next round of training.

[0028] Preferably, the method of using the virtual character facial animation corresponding to the error discrimination result to assist the training object, so that the eye movement trajectory data of the training object when observing the virtual character facial animation reaches a threshold condition, includes:

[0029] The virtual character's facial expression animation corresponding to the incorrect judgment result is re-exposed to the training subject. At the same time, gaze navigation auxiliary boxes are added to the virtual character's facial expression animation in a predetermined order. The gaze navigation auxiliary boxes are used to indicate the gaze area.

[0030] The eye-tracking data of the trainee when observing the facial expressions of virtual characters of different emotional types is monitored in real time. Based on the eye-tracking data, it is determined whether the order in which the trainee observes the gaze area is consistent with the order in which the gaze navigation auxiliary box is added.

[0031] If yes, end the assisted training; if not, repeat the above steps.

[0032] This application also discloses an adaptive training device for emotion recognition ability based on eye-tracking feedback, the device comprising:

[0033] An animation acquisition unit is used to acquire visual stimulus materials for this round of training, wherein the visual stimulus materials include at least two different types of virtual character facial expression animations.

[0034] The user interaction unit is used to display virtual character facial animations of different emotion types to the training subject in a random order, and when the virtual character facial animation of each emotion type is displayed, it issues an instruction to the training subject to judge the emotion category corresponding to each virtual character facial animation.

[0035] The eye-tracking unit is used to acquire in real time the eye movement trajectory data of the training subject when observing virtual character facial expressions of different emotion types, and to acquire the discrimination results of the training subject for each virtual character facial expression.

[0036] The data processing unit is used to obtain the similarity of virtual character facial expressions for each emotion type based on the eye movement trajectory data and judgment results.

[0037] The parameter adjustment unit is used to adjust the expression intensity parameters of the virtual character facial animation of each emotion type according to the similarity of each emotion type, and use the various virtual character facial animations after adjusting the expression intensity parameters as visual stimulus materials for the next round of training. The above steps are repeated until the training completion conditions are met.

[0038] This application also discloses a computer-readable storage medium storing an adaptive training program for emotion recognition ability based on eye-tracking feedback. When the adaptive training program for emotion recognition ability based on eye-tracking feedback is executed by a processor, it implements the above-described adaptive training method for emotion recognition ability based on eye-tracking feedback.

[0039] This application also discloses a computer device, which includes a computer-readable storage medium, a processor, and an eye-tracking feedback-based adaptive training program for emotion recognition ability stored in the computer-readable storage medium. When the eye-tracking feedback-based adaptive training program is executed by the processor, it implements the above-described eye-tracking feedback-based adaptive training method for emotion recognition ability.

[0040] (III) Beneficial Effects

[0041] This invention discloses an adaptive training method for emotion recognition ability based on eye-tracking feedback, which has the following technical advantages compared to traditional training methods:

[0042] Regarding the stimulus materials used in training, virtual character avatar animations ranging from exaggerated to subtle intensity are employed as visual stimuli. These stimuli can be adaptively selected based on the autistic patient's training progress, facilitating the patient's ability to distinguish and learn subtle changes in facial expressions and movements, and enhancing their emotional perception meta-learning abilities. In terms of training task design, the tasks in this embodiment range from simple to complex, and the training effect on the trainees can be evaluated using the proposed similarity calculation method, allowing for adaptive and dynamic adjustments to the training tasks. Attached Figure Description

[0043] Figure 1 This is a flowchart of an adaptive training method for emotion recognition ability based on eye-tracking feedback, according to Embodiment 1 of the present invention.

[0044] Figure 2 This is a flowchart of the similarity calculation method according to Embodiment 1 of the present invention;

[0045] Figure 3 This is a schematic diagram of visual element division according to Embodiment 1 of the present invention;

[0046] Figure 4 This is a flowchart illustrating the generation process of virtual character facial expression animation according to Embodiment 1 of the present invention.

[0047] Figure 5 This is a schematic diagram of the auxiliary training process according to Embodiment 1 of the present invention;

[0048] Figure 6 This is a schematic diagram of the principle framework of the emotion recognition ability adaptive training device based on eye-tracking feedback according to Embodiment 2 of the present invention.

[0049] Figure 7 This is a schematic diagram of a computer device according to Embodiment 4 of the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0051] Before describing the various embodiments of this application in detail, the inventive concept of this application is first briefly described: In the prior art, when training autistic patients, the visual stimulation materials used are generally fixed pictures and videos, but patients have difficulty obtaining effective training. This application proposes an adaptive training method for emotion recognition ability based on eye-tracking feedback. Virtual character facial animation is used as visual stimulation material. During the training process, the eye-tracking trajectory data and emotion type discrimination results of the trainee when observing the virtual character facial animation are monitored in real time. The similarity is further calculated, and the facial intensity parameters of the virtual character facial animation to be used in the next round of training are adjusted in real time according to the similarity. Thus, the facial intensity of the visual stimulation material is adjusted in each round of training. The facial intensity of the stimulation material changes from exaggerated to slight intensity throughout the process, which is conducive to the trainee distinguishing and learning subtle changes in facial expressions and enhancing the patient's emotional perception meta-learning ability.

[0052] like Figure 1 As shown, the adaptive training method for emotion recognition ability based on eye-tracking feedback in this embodiment includes the following steps:

[0053] Step S10: Obtain the visual stimulus material for this round of training, wherein the visual stimulus material includes at least two different types of virtual character facial expression animations;

[0054] Step S20: Show the training subject virtual character facial animations of different emotion types in a random order, and when the virtual character facial animation of each emotion type is finished being shown, issue an instruction to the training subject to identify the emotion category corresponding to each virtual character facial animation.

[0055] Step S30: In real time, acquire the eye movement trajectory data of the training subject when observing virtual character facial expression animations of different emotion types, and acquire the discrimination results of the training subject for each emotion type of virtual character facial expression animation;

[0056] Step S40: Based on the eye movement trajectory data and judgment results corresponding to the virtual character facial expression animations of each emotion type, obtain the similarity corresponding to each emotion type of virtual character facial expression animation;

[0057] Step S50: Adjust the expression intensity parameters of the virtual character facial expression animations of each emotion type according to the similarity of each emotion type, and use the various virtual character facial expression animations after adjusting the expression intensity parameters as visual stimulus materials for the next round of training. Repeat the above steps until the training completion conditions are met.

[0058] For example, during initial training, the facial expression intensity parameters of the visual stimulus material are set to K=1 and T=5, where K represents the expression density coefficient and T represents the duration. K=1 represents an exaggerated expression, and T=5 represents that the expression lasts for 5 seconds. To achieve good training results, each set of visual stimulus materials needs to include several virtual character facial expression animations of different emotion types, and these need to be shown to the training subjects in a random order. For example, the visual stimulus material for initial training contains six types of virtual character facial expression animations: happy, sad, neutral, angry, disgusted, and surprised.

[0059] Furthermore, in practice, virtual character facial animations are played on a display screen. An eye-tracking device can be installed above the screen to monitor the eye movement data of the training subject in real time as they observe virtual character facial animations of different emotion types. After each segment of virtual character facial animation finishes playing, the screen displays a prompt indicating the corresponding emotion category. The training subject can select the emotion category by touching the screen or clicking with a mouse, thereby obtaining the training subject's judgment result for each emotion type of virtual character facial animation. It should be noted that the instruction to the training subject to judge the emotion category corresponding to each virtual character facial animation can also take other forms, such as voice prompts, and this embodiment is not limited to these.

[0060] After a set of virtual character facial expression animations finished playing, eye-tracking data and discrimination results were obtained for each emotion type of virtual character facial expression animation. The discrimination results may include correct and incorrect results. When an incorrect result exists, the similarity of the virtual character facial expression animation corresponding to the incorrect result is set to zero. A similarity of zero indicates that the expression intensity parameter of the virtual character facial expression animation remains unchanged. If an incorrect result exists, it means that the training subject cannot recognize the expression of that emotion type at all. In the next round of training, it is necessary to repeat the training using virtual character facial expression animations of the same intensity; therefore, the similarity is set to zero.

[0061] Furthermore, a correct judgment indicates that the trained subject can recognize the expression of that emotion type. However, it is necessary to further determine the degree to which the trained subject recognizes the expression to determine the intensity of that type of expression in the next round of training. Similarity is used to represent the degree of recognition. Specifically, the similarity calculation method for the virtual character's facial expression animation corresponding to the correct judgment includes the following steps:

[0062] Step S41: Extract single-person gaze trajectory features and single-person gaze trajectory path length from the eye movement trajectory data of the virtual character's facial expression animation corresponding to the correct judgment result;

[0063] Step S42: Obtain the editing distance based on the single person's gaze trajectory characteristics and the pre-acquired normal person's gaze trajectory characteristics;

[0064] Step S43: Calculate the similarity of virtual character facial animations for each emotion type based on the edit distance and the path length of the single-person gaze trajectory.

[0065] For example, in step S41, since the eye-tracking data includes the gaze projection position, this embodiment first segments the virtual character's facial expression animation as follows: Figure 3 The visual element areas shown are intended to capture the patient's focus and path to the virtual character's facial expressions. There are three areas in total: the facial expression generation area [A, B, C, D, E, F, G], the non-facial expression generation area [J, K, L, M, N, O], and other irrelevant areas [P, Q, R, S, T, U].

[0066] Furthermore, the training subject's gaze duration on the screen display content and the individual visual elements of the display are taken as input, and then the single-person gaze trajectory is represented by the visual elements and the gaze duration on these elements. For example, when a single-person gaze trajectory is represented as A[250ms]B[150ms]C[200ms], it means that the person viewed visual elements A, B, and C in sequence for 250ms, 150ms, and 200ms, respectively. Visual elements can exist continuously (ABBC) or discontinuously (ABAC) in a single-person gaze trajectory. Each discontinuous occurrence of a visual element along a path is called an instance of that element. Instances of a particular visual element are distinguished from each other using numbers, with the longest instance represented by the subscript 1. For example, A2

[150] B1

[150] B1

[200] A1

[250] , then consecutive identical elements are merged, such as A2

[150] B1

[150] B1

[200] A1

[250] →A2

[150] B1

[350] A1

[250] . This path is a single-person gaze trajectory feature. The path length of the single-person gaze trajectory is the length of the string. For example, the path length of A2

[150] B1

[350] A1

[250] is 3.

[0067] In step S42, since this embodiment requires comparing the single gaze trajectory of the training subject with the gaze trajectory of normal people, it is necessary to first count the gaze trajectories of multiple normal people on the virtual character's facial expression animation. Before training the training subject, the same visual stimulus material is shown to multiple normal people to obtain the eye movement trajectory data of each normal person. The gaze trajectory features of each normal person are obtained according to the above extraction method, and then the gaze trajectory trend features of normal people are calculated.

[0068] Specifically, the formula for calculating the priority value of each visual element in the gaze trajectory features of each normal person is as follows:

[0069]

[0070] Among them Ψ i P represents the order priority value of the i-th visual element in the gaze trajectory features of each normal person. i L1 represents the position of the visual element in the gaze trajectory features of each normal person (starting from 0), L1 represents the length of the gaze trajectory path of a normal person, and max represents the position of the visual element. i This represents the maximum order priority value, which defaults to 1. i The minimum priority value is represented by the following formula:

[0071] min i =1 / L1

[0072] For example, when L1 = 10, min i =1. Therefore, we can obtain the order priority value of each instance in each path.

[0073] For example, suppose the gaze trajectory of normal person A is “ABCDE”, the path length is 5, and the positions of visual elements A, B, C, D, and E are P1=0, P2=1, P3=2, P4=3, and P5=4, respectively. Therefore, according to the above calculation formula, the priority values ​​of visual elements A, B, C, D, and E of A are Ψ1=1 (i.e., A=1), B=0.8, C=0.6, D=0.4, and E=0.2, respectively. Suppose the gaze trajectory of normal person B is “ACBD”. According to the above calculation formula, the priority values ​​of visual elements B are A=1, C=0.75, B=0.5, and D=0.25.

[0074] Secondly, based on the order priority values ​​of visual elements from multiple normal individuals, the total order priority value of each visual element is calculated. Let there be a total of u visual elements, y = {y1, y2, ... y}. u Let} be the set of all visual elements, then the calculation can be performed using the following formula:

[0075]

[0076] Calculate the total order priority value for each visual element, where This represents the total priority value of visual element y1. Let y1 represent the order priority value of the visual element y1 of the i-th person. Then, sort the visual elements in descending order according to their total order priority values ​​to generate the normal person's gaze trajectory trend feature. For the same priority value, the total duration and the total number of instances are also considered. Then, remove the instance's sub-indicator, such as A1→A, and exclude duplicates such as ABBC→ABC. Finally, the trend feature is expressed by the visual elements. For example, based on the order priority values ​​of normal persons A and B calculated above, the total order priority values ​​of each visual element for A and B are A = 2, B = 1.3, C = 1.35, D = 0.65, and E = 0.2, respectively. Therefore, the normal person's gaze trajectory trend feature obtained from the gaze trajectory features of A and B is ACBDE.

[0077] Further, in step S43, the edit distance D represents the minimum number of editing operations (addition, deletion, and / or replacement) required to convert one string into another. For example, if the strings representing the single-view gaze trajectory features of the training object and the trajectory features of a normal person are "ABCDE" and "BD" respectively, and A, C, and E are deleted, then the edit distance D = 3. The formula for calculating similarity is as follows:

[0078]

[0079] S represents the similarity, and L represents the single-person gaze trajectory path length of the training object. In the above embodiment, the similarity is S = 100 * (1 - 3 / 5) = 40, that is, the similarity between "ABCDE" and "BD" is 40%.

[0080] Further, in step S50, based on a pre-defined mapping relationship, the facial expression intensity parameters of the visual stimulus material for the next round of training are determined according to the similarity. The mapping relationship is as follows:

[0081]

[0082]

[0083] Where k represents the expression concentration coefficient and T represents the expression duration. The virtual character's facial expression animation intensity corresponding to S∈[0,25) is exaggerated, the virtual character's facial expression animation intensity corresponding to S∈[25,50) is normal, and the virtual character's facial expression animation intensity corresponding to S∈[50,75) is minimal.

[0084] Furthermore, the mapping relationship between the expression density coefficient k and the facial control point coordinates p is as follows:

[0085]

[0086] Where, p i d represents the coordinates of the i-th facial control point under the current expression density coefficient k. i This represents the distance between the coordinates of the i-th facial control point when it has no expression and the coordinates when it has an exaggerated expression. This represents the coordinates of the i-th facial control point when it is expressionless. Therefore, when k equals 1, it represents the coordinates of the exaggeration level intensity of the i-th facial control point.

[0087] Furthermore, based on the obtained facial control point coordinates, the new coordinates of the virtual character's facial deformation points are obtained using the Dirichlet free deformation algorithm. This allows for the synthesis of a virtual character's facial animation with adjusted expression intensity parameters based on the new coordinates of these deformation points and the duration of the expression. The specific process is as follows: Figure 4 As shown, the Dirichlet Free Deformation (DFFD) algorithm is existing technology. This embodiment only describes the main steps of the algorithm, including the following:

[0088] Given a set of control points P = {p1, p2, ..., p...} n Let P be a set of points P and any point x. The coordinates of any point x can be represented by the coordinate differences between points in the set P. Sibson coordinates are a local coordinate system. Point x is only affected by the control points around it that can influence it. Therefore, the set of control points affecting point x may be all points in set P or a subset of P. Assuming that all points in set P are control points affecting x, then x can be represented by the following formula.

[0089]

[0090] in, andu i >0 (0≤i≤n). Let P′={p′1,p′2,…,p′ n Let} be the point set P = {p1, p2, ..., p n Let x be the coordinates of the control point after it has moved. As the control point moves, the coordinates of point x also change accordingly. Let the new coordinates of point x after the move be x′, and Δx be the displacement difference, i.e., x′=x+Δx. Therefore, to calculate x′, we only need to find Δx. The process for calculating Δx is given below.

[0091] Assume the set of control points affecting the movement of point x is P = {p1, p2, ..., p...} n Its coordinates after the movement are represented as P′={p′1,p′2,…,p′}.n}, where p′ i For p i The coordinates after the movement, Δp i Representing point p′ i With point p i The displacement difference, i.e., p′ i =p i +Δp i The following section first uses the point set P = {p1, p2, ..., p...} n Construct an n-dimensional, m-th degree Bezier simplex, where m is a constant, typically 1 or 2.

[0092] Let B = {q} I}(I=(i1,i2,…,i n ),|I|=i1+i2+…+i n =m) is the control point set of an n-dimensional m-order Bezier simplex, and its calculation method is given by the following formula (when m=1, then B==P, and this step can be omitted).

[0093]

[0094] Where, p J (j1,j2,…,j n ),|J|=j1+j2+…+j n =1) is p i (p i Another representation of ∈P, namely p (1,0,…,0) =p1,p (0,1,…,0) =p2,p (0,0,…,n) =p n . These are Bernstein basis functions, given by the following formula.

[0095]

[0096] Where i+j+k=n, r+s+t=1. Let B={q′ I}(I=(i1,i2,…,i n ),|I|=i1+i2+…+i n =m) is the set of control points of the n-dimensional m-order Bezier simplex formed by the movement of point set P′, where q′ I =q I +Δq I Therefore, Δx can be calculated using the following formula.

[0097]

[0098] Where U represents the control point q corresponding to point x. IThe Sibson coordinates of point x are obtained. Therefore, by calculating the neighboring control points and their corresponding Sibson coordinates, Δx can be calculated using the above method. Then, x′ can be calculated using x′=x+Δx, which is the new coordinate value of the point to be deformed on the virtual character's face. Based on the input facial control point coordinates, the new coordinates after each deformation can be calculated. Combined with the expression duration, OpenGL technology can be used to generate new virtual character facial animations. The key to implementing the Dirichlet free deformation algorithm lies in the Delaunay partitioning in three-dimensional coordinates and the calculation of Sibson coordinates. These two parts are existing technologies and will not be elaborated upon here.

[0099] According to steps S10 to S50 above, adaptive selection of visual stimulus materials of different intensities can be achieved. The virtual character facial expression animation mainly used in this embodiment includes three intensity levels: exaggerated (k=1, T=5), normal (k=0.6, T=3), and subtle (k=0.3, T=1). When the similarity of the gaze trajectory obtained after training is between [0, 25), the exaggerated intensity facial expression animation is played. This is because the main areas where people's expressions are generated are the mouth, eyebrows, eyes, forehead, and cheeks. A similarity close to 0 means that the trainee has not paid attention to or has not followed the normal person's strategy of paying attention to the expression generation area. Therefore, the exaggerated intensity animation is played to train the patient. If the similarity is between [25, 50), the normal intensity facial expression animation is played; if the similarity is between [50, 75), the smiling intensity facial expression animation is played.

[0100] Furthermore, in another embodiment, when there are erroneous judgment results, the judgment results include correct judgment results and / or erroneous judgment results. After obtaining the judgment results of the training object for each virtual character facial animation, the method further includes: using the virtual character facial animation corresponding to the erroneous judgment results to perform auxiliary training on the training object, so that the eye movement trajectory data of the training object when observing the virtual character facial animation reaches a threshold condition. Specifically, the auxiliary training includes the following steps:

[0101] Step 1: Re-show the virtual character's facial expression animation corresponding to the incorrect judgment result to the training subject, and add gaze navigation auxiliary boxes to the virtual character's facial expression animation in a predetermined order. The gaze navigation auxiliary boxes are used to indicate the gaze area.

[0102] Step 2: Monitor the eye-tracking data of the trainee when observing the virtual character's facial expressions of different emotion types in real time, and determine whether the order in which the trainee observes the gaze area is consistent with the order in which the gaze navigation aid box is added based on the eye-tracking data.

[0103] Step 3: If yes, end the assisted training; if no, repeat the above steps.

[0104] In actual operation, such as Figure 5 As shown, when the visual stimulus material begins to play, a gaze navigation aid is superimposed on the visual stimulus material according to the gaze trajectory sequence of a normal person, emphasizing the gaze area. Based on real-time eye-tracking data, the gaze navigation aid disappears when the patient projects their gaze onto the emphasized gaze area. When all gaze navigation aids disappear correctly in sequence, the screen displays "You're awesome!" and the corresponding emoticon label. If the patient does not project their gaze onto the emphasized gaze area in sequence, a "Failed, please try again" prompt appears after the animation finishes. Through assisted training, the trainee's ability to recognize the virtual character's facial expressions of this emotion type can be further improved.

[0105] Furthermore, in another embodiment, after obtaining the similarity scores corresponding to the virtual character facial expressions for each emotion type, the method further includes:

[0106] The similarity score of the virtual character facial expressions for each emotion type is determined to be greater than or equal to a predetermined value. If yes, the virtual character facial expressions with a similarity score greater than or equal to the predetermined value are removed from the visual stimulus material in the next round of training; otherwise, they are retained in the visual stimulus material in the next round of training. For example, the predetermined value is set to 75. After two rounds of training, the similarity scores of the six types of virtual character facial expressions are S... happy =80, S sad =78, S neutral =50, S angry =60, S disgust =30 and S surprise =20. The similarity of virtual character facial expressions for the two emotion types, happiness and sadness, is greater than 75, indicating that the training subject's ability to recognize these two emotions is strong enough. There is no need to train virtual character facial expressions of this emotion type in the next round of training. Virtual character facial expressions of this emotion type can be removed in the next round of training.

[0107] The adaptive training method for emotion recognition ability based on eye-tracking feedback disclosed in Embodiment 1 uses virtual character avatar animations ranging from exaggerated to subtle intensity as visual stimuli. These stimuli can be adaptively selected based on the training progress of autistic patients, which helps them distinguish and learn subtle changes in facial expressions and enhances their meta-learning ability in emotion perception. Regarding training task design, the tasks in this embodiment range from simple to complex, and the training effect on the trainees can be evaluated using the proposed similarity calculation method, allowing for adaptive dynamic adjustment of the training tasks.

[0108] Furthermore, such as Figure 6 As shown in the figure, this embodiment 2 also discloses an adaptive training device for emotion recognition ability based on eye-tracking feedback. The device includes an animation acquisition unit 100, a user interaction unit 200, an eye-tracking unit 300, a data processing unit 400, and a parameter adjustment unit 500. Animation acquisition unit 100 is used to acquire visual stimulus materials for the current training round, including at least two different types of virtual character facial expression animations. User interaction unit 200 is used to display virtual character facial expression animations of different emotional types to the training subject in a random order, and when the display of each type of virtual character facial expression animation is completed, it issues an instruction to the training subject to judge the emotion category corresponding to each type of virtual character facial expression animation. Eye tracking unit 300 is used to acquire in real time the eye movement trajectory data corresponding to the training subject when observing virtual character facial expression animations of different emotional types, and to acquire the judgment result of the training subject for each type of virtual character facial expression animation. Data processing unit 400 is used to obtain the similarity corresponding to each type of virtual character facial expression animation based on the eye movement trajectory data and judgment result. Parameter adjustment unit 500 is used to adjust the expression intensity parameter of the virtual character facial expression animation of each emotion type according to the similarity of each emotion type, and use the various virtual character facial expression animations after adjusting the expression intensity parameter as visual stimulus materials for the next round of training, repeating the above steps until the training completion condition is met.

[0109] Furthermore, the judgment result includes a correct judgment result and / or an incorrect judgment result. When an incorrect judgment result exists, the user interaction unit 200 is also used to assist the training object by using the virtual character facial expression animation corresponding to the incorrect judgment result, so that the eye movement trajectory data of the training object when observing the virtual character facial expression animation reaches the threshold condition. Specifically, the user interaction unit 200 is also used to re-show the virtual character facial expression animation corresponding to the incorrect judgment result to the training object, and add gaze navigation auxiliary boxes to the virtual character facial expression animation in a predetermined order. The gaze navigation auxiliary boxes are used to indicate the gaze area. The eye tracking unit 300 is also used to monitor the eye tracking data corresponding to the training object when observing virtual character facial expression animations of different emotion types in real time, and determine whether the observation order of the gaze area by the training object is consistent with the addition order of the gaze navigation auxiliary boxes based on the eye tracking data. If yes, the assisted training ends; if not, the above steps are repeated.

[0110] Furthermore, when an incorrect judgment result exists, the data processing unit 400 is also used to set the similarity of the virtual character's facial expression animation corresponding to the incorrect judgment result to zero. A similarity of zero indicates that the facial expression intensity parameter of the virtual character's facial expression animation remains unchanged. When a correct judgment result exists, the data processing unit 400 is also used to extract single-person gaze trajectory features and single-person gaze trajectory path length based on the eye movement trajectory data of the virtual character's facial expression animation corresponding to the correct judgment result; obtain the edit distance based on the single-person gaze trajectory features and the pre-acquired normal person gaze trajectory trend features; and calculate the similarity corresponding to each type of virtual character's facial expression animation based on the edit distance and the single-person gaze trajectory path length.

[0111] Further, the parameter adjustment unit 500 is used to: determine the expression intensity parameter based on a preset conversion relationship between similarity and expression intensity parameter, wherein the expression intensity parameter includes expression density coefficient and expression duration; obtain facial control point coordinate values ​​based on the expression density coefficient, and obtain new coordinate values ​​of the virtual character's facial deformation points using the Dirichlet free deformation algorithm; and synthesize the adjusted virtual character expression animation based on the new coordinate values ​​of the virtual character's facial deformation points and expression duration. Further, after obtaining the similarity corresponding to the virtual character expression animation for each emotion type, the parameter adjustment unit 500 is also used to: determine whether the similarity corresponding to the virtual character expression animation for each emotion type is greater than or equal to a predetermined value; if so, delete the virtual character expression animation with a similarity greater than or equal to the predetermined value from the visual stimulus material in the next round of training.

[0112] The more detailed working process of the animation acquisition unit 100, user interaction unit 200, eye tracking unit 300, data processing unit 400 and parameter adjustment unit 500 can be referred to the relevant description in Embodiment 1, and will not be repeated here.

[0113] This embodiment also discloses a computer-readable storage medium storing an adaptive training program for emotion recognition ability based on eye-tracking feedback. When the adaptive training program for emotion recognition ability based on eye-tracking feedback is executed by a processor, it implements the above-described adaptive training method for emotion recognition ability based on eye-tracking feedback.

[0114] This fourth embodiment also discloses a computer device, at the hardware level, such as... Figure 7As shown, the terminal includes a processor 12, an internal bus 13, a network interface 14, and a computer-readable storage medium 11. The processor 12 reads the corresponding computer program from the computer-readable storage medium and runs it, forming a request processing device at the logical level. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices. The computer-readable storage medium 11 stores an adaptive training program for emotion recognition ability based on eye-tracking feedback. When the processor executes the adaptive training program for emotion recognition ability based on eye-tracking feedback, it implements the above-described adaptive training method for emotion recognition ability based on eye-tracking feedback.

[0115] Computer-readable storage media include both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer-readable storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage, quantum memory, graphene-based storage media or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0116] The specific embodiments of the present invention have been described in detail above. Although some embodiments have been shown and described, those skilled in the art should understand that modifications and improvements can be made to these embodiments without departing from the principles and spirit of the present invention as defined by the claims and their equivalents, and such modifications and improvements should also be within the protection scope of the present invention.

Claims

1. An adaptive training method for emotion recognition ability based on eye-tracking feedback, characterized in that, The method includes: Obtain visual stimulus materials for this round of training, wherein the visual stimulus materials include virtual character facial expression animations of at least two different emotion types; The system displays virtual character facial animations of different emotion types to the trainees in a random order, and when the virtual character facial animation of each emotion type is displayed, it issues an instruction to the trainees to identify the emotion type corresponding to each virtual character facial animation. The eye-tracking data of the training subject when observing virtual character facial expressions of different emotion types are acquired in real time, and the discrimination results of the training subject for each type of virtual character facial expression are acquired. The discrimination results include correct discrimination results and / or incorrect discrimination results. The similarity of virtual character facial animations for each emotion type is obtained based on the eye-tracking trajectory data and discrimination results corresponding to the facial expressions of virtual characters for each emotion type. When there are incorrect discrimination results, the method for obtaining the similarity of virtual character facial animations for each emotion type based on the eye-tracking trajectory data and discrimination results includes: setting the similarity of the virtual character facial animations corresponding to the incorrect discrimination results to zero, where zero similarity indicates that the expression intensity parameter of the virtual character facial animation remains unchanged. When there are correct discrimination results, the method for obtaining the similarity of virtual character facial animations for each emotion type based on the eye-tracking trajectory data and discrimination results includes: extracting single-person gaze trajectory features and single-person gaze trajectory path length from the eye-tracking trajectory data of the virtual character facial animations corresponding to the correct discrimination results; obtaining the edit distance based on the single-person gaze trajectory features and pre-acquired normal person gaze trajectory trend features; and calculating the similarity of virtual character facial animations for each emotion type based on the edit distance and the single-person gaze trajectory path length. The eye-tracking trajectory data includes gaze projection positions. Virtual character facial animations are segmented into visual element regions. The training subject's gaze duration on the displayed content and each visual element are used as input. The single-person gaze trajectory is then represented by the visual elements and the gaze duration on those elements. Visual elements may exist continuously or discontinuously in the single-person gaze trajectory. Each discontinuous occurrence of a visual element along a path is called an instance of that element. Instances of a specific visual element are distinguished by numbers, with the longest instance represented by index 1. Continuously existing identical elements are then merged. This path is the single-person gaze trajectory feature, and its length is equal to the length of the string. The order priority value of each visual element in the gaze trajectory feature of each normal person is calculated. Based on the order priority values ​​of visual elements from multiple normal people, the total order priority value of each visual element is calculated. The visual elements are then sorted in descending order according to their total order priority values ​​to generate the normal person gaze trajectory trend feature. Based on the similarity of the virtual character facial expressions for each emotion type, adjust the expression intensity parameters of the virtual character facial expressions for that emotion type. Use the adjusted virtual character facial expressions as visual stimuli for the next round of training. Repeat the above steps until the training completion conditions are met. The method of adjusting the expression intensity parameters of virtual character expression animations for each emotion type based on their similarity, and using the adjusted virtual character expression animations as visual stimuli for the next round of training, includes: determining the expression intensity parameters based on a preset conversion relationship between similarity and expression intensity parameters, wherein the expression intensity parameters include an expression density coefficient and an expression duration; obtaining facial control point coordinates based on the expression density coefficient, and using the Dirichlet free deformation algorithm to obtain new coordinates of the virtual character's facial deformation points; and synthesizing the adjusted virtual character expression animations based on the new coordinates of the virtual character's facial deformation points and the expression duration.

2. The adaptive training method for emotion recognition ability based on eye-tracking feedback according to claim 1, characterized in that, When erroneous judgment results exist, after obtaining the judgment results of the training object for each virtual character facial animation, the method further includes: The training object is assisted by using the virtual character facial expression animation corresponding to the error discrimination result, so that the eye movement trajectory data of the training object when observing the virtual character facial expression animation reaches the threshold condition.

3. The adaptive training method for emotion recognition ability based on eye-tracking feedback according to claim 1, characterized in that, After obtaining the similarity scores for virtual character facial expressions of each emotion type, the method further includes: Determine whether the similarity of the virtual character's facial expression animation for each emotion type is greater than or equal to a predetermined value; If so, the virtual character facial animation with a similarity greater than or equal to a predetermined value will be removed from the visual stimulus material in the next round of training.

4. The adaptive training method for emotion recognition ability based on eye-tracking feedback according to claim 2, characterized in that, The method of using virtual character facial animations corresponding to incorrect discrimination results to assist in training a training subject, so that the eye movement trajectory data of the training subject when observing the virtual character facial animations reaches a threshold condition, includes: The virtual character's facial expression animation corresponding to the incorrect judgment result is re-exposed to the training subject. At the same time, gaze navigation auxiliary boxes are added to the virtual character's facial expression animation in a predetermined order. The gaze navigation auxiliary boxes are used to indicate the gaze area. The eye-tracking data of the trainee when observing the facial expressions of virtual characters of different emotional types is monitored in real time. Based on the eye-tracking data, it is determined whether the order in which the trainee observes the gaze area is consistent with the order in which the gaze navigation aid box is added. If yes, end the assisted training; if not, repeat the above steps.

5. An adaptive training device for emotion recognition ability based on eye-tracking feedback, characterized in that, The device includes: An animation acquisition unit is used to acquire visual stimulus materials for this round of training, wherein the visual stimulus materials include at least two different types of virtual character facial expression animations. The user interaction unit is used to display virtual character facial animations of different emotion types to the training subject in a random order, and when the virtual character facial animation of each emotion type is displayed, it sends an instruction to the training subject to identify the emotion type corresponding to each virtual character facial animation. An eye-tracking unit is used to acquire in real time the eye-tracking trajectory data of the training subject when observing virtual character facial expressions of different emotion types, and to acquire the discrimination results of the training subject for each type of virtual character facial expression, the discrimination results including correct discrimination results and / or incorrect discrimination results; The data processing unit is used to obtain the similarity of virtual character facial animations for each emotion type based on the eye-tracking trajectory data and discrimination results corresponding to the virtual character facial animations for each emotion type. When there is an incorrect discrimination result, the method for obtaining the similarity of virtual character facial animations for each emotion type based on the eye-tracking trajectory data and discrimination results includes: setting the similarity of the virtual character facial animations corresponding to the incorrect discrimination result to zero, where zero similarity indicates that the facial expression intensity parameter of the virtual character facial animation remains unchanged. When there is a correct discrimination result, the method for obtaining the similarity of virtual character facial animations for each emotion type based on the eye-tracking trajectory data and discrimination results includes: extracting single-person gaze trajectory features and single-person gaze trajectory path length from the eye-tracking trajectory data of the virtual character facial animations corresponding to the correct discrimination result; obtaining the edit distance based on the single-person gaze trajectory features and pre-acquired normal person gaze trajectory trend features; and calculating the similarity of virtual character facial animations for each emotion type based on the edit distance and the single-person gaze trajectory path length. The eye-tracking trajectory data includes gaze projection positions. Virtual character facial animations are segmented into visual element regions. The training subject's gaze duration on the displayed content and each visual element are used as input. The single-person gaze trajectory is then represented by the visual elements and the gaze duration on those elements. Visual elements may exist continuously or discontinuously in the single-person gaze trajectory. Each discontinuous occurrence of a visual element along a path is called an instance of that element. Instances of a specific visual element are distinguished by numbers, with the longest instance represented by index 1. Continuously existing identical elements are then merged. This path is the single-person gaze trajectory feature, and its length is equal to the length of the string. The order priority value of each visual element in the gaze trajectory feature of each normal person is calculated. Based on the order priority values ​​of visual elements from multiple normal people, the total order priority value of each visual element is calculated. The visual elements are then sorted in descending order according to their total order priority values ​​to generate the normal person gaze trajectory trend feature. The parameter adjustment unit is used to adjust the expression intensity parameters of the virtual character expression animations for each emotion type based on the similarity to the corresponding virtual character expression animations. The adjusted virtual character expression animations are then used as visual stimuli for the next round of training. This process is repeated until the training completion conditions are met. The method of adjusting the expression intensity parameters of virtual character expression animations for each emotion type based on their similarity, and using the adjusted virtual character expression animations as visual stimuli for the next round of training, includes: determining the expression intensity parameters based on a preset conversion relationship between similarity and expression intensity parameters, wherein the expression intensity parameters include an expression density coefficient and an expression duration; obtaining facial control point coordinates based on the expression density coefficient, and using the Dirichlet free deformation algorithm to obtain new coordinates of the virtual character's facial deformation points; and synthesizing the adjusted virtual character expression animations based on the new coordinates of the virtual character's facial deformation points and the expression duration.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores an adaptive training program for emotion recognition ability based on eye-tracking feedback, which, when executed by a processor, implements the adaptive training method for emotion recognition ability based on eye-tracking feedback as described in any one of claims 1 to 4.

7. A computer device, characterized in that, The computer device includes a computer-readable storage medium, a processor, and an eye-tracking feedback-based adaptive training program for emotion recognition ability stored in the computer-readable storage medium, wherein the eye-tracking feedback-based adaptive training program for emotion recognition ability, when executed by the processor, implements the eye-tracking feedback-based adaptive training method for emotion recognition ability according to any one of claims 1 to 4.

Images