Animation character interactive control device based on eye tracking

Abstract

The present invention relates to the field of human-computer interaction technology, and discloses an interactive control device for an animated character based on eye tracking, comprising a mask module for collecting user facial information, a stable wearing component, and a collection component for collecting the user's eye state; the mask module comprises a cover body, and the upper and lower parts of the cover body are respectively installed with multiple groups of camera group one and camera group two, and the cover body is installed with a front camera on the opposite side of the camera group one, and the collection component comprises a frame, and the frame is fixedly installed on the surface of the cover body, and multiple eye movement collection cameras are evenly arranged in a ring on the outside of the frame, and a mounting seat is fixedly installed on the outside of the frame. Through the collaborative collection of facial expressions, eye movement data, voice information and the front visual environment by multiple groups of cameras and eye movement collection components, the accuracy and integrity of data collection are significantly improved, high-quality input is provided for the intelligent interaction of animated characters, and the naturalness and immersion of interaction are enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of human-computer interaction, and in particular to an animation character interaction control device based on eye tracking. Background Art

[0002] With the rapid development of virtual reality, augmented reality and artificial intelligence technologies, the way of human-computer interaction is changing from traditional mouse and keyboard operation to a more natural and efficient multimodal interaction mode. As an important branch of it, eye tracking-based interaction technology has attracted widespread attention due to its non-contact, intuitive and high real-time performance, and has gradually become an important research direction of modern human-computer interaction systems. Eye tracking technology can perceive the user's attention distribution, emotional state and behavioral intention in real time by collecting and analyzing the user's eye movement data (such as gaze point, pupil diameter, gaze time, etc.). This technology has been widely used in medical diagnosis, market analysis, education and training, and immersive interaction systems in virtual reality. However, traditional eye tracking-based interaction devices are often limited to simple gaze point selection or target tracking, lacking in-depth integration with multimodal inputs such as user emotions, voice and actions, making it difficult to achieve more complex and natural interactions.

[0003] As an emerging entertainment and educational tool, the demand for the virtual interactive system of anime characters is gradually developing towards high immersion and intelligence. In the existing technology, the interaction mode of virtual characters mainly relies on single or low-correlation interaction modes such as mouse, gestures and voice. This interaction method has great limitations: first, the single-modal interaction lacks a multi-dimensional understanding of user behavior, resulting in low interaction accuracy and naturalness; second, the traditional system responds slowly to the user's gaze intention, and it is difficult to dynamically adjust the virtual character's actions and emotional feedback; in addition, the existing system lacks accurate understanding and personalized response to the user's emotional state, and the interaction process appears mechanical and monotonous, and cannot provide a high-quality user experience. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides an interactive control device for anime characters based on eye tracking, which solves the problem that the single-modal interaction method in the existing virtual character interaction technology is difficult to achieve accurate understanding and real-time response to the user's multi-dimensional behaviors and emotions, resulting in low interaction accuracy, insufficient emotional feedback and poor user experience.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: an interactive control device for anime characters based on eye tracking, comprising a mask module for collecting user facial information, a stable wearing component, and a collection component for collecting the user's eye status;

[0006] The mask module includes a mask body, and the upper and lower parts of the mask body are respectively installed with multiple groups of camera group one and camera group two, and a front-end camera is installed on the side of the mask body opposite to the camera group one.

[0007] Preferably, the acquisition component includes a frame, which is fixedly mounted on the surface of the cover body, and a plurality of eye movement acquisition cameras are evenly arranged in a ring shape on the outside of the frame, a mounting seat is fixedly mounted on the outside of the frame, a side camera is mounted on the inside of the mounting seat, and a gyroscope is mounted in the middle of the frame.

[0008] Preferably, the mask module also includes a microphone, which is installed on the lower half of the mask body and is used to collect user voice information.

[0009] Preferably, the wearing assembly includes a fixing seat, which is fixedly connected to the surface of the cover body, a wearing hoop is rotatably connected to the middle of the fixing seat, an adjusting bolt is threadedly connected to the outer side of the wearing hoop, one end of the adjusting bolt is rotatably connected, and both ends are fixedly connected to the inner side of the wearing hoop.

[0010] Preferably, the bottom of the wearing hoop 1 is fixedly connected to the wearing hoop 2, the middle part of the wearing hoop 2 is threadedly connected to the adjusting bolt 2, one end of the adjusting bolt 2 is rotatably connected to the flexible strap 2, and both ends of the flexible strap 2 are fixedly connected to the inner side of the wearing hoop 2.

[0011] Preferably, a notch is provided in the middle of the frame, and a nose pad is fixedly connected to the inner side of the frame.

[0012] Animation character interactive control system based on eye tracking, including:

[0013] A data collection module, used to collect the user's eye movement data, facial expression data, head posture data and voice data;

[0014] A data processing module, used for preprocessing, feature extraction and time alignment of the eye movement data, facial expression data, head posture data and voice data;

[0015] A multimodal fusion module, used to fuse the multimodal features extracted by the data processing module into a unified feature representation;

[0016] A user intention recognition module, used to recognize the user's target object and interaction intention based on the unified feature representation;

[0017] The cartoon character control module is used to generate the actions, expressions and voice feedback of the cartoon character according to the user's intention, so as to realize the interactive control between the user and the cartoon character.

[0018] Preferably, the multimodal fusion module includes:

[0019] Self-attention mechanism for feature fusion based on the contextual relationship of multimodal data;

[0020] A multi-layer feature encoder is used to encode eye movements, facial expressions, head postures, and voice features into a unified feature representation and output the user's comprehensive behavioral intentions.

[0021] Preferably, the animation character control module includes:

[0022] An action generation unit, used to generate the action of the animation character according to the user's gaze target and interaction intention;

[0023] An expression feedback unit, used to adjust the expression of the cartoon character according to the user's emotional state;

[0024] A voice feedback unit, used to generate a matching voice response according to the user's emotions and voice content;

[0025] The motion smoothing unit is used to smooth the motion transition of animation characters through motion optimization algorithms to improve the naturalness of interaction.

[0026] Preferably, the user intention recognition module adopts a multi-layer perceptron or a deep learning model to realize target object recognition and emotion classification according to multimodal fusion features, and drives the response behavior of the animation character through the classification results.

[0027] Working principle: Wear the acquisition component on the bridge of the nose through the notch and the nose pad, and then adjust the bolts one and two to deform the flexible strap two, so as to fix the device. Use multiple eye movement acquisition cameras on the surface of the frame to identify the direction of the user's eyeballs to complete the basic interaction with the anime character. At this time, use camera group one and camera group two on the surface of the mask to collect data from the upper and lower parts of the user's face, including eyebrows, corners of the mouth, and lip movements. Use the side camera to collect the side features of the user's face. In this way, through multi-point positioning, accurate recognition of facial expressions is achieved to further improve the accuracy of interactive control and improve the user experience. Secondly, during use, the microphone is used to collect the user's voice signal, and the gyroscope is used to detect the user's head posture to enrich the interactive control content.

[0028] The present invention provides an animation character interactive control device based on eye tracking. It has the following beneficial effects:

[0029] 1. The present invention uses multiple groups of cameras and eye movement acquisition components to collaboratively collect facial expressions, eye movement data, voice information and the front visual environment, significantly improving the accuracy and integrity of data collection, providing high-quality input for the intelligent interaction of animated characters, and enhancing the naturalness and immersion of the interaction.

[0030] 2. The present invention uses multimodal fusion algorithms and artificial intelligence technology to enable virtual animation characters to dynamically generate actions, expressions and voice feedback according to user intentions and emotions. The character's response is more intelligent and personalized, and can adapt to the needs of different users. It is widely applicable to various scenarios such as virtual reality entertainment, education and training, and psychological counseling. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A perspective view of the present invention;

[0032] Figure 2 It is a structural schematic diagram of the mask module in the present invention;

[0033] Figure 3 It is a schematic diagram of the structure explosion of the present invention;

[0034] Figure 4 It is a schematic diagram of the acquisition component in the present invention.

[0035] Among them, 1. mask module; 11. mask body; 12. camera group 1; 13. camera group 2; 14. front-end camera; 15. microphone; 2. wearing component; 21. fixing seat; 22. wearing hoop 1; 23. flexible strap 2; 24. adjusting bolt 1; 25. wearing hoop 2; 26. flexible strap 2; 27. adjusting bolt 2; 3. acquisition component; 31. frame; 32. eye movement acquisition camera; 33. nose pad; 34. notch; 35. mounting seat; 36. side camera; 37. gyroscope. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the specification of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0037] Example:

[0038] Please refer to the attached Figure 1 -Attached Figure 2The embodiment of the present invention provides an interactive control device for an animated character based on eye tracking, comprising a mask module 1 for collecting user facial information, the mask module 1 comprising a mask body 11, the upper and lower parts of the mask body 11 are respectively equipped with a plurality of camera groups 1 12 and camera groups 2 13, the camera group 1 12 is used to collect image data of the upper half of the user's face, including changes in eyebrows and cheekbones; the camera group 2 13 is used to collect image data of the lower half of the user's face, including changes in the corners of the mouth, changes in lips, etc., the camera group 12 and the camera group 2 13 cooperate with each other to realize the collection of the user's facial expression. A front-end camera 14 is installed on the side opposite to the mask body 11 and the camera group 1 12, and the front-end camera 14 is used to collect visual information in front of the user; to determine the interaction between eyeball gaze and external vision. The mask module 1 also includes a microphone 15, which is installed on the lower part of the mask body 11 and is used to collect user voice information, and the voice information collected by the microphone 15 is used to realize the dialogue interaction with the animated character.

[0039] Please refer to the attached Figure 2 -Attached Figure 3 , and also includes a stable wearing component 2; the control device can be worn on the user's head through the wearing component 2, and the wearing component 2 includes a fixing seat 21, the fixing seat 21 is fixedly connected to the surface of the cover body 11, and the middle part of the fixing seat 21 is rotatably connected to a wearing hoop 22, and the outer side of the wearing hoop 22 is threadedly connected to an adjusting bolt 24, and one end of the adjusting bolt 24 is rotatably connected to 23, and both ends of 23 are fixedly connected to the inner side of the wearing hoop 22, and the deformation amplitude of 23 can be adjusted by rotating the adjusting bolt 24, so as to adjust the size of the internal space of the wearing hoop 22 to adjust the size of the internal space of the wearing hoop 22 according to the wearing Adjustment is needed; a wearing hoop 25 is fixedly connected to the bottom of the wearing hoop 22, an adjusting bolt 27 is threadedly connected to the middle of the wearing hoop 25, one end of the adjusting bolt 27 is rotatably connected to a flexible strap 26, both ends of the flexible strap 26 are fixedly connected to the inner side of the wearing hoop 25, and the adjusting bolt 27 is rotated in the same way to adjust the deformation amplitude of the flexible strap 26, so as to complete the overall wearing, 23 and the flexible strap 26 are made of hard rubber material, and are deformed after being squeezed by the adjusting bolt 27 and the adjusting bolt 1 24 to provide clamping force.

[0040] Please refer to the attached Figure 2 -Attached Figure 4, and also includes a collection component 3 for collecting the user's eye status; the collection component 3 includes a frame 31, which is fixedly installed on the surface of the mask body 11, and a notch 34 is opened in the middle of the frame 31. A nose pad 33 is fixedly connected to the inner side of the frame 31. The notch 34 is placed on the bridge of the user's nose, and then the collection component 3 is lifted as a whole by the nose pad 33. The wearing and positioning of the device can be completed in conjunction with the wearing component 2. A plurality of eye movement collection cameras 32 are evenly arranged in a ring shape on the outer side of the frame 31. The eye movement collection cameras 32 are all facing the user's eyeballs to collect the user's eye movement data. A mounting seat 35 is fixedly installed on the outer side of the frame 31, and a side camera 36 is installed on the inner side of the mounting seat 35. The side camera 36 is used to collect the user's side face information, and the collection component 3 and the mask module 1 are used to collect facial expressions.

[0041] As part of this application, the present invention also provides an animation character interactive control system based on eye tracking, comprising:

[0042] A data collection module, used to collect the user's eye movement data, facial expression data, head posture data and voice data;

[0043] The eye movement acquisition camera 32 is used to collect the user's eye movement data and collect the gaze point (x g ,y g ), pupil diameter change (Δd p ), and eye-saccade data (Δx g ,Δy g ). The key formula is as follows:

[0044] Δx g =x g,t -x g,t-1 , Δy g =y g,t -y g,t-1

[0045] Δd p =d p,t -d p,t-1

[0046] Where t represents the time frame

[0047] Camera group one 12 and camera group two 13 are used to collect the user's facial expression.

[0048] Based on high-definition camera and OpenCV library, real-time capture of facial key points K = {k1, k2, ..., k n}:

[0049]

[0050] Calculate facial expression features through key point changes, such as the angle of the mouth corner upward θ mouth :

[0051]

[0052] The gyroscope (37) is used to collect the user's head posture.

[0053] Get the head rotation angle (Pitch, Yaw, Roll):

[0054] H=[φ,θ,ψ]

[0055] Correct the gaze point by using the coordinate transformation matrix R:

[0056]

[0057] The user's voice data is collected using a microphone 15. The voice signal s(t) is collected and the Mel Frequency Cepstral Coefficient (MFCC) feature F is extracted. mfcc :

[0058] F mfcc =MFCC(s(t))

[0059] A data processing module, used for preprocessing, feature extraction and time alignment of the eye movement data, facial expression data, head posture data and voice data;

[0060] Eye movement data denoising: Use Kalman filtering to correct gaze jitter:

[0061]

[0062] Among them, K t is the Kalman gain, z t is the observed value.

[0063] Multimodal time alignment: Align multimodal data to the unified time axis t through linear interpolation:

[0064]

[0065] A multimodal fusion module, used to fuse the multimodal features extracted by the data processing module into a unified feature representation;

[0066] The multimodal fusion module comprises:

[0067] Self-attention mechanism for feature fusion based on the contextual relationship of multimodal data;

[0068] A multi-layer feature encoder is used to encode eye movements, facial expressions, head postures, and voice features into a unified feature representation and output the user's comprehensive behavioral intentions.

[0069] Use the Transformer model to fuse multimodal data:

[0070] Input Embedding Layer

[0071] Multimodal feature embedding:

[0072] X=[X eye ,X face ,X head ,X audio ]

[0073] Self-Attention Mechanism:

[0074]

[0075] Among them, Q, K, V are query, key and value vectors.

[0076] Fusion output:

[0077] Y = Transformer(X)

[0078] A user intention recognition module is used to recognize the user's target object and interaction intention based on the unified feature representation; the user intention recognition module adopts a multi-layer perceptron or a deep learning model to realize target object recognition and emotion classification according to multimodal fusion features, and drives the response behavior of the animation character through the classification results.

[0079] User status classification

[0080] Output the user's current state S through the multimodal fusion model:

[0081] S={T target ,C emotion}

[0082] Among them, T target To look at the target, C emotion For emotional state.

[0083] Action generation strategy:

[0084] Generate animation character actions based on reinforcement learning:

[0085] State-Action Model:

[0086] State S corresponds to character action A:

[0087] A=π(S)

[0088] Among them, π is the policy function.

[0089] Action optimization:

[0090] Optimizing action selection using a Deep Q Network (DQN):

[0091]

[0092] The cartoon character control module is used to generate the actions, expressions and voice feedback of the cartoon character according to the user's intention, so as to realize the interactive control between the user and the cartoon character.

[0093] The animation character control module includes:

[0094] The action generation unit is used to generate the actions of the animation character according to the user's gaze target and interaction intention; natural actions are generated through GAN:

[0095] Generator G: G(z) = Action(z)

[0096] Where z is random noise.

[0097] Discriminator D:

[0098] An expression feedback unit, used to adjust the expression of the cartoon character according to the user's emotional state;

[0099] Character expression adjustment based on user emotional state C emotion :

[0100]

[0101] in, is the expression mapping function.

[0102] A voice feedback unit, used to generate a matching voice response according to the user's emotions and voice content;

[0103] The motion smoothing unit is used to smooth the motion transition of animation characters through motion optimization algorithms to improve the naturalness of interaction.

[0104] Use Bezier curves to smoothly transition actions:

[0105] P(t)=(1-t) 2 P0+2(1-t)tP1+t 2 P2,t∈[0,1]

[0106] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. Animation character interactive control device based on eye tracking, characterized in that: It comprises a mask module (1) for collecting facial information of a user, a stable wearing component (2) and a collecting component (3) for collecting the eye status of the user; The mask module (1) comprises a mask body (11), wherein the upper and lower parts of the mask body (11) are respectively installed with a plurality of camera groups 1 (12) and camera groups 2 (13), and a front-end camera (14) is installed on the side of the mask body (11) opposite to the camera group 1 (12).

2. The animation character interactive control device based on eye tracking according to claim 1, characterized in that: The acquisition component (3) comprises a lens frame (31), the lens frame (31) is fixedly mounted on the surface of the cover body (11), a plurality of eye movement acquisition cameras (32) are evenly arranged in a ring shape on the outside of the lens frame (31), a mounting seat (35) is fixedly mounted on the outside of the lens frame (31), a side camera (36) is mounted on the inside of the mounting seat (35), and a gyroscope (37) is mounted in the middle of the lens frame (31).

3. The animation character interactive control device based on eye tracking according to claim 1, characterized in that: The mask module (1) also includes a microphone (15), which is installed on the lower half of the mask body (11) and is used to collect user voice information.

4. The animation character interactive control device based on eye tracking according to claim 1, characterized in that: The wearing assembly (2) comprises a fixing seat (21), the fixing seat (21) is fixedly connected to the surface of the cover body (11), a wearing hoop (22) is rotatably connected to the middle part of the fixing seat (21), an adjusting bolt (24) is threadedly connected to the outer side of the wearing hoop (22), one end of the adjusting bolt (24) is rotatably connected to (23), and both ends of the (23) are fixedly connected to the inner side of the wearing hoop (22).

5. The animation character interactive control device based on eye tracking according to claim 4, characterized in that: The bottom of the wearing hoop 1 (22) is fixedly connected to the wearing hoop 2 (25), the middle of the wearing hoop 2 (25) is threadedly connected to the adjusting bolt 2 (27), one end of the adjusting bolt 2 (27) is rotatably connected to the flexible binding belt 2 (26), and both ends of the flexible binding belt 2 (26) are fixedly connected to the inner side of the wearing hoop 2 (25).

6. The animation character interactive control device based on eye tracking according to claim 2, characterized in that: A notch (34) is provided in the middle of the frame (31), and a nose pad (33) is fixedly connected to the inner side of the frame (31).

7. Animation character interactive control system based on eye tracking, characterized in that: The animation character interactive control device based on eye tracking according to any one of claims 1 to 6 comprises: A data collection module, used to collect the user's eye movement data, facial expression data, head posture data and voice data; A data processing module, used for preprocessing, feature extraction and time alignment of the eye movement data, facial expression data, head posture data and voice data; A multimodal fusion module, used to fuse the multimodal features extracted by the data processing module into a unified feature representation; A user intention recognition module, used to recognize the user's target object and interaction intention based on the unified feature representation; The cartoon character control module is used to generate the actions, expressions and voice feedback of the cartoon character according to the user's intention, so as to realize the interactive control between the user and the cartoon character.

8. The cartoon character interactive control system based on eye tracking according to claim 7, characterized in that: The multimodal fusion module comprises: Self-attention mechanism for feature fusion based on the contextual relationship of multimodal data; A multi-layer feature encoder is used to encode eye movements, facial expressions, head postures, and voice features into a unified feature representation and output the user's comprehensive behavioral intentions.

9. The cartoon character interactive control system based on eye tracking according to claim 7, characterized in that: The animation character control module includes: An action generation unit, used to generate the action of the animation character according to the user's gaze target and interaction intention; An expression feedback unit, used to adjust the expression of the cartoon character according to the user's emotional state; A voice feedback unit, used to generate a matching voice response according to the user's emotions and voice content; The motion smoothing unit is used to smooth the motion transition of animation characters through motion optimization algorithms to improve the naturalness of interaction.

10. The cartoon character interactive control system based on eye tracking according to claim 7, characterized in that: The user intention recognition module adopts a multi-layer perceptron or a deep learning model to realize target object recognition and emotion classification according to multimodal fusion features, and drives the response behavior of the animation character through the classification results.

Images