Apparatus and method for generating avatar lip-sync animations based on multimodal biosignals.

Abstract

To provide a device for receiving a biological signal including brain waves when a user imagines speaking as an input and outputting an avatar lip sync animation, and a method for generating the same.SOLUTION: The avatar lip-sync animation generation apparatus includes a multimodal data collection unit, a preprocessing unit for preprocessing multimodal data, a feature extraction unit for extracting a feature vector including a biological signal feature and a facial feature of a user from the preprocessed multimodal data, and an avatar generation unit for generating an avatar that replaces the appearance of the user. The system includes a lip sync reconstruction unit for predicting the shape of the mouth and the movement of the face when the user imagines speaking, and a lip sync animation implementation unit for implementing the lip sync animation of the avatar by applying the predicted shape of the mouth and movement of the face to the avatar generated by the lip sync reconstruction unit and the avatar generation unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an apparatus and a method for generating avatar lip-sync animation based on multimodal biological signals, and more particularly, to an apparatus and a method for generating avatar lip-sync animation based on multimodal biological signals, which can generate an avatar corresponding to a user's facial image and implement avatar lip-sync animation based on multimodal biological signals generated when the user imagines speaking by using a pre-established lip-sync reconstruction model. [Background Art]

[0002] Brain-Computer Interface (BCI) is a technology that directly connects neurological signals of the brain to a computer system to enable communication and control. For this purpose, various biological signal measurement technologies are used to identify brain activities such as the user's thinking, concentration and imagination, and convert them into digital commands. Brain-computer interfaces have become a very important tool especially for people with limited motor ability, enabling them to perform activities such as using computers, moving robotic arms, and even controlling wheelchairs. This technology provides new interaction methods in various fields, ranging from virtual reality, video games, neuroscience research, to even the creation of art and music. Recently, along with the development of algorithms for analyzing brain signals, brain-computer interface technology has become more sophisticated, which has the potential to innovatively change the future interaction between humans and machines. On the other hand, recently, research has been conducted on a method of implementing a speaking person's face through lip-sync between a face synthesized by computer graphics and human speech. As a prior art, Korean Patent Publication No. 10-2006-0031449 (published on April 12, 2006) proposes "Apparatus and method for automatic speech-based lip-sync animation, and recording medium". Existing lip-sync animation technologies, including the aforementioned conventional technologies, have primarily relied on methods that reconstruct mouth shapes based on input audio data. However, existing methods for generating speaking faces through lip-sync animation require the use of recorded audio data of the user speaking directly. Therefore, existing systems have limitations, such as being unable to utilize audio data in patients who have difficulty speaking or in quiet situations, and being unable to express subtle emotions such as the user's facial expressions and nuances. On the other hand, another prior art example is the "Brain-Computer Interface System and User Dialogue Intent Recognition Method Using the Same," which was proposed in the Korean Patent Publication No. 10-2020-0052807 (published May 15, 2020). However, communication systems based on brain-computer interfaces, including the aforementioned other conventional technologies, have primarily relied on passively reading and transmitting user intentions, such as simple class classification and text generation, through brainwaves during speech. Communication systems that utilize brain signals have been developed extensively in the field of brain-computer interfaces in recent years, and are being integrated with the field of artificial intelligence, leading to the development of diverse methodologies.

[0003] Among these, speech-imagination-based user communication technology has the advantage of being able to communicate the user's intentions without the user's direct utterance. However, in brain-computer interfaces, user communication technologies still face limitations, such as reduced real-time decoding performance, low recognition rates, and difficulty in achieving understandable speech synthesis. Although various methods have been proposed to improve performance, the technology of measuring invasive electroencephalograms and transmitting the results to doctors has limitations, such as high costs and difficulty in practical use in real life, resulting in a low utilization rate as a method for recommending surgery to the general public. While technologies have been developed that synthesize speech using brainwaves during speech, these have limitations, such as being restricted to patients who have difficulty speaking or to quiet environments where speaking is difficult. Therefore, there is a need for the development of new technology that can take biosignals from the time of speech imagination, rather than brainwaves during speech and recorded speech data as input, and output avatar lip-sin animation. [Overview of the project] [Problems that the invention aims to solve]

[0004] This invention overcomes the limitations of the prior art described above and was devised in response to the demand for new technological development. The object of this invention is to provide a multimodal biosignal-based avatar lip-sync animation generation device and method that can receive biosignals, including brainwaves during the user's speech imagination, as input and output avatar lip-sync animation. [Means for solving the problem]

[0005] To achieve the above objectives, the avatar lip-sync animation generation apparatus based on a multimodal biosignal according to the present invention includes: a multimodal data collection unit that collects biosignal data including brainwaves when the user imagines speaking, and multimodal data including video data; a preprocessing unit that preprocesses the multimodal data; a feature extraction unit that extracts feature vectors including the user's biosignal features and facial features from the preprocessed multimodal data; an avatar generation unit that generates an avatar that represents the user; a lip-sync reconstruction unit that inputs the extracted feature vectors into a pre-provided lip-sync reconstruction model to predict the mouth pattern and facial movements when the user imagines speaking; and a lip-sync animation implementation unit that applies the lip-sync reconstruction unit to the avatar generated by the avatar generation unit, and the mouth shape and facial movements predicted by the lip-sync reconstruction unit to the avatar generated by the avatar generation unit, to realize the avatar's lip-sync animation. Here, the avatar generation unit generates a 2D or 3D avatar from the user's video data using computer vision technology, maps the user's facial features extracted by the feature extraction unit to the generated avatar to specify a facial landmark, and the lip-sync animation implementation unit implements an avatar lip-sync animation by applying the mouth shape and facial movements predicted by the lip-sync reconstruction unit based on the coordinate values ​​of the facial landmark to the avatar generated by the avatar generation unit. Furthermore, the multimodal biosignal-based avatar lip-sync animation generation device according to the present invention further includes a feature fusion unit that fuses the feature vectors extracted by the feature extraction unit and converts them into embedded fusion vectors, and the lip-sync reconstruction unit is characterized in that it inputs the embedded fusion vectors into a pre-provided lip-sync reconstruction model to predict the mouth patterns and facial movements when the user imagines speaking. Here, the multimodal data acquisition unit is characterized by including a presentation text transmission display module that transmits a presentation text to the user for speech imagination; a biosignal acquisition module that measures biosignals including the user's brain waves and collects biosignal data; an image acquisition module that captures an image of the user's face and collects image data; and a data storage module that stores the biosignal data and image data of the user as they imagine speech in response to the transmitted presentation text, along with a trigger value recorded over time. Herein, the biosignal acquisition module further includes electromyography in the biosignals of the user to be measured, and the lip-sync reconstruction unit predicts mouth patterns and facial movements by inferring the movement trajectories of the articulatory organs corresponding to speech imagination based on the electromyography. Here, the feature fusion unit is characterized by applying a weighted value according to a predetermined criterion to the feature vector extracted from the feature extraction unit, and fusing the feature vector to which the weighted value has been applied to convert it into an embedded fusion vector. Here, the lip-sync reconstruction model is characterized by comprising either a first prediction model that predicts the mouth shape and facial movements of the user when imagining speech from the extracted feature vectors, or a second prediction model that grasps and classifies the user's intention from the extracted feature vectors and predicts the mouth shape and facial movements of the user when imagining speech according to the classified intention. Furthermore, the avatar lip-sync animation generation method on a multimodal biosignal base according to the present invention includes a multimodal data collection step of collecting biosignal data including brainwave data and video data when the user imagines speaking; a preprocessing step of preprocessing the multimodal data; a feature extraction step of extracting feature vectors including the user's biosignal features and facial features from the preprocessed multimodal data; an avatar generation step of generating an avatar that replaces the user's appearance based on the facial features among the extracted feature vectors; a lip-sync reconstruction step of inputting the extracted feature vectors into a pre-prepared lip-sync reconstruction model to predict the shape of the mouth and facial movements when the user imagines speaking; and a lip-sync animation realization step of applying the shape of the mouth and facial movements predicted in the lip-sync reconstruction step to the avatar generated in the avatar generation step to realize the avatar's lip-sync animation.

[0006] Furthermore, the multimodal biosignal-based avatar lip-sync animation generation method according to the present invention further includes a feature fusion step in which the feature vectors extracted in the feature extraction step are fused and converted into an embedded fusion vector, and the lip-sync reconstruction step is characterized in that the embedded fusion vector is input to a pre-established lip-sync reconstruction model to predict the mouth shape and facial movements when the user imagines speaking. [Effects of the Invention]

[0007] With the configuration described above, the multimodal biosignal-based avatar lip-sync animation generation apparatus and method according to the present invention has the advantage of being able to grasp the user's intention from the biosignals when the user imagines speaking and provide it as an avatar lip-sync animation. Furthermore, the avatar lip-sync animation generation apparatus and method based on multimodal biosignals according to the present invention can be developed into a system without constraints on various applications by using biosignals including non-invasive speech-imaginary electroencephalograms. It has the advantage of enabling lip-sync animation that can visually convey the user's intentions without the user having to speak directly by extracting speech and facial reconstruction information embedded in the biosignals, expressing and conveying the user's emotions and intentions through facial expressions, and enabling future-oriented technology by utilizing avatars to enable factual and dynamic communication in the next-generation digital world. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a configuration diagram of a multimodal biosignal-based avatar lip-sync animation generation device according to one embodiment of the present invention. [Figure 2] Figure 2 is a diagram showing the configuration of a multimodal data acquisition unit according to one embodiment of the present invention. [Figure 3] Figure 3 is an example diagram of an avatar generation unit according to one embodiment of the present invention. [Figure 4] Figure 4 is a data collection and processing flowchart for lip-sync reconstruction according to one embodiment of the present invention. [Figure 5] Figure 5 is a data processing flowchart for realizing avatar lip-sync animation according to one embodiment of the present invention. [Figure 6] Figure 6 is a flowchart of a method for generating multimodal biosignal-based avatar lip-sync animation according to one embodiment of the present invention. [Modes for carrying out the invention]

[0009] The multimodal biosignal-based avatar lip-sync animation generation apparatus and method according to the present invention will be described in more detail below with reference to the embodiments shown in the drawings. Figure 1 is a configuration diagram of a multimodal biosignal-based avatar lip-sync animation generation device according to one embodiment of the present invention, Figure 2 is a configuration diagram of a multimodal data acquisition unit according to one embodiment of the present invention, Figure 3 is an example diagram of an avatar generation unit according to one embodiment of the present invention, Figure 4 is a data acquisition and processing flowchart for lip-sync reconstruction according to one embodiment of the present invention, and Figure 5 is a data processing flowchart for realizing avatar lip-sync animation according to one embodiment of the present invention. Referring to Figure 1, a multimodal biosignal-based avatar lip-sync animation generation apparatus according to one embodiment of the present invention includes a multimodal data acquisition unit 10, a preprocessing unit 20, a feature extraction unit 30, an avatar generation unit 40, a feature fusion unit 50, a lip-sync reconstruction unit 60, and a lip-sync animation implementation unit 70. The multimodal data acquisition unit 10 is configured to collect multimodal data, including electroencephalograms (EEGs) and electromyograms (EMGs) of the user as they imagine speaking, as well as video data such as images of the user's face. In one embodiment of the present invention, the multimodal data acquisition unit 10 may include a text presentation display module 11, a biosignal acquisition module 12, an image acquisition module 13, and a data storage module 14, as shown in Figure 2. The aforementioned presentation text display module 31 is configured to convey presentation text for speech imagination to the user via a screen. In one embodiment of the present invention, the presentation text transmission display module 31 may be configured to transmit to the user a guideline video that includes a reference mouth shape for the presentation text. The biosignal acquisition module 12 is configured to collect biosignal data by measuring biosignals such as electroencephalograms and electromyograms of the user. Electroencephalography (EEG), as a biological signal, refers to the electrical activity of the brain measured via electrodes attached to the scalp. This signal is used as an important tool for understanding various states and activities of the brain, and particularly has the advantage of enabling temporally precise tracking of functional changes in the brain. EEG is widely used not only in neuroscience research, clinical diagnosis, and neuropsychology, but also in the development of brain-computer interfaces and other fields. Particularly in the field of brain-computer interfaces, the intentions and thoughts of a user are recognized and converted into machine commands, which enables users to control external devices merely through imagination, thereby providing new methods of communication and interaction. Such EEG signal utilization technology can analyze brain activity patterns related to speech imagination and be applied to complex tasks such as real-time lip-syncing for digital avatars and animations. The biological signal acquisition module 12 for measuring the user's EEG may be configured as a wearable non-invasive device for measuring speech imagination-based EEG. In one embodiment, the module may be configured such that a user wears a cap-shaped device with a total of 128 electrodes attached to the outside of the scalp to measure real-time EEG data based on the user's biological signal. A gel-like conductive substance is applied to the scalp to match the electrodes so that EEG can be measured accurately.

[0010] At this time, the biological signal acquisition module 12 for measuring the user's EEG also measures speech attempt-based EEG together for analysis and comparison of speech imagination-based EEG, and measures EEG when only the mouth shape moves without producing sound. For the measured EEG, the required duration is recorded along with trigger values. The EEG data is stored in a designated database path of the data storage module 14 described later, and is preferably backed up together to an external storage device for data preservation. On the other hand, electromyography (EMG), as a biosignal, refers to electrical signals associated with muscle activity, measuring electrical changes due to muscle contraction and relaxation. This signal plays an important role in evaluating the functional state of muscles and the health of the nervous system, and is widely used in medical diagnosis, rehabilitation, sports science, and bioepidemiology research. In particular, EMG measurement technology is extremely useful in research on muscle control by precisely monitoring muscle activity. It can be used to integrate natural human movements into digital avatars and robotic technology, and facial movements and changes in expression can be reconstructed based on the movements of specific muscles of the user. It can also be used in a variety of fields, such as intuitive communication systems that synthesize speech based on the movements of articulatory muscles in speech situations. The electromyograms measured as described above are used in the lip-sync reconstruction unit 60, described later, to infer the movement trajectories of the articulatory organs corresponding to speech imagination based on the electromyograms, thereby enabling the prediction of mouth patterns and facial movements. Here, Articulatory Kinematic Trajectories (AKTs) are precisely recorded information about the movement of articulatory organs during speech production. This plays a crucial role in understanding how articulatory organs such as the lips, tongue, and jaw move to produce sound. By analyzing articulatory kinematics, it is widely used in the fields of linguistics, phonetics, medicine, and computer science. For example, AKTs can be used to analyze the articulation patterns of people with speech disorders and to develop treatment methods. It is also importantly applied in combination with artificial intelligence technology to real-time lip-sync animation, sophisticated avatar representation, and improve the accuracy of speech recognition systems. By providing highly detailed articulatory data, articulatory trajectory information plays an essential role in deepening our understanding of the human speech process and, based on this, developing more natural and realistic communication technologies. The aforementioned video acquisition module 13 is configured to capture video footage of the user's face and collect video data. Said video acquisition module 13 records face videos of a user when the user attempts to speak or imagines speaking by means of a camera attached to a display screen unit. Moving videos and still images are recorded in real time, and the same trigger value is recorded over time so as to match electroencephalogram data recording. The face video data is stored in a database path specified by the data storage module 14, and is preferably backed up together in an external storage device for data preservation. Said data storage module 14 is configured to store said biological signal data and said video data of a user who imagines speech in response to the transmitted prompt text together with trigger values recorded over time.

[0011] Said preprocessing unit 20 is configured to preprocess said multimodal data. As an example, preprocessing of electroencephalogram data recorded over continuous time among said multimodal data comprises dividing the data according to the co-recorded trigger values, separating only electroencephalograms generated during actual speech attempts and imagination for about 1.5 seconds, and obtaining and storing frequency value information of the corresponding portion in a two-dimensional form of time and channels. Said feature extraction unit 30 is configured to extract feature vectors including user biological signal features and face features from said preprocessed multimodal data. Herein, the biological signal features may include electroencephalogram features and electromyogram features, and the face features may include face shape features, mouth shape features and face movement features. For feature vector extraction in said feature extraction unit 30, common spatial pattern (CSP) analysis and linear discriminant analysis (LDA) methods are generally used. The methods are used to better distinguish motion-related characteristics from biological signals and better obtain information on face features and landmark features from face video data, and the extracted feature vectors are used for artificial intelligence neural network model learning, which advantageously contributes to achieving high prediction performance. For example, the feature extraction unit 30 can be configured to extract phoneme-specific and mouth-shape-specific motion feature vectors from the imagined speech brainwaves in the case of brainwave features, and to extract face feature vectors in the case of face features, and then to fuse the phoneme-specific and mouth-shape-specific motion feature vectors and face feature vectors together in the feature fusion unit 50 described later and convert them into an embedded fusion vector. As shown in Figure 3, the avatar generation unit 40 is configured to generate an avatar that represents the user's appearance. Here, "avatar" refers to a virtual character or image that functions as a representative or spokesperson for a user in the digital world. It primarily refers to digital agents that reflect a user's physical, emotional, or personal characteristics in online games, virtual reality, and social media. Avatars have diverse appearances and characteristics, allowing users to express their unique identity and personality as a means of self-expression and interaction in the digital environment. Recently, with the development of artificial intelligence technology, avatars are becoming more realistic by mimicking users' real-time movements, facial expressions, and even speech patterns. Such developments enrich the virtual and augmented reality experience, opening up new possibilities in online communication and interactive entertainment. In one embodiment of the present invention, the avatar generation unit 40 is configured to generate a 2D or 3D avatar from the user's video data using computer vision technology, and to map the user's facial features, particularly facial shape features, extracted by the feature extraction unit 30, onto the generated avatar to specify a facial landmark.

[0012] Here, a face landmark is a point that indicates the main feature of a face and is widely used in computer vision research and image processing. Such landmarks refer to points corresponding to important parts of the face such as the eyes, nose, mouth, and jawline, and these play a core role in various applications such as face recognition, emotion analysis, face tracking, lip-syncing, and avatar generation. Accurately identifying and tracking face landmarks is essential for understanding and interpreting facial morphology and movement, and through this, even subtle changes in facial expression and eye movements can be captured. Recently, advances in artificial intelligence technology have made even more precise and rapid face landmark detection possible, enabling innovative applications in various fields such as virtual reality, augmented reality, interactive games, and security systems. The avatar generation unit 40 can utilize open-source software for generating avatars, with examples including avatars in the form of Apple's Mimoticons. It can also be applied to character avatars other than the user's personal face, in which case a correspondence between the user's face landmark and the new character avatar's landmark coordinates is required. The face landmark specified in the avatar generation unit 40 serves as the reference for coordinate values ​​in the lip-sync animation implementation unit 70, which will be described later. A detailed explanation of this will be provided later. The feature fusion unit 50 is configured to fuse the feature vectors extracted from the feature extraction unit 30 and convert them into an embedded fusion vector. In another embodiment, the feature fusion unit 50 may be configured to apply weighted values ​​based on predetermined criteria to the feature vectors extracted by the feature extraction unit 30, fuse the weighted feature vectors, and convert them into an embedded fusion vector. Here, the criteria for applying weighted values ​​may be user-selected, or high-importance multimodal data may be pre-defined. The lip-sync reconstruction unit 60 is configured to predict the mouth shape and facial movements of the user when they imagine speaking by inputting the extracted feature vectors into a pre-prepared lip-sync reconstruction model. In one embodiment of the present invention, the lip-sync reconstruction unit 60 may be configured to predict the mouth shape and facial movements of the user when they are about to speak by inputting an embedded fusion vector, in which features have been fused by the feature fusion unit 50, into a pre-provided lip-sync reconstruction model. Such lip-sync reconstruction models can be implemented using deep learning models or autoencoders, such as DNNs (Deep Neural Networks), CNNs (Convolutional Neural Networks), and RNNs (Recurrent Neural Networks). Furthermore, in one embodiment of the present invention, the lip-sync reconstruction model may consist of either a first prediction model that immediately predicts the mouth shape and facial movements of the user when imagining speech from the extracted feature vectors, or a second prediction model that grasps and classifies the user's intention from the extracted feature vectors and predicts the mouth shape and facial movements of the user when imagining speech according to the classified intention. On the other hand, the lip-sync reconstruction unit 60 may be configured to either select only the user's brainwaves as input values ​​from the biosignals measured when the user is imagining speech, or to select the user's brainwaves along with other biosignals as multiple input values. For example, in the case of electromyography as another biosignal, the lip-sync reconstruction unit 60 may be configured to predict mouth patterns and facial movements by inferring the movement trajectories of the articulatory organs corresponding to speech imagination based on the electromyography.

[0013] In this case, when other biological signals such as electromyography are selected as multiple input values ​​along with the user's electroencephalogram, the feature fusion unit 50 is configured to apply a weighting value corresponding to the importance of the biological signals selected as input values ​​or a user-set value to the feature vector extracted by the feature extraction unit 30, fuse the weighted feature vectors to convert them into an embedded fusion vector, and the lip-sync reconstruction unit 60 may be configured to receive this embedded fusion vector, which is the fusion of the weighted feature vectors, and predict the standing posture and facial movements corresponding to imagined speech. Figure 4 shows a data acquisition and processing flowchart for lip-sync reconstruction according to one embodiment of the present invention. The following describes, with reference to Figure 4, a methodology for generating avatars and performing lip-sync animation to visually convey the user's intentions from non-invasive electroencephalogram (EEG) data during speech imagination. For lip-sync animation, the mouth shape and facial movements are reconstructed from speech-imagining EEG data, and this is applied to a newly generated avatar from the user's facial video data to realize a speaking avatar. Compared to the aforementioned existing speech-imagining-based brain-computer interface technologies, this has greater scalability because it can regenerate mouth shape combinations corresponding to all human voices by decoding only about 15 busy (Visme) values ​​that indicate mouth shape, and can construct moving mouth shapes even for sentences that are not predefined. Furthermore, the regeneration performance is also higher than existing EEG-based systems by utilizing about 15 transparency-related features. Speech-imagination EEG involves wearing an EEG device and performing both a speech attempt and a speech-imagination attempt. The speech attempt is recorded simultaneously to supplement the EEG data learning for speech-imagination, supplementing the feature information for words and sentences that the user attempts to speak by moving only the mouth shape without making any sound. Learning of phoneme units that make up words and sentences is prioritized, and both the speech attempt and the speech-imagination attempt take place within approximately 1.5 seconds. Furthermore, actual mouth shape classes used when actually speaking words and sentences are defined, and learning of mouth shape units is added, but the speech attempt and speech-imagination attempt take place simultaneously within the same time frame. This process is repeated n times to collect user-specific speech-imagination EEG data. EEG is measured using a total of 128 channels, including the Broca-Wernicke region, where speech-imagining EEG is characteristically expressed. Generally, EEG data is collected at a sampling frequency of 1,000 Hz, but this value may vary depending on the situation. While measuring EEG data, blinking and unintentional, unnecessary muscle movements may be recorded as noise, but this noise is removed through a preprocessing process via the preprocessing unit 20. The EEG data over time is divided based on different trigger values, and this can be used to collect data suitable for learning. The finally obtained EEG data is extracted using the time-frequency characteristics.

[0014] Simultaneously, while measuring speech-imagining EEG data, a camera placed in front of the user records the user's face, securing facial features and video data. The system can be implemented using this video information, or even just a single 2D photograph provided by the user later. Information on mouth patterns and facial movements during speech attempts can be obtained and used as additional features during the learning process. Approximately 68 facial landmarks can be obtained from the user's face recorded in the facial video data, which will later be used to apply lip-sync animation to the avatar. The pre-processed data is converted into feature vectors that can represent mouth patterns and facial movements. EEG data is converted into biosignal embedding vectors via a biosignal encoder, and video data is converted into facial feature embedding vectors via a facial feature encoder. These embedding vectors are stored in a fused vector form for subsequent learning, and the embedding vector conversion process acts as a process to better represent the features of mouth shapes in relation to the phonemes and deletions of imagined speech, enabling better differentiation of facial features and high-quality reconstruction. This method analyzes information about the movement of the articulatory organs to achieve lip-syncing from electroencephalogram (EEG) data. Based on existing research that the movement trajectories of the articulatory organs are encoded in the speech sensorimotor cortex of the brain, it is possible to infer the movement of the articulatory organs of an independent user. It is possible to analyze 12-dimensional movement trajectories of the articulatory organs, which correspond to x and y displacement values ​​for many parts of the tongue and the upper and lower parts of the lips. The transformed embedding vectors are learned through a lip-sync reconstruction model to derive results for mouth shape or facial movement. The lip-sync reconstruction model for learning can be adjusted to produce optimal learning results by tuning parameters such as the learning rate and placement size. Movement changes are represented by the repositioning of positional values ​​in the trajectory form of facial landmarks, and complex speech face lip-sync animation is realized through the reconstruction of the corresponding landmarks. The generated avatar has N (N>10) landmarks and corresponding coordinates, and these coordinates are applied to match the movement of the reconstructed landmarks, resulting in a natural avatar lip-sync animation. The process works by having a model trained on collected user-specific data. When a user uses the system, brainwaves and biosignals are fed in real time to the already trained model, and a moving face synthesized from the given biosignals is output to the screen in real time. At this time, the user's avatar output can be selected and applied in a manner desired by the user, using either a personalized avatar generated from the user's facial video data or a 2D photograph provided by the user. The lip-sync animation implementation unit 70 is configured to implement avatar lip-sync animation by applying the mouth shape and facial movements predicted by the lip-sync reconstruction unit to the avatar generated by the avatar generation unit. More specifically, the lip-sync animation implementation unit 70 can implement avatar lip-sync animation by applying the mouth shape and facial movements predicted by the lip-sync reconstruction unit 60 based on the coordinate values ​​of the face landmark to the avatar generated by the avatar generation unit 40.

[0015] Figure 5 shows a data processing flowchart for realizing avatar lip-sync animation according to one embodiment of the present invention. Referring to Figure 5, preprocessed biosignal data is processed through a biosignal encoder, biosignal embedding vector, and lip-sync reconstruction decoder to generate mouth patterns and facial movements during the user's imagined speech. Preprocessed video data is processed through a facial feature encoder, facial feature embedding vector, and avatar generation decoder to generate an avatar. The generated mouth patterns and facial movements are then coupled with the avatar to realize the final avatar lip-sync animation. The lip-sync animation implementation unit 70 can factually implement the information derived from the lip-sync reconstruction unit 60 based on the landmark coordinate values ​​of the generated avatar. The avatar can lip-sync to say the presented sentence as imagined by the user, and output actual audio along with it, thereby implementing an avatar animation that appears as if the user is actually speaking. The system can be configured to display the result on the screen in real time to allow the user to see the result and provide feedback to prove that their communication is accurate. The above describes the multimodal biosignal-based avatar lip-sync animation generation apparatus according to the present invention, and the following describes the multimodal biosignal-based avatar lip-sync animation generation method according to the present invention. Figure 6 is a flowchart of a method for generating multimodal biosignal-based avatar lip-sync animation according to one embodiment of the present invention. Referring to Figure 6, a multimodal biosignal-based avatar lip-sync animation generation method according to one embodiment of the present invention includes a multimodal data acquisition step (S10), a preprocessing step (S20), a feature extraction step (S30), an avatar generation step (S40), a feature fusion step (S50), a lip-sync reconstruction step (S60), and a lip-sync animation implementation step (S70). The specific details of each stage have already been described in the multimodal biosignal-based avatar lip-sync animation generation device according to the present invention, so below we will only briefly explain the characteristics of each stage.

[0016] The aforementioned multimodal data acquisition step (S10) is a step in which multimodal data is collected, including biosignal data such as brainwaves during the user's speech imagination, and video data. The aforementioned preprocessing step (S20) is a step in which the multimodal data is preprocessed. The feature extraction step (S30) is a step in which feature vectors, including the user's biosignal features and facial features, are extracted from the pre-processed multimodal data. The avatar generation step (S40) is a step in which an avatar is generated that represents the user's appearance based on the facial features among the extracted feature vectors. The feature fusion step (S50) is a step in which the feature vectors extracted in the feature extraction step are fused and converted into an embedded fusion vector. The lip-sync reconstruction step (S60) is a step in which the extracted feature vectors are input into a pre-established lip-sync reconstruction model to predict the mouth shape and facial movements when the user imagines speaking. In one embodiment of the present invention, the lip-sync reconstruction step (S60) can be configured to predict the mouth shape and facial movements of the user when they imagine speaking by inputting an embedded fusion vector, in which features were fused in the feature fusion step (S50), into a pre-provided lip-sync reconstruction model. The lip-sync animation implementation stage (S70) is a stage in which the mouth patterns and facial movements predicted in the lip-sync reconstruction stage (S60) are applied to the avatar generated in the avatar generation stage (S40) to implement the avatar lip-sync animation. The multimodal biosignal-based avatar lip-sync animation generation apparatus and method according to the present invention, having the configuration described above, offers innovative applicability in the fields of medicine, assistive devices, and everyday communication. In particular, as a customized solution for individuals with physical or linguistic limitations, it enables them to express their intentions in a direct and efficient manner. This technology can be integrated with a variety of interfaces such as voice generation devices, virtual and augmented reality systems, and next-generation communication systems, and can be used to interpret user thoughts in real time and control various digital devices. Such brain-computer interface technologies can also serve as rehabilitation tools for stroke and severe muscle injury patients, representing a major turning point in health management and disability overcoming. Furthermore, because this technology can be applied to general mass markets such as education, entertainment, and personal productivity enhancement, it is expected to become deeply integrated into daily life and accessible to everyone as technology advances. The multimodal biosignal-based avatar lip-sync animation generation apparatus and method shown in the drawings above are merely one embodiment for carrying out the present invention and should not be construed as limiting the technical idea of ​​the present invention. The scope of protection of the present invention is defined solely by the matters described in the following claims, and improved and modified embodiments without departing from the spirit of the present invention are said to fall within the scope of protection of the present invention insofar as they are obvious to a person with ordinary skill in the art to which the present invention pertains. [Explanation of Symbols]

[0017] 10. Multimodal Data Collection Unit 20 Pre-processing 30 Feature extraction unit 40 Avatar Generation Section 50 Feature Fusion Section 60 Lip Sync Reconstruction Unit 70 Lip-Sync Animation Implementation Department

Claims

1. A multimodal data acquisition unit that collects biosignal data, including brainwaves, during the user's speech imagination, and multimodal data, including video data, A preprocessing unit for preprocessing the multimodal data, A feature extraction unit extracts feature vectors including the user's biosignal features and facial features from the pre-processed multimodal data. An avatar generation unit that generates an avatar that represents the user's appearance, A lip-sync reconstruction unit inputs an embedded fusion vector, which is obtained by fusing the extracted feature vectors into a pre-prepared lip-sync reconstruction model, to predict the mouth shape and facial movements of the user when they imagine speaking. The system includes a lip-sync animation implementation unit that applies the mouth shape and facial movements predicted by the lip-sync reconstruction unit to the avatar generated by the avatar generation unit to realize an avatar lip-sync animation, and further includes, The lip-sync reconstruction unit includes a feature fusion unit that fuses the feature vectors extracted by the feature extraction unit and converts them into the embedding fusion vector, and the lip-sync reconstruction unit inputs the embedding fusion vector into a pre-provided lip-sync reconstruction model to predict the mouth shape and facial movements of the user when imagining speech, The feature fusion unit is characterized by applying weighted values ​​according to a predetermined criterion to the feature vectors extracted by the feature extraction unit, fusing the feature vectors to which the weighted values ​​have been applied, and converting them into an embedded fusion vector. A multimodal biosignal-based avatar lip-sync animation generation device.

2. The multimodal biosignal-based avatar lip-sync animation generation apparatus according to claim 1, characterized in that the avatar generation unit generates a 2D or 3D avatar from the user's video data using computer vision technology, maps the user's facial features extracted by the feature extraction unit to the generated avatar to specify facial landmarks, and the lip-sync animation implementation unit implements an avatar lip-sync animation by applying the mouth shape and facial movements predicted by the lip-sync reconstruction unit based on the coordinate values ​​of the facial landmarks to the avatar generated by the avatar generation unit.

3. The lip-sync reconstruction model is a first predictive model that immediately predicts the mouth shape and facial movements of the user when imagining speech from the extracted feature vectors, or The multimodal biosignal-based avatar lip-sync animation generation apparatus according to claim 1, characterized in that it is composed of one of a second prediction model that grasps and classifies the user's intent from the extracted feature vectors and predicts the mouth shape and facial movements of the user when imagining speech according to the classified intent.

4. A multimodal data acquisition stage in which biosignal data, including brainwaves, and video data are collected during the user's speech imagination, A preprocessing step of preprocessing the multimodal data, and a feature extraction step of extracting feature vectors including the user's biosignal features and facial features from the preprocessed multimodal data. The process includes an avatar generation step in which an avatar is generated based on facial features from the extracted feature vectors to represent the user's appearance, and a lip-sync reconstruction step in which the extracted feature vectors are fused and transformed into an embedded fusion vector which is input into a pre-prepared lip-sync reconstruction model to predict the mouth shape and facial movements when the user imagines speaking. A multimodal biosignal-based avatar lip-sync animation generation method, comprising a lip-sync animation realization step of applying the mouth shape and facial movements predicted in the lip-sync reconstruction step to the avatar generated in the avatar generation step to realize the avatar's lip-sync animation, wherein The generation method further includes, The feature fusion step includes fusing the feature vectors extracted in the feature extraction step and converting them into the embedding fusion vector, The lip-sync reconstruction step involves inputting the embedded fusion vector into a pre-defined lip-sync reconstruction model to predict the mouth shape and facial movements of the user when imagining speech. A method for generating multimodal biosignal-based avatar lip-sync animation, characterized in that the feature fusion step involves applying weighted values ​​according to predetermined criteria to the feature vectors extracted in the feature extraction step, and fusing the weighted feature vectors to convert them into embedded fusion vectors.

Images