A shadow play thought control system and method based on electroencephalogram and virtual reality

Abstract

The application discloses a shadow play idea control system and method based on electroencephalogram and virtual reality, and relates to the technical field of intelligent control. In the system, a signal acquisition module synchronously acquires electroencephalogram signals and electrooculogram signals of a user; an electroencephalogram decoding unit of a signal processing and control module is configured to process the electroencephalogram signals based on an EBLM mixed decoding model to output a continuous speed vector for controlling the displacement of a virtual shadow play character; an electrooculogram detection unit is configured to process the electrooculogram signals based on an anti-mis-touch electrooculogram detection algorithm to output an effective blink event trigger signal; a shadow play action synthesis unit in a virtual reality engine module is configured to receive the continuous speed vector to control the continuous displacement of the torso of the virtual shadow play character in a two-dimensional plane; and in response to the effective blink event trigger signal, an instantaneous physical impact force conforming to a multi-body dynamics model is applied to the limbs of the virtual shadow play character, thereby realizing continuous and smooth two-dimensional displacement control of the shadow play character and stable and reliable blink triggering.

Description

Technical Field

[0001] This application relates to the field of intelligent control technology, specifically to a shadow puppetry thought control system and method based on electroencephalography (EEG) and virtual reality. Background Technology

[0002] The transmission and physical preservation of traditional shadow puppetry skills face severe challenges. The oral transmission model is no longer sustainable, and physical shadow puppets are easily damaged by environmental erosion. Existing digital preservation methods, such as virtual reality (VR) interaction based on controllers or motion-sensing devices, are insufficient to capture and reproduce the essence of traditional performance—"controlling form with intention."

[0003] While brain-computer interface technology offers new possibilities, significant bottlenecks remain in practical applications. Existing technologies struggle to reliably convert ambiguous EEG signals into high-precision, continuous, multi-dimensional artistic movements; decoding methods based on single classifiers are susceptible to interference, leading to unstable outputs and discrete movements; auxiliary signals (such as electrooculography) are prone to false triggering; and the system also suffers from poor real-time synchronization and a lack of physical texture in virtual shadow puppets, all of which restrict the practical application of the technology and its artistic expression. Summary of the Invention

[0004] The purpose of this application is to provide a shadow puppetry thought control system and method based on EEG and virtual reality to solve the problems mentioned in the background art.

[0005] In a first aspect, one embodiment of this application provides a shadow puppetry thought control system based on EEG and virtual reality. The system includes: a signal acquisition module, a signal processing and control module, and a virtual reality engine module. The signal acquisition module is used to simultaneously acquire the user's EEG and EEG signals; the signal processing and control module is communicatively connected to the signal acquisition module and is used to process the EEG and EEG signals and generate control commands; the virtual reality engine module is communicatively connected to the signal processing and control module and is used to drive the virtual shadow puppets to perform interactive performances according to the control commands; wherein, the signal processing and control module includes an EEG decoding unit configured to process EEG signals based on an EBLM hybrid decoding model to output a continuous velocity vector for controlling the displacement of the virtual shadow puppet's character; an EEG detection unit configured to process EEG signals based on an anti-mistouch EEG detection algorithm to output a valid blink event trigger signal; the virtual reality engine module includes a shadow puppet motion synthesis unit configured to: receive the continuous velocity vector to control the virtual shadow puppet's torso to perform continuous displacement in a two-dimensional plane; and, in response to a valid blink event trigger signal, apply an instantaneous physical force conforming to a multibody dynamics model to the limbs of the virtual shadow puppet.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, the EBLM hybrid decoding model includes a clustering sub-model and a linear support vector machine (SVM) classification sub-model. The clustering sub-model is configured to perform unsupervised clustering on feature vectors extracted from EEG signals to generate cluster membership features that characterize the strength of sample affiliation to each cluster center; the linear support vector machine (SVM) classification sub-model is configured to receive a fused feature vector formed by fusing the original EEG features and the cluster membership features, and perform multi-directional continuous classification mapping to output discrete motion direction category labels.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the EEG decoding unit is further configured to: map the discrete motion direction category labels output by the linear support vector machine classification sub-model into continuous velocity vectors through velocity smoothing processing. The velocity smoothing processing includes: weighting and combining the base direction vector obtained by mapping the discrete motion direction category labels at the current moment with the two-dimensional velocity vector output at the previous moment; wherein, the weights of the velocity vector at the previous moment and the base direction vector at the current moment in the combination are controlled by a preset smoothing coefficient to generate a smooth and continuous two-dimensional velocity vector at the current moment.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the anti-accidental eye contact detection algorithm includes: generating a binary signal of the closed eye state based on the instantaneous voltage difference between the left and right eye electroencephalogram (EEG) signals and the eye-closing threshold; integrating the binary signal of the closed eye state within a preset integration time window to obtain a continuous eye-closing integrated signal; calculating the rate of change of the continuous eye-closing integrated signal; and determining a valid blinking event and outputting a trigger signal when the continuous eye-closing integrated signal exceeds the minimum eye-closing time threshold and the rate of change meets a preset triggering condition.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the shadow puppet model is a multibody dynamics model built based on a physics engine, wherein: the torso of the shadow puppet is defined as a suspended base, and the torso of the shadow puppet is configured to move freely in a two-dimensional plane under the drive of a continuous velocity vector; the limbs of the shadow puppet are connected to the torso through hinge joints, and the hinge joints are configured with specific rotational damping and gravity scaling coefficients; instantaneous physical impulse is applied to the rigid body corresponding to the limb to drive the limb to swing and naturally return under the combined action of rotational damping and simulated gravity.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the signal processing and control module and the virtual reality engine module synchronize instructions through an asynchronous double-buffered synchronous architecture; wherein, the asynchronous double-buffered synchronous architecture has an instruction buffer queue for receiving and buffering control instructions generated by the signal processing and control module; wherein, the virtual reality engine module reads control instructions from the instruction buffer queue at a predetermined rendering frame rate to complete the logical decoupling between the decoding instruction stream of the signal processing and control module and the graphics rendering frame rate of the virtual reality engine module.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, the asynchronous double-buffered synchronous architecture is further configured to perform time smoothing and linear interpolation on the control instructions read from the instruction buffer queue to generate smooth and continuous motion control signals.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, the system further includes: an event marking module, set in the virtual reality engine module, used to immediately generate a synchronous marking signal corresponding to the state change the instant the virtual shadow puppet's state changes and is presented on the virtual reality screen, and send the synchronous marking signal as a high-precision timestamp to a device for recording EEG and EEG signals, so as to align the visual event of the state change with the corresponding original EEG and EEG signals on the time axis.

[0013] Secondly, one embodiment of this application provides a method for controlling shadow puppetry based on electroencephalography (EEG) and virtual reality. The method includes: synchronously acquiring the user's EEG and electrooculography (EOG) signals; processing the EEG signals based on an EBLM hybrid decoding model to decode a continuous velocity vector for controlling the displacement of the virtual shadow puppet character; processing the EOG signals based on an anti-accidental touch EOG detection algorithm to determine and generate a valid blink event trigger signal; controlling the torso of the virtual shadow puppet to perform two-dimensional continuous displacement based on the continuous velocity vector in a virtual reality environment; and in response to the valid blink event trigger signal, applying an instantaneous physical force to the limbs of the virtual shadow puppet based on a multibody dynamics model to drive the limbs to produce a swinging motion.

[0014] In conjunction with the second aspect, in some implementations of the second aspect, the EEG signal is processed based on the EBLM hybrid decoding model to decode the continuous velocity vector used to control the displacement of the virtual shadow puppet. This includes: extracting features from the EEG signal to obtain the original feature vector; performing K-means clustering on the original feature vector, calculating the Euclidean distance between the original feature vector and each cluster center, and converting it into a cluster membership feature vector; fusing the original feature vector and the cluster membership feature vector to form a fused feature vector; inputting the fused feature vector into a pre-trained linear support vector machine classifier to obtain discrete motion direction category labels; and converting the discrete motion direction category labels into a continuous velocity vector using a velocity smoothing formula.

[0015] Compared with the prior art, the beneficial effects of this application are: The shadow puppetry intention control system provided in this application embodiment, based on the EBLM hybrid decoding model and the anti-accidental touch electrooculography detection algorithm, realizes continuous and smooth two-dimensional displacement control of shadow puppet characters and stable and reliable blink triggering, overcoming the problems of discrete and unstable output of traditional single classifiers; based on the multibody dynamics model, it accurately reproduces the motion inertia of shadow puppets, and combined with the asynchronous double buffer synchronous architecture, it controls the end-to-end system latency to within 50ms, significantly improving the smoothness and realism of operation. This immersive interactive method of "controlling form with intention" enhances the user's deep resonance with intangible cultural heritage. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the structure of a shadow puppetry thought control system based on electroencephalography and virtual reality, provided as an embodiment of this application.

[0017] Figure 2 The diagram shown is a structural schematic of a shadow puppet model in one embodiment of this application.

[0018] Figure 3 This is a schematic diagram of the structure of an EBLM hybrid decoding model provided in another embodiment of this application.

[0019] Figure 4 This is a flowchart illustrating an anti-accidental touch electrooculography detection algorithm provided in another embodiment of this application.

[0020] Figure 5 A schematic diagram of the structure of a shadow puppetry thought control system based on electroencephalography and virtual reality, provided as another embodiment of this application.

[0021] Figure 6 This is a flowchart illustrating a method for controlling the mind in shadow puppetry based on electroencephalography and virtual reality, provided in an embodiment of this application.

[0022] Figure 7 This is a schematic diagram illustrating the process of processing EEG signals based on an EBLM hybrid decoding model to decode a continuous velocity vector used to control the displacement of a virtual shadow puppet, according to one embodiment of this application.

[0023] Figure 8 This is an overall experimental schematic diagram of a shadow puppetry thought control method based on EEG and virtual reality provided in an embodiment of this application.

[0024] Figure 9 The diagram shown is a schematic diagram of the accuracy of motor imagery (MI) classification in one embodiment of this application.

[0025] Figure 10A line graph showing the trend of increasing number of people and changes in classification accuracy.

[0026] Figure 11 Line graph showing the change in completion time for Group A and Group B as the number of participants.

[0027] Figure 12 The image shown is a radar chart of the multidimensional assessment. Detailed Implementation

[0028] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0029] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application's specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The term "and / or" as used in this application includes any and all combinations of one or more of the associated listed items.

[0030] Figure 1 This is a schematic diagram of the structure of a shadow puppetry thought control system based on electroencephalography and virtual reality, provided as an embodiment of this application. Figure 1 As shown, this embodiment provides a shadow puppetry thought control system based on EEG and virtual reality. The system includes a signal acquisition module, a signal processing and control module, and a virtual reality engine module. The signal processing and control module is communicatively connected to the signal acquisition module, and the virtual reality engine module is communicatively connected to the signal processing and control module.

[0031] The signal acquisition module is used to simultaneously acquire the user's electroencephalogram (EEG) and electrooculogram (EOG) signals. The signal processing and control module is used to process the EEG and EOG signals and generate control commands. The virtual reality engine module is used to drive the virtual shadow puppets to perform interactive shows based on the control commands.

[0032] The signal processing and control module includes an EEG decoding unit and an EEG detection unit. The EEG decoding unit is configured to process EEG signals using an Energy-Based Language Model (EBLM) hybrid decoding model to output a continuous velocity vector for controlling the displacement of the virtual shadow puppet character. The EEG detection unit is configured to process EEG signals based on an anti-mistouch EEG detection algorithm to output a valid blink event trigger signal.

[0033] The virtual reality engine module includes a shadow puppet motion synthesis unit, which is configured to receive continuous velocity vectors to control the torso of the virtual shadow puppet to make continuous displacement in a two-dimensional plane; and to apply instantaneous physical force conforming to a multibody dynamics model to the limbs of the virtual shadow puppet in response to a valid blink event trigger signal.

[0034] For example, EEG signals are weak potential fluctuations generated by the electrical activity of neurons in the brain. They are collected through scalp electrodes and can be decoded into cognitive states such as the user's motor intentions.

[0035] For example, electrooculography (EOG) signals are the corneal-retinal potential difference generated by eye movements and eyelid opening and closing, measured by periocular electrodes, and are often used to detect events such as blinking.

[0036] For example, the control command is a computer command generated by the signal processing and control module after decoding, fusing and judging the collected EEG and EEG signals. It is used to drive the virtual shadow puppet's movements. The control command includes a continuous two-dimensional velocity vector generated based on EEG decoding (controlling the shadow puppet's displacement) and a trigger signal generated based on blink detection (controlling the shadow puppet's specific movements).

[0037] It should be understood that Figure 2 The diagram shown is a structural schematic of a shadow puppet model in one embodiment of this application. The virtual shadow puppet is a digital shadow puppet model created in a virtual reality engine. Its movement is simulated based on a multibody dynamics physical model, including a torso unaffected by gravity and displaced by brainwaves, and arms connected by hinge joints, driven by impulses triggered by blinking, and naturally swinging under the influence of gravity and damping. Users control the virtual shadow puppet's continuous movement in a two-dimensional VR scene through thought (motor imagery), and trigger specific limb movements such as arm swings by blinking, thereby completing preset tasks such as spatial positioning and collaboration, or engaging in free exploration on a virtual stage, providing an immersive experience.

[0038] The shadow puppetry intention control system provided in this application embodiment, based on the EBLM hybrid decoding model and the anti-accidental touch electrooculography detection algorithm, realizes continuous and smooth two-dimensional displacement control of shadow puppet characters and stable and reliable blink triggering, overcoming the problems of discrete and unstable output of traditional single classifiers; based on the multibody dynamics model, it accurately reproduces the motion inertia of shadow puppets, and combined with the asynchronous double buffer synchronous architecture, it controls the end-to-end system latency to within 50ms, significantly improving the smoothness and realism of operation. This immersive interactive method of controlling form with intention enhances the user's deep resonance with intangible cultural heritage.

[0039] Figure 3 This is a schematic diagram of the structure of an EBLM hybrid decoding model provided in another embodiment of this application. Figure 3As shown in the embodiments of this application, the EBLM hybrid decoding model includes: a clustering sub-model and a linear support vector machine classification sub-model.

[0040] The clustering sub-model is configured to perform unsupervised clustering on feature vectors extracted from EEG signals to generate cluster membership features that characterize the strength of a sample’s affiliation to each cluster center.

[0041] For example, the EEG decoding unit is configured to process EEG signals based on an EBLM hybrid decoding model that combines unsupervised and supervised learning. By introducing unsupervised clustering before supervised classification, smooth and continuous movement of virtual shadow puppet characters in two-dimensional space can be achieved.

[0042] The linear support vector machine classification sub-model is configured to receive a fused feature vector, which is formed by fusing raw EEG features with cluster membership features, and perform multi-directional continuous classification mapping to output discrete motion direction category labels.

[0043] First, the K-means clustering space is partitioned to separate the raw EEG feature vectors containing different motor intentions (such as imagining moving left, right, up, or down) and resting states. Input the data into the K-means algorithm, which divides the sample into K clusters. The optimization objective of the K-means algorithm is to minimize the sum of squared Euclidean distances between all samples and their corresponding cluster centers. The algorithm formula is: ; : Represents the centroid (center of the k-th cluster), which is a parameter that needs to be optimized; : Represents the set of all samples belonging to the k-th cluster; : Represents the sample feature vector With cluster center The Euclidean distance between them.

[0044] This step generates a novel feature representation based on the membership degree of each cluster center to the sample, which improves the classifier's ability to distinguish different directions of motion.

[0045] The Euclidean distance between the sample and each cluster center is converted into similarity weights, and then normalized to obtain the membership degree of the sample to each cluster center (used to represent the strength of the sample's belonging to each cluster center). Each sample will obtain a K-dimensional vector, where each value in the vector corresponds to the degree of membership to a cluster center, and the sum of all values ​​is 1.

[0046] After obtaining the original EEG feature vector and the corresponding cluster membership vector, a feature-level fusion strategy is adopted to concatenate the two to form a unified extended feature representation. The fused feature vector is then uniformly standardized. Finally, the generated fused feature vector is used as input for training and prediction of the Linear Support Vector Machine (LSVM) classifier.

[0047] Specifically, the feature vectors will be fused. The input is used to perform multi-directional continuous classification mapping in an LSVM. The classification function of the LSVM is defined as: ; Where w is the normal vector of the classification hyperplane, which is the weight parameter that the model needs to learn; b is the bias term of the classification hyperplane.

[0048] In a given label On a training dataset where +1 represents "left" and -1 represents "right," the optimization objective of LSVM is to find an optimal hyperplane that maximizes the classification margin while minimizing the classification error. Its mathematical expression is: ； ； in, This is a regularization parameter used to balance the two objectives of "maximizing the classification margin" and "minimizing the classification error". This represents the total number of training samples. and These are the weight vector and bias term of the classification hyperplane, respectively; It is a slack variable. It refers to the directional category label. The optimization process of LSVM is to ensure that each sample is either correctly classified and has a sufficient margin (by 1). Given the definition of ξi, and assuming a penalty (ξi increases), we find a hyperplane that achieves the optimal balance between maximizing the classification margin and minimizing the classification error penalty by adjusting w and b. This balance is controlled by the regularization parameter C. The final w and b define the found optimal classification hyperplane.

[0049] Finally, the LSVM classifier classifies each input sample into a predefined (Up, Down, Left, Right, Rest) direction category. The output at each time step is a discrete category label representing the currently identified motion intention direction.

[0050] In one specific embodiment, the EEG decoding unit is further configured to: map the discrete motion direction category labels output by the linear support vector machine classification sub-model into continuous velocity vectors through velocity smoothing processing. The velocity smoothing processing includes: weighting and combining the base direction vector obtained by mapping the discrete motion direction category labels at the current moment with the two-dimensional velocity vector output at the previous moment; wherein, the weights of the velocity vector at the previous moment and the base direction vector at the current moment in the combination are controlled by a preset smoothing coefficient to generate a smooth and continuous two-dimensional velocity vector at the current moment.

[0051] Discrete motion direction category labels are transformed into continuous two-dimensional velocity vectors through linear interpolation. The calculation formula is as follows: ; Here, α is a smoothing coefficient, which ensures the continuity and naturalness of the output velocity vector. At the current moment A two-dimensional velocity vector output from decoded and smoothed EEG signals. At the previous moment A two-dimensional velocity vector output from decoded and smoothed EEG signals. The linear support vector machine classifier at the current time step The output is a raw, unsmoothed discrete direction vector.

[0052] Final velocity output at the current moment It is the smooth output of the previous moment. Compared with the original instructions at the current moment The weighted sum. When the value of α is large, the formula "relies" more on historical states and current new instructions. The instantaneous changes in α have a relatively small impact on the final output, resulting in a very smooth and highly inertial change in the velocity vector. This effectively suppresses command jitter caused by fluctuations in EEG signals or momentary misjudgments by the classifier. When the α value is small, the formula "closely follows" the current command and is more responsive, but the smoothing effect is weakened, potentially retaining more high-frequency fluctuations from the original signal. This linear interpolation (smoothing) formula solves the problem of severe discretization in the output of traditional single classifiers by integrating "historical smoothed velocity" with "current original command," achieving a natural transition from "discrete direction" to "continuous velocity vector" in control commands, and ensuring the smoothness of the shadow puppet character's movement.

[0053] Figure 4 This is a flowchart illustrating an anti-accidental touch electrooculography (EOG) detection algorithm provided in another embodiment of this application. Figure 4 As shown in the embodiments of this application, the anti-accidental touch electrooculography detection algorithm includes the following steps.

[0054] Step 401: Generate a binary signal for the closed-eye state based on the instantaneous voltage difference between the left and right eye electroencephalograms and the eye-closing threshold.

[0055] Blinking is used as an input signal to trigger specific behaviors or events. To achieve accurate blink detection, a complete detection process can be constructed based on the voltage difference between the left and right eyes. First, the instantaneous voltage difference is defined: ; This difference reflects changes in the electromyographic signals and ocular potentials of the left and right eyes, and is a fundamental indicator for determining the state of eye closure. Through observation... The fluctuations can initially identify voltage differences when eyes are closed, but this is prone to misjudgment. Therefore, further processing is needed to enhance the stability of the judgment.

[0056] Therefore, a method of continuous eye-closing integration is introduced, and a binary eye-closing label is defined. : ; in The threshold for closing the eyes is defined as follows: when the instantaneous voltage difference is below the threshold, the eyes are considered to be in a closed state; when it is above the threshold, the eyes are considered to be open. The threshold for closing the eyes can be set according to actual conditions, and this application does not impose specific limitations on it.

[0057] Step 402: Integrate the binary signal of the closed-eye state within a preset integration time window to obtain a continuous closed-eye integrated signal.

[0058] Specifically, the binary signal indicating closed eyes is integrated and accumulated to form a continuous integrated signal indicating closed eyes. The formula is as follows: ； in For integration time window, integration time window It is usually set to 200-500ms to cover a complete blink. The sampling time interval is defined as . Through continuous integration, the system can effectively reduce instantaneous noise interference and reflect the duration of eye closure, providing a reliable basis for action triggering.

[0059] Step 403: Calculate the rate of change of the continuous closed-eye integral signal.

[0060] To further improve the robustness of the judgment, the system introduces derivative determination of the closed-eye integral signal. Using derivative information can avoid false triggering due to instantaneous integral fluctuations. In the actual triggering logic, when continuous closed-eye integration occurs... Exceeding the minimum eye-closing time threshold And derivative Greater than the set threshold When the action trigger condition is met, that is ； in, It is typically set to a lower limit slightly shorter than the duration of a natural blink (e.g., 50-100ms) to filter out brief interruptions. Derivative Integral signal The rate of change.

[0061] Step 404: When the continuous eye-closing integral signal exceeds the minimum eye-closing time threshold and the rate of change meets the preset triggering condition, it is determined to be a valid blinking event and a trigger signal is output.

[0062] During blinking, when the eyes begin to open after closing, the instantaneous binary signal... It will jump from 1 back to 0, which causes the oldest "1" in the integration window to be removed. There will be a sudden, sharp drop, corresponding to a negative derivative with a large absolute value. Therefore, The condition >δ (usually set to a negative threshold, such as -0.5) precisely captures the transition moment between the end of a blink and the opening of the eyes. Combining it with the duration condition means that the system only triggers an action at the instant when it detects a sufficiently long closed eye state that has just ended. This simulates the natural human intuition of performing an action after blinking and completely avoids false triggers during the closed eye process or caused by noise.

[0063] This scheme ensures that the corresponding action is triggered only when the eyes are continuously and stably closed, effectively suppressing false actions caused by brief blinks or noise interference. To minimize the jitter and fluctuation of the action signal, the system uses an exponential moving average (EMA) to smooth the action trigger signal, further reducing false triggers caused by signal jitter and improving the robustness and practicality of the system.

[0064] In one embodiment, the shadow puppet model is a multibody dynamics model built based on a physics engine. The torso of the shadow puppet is defined as a suspended base and is configured to move freely in a two-dimensional plane under the drive of a continuous velocity vector. The limbs of the shadow puppet are connected to the torso through hinge joints, which are configured with specific rotational damping and gravity scaling coefficients. Instantaneous physical impulse is applied to the rigid body corresponding to the limb to drive the limb to swing and naturally return under the combined action of rotational damping and simulated gravity.

[0065] To reproduce the essence of the "shadow puppetry" manipulation and motion inertia of Central Plains shadow puppetry in digital space, this system establishes a simplified multibody dynamics model based on a physics engine. The specific implementation scheme is as follows: The puppet's torso is defined as the system's floating base, and the gravitational acceleration effect under its rigid body properties is removed in the Unity engine to simulate the vertical support force provided by the main control rod (main rod) in traditional performances. This design ensures that the torso can move freely within the two-dimensional virtual screen plane driven by brainwave commands without causing unexpected falls.

[0066] The left and right arms are designed as independent dynamic rigid body units, connected to the shoulder joint coordinates above the torso via a Hinge Joint. The system configures specific rotational damping (Angular Drag) and gravity scaling factor for this joint. In Unity, Angular Drag is typically set to 0.2–3.5. To ensure that the arm produces 1–2 natural decaying swings after impact and to simulate the effects of air resistance and joint friction in limb movement, 1.35 was selected through multiple tests to achieve the best visual performance. The gravity scaling factor was set to 1.2 to utilize natural gravity to achieve automatic swingback of the arm after pulse triggering.

[0067] When the system detects When a valid blink event occurs (=1), an instantaneous upward impulse is applied to the rigid body of the arm. After the external force-driven pulse ends, the arm hangs down under the natural action of simulated gravity and swings back to the vertical equilibrium position according to the damping settings. Specifically, within a single physical step, an initial angular momentum is given to the arm by directly applying a momentum impulse Δp, and then the power is immediately cut off, allowing the physics engine to take over the subsequent pure physical swing, thus visually accurately reproducing the agile yet not loose inertial dynamic beauty of traditional shadow puppetry performances.

[0068] In one specific embodiment, the signal processing and control module and the virtual reality engine module synchronize instructions via an asynchronous double-buffered synchronization architecture. This architecture includes an instruction buffer queue for receiving and buffering control instructions generated by the signal processing and control module. The virtual reality engine module reads control instructions from the instruction buffer queue at a predetermined rendering frame rate, thus logically decoupling the decoded instruction stream of the signal processing and control module from the graphics rendering frame rate of the virtual reality engine module. The asynchronous double-buffered synchronization architecture is further configured to perform time smoothing and linear interpolation processing on the control instructions read from the instruction buffer queue to generate smooth and continuous motion control signals.

[0069] The system hardware layer transmits signals to the backend in real time through the Lab Streaming Layer (LSL) communication protocol to achieve ultra-low latency processing at the millisecond level.

[0070] In terms of rendering optimization, this application designs an asynchronous double-buffered architecture, specifically by establishing an instruction buffer queue for the system. The backend EBLM algorithm generates a velocity vector at its own pace. and blinking commands The EEG decoding command stream is first written to a buffer. The front-end VR rendering engine (at a fixed frame rate, such as 90Hz) reads the latest processed commands from another buffer. These two buffers work alternately, logically decoupling the EEG decoding command stream from the graphics rendering frame rate, avoiding stuttering caused by direct impact on the rendering thread due to decoding rate fluctuations. The time smoothing algorithm can be based on the exponential moving average algorithm. Linear interpolation is applied to the discrete motion velocity vector signal decoded by the EBLM algorithm; this processing occurs in the read buffer of the Unity 3D engine's built-in command buffer queue.

[0071] By introducing time smoothing algorithms and linear interpolation, the system can effectively counteract the command jitter caused by the non-stationarity of bioelectric signals, thereby enhancing the stability of motion transitions and controlling the end-to-end system delay to within 50ms.

[0072] Figure 5 A schematic diagram of a shadow puppetry thought control system based on electroencephalography and virtual reality, provided as another embodiment of this application. Figure 5 As shown in the embodiment of this application, the system further includes an event marking module. The event marking module is located in the virtual reality engine module and is used to immediately generate a synchronous marking signal corresponding to the state change the instant the virtual shadow puppet's state undergoes a specific change and is displayed on the virtual reality screen. The synchronous marking signal is then sent as a high-precision timestamp to a device used to record electroencephalogram (EEG) and electrooculogram (EOG) signals, so that the visual event of the state change is aligned with the corresponding original EEG and EOG signals on the timeline.

[0073] Regarding the synchronization mechanism, the system has a built-in event tagging and sending module in the Unity 3D engine. When a specific visual stimulus (change in the state of the shadow puppet) is presented on the screen, the module synchronously sends the tagging signal corresponding to the visual stimulus through the LSL protocol, embedding it as a high-precision timestamp into the bio-EEG / EEG data stream, thereby ensuring strict alignment between the visual presentation and the original EEG / EEG signals on the timeline.

[0074] Figure 6 This is a flowchart illustrating a thought control method for shadow puppetry based on electroencephalography (EEG) and virtual reality, provided as an embodiment of this application. Figure 6As shown in the embodiments of this application, the method includes the following steps: Step 100: Simultaneously collect the user's electroencephalogram (EEG) and electrooculogram (EOG) signals.

[0075] Step 200: Process EEG signals based on the EBLM hybrid decoding model to decode the continuous velocity vector used to control the displacement of the virtual shadow puppet.

[0076] Step 200 decodes a smooth, continuous velocity vector from the EEG signal using the EBLM hybrid decoding model, solving the common problems of discontinuous commands and discrete outputs in traditional brain-computer interface control. This enables the virtual shadow puppet torso to move naturally and smoothly in two-dimensional space, laying the foundation for precise stage positioning and movement.

[0077] Step 300: Process the electrooculogram signal based on the anti-accidental touch electrooculogram detection algorithm, determine and generate a valid blink event trigger signal.

[0078] The anti-accidental triggering electrooculography detection algorithm introduced in step 300 effectively filters out noise and transient eye fluctuations by combining integral and derivative determination mechanisms, significantly reducing the false triggering rate and enabling the natural behavior of blinking to be reliably converted into a trigger switch for limb movements.

[0079] Step 400: In the virtual reality environment, the torso of the virtual shadow puppet is controlled to perform two-dimensional continuous displacement based on the continuous velocity vector.

[0080] Step 500: In response to a valid blink event trigger signal, an instantaneous physical force is applied to the limbs of the virtual shadow puppet based on a multibody dynamics model to drive the limbs to produce a swinging motion.

[0081] Steps 400 and 500 map the two control commands mentioned above to the torso movement and limb swing of the virtual shadow puppet, respectively, and render them using a multi-body dynamics model based on a physics engine, so that the shadow puppet's movement presents realistic physical inertia, damping, and gravity effects. This design not only achieves a hands-free immersive interaction of "controlling form with intention," but also greatly enhances the user's sense of presence, control, and artistic expression in the experience of intangible cultural heritage by coupling bioelectric control signals with real physical motion laws.

[0082] Figure 7 This is a schematic diagram illustrating a process for processing electroencephalogram (EEG) signals using an EBLM hybrid decoding model, as provided in one embodiment of this application, to decode continuous velocity vectors used to control the displacement of a virtual shadow puppet character. (See attached diagram.) Figure 7 As shown in the embodiment of this application, the EEG signal is processed based on the EBLM hybrid decoding model to decode the continuous velocity vector used to control the displacement of the virtual shadow puppet, including the following steps.

[0083] Step 701: Extract features from the EEG signal to obtain the original feature vector.

[0084] Step 702: Perform K-means clustering on the original feature vector, calculate the Euclidean distance between the original feature vector and each cluster center, and convert it into a cluster membership feature vector.

[0085] Step 703: The original feature vector is fused with the cluster membership feature vector to form a fused feature vector.

[0086] Steps 701-703 introduce K-means clustering, which, based on traditional feature extraction, maps the original features to a clustering space that better represents the inherent structure of different motion intentions, and then fuses the obtained cluster membership features with the original features. This feature enhancement mechanism provides the subsequent classifier with more easily separable and information-rich feature representations, thereby effectively improving the model's ability to distinguish between different motion intention categories (such as front, back, left, right, etc.).

[0087] Step 704: Input the fused feature vector into the pre-trained linear support vector machine classifier to obtain discrete motion direction category labels.

[0088] Step 704 uses a linear support vector machine (LSVM) to classify the enhanced fused features. Because of its advantage of minimizing structural risk, it can find the optimal classification surface in the high-dimensional feature space, thus ensuring the generalization performance of the model.

[0089] Step 705: Convert the discrete motion direction category labels into continuous velocity vectors using a velocity smoothing formula.

[0090] Step 705 cleverly transforms the discrete direction labels output by the classifier into continuous two-dimensional velocity vectors by introducing a velocity smoothing formula. This eliminates the sense of stepping and mechanicalness in instructions caused by simple classification, making the movement transition of the virtual character smooth and natural. It solves the technical bottleneck of discrete output and stiff control that is common in traditional motor imagination brain-computer interfaces in continuous control tasks.

[0091] This application creatively integrates unsupervised and supervised learning to construct an EBLM hybrid decoding model, significantly improving the accuracy, stability, and continuity of EEG decoding. The entire process works together to provide an efficient and robust solution for the continuous, stable, and precise control of virtual characters driven by EEG signals.

[0092] Figure 8This is a schematic diagram of an overall experiment for a shadow puppetry thought control method based on EEG and virtual reality, provided in one embodiment of this application. The experiment was conducted in a controlled laboratory environment to minimize the impact of external interference on EEG signals and behavioral performance. Group A subjects wore a 32-channel non-invasive EEG acquisition device and an HTC Vive headset; the system simultaneously acquired EEG and EOG signals. Group B subjects wore only a VR headset and used a standard controller for interactive operation. The complete experimental process lasted approximately 60 minutes and included three consecutive phases (such as...). Figure 8 (As shown).

[0093] The first phase was the system calibration and familiarization phase. In this phase, Group A participants performed a motion visualization task in four directions based on visual cues in the virtual environment. Some of the collected data was used for personalized fine-tuning of the pre-trained EBLM model and for practicing blink-triggered operations. Group B participants, on the other hand, familiarized themselves with basic operations such as joystick movement and button triggering during this phase, aiming to reduce the interference of the new interaction method on subsequent experimental results. The second phase was the standardized task testing phase. Both groups of participants completed uniform operational tasks within the same virtual shadow puppetry scene: Task 1 (spatial positioning) required participants to move the shadow puppet character from the left side of the stage to a designated area in the center of the screen; Task 2 (collaborative performance) required triggering the shadow puppet's waving gesture. After completing the spatial positioning, participants needed to interact with the character on the virtual stage within a limited time. Each task was repeated 20 times to record task completion performance and learning trends. The third stage is the subjective assessment and free exploration phase: Participants first experience the system's interactive methods and operational atmosphere through a short period of free exploration. They then independently complete a system usability scale, a NASA-TLX cognitive load scale, and a presence questionnaire, followed by a brief semi-structured interview. The interview focused on the intuitiveness of intention control, the immersive experience, and its impact on the understanding of shadow puppetry culture.

[0094] Figure 9 The diagram shown is a schematic representation of the accuracy of motor imagery (MI) classification in one embodiment of this application. Figure 9 As shown, the horizontal axis represents different subjects, and the vertical axis represents classification accuracy. Statistical results show that the average overall accuracy of the subjects was 82.7%, with a standard deviation of 4.5%, and the accuracy ranged from a minimum of 75.2% to a maximum of 89.6%. Notably, most subjects achieved stable and usable decoding performance without long-term training, and the accuracy distribution among subjects showed a relatively concentrated trend, with no significantly extreme low values. These results indicate that the EBLM framework has good versatility for novice users and does not impose excessive restrictions.

[0095] In one experiment, the model maintained a balanced accuracy in recognizing "up, down, left, right" motion commands (81.5%~83.2%), with the highest accuracy in the resting state (87.4%). The high distinguishability between the resting state and motion imagination maintained a low false trigger rate (3.6%), providing a reliable foundation for continuous interaction.

[0096] Figure 10 A line graph showing the trend of increasing number of users and changes in classification accuracy. (Example) Figure 10 As shown, the horizontal axis represents the number of participants (from 1 to 20), and the vertical axis represents the classification accuracy. Figure 10 The midline shows that the accuracy rate generally shows a continuous upward trend: it gradually increases from 76.8% for the first person, to 80.2% for the sixth person, 82.6% for the eleventh person, and finally reaches 84.5% for the twentieth person. This indicates that as the sample size increases, the classification performance of the model or system is steadily optimized, especially with rapid growth in the early stages, followed by a slower growth rate and eventual stabilization.

[0097] Figure 11 Line graphs showing the change in completion time for Group A and Group B as a function of the number of participants are shown, with the horizontal axis representing the number of participants (1–20) and the vertical axis representing the completion time (seconds). Due to the specificity of intention control and cognitive load, Group A showed a significantly longer average completion time (22.4 seconds) in the initial trial phase (Group B: 10.5 seconds; t(18)=8.52, p<0.001, d=2.65). After 20 training trials, Group A showed a significant learning effect, with its average completion time shortened to 13.2 seconds.

[0098] Although Group B performed better in the final trial (7.6 seconds) due to the immediate feedback from the physical controller, statistical analysis showed a significant difference between the two groups (t(18)=5.24, p<0.001). However, the time reduction rate in Group A (41.1%) was much higher than that in Group B (27.6%), demonstrating that intention-controlled therapy has strong adaptability and therapeutic potential. The comparison results are shown in [link to comparison results]. Figure 12 .

[0099] Figure 12 The radar chart shown compares the performance of Group A and Group B across four dimensions: mental stress (NASA-TLX), cultural presence, spatial presence (IPQ), and system usability (SUS). Each axis in the chart represents an assessment dimension, with higher values ​​indicating better performance (e.g., higher SUS scores are better, and lower mental stress is better).

[0100] As can be seen from the figure: Group A scored higher in mental load (52.7) and spatial immersion (86.1), but slightly lower in cultural immersion (79.3) and system availability (82.3) than Group B; Group B performed better in cultural immersion (90) and system availability (82.3), but slightly weaker in mental load (54.3) and spatial immersion (79.3).

[0101] Overall, Group A focuses more on immersion and cognitive load, while Group B emphasizes usability and cultural experience. Each has its own advantages in different dimensions, forming a complementary user experience structure.

[0102] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A shadow puppetry thought control system based on electroencephalography (EEG) and virtual reality, characterized in that, include: The signal acquisition module is used to simultaneously acquire the user's electroencephalogram (EEG) and electrooculogram (EOG) signals; The signal processing and control module is communicatively connected to the signal acquisition module and is used to process the electroencephalogram (EEG) signals and electrooculogram (EOG) signals, and generate control commands. The virtual reality engine module is communicatively connected to the signal processing and control module and is used to drive the virtual shadow puppets to perform interactive shows according to the control commands. The signal processing and control module includes: The EEG decoding unit is configured to process the EEG signals based on the EBLM hybrid decoding model to output a continuous velocity vector for controlling the displacement of the virtual shadow puppet character; The electrooculogram (EOG) detection unit is configured to process the EOG signal based on an anti-accidental touch EOG detection algorithm to output a valid blink event trigger signal; The virtual reality engine module includes: The shadow puppet motion synthesis unit is configured to: receive the continuous velocity vector to control the torso of the virtual shadow puppet to perform continuous displacement in a two-dimensional plane; and, in response to the effective blink event trigger signal, apply an instantaneous physical force conforming to a multibody dynamics model to the limbs of the virtual shadow puppet.

2. The system according to claim 1, characterized in that, The EBLM hybrid decoding model includes: A clustering sub-model is configured to perform unsupervised clustering on feature vectors extracted from the EEG signals to generate cluster membership features that characterize the strength of a sample’s affiliation to each cluster center. The linear support vector machine classification sub-model is configured to receive a fused feature vector formed by fusing the original EEG features with the cluster membership features, perform multi-directional continuous classification mapping, and output discrete motion direction category labels.

3. The system according to claim 2, characterized in that, The EEG decoding unit is further configured to: The discrete motion direction category labels output by the linear support vector machine classification sub-model are mapped to the continuous velocity vector through velocity smoothing processing. The speed smoothing process includes: The base direction vector obtained by mapping the discrete motion direction category label at the current moment is weighted and combined with the two-dimensional velocity vector output at the previous moment. Specifically, a preset smoothing coefficient is used to control the weight of the velocity vector from the previous moment and the base direction vector from the current moment in the combination, so as to generate a smooth and continuous two-dimensional velocity vector at the current moment.

4. The system according to claim 1, characterized in that, The electrooculogram detection algorithm for preventing accidental contact includes: Based on the instantaneous voltage difference between the left and right eye electrical signals and the eye-closing threshold, a binary signal for the closed eye state is generated. The binary signal of the closed-eye state is integrated within a preset integration time window to obtain a continuous closed-eye integrated signal. Calculate the rate of change of the continuous closed-eye integral signal; When the continuous eye-closing integral signal exceeds the minimum eye-closing time threshold and the rate of change meets the preset triggering condition, it is determined to be a valid blinking event and a trigger signal is output.

5. The system according to claim 1, characterized in that, The shadow puppet model is a multibody dynamics model built based on a physics engine, wherein: The torso of the shadow puppet is defined as a suspended base, and the torso of the shadow puppet is configured to move freely in a two-dimensional plane under the drive of the continuous velocity vector; The shadow puppet's limbs are connected to the torso via hinge joints, which are configured with specific rotational damping and gravitational scaling factors. The instantaneous physical force is applied to the rigid body corresponding to the limb to drive the limb to swing and naturally return under the combined action of rotational damping and simulated gravity.

6. The system according to claim 1, characterized in that, The signal processing and control module and the virtual reality engine module synchronize instructions through an asynchronous double-buffered synchronous architecture. The asynchronous double-buffered synchronous architecture includes an instruction buffer queue for receiving and buffering control instructions generated by the signal processing and control module. The virtual reality engine module reads control commands from the instruction buffer queue at a predetermined rendering frame rate to achieve logical decoupling between the decoding instruction stream of the signal processing and control module and the graphics rendering frame rate of the virtual reality engine module.

7. The system according to claim 6, characterized in that, The asynchronous double-buffered synchronous architecture is further configured to perform time smoothing and linear interpolation processing on the control instructions read from the instruction buffer queue to generate smooth and continuous motion control signals.

8. The system according to any one of claims 1 to 7, characterized in that, The system also includes: An event tagging module, located in the virtual reality engine module, is used to immediately generate a synchronous tagging signal corresponding to the state change the instant the virtual shadow puppet's state changes and is displayed on the virtual reality screen. The synchronous tagging signal is then sent as a high-precision timestamp to a device for recording the electroencephalogram (EEG) signal and the electrooculogram (EOG) signal, so as to align the visual event of the state change with the corresponding original EEG and EOG signals on the time axis.

9. A method for controlling shadow puppetry based on electroencephalography (EEG) and virtual reality, characterized in that, include: Simultaneously collect the user's electroencephalogram (EEG) and electrooculogram (EOG) signals; The EEG signals are processed using the EBLM hybrid decoding model to decode the continuous velocity vector used to control the displacement of the virtual shadow puppet character. The electrooculogram (EOG) signal is processed based on the anti-accidental touch EOG detection algorithm to determine and generate a valid blink event trigger signal; In a virtual reality environment, the torso of the virtual shadow puppet is controlled to perform two-dimensional continuous displacement based on the continuous velocity vector; In response to the effective blink event trigger signal, an instantaneous physical force is applied to the limbs of the virtual shadow puppet based on a multibody dynamics model to drive the limbs to produce a swinging motion.

10. The method according to claim 9, characterized in that, The EBLM-based hybrid decoding model processes the EEG signals to decode a continuous velocity vector used to control the displacement of the virtual shadow puppet, including: Feature extraction is performed on the electroencephalogram (EEG) signal to obtain the original feature vector; K-means clustering is performed on the original feature vector, the Euclidean distance between the original feature vector and each cluster center is calculated, and the result is converted into a cluster membership feature vector. The original feature vector is fused with the clustering membership feature vector to form a fused feature vector; The fused feature vector is input into a pre-trained linear support vector machine classifier to obtain discrete motion direction category labels; The discrete motion direction category labels are converted into the continuous velocity vector using a velocity smoothing formula.

Images