Method, device and storage medium for controlling virtual home based on brain-computer interface

Abstract

This invention provides a method, device, and storage medium for controlling a virtual home environment based on a brain-computer interface (BCI). The method includes: loading a virtual scene simulating a home environment; distributing virtual device models of simulated home appliances within the virtual scene; collecting brainwave signals from a user via the BCI; identifying device operation information triggered by the user based on the brainwave signals; generating device operation instructions for the virtual device models based on the device operation information; and executing the device operation instructions to render an operation animation of the virtual device models within the virtual scene. This embodiment provides a low-cost and secure experimental platform for users to practice controlling home appliances via BCI, updating models related to controlling home appliances via BCI, and for developers to develop control logic for home appliances, based on virtual simulation technology.

Description

Technical Field

[0001] This invention belongs to the technical field of brain-computer interfaces, and in particular relates to a method, device and storage medium for controlling virtual home based on a brain-computer interface. Background Technology

[0002] One direction for smart home devices is the integration of brain-computer interface (BCI) technology, which allows users to control home devices through non-invasive brain-computer interfaces (BCI), providing convenience for users (especially those with physical disabilities).

[0003] Typically, an offline, cross-subject approach is used to train a general model to recognize user intentions by using sample data from a group of users controlling home appliances via brain-computer interfaces. Subsequently, the model is updated online using sample data from individual users controlling home appliances via brain-computer interfaces.

[0004] However, due to factors such as user proficiency and model accuracy, users may make mistakes when controlling home appliances via brain-computer interfaces during the online model update process. This can easily lead to damage to home appliances or even safety accidents. After user feedback, developers modify the control logic of the home appliances. This process is repeated until the control logic meets all requirements. This restricts the practical application and industrialization of brain-computer interfaces in home appliances. Summary of the Invention

[0005] In view of this, the present invention provides a method, device and storage medium for controlling virtual home devices based on brain-computer interface, so as to improve the security of users controlling home devices based on brain-computer interface.

[0006] A first aspect of the present invention provides a method for controlling a virtual home based on a brain-computer interface, comprising: A virtual scene simulating a home environment is loaded; virtual device models simulating home appliances are distributed within the virtual scene; Use a brain-computer interface to collect brainwave signals from the user; The device operation information triggered by the user is identified based on the electroencephalogram (EEG) signals. Based on the device operation information, generate device operation instructions for the virtual device model; The device operation instructions are executed to render an operation animation of the virtual device model in the virtual scene.

[0007] A second aspect of the present invention provides a device for controlling a virtual home based on a brain-computer interface, comprising: A virtual scene loading module is used to load a virtual scene simulating a home environment; the virtual scene contains virtual device models simulating home appliances. The EEG signal acquisition module is used to call the brain-computer interface to collect EEG signals from the user; The device operation information recognition module is used to recognize the device operation information triggered by the user based on the electroencephalogram (EEG) signal. The device operation instruction generation module is used to generate device operation instructions for the virtual device model based on the device operation information. The device operation instruction execution module is used to execute the device operation instructions to render the operation animation of the virtual device model in the virtual scene.

[0008] A third aspect of the present invention provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method for controlling a virtual home based on a brain-computer interface as described in the first aspect above.

[0009] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method for controlling a virtual home based on a brain-computer interface as described in the first aspect above.

[0010] The fifth aspect of the present invention provides a computer program product that, when run on a computer, causes the computer to perform the brain-computer interface-based method for controlling a virtual home as described in the first aspect above.

[0011] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: In this embodiment, a virtual scene simulating a home environment is loaded; virtual device models simulating home appliances are distributed within the virtual scene; a brain-computer interface (BCI) is invoked to collect the user's electroencephalogram (EEG) signals; device operation information triggered by the user is identified based on the EEG signals; device operation instructions are generated for the virtual device models based on the device operation information; and the device operation instructions are executed to render the operation animation of the virtual device models in the virtual scene. This embodiment provides a low-cost and secure experimental platform based on virtual simulation technology for users to practice controlling home appliances using BCI, updating models related to controlling home appliances using BCI, and for developers to develop control logic for home appliances. It allows for large-scale verification of the control logic of home appliances and rapid deployment when the control logic of home appliances meets various requirements, which helps to improve the practicality and industrialization of BCI in home appliances. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of a method for controlling a virtual home based on a brain-computer interface, provided in an embodiment of the present invention; Figure 2 This is an example diagram of a virtual scene provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of another method for controlling a virtual home based on a brain-computer interface provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of a device for controlling a virtual home based on a brain-computer interface, provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0014] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the present invention. However, those skilled in the art will recognize that the present application may be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted to avoid unnecessary detail that could obscure the description of the present application.

[0015] The technical solution of the present invention will be illustrated below through specific embodiments.

[0016] Reference Figure 1 The diagram illustrates a method for controlling a virtual home based on a brain-computer interface according to an embodiment of the present invention, which may specifically include the following steps: Step 101: Load the virtual scene of the simulated home environment.

[0017] In this embodiment, one or more virtual scenes simulating a real home environment, such as a living room, bedroom, kitchen, and bathroom, can be constructed based on a virtual engine such as Unity.

[0018] One or more virtual device models are distributed in a virtual scene to simulate home appliances that can be set up in the home environment. This provides users with a simulated environment to practice / exercise by controlling home appliances based on brain-computer interfaces, and also provides developers with a simulated environment to develop control logic for home appliances.

[0019] Each virtual device model has a unique device identifier, control logic, and operation animations (with paths) that simulate changes in home appliances.

[0020] For example, such as Figure 2 As shown, the virtual scene of the simulated living room includes: a virtual device model of a simulated television ①, a virtual device model of a simulated air conditioner ②, a virtual device model of a simulated ceiling light ③, a virtual device model of a simulated curtain ④, and so on.

[0021] For example, virtual device model ① (device identifier: TV) supports simulated on / off operation animations, virtual device model ② (device identifier: CUR) supports simulated on / off operation animations, virtual device model ③ (device identifier: LIT) supports simulated on / off operation animations, virtual device model ④ (device identifier: AIR) supports simulated on / off operation animations, and so on.

[0022] Developers can map new attributes in the configuration file to enable dynamic addition, deletion, and modification of virtual device models in the virtual scene without modifying the virtual scene code. This approach offers good compatibility, scalability, and low deployment costs, thus improving development efficiency.

[0023] Step 102: Use the brain-computer interface to collect EEG signals from the user according to the steady-state visual evoked potential paradigm.

[0024] In this embodiment, the user can wear a head-mounted device (such as a headband, helmet, etc.). The head-mounted device is equipped with wired / wireless (such as Bluetooth) transmitters, brain-computer interfaces and other components. It can call the brain-computer interface to collect single-channel or multi-channel EEG signals from the user (represented by data such as ID and account) according to paradigms such as steady-state visual evoked potentials.

[0025] Among them, steady-state visual evoked potentials (SSVEPs) are stable brain electrical oscillations generated by the brain in response to periodic visual flashes at a fixed frequency, which are locked to the frequency / phase of the stimulus.

[0026] When a user focuses on a periodic flashing stimulus (such as a light or a square on a screen) in the 3.5Hz-75Hz range, neurons in the occipital visual cortex are synchronously driven, generating a phase-locked, steady-state potential on the scalp that matches the stimulus frequency (and harmonics). At this time, the brain-computer interface can be invoked to collect multi-channel EEG signals from the user.

[0027] Taking a five-channel EEG signal as an example, a brain-computer interface includes components such as electrodes, a signal processor, and an A / D (Analogue to Digital) converter.

[0028] The electrodes include a reference electrode, a ground electrode, and five signal channels of EEG electrodes. The ground electrode is used to determine the zero potential of the EEG signal. The reference electrode and the ground electrode are placed on the temples on both sides of the user's head. The EEG electrodes are placed in the prefrontal lobe (2 channels), occipital lobe (2 channels), and parietal lobe (1 channel).

[0029] The signal processor is used to amplify the acquired EEG signal through an amplifier at a sampling frequency of 250Hz, and to filter out 50Hz power frequency noise in the EEG signal through a notch filter, and then filter out DC components and high frequency noise through a bandpass filter of 0.1-50Hz.

[0030] The A / D converter uses a 24-bit resolution digital-to-analog converter chip to convert the amplified and filtered EEG signals from analog signals into digital signals.

[0031] At this point, the head-mounted device can transmit brainwave signals to the brain-computer interface platform via wired or wireless means.

[0032] Step 103: Identify device operation information triggered by the user based on EEG signals.

[0033] An operation detection model can be pre-set in the brain-computer interface platform. The operation detection model is used to identify the user's operation intention triggered by home devices (including real home devices and simulated virtual device models) based on EEG signals under the SSVEP paradigm.

[0034] The operation detection model can be a machine learning model, such as Support Vector Machine (SVM), decision tree, random forest, etc., or a deep learning model, such as Convolutional Neural Networks (CNN), Long Short Term Memory (LSTM), Transformer (a neural network architecture based on self-attention mechanism), etc.

[0035] For deep learning models, the structure of device operation information is not limited to artificially designed neural networks, such as EEGNet (electroencephalography model). It can also be a neural network optimized by model quantization methods, a neural network searched by NAS (Neural Architecture Search) methods based on the characteristics of user operation of home devices, and so on. This embodiment does not impose any restrictions on this.

[0036] Based on the current EEG signal, the EEG signal can be divided into multiple segments, and the EEG signal of each segment can be input into the operation detection model to identify (classify) the user's operation intention triggered by home devices.

[0037] At this point, the operation intent can be mapped to standardized device operation information according to the preset mapping relationship in the configuration file. The device operation information includes parameters such as the device identifier and operation action of the virtual device model.

[0038] For example, if the user's intention is to "turn on the TV", it will be mapped to the device operation information "TV" (device identifier) ​​and "on" (operation action). If the user's intention is to "close the curtains", the user's intention will be "CUR" (device identifier) ​​and "off" (operation action), and so on.

[0039] Furthermore, the brain-computer interface platform can encapsulate the function of recognizing user-triggered device operation information into an API (Application Programming Interface) for developers to call directly, allowing developers to focus on business and improve development efficiency.

[0040] Step 104: Generate device operation instructions for the virtual device model based on the device operation information.

[0041] In this embodiment, standardized device operation instructions are generated for the virtual device model using device operation information according to a preset interaction protocol.

[0042] In general, device operation commands are structured data in formats such as JOSN (JavaScript Object Notation), and are sent to the virtual engine via protocols such as Socket.

[0043] In addition to device operation information, other parameters can be encapsulated into the device operation instructions, such as attention value, attention level, timestamp, etc. This embodiment does not limit this.

[0044] In one method of calculating attention values, an attention detection model can be pre-configured in the brain-computer interface platform. The attention detection model is used to identify the user's attention value based on EEG signals, representing the user's level of attention concentration.

[0045] The attention detection model can be a machine learning model, such as SVM, decision tree, random forest, etc., or a deep learning model, such as CNN, LSTM, Transformer, etc.

[0046] For deep learning models, the structure of device operation information is not limited to manually designed neural networks, such as EEGNet. It can also be a neural network optimized by model quantization methods, a neural network searched for the characteristics of user attention by NAS methods, and so on. This embodiment does not impose any restrictions on this.

[0047] Based on the current EEG signals, the EEG signals of each segment can be input into the attention detection model to detect the user's attention value.

[0048] In another way of calculating attention value, the attention value can be quantified based on the time-domain energy value of the EEG signal in the prefrontal cortex channel and the proportion of frequency-domain alpha waves.

[0049] The user's attention value is mapped to an attention level by means of piecewise functions, and the attention value and attention level are encapsulated into the device operation instructions.

[0050] For example, if the attention value is ≥50, the attention level is high; if 40≤attention<50, the attention level is medium; if the attention value is <40, the attention level is low, and so on.

[0051] Furthermore, the brain-computer interface platform can encapsulate the function of generating device operation instructions into an API (Application Programming Interface) for developers to call directly, allowing developers to focus on business and improve development efficiency.

[0052] Step 105: Execute device operation commands to render operation animations of the virtual device model in the virtual scene.

[0053] An actuator is set up in the virtual engine to execute device operation instructions and render operation animations of virtual device models in the virtual scene. The operation animations include simulated audio data and / or simulated video data, thereby simulating the effect of operating home appliances.

[0054] In a specific implementation, if the pre-loaded resource mode is applied in this embodiment, it can be determined whether the device operation information is compatible with the pre-loaded operation animation of the virtual device model.

[0055] If so, execute the device operation instructions to render the pre-loaded operation animation of the virtual device model in the virtual scene.

[0056] If not, release the pre-loaded operation animation of the virtual device model, execute the device operation command, reload the operation animation of the virtual device model that is adapted to the device operation information according to the path and other information, and render the reloaded operation animation of the virtual device model in the virtual scene.

[0057] In this embodiment, a virtual scene simulating a home environment is loaded; virtual device models simulating home appliances are distributed within the virtual scene; a brain-computer interface (BCI) is invoked to collect the user's electroencephalogram (EEG) signals; device operation information triggered by the user is identified based on the EEG signals; device operation instructions are generated for the virtual device models based on the device operation information; and the device operation instructions are executed to render the operation animation of the virtual device models in the virtual scene. This embodiment provides a low-cost and secure experimental platform based on virtual simulation technology for users to practice controlling home appliances using BCI, updating models related to controlling home appliances using BCI, and for developers to develop control logic for home appliances. It allows for large-scale verification of the control logic of home appliances and rapid deployment when the control logic of home appliances meets various requirements, which helps to improve the practicality and industrialization of BCI in home appliances.

[0058] Reference Figure 3 The diagram illustrates another method for controlling a virtual home based on a brain-computer interface, as provided in an embodiment of the present invention. Specifically, it may include the following steps: Step 301: Load the virtual scene of the simulated home environment.

[0059] Among them, virtual device models simulating home appliances are distributed in the virtual scene.

[0060] Step 302: Use the brain-computer interface to collect EEG signals from the user.

[0061] Step 303: Identify device operation information triggered by the user based on EEG signals.

[0062] Step 304: Generate equipment operation instructions for the virtual equipment model based on the equipment operation information.

[0063] Step 305: Read the user's attention value represented by the EEG signal in the device operation instructions.

[0064] If multiple virtual device models of simulated home appliances are distributed in a virtual scene, and the virtual engine's executor adopts a passive response mode of single device operation command - single rendering action, the operation animation of the virtual device model is reloaded every time the device operation command is executed. The delay of continuous operation of multiple virtual device models is linearly superimposed, which has a certain impact on the smoothness and real-time performance of the interaction.

[0065] In this embodiment, a pre-loaded resource mode is applied, that is, the operation animations of virtual devices that the user may perform in subsequent potential operations are pre-loaded, reducing the linear superposition of latency during continuous operation of multiple virtual device models and improving the smoothness and real-time performance of the interaction.

[0066] In the executor of the virtual engine, data of the specified first field can be read from the device operation instructions according to the preset interaction protocol to obtain the user's attention value represented by the current EEG signal.

[0067] Step 306: Calculate the original confidence level of the user's historical operation information for triggering other device operation information after the current device operation information.

[0068] In this embodiment, the combinations between current device operation information and other device operation information can be traversed, and the original confidence level of triggering other device operation information after the current device operation information is calculated for the user (represented by data such as ID and account). Based on the user's identification of the correlation between the current device operation information and other device operation information, the operation habit of continuously triggering other device operation information after the current device operation information can be characterized.

[0069] For example, the statistical history of a user (represented by data such as ID or account) triggers the operation information of other devices at a first frequency after the current device operation information, and the statistical history of a user (represented by data such as ID or account) triggers the current device operation information at a second frequency.

[0070] For the same operation information of other devices, calculate the ratio between the first frequency and the second frequency, and use it as the original confidence level of the user (represented by data such as ID and account) in triggering operation information of other devices after the current operation information.

[0071] In this example, let A represent the current device operation information, and B represent other device operation information besides A. Then, the initial confidence level C for a user's history triggering device operation information B after device operation information A is... int It can be represented as: C int = P A-B / P A ×100%, where P A-B P represents the first frequency at which device operation information B is triggered after device operation information A in the user's history. A This is the second frequency of historical trigger device operation information A.

[0072] Step 307: In the virtual scene, generate the original correction coefficients for triggering operation information on other devices after the user's operation information on the current device, based on the attention value.

[0073] In general, users tend to focus their attention on triggering certain device operation information. Therefore, in a virtual environment, at least based on the attention value of the current device operation information, the original correction coefficient for the user (represented by data such as ID and account) to trigger other device operation information after the current device operation information can be generated. This is used to reflect the stability and effectiveness of the user's (represented by data such as ID and account) inherent attention in a specific virtual scenario.

[0074] In one embodiment of the present invention, step 307 may include the following steps: Step 3071: Use the user history to trigger the attention value of other device operation information after the current device operation information to generate the first corrected weight.

[0075] At the user level, the first corrected weight can be generated using the attention value of the user's (represented by data such as ID and account) historical actions triggered after the current device's actions.

[0076] For example, in scenarios where a user's history triggers other device operation information after the current device operation information, statistical values ​​(such as average, maximum, minimum, etc.) can be generated for the attention value corresponding to the current device operation information and the attention value corresponding to other device operation information, and used as the attention value when the user's history triggers other device operation information after the current device operation information. Alternatively, the attention value corresponding to the current device operation information or the attention value corresponding to other device operation information can be directly selected as the attention value when the user's history triggers other device operation information after the current device operation information.

[0077] Count the number of attention values ​​a user has historically generated when triggering other device operation information after the current device operation information.

[0078] If the number of attention values ​​triggered by a user's history when triggering other device operation information after the current device operation information is greater than or equal to the preset sample size (e.g., 3), then the user's history of attention values ​​triggered by triggering other device operation information after the current device operation information is determined to be valid.

[0079] If the number of attention values ​​for users triggering other device operation information after the current device operation information is less than the preset sample size, then a specified value (such as 0.5) is filled in as the attention value for users triggering other device operation information after the current device operation information, until the number of attention values ​​for users triggering other device operation information after the current device operation information is greater than or equal to the preset sample size.

[0080] The average and standard deviation of the attention values ​​of user history triggered by other device operation information after the current device operation information are calculated respectively. Among them, for the user history triggered by other device operation information after the current device operation information, the average value can be normalized to a specified first range (such as [0, 1]) and the standard deviation can be normalized to a specified second range (such as [0, 1]) using algorithms such as Max-Min.

[0081] The product of the average value, the stability coefficient, and the preset first benchmark weight is calculated as the first correction weight; where the stability coefficient is the difference between 1 and the standard deviation, which can characterize the stability and concentration of the user's historical attention value in triggering other device operation information after the current device operation information.

[0082] In this example, the first corrected weight W his It can be represented as: W his =S×(1-σ)×k1, where S is the mean, σ is the standard deviation, and k1 is the first benchmark weight (e.g., 0.4).

[0083] Step 3072: Generate a second correction weight using the correlation between the current device operation information and other device operation information in the virtual scene.

[0084] In the dimension of virtual scene, a second correction weight can be generated by the correlation between the current device operation information and other device operation information in the current virtual scene, which reflects the degree of continuous correlation between the state changes of the current device operation information and other device operation information in the current virtual scene.

[0085] For example, a first scene weight is determined for configuring the current device operation information and a second scene weight is determined for configuring the operation information of other devices in the current virtual scene.

[0086] The independent coefficients are calculated using the negative of the product between the weights of the first scene and the weights of the second scene as the exponent and a natural number as the base.

[0087] The product of the correlation coefficient and the preset second benchmark weight is calculated as the second correction weight; wherein, the correlation coefficient is the difference between 1 and the independence coefficient, which is used to quantify the correlation between the current device operation information and other device operation information.

[0088] In this example, let A represent the current device operation information, and B represent the operation information of other devices besides A. Then the second correction weight W... sce It can be represented as: W sce =(1-e -(wA×WB))×k2, where wA is the first scene weight of device operation information A, wB is the first scene weight of device operation information B, and k2 is the second baseline weight (e.g., 0.3).

[0089] Step 3073: Generate the third corrected weight using the current attention value.

[0090] In terms of device operation information, a third correction weight can be generated using the attention value corresponding to the current device operation information. This is a real-time trigger condition for confidence correction and reflects the user's current operation status.

[0091] For example, the product between the attention value corresponding to the current device operation information and the preset third baseline weight is calculated as the third correction weight.

[0092] In this embodiment, the third correction weight K curr It can be represented as K curr = (A curr / 100)×k3 where, A curr k3 is the attention value corresponding to the current device operation information, and k3 is the third baseline weight (e.g., 0.3).

[0093] Step 3074: Linearly merge the first correction weight, the second correction weight, and the third correction weight into the original correction coefficient for triggering other device operation information after the user's current device operation information.

[0094] In this embodiment, the first correction weight, the second correction weight, and the third correction weight are linearly fused to obtain the original correction coefficient for when the user triggers other device operation information after the current device operation information.

[0095] For example, the original correction coefficient K for when a user triggers other device operation information after the current device operation information. total It can be represented as: K total =W his +W sce +K curr .

[0096] This embodiment comprehensively generates the original correction coefficients for triggering other device operation information after the user's current device operation information from dimensions such as user, virtual scene, and device operation information. It adapts to the user's real-time physiological state and home operation scene patterns, effectively improving the accuracy of subsequent target confidence, accurately preloading the operation animation of virtual device models, reducing resource waste, reducing the response latency of a single device operation command from more than 0.4s to less than 0.25s, and reducing the total latency of continuous operation of multiple virtual device models by about 50%, greatly improving the smoothness and real-time performance of the interaction.

[0097] In this context, it can automatically adapt to the attentional physiological habits and home operation scenarios of different users, providing each user with personalized predictive services that fit their operating habits, physiological state, and home environment.

[0098] Step 308: For the same operation information of other devices, correct the original confidence level to the target confidence level based on the original correction coefficient.

[0099] In this embodiment, the operation information of each device other than the current device operation information can be traversed. For the same other device operation information, the original confidence level is corrected using the original correction coefficient to obtain the target confidence level. This allows the target confidence level to form a strong intrinsic correlation with the original correction coefficient, effectively filtering out random operation interference when the user's attention is scattered.

[0100] In practical implementation, for the same operation information of other devices, the original correction coefficient K is... total Compare with a preset correction threshold (e.g., 0.2).

[0101] If the original correction coefficient K total Less than the correction threshold (e.g., K) total If the confidence level is less than 0.2, then the original confidence level C is ignored. int Set the target confidence level to C final It is 0.

[0102] If the original correction coefficient K total Greater than or equal to the correction threshold (e.g., K) total If the value is ≥0.2), then the original correction factor K is used. total The target correction coefficient is generated exponentially using a preset correction factor λ as the base; where the correction factor λ is greater than 0 and less than or equal to less than 1, that is, λ∈(0,1], such as 0.5, and is used to adjust the original correction coefficient K during the nonlinear correction process. total Balance retention and filtering.

[0103] Calculate the original confidence level C int The product of the target correction factor and the target confidence level C is used as the target confidence level C. final .

[0104] Then, the target confidence level C final It can be represented as: C final =C int ×K total λ ×100%.

[0105] Generally, it can be based on the target confidence level C. final Sort the operation information of other devices from largest to smallest, and retain the operation information of other devices in the first n positions (n ​​is a positive integer) of the sorted information to reduce the amount of computation.

[0106] Step 309: Based on the original correction coefficients and target confidence level, preload the operation animation of the virtual device model adapted to the operation information of other devices.

[0107] In this embodiment, by comprehensively measuring the original correction coefficient and the target confidence level, the operation animation of the virtual device model, which is adapted to the operation information of other devices that the user may trigger next, is preloaded using information such as the path.

[0108] In one scenario, if the target confidence level C final Greater than or equal to a preset first confidence threshold (e.g., C) final If the percentage is ≥80%, then the operation animation of the virtual device model adapted to other device operation information is preloaded. The preloaded operation animation of the virtual device model is configured with a first effective time according to the original correction coefficient. The first effective time is the duration for which the preloaded operation animation of the virtual device model is retained, and it is proportional to the original correction coefficient K. total Positive correlation, such as first effective time = 15s × K total .

[0109] Additionally, it generates prompts for the pre-loaded operation animations of the virtual device model. Generally, the prompts do not conflict with the stimulation frequency band of SSVEP, such as a flashing frequency of 2Hz and a transparency of 30%.

[0110] In another case, if the target confidence level C final Greater than or equal to the preset second confidence threshold, and less than the preset first confidence threshold (e.g., 60% ≤ C) final If the percentage is less than 80%, then the operation animation of the virtual device model adapted to other device operation information is preloaded. A second effective time is configured for the preloaded operation animation of the virtual device model according to the original correction coefficient. The second effective time is the duration for which the preloaded operation animation of the virtual device model is retained, and it is proportional to the original correction coefficient K. total There is a positive correlation, and the first effective time is greater than the second effective time, such as the second effective time = 10s × K. total .

[0111] In another case, if the target confidence level C final Less than the preset second confidence threshold (e.g., C) final If the percentage is less than 60%, then preloading the operation animation of the virtual device model that is adapted to the operation information of other devices is prohibited.

[0112] In addition, the attention level mapped by the attention value can be obtained by reading the data of the specified second field in the device operation instructions according to the interaction protocol.

[0113] If at least one of the following—attention level, the duration between triggering the current device operation information and triggering the previous device operation information—satisfies a preset continuity condition, such as the attention level being medium / high, or the duration between triggering the current device operation information and triggering the previous device operation information being ≤10s, or the original correction coefficient K… total If the value is ≥0.4, then preloading the operation animation of the virtual device model is allowed, that is, steps 305-309 are allowed to be executed.

[0114] If the attention level, the duration of the attention level, and the original correction coefficient all meet the preset termination conditions, such as the attention level being low, the duration of the attention level being ≥5s, and the original correction coefficient K being... total If the value is less than 0.2, then stop preloading the operation animation of the virtual device model, that is, stop executing steps 305-309. At this time, the operation animation that has been preloaded into the virtual device model can be released.

[0115] If at least one of the changes in the attention value or the decrease in the original correction coefficient meets the preset abnormal conditions, such as the change in the attention value ≥ 30, or the decrease in the original correction coefficient ≥ 0.3, then the operation animation of the preloaded virtual device model is stopped, that is, steps 305-309 are stopped.

[0116] If the attention level, the duration of the attention level, and the original correction coefficient all meet the preset normal conditions, such as the attention level being high, the duration of the attention level being ≥2s, and the original correction coefficient K... total If the value is ≥0.3, then the operation animation of the preloaded virtual device model will be restored, that is, the execution steps 305-309 will be restored.

[0117] This embodiment uses a preloading start-stop mechanism based on multi-dimensional attention features to avoid meaningless predictions and resource consumption under low attention and low feature matching conditions. The peak memory usage of the executor is reduced by 35%, and the dynamic retention time strategy of preloaded resources further improves resource utilization and ensures stable system operation.

[0118] In this embodiment, the example is taken as a user completing a series of brain-controlled operations in a virtual living room (virtual scene), such as turning on the TV, closing the curtains, and dimming the lights.

[0119] S1. Acquiring EEG signals and generating structured instructions The user wears a headband and maintains an attention span of 65 points or higher for 3 consecutive seconds while looking at the SSVEP visual stimulation interface corresponding to the preset "Turn on the TV" command on the brain-computer interface platform. The headband collects the user's EEG signals based on five channels and uploads them to the brain-computer interface platform. The brain-computer interface platform recognizes the device operation information based on the EEG signals and generates device operation instructions in JSON format. The device operation instructions include the device operation information "Turn on the TV", the real-time attention value of 65, the attention level "high", and the collection timestamp "2024-06-10 19:08:00".

[0120] 2. Command Transmission and Parsing The Socket server in the brain-computer interface platform sends device operation instructions to the virtual actuator (i.e., the Socket client) via TCP / IP (Transmission Control Protocol / Internet Protocol). After the protocol verification is successful, it extracts parameters such as device operation information, attention value, and attention level from the device operation instructions and matches the target device as a TV (virtual device model).

[0121] 3. Predictive Start-up Judgment Retrieve historical six-tuple time-series data (timestamp, current device operation information, other device operation information, device state changes, attention value when executing the current device operation information and other device operation information, scene weight coefficients (first baseline weight, second baseline weight, third baseline weight, first scene weight, second scene weight, etc.)) to calculate the first corrected weight W when the user triggers "turn on TV" and then "close curtains". his =0.27.

[0122] 4. Multi-dimensional confidence correction and intent screening The initial confidence level C of the "close curtains" action triggered after the user triggers "turn on the TV". init1 =85%; the original confidence level C of "dimming the lights" triggered after the user triggers "turn on the TV". init2 =78%.

[0123] In the living room, the second modified weight W represents the correlation between "turning on the TV" and "closing the curtains". sce1 =0.13, representing the second adjustment weight W that characterizes the correlation between "turning on the TV" and "dimming the lights". sce2 =0.13.

[0124] The third corrected weight K for "turning on the TV" curr =65 / 100×0.3=0.195.

[0125] The original correction factor K for "closing the curtains" total1 =0.27+0.13+0.195=0.595, the original correction coefficient K for "dimming the lights". total2 =0.25+0.13+0.195=0.575.

[0126] Target confidence level C for "closing the curtains" final1 =85%×0.595≈65.54%, the target confidence level C for "dimming the lights" final2 =78%×0.575≈59.22%.

[0127] 5. Tiered resource preloading The "close the curtains" animation is preloaded, with a resource retention time of 10s × 0.595 ≈ 5.95s; the "dim the lights" animation is not preloaded.

[0128] 6. Continuous Operations and Preloaded Resource Calls After the TV (virtual device model) is turned on, the user's attention is maintained at 70 points, triggering the device operation information "close the curtains". The actuator directly calls the pre-loaded "close the curtains" operation animation, and the response latency of the operation animation is reduced to <0.25s. At the same time, using "close the curtains" as the current device operation information, the target confidence C of other device operation information "dimming the lights" is calculated. final =74.5%, preload the "dim the lights" operation animation, resource retention time = 10s × 0.66 = 6.6s.

[0129] 7. Subsequent Operations and Resource Requests When a user triggers the device operation message "dim the lights", the actuator directly calls the pre-loaded "dim the lights" operation animation to achieve smooth execution of continuous operations.

[0130] 8. Time series data update After all animations are completed, update the device status of the TV, curtains, and ceiling light, and record the six-tuple time-series data of this operation; incrementally update historical attention values ​​and association rules, and maintain the attributes of strong association rules.

[0131] 9. Prediction terminated After the user completes the operation, their attention score remains at 38 points for 5 consecutive seconds, and the original correction coefficient K... total =0.18, terminate prediction, release all preloaded resources; the executor continues to monitor subsequent EEG information, and restarts the prediction process after the prediction start conditions are met.

[0132] It should be noted that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0133] Reference Figure 4 The diagram illustrates a device for controlling a virtual home based on a brain-computer interface, according to an embodiment of the present invention. Specifically, it may include the following modules: The virtual scene loading module 401 is used to load a virtual scene simulating a home environment; the virtual scene contains virtual device models simulating home appliances. The EEG signal acquisition module 402 is used to call the brain-computer interface to acquire EEG signals from the user. The device operation information recognition module 403 is used to recognize the device operation information triggered by the user based on the electroencephalogram (EEG) signal. The device operation instruction generation module 404 is used to generate device operation instructions for the virtual device model based on the device operation information. The device operation instruction execution module 405 is used to execute the device operation instructions to render the operation animation of the virtual device model in the virtual scene.

[0134] In one embodiment of the present invention, the device operation instruction execution module 405 is further configured to: Determine whether the device operation information matches the pre-loaded operation animation of the virtual device model; If so, the device operation instructions are executed to render the pre-loaded operation animation of the virtual device model in the virtual scene; If not, release the pre-loaded operation animation of the virtual device model, execute the device operation command, reload the operation animation of the virtual device model adapted to the device operation information, and render the reloaded operation animation of the virtual device model in the virtual scene.

[0135] In one embodiment of the present invention, the device further includes: An attention value reading module is used to read the user's attention value represented by the electroencephalogram (EEG) signal in the device operation instructions; The original confidence calculation module is used to calculate the original confidence of the user's history of triggering other device operation information after the current device operation information; The original correction coefficient calculation module is used to generate, based on the attention value, the original correction coefficient for the user to trigger other device operation information after the current device operation information in the virtual scene; The target confidence correction module is used to correct the original confidence level to a target confidence level based on the original correction coefficient for the same other device operation information. An operation animation preloading module is used to preload the operation animation of the virtual device model that is adapted to other device operation information based on the original correction coefficient and the target confidence level.

[0136] In one embodiment of the present invention, the original confidence calculation module is further configured to: The first frequency at which the user's historical statistics trigger other device operation information after the current device operation information; The user's statistical history triggers the second frequency of the current device operation information; For the same other device operation information, the ratio between the first frequency and the second frequency is calculated as the original confidence level of the user history in triggering other device operation information after the current device operation information.

[0137] In one embodiment of the present invention, the original correction coefficient calculation module is further configured to: The user history is used to generate a first corrected weight by triggering the attention value of other device operation information after the current device operation information using the current device operation information; A second correction weight is generated using the correlation between the current device operation information and other device operation information in the virtual scenario; A third corrected weight is generated using the attention value described above; The first correction weight, the second correction weight, and the third correction weight are linearly fused together to form the original correction coefficient for the user to trigger other device operation information after the current device operation information.

[0138] In one embodiment of the present invention, the original correction coefficient calculation module is further configured to: The step of using the user history to trigger the attention value of other device operation information after the current device operation information to generate a first corrected weight includes: Calculate the average and standard deviation of the attention values ​​of the user's history that trigger other device operation information after the current device operation information; The product of the average value, the stability coefficient, and the preset first benchmark weight is calculated as the first correction weight; wherein the stability coefficient is the difference between 1 and the standard deviation.

[0139] In one embodiment of the present invention, the original correction coefficient calculation module is further configured to: Determine the first scene weight configured for the current device operation information and the second scene weight configured for other device operation information in the virtual scene; The independent coefficients are calculated using the negative of the product between the weights of the first scene and the weights of the second scene as the exponent and a natural number as the base. The product of the correlation coefficient and the preset second benchmark weight is calculated as the second correction weight; wherein the correlation coefficient is the difference between 1 and the independence coefficient.

[0140] In one embodiment of the present invention, the original correction coefficient calculation module is further configured to: The product of the current attention value and the preset third baseline weight is calculated as the third correction weight.

[0141] In one embodiment of the present invention, the target confidence correction module is further configured to: For the same other device operation information, the original correction coefficient is compared with a preset correction threshold; If the original correction coefficient is less than the correction threshold, then the original confidence level is ignored and the target confidence level is set to 0. If the original correction coefficient is greater than or equal to the correction threshold, then a target correction coefficient is generated with the original correction coefficient as the base and a preset correction factor as the exponent; wherein the correction factor is greater than 0 and less than or equal to 1; The product of the original confidence level and the target correction coefficient is calculated as the target confidence level.

[0142] In one embodiment of the present invention, the operation animation preloading module is further configured to: If the target confidence level is greater than or equal to a preset first confidence threshold, then the operation animation of the virtual device model adapted to other device operation information is preloaded, a first effective time is configured for the preloaded operation animation of the virtual device model according to the original correction coefficient, and a prompt message is generated for the preloaded operation animation of the virtual device model; the first effective time is positively correlated with the original correction coefficient. If the target confidence level is greater than or equal to a preset second confidence threshold and less than a preset first confidence threshold, then the operation animation of the virtual device model adapted to other device operation information is preloaded, and a second effective time is configured for the preloaded operation animation of the virtual device model according to the original correction coefficient; the second effective time is positively correlated with the original correction coefficient; the first effective time is greater than the second effective time; If the target confidence level is less than a preset second confidence threshold, then preloading the operation animation of the virtual device model that is adapted to other device operation information is prohibited.

[0143] In one embodiment of the present invention, the device further includes: An attention level reading module is used to read the attention level mapped by the attention value from the device operation instructions; The preloading permission module is configured to allow the preloading of the operation animation of the virtual device model if at least one of the attention level, the duration of the interval between triggering the current device operation information and triggering the previous device operation information, and the original correction coefficient satisfies a preset continuity condition. The first preloading stop module is used to stop preloading the operation animation of the virtual device model if the attention level, the duration of the attention level, and the original correction coefficient all meet the preset termination conditions. The second preloading stop module is used to stop preloading the operation animation of the virtual device model if at least one of the change range of the attention value and the decrease range of the original correction coefficient meets a preset abnormal condition. The preload recovery module is used to restore the operation animation of the preloaded virtual device model if the attention level, the duration of the attention level, and the original correction coefficient all meet preset normal conditions.

[0144] This invention provides a device for controlling virtual home based on brain-computer interface. By using this device, the steps in the aforementioned methods for controlling virtual home based on brain-computer interface can be implemented.

[0145] It should be noted that the module division in the various brain-computer interface-based virtual home control devices provided in the above embodiments is illustrative and only represents one logical functional division. In actual implementation, other division methods may also be used. Furthermore, the functional modules in the various embodiments of this invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0146] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of the embodiments of the present invention can be embodied in the form of a computer program product, which is stored in a computer storage medium and includes several instructions to cause an electronic device or processor to execute all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned computer storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0147] Furthermore, the device for controlling virtual home based on brain-computer interface provided in the above embodiments and the method embodiment for controlling virtual home based on brain-computer interface belong to the same concept. For details of their specific implementation process, please refer to the method embodiment, which will not be repeated here.

[0148] Reference Figure 5 The diagram illustrates an electronic device according to an embodiment of the present invention. Figure 5 As shown, the electronic device in this embodiment of the invention includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the above-described method embodiment for controlling a virtual home based on a brain-computer interface. Alternatively, when the processor executes the computer program, it implements the functions of each module in the above-described device embodiment for controlling a virtual home based on a brain-computer interface.

[0149] For example, the computer program may be divided into one or more modules, which are stored in the memory and executed by the processor to complete this application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which can be used to describe the execution process of the computer program in the electronic device.

[0150] The electronic device may be a desktop computer, a cloud server, or other computing device. The electronic device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 5 This is merely one example of an electronic device and does not constitute a limitation on the electronic device. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the electronic device may also include input / output devices, network access devices, buses, etc.

[0151] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0152] The memory can be an internal storage unit of the electronic device, such as a hard drive or RAM. Alternatively, it can be an external storage device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc. Furthermore, the memory can include both internal and external storage units. The memory is used to store the computer program and other programs and data required by the electronic device. The memory can also be used to temporarily store data that has been output or will be output.

[0153] This invention also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the brain-computer interface-based method for controlling a virtual home as described in the foregoing embodiments.

[0154] This invention also discloses a computer-readable storage medium storing a computer program that, when executed by a processor, implements the brain-computer interface-based method for controlling a virtual home as described in the foregoing embodiments.

[0155] This invention also discloses a computer program product that, when run on a computer, causes the computer to execute the brain-computer interface-based virtual home control method described in the foregoing embodiments.

[0156] The embodiments described above are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for controlling virtual home furnishings based on a brain-computer interface, characterized in that, include: A virtual scene simulating a home environment is loaded; virtual device models simulating home appliances are distributed within the virtual scene; Use a brain-computer interface to collect brainwave signals from the user; The device operation information triggered by the user is identified based on the electroencephalogram (EEG) signals. Based on the device operation information, generate device operation instructions for the virtual device model; The device operation instructions are executed to render an operation animation of the virtual device model in the virtual scene.

2. The method according to claim 1, characterized in that, The execution of the device operation instructions to render the operation animation of the virtual device model in the virtual scene includes: Determine whether the device operation information matches the pre-loaded operation animation of the virtual device model; If so, the device operation instructions are executed to render the pre-loaded operation animation of the virtual device model in the virtual scene; If not, release the pre-loaded operation animation of the virtual device model, execute the device operation command, reload the operation animation of the virtual device model adapted to the device operation information, and render the reloaded operation animation of the virtual device model in the virtual scene.

3. The method according to claim 1 or 2, characterized in that, The method further includes: The user's attention value, represented by the electroencephalogram (EEG) signal, is read from the device operation instructions; The original confidence level of the user's calculation history in triggering other device operation information after the current device operation information; In the virtual scene, based on the attention value, an original correction coefficient is generated for the user to trigger other device operation information after the current device operation information. For the same other device operation information, the original confidence level is corrected to the target confidence level based on the original correction coefficient; Based on the original correction coefficients and the target confidence level, the operation animation of the virtual device model, adapted to other device operation information, is preloaded.

4. The method according to claim 3, characterized in that, The step of generating the original correction coefficient for the user triggering other device operation information after the current device operation information in the virtual scene based on the attention value includes: The user history is used to generate a first corrected weight by triggering the attention value of other device operation information after the current device operation information using the current device operation information; A second correction weight is generated using the correlation between the current device operation information and other device operation information in the virtual scenario; A third corrected weight is generated using the attention value described above; The first correction weight, the second correction weight, and the third correction weight are linearly fused together to form the original correction coefficient for the user to trigger other device operation information after the current device operation information.

5. The method according to claim 4, characterized in that, The step of using the user history to trigger the attention value of other device operation information after the current device operation information to generate a first corrected weight includes: Calculate the average and standard deviation of the attention values ​​of the user's history that trigger other device operation information after the current device operation information; The product of the average value, the stability coefficient, and the preset first benchmark weight is calculated as the first correction weight; wherein, the stability coefficient is the difference between 1 and the standard deviation; The step of generating a second correction weight using the correlation between the current device operation information and other device operation information in the virtual scene includes: Determine the first scene weight configured for the current device operation information and the second scene weight configured for other device operation information in the virtual scene; The independent coefficients are calculated using the negative of the product between the weights of the first scene and the weights of the second scene as the exponent and a natural number as the base. The product of the correlation coefficient and the preset second benchmark weight is calculated as the second correction weight; wherein the correlation coefficient is the difference between 1 and the independence coefficient; The step of generating a third corrected weight using the current attention value includes: The product of the current attention value and the preset third baseline weight is calculated as the third correction weight.

6. The method according to claim 3, characterized in that, The step of correcting the original confidence level to the target confidence level based on the original correction coefficient for the same other device operation information includes: For the same other device operation information, the original correction coefficient is compared with a preset correction threshold; If the original correction coefficient is less than the correction threshold, then the original confidence level is ignored and the target confidence level is set to 0. If the original correction coefficient is greater than or equal to the correction threshold, then a target correction coefficient is generated with the original correction coefficient as the base and a preset correction factor as the exponent; wherein the correction factor is greater than 0 and less than or equal to 1; The product of the original confidence level and the target correction coefficient is calculated as the target confidence level.

7. The method according to claim 3, characterized in that, The operation animation of the virtual device model, preloaded based on the original correction coefficients and the target confidence level and adapted to other device operation information, includes: If the target confidence level is greater than or equal to a preset first confidence threshold, then the operation animation of the virtual device model adapted to other device operation information is preloaded, a first effective time is configured for the preloaded operation animation of the virtual device model according to the original correction coefficient, and a prompt message is generated for the preloaded operation animation of the virtual device model; the first effective time is positively correlated with the original correction coefficient. If the target confidence level is greater than or equal to a preset second confidence threshold and less than a preset first confidence threshold, then the operation animation of the virtual device model adapted to other device operation information is preloaded, and a second effective time is configured for the preloaded operation animation of the virtual device model according to the original correction coefficient; the second effective time is positively correlated with the original correction coefficient; the first effective time is greater than the second effective time; If the target confidence level is less than a preset second confidence threshold, then preloading the operation animation of the virtual device model that is adapted to other device operation information is prohibited.

8. The method according to claim 3, characterized in that, The original confidence level of the user's calculation history in triggering other device operation information after the current device operation information includes: The first frequency at which the user's historical statistics trigger other device operation information after the current device operation information; The user's statistical history triggers the second frequency of the current device operation information; For the same other device operation information, calculate the ratio between the first frequency and the second frequency, and use it as the original confidence level of the user history in triggering other device operation information after the current device operation information; The method further includes: The attention level mapped by the attention value is read from the device operation instructions; If at least one of the attention level, the duration between triggering the current device operation information and triggering the previous device operation information, and the original correction coefficient satisfies a preset continuity condition, then the operation animation of the virtual device model is allowed to be preloaded. If the attention level, the duration of the attention level, and the original correction coefficient all meet the preset termination conditions, then the preloading of the operation animation of the virtual device model will stop. If at least one of the changes in the attention value and the decrease in the original correction coefficient meets a preset abnormal condition, then the preloading of the operation animation of the virtual device model will stop. If the attention level, the duration of the attention level, and the original correction coefficient all meet the preset normal conditions, then the operation animation of the preloaded virtual device model is restored.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method for controlling a virtual home based on a brain-computer interface as described in any one of claims 1-8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for controlling a virtual home based on a brain-computer interface as described in any one of claims 1-8.

Images