Hybrid intention decoding and man-machine interaction method assisted by smart watch

Abstract

The invention relates to the technical field of intention decoding and man-machine interaction, in particular to a smartwatch-assisted hybrid intention decoding and man-machine interaction method, which comprises the following steps of: acquiring interaction characteristic data acquired by a smartwatch sensor and electroencephalogram signal data acquired by an electroencephalogram signal acquisition device; based on the interaction feature data, executing coarse-grained action recognition; based on the electroencephalogram signal data, executing fine-grained intention analysis; generating a mixed intention decoding result according to the coarse-grained action recognition result and the fine-grained intention analysis result; according to the method, the action data collected by the intelligent watch sensor and the electroencephalogram data obtained by the special electroencephalogram signal collecting device are utilized at the same time, and organic combination of multi-modal information is achieved. The fusion gives full play to the advantages of different modal data, the action data provides visual behavior information, and the electroencephalogram data can capture deeper intention information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intention decoding and human-computer interaction, and more particularly, to a hybrid intention decoding and human-computer interaction method assisted by a smart watch. BACKGROUND

[0002] With the rapid development of smart wearable devices, smart watches as portable human-computer interaction devices are becoming increasingly popular. However, due to the limited screen size of smart watches, traditional touch-based interaction methods are not convenient and natural in some scenarios. Therefore, the industry has been exploring more intuitive and efficient human-computer interaction methods to improve user experience.

[0003] In recent years, intention recognition technology based on bioelectric signals has attracted widespread attention. These technologies mainly use electroencephalogram (EEG), electromyogram (EMG) and other bioelectric signals to analyze user's intentions, thereby realizing touchless human-computer interaction. However, existing intention recognition methods mainly have the following problems:

[0004] Firstly, single-modal intention recognition methods are difficult to meet the needs of complex scenarios. For example, methods relying only on EEG signals can capture user's thinking activities, but are easily disturbed by external interference, and the signal collection device is usually bulky. Methods based only on EMG or motion sensors may have difficulty in recognizing fine intention differences.

[0005] Secondly, existing multi-modal fusion methods often use simple feature concatenation or decision layer fusion, and fail to fully utilize the complementarity and synergy between different modal signals. This leads to limited recognition effect after fusion, making it difficult to cope with complex and variable actual use scenarios.

[0006] In addition, most existing methods lack adaptive learning ability and cannot be personalized optimized according to user's usage habits and preferences. This makes it difficult to guarantee the long-term use experience of the system, especially when it comes to different user groups.

[0007] Finally, existing intention recognition systems usually concentrate all computing tasks on terminal devices or cloud servers, which makes it difficult to balance real-time and complex computing ability at the same time, limiting the overall performance and scalability of the system.

[0008] In view of the above problems, there is an urgent need for a human-computer interaction method that can comprehensively utilize multi-modal data, realize accurate intention decoding, have adaptive learning ability, and efficiently coordinate terminal and cloud resources. Such a method should be able to be implemented on small wearable devices such as smart watches to meet the interaction needs of users in various scenarios. SUMMARY

[0009] The application provides a smart watch assisted hybrid intention decoding and human-computer interaction method, aiming at solving the above-mentioned problems existing in the prior art. The method realizes accurate decoding of user intention and natural interaction by fusing smart watch sensor data and electroencephalogram signals and combining advanced machine learning algorithms.

[0010] To solve the above technical problems, the application adopts the following technical solutions:

[0011] The smart watch assisted hybrid intention decoding and human-computer interaction method comprises:

[0012] The acquisition step comprises:

[0013] Acquiring interaction feature data collected by a smart watch sensor and electroencephalogram data collected by an electroencephalogram signal acquisition device;

[0014] The processing step comprises:

[0015] Performing coarse-grained action recognition based on the interaction feature data;

[0016] Performing fine-grained intention analysis based on the electroencephalogram data;

[0017] Generating a hybrid intention decoding result according to the coarse-grained action recognition result and the fine-grained intention analysis result;

[0018] The output step comprises:

[0019] Sending the hybrid intention decoding result to a control module for performing corresponding human-computer interaction operations.

[0020] As a preferred embodiment, the acquisition step specifically comprises:

[0021] Collecting interaction feature data through an acceleration sensor, a gyroscope sensor, a pressure sensor and an electromyography sensor of a smart watch;

[0022] Collecting electroencephalogram data through a wearable brain-computer signal acquisition device.

[0023] As a preferred embodiment, the coarse-grained action recognition comprises:

[0024] Extracting time sequence features by using an LSTM network and an attention mechanism based on the interaction feature data;

[0025] Determining an action category through a Softmax classifier according to the time sequence features.

[0026] As a preferred embodiment, the fine-grained intention analysis comprises:

[0027] Performing independent component analysis on the electroencephalogram data to remove interference signals;

[0028] Based on the interference-removed electroencephalogram signal data, a CNN-LSTM hybrid network is used to extract spatio-temporal features;

[0029] According to the spatio-temporal features, an intent analysis is realized through a multi-layer attention mechanism.

[0030] As a preferred, the step of generating a hybrid intent decoding result comprises:

[0031] Based on the integration strategy generator, the coarse-grained action recognition result and the fine-grained intent analysis result are adaptively weighted and fused;

[0032] According to the fusion result, a final hybrid intent decoding result is determined.

[0033] As a preferred, it further comprises an adaptive interaction mode switching step:

[0034] Based on the smart watch sensor data, a user motion state is detected in real time;

[0035] According to the user motion state, a brain-electrode and muscle-electrode dual-mode interaction mode is adopted in a static state, and a gesture-based simplified interaction mode is switched to in a motion state.

[0036] As a preferred, it further comprises an incremental learning step:

[0037] User interaction data and corresponding intent labels are obtained;

[0038] Based on the user interaction data and intent labels, an incremental training is performed on the hybrid intent decoding model;

[0039] According to the incremental training result, the hybrid intent decoding model parameters are updated.

[0040] As a preferred, the incremental training comprises:

[0041] A forgetting mechanism is used to control the influence of historical data on the model;

[0042] Batch normalization technology is used to improve the stability of incremental learning;

[0043] Based on user features, a grouping training is performed to realize personalized model optimization.

[0044] As a preferred, it further comprises an edge computing and cloud collaborative processing step:

[0045] A lightweight model is deployed on the smart watch to perform real-time action recognition;

[0046] Complex computing tasks are sent to the cloud for fine-grained intent analysis;

[0047] Based on an asynchronous updating mechanism, data interaction and model updating between the smart watch end and the cloud end are coordinated.

[0048] As preferred, the performing of the corresponding human-computer interaction operation comprises:

[0049] According to the mixed intention decoding result, a corresponding interaction mode is selected;

[0050] Based on the selected interaction mode, control instructions or feedback information are generated;

[0051] The control instructions or feedback information are output to the user through a display screen or a vibration module of the smart watch.

[0052] The method of the present application has the following remarkable technical effects:

[0053] 1. Multimodal data fusion: This method simultaneously utilizes action data collected by smart watch sensors and electroencephalogram data obtained by special electroencephalogram acquisition devices, realizing the organic combination of multimodal information. This fusion fully utilizes the advantages of different modal data, with action data providing intuitive behavior information and electroencephalogram data capturing deeper intention information. Through this complementary combination, the accuracy and robustness of intention recognition are greatly improved.

[0054] 2. Hierarchical intention decoding: This method innovatively proposes a hierarchical decoding strategy combining coarse-grained action recognition and fine-grained intention analysis. Coarse-grained recognition quickly captures the basic action intention of the user, while fine-grained analysis further extracts detailed intention information. This hierarchical design not only improves the recognition accuracy, but also adapts to different complexity of interaction scenarios, greatly enhancing the flexibility and applicability of the system.

[0055] 3. Adaptive learning ability: By introducing an incremental learning mechanism, this method can continuously optimize model parameters according to user usage data, realizing personalized intention recognition. This adaptive ability enables the system to continuously improve performance over time, providing users with increasingly accurate and natural interaction experiences.

[0056] 4. Edge computing and cloud collaboration: This method skillfully utilizes the local computing capability of the smart watch and the powerful processing capability of the cloud, performing real-time preliminary recognition on the smart watch end, and handing over complex intention analysis tasks to the cloud. This distributed computing architecture not only guarantees the real-time nature of interaction, but also is not limited by the computing capability of the terminal device, making it possible to accurately analyze complex intentions.

[0057] 5. Scene-adaptive interaction: This method can automatically switch different interaction modes according to the user's motion state. In the static state, it makes full use of EEG signals for fine interaction, while in the motion state, it focuses on stable and reliable motion recognition. This flexible interaction strategy greatly improves the practicality and user-friendliness of the system in various usage scenarios.

[0058] 6. Improve interaction efficiency: Through accurate intention decoding and natural interaction methods, this method significantly reduces the user's need for complex operations. Users can complete various tasks through simple actions or thoughts, greatly improving interaction efficiency, especially suitable for use on devices with limited screen size such as smart watches.

[0059] 7. Expand application scenarios: Thanks to its powerful intention decoding ability and flexible interaction methods, this method can be widely used in smart home control, virtual reality interaction, assisted medical care and other fields. It provides a solid technical foundation for developing more intelligent and intuitive human-computer interaction systems.

[0060] 8. Personalized experience: Through continuous learning and analysis of user data, this method can gradually adapt to each user's unique usage habits and preferences, providing highly personalized interaction experience. This not only improves user satisfaction, but also points the way for the development of more intelligent and humanized interaction systems.

[0061] In summary, the smart watch assisted hybrid intention decoding and human-computer interaction method provided by the present application effectively solves the problems of insufficient accuracy, poor adaptability, and low computational efficiency in existing technologies through multi-modal data fusion, hierarchical intention decoding, adaptive learning and other innovative technologies. This method not only significantly improves the naturalness and efficiency of human-computer interaction, but also opens up new possibilities for future more intelligent and personalized interaction systems, with important theoretical significance and broad application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0062] Figure 1 The method flowchart of the present application.

[0063] Figure 2 The data acquisition flowchart of the present application.

[0064] Figure 3 The coarse-grained motion recognition flowchart of the present application.

[0065] Figure 4 The fine-grained intention analysis flowchart of the present application.

[0066] Figure 5 The hybrid intention decoding flowchart of the present application.

[0067] Figure 6Flowchart of adaptive interaction mode switching of the present application.

[0068] Figure 7 Flowchart of incremental learning of the present application. DETAILED DESCRIPTION

[0069] As Figures 1-7 shown, the present application relates to a smartwatch-assisted hybrid intent decoding and human-computer interaction method. The method collects multi-modal data of the user through a smartwatch and a wearable brain-computer interface device, and realizes accurate decoding of the user's intent and natural interaction by combining advanced machine learning algorithms. The specific embodiments of the present application will be described in detail below.

[0070] Firstly, the method proposed by the present application includes an acquisition step, a processing step and an output step. In the acquisition step, the system acquires interaction feature data collected by a smartwatch sensor and electroencephalogram data collected by an electroencephalogram signal acquisition device. Preferably, the smartwatch can integrate multiple sensors, such as acceleration sensors, gyroscope sensors, pressure sensors and electromyography sensors, etc., to comprehensively capture the user's actions and physiological information. For example, the sampling frequency of the acceleration sensor can be set to 100 Hz, and the sampling frequency of the gyroscope sensor can be set to 20 Hz. Such settings can ensure data accuracy while avoiding excessive data volume. At the same time, the wearable brain-computer signal acquisition device collects the user's electroencephalogram signals, and the sampling frequency is preferably 200 Hz, which is sufficient to capture most meaningful electroencephalogram activities.

[0071] In the processing step, the method of the present application performs coarse-grained action recognition based on the acquired interaction feature data, and performs fine-grained intent analysis based on the electroencephalogram signal data. The coarse-grained action recognition uses LSTM network and attention mechanism to extract time sequence features. This structure can effectively capture the time dependence of action data. Specifically, the structure of the LSTM network can be represented as follows:

[0072] f t =σ(W f ·[h t-1 ,x t ]+b f )

[0073] i t =σ(W i ·[h t-1 ,x t ]+b i )

[0074]

[0075]

[0076] o t= σ(W o · [h t-1 , x t ] + b o )

[0077] h t = o t * tanh(C t )

[0078] where f t , i t , o t are forget gate, input gate and output gate respectively, C t is cell state, h t is hidden state, W and b are weight and bias parameters, and σ is sigmoid activation function.

[0079] The attention mechanism can be represented as:

[0080] α t = softmax(v T tanh(Wh t +b))

[0081] where α is attention weight, and v, W, b are learnable parameters.

[0082] For fine-grained intent parsing, the application adopts a CNN-LSTM hybrid network to extract the spatio-temporal features of the EEG signal. The CNN part is used to extract spatial features, and the LSTM part is used to capture time dependence. The multi-layer attention mechanism further enhances the model's ability to identify key information.

[0083] In an embodiment of the application, the convolutional layer of the CNN can be represented as:

[0084]

[0085] where y is the output of the (i, j) position of the l-th layer, is the convolution kernel weight, is the input of the previous layer, b l is the bias, and f is the activation function.

[0086] Next, the method of the application generates a hybrid intent decoding result according to the coarse-grained action recognition result and the fine-grained intent parsing result. This step adopts an integration strategy generator to combine the recognition results of the two levels through adaptive weighted fusion. The fusion strategy can be represented as:

[0087] I = αR c + (1-α)R f

[0088] where I is the final mixed-intent decoding result, R c is the coarse-grained recognition result, R f is the fine-grained parsing result, and a is the adaptive weight, which can be dynamically adjusted according to the current scenario and recognition confidence.

[0089] In the output step, the method of the present application sends the mixed-intent decoding result to the control module for performing the corresponding human-computer interaction operation. This can include but is not limited to controlling smart home devices, responding to voice commands, or interacting in a virtual / augmented reality environment, etc.

[0090] The method of the present application also includes an adaptive interaction mode switching function. The system will detect the user's motion state in real time and select the appropriate interaction mode according to the state. For example, when it is detected that the user is in a stationary state (judged by acceleration sensor data, if the acceleration change is less than 0.1 m / s 2 over a continuous 1-second period, the system will enable the dual-mode interaction mode of electroencephalogram and electromyogram to obtain more rich intent information. When the user is in a motion state, the system will switch to a simplified interaction mode based on gestures to ensure the stability and reliability of the interaction.

[0091] In addition, the present application also introduces an incremental learning mechanism, which enables the system to continuously adapt to the user's personalized usage habits. In actual application, the system will regularly collect the user's interaction data and corresponding intent labels, and use these data to perform incremental training on the mixed-intent decoding model. During the incremental training process, a forgetting mechanism and batch normalization technology are used to balance the weights of new and old knowledge and improve the stability of learning. Specifically, the forgetting mechanism can be expressed as:

[0092] W t t+1 = W t-1 t * e -λt

[0093] where W t t is the model weight at time t, and λ is the forgetting rate, usually set between 0.001 and 0.01, and the specific value can be adjusted according to the application scenario.

[0094] The calculation formula of batch normalization is:

[0095]

[0096] where μ B and are the mean and variance of the mini-batch, γ and β are the learnable scaling and shifting parameters, and ∈ is a small constant to prevent division by zero, usually set to 1e -5 .

[0097] The method of the present application realizes complex intent analysis while ensuring response speed through edge computing and cloud processing. A lightweight model is deployed on the smart watch to perform real-time action recognition, while complex intent analysis tasks are sent to the cloud for processing. This distributed architecture fully utilizes the real-time nature of edge devices and the powerful computing capabilities of the cloud, enabling the system to maintain high efficiency in various scenarios.

[0098] In summary, the smart watch assisted hybrid intent decoding and human-computer interaction method proposed in the present application realizes accurate understanding of user intent and natural interaction through multi-modal data fusion, deep learning algorithms and distributed computing architecture. This method not only improves the accuracy and fluency of interaction, but also has good adaptability and scalability, laying the foundation for future more intelligent and natural human-computer interaction systems. In a preferred embodiment of the present application, the fine-grained intent analysis step further includes independent component analysis of the electroencephalogram data to remove interference signals. This step is crucial for improving the quality of electroencephalogram signals, as they often contain various noise and artifacts. The independent component analysis (ICA) technique can effectively separate the true electroencephalogram activity signals.

[0099] Specifically, ICA can be represented as:

[0100] X = AS

[0101] Where X is the observed mixed signal, A is the mixing matrix, and S is the source signal. The method of the present application estimates the inverse matrix W of the mixing matrix A to obtain independent source signals:

[0102] S = WX

[0103] In practical applications, the FastICA algorithm can be used to implement this process. This algorithm estimates independent components by maximizing non-Gaussianity, and its iterative process can be represented as:

[0104] w + = E{xg(w T x)} - E{g'(w T x)}w

[0105] Where w is the independent component to be estimated, g is a nonlinear function, and g(u) = tanh(au) is usually selected, where a is a constant, generally taking a value between 1 and 2.

[0106] After removing interference signals, the method of the present application uses a CNN-LSTM hybrid network to extract spatio-temporal features. The CNN part is mainly responsible for extracting spatial features, while the LSTM captures temporal dependencies. This structure is particularly suitable for processing data such as electroencephalogram signals that have complex spatio-temporal features.

[0107] In the CNN part, the present application preferably uses a multi-layer convolutional structure. Each layer of convolution can be represented as:

[0108]

[0109] wherein, is the jth feature map of the lth layer, M j is the selection of input feature maps, is the convolution kernel, is the bias term, and f is the activation function, which is usually selected as the ReLU function: f(x) = max(0, x).

[0110] The structure of the LSTM part is the same as described above and will not be repeated. It is worth noting that between the CNN and the LSTM, the method of the present application adds a fully connected layer for converting the spatial features extracted by the CNN into a sequence form that can be processed by the LSTM.

[0111] The method of the present application also introduces a multi-layer attention mechanism, which further enhances the model's ability to identify key information. After each layer of LSTM, an attention layer is applied. The calculation of the attention mechanism can be represented as:

[0112] e t = v T tanh(Wh t +b)

[0113]

[0114]

[0115] wherein h t is the hidden state of the LSTM, v, W and b are learnable parameters, a t is the attention weight, and c is the context vector. In another embodiment of the present application, the generation of the hybrid intention decoding result is achieved by an integration strategy generator. This generator uses an adaptive weighted fusion method to combine the coarse-grained action recognition result and the fine-grained intention parsing result. The specific fusion strategy can be represented as:

[0116] I = aR c + (1-a)R f + b(R c R f )

[0117] wherein I is the final hybrid intention decoding result, R c is the coarse-grained recognition result, and R fis the fine-grained parsing result, and and are adaptive weights, and denotes element-wise multiplication. This fusion method not only considers the linear combination of the two results, but also introduces their interaction terms, which can better capture complex intent information.

[0118] The values of the weights and are dynamically adjusted and can be determined according to the current scene and the confidence of recognition. For example, the following settings can be made:

[0119] α = σ(w1c c +w2c f +b1)

[0120] β = σ(w3c c +w4c f +b2)

[0121] where c c and c f are the confidence of coarse-grained and fine-grained recognition, w1, w2, w3, w4, b1 and b2 are learnable parameters, and σ is the sigmoid function.

[0122] After generating the mixed intent decoding result, the method of the present application sends it to the control module for performing corresponding human-computer interaction operations. These operations can include but are not limited to: controlling smart home devices (such as adjusting indoor temperature, turning on / off lamps, etc.), responding to voice commands (such as setting an alarm, sending a message, etc.), interacting in a virtual or augmented reality environment (such as selecting a virtual object, adjusting the viewing angle, etc.).

[0123] Preferably, the method of the present application further includes an adaptive interaction mode switching function. The system will detect the user's motion state in real time and select the most suitable interaction mode according to the state. For example, when it is detected that the user is in a stationary state, the system will enable the dual-mode interaction mode of electroencephalogram and electromyogram. The condition for judging the stationary state can be set as: the standard deviation of the acceleration sensor data within 2 seconds is less than 0.05 m / s 2 In this mode, the system can obtain more rich intent information, thus realizing more accurate interaction.

[0124] On the contrary, when the user is in a motion state, the system will automatically switch to a simplified interaction mode based on gestures. This mode mainly relies on the data of acceleration sensor and gyroscope sensor, which can ensure the stability and reliability of interaction when the user is moving. The condition for judging the motion state can be set as: the standard deviation of the acceleration sensor data within 1 second is greater than 0.2 m / s 2 .

[0125] Through this adaptive interaction mode switching, the method of the present application can maintain good interaction experience in various use scenarios, greatly improving the practicality and user friendliness of the system. In another preferred embodiment of the present application, the incremental training process contains several key technical points aimed at improving the adaptability and learning efficiency of the model. First, the present method uses a forgetting mechanism to control the influence of historical data on the model. This mechanism can help the model better adapt to changes in user habits and avoid over-reliance on old data. Specifically, the forgetting mechanism can be implemented through the following formula:

[0126]

[0127] where x i is the input, μ B and are the mean and variance of the mini-batch, ∈ is a small constant (usually set to 1e-5) for numerical stability, and γ and β are learnable scaling and shifting parameters.

[0128] In addition, the method of the present application also performs group training based on user features to achieve personalized model optimization. This strategy can better capture the behavior patterns of different user groups and improve the generalization ability of the model. For example, users can be divided into different groups according to their age, usage frequency, main usage scenarios, etc., and then a special model is trained for each group. This group training method can be implemented through a multi-task learning framework:

[0129]

[0130] where L g is the loss function of the gth group of users, α g is the weight of each group, Ω(W) is the regularization term, and λ is the regularization coefficient. Preferably, α g can be dynamically adjusted according to the number of users in each group to balance the contributions of different groups.

[0131] In addition, the method of the present application also performs group training based on user features to achieve personalized model optimization. This strategy can better capture the behavior patterns of different user groups and improve the generalization ability of the model. For example, users can be divided into different groups according to their age, usage frequency, main usage scenarios, etc., and then a special model is trained for each group. This group training method can be implemented through a multi-task learning framework:

[0132]

[0133] where L g is the loss function of the gth group of users, α gis the weight of each group, Ω(W) is the regularization term, and λ is the regularization coefficient. Preferably, α g It can be dynamically adjusted according to the number of users in each group to balance the contribution of different groups.

[0134] In another embodiment of the present application, the edge computing and cloud collaborative processing step further optimizes the performance and efficiency of the system. On the smart watch side, a lightweight model is deployed to perform real-time action recognition. This model can be a small neural network pruned and quantized, such as a variant of MobileNet or SqueezeNet. Such design can ensure fast response under limited computing resources.

[0135] At the same time, complex computing tasks such as fine-grained intent parsing are sent to the cloud for execution. The cloud has more powerful computing capabilities and can run more complex models such as the CNN-LSTM hybrid network described earlier. This distributed architecture takes full advantage of the real-time performance of edge devices and the powerful computing capabilities of the cloud, enabling the system to maintain high efficiency in various scenarios.

[0136] To coordinate data interaction and model updates between the smart watch side and the cloud, the method of the present application adopts an asynchronous update mechanism. This mechanism can be represented as:

[0137]

[0138] where θ t is the model parameter at time t, η is the learning rate, f i is the loss function of the i-th data sample. In practical applications, the Adam optimizer can be used to implement this update process, and the update rule of the Adam optimizer is as follows:

[0139]

[0140] where m t and v t are the first and second moment estimates of the gradient, β1 and β2 are the decay rates (usually set to 0.9 and 0.999), and ∈ is a small constant (usually set to 1e -8 ).

[0141] Finally, the method of the present application adopts a flexible strategy when performing corresponding human-computer interaction operations. First, the system will select the appropriate interaction mode according to the mixed intent decoding result. For example, if the decoding result indicates that the user's intent is to perform precise control, the system may choose a fine gesture-based interaction mode; if the intent is a quick command, the system may choose a voice interaction mode.

[0142] Then, based on the selected interaction mode, the system generates corresponding control instructions or feedback information. These instructions or information can include device control commands, user interface update instructions, or voice feedback content, etc. Finally, the system outputs these control instructions or feedback information to the user through the display screen or vibration module of the smart watch.

[0143] Preferably, the method of the present application further includes an adaptive feedback mechanism. The system monitors the user's reaction to each interaction, and if it finds that the user frequently corrects certain operations, it automatically adjusts the relevant recognition threshold or interaction strategy. For example, if it finds that the user often needs to repeat a certain voice command, the system may lower the recognition threshold for that command, or provide more visual feedback to confirm that the command has been correctly recognized.

[0144] In this way, the method of the present application not only achieves accurate intent recognition and natural human-computer interaction, but also continuously adapts to the user's usage habits, providing personalized interaction experience. This intelligent and adaptive interaction method greatly improves the user's satisfaction, laying the foundation for future more advanced human-computer interaction systems.

[0145] The above description is only a preferred embodiment of the present application, but the protection scope of the present application is not limited thereto; any person skilled in the art, according to the scheme and the improved concept of the present application, makes equivalent replacement or change within the scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A hybrid intent decoding and human-computer interaction method assisted by a smartwatch, characterized in that, include: The acquisition steps include: Acquire interactive feature data collected by the smartwatch's sensors and EEG signal data collected by the EEG signal acquisition device; The processing steps include: Based on the interaction feature data, perform coarse-grained action recognition; Based on the aforementioned EEG signal data, fine-grained intent parsing is performed; Based on the coarse-grained action recognition result and the fine-grained intent parsing result, a hybrid intent decoding result is generated; Output steps, including: The hybrid intent decoding result is sent to the control module to perform the corresponding human-computer interaction operation.

2. The method according to claim 1, characterized in that, The acquisition steps specifically include: Interaction feature data is collected through the accelerometer, gyroscope, pressure sensor and electromyography sensor of the smartwatch; Brainwave signal data is collected using a wearable brain-computer interface device.

3. The method according to claim 1, characterized in that, The coarse-grained action recognition includes: Based on the interaction feature data, temporal features are extracted using an LSTM network and an attention mechanism; Based on the temporal features, the action category is determined using a Softmax classifier.

4. The method according to claim 1, characterized in that, The fine-grained intent parsing includes: Independent component analysis was performed on the EEG signal data to remove interference signals; Based on the interference-free EEG signal data, spatiotemporal features are extracted using a CNN-LSTM hybrid network. Based on the aforementioned spatiotemporal characteristics, intent parsing is achieved through a multi-layered attention mechanism.

5. The method according to claim 1, characterized in that, The steps for generating the hybrid intent decoding result include: Based on the integrated strategy generator, the coarse-grained action recognition results and the fine-grained intent parsing results are adaptively weighted and fused. Based on the fusion results, the final hybrid intent decoding result is determined.

6. The method according to claim 1, characterized in that, It also includes the adaptive interaction mode switching steps: Based on the sensor data from the smartwatch, the user's movement status is detected in real time; Based on the user's movement state, a dual-modal interaction mode using EEG and EMG is adopted when the user is at rest, and a simplified interaction mode based on gestures is switched when the user is in motion.

7. The method according to claim 1, characterized in that, It also includes incremental learning steps: Obtain user interaction data and corresponding intent tags; Based on the user interaction data and intent tags, the hybrid intent decoding model is incrementally trained; Update the parameters of the hybrid intent decoding model based on the incremental training results.

8. The method according to claim 7, characterized in that, The incremental training includes: A forgetting mechanism is used to control the impact of historical data on the model; Use batch normalization techniques to improve the stability of incremental learning; Group training based on user characteristics enables personalized model optimization.

9. The method according to claim 1, characterized in that, It also includes the steps of edge computing and cloud-based collaborative processing: Deploy a lightweight model on a smartwatch to perform real-time motion recognition; Send complex computing tasks to the cloud for fine-grained intent parsing; Based on an asynchronous update mechanism, it coordinates data interaction and model updates between the smartwatch and the cloud.

10. The method according to claim 1, characterized in that, The execution of the corresponding human-computer interaction operations includes: Based on the result of the hybrid intent decoding, select the corresponding interaction mode; Based on the selected interaction mode, generate control commands or feedback information; The control commands or feedback information are output to the user through the smartwatch's display screen or vibration module.

Images