Cooperative interface generation method and system integrating eye movement interaction and touch perception
By recognizing eye movement patterns and predicting tactile feedback patterns, a tactile perception strategy is generated, which solves the problem of insufficient integration of eye movement interaction and tactile perception in existing technologies, and achieves a more intelligent and intuitive human-computer interaction experience.
Patent Information
- Application Number
- CN202311611194.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-29
- Publication Date
- 2026-02-03
AI Technical Summary
Existing technologies rarely combine eye-tracking interaction and tactile perception, failing to meet users' needs for a more intelligent and intuitive human-computer interaction method.
By identifying users' eye movement patterns from eye-tracking interaction data, and utilizing the pre-built correlation between user eye movements and tactile perception, the system predicts the next tactile feedback pattern, calculates the correlation score, generates tactile perception strategies and eye-tracking interaction strategies, and constructs a collaborative interface that meets user needs.
It improves the efficiency and accuracy of interactive operations, meeting users' needs for a more intelligent and intuitive human-computer interaction method.
Smart Images
Figure CN121455313A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the technical field of human-computer interaction, and particularly relates to a collaborative interface generation method and system integrating eye movement interaction and tactile perception. BACKGROUND
[0002] In the wave of modern technological development, human-computer interaction technology has been widely applied and researched. Traditional human-computer interaction methods mainly include keyboard, mouse and touch screen, etc., which cannot meet the needs of users for interaction experience to some extent. Therefore, new human-computer interaction methods are widely concerned and researched to meet the needs of users for more natural, efficient and intelligent interaction methods.
[0003] Eye movement technology as a new human-computer interaction method realizes interaction control by tracking the eye movement of users, and has been widely applied in virtual reality, smart home and medical treatment, etc. Tactile perception technology as another important human-computer interaction method can bring users more intuitive and immersive interaction experience by simulating the tactile perception system of human beings.
[0004] However, there are few technologies on the market that combine eye movement interaction and tactile perception to provide users with more rich and diverse interaction experience. Therefore, it is necessary to provide a collaborative interface generation method integrating eye movement interaction and tactile perception to meet the needs of users for more intelligent and intuitive human-computer interaction methods. SUMMARY
[0005] The purpose of the application is to provide a collaborative interface generation method and system integrating eye movement interaction and tactile perception to solve the problems in the prior art, improve the efficiency and accuracy of interaction operation, and meet the needs of users for more intelligent and intuitive human-computer interaction methods.
[0006] One embodiment of the present application provides a collaborative interface generation method integrating eye movement interaction and tactile perception, which comprises:
[0007] identifying the eye movement pattern of the user from the eye movement interaction data;
[0008] using the pre-constructed correlation between the eye movement of the user and the tactile perception to predict the tactile feedback pattern of the next step according to the eye movement pattern of the user, wherein the tactile feedback pattern comprises predicted tactile perception data;
[0009] calculating the correlation score between the eye movement pattern of the user and the predicted tactile feedback pattern;
[0010] judging whether the correlation score reaches a threshold score;
[0011] If the predicted tactile feedback pattern is achieved, a tactile perception strategy is generated based on the predicted tactile feedback pattern; otherwise, the process returns to the step of using the pre-built correlation between user eye movement and tactile perception to predict the next tactile feedback pattern based on the user's eye movement pattern, until the correlation score between the eye movement pattern and the predicted tactile feedback pattern reaches the threshold score.
[0012] An eye-tracking interaction strategy is generated based on the eye-tracking pattern, and a collaborative interface that meets user needs is generated based on the eye-tracking interaction strategy and the tactile perception strategy.
[0013] Optionally, identifying the user's eye movement pattern from the eye-tracking interaction data includes:
[0014] The fixation points and saccade paths in the eye-tracking interaction data are used as nodes, and a graph structure is constructed based on their temporal relationship; wherein, the connections between the nodes are determined according to the temporal order and distance factors.
[0015] Based on the constructed eye-tracking interaction data map, features of the map are extracted, wherein the features can reflect the spatiotemporal relationship of eye-tracking patterns and the complexity of saccade paths.
[0016] Complex network analysis is performed on the extracted graph features. Based on the results of the complex network analysis, the features of eye movement patterns are identified, and the eye movement patterns of different users are distinguished by these features.
[0017] Optionally, the step of predicting the next tactile feedback pattern based on the user's eye movement pattern by utilizing a pre-built correlation between user eye movement and tactile perception includes:
[0018] The user's eye movement pattern is input into a pre-trained regression model to obtain an intermediate representation; wherein, the regression model is able to learn the correlation between the user's eye movement pattern and haptic feedback pattern;
[0019] The intermediate representation is mapped onto the feature space of tactile perception to obtain the predicted tactile feedback pattern.
[0020] Optionally, the formula for calculating the correlation score between the user's eye movement pattern and the predicted tactile feedback pattern includes:
[0021] r=(n(Σxy)-(Σx)(Σy)) / sqrt((n(Σx^2)-(Σx)^2)(n(Σy^2)-(Σy)^2))
[0022] Where r represents the associated score, n represents the number of sample data points, Σ represents the summation, x and y represent the sample data points of the eye-tracking pattern and the predicted tactile feedback pattern, respectively, Σxy represents the sum of the products of the corresponding data points of x and y, Σx and Σy represent the sum of x and y, respectively, and Σx^2 and Σy^2 represent the sum of the squares of x and y, respectively.
[0023] Optionally, generating a collaborative interface that meets user needs based on the eye-tracking interaction strategy and the tactile perception strategy includes:
[0024] Eye-tracking patterns and predicted haptic feedback patterns are used as inputs to a portion of the state space to construct a state representation for the interface generation model.
[0025] Based on eye-tracking interaction strategy and tactile perception strategy, determine the action space for the interface generation model;
[0026] Determine the reward function used to evaluate the model's behavior in the environment;
[0027] The model is trained using a state space, an action space, and a reward function. During training, the model is used to find the optimal eye-tracking interaction strategy and haptic perception strategy to maximize the reward.
[0028] Based on the trained model, a collaborative interface that meets user needs is generated using the current eye-tracking interaction strategy and tactile perception strategy.
[0029] Another embodiment of this application provides a collaborative interface generation system integrating eye-tracking interaction and tactile perception, the system comprising:
[0030] The recognition module is used to identify the user's eye movement patterns from eye-tracking interaction data;
[0031] The prediction module is used to predict the next tactile feedback pattern based on the user's eye movement pattern by utilizing the pre-built correlation between the user's eye movement and tactile perception. The tactile feedback pattern includes: predicted tactile perception data.
[0032] The calculation module is used to calculate the correlation score between the user's eye movement pattern and the predicted tactile feedback pattern;
[0033] The judgment module is used to determine whether the associated score has reached the threshold score;
[0034] The first generation module is used to generate a tactile perception strategy based on the predicted tactile feedback pattern if the correlation score reaches the threshold score; otherwise, it returns to the step of using the pre-built correlation between user eye movement and tactile perception to predict the next tactile feedback pattern based on the user's eye movement pattern, until the correlation score between the eye movement pattern and the predicted tactile feedback pattern reaches the threshold score.
[0035] The second generation module is used to generate an eye-tracking interaction strategy based on the eye-tracking pattern, and to generate a collaborative interface that meets user needs based on the eye-tracking interaction strategy and the tactile perception strategy.
[0036] Another embodiment of this application provides a storage medium storing a computer program, wherein the computer program is configured to execute the method described in any of the preceding claims when running.
[0037] Another embodiment of this application provides an electronic device including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the method described in any of the preceding claims.
[0038] Compared with existing technologies, this invention provides a collaborative interface generation method integrating eye-tracking interaction and tactile perception. This method identifies the user's eye-tracking patterns from eye-tracking interaction data; predicts the next tactile feedback pattern using a pre-built correlation between the user's eye-tracking and tactile perception; calculates a correlation score between the eye-tracking pattern and the predicted tactile feedback pattern; determines whether the correlation score reaches a threshold score; if so, generates a tactile perception strategy based on the predicted tactile feedback pattern; generates an eye-tracking interaction strategy based on the eye-tracking pattern; and generates a collaborative interface that meets the user's needs based on the eye-tracking interaction strategy and the tactile perception strategy. This improves the efficiency and accuracy of interactive operations, satisfying users' needs for a more intelligent and intuitive human-computer interaction method. Attached Figure Description
[0039] Figure 1 A flowchart illustrating a collaborative interface generation method integrating eye-tracking interaction and tactile perception provided in an embodiment of the present invention;
[0040] Figure 2 A schematic diagram of the structure of a collaborative interface generation system integrating eye-tracking interaction and tactile perception provided in an embodiment of the present invention;
[0041] Figure 3 This is a hardware structure block diagram of a computer terminal that provides a collaborative interface generation method integrating eye-tracking interaction and tactile perception, as provided in an embodiment of the present invention. Detailed Implementation
[0042] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.
[0043] See Figure 1 The present invention provides a method for generating a collaborative interface integrating eye-tracking interaction and tactile perception, the method comprising the following steps:
[0044] S101, Identify the user's eye movement patterns from eye-tracking interaction data;
[0045] Specifically, the fixation points and saccade paths in the eye-tracking interaction data can be used as nodes to construct a graph structure based on their temporal relationship; wherein the connections between the nodes are determined according to the temporal order and distance factors.
[0046] Extract fixation point and saccade path information from eye-tracking interaction data.
[0047] Using fixation points and saccade paths as nodes, a graph structure is constructed based on their temporal relationship.
[0048] The connection method between nodes can be determined based on time sequence and distance factors.
[0049] For chronological order, the connection order can be determined based on the order in which the fixation point and saccade path appear on the timeline. For example, the connection strength can be determined by connecting lines in chronological order or by time intervals.
[0050] For distance factors, the connection method can be determined based on the physical distance between the gaze point and the saccade path. For example, the connection weight can be determined based on the distance, or the connection distance threshold can be used to determine whether a connection exists.
[0051] Other factors, such as gaze duration, can also be considered to comprehensively determine the connection relationships between nodes.
[0052] The completed graph structure can be used for subsequent data analysis, visualization, or fusion with tactile perception data to generate more intelligent and personalized interactive interfaces.
[0053] Based on the constructed eye-tracking interaction data map, features of the map are extracted, wherein the features can reflect the spatiotemporal relationship of eye-tracking patterns and the complexity of saccade paths.
[0054] Density distribution features of fixation points and saccade paths are extracted. Statistical analysis of node distribution reflects the spatial distribution of eye movement patterns.
[0055] Calculate distance features. Distance features between fixation points and saccade paths can be calculated based on graph-structured distance metrics to reflect the spatial relationships between eye movement patterns.
[0056] Extract temporal features. Analyze the temporal sequence of fixation points and saccades to reflect the temporal characteristics of eye movement patterns.
[0057] Consider the complexity of scan paths. The complexity of scan paths can be characterized by graph connectivity analysis or by metrics such as path length and number of branches.
[0058] High-level features can be extracted using machine learning or deep learning methods. Modern machine learning techniques, such as neural networks and convolutional neural networks, can be used to extract and learn features from eye-tracking interaction map data to uncover more complex and abstract eye-tracking pattern features.
[0059] The extracted features are used for subsequent eye-tracking interaction analysis, interface generation, or pattern recognition to achieve a deeper understanding and application of the spatiotemporal relationships and complexity of eye-tracking interaction patterns.
[0060] Complex network analysis is performed on the extracted graph features. Based on the results of the complex network analysis, the features of eye movement patterns are identified, and the eye movement patterns of different users are distinguished by these features.
[0061] A complex network model of eye-tracking interaction data is constructed. First, based on the feature extraction results of the eye-tracking interaction map, a complex network model is constructed, where nodes represent gaze points and saccade paths, and edges represent the relationships between them.
[0062] Complex network analysis methods can be applied to the constructed eye-tracking interactive complex network model, including indices such as degree distribution, clustering coefficient, and betweenness centrality, to analyze the network characteristics.
[0063] Identify eye-tracking pattern characteristics. Through complex network analysis, identify key network indicators that represent eye-tracking pattern characteristics, such as the degree of clustering in the network and the importance of nodes.
[0064] Differentiating eye movement patterns among different users. Based on the characteristics of eye movement patterns, machine learning methods, such as cluster analysis and classifiers, can be used to identify and differentiate the eye movement patterns of different users.
[0065] Establish a correlation between eye-tracking pattern features and user identity. By modeling and inductively analyzing the eye-tracking pattern features of different users, a correlation between eye-tracking patterns and user identity can be established, thereby enabling the differentiation and identification of eye-tracking patterns among different users.
[0066] Finally, by utilizing the identified user eye-tracking patterns, interactive interfaces can be customized for different users, providing a more intelligent and personalized user experience.
[0067] S102, using the pre-built correlation between user eye movement and tactile perception, predict the next tactile feedback pattern based on the user's eye movement pattern, wherein the tactile feedback pattern includes: predicted tactile perception data;
[0068] Specifically, the user's eye movement pattern can be input into a pre-trained regression model to obtain an intermediate representation; wherein, the regression model can learn the correlation between the user's eye movement pattern and haptic feedback pattern;
[0069] Data preparation and feature extraction: First, collect users' eye movement data and corresponding tactile feedback data. Then, extract features related to eye movement patterns and tactile feedback patterns from this data, such as the duration of eye fixation, the length of saccade paths, and the intensity of tactile feedback.
[0070] Building a regression model: Using machine learning methods, such as linear regression, support vector regression (SVR), neural networks, etc., a regression model is built. The extracted eye movement pattern features are used as input and the haptic feedback pattern is used as output. The model is trained to learn the correlation between the user's eye movement pattern and haptic feedback pattern.
[0071] Feature selection and model optimization: Before building a regression model, feature selection and dimensionality reduction may be necessary to eliminate irrelevant or redundant features and improve the model's predictive performance. Furthermore, the model can be optimized through methods such as cross-validation to improve its generalization ability.
[0072] Model training and evaluation: The constructed regression model is trained using existing eye-tracking and haptic feedback data, and validated and evaluated using another set of data to ensure that the model can accurately learn the correlation between the user's eye-tracking patterns and haptic feedback patterns.
[0073] Prediction and Application: After the model training and evaluation are completed, the new eye movement pattern can be input into the pre-trained regression model to obtain an intermediate representation. This intermediate representation can be used to predict the corresponding tactile feedback pattern, thereby realizing the correlation prediction between eye movement pattern and tactile feedback pattern.
[0074] Further research and applications: The predicted intermediate representations can also be used in fields such as personalized user experience and intelligent assistance technology to provide users with a more personalized and intelligent interactive experience.
[0075] The intermediate representation is mapped onto the feature space of tactile perception to obtain the predicted tactile feedback pattern.
[0076] Mapping to the feature space of tactile perception: Using mapping methods, the intermediate representations obtained from the regression model are mapped to the feature space of tactile perception. This may involve techniques such as feature transformation, dimensionality reduction, and feature extraction to ensure that the intermediate representations accurately reflect the relevant features of tactile feedback.
[0077] Extraction of tactile perception features: Design appropriate algorithms or models to extract tactile perception-related features from tactile feedback data, such as tactile intensity, tactile pattern, and tactile frequency.
[0078] Establish a mapping model: Based on the feature space mapped to tactile perception and the extracted tactile perception features, establish a model, which can be a regression model, a classification model or other suitable model, to learn the mapping relationship between intermediate representations and tactile perception features.
[0079] Model training and evaluation: The mapping model is trained using existing intermediate representation data and corresponding tactile perception feature data, and then validated and evaluated using another set of data to ensure that the mapping model can accurately map the intermediate representations to the tactile perception feature space.
[0080] Predicting tactile feedback patterns: The intermediate representation is input into the pre-trained mapping model to obtain the predicted tactile perception features, thereby enabling the prediction of tactile feedback patterns.
[0081] Further research and applications: The predicted haptic feedback patterns can be used in personalized user experiences, virtual reality, augmented reality, and other fields to provide users with a more personalized and richer perceptual experience. Simultaneously, the mapping model can be continuously optimized and improved to enhance its predictive performance and generalization ability.
[0082] S103, calculate the correlation score between the user's eye movement pattern and the predicted tactile feedback pattern;
[0083] Specifically, the formula for calculating the correlation score may include:
[0084] r=(n(Σxy)-(Σx)(Σy)) / sqrt((n(Σx^2)-(Σx)^2)(n(Σy^2)-(Σy)^2))
[0085] Where r represents the associated score, n represents the number of sample data points, Σ represents the summation, x and y represent the sample data points of the eye-tracking pattern and the predicted tactile feedback pattern, respectively, Σxy represents the sum of the products of the corresponding data points of x and y, Σx and Σy represent the sum of x and y, respectively, and Σx^2 and Σy^2 represent the sum of the squares of x and y, respectively.
[0086] By calculating this formula, a correlation coefficient r can be obtained that ranges from -1 to 1. When r approaches 1, it indicates a strong positive correlation between x and y; when r approaches -1, it indicates a strong negative correlation between x and y; and when r approaches 0, it indicates that there is no linear correlation between x and y.
[0087] S104, determine whether the associated score has reached the threshold score;
[0088] The calculated score is compared with a pre-set threshold to determine whether the threshold score has been reached. If the threshold score is reached or exceeded, it indicates that the correlation between the eye movement pattern and the predicted tactile feedback pattern has met the requirements.
[0089] S105, if the threshold score is reached, generate a tactile perception strategy based on the predicted tactile feedback pattern; otherwise, return to the step of using the pre-built correlation between user eye movement and tactile perception to predict the next tactile feedback pattern based on the user's eye movement pattern, until the correlation score between the eye movement pattern and the predicted tactile feedback pattern reaches the threshold score.
[0090] If the threshold score is not reached, a new tactile feedback pattern needs to be predicted again based on the correlation until the correlation score between the eye movement pattern and the predicted new tactile feedback pattern reaches the threshold score, ensuring that the prediction reaches a certain level of accuracy.
[0091] S106, generate an eye-tracking interaction strategy based on the eye-tracking pattern, and generate a collaborative interface that meets user needs based on the eye-tracking interaction strategy and the tactile perception strategy.
[0092] Specifically, eye-tracking patterns and predicted haptic feedback patterns can be used as inputs to a portion of the state space to construct a state representation for the interface generation model.
[0093] Representation of eye movement patterns: Encoding the user's eye movement patterns can be done using feature vectors. Eye movement data can be processed and features extracted, such as the length, velocity, and acceleration of the eye movement trajectory, to represent the eye movement pattern.
[0094] Representation of predicted haptic feedback patterns: The predicted haptic feedback patterns can be encoded, or feature vectors can be used to process and extract features from the predicted haptic feedback data. For example, features such as frequency, amplitude, and duration of the predicted haptic data can be used to represent the haptic feedback patterns.
[0095] Constructing a state representation: By taking the eye-tracking pattern and the predicted tactile feedback pattern as inputs to a portion of the state space, the encoded eye-tracking pattern and the predicted tactile feedback pattern can be combined into a state vector as part of the state space.
[0096] Combining state representations: Add other related states to the state space, such as environment state and user preference state, to construct a complete state space.
[0097] Determine the dimensions and extent of the state space: When incorporating eye-tracking patterns and predicted haptic feedback patterns as part of the state space, it is necessary to ensure that the dimensions and extent of the state space are reasonable so that the interface generation model can effectively model the state space and generate the interface.
[0098] Based on eye-tracking interaction strategy and tactile perception strategy, determine the action space for the interface generation model;
[0099] Representation of eye-tracking interaction strategies: Encoding eye-tracking interaction strategies allows them to be represented as part of the action space based on specific eye-tracking behaviors, such as fixation, saccades, and blinking. Different eye-tracking interaction strategies can be represented using discrete actions.
[0100] Representation of tactile perception strategies: Tactile perception strategies are encoded and represented as another part of the action space based on specific tactile feedback types and operation methods, such as touch, vibration, and force feedback. Different tactile perception strategies can also be represented using discrete actions.
[0101] Determine the dimensions and scope of the action space: When using eye-tracking interaction strategies and tactile perception strategies as the action space of the interface generation model, it is necessary to ensure that the dimensions and scope of the action space are reasonable so that the interface generation model can effectively model and generate the action space.
[0102] Combination of action space: The encoded eye-tracking interaction strategy and tactile perception strategy are combined into action vectors as part of the action space.
[0103] Consider a reasonable action selection mechanism: When determining the specific implementation method of the action space, a reasonable action selection mechanism can be introduced. For example, when generating a collaborative interface, an appropriate eye-tracking interaction strategy and tactile perception strategy can be selected based on the current state.
[0104] Determine the reward function used to evaluate the model's behavior in the environment;
[0105] Based on the goal of generating the collaborative interface, the definition of the reward function is determined: the reward function should be defined according to the specific goal of generating the collaborative interface. For example, user satisfaction, efficiency, accuracy and other indicators of the collaborative interface can be set as part of the reward function.
[0106] Establish reward function calculation rules: Based on the characteristics of the collaborative interface generation task, establish reward function calculation rules. For example, the reward value can be calculated based on factors such as the degree of matching between the interface generation result and user needs, the time consumed, and user satisfaction.
[0107] Choose an appropriate reward function: Based on the specific collaborative interface that generates the task, select a suitable reward function, which can be a linear function, a non-linear function, or other appropriate function forms.
[0108] Optimizing the reward function using reinforcement learning algorithms: Reinforcement learning algorithms can be used to continuously adjust the reward function to maximize the reward value. This includes using reward-feedback loops to adjust and learn the model's behavior towards the environment to achieve better collaborative interface generation results.
[0109] Consider a compound reward function: If multiple factors need to be considered to evaluate the model’s behavior in the environment, multiple reward factors can be combined to construct a compound reward function to comprehensively consider various factors.
[0110] The model is trained using a state space, an action space, and a reward function. During training, the model is used to find the optimal eye-tracking interaction strategy and haptic perception strategy to maximize the reward.
[0111] Data collection and preprocessing: First, it is necessary to collect user eye-tracking interaction data and tactile perception data, and then preprocess and extract features from them to convert them into a representation of the state space required for model training.
[0112] Model selection: Choose a model that is suitable for handling the state space and action space, such as Q-learning, Deep Q Network (DQN), and Policy Gradient in reinforcement learning.
[0113] Define the action selection strategy: Based on the selected model, define the model's action selection strategy, which involves choosing the next eye-tracking interaction strategy and tactile perception strategy according to the current state. This may involve a balance between exploration and exploitation to ensure that the model can not only utilize known eye-tracking interaction strategies and tactile perception strategies, but also try new strategies for improvement.
[0114] Model training: Using the state space, action space, and reward function, the selected reinforcement learning model is trained. The model will try different eye-tracking interaction strategies and tactile perception strategies, and adjust the strategy based on the feedback from the reward function to maximize the overall reward.
[0115] Hyperparameter optimization: During training, it may be necessary to adjust and optimize the model's hyperparameters to obtain the best training results.
[0116] Model evaluation and validation: During the training process, the model needs to be evaluated and validated to ensure that the trained model can effectively generate the best eye-tracking interaction strategy and tactile perception strategy in the real environment.
[0117] Based on the trained model, a collaborative interface that meets user needs is generated using the current eye-tracking interaction strategy and tactile perception strategy.
[0118] State space representation: Encode the current interface state, extract and represent various features of the interface, and use them as input to the model, such as the position, size, and color of interface elements.
[0119] Action space selection: Using the trained model, select the optimal eye-tracking interaction strategy and tactile perception strategy based on the current interface state. This can be achieved by directly generating eye-tracking interaction paths or tactile feedback methods.
[0120] Interface Generation: The selected eye-tracking interaction strategy and haptic perception strategy are applied to the current interface state to generate a collaborative interface that meets the user's needs. This may involve adjustments to elements such as layout, color, size, and animation effects.
[0121] User feedback: Provide user feedback on the generated collaborative interface. This can be done through actual user testing or simulated user interaction to obtain user evaluations and opinions on the generated interface.
[0122] Iterative optimization: Based on user feedback, the model is iteratively optimized to continuously improve the eye-tracking interaction strategy and tactile perception strategy, so that the generated collaborative interface better meets user needs.
[0123] As can be seen, by identifying the user's eye movement patterns from eye-tracking interaction data; predicting the next tactile feedback pattern using a pre-built correlation between user eye movement and tactile perception; calculating the correlation score between the eye movement pattern and the predicted tactile feedback pattern; determining whether the correlation score reaches a threshold score; if so, generating a tactile perception strategy based on the predicted tactile feedback pattern; generating an eye-tracking interaction strategy based on the eye movement pattern; and generating a collaborative interface that meets user needs based on the eye-tracking interaction strategy and the tactile perception strategy, the efficiency and accuracy of interactive operations can be improved, meeting users' needs for a more intelligent and intuitive human-computer interaction method.
[0124] Another embodiment of the present invention provides a collaborative interface generation system integrating eye-tracking interaction and tactile perception, see [link to relevant documentation]. Figure 2 The system may include:
[0125] Recognition module 201 is used to recognize the user's eye movement pattern from eye movement interaction data;
[0126] The prediction module 202 is used to predict the next tactile feedback pattern based on the user's eye movement pattern by utilizing the pre-built correlation between the user's eye movement and tactile perception. The tactile feedback pattern includes: predicted tactile perception data.
[0127] Calculation module 203 is used to calculate the correlation score between the user's eye movement pattern and the predicted tactile feedback pattern;
[0128] The judgment module 204 is used to determine whether the associated score has reached the threshold score;
[0129] The first generation module 205 is used to generate a tactile perception strategy based on the predicted tactile feedback pattern if the correlation score reaches the threshold score; otherwise, it returns to the step of using the pre-built correlation between user eye movement and tactile perception to predict the next tactile feedback pattern based on the user's eye movement pattern, until the correlation score between the eye movement pattern and the predicted tactile feedback pattern reaches the threshold score.
[0130] The second generation module 206 is used to generate an eye-tracking interaction strategy based on the eye-tracking pattern, and generate a collaborative interface that meets user needs based on the eye-tracking interaction strategy and the tactile perception strategy.
[0131] As can be seen, by identifying the user's eye movement patterns from eye-tracking interaction data; predicting the next tactile feedback pattern using a pre-built correlation between user eye movement and tactile perception; calculating the correlation score between the eye movement pattern and the predicted tactile feedback pattern; determining whether the correlation score reaches a threshold score; if so, generating a tactile perception strategy based on the predicted tactile feedback pattern; generating an eye-tracking interaction strategy based on the eye movement pattern; and generating a collaborative interface that meets user needs based on the eye-tracking interaction strategy and the tactile perception strategy, the efficiency and accuracy of interactive operations can be improved, meeting users' needs for a more intelligent and intuitive human-computer interaction method.
[0132] The following detailed explanation uses a computer terminal as an example. Figure 3 This is a hardware structure block diagram of a computer terminal for a collaborative interface generation method integrating eye-tracking interaction and tactile perception, provided as an embodiment of the present invention. (See diagram below.) Figure 3 As shown, a computer terminal may include one or more ( Figure 3 Only one is shown in the diagram. A processor 302 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 304 for storing data are also shown. Optionally, the computer terminal may further include a transmission device 306 for communication functions and an input / output device 308. Those skilled in the art will understand that... Figure 3The structure shown is for illustrative purposes only and does not limit the structure of the computer terminal described above. For example, the computer terminal may also include components that are more complex than those described above. Figure 3 The more or fewer components shown, or having the same Figure 3 The different configurations shown.
[0133] The memory 304 can be used to store software programs and modules of application software, such as the program instructions / modules corresponding to the collaborative interface generation method integrating eye-tracking interaction and tactile perception in this embodiment of the application. The processor 302 executes various functional applications and data processing by running the software programs and modules stored in the memory 304, thereby implementing the above-mentioned method. The memory 304 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 304 may further include memory remotely located relative to the processor 302, and these remote memories can be connected to a computer terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0134] The transmission device 306 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer terminal. In one example, the transmission device 306 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 306 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.
[0135] This invention also provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above method embodiments when running.
[0136] Specifically, in this embodiment, the storage medium can be configured to store a computer program for performing the following steps:
[0137] S101, Identify the user's eye movement patterns from eye-tracking interaction data;
[0138] S102, using the pre-built correlation between user eye movement and tactile perception, predict the next tactile feedback pattern based on the user's eye movement pattern, wherein the tactile feedback pattern includes: predicted tactile perception data;
[0139] S103, calculate the correlation score between the user's eye movement pattern and the predicted tactile feedback pattern;
[0140] S104, determine whether the associated score has reached the threshold score;
[0141] S105, if the threshold score is reached, generate a tactile perception strategy based on the predicted tactile feedback pattern; otherwise, return to the step of using the pre-built correlation between user eye movement and tactile perception to predict the next tactile feedback pattern based on the user's eye movement pattern, until the correlation score between the eye movement pattern and the predicted tactile feedback pattern reaches the threshold score.
[0142] S106, generate an eye-tracking interaction strategy based on the eye-tracking pattern, and generate a collaborative interface that meets user needs based on the eye-tracking interaction strategy and the tactile perception strategy.
[0143] Specifically, in this embodiment, the storage medium may include, but is not limited to, USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks, and other media capable of storing computer programs.
[0144] As can be seen, by identifying the user's eye movement patterns from eye-tracking interaction data; predicting the next tactile feedback pattern using a pre-built correlation between user eye movement and tactile perception; calculating the correlation score between the eye movement pattern and the predicted tactile feedback pattern; determining whether the correlation score reaches a threshold score; if so, generating a tactile perception strategy based on the predicted tactile feedback pattern; generating an eye-tracking interaction strategy based on the eye movement pattern; and generating a collaborative interface that meets user needs based on the eye-tracking interaction strategy and the tactile perception strategy, the efficiency and accuracy of interactive operations can be improved, meeting users' needs for a more intelligent and intuitive human-computer interaction method.
[0145] This invention also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.
[0146] Specifically, the aforementioned electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the aforementioned processor, and the input / output device is connected to the aforementioned processor.
[0147] Specifically, in this embodiment, the processor can be configured to perform the following steps via a computer program:
[0148] S101, Identify the user's eye movement patterns from eye-tracking interaction data;
[0149] S102, using the pre-built correlation between user eye movement and tactile perception, predict the next tactile feedback pattern based on the user's eye movement pattern, wherein the tactile feedback pattern includes: predicted tactile perception data;
[0150] S103, calculate the correlation score between the user's eye movement pattern and the predicted tactile feedback pattern;
[0151] S104, determine whether the associated score has reached the threshold score;
[0152] S105, if the threshold score is reached, generate a tactile perception strategy based on the predicted tactile feedback pattern; otherwise, return to the step of using the pre-built correlation between user eye movement and tactile perception to predict the next tactile feedback pattern based on the user's eye movement pattern, until the correlation score between the eye movement pattern and the predicted tactile feedback pattern reaches the threshold score.
[0153] S106, generate an eye-tracking interaction strategy based on the eye-tracking pattern, and generate a collaborative interface that meets user needs based on the eye-tracking interaction strategy and the tactile perception strategy.
[0154] Specifically, the specific examples in this embodiment can be referred to the examples described in the above embodiments and optional implementations, and will not be repeated here.
[0155] As can be seen, by identifying the user's eye movement patterns from eye-tracking interaction data; predicting the next tactile feedback pattern using a pre-built correlation between user eye movement and tactile perception; calculating the correlation score between the eye movement pattern and the predicted tactile feedback pattern; determining whether the correlation score reaches a threshold score; if so, generating a tactile perception strategy based on the predicted tactile feedback pattern; generating an eye-tracking interaction strategy based on the eye movement pattern; and generating a collaborative interface that meets user needs based on the eye-tracking interaction strategy and the tactile perception strategy, the efficiency and accuracy of interactive operations can be improved, meeting users' needs for a more intelligent and intuitive human-computer interaction method.
[0156] The above description, based on the embodiments shown in the figures, details the structure, features, and effects of the present invention. The above description is only a preferred embodiment of the present invention, but the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or equivalent embodiments modified to have equivalent changes, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.
Claims
1. A collaborative interface generation method integrating eye-tracking interaction and tactile perception, characterized in that, The method includes: Identify users' eye movement patterns from eye-tracking interaction data; By utilizing the pre-built correlation between user eye movement and tactile perception, the next tactile feedback pattern is predicted based on the user's eye movement pattern, wherein the tactile feedback pattern includes: predicted tactile perception data; Calculate the correlation score between the user's eye movement pattern and the predicted haptic feedback pattern; Determine whether the associated score has reached the threshold score; If the predicted tactile feedback pattern is achieved, a tactile perception strategy is generated based on the predicted tactile feedback pattern; otherwise, the process returns to the step of using the pre-built correlation between user eye movement and tactile perception to predict the next tactile feedback pattern based on the user's eye movement pattern, until the correlation score between the eye movement pattern and the predicted tactile feedback pattern reaches the threshold score. An eye-tracking interaction strategy is generated based on the eye-tracking pattern, and a collaborative interface that meets user needs is generated based on the eye-tracking interaction strategy and the tactile perception strategy.
2. The method according to claim 1, characterized in that, The process of identifying the user's eye movement patterns from eye-tracking interaction data includes: The fixation points and saccade paths in the eye-tracking interaction data are used as nodes, and a graph structure is constructed based on their temporal relationship; wherein, the connections between the nodes are determined according to the temporal order and distance factors. Based on the constructed eye-tracking interaction data map, features of the map are extracted, wherein the features can reflect the spatiotemporal relationship of eye-tracking patterns and the complexity of saccade paths. Complex network analysis is performed on the extracted graph features. Based on the results of the complex network analysis, the features of eye movement patterns are identified, and the eye movement patterns of different users are distinguished by these features.
3. The method according to claim 2, characterized in that, The method of predicting the next tactile feedback pattern based on the user's eye movement pattern by utilizing a pre-built correlation between user eye movement and tactile perception includes: The user's eye movement pattern is input into a pre-trained regression model to obtain an intermediate representation; wherein, the regression model is able to learn the correlation between the user's eye movement pattern and haptic feedback pattern; The intermediate representation is mapped onto the feature space of tactile perception to obtain the predicted tactile feedback pattern.
4. The method according to claim 3, characterized in that, The formula for calculating the correlation score between the user's eye movement pattern and the predicted tactile feedback pattern includes: r=(n(Σxy)-(Σx)(Σy)) / sqrt((n(Σx^2)-(Σx)^2)(n(Σy^2)-(Σy)^2)) Where r represents the associated score, n represents the number of sample data points, Σ represents the summation, x and y represent the sample data points of the eye-tracking pattern and the predicted tactile feedback pattern, respectively, Σxy represents the sum of the products of the corresponding data points of x and y, Σx and Σy represent the sum of x and y, respectively, and Σx^2 and Σy^2 represent the sum of the squares of x and y, respectively.
5. The method according to claim 4, characterized in that, The generation of a collaborative interface that meets user needs based on the eye-tracking interaction strategy and the tactile perception strategy includes: Eye-tracking patterns and predicted haptic feedback patterns are used as inputs to a portion of the state space to construct a state representation for the interface generation model. Based on eye-tracking interaction strategy and tactile perception strategy, determine the action space for the interface generation model; Determine the reward function used to evaluate the model's behavior in the environment; The model is trained using a state space, an action space, and a reward function. During training, the model is used to find the optimal eye-tracking interaction strategy and haptic perception strategy to maximize the reward. Based on the trained model, a collaborative interface that meets user needs is generated using the current eye-tracking interaction strategy and tactile perception strategy.
6. A collaborative interface generation system integrating eye-tracking interaction and tactile perception, characterized in that, The system includes: The recognition module is used to identify the user's eye movement patterns from eye-tracking interaction data; The prediction module is used to predict the next tactile feedback pattern based on the user's eye movement pattern by utilizing the pre-built correlation between the user's eye movement and tactile perception. The tactile feedback pattern includes: predicted tactile perception data. The calculation module is used to calculate the correlation score between the user's eye movement pattern and the predicted tactile feedback pattern; The judgment module is used to determine whether the associated score has reached the threshold score; The first generation module is used to generate a tactile perception strategy based on the predicted tactile feedback pattern if the correlation score reaches the threshold score; otherwise, it returns to the step of using the pre-built correlation between user eye movement and tactile perception to predict the next tactile feedback pattern based on the user's eye movement pattern, until the correlation score between the eye movement pattern and the predicted tactile feedback pattern reaches the threshold score. The second generation module is used to generate an eye-tracking interaction strategy based on the eye-tracking pattern, and to generate a collaborative interface that meets user needs based on the eye-tracking interaction strategy and the tactile perception strategy.
7. The system according to claim 6, characterized in that, The identification module is specifically used for: The fixation points and saccade paths in the eye-tracking interaction data are used as nodes, and a graph structure is constructed based on their temporal relationship; wherein, the connections between the nodes are determined according to the temporal order and distance factors. Based on the constructed eye-tracking interaction data map, features of the map are extracted, wherein the features can reflect the spatiotemporal relationship of eye-tracking patterns and the complexity of saccade paths. Complex network analysis is performed on the extracted graph features. Based on the results of the complex network analysis, the features of eye movement patterns are identified, and the eye movement patterns of different users are distinguished by these features.
8. The system according to claim 7, characterized in that, The prediction module is specifically used for: The user's eye movement pattern is input into a pre-trained regression model to obtain an intermediate representation; wherein, the regression model is able to learn the correlation between the user's eye movement pattern and haptic feedback pattern; The intermediate representation is mapped onto the feature space of tactile perception to obtain the predicted tactile feedback pattern.
9. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the method of any one of claims 1-5 when it is run.
10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method of any one of claims 1-5.