Touch display screen panoramic interaction control system based on virtual reality

Abstract

The invention discloses a touch display screen panoramic interaction control system based on virtual reality, and relates to the technical field of virtual reality, the touch display screen panoramic interaction control system comprises a data acquisition module, a data processing module, a management analysis module and an interaction control module, and the management analysis module comprises an action interaction sub-module, an eye movement analysis sub-module and an interaction immersion sub-module. Multiple sub-modules in the management analysis module are mutually associated and matched, the action interaction sub-module outputs an action complexity value so as to integrate multiple factors to quantify the action complexity, adjustment and man-machine interaction of the system are facilitated, the eye movement analysis sub-module outputs eye movement attention comprehensive regulation factors, and the eye movement attention comprehensive regulation factors output by the eye movement analysis sub-module are more accurate. The interactive immersion sub-module outputs a human-computer interaction immersion index to integrate multiple factors to measure the immersion degree, help the system to optimize experience and improve the environment, and make interaction more efficient, and eye movement attention is quantified by considering multiple factors such as the gazing frequency and the like to optimize virtual reality scene display, and the interactive immersion sub-module outputs a human-computer interaction immersion index to integrate multiple factors to measure the immersion degree.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, specifically to a virtual reality-based touchscreen panoramic interactive control system. Background Technology

[0002] Virtual reality (VR) technology is a computer simulation system that can create and experience virtual worlds. It uses computers to generate a simulated environment, allowing users to immerse themselves in that environment. VR technology is usually achieved through external devices applied to users to enable human-computer interaction in virtual reality. These devices include head-mounted displays, full-body auto-trackers, controllers, and eye trackers. Through these devices, users can immerse themselves in the virtual world and achieve a sense of being there. VR technology has broad application prospects in fields such as gaming, education, healthcare, and training.

[0003] However, current panoramic interactive control systems have a relatively simple evaluation of user actions, mostly limited to the recognition of basic action types. When interacting with virtual reality on a touch screen, they may lack consideration for factors such as the length of the user's action path, the rate of change of direction, the smoothness, and the diversity. For example, when performing complex panoramic interactive operations, the system cannot accurately measure the complexity of the user's actions, which may lead to misjudgment of the user's intentions and affect the accuracy of the interaction. Furthermore, when using a touch screen for panoramic interaction, some users have large and fast movements, while others have more delicate and slow movements. Existing systems cannot provide personalized interactive experiences for these different habits, which reduces the applicability of the system and user satisfaction.

[0004] Furthermore, existing systems may lack analysis of user eye movement and attention, and may lack in-depth analysis of the degree of user attention and attention dispersion in different areas of the screen. In panoramic interaction on touch screens, when faced with rich panoramic content, the system cannot accurately determine the user's key focus area, making it difficult to effectively guide the user's attention and affecting the interaction effect.

[0005] In actual interaction, user actions and eye movements influence each other, but existing systems may not be able to comprehensively consider the combined effect of the two, may not consider the impact of action complexity on eye movement attention distribution, and the feedback effect of eye movements on action execution. This makes it impossible for the system to fully understand user interaction behavior and provide intelligent interaction services based on action and eye movement. Summary of the Invention

[0006] The purpose of this invention is to provide a virtual reality-based touch screen panoramic interactive control system, which solves the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a virtual reality-based touchscreen panoramic interactive control system, comprising:

[0008] Data acquisition module: used to collect motion data, eye movement data, and motion type data;

[0009] Motion data information includes: the coordinates of the user's hand in three-dimensional space, the displacement vector between adjacent coordinate points, and the velocity change between two adjacent action time points;

[0010] Eye-tracking data includes: the number of fixations and total fixation time for the user in the j1st subregion;

[0011] Action type data information includes: the number of action types.

[0012] Data processing module: This module is used to input the data collected by the data acquisition module, clean and correct the input data, and input the processed data into the management and analysis module.

[0013] Management and analysis module;

[0014] Based on the user's hand coordinates in three-dimensional space, the displacement vector between adjacent coordinate points, the velocity change between two adjacent action time points, and the number of action types, the action path length value, the action direction change value, the action smoothness factor, and the action diversity factor are calculated respectively. The calculated values ​​are then weighted and integrated to output the action complexity value.

[0015] Based on the number of fixations in the j1st subregion, the fixation frequency and eye-tracking attention factor are calculated. Then, based on the total fixation time in the j1st subregion, the fixation duration factor is calculated. Finally, the eye-tracking attention factor and the fixation duration factor are weighted and integrated to output the comprehensive eye-tracking attention regulation factor.

[0016] The human-computer interaction immersion index is output by weighting and combining factors such as motion complexity, eye-tracking attention regulation, and environmental adaptation.

[0017] Interactive control module: Used to input the output values ​​of the management and analysis module, and to optimize the display layout and content presentation of the virtual environment accordingly.

[0018] Optionally, the management and analysis module includes: a motion interaction submodule, an eye-tracking analysis submodule, and an interactive immersion submodule.

[0019] Optionally, the processing procedure of the action interaction submodule is as follows:

[0020] S1, In a virtual reality environment, the user's actions are captured by motion capture devices in the data acquisition module, obtaining a sequence of coordinates in three-dimensional space. Let the user's actions occur within the time interval [t0, t...]. n The sequence of action coordinates within the range is (x1, y1, z1), (x2, y2, z2), ..., (x... n y n , z n ), to obtain the coordinates (x, y) of the user's hand in three-dimensional space at time point i. i y i , z i );

[0021] S2, through the collected coordinates (x i y i , z i ) Analyze the displacement components between two adjacent action key points and square them to eliminate the influence of positive and negative displacement components. Then, consider the displacements in three directions together to obtain the actual displacement distance between two adjacent key points. Finally, sum the displacement distances between all adjacent time points to output the motion path length value of the user in the entire action process.

[0022] S3, through the collected coordinates (x i y i , z i Analyze the displacement vectors between adjacent coordinate points and the displacement vectors between the next set of adjacent coordinate points, and calculate the magnitude of the corresponding vectors. Then, use the vector dot product operation to process the vectors, divide the dot product result by the product of the magnitudes of the two vectors, and then use the inverse cosine function to process the vectors, and use the summation function to sum the vectors to obtain the cumulative angle of the direction change during the entire action process, and calculate the average value to output the degree of change in the direction of the action.

[0023] S4 obtains an approximate value of acceleration by using the velocity change between two adjacent action time points and the time interval between two adjacent action time points, and places the approximate value of acceleration in the denominator to reflect the negative impact of acceleration on motion smoothness, so as to output the motion smoothness factor.

[0024] S5 combines the occurrence count of the k-th action type with the average occurrence count of the action type, and then sums the difference to obtain the dispersion of the occurrence count of the action, so as to output the action diversity factor.

[0025] S6 combines the motion path length value, motion direction change value, motion smoothness factor, and motion diversity factor in a weighted manner to output the motion complexity value.

[0026] Optionally, the processing procedure of the eye-tracking analysis submodule is as follows:

[0027] S1, the fixation frequency is analyzed by dividing the number of fixations in the j1th small region by the total number of fixations.

[0028] S2 outputs the eye-tracking attention factor by quantifying the fixation frequency using a logarithmic function;

[0029] S3 analyzes the relative duration of each gaze in the target small area by the user's total gaze time in the j1-th small area and the duration of the user's i1-th gaze in the j1-th small area, and calculates the average value to output the gaze duration factor.

[0030] S4 combines the eye-tracking attention factor, fixation duration factor, and motion complexity value in a weighted manner to output a comprehensive eye-tracking attention regulation factor.

[0031] Optionally, the processing procedure of the interactive immersion submodule is as follows:

[0032] S1, weighted combination of eye-tracking attention adjustment factor and motion complexity value, input to interactive immersion submodule;

[0033] S2 compares user action data with template action data based on the accuracy, completeness, and fluency of user actions to output an environmental adaptation factor. Then, the environmental adaptation factor, eye-tracking attention comprehensive adjustment factor, and action complexity value are combined to output the human-computer interaction immersion index.

[0034] Optionally, the interactive control module includes: a motion interaction submodule analysis section, an eye-tracking analysis submodule analysis section, and an interactive immersion submodule analysis section;

[0035] The analysis section of the action interaction submodule specifically includes:

[0036] Based on the calculation results of the action complexity value, the task difficulty and action requirements in the virtual environment are adjusted. When the action complexity value is high, it will cause users to feel tired. The operation steps are simplified and unnecessary steps are removed. When the action complexity value is low, the user's operation is too simple and cannot fully utilize the system's functions, resulting in a monotonous user experience and reduced interest in the system. The action complexity is increased to improve user participation and immersion.

[0037] Optionally, the eye-tracking analysis submodule specifically includes:

[0038] Based on the calculation results of the eye-tracking attention comprehensive adjustment factor, the display layout and content presentation of the virtual environment are optimized. When the eye-tracking attention comprehensive adjustment factor is high, it means that the user's gaze on the screen is relatively scattered, and the user cannot concentrate on important information, which affects the interaction effect. The display position and method of key information should be adjusted to attract the user's attention. When the eye-tracking attention comprehensive adjustment factor is low, the display content should be further enriched to enhance the user's desire to explore.

[0039] Optionally, the analysis section of the interactive immersion submodule specifically includes:

[0040] Based on the calculation results of the human-computer interaction immersion index, the entire virtual reality touch screen panoramic interactive control system is comprehensively optimized. When the human-computer interaction immersion index is high, it indicates that the user's immersion is high and they are very satisfied with the interactive experience. On the basis of maintaining the current good experience, we should further explore and optimize the interaction methods to continuously improve user participation. When the human-computer interaction immersion index is low, it indicates that the user's immersion is low and the user is not satisfied with the current interactive experience. At this time, the entire virtual reality touch screen panoramic interactive control system needs to be comprehensively optimized. We need to adjust the ambient sound effects, enhance the visual effects, and improve the tactile feedback to improve the user's immersion.

[0041] Compared with the prior art, the present invention has the following beneficial effects:

[0042] I. This invention outputs a motion complexity value through a motion interaction submodule. This submodule quantifies the complexity of user actions by comprehensively considering multiple factors: motion path length reflects the range of motion, direction change rate reflects the frequency of changes in motion direction, fluency factor measures the smoothness of motion speed changes, and diversity factor considers the richness of motion types. In a touchscreen panoramic interactive control system, the system can dynamically adjust the interaction content or feedback method according to the user's motion difficulty to improve the user experience. It can also assess the user's skill level and provide a basis for personalized training and guidance. This module calculates the direction change rate using vector dot product and inverse cosine function, and calculates the fluency factor using speed change and time interval, etc., using precise calculation and analysis methods to comprehensively consider all aspects of the motion and quantify relevant factors into motion complexity indicators, providing the system with more detailed and accurate motion information.

[0043] Second, this invention outputs a comprehensive eye-tracking attention adjustment factor through an eye-tracking analysis sub-module. This sub-module quantifies the distribution of the user's eye-tracking attention by using fixation frequency, eye-tracking attention entropy, and fixation duration factor. Fixation frequency reflects the proportion of the user's attention to different areas, eye-tracking attention entropy measures the degree of eye movement dispersion, and fixation duration factor reflects the degree of fixation concentration. Based on this, the control system can optimize the layout and content display of the virtual scene according to the user's eye-tracking attention distribution, thereby improving the efficiency of the user's information acquisition and enhancing the sense of immersion. This sub-module comprehensively analyzes the user's eye-tracking attention from multiple perspectives, such as fixation frequency, eye-tracking attention entropy, and fixation duration factor.

[0044] Third, this invention outputs a human-computer interaction immersion index through an interactive immersion submodule. This index comprehensively considers factors such as motion complexity, eye-tracking attention, and environmental adaptability, serving as a crucial indicator of user immersion during human-computer interaction. The control system can optimize the interactive experience based on this index value to enhance user immersion. Based on the calculation results of this submodule, improvements can be made in various aspects, such as increasing the detail of the virtual environment and optimizing lighting effects. The virtual presentation environment can be adjusted in real-time according to user motion and eye-tracking data to increase user engagement. This module helps the system evaluate the user's immersive experience and adjust system parameters based on the calculation results. For example, by optimizing the interactive performance of the touchscreen display, such as response speed and tactile feedback, and the content presentation effects, such as image quality and color reproduction, the smoothness of user movements and the level of eye-tracking attention can be improved, thereby enhancing human-computer interaction immersion and making the interaction more natural and efficient. This module integrates multiple factors through a weighted summation method and flexibly adjusts them according to the importance of the factors, more accurately reflecting the user's immersive experience. Attached Figure Description

[0045] Figure 1 This is a flowchart illustrating the method steps of a virtual reality-based touchscreen panoramic interactive control system.

[0046] Figure 2 This is a schematic diagram of the structural composition of the virtual reality-based touch screen panoramic interactive control system. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Please see Figures 1 to 2This implementation provides a virtual reality-based touchscreen panoramic interactive control system, including:

[0049] Data acquisition module: used to collect motion data, eye movement data, and motion type data;

[0050] Motion data information includes: the coordinates of the user's hand in three-dimensional space, the displacement vector between adjacent coordinate points, and the velocity change between two adjacent action time points;

[0051] Eye-tracking data includes: the number of fixations and total fixation time for the user in the j1st subregion;

[0052] Action type data information includes: the number of action types.

[0053] Data processing module: This module is used to input the data collected by the data acquisition module, clean and correct the input data, and input the processed data into the management and analysis module.

[0054] Management and analysis module;

[0055] Based on the user's hand coordinates in three-dimensional space, the displacement vector between adjacent coordinate points, the velocity change between two adjacent action time points, and the number of action types, the action path length value, the action direction change value, the action smoothness factor, and the action diversity factor are calculated respectively. The calculated values ​​are then weighted and integrated to output the action complexity value.

[0056] Based on the number of fixations in the j1st subregion, the fixation frequency and eye-tracking attention factor are calculated. Then, based on the total fixation time in the j1st subregion, the fixation duration factor is calculated. Finally, the eye-tracking attention factor and the fixation duration factor are weighted and integrated to output the comprehensive eye-tracking attention regulation factor.

[0057] The human-computer interaction immersion index is output by weighting and combining factors such as motion complexity, eye-tracking attention regulation, and environmental adaptation.

[0058] Interactive control module: Used to input the output values ​​of the management and analysis module, and to optimize the display layout and content presentation of the virtual environment accordingly;

[0059] Based on the calculation results of the action complexity value, the task difficulty and action requirements in the virtual environment are adjusted. When the action complexity value is high, it will cause users to feel tired. The operation steps are simplified and unnecessary steps are removed. When the action complexity value is low, the user's operation is too simple and cannot fully utilize the system's functions, resulting in a monotonous user experience and reduced interest in the system. The action complexity is increased to improve user participation and immersion.

[0060] Based on the calculation results of the eye-tracking attention comprehensive adjustment factor, the display layout and content presentation of the virtual environment are optimized. When the eye-tracking attention comprehensive adjustment factor is high, it means that the user's gaze on the screen is relatively scattered, and the user cannot concentrate on important information, which affects the interaction effect. The display position and method of key information should be adjusted to attract the user's attention. When the eye-tracking attention comprehensive adjustment factor is low, the display content should be further enriched to enhance the user's desire to explore.

[0061] Based on the calculation results of the human-computer interaction immersion index, the entire virtual reality touch screen panoramic interactive control system is comprehensively optimized. When the human-computer interaction immersion index is high, it indicates that the user's immersion is high and they are very satisfied with the interactive experience. On the basis of maintaining the current good experience, we should further explore and optimize the interaction methods to continuously improve user participation. When the human-computer interaction immersion index is low, it indicates that the user's immersion is low and the user is not satisfied with the current interactive experience. At this time, the entire virtual reality touch screen panoramic interactive control system needs to be comprehensively optimized. We need to adjust the ambient sound effects, enhance the visual effects, and improve the tactile feedback to improve the user's immersion.

[0062] The management and analysis module includes: a motion interaction submodule, an eye-tracking analysis submodule, and an interactive immersion submodule.

[0063] In this embodiment: The virtual reality-based touch screen panoramic interactive control system can dynamically adjust the content and difficulty of the virtual scene. When the Human-Computer Interaction Immersion Index (IOGP) is low, it indicates that the user's immersion is insufficient, and the user may not be able to fully engage in the virtual interaction, resulting in a poor experience with the system and potentially reducing the user's willingness to use the system. At this time, the control system can use a higher resolution and wider field of view to improve the visual experience, enhance the realism of the virtual environment, and provide diverse interaction methods, such as gesture control, voice commands, and body motion recognition, to meet the needs of different users, increase user participation, and increase the fun and challenge of the scene displayed on the screen. When the Human-Computer Interaction Immersion Index (IOGP) is high, it indicates that the user is already highly immersed, and the difficulty of the current scene can be appropriately maintained to keep the user interested.

[0064] This virtual reality-based touchscreen panoramic interactive control system optimizes the interaction method based on the Action Complexity Value (ASS) and the Eye Attention Comprehensive Adjustment Factor (BSS). High action complexity may indicate excessive steps, large movements, or frequent directional changes, increasing user difficulty and system processing burden. The control system can streamline steps by eliminating unnecessary steps, such as merging multiple consecutive small operations into one large operation to reduce user interaction. Conversely, a low BSS indicates that the user's eye attention is overly focused on one or a few areas, potentially overlooking other important information and resulting in incomplete information acquisition. Important information can be distributed across different areas of the screen to avoid over-concentration. For example, related operation buttons can be distributed across different functional modules, and dynamic elements such as flashing icons and animation effects can be added to different areas of the screen to attract the user's attention and expand their focus.

[0065] Based on the calculation results of multiple sub-modules in the system's management and analysis module, the system can adjust the feedback method of VR devices. For example, when the motion complexity value ASS is high, the vibration feedback intensity of the controller is increased, allowing users to more intuitively feel the effect of their actions. When the eye-tracking attention comprehensive adjustment factor BSS is low, the brightness, contrast and other parameters of the screen are adjusted through the head-mounted display to improve the visual experience. In addition, through the above optimized formulas and application methods, the system can be made more intelligent and personalized, providing users with a higher quality virtual reality touch screen panoramic interactive experience.

[0066] Based on the motion complexity value (ASS) and the eye-tracking attention comprehensive adjustment factor (BSS), this system can provide personalized interactive experiences for different users. For example, it can reduce the difficulty of operation for users who are not familiar with the motion and optimize the information display method for users who are easily distracted.

[0067] The calculation results of Eye-tracking Attention Comprehensive Modulation Factor (BSS) and Human-Computer Interaction Immersion Index (IOGP) can provide a reference for system interface design, scene layout and task design, improve the system's usability and attractiveness. The system can dynamically adjust the interaction content and feedback method based on the real-time calculation results, such as providing reminders when the user's attention declines and providing assistance when the complexity of the action exceeds the user's ability.

[0068] This system comprises multiple interconnected and complementary sub-modules that comprehensively evaluate user interaction with the system from three levels: action, eye movement, and overall immersion. Action complexity affects eye movement attention distribution and immersion, while eye movement attention is closely related to immersion. Through combined calculations, more accurate and comprehensive user interaction information can be obtained, enabling the control system to achieve more intelligent and personalized interactions. Based on the comprehensive calculation results, the system can adjust the interaction content, difficulty, and feedback methods in real time to improve user immersion and operating experience, reduce user fatigue and frustration, enhance system usability and attractiveness, and thus improve the overall system performance and user satisfaction.

[0069] Please see Figures 1 to 2 The action interaction submodule is specifically as follows:

[0070] First, in a virtual reality environment, the user's movements are captured by motion capture devices in the data acquisition module, obtaining a sequence of coordinates in three-dimensional space. Let the user's movements occur within the time interval [t0, t...]. n The sequence of action coordinates within the range is (x1, y1, z1), (x2, y2, z2), ..., (x... n y n , z n ), to obtain the coordinates (x, y) of the user's hand in three-dimensional space at time point i. i y i , z i );

[0071] Second, through the collected coordinates (x i y i , z i ) Analyze the displacement components between two adjacent action key points and square them to eliminate the influence of positive and negative displacement components. Then, consider the displacements in three directions together to obtain the actual displacement distance between two adjacent key points. Finally, sum the displacement distances between all adjacent time points to output the motion path length value of the user in the entire action process.

[0072] Third, through the collected coordinates (x i y i , z i Analyze the displacement vectors between adjacent coordinate points and the displacement vectors between the next set of adjacent coordinate points, and calculate the magnitude of the corresponding vectors. Then, use the vector dot product operation to process the vectors, divide the dot product result by the product of the magnitudes of the two vectors, and then use the inverse cosine function to process the vectors, and use the summation function to sum the vectors to obtain the cumulative angle of the direction change during the entire action process, and calculate the average value to output the degree of change in the direction of the action.

[0073] Fourth, by using the velocity change between two adjacent action time points and the time interval between two adjacent action time points, an approximate value of acceleration is obtained, and the approximate value of acceleration is placed in the denominator to reflect the negative impact of acceleration on the smoothness of the action, so as to output the smoothness factor of the action.

[0074] Fifth, the frequency of occurrence of the k-th action type is combined with the average frequency of occurrence of the action type, and then the difference is summed to obtain the dispersion of the frequency of occurrence of the action, so as to output the action diversity factor.

[0075] Sixth, the motion path length value, motion direction change value, motion smoothness factor, and motion diversity factor are weighted and combined to output the motion complexity value.

[0076] The calculation formula for the action interaction submodule is as follows:

[0077] ASS=A1×AL+A2×AD+A3×(1-AS)+A4×(1-AU);

[0078]

[0079] in:

[0080] ASS refers to the action complexity value;

[0081] The motion complexity value (ASS) is an indicator that comprehensively considers factors such as motion path length, direction change rate, smoothness, and diversity. Its value range is usually between 0 and 1. This embodiment provides possible threshold values.

[0082] Low complexity threshold: ASS < 0.3;

[0083] When the Action Complexity (ASS) value is below 0.3, it indicates that the action is relatively simple and users may feel that it lacks challenge and have low participation. At this time, it is advisable to increase the complexity of the action, such as increasing the path length of the action, increasing the rate of change of direction, or introducing more diverse action types. For example, the system can increase the depth of operation, provide users with more advanced and complex function options, and encourage users to try diverse operations, such as adding gesture combination operations on the basis of simple touch operations, setting challenge tasks, and stimulating users' desire to explore.

[0084] High complexity threshold: ASS > 0.7;

[0085] When the action complexity value (ASS) is higher than 0.7, it indicates that the action is complex and the user may feel tired or find it difficult to complete the task. In this case, the task difficulty should be appropriately reduced, such as reducing the path length of the action, reducing the frequency of directional changes, or simplifying the diversity of actions. The system can simplify the operation process based on this result, remove unnecessary operation steps, such as merging multiple consecutive small operations into one large operation to reduce the difficulty of user operation. It can also provide operation prompts, guiding users to complete the operation through intuitive icons and symbols, just like when the user performs a complex gesture operation, the screen displays an animated demonstration of the standard gesture.

[0086] This submodule uses weighted summation to comprehensively consider factors such as motion path length, direction change rate, smoothness, and diversity to obtain a comprehensive motion complexity index.

[0087] AL refers to the motion path length value, which reflects the total displacement length of the user's action in space.

[0088] AD refers to the degree of change in the direction of the movement, reflecting the extent of the change in the direction of the movement.

[0089] AS stands for Motion Smoothness Factor, which reflects the smoothness of user actions. The closer the value of the Motion Smoothness Factor AS is to 1, the smoother the action.

[0090] AU stands for Motion Diversity Factor. The closer the value of the Motion Diversity Factor AU is to 1, the higher the diversity of motions.

[0091] A1, A2, A3, and A4 respectively represent the weight coefficients of the motion path length value, the motion direction change value, the motion smoothness factor, and the motion diversity factor;

[0092] n refers to the number of recorded action time points;

[0093] i refers to the sequence number in the time series, which varies from 1 to n and is used to mark the temporal order of adjacent action key point vectors.

[0094] (x i y i , z i This refers to the coordinates of a user's hand or key body points in three-dimensional space at time point i, measured in meters (m). These coordinates are obtained through motion capture devices. Common motion capture devices include wearable sensors based on inertial measurement units (IMUs), such as the IMU built into VR controllers, which can measure the position of the controllers in space in real time. There are also devices based on optical tracking technology, which place multiple cameras in space and calculate the three-dimensional coordinates of markers installed on the human body or objects using the principle of triangulation. There are also self-positioning full-body trackers worn in multiple locations on the body. The richer the user position information detected, the more accurate the analysis of user interaction information based on this system.

[0095] In a virtual reality environment, a user's actions can be captured by motion capture devices to obtain their coordinate sequence in three-dimensional space. Let the user's actions occur within a time interval [t0, tn], and the coordinate sequence within this time interval be (x1, y1, z1), (x2, y2, z2), ..., (x...). n y n , z n );

[0096] (x i-1 y i-1 , z i-1 (i) refers to the coordinates of the user's hand in three-dimensional space at time point i-1;

[0097] Refers to the displacement vector between adjacent coordinate points, in meters;

[0098] This represents the direction and magnitude of the user's movement within adjacent time periods;

[0099] Refers to the displacement vector between the next set of adjacent coordinate points, calculated in the same way as... They are the same, only the time point is shifted one position forward;

[0100] Referential vector The modulus, in meters, can be obtained from the formula for calculating the vector modulus, which is existing technology and will not be elaborated here.

[0101] Referential vector The module length, in meters;

[0102] AO refers to the preset adjustment weight coefficient, which is a pre-set constant that can be adjusted according to the actual situation. It is used to control the degree of influence of speed changes on the smoothness factor.

[0103] AU i This refers to the change in velocity between two adjacent points in time, measured in meters per second. It can be calculated by first determining the displacement of adjacent coordinate points, such as... Divide the modulus length by the time interval AW i By obtaining the speed of adjacent key points and then calculating the difference between adjacent speeds, the speed change between two adjacent action key points is obtained, which reflects the change in the user's speed within adjacent time periods.

[0104] AW i It refers to the time interval between two adjacent action time points, in seconds, and is obtained by the time recording function of the motion capture device, that is, the time difference between the data acquisition of two adjacent key points.

[0105] m1 refers to the number of action types, which is determined according to the classification criteria of the action. For example, if the action is divided into different types such as raising the hand, turning around, and kicking, then m1 is the total number of these types.

[0106] k refers to the index number of different action types, which varies from 1 to m1 and is used to distinguish different action types;

[0107] AUF refers to the average number of occurrences of a particular action type, which is obtained by summing the occurrences of all action types and then dividing by the total number of action types.

[0108] AUG k The number of occurrences of the k-th action type is obtained by classifying and statistically analyzing motion capture data. The action of each key point is compared with the preset action type template to count the number of occurrences of each action type.

[0109] (x i -x i-1 ), (y i -y i-1 ), (z i -x i-1i The ) refers to the displacement components between two adjacent action key points on the three coordinate axes x, y, and z, respectively. This is to measure the change of the user's position in each direction from one point in time to the next in three-dimensional space.

[0110] (x i -x i-1 ) 2 +(y i -y i-1 ) 2 +(z i -x i-1i ) 2 This term is in three-dimensional space. This formula calculates the square of the displacement between two adjacent key points. This is done to eliminate the influence of positive and negative displacement components and to consider the displacement in three directions together. Then, the square root of the above result is taken to obtain the actual displacement distance between two adjacent key points. The summation function of the previous term sums the displacement distances between all adjacent key points, thereby obtaining the total path length of the user in the entire action process. The larger this value is, the wider the range of action of the user in the virtual environment.

[0111] In this item This is a vector dot product operation. Here, the dot product yields the product of the cosine of the angle between two vectors and the magnitude of the two vectors. Dividing the dot product by the product of the magnitudes of the two vectors gives the cosine of the angle between the two vectors, cosθ. This is done to eliminate the influence of vector length and focus only on the directional relationship between the two vectors.

[0112] This section uses the inverse cosine function to convert the cosine value into the angle θ between two vectors, in radians. This angle represents the change in the user's direction of movement within adjacent time periods. The absolute value is used to ensure that the angle value is non-negative, because the magnitude of the change in direction is a positive number, regardless of whether the direction change is positive or negative.

[0113] The summation function of the preceding term in this section sums the angles of directional changes over all adjacent time intervals to obtain the cumulative angle of directional changes throughout the entire action.

[0114] This refers to calculating the average value. Since there are a total of n-1 adjacent time periods, the average rate of change of direction in each time period is obtained by dividing by n-1. The larger this value is, the more frequently the user's action direction changes.

[0115] This item is the change in velocity divided by the time interval to obtain an approximate value of acceleration. The greater the acceleration, the more drastic the change in the user's movement speed, and the less smooth the movement may be.

[0116] This item refers to This value decreases as acceleration increases, reflecting the negative impact of acceleration on the smoothness of movement. When the acceleration is 0, this term is 1, indicating that the movement is completely smooth. The summation function of the preceding term of this term sums the smoothness impact values ​​in all adjacent time periods.

[0117] This refers to calculating the square of the deviation between the frequency of each action type and the average value. Squaring is used to eliminate the positive and negative effects of the deviation and highlight its magnitude. The summation function sums the squares of the deviations of all action types to obtain the dispersion of the frequency of action occurrences. The greater the dispersion, the more uneven the distribution of action types.

[0118] m1×AUF 2 It refers to the process of normalizing the degree of dispersion so that the magnitudes of the denominator and numerator match.

[0119] The coefficient of variation is used to obtain the distribution of action types. The larger the value, the more concentrated the action types are on a few types. Subtracting the coefficient of variation from 1 gives the action diversity factor. The closer this value is to 1, the more diverse the action types are.

[0120] 1-AS refers to the degree of non-fluency because S is the fluency factor. The higher the degree of non-fluency, the more complex the action.

[0121] 1-AU refers to the degree of insufficiency of movement diversity because AU is the movement diversity factor. The greater the insufficiency, the more complex the movement.

[0122] In this embodiment: When a user operates on the touch screen, various actions are generated, such as finger swiping, clicking, zooming, etc. This submodule can quantitatively evaluate the path length, direction change rate, smoothness and diversity of these actions. For example, when a user performs complex multi-touch operations on the screen, this submodule can accurately calculate the complexity of the action. The system can adjust the interaction strategy based on this result, such as providing more appropriate feedback prompts or adjusting the difficulty of operation.

[0123] In this submodule, the motion path length value AL measures the total distance that the user's hand or body key points move in three-dimensional space during the execution of an action. It intuitively reflects the spatial coverage of the action. An excessively large motion path length value AL means that the user's operation range on the touch screen is wide and the range of motion is large. This may be because the user is making a large selection, drawing complex graphics, or performing a multi-step operation. In this case, the system can provide a more lenient operation error tolerance mechanism to avoid misoperation caused by a large operation range. For example, the response range of the operable area can be appropriately expanded. At the same time, the system can intelligently predict the user's next operation based on the user's motion path and provide corresponding prompts, such as popping up guiding icons or text prompts. An excessively small motion path length value AL indicates that the user's operation is more concentrated and may be performing some fine operations, such as clicking small icons, entering precise data, or making fine adjustments. The system can improve the accuracy of the operation and reduce the size of the operation response area to meet the user's needs for fine operation. For example, the error range of the icon click area can be reduced, or auxiliary tools such as a magnifying glass can be provided to facilitate users to operate more accurately.

[0124] The AD value, representing the degree of change in the direction of a user's actions, describes the frequency and magnitude of these changes. It reflects the flexibility and variability of the action in terms of direction. The more frequent and larger the changes in direction, the more frequently the user's actions on the touchscreen change direction. The operation is relatively flexible but may also be somewhat arbitrary. The user may be trying different operation methods or is in an exploratory operation phase. In this case, the system needs to analyze the user's operation intentions more deeply, combining context and historical operation records to accurately understand the user's needs. This can provide a more guided interactive interface, such as displaying the standard operation process or recommended operation path, or popping up a prompt box to ask the user about their specific operation goal. The more frequent and smaller the changes in direction, the more stable the user's action direction is. The user may be performing simple operations such as straight-line sliding or continuous clicking. The system can simplify the operation feedback process and improve the response speed to enhance the user's sense of smoothness. For example, it can reduce the number of operation confirmation steps and automatically merge and process continuous simple operations to reduce waiting time.

[0125] The motion smoothness factor (AS) measures the smoothness of motion execution, reflecting whether the motion is continuous and natural. Low smoothness indicates that the user's actions may be paused, hesitant, or slow, and the operation is not smooth enough. This may be due to the difficulty of the operation, the unreasonable interface layout, or the user's unfamiliarity with the operation. The control system can provide more operation prompts, optimize the interface layout, and reduce unnecessary operation steps to improve the user's operation smoothness. For example, display operation instructions near the operation area and readjust the position and size of interface elements to make the operation more convenient. High smoothness means that the user's actions are fast and continuous. Smooth operation indicates that the user is more proficient in the operation or the operation itself is simple and direct. The control system can maintain the current interaction rhythm and provide timely feedback to maintain a good user experience. For example, it can quickly respond to operations and give clear operation result prompts. It can also appropriately increase the difficulty or complexity of the operation according to the user's operation habits, such as providing more advanced function options or challenge modes.

[0126] The motion diversity factor AU reflects the richness of different types of motions contained in an action. When the motion type is simple, users are likely to form fixed motion patterns and the complexity is low. However, when the motion type is rich and diverse, users need to master more motion skills and switching methods, which increases the difficulty and complexity of the motion. Introducing this parameter can evaluate the complexity of the motion from the perspective of the richness of motion types.

[0127] The calculation result of this submodule, the Action Complexity Value (ASS), helps the control system understand the complexity of the user's current action. Based on the calculation result, the system can accurately adjust the operation process and difficulty to adapt to different users' skill levels and usage habits. Novice users will experience simpler and more direct operations, while experienced users will have the opportunity to explore more complex and advanced functions, making the system more widely applicable. This avoids users performing overly complex operations, thereby reducing the burden on the system when processing complex instructions, reducing system lag and response delays caused by complex operations, ensuring the smoothness and stability of system operation, helping to identify and eliminate unnecessary operation steps, simplifying operation logic, and making the interaction process more efficient. Users can complete tasks faster and improve operational efficiency.

[0128] When the action complexity value (ASS) is too high, the calculation result shows that the action complexity is high. The system can identify that the user's operation is difficult and can simplify the operation process accordingly. For example, multiple consecutive small operations can be combined into one large operation to reduce the user's operation steps. At the same time, operation prompts can be provided, such as displaying standard gesture animations when performing complex gesture operations, making the interaction smoother, reducing the user's learning cost, and improving the operation efficiency. If the action complexity value (ASS) is low, the system displays low action complexity and can increase the depth and diversity of operations to provide users with a richer interactive experience. For example, it can add gesture combination operations, multi-mode operations, etc., to meet the needs of different users and enable the system to adapt to more diverse user operation habits.

[0129] This submodule comprehensively considers factors such as motion path length, direction change rate, smoothness, and diversity to quantify the complexity of user actions. Path length reflects the range of actions, direction change rate reflects the frequency of changes in action direction, smoothness factor measures the stability of action speed changes, and diversity factor considers the richness of action types. In a touch screen panoramic interactive control system, understanding the complexity of user actions helps the system dynamically adjust the interaction content or feedback method according to the difficulty of user actions. For example, when the complexity of actions is high, the system can appropriately reduce the difficulty of operation or provide more prompts to improve the user experience. It can also assess the user's skill level based on the complexity of actions, providing a basis for personalized training or guidance.

[0130] Existing technologies may only focus on certain characteristics of a movement, such as movement speed or simple movement types. However, this formula comprehensively considers all aspects of a movement by using movement path length, rate of change of direction, smoothness factor, and diversity factor. This submodule quantifies these factors into movement complexity indicators through precise calculation and analysis. For example, it uses vector dot product and inverse cosine function to calculate the rate of change of direction, accurately measuring the change in movement direction; and it uses speed change and time interval to calculate the smoothness factor, scientifically reflecting the smoothness of the movement. This calculation method is more detailed and accurate, providing the system with more comprehensive movement information.

[0131] Please see Figures 1 to 2 The eye-tracking analysis submodule specifically includes:

[0132] First, the fixation frequency is analyzed by dividing the number of fixations in the j1th small region by the total number of fixations.

[0133] Second, the eye-tracking attention factor is output by measuring the fixation frequency and quantifying it numerically using a logarithmic function.

[0134] Third, by analyzing the user's total fixation time in the j1st small area and the duration of the user's i1st fixation in the j1st small area, the relative duration of each fixation in the target small area is analyzed, and the average value is calculated to output the fixation duration factor.

[0135] Fourth, the eye-tracking attention factor, fixation duration factor, and motion complexity value are weighted and combined to output a comprehensive eye-tracking attention regulation factor.

[0136] The calculation process of the eye-tracking analysis submodule is as follows:

[0137] BSS=B1×BE×(1+B2×ASS)×(1-B3×BT);

[0138]

[0139] in:

[0140] BSS stands for Eye Movement Attention Integration Factor;

[0141] The Eye Movement Attention Synchronization Factor (BSS) measures the degree of distraction in a user's eye movements, and its value typically ranges from 0 to 1. This embodiment provides a possible threshold:

[0142] Low entropy threshold: BSS < 0.2;

[0143] When the Eye Movement Attention Strain (BSS) value is below 0.2, it indicates that the user's eye movements are highly focused and the user pays close attention to a specific area. In this case, the displayed content can be further enriched, for example, by adding more relevant details around key information or guiding the user to explore other areas to increase the user's desire to explore.

[0144] The system can distribute information layout, rationally distributing important information in different areas of the screen, and introduce dynamic elements, such as flashing icons and animation effects, to attract users' attention; it also regularly updates the interface content to avoid user eye fatigue.

[0145] High entropy threshold: BSS > 0.6;

[0146] When the Eye Movement Attention Strain (BSS) value is higher than 0.6, it indicates that the user's eye movements are relatively scattered and it is difficult to concentrate. At this time, it is necessary to adjust the display position and method of key information, such as enlarging the key information, changing the color, or adding animation effects to attract the user's attention.

[0147] BE stands for Eye Movement Attention Factor, which is the original Eye Movement Attention Modulation Factor, reflecting the degree of eye movement distraction;

[0148] BT refers to the fixation duration factor;

[0149] B1, B2, and B3 refer to the weighting coefficients of the eye-tracking attention factor, the motion complexity value, and the fixation duration factor, respectively.

[0150] N represents the number of small regions, determined by the division of the virtual environment's display area. The eye tracker of the virtual reality device is used to obtain the user's gaze coordinates (x, y) within the virtual environment. i ,y i This divides the display area of ​​the virtual environment into N smaller areas;

[0151] j1 refers to the sub-region number of the virtual environment display area. It is an integer, ranging from 1 to N, and has no unit. It is used to distinguish different sub-regions.

[0152] BF j1 Refers to fixation frequency; it represents the proportion of the number of fixations in the j1-th sub-region to the total number of fixations, reflecting the degree of attention the user pays to that sub-region.

[0153] BN j1 This refers to the number of user fixation points in the j1th small region. The coordinates of the user's fixation points are recorded by an eye tracker. Then, based on the pre-divided small regions, the virtual environment display area is divided into grids. The small region to which each fixation point belongs is determined and the results are statistically obtained.

[0154] m2 refers to the total number of fixations of the user, which is the total number of fixations recorded by the eye tracker;

[0155] BTW j1 This refers to the total fixation time of the user in the j1-th subregion, in seconds, calculated by the BRR of each fixation duration within that subregion. i1，j1 Summation yields the result;

[0156] i1 refers to the user's gaze sequence number in the j1-th sub-region, used to mark different gaze events within that sub-region;

[0157] BRR i1，j1 The duration of the i1th fixation of the user in the j1th small area is measured in seconds and is recorded by the eye tracker. It is the length of time from when the user begins to fixate on the small area until the user shifts their gaze.

[0158] log2(BF j1 Taking the logarithm to base 2 is a common practice in information processing. In information processing, the logarithmic function is used to measure the uncertainty of an event. j1 This represents the user's gaze frequency in the j1-th subregion, when BF j1 When the value is close to 1, it indicates that most of the user's gaze is concentrated in this area, with low uncertainty. When BF j1 When the value is close to 0, it indicates that the user rarely looks at the area, and the uncertainty is high. The logarithmic function can quantify this uncertainty.

[0159] The fixation frequency is multiplied by its logarithm, and the uncertainty of each small region is weighted. Regions with higher fixation frequencies contribute more to the overall entropy. The summation is then taken as a summation to integrate the uncertainties of all small regions, and the negative sign is because the logarithmic function is effective when 0 < BF. j1 The value of <1 is negative. Taking the negative sign makes the eye-tracking attention factor value non-negative. The larger the eye-tracking attention factor BE is, the more scattered the user's eye movements are and the less focused their attention is. The smaller the eye-tracking attention factor BE is, the more concentrated the user's eye movements are in certain areas and the more focused their attention is.

[0160] It refers to the average relative duration of each fixation by a user within a small target area;

[0161] This refers to calculating the average value. Since there are N small regions in total, dividing by N gives the average relative duration factor of each fixation during the entire eye movement process. The larger this value is, the longer the user's fixation duration is, and the more focused their attention may be.

[0162] (1+B2×ASS) refers to the complexity of the action. The more complex the action, the more it may affect the distribution of the user's eye movement attention, making the eye movement more scattered. Therefore, (1+B2×ASS) is used to amplify the eye movement attention factor BE.

[0163] The calculation of (1-B3×BT) is because BT is the fixation duration factor. (1-B3×BT) represents the degree of fixation duration insufficiency. The greater the fixation duration insufficiency, the more scattered the eye movements may be, and the more the eye movement attention factor BE will be amplified.

[0164] In this embodiment: the panoramic content presented on the touch screen will attract the user's attention, and the user's eye movement behavior will leave gaze points on the screen. This submodule can analyze these gaze points and calculate indicators such as the frequency of the user's gaze and attention to different areas of the screen. The system can understand the user's focus based on these indicators, optimize the layout and display of the screen content, and improve the interaction efficiency.

[0165] The gaze frequency BF calculated by this submodule j1 The fixation frequency (BF) represents the proportion of a user's gaze points within a small area of ​​a virtual environment out of the total number of gaze points, reflecting the degree of user attention to that area. j1 A high gaze frequency indicates that the user is more interested in or needs more attention from that area. This area may contain important information or key task objectives. The system can further highlight the information in this area, such as increasing brightness, enlarging the display, or changing the color. It can also push relevant content or services based on the user's gaze preferences, for example, displaying more relevant detailed information or recommended content around the area. j1 A low value indicates that the user's attention to this area is low. This may be because the information in this area is not attractive enough or the layout is unreasonable. The system can consider adjusting the content or position of this area, or optimizing the display effect. Introducing this parameter can help understand the distribution of the user's attention in the virtual environment.

[0166] Eye Attention Factor (BE) is an indicator that measures the degree of eye distraction in a user. It is calculated based on fixation frequency. A higher BE value indicates that the user's eye movements are more scattered and their attention is not focused enough. This may be due to overly complex interface content or a lack of clear guidance. Control systems can simplify interface content, reduce distracting information, and highlight key areas. They can also use dynamic guidance to attract the user's attention to key information, such as using animation effects or flashing prompts to guide the user's gaze. A lower BE value indicates that the user's eye movements are more focused on certain areas, and their attention is more concentrated. The system can provide more in-depth interactive services based on the content in these areas, such as detailed information displays and related operation suggestions. For example, a detailed explanation box or more operation options can be displayed in the area that the user is focused on. In human-computer interaction, the BE value can reflect the user's information processing method and cognitive load in the virtual environment. A high BE value may indicate that the user is searching for information or is confused about the environment, while a low BE value may indicate that the user has clearly identified the focus. Introducing this parameter can quantify the degree of eye distraction in a user, providing a reference for assessing the user's attention state and optimizing interface design.

[0167] The gaze duration factor (BT) measures the average gaze duration of a user in each small area, reflecting the depth of the user's attention to different areas. A longer gaze duration may indicate that the user has engaged in in-depth information processing or thinking in that area, while a shorter gaze duration may indicate that the user is just browsing quickly. The gaze duration factor can help determine the user's interest and importance in different areas. Introducing this parameter can further analyze the user's attention distribution from the dimension of gaze duration.

[0168] The Eye-Motion Attention Comprehensive Modulation Factor (BSS) calculated by this submodule reflects the user's degree of eye distraction and concentration. The control system can adjust the key content of the display interface based on the value of the BSS. When the BSS is large, it indicates that the user's attention is distracted, and key information is highlighted or prompts are added. In virtual teaching systems in the field of education, the teaching progress and methods are adjusted according to the student's BSS.

[0169] When the Eye Movement Attention Strain Factor (BSS) is too high, the interface design can be optimized to highlight key information; reminder mechanisms, such as sound prompts or flashing effects, can be added to attract the user's attention. Generally, the lower the BSS, the better. A lower BSS indicates that the user's eye movements are more focused and their attention is more concentrated, enabling them to better participate in the human-computer interaction process. This submodule quantifies the distribution of the user's eye movement attention through fixation frequency, eye movement attention entropy, and fixation duration factor. Fixation frequency reflects the proportion of the user's attention to different areas, the BSS measures the degree of eye movement dispersion, and the fixation duration factor (BT) reflects the degree of fixation concentration. This control system can optimize the layout and content display of the virtual scene according to the user's eye movement attention distribution, such as placing important information in areas where the user's attention is concentrated, or dynamically adjusting the scene content according to the user's fixation, thereby improving the efficiency of the user's information acquisition and enhancing the sense of immersion.

[0170] Existing technologies may pay less attention to the details and distribution of eye movements, while this submodule introduces fixation frequency, eye-movement attention entropy, and fixation duration factors to analyze the user's eye-movement attention fixation frequency (BF) from multiple perspectives. j1 The eye-tracking attention factor (BE) and fixation duration factor (BT) are used to analyze the user's eye-tracking attention from multiple perspectives.

[0171] Please see Figures 1 to 2 The interactive immersive submodule is specifically as follows:

[0172] First, the eye-tracking attention modulation factor and the motion complexity value are weighted and combined, and then input into the interactive immersion submodule;

[0173] Second, based on the accuracy, completeness, and fluency of user actions, user action data is compared with template action data to output an environmental adaptation factor. Then, the environmental adaptation factor, eye-tracking attention comprehensive adjustment factor, and action complexity value are combined to output a human-computer interaction immersion index.

[0174] The computational processing of the interactive immersion submodule is as follows:

[0175] IOGP=C1×ASS+C2×(1-BSS)+C3×IOA;

[0176] in:

[0177] IOGP refers to the Human-Computer Interaction Immersion Index;

[0178] The Immersion Index for Human-Computer Interaction (IOGP) is a comprehensive indicator that takes into account factors such as motion complexity, eye-tracking attention concentration, and environmental adaptability. Its value range is generally between 0 and 1. This embodiment provides a possible threshold.

[0179] Low immersion threshold: IOGP < 0.4;

[0180] When the IOGP value of human-computer interaction is lower than 0.4, it indicates that the user's immersion is low and may be dissatisfied with the current interactive experience. At this time, the entire virtual reality touch screen panoramic interactive control system can be fully optimized, such as adjusting the ambient sound effects, enhancing the visual effects and improving the tactile feedback, in order to improve the user's immersion.

[0181] High immersion threshold: IOGP > 0.8;

[0182] When the Immersion Index (IOGP) of Human-Computer Interaction is higher than 0.8, it indicates that the user is highly immersed and very satisfied with the interaction experience. On the basis of maintaining the current good experience, we can further explore new interaction methods or add more challenges to continuously improve user engagement.

[0183] IOA stands for Environment Adaptability Factor, which is obtained by evaluating the similarity between user actions and preset action templates in the virtual environment. User action data can be compared with preset templates. The higher the similarity, the closer the IOA value is to 1. It can be comprehensively evaluated from three aspects: accuracy, completeness and fluency of the action.

[0184] The Environmental Adaptation Factor (IOA) is further explained below;

[0185] Suppose we decompose the preset actions in the virtual environment into V basic action units, and when the user completes the corresponding action, there is a corresponding evaluation index for each basic action unit;

[0186] Let aS represent the score of the user completing the S-th basic action unit, with a value range of [0,1]. The specific scoring rules are as follows:

[0187] The accuracy score aS1 measures the degree of matching between a user's action and a preset action in terms of spatial position and posture. It is obtained by acquiring the coordinate information of the user's action through motion capture equipment, comparing it with the standard coordinates of the preset action, and calculating the error between the two. The smaller the error, the higher the accuracy score. Assuming the error is ek and emax is the preset maximum allowable error, the formula for calculating the accuracy score aS1 is as follows:

[0188]

[0189] The completeness score aS2 is used to determine whether a user has completed all the key steps of a preset action. If the user has completed all the key steps of the basic action unit, then aS2 = 1. If some steps are missing, points are deducted according to the proportion of the missing steps. For example, if the proportion of missing steps is pk, then the formula for calculating the completeness score aS2 is as follows:

[0190] aS2 = 1 - pk;

[0191] The fluency score aS3 assesses the smoothness of a user's actions. It is scored based on factors such as speed variations and the number of pauses during the action. A higher fluency score is achieved with smoother speed variations and fewer pauses. A fluency metric sk can be defined, and its relationship with the fluency score can be determined through experimentation or experience. smax is the maximum fluency metric value. The formula for calculating the fluency score aS3 is as follows:

[0192]

[0193] The formula for calculating the score of the Sth basic action unit is as follows:

[0194] aS=a1×aS1+a2×aS2+a3×aS3;

[0195] Where a1, a2 and a3 are the weight coefficients of aS1, aS2 and aS3, respectively;

[0196] In summary, the formula for calculating the Environmental Adaptability Factor (IOA) is as follows:

[0197]

[0198] C1, C2, C3, and C4 refer to the weighting coefficients of the motion complexity value ASS, the eye-tracking attention comprehensive adjustment factor BSS, and the environmental adaptation factor IOA, respectively.

[0199] (1-BSS) refers to the degree of eye-tracking attention concentration because BSS is the comprehensive regulatory factor for eye-tracking attention. The more concentrated the attention, the stronger the sense of immersion.

[0200] In this embodiment: This submodule comprehensively considers action complexity, eye movement attention and other factors to evaluate the user's immersion during interaction with the touch screen. By optimizing the interactive performance of the touch screen, such as response speed and tactile feedback, and the content presentation effect, such as image quality and color reproduction, the user's action fluency and eye movement attention concentration can be improved, thereby enhancing the immersive experience of human-computer interaction.

[0201] In the calculation of human-computer interaction immersion, action complexity is a factor that reflects the difficulty of the user's actions during operation. Appropriate action complexity can increase the user's sense of participation and challenge, thereby improving immersion. However, if the actions are too complex, the user may feel frustrated and tired, reducing immersion. Therefore, action complexity needs to be controlled within a suitable range. Introducing this parameter can affect the immersion of human-computer interaction from the perspective of action difficulty.

[0202] (1-BSS) represents the degree of eye-tracking attention concentration, which reflects whether the user can concentrate in a virtual environment. The more concentrated the attention, the more fully the user can immerse themselves in the interaction, and the stronger the sense of immersion. When the user focuses on completing a virtual task, the eye-tracking attention is concentrated, and the user can better experience the virtual environment. Introducing this parameter can affect the sense of immersion in human-computer interaction from the perspective of attention concentration.

[0203] The Environment Adaptability Factor (IOA) measures the similarity between user actions and preset action templates in the virtual environment, reflecting the degree of fit between the user and the virtual environment. When the user's actions are highly adapted to the virtual environment, the user will feel that they are truly integrated into the virtual world, enhancing the sense of immersion. In virtual reality games, if the player's actions can naturally match the actions of the game character, the player will have a more realistic experience. Based on this interactive immersion submodule, the sense of immersion in human-computer interaction can be affected from the perspective of the user's interaction adaptability to the environment.

[0204] This submodule comprehensively considers factors such as motion complexity, eye-tracking attention, and environmental adaptability. It is an important indicator for measuring the user's immersion during human-computer interaction. The control system can optimize the interactive experience based on the IOGP value, such as adjusting the sound effects and lighting effects of the virtual environment to enhance the user's immersion. Based on the calculation results of this submodule, the quality of the virtual environment can be improved, such as adding details and optimizing lighting effects; the challenge of tasks can be adjusted to be neither too easy nor too difficult; and the environment and tasks can be adjusted in real time based on the user's motion and eye-tracking data to increase user participation. For example, in an immersive movie experience, when the user's immersion is low, the rhythm of the movie plot can be adjusted or interactive elements can be added. When the IOGP is high, it indicates that the user is highly immersed in the human-computer interaction process, has a good experience, and can participate more deeply in the virtual environment. This submodule can help the system evaluate the user's immersion experience and adjust system parameters based on the calculation results, such as optimizing the realism of the virtual environment to enhance the user's immersion and make the interaction more natural and efficient.

[0205] Existing technologies may have a relatively singular approach to assessing the immersion of human-computer interaction, focusing primarily on one aspect, such as visual effects or auditory feedback. This submodule comprehensively considers multiple factors, including motion complexity, eye movement attention, environmental adaptability, and task completion, to more fully measure immersion. By integrating multiple factors through a weighted summation method, it can flexibly adjust according to the importance of different factors, thus more accurately reflecting the user's immersive experience during the interaction process.

[0206] It is worth noting that further calculations based on the Human-Computer Interaction Immersion Index (IOGP) are used to influence the weight coefficient A1 of the motion path length value in the motion interaction sub-module, thereby adjusting the influence ratio of the motion path length value AL in the motion interaction sub-module, and thus optimizing and adjusting the calculation results of the motion complexity value ASS.

[0207] First: A1 new =A1 old +[α×(IOGP-IOGP)];

[0208] Secondly: Set the iteration termination condition:

[0209] Termination condition 1: The number of iterations is 20;

[0210] Termination condition two: |ROE new -ROE old | < 0.001;

[0211] in:

[0212] A1 new The weight coefficient refers to the action path length value after iteration;

[0213] A1 old The weight coefficient refers to the action path length value before the iteration.

[0214] α refers to the adjustment coefficient, which ranges from 0 to 1 and is used to control the adjustment range of the action path length value weight coefficient in each iteration;

[0215] IOGP refers to the threshold for comparing the immersion index of human-computer interaction.

[0216] In this embodiment: This iterative form, by continuously iterating the motion path length value weight coefficient A1new, can gradually adjust the influence of the motion path length value AL on the motion complexity value ASS, thereby optimizing the human-computer interaction immersion index IOGP. As the iteration proceeds, the human-computer interaction immersion index IOGP will gradually approach a local optimum, thereby improving the immersion of human-computer interaction. This iterative form directly targets the key factor affecting the immersion of human-computer interaction, adjusting the motion path length weight in motion complexity, and can more effectively optimize the immersion index.

[0217] The Immersive Human-Computer Interaction Index (IOGP) in the Immersive Interaction submodule includes the Action Complexity Value (ASS). The motion path length value (AL) weighting coefficient (A1) in the Action Interaction submodule is one of the weights affecting the ASS value. When the motion path length value weighting coefficient A1 increases, the influence of the motion path length value (AL) on the ASS value increases, which may in turn affect the ASS value. Since the Immersive Human-Computer Interaction Index (IOGP) and the ASS value are positively correlated, changes in the motion path length value weighting coefficient A1 will indirectly affect the Immersive Human-Computer Interaction Index (IOGP). The IOGP (Intensity of Gaining Perception) index is determined by the motion path length value. Therefore, adjusting the motion path length value weighting coefficient A1 can optimize the IOGP. The reason for choosing the motion path length value weighting coefficient A1 instead of other parameters is that the motion path length value AL is a relatively intuitive and important factor affecting motion complexity. Adjusting its weight can more directly affect motion complexity and thus affect the immersion of human-computer interaction. Although other parameters also affect motion complexity and immersion of human-computer interaction, the correlation between the motion path length value weighting coefficient A1 and the IOGP is relatively more direct and obvious.

[0218] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A virtual reality-based touchscreen panoramic interactive control system, characterized in that, include: Data acquisition module: used to collect motion data, eye movement data, and motion type data; Motion data information includes: the coordinates of the user's hand in three-dimensional space, the displacement vector between adjacent coordinate points, and the velocity change between two adjacent action time points; Eye-tracking data includes: the number of fixations and total fixation time for the user in the j1st subregion; Action type data information includes: the number of action types. Data processing module: This module is used to input the data collected by the data acquisition module, clean and correct the input data, and input the processed data into the management and analysis module. Management and analysis module; Based on the user's hand coordinates in three-dimensional space, the displacement vector between adjacent coordinate points, the velocity change between two adjacent action time points, and the number of action types, the action path length value, the action direction change value, the action smoothness factor, and the action diversity factor are calculated respectively. The calculated values ​​are then weighted and integrated to output the action complexity value. Based on the number of fixations in the j1st subregion, the fixation frequency and eye-tracking attention factor are calculated. Then, based on the total fixation time in the j1st subregion, the fixation duration factor is calculated. Finally, the eye-tracking attention factor and the fixation duration factor are weighted and integrated to output the comprehensive eye-tracking attention regulation factor. The human-computer interaction immersion index is output by weighting and combining factors such as motion complexity, eye-tracking attention regulation, and environmental adaptation. Interactive control module: Used to input the output values ​​of the management and analysis module, and to optimize the display layout and content presentation of the virtual environment accordingly.

2. The virtual reality-based touchscreen panoramic interactive control system according to claim 1, characterized in that: The management and analysis module includes: a motion interaction submodule, an eye-tracking analysis submodule, and an interactive immersion submodule.

3. The virtual reality-based touchscreen panoramic interactive control system according to claim 2, characterized in that: The processing procedure of the action interaction submodule is as follows: S1, In a virtual reality environment, the user's actions are captured by motion capture devices in the data acquisition module, obtaining a sequence of coordinates in three-dimensional space. Let the user's actions occur within the time interval [t0, t...]. n The sequence of action coordinates within the range is (x1, y1, z1), (x2, y2, z2), ..., (x... n y n , z n ), to obtain the coordinates (x, y) of the user's hand in three-dimensional space at time point i. i y i , z i ); S2, through the collected coordinates (x i y i , z i ) Analyze the displacement components between two adjacent action key points and square them to eliminate the influence of positive and negative displacement components. Then, consider the displacements in three directions together to obtain the actual displacement distance between two adjacent key points. Finally, sum the displacement distances between all adjacent time points to output the motion path length value of the user in the entire action process. S3, through the collected coordinates (x i y i , z i Analyze the displacement vectors between adjacent coordinate points and the displacement vectors between the next set of adjacent coordinate points, and calculate the magnitude of the corresponding vectors. Then, use the vector dot product operation to process the vectors, divide the dot product result by the product of the magnitudes of the two vectors, and then use the inverse cosine function to process the vectors, and use the summation function to sum the vectors to obtain the cumulative angle of the direction change during the entire action process, and calculate the average value to output the degree of change in the direction of the action. S4 obtains an approximate value of acceleration by using the velocity change between two adjacent action time points and the time interval between two adjacent action time points, and places the approximate value of acceleration in the denominator to reflect the negative impact of acceleration on motion smoothness, so as to output the motion smoothness factor. S5 combines the occurrence count of the k-th action type with the average occurrence count of the action type, and then sums the difference to obtain the dispersion of the occurrence count of the action, so as to output the action diversity factor. S6 combines the motion path length value, motion direction change value, motion smoothness factor, and motion diversity factor in a weighted manner to output the motion complexity value.

4. The virtual reality-based touchscreen panoramic interactive control system according to claim 3, characterized in that: The processing procedure of the eye-tracking analysis submodule is as follows: S1, the fixation frequency is analyzed by dividing the number of fixations in the j1th small region by the total number of fixations. S2 outputs the eye-tracking attention factor by quantifying the fixation frequency using a logarithmic function; S3 analyzes the relative duration of each gaze in the target small area by the user's total gaze time in the j1-th small area and the duration of the user's i1-th gaze in the j1-th small area, and calculates the average value to output the gaze duration factor. S4 combines the eye-tracking attention factor, fixation duration factor, and motion complexity value in a weighted manner to output a comprehensive eye-tracking attention regulation factor.

5. The virtual reality-based touchscreen panoramic interactive control system according to claim 4, characterized in that: The processing procedure of the interactive immersive submodule is as follows: S1, weighted combination of eye-tracking attention adjustment factor and motion complexity value, input to interactive immersion submodule; S2 compares user action data with template action data based on the accuracy, completeness, and fluency of user actions to output an environmental adaptation factor. Then, the environmental adaptation factor, eye-tracking attention comprehensive adjustment factor, and action complexity value are combined to output the human-computer interaction immersion index.

6. The virtual reality-based touchscreen panoramic interactive control system according to claim 5, characterized in that: The interactive control module includes: a motion interaction submodule analysis section, an eye-tracking analysis submodule analysis section, and an interactive immersion submodule analysis section; The analysis section of the action interaction submodule specifically includes: Based on the calculation results of the action complexity value, the action requirements in the virtual environment are adjusted. When the action complexity value is high, it will cause users to feel tired. The operation steps are simplified and unnecessary steps are removed. When the action complexity value is low, the user's operation is too simple and cannot fully utilize the system's functions, resulting in a monotonous user experience and reduced interest in the system. The action complexity is increased to improve user participation and immersion.

7. The virtual reality-based touchscreen panoramic interactive control system according to claim 6, characterized in that: The eye-tracking analysis submodule specifically includes the following analysis components: Based on the calculation results of the eye-tracking attention comprehensive adjustment factor, the display layout and content presentation of the virtual environment are optimized. When the eye-tracking attention comprehensive adjustment factor is high, it means that the user's gaze on the screen is relatively scattered, and the user cannot concentrate on important information, which affects the interaction effect. The display position and method of key information should be adjusted to attract the user's attention. When the eye-tracking attention comprehensive adjustment factor is low, the display content should be further enriched to enhance the user's desire to explore.

8. The virtual reality-based touchscreen panoramic interactive control system according to claim 6, characterized in that: The analysis section of the interactive immersion submodule specifically includes: Based on the calculation results of the human-computer interaction immersion index, the entire virtual reality touch screen panoramic interactive control system is comprehensively optimized. When the human-computer interaction immersion index is high, it indicates that the user's immersion is high and they are very satisfied with the interactive experience. On the basis of maintaining the current good experience, we should further explore and optimize the interaction methods to continuously improve user participation. When the human-computer interaction immersion index is low, it indicates that the user's immersion is low and the user is not satisfied with the current interactive experience. At this time, the entire virtual reality touch screen panoramic interactive control system needs to be comprehensively optimized. We need to adjust the ambient sound effects, enhance the visual effects, and improve the tactile feedback to improve the user's immersion.

Images