A VR / AR interaction method, system and storage medium

Abstract

The application discloses a VR / AR interaction method and system and a storage medium, and relates to the technical field of VR / AR human-computer interaction; the technical points are as follows: by constructing an invalid interaction infection chain map, originally isolated invalid interaction events are associated into a network with causal propagation relationship; in the interaction intention recognition process, not only the characteristic performance of the current interaction is analyzed, but also the position and propagation path of the current interaction in the infection chain are tracked; potential risks can be identified and blocked in the early stage of the chain mis-touch, and the problem that the prior art cannot cope with continuous mis-touch is solved; secondly, the traditional binary processing logic of positive effective feature matching plus single-point noise filtering is abandoned, and a technical route of calculating preliminary confidence first and then performing reverse calibration through the infection chain is adopted; even in the case of systematic abnormality of full-modal data, abnormal interaction modes can be identified through the propagation law of the infection chain, and the interaction stability in a complex environment is greatly improved.

Description

Technical Field

[0001] This invention relates to the field of VR / AR human-computer interaction technology, specifically to a VR / AR interaction method, system, and storage medium. Background Technology

[0002] With the deep application of VR / AR technology in consumer entertainment, industrial simulation, education and training, and other fields, multimodal interaction has become the mainstream interaction method to replace traditional controller operation. Existing technologies generally follow a standardized processing flow when implementing multimodal interaction: First, various sensors simultaneously collect multimodal raw data such as gestures, eye movements, and voice; feature vectors that match preset valid interaction actions are extracted; the feature vectors are input into a pre-trained classification model for matching calculation; the initial confidence level of the interaction intent is output; finally, a fixed threshold is used to filter out single-point noise data with low confidence levels, and the data is determined to be valid interaction and the corresponding operation is executed.

[0003] However, during the implementation of the above technical solution, at least the following technical problems were discovered: First, it assumes that all interactive events are independent random events, ignoring the causal propagation relationship between invalid interactions. For example, when a user accidentally touches something due to hand tremors or eye shift, similar actions that occur consecutively in a short period of time will be repeatedly judged by the system as independent events, which cannot stop the chain reaction and leads to frequent accidental touches. Secondly, a binary processing logic combining positive effective feature matching and single-point noise filtering is adopted. All filtering mechanisms are designed for single isolated invalid interactions. When the scene environment changes abruptly, such as strong light shining directly on the sensor or the device being violently impacted, it will cause systemic anomalies in the full modality data. At this time, the single-point filtering mechanism will fail completely, thus triggering a wide range of erroneous operations. To address this, we provide a VR / AR interaction method, system, and storage medium. Summary of the Invention

[0004] To achieve the above objectives, the present invention provides the following technical solution: A VR / AR interaction method, the steps of which are as follows: Time synchronization is achieved by using a globally unified clock to collect multiple sets of multimodal interaction data in VR / AR interaction scenarios. Based on a pre-built invalid interaction contagion chain graph, the propagation characteristics of all active contagion chain nodes in the current scenario are extracted to generate a differentiated interaction processing cycle adapted to the current scenario. Based on the differentiated interaction processing cycle, and taking the complete time window of the infection chain node as the benchmark, a multi-batch, multi-dimensional interaction feature set containing the characteristics of the infection chain node is extracted; A dynamic weight fusion strategy based on the node level of the infection chain is adopted to perform hierarchical integration and analysis on multi-batch multi-dimensional interaction feature sets to obtain multi-dimensional interaction feature fusion values. Combining the standard temporal causal logic of effective interaction, the initial confidence level of interaction intent is calculated based on the fusion value of multi-dimensional interaction features. By matching the complete propagation path of the invalid interaction contagion chain and calculating the intent decay gradient, the confidence of the initial interaction intent is reverse-calibrated to generate the final interaction intent confidence. Based on the real-time activity of the invalid interaction contagion chain, the initial interaction trigger threshold is dynamically adjusted according to the preset threshold adjustment rules. The final interaction intent confidence is compared with the dynamically adjusted interaction trigger threshold. If the final interaction intent confidence is greater than the dynamically adjusted interaction trigger threshold, it is determined to be a valid interaction intent and the corresponding interaction response operation is executed. Otherwise, it is determined to be an invalid interaction intent and the current VR / AR scene state is maintained. At the same time, the invalid interaction is updated to the invalid interaction contagion chain graph as a new node according to the preset rules.

[0005] Furthermore, the steps for processing multiple sets of multimodal interaction data in VR / AR interaction scenarios using a globally unified clock are as follows: Identify various multimodal sensor ports in VR / AR interactive scenarios; All sensor ports are controlled by a global unified clock to synchronously collect multiple sets of raw interaction data and corresponding interaction context associated data. The interaction context associated data includes continuous historical interaction data for a preset duration before the interaction occurs, scene environment data when the interaction occurs, and continuous subsequent interaction data for a preset duration after the interaction occurs. Multi-dimensional physical features of each set of original interaction data were extracted and divided into gesture action data, eye tracking data, voice command data, tactile feedback data, and spatial positioning data.

[0006] Furthermore, the step of generating a differentiated interaction processing cycle adapted to the current scenario is as follows: From the invalid interaction infection chain graph, all nodes that have triggered invalid interactions within a preset statistical period are selected as active infection chain nodes; Calculate the propagation speed of each active infection chain node, where the propagation speed is the average time interval from the node's first appearance to triggering the first subsequent invalid interaction; The basic processing cycle for each active infection chain node is determined based on its corresponding data type. Based on the hierarchical relationship of the infection chain nodes, the basic processing cycle is shortened and adjusted step by step, with different levels of nodes corresponding to different shortening rates; By integrating and adjusting various data processing cycles, a differentiated interactive processing cycle adapted to the current scenario is generated.

[0007] Furthermore, the steps for extracting multi-batch, multi-dimensional interaction feature sets containing infectious chain node features are as follows: Based on the differentiated interaction processing cycle, assign independent data processing tasks to each type of multimodal data and configure the corresponding processing parameters; Locate the original data storage nodes corresponding to various types of data according to the processing parameters, and formulate a data processing sequence for each original data storage node, including processing order, number of processing times and processing interval; Based on the data processing sequence, the original data is subjected to batch feature extraction to obtain the single-dimensional interaction features corresponding to each type of data; and further, the infection chain node features corresponding to each single-dimensional interaction feature are extracted. The infection chain node features include the similarity between the feature and historical invalid interaction features, and the spatiotemporal matching degree between the feature and the infection chain node. The single-dimensional interaction features and infection chain node features of various types of data are aligned in time and spatially associated with the timestamp of a globally unified clock to form a multi-batch, multi-dimensional interaction feature set.

[0008] Furthermore, the steps for obtaining the multi-dimensional interactive feature fusion value are as follows: Feature structure information and feature content information are extracted from multiple batches of multi-dimensional interactive feature sets. Specifically, feature structure consistency evaluation value is calculated based on feature structure information, and multi-dimensional content quality indicators are obtained based on feature content information. From the invalid interaction contagion chain graph, identify all contagion chain nodes whose feature matching degree with the current feature set reaches a preset matching threshold; The weighting coefficients of each dimension of content quality indicators are dynamically adjusted based on the level and propagation intensity of each matching node. By combining the evaluation values ​​of each dimension with the adjusted weight coefficients, the feature content validity evaluation value is calculated; then, the feature structure consistency evaluation value and the feature content validity evaluation value are weighted and fused to obtain the multi-dimensional interactive feature fusion value.

[0009] Furthermore, the final interaction intent confidence generation step is as follows: Extract all complete transmission paths containing a preset number of nodes from the invalid interaction transmission chain graph; The temporal and spatial features of the interaction feature set to be verified are matched with all complete transmission chains in a step-by-step sliding match. If the matching degree of any segment reaches the preset transmission chain matching threshold, it is determined that the initial interaction intention is on the transmission path of the invalid interaction transmission chain, and the transmission distance of the interaction feature on the transmission path is calculated. The intention decay gradient coefficient is calculated based on the propagation distance. Then, the initial interaction intention confidence is multiplied by the intention decay gradient coefficient to obtain the final interaction intention confidence. If the matching degree of all segments does not reach the preset infection chain matching threshold, the initial interaction intention confidence is directly used as the final interaction intention confidence.

[0010] Furthermore, the step of dynamically adjusting the initial interaction trigger threshold according to the real-time activity of the invalid interaction contagion chain and a preset threshold adjustment rule is as follows: The total number of times all active infection chain nodes in the invalid interaction infection chain graph are triggered within the preset statistical period is used as the real-time activity of the infection chain. The threshold adjustment coefficient is calculated based on the real-time activity of the infection chain; the initial interaction trigger threshold is multiplied by the threshold adjustment coefficient to obtain the dynamically adjusted interaction trigger threshold. When the real-time activity of the infection chain is 0, the dynamically adjusted interaction trigger threshold is equal to the initial interaction trigger threshold.

[0011] Furthermore, the step of updating the invalid interaction infection chain graph as a new node according to preset rules is as follows: Extract the multi-dimensional features of the invalid interaction and the corresponding interaction context association data, generate candidate infection chain nodes, and calculate the feature similarity between the candidate infection chain nodes and all existing nodes in the invalid interaction infection chain graph. If there are existing nodes with a similarity threshold that reach the preset merging threshold, the candidate node is merged into the existing node, and the propagation strength and trigger count of the existing node are updated; if there are no existing nodes with a similarity threshold that reach the preset merging threshold, the candidate node is added as a first-level infection node to the invalid interaction infection chain graph. If the invalid interaction is caused by other invalid interactions, a propagation association is established between the candidate node and the initiating node, and the level of the candidate node is set to the level of the initiating node plus 1.

[0012] A VR / AR interaction system, the interaction system comprising: The multimodal data acquisition module uses a globally unified clock to perform high-precision time synchronization acquisition of multiple sets of multimodal interaction data in VR / AR interaction scenarios; The differentiated cycle generation module extracts the propagation characteristics of all active infection chain nodes in the current scenario based on a pre-built invalid interaction infection chain graph, and generates a differentiated interaction processing cycle that adapts to the current scenario. The interaction feature extraction module extracts multi-batch, multi-dimensional interaction feature sets containing the features of the infection chain nodes, based on the differentiated interaction processing cycle and the complete time window of the infection chain nodes. The feature fusion calculation module adopts a dynamic weight fusion strategy based on the node level of the infection chain to perform hierarchical integration and analysis on multiple batches of multi-dimensional interaction feature sets to obtain multi-dimensional interaction feature fusion values. The preliminary confidence calculation module combines the standard temporal causal logic of effective interactions and calculates the preliminary interaction intent confidence based on the fusion value of multi-dimensional interaction features. The confidence reverse calibration module performs reverse calibration on the initial interaction intent confidence by matching the complete propagation path of the invalid interaction contagion chain and calculating the intent decay gradient, and generates the final interaction intent confidence. The trigger threshold adjustment module dynamically adjusts the initial interaction trigger threshold according to the real-time activity of the invalid interaction infection chain and the preset threshold adjustment rules. The interaction intent determination module compares the final interaction intent confidence level with the dynamically adjusted interaction trigger threshold to determine the validity of the interaction intent. The interaction response execution module executes the corresponding interaction response operation or maintains the current VR / AR scene state based on the interaction intent determination result. The infection chain graph update module updates the invalid interaction infection chain graph by treating the interaction data determined to be invalid as new nodes according to preset rules.

[0013] A computer-readable storage medium storing a computer program, the steps of a VR / AR interaction method implemented when the computer program is executed by a processor.

[0014] This invention provides a VR / AR interaction method, system, and storage medium, which have the following beneficial effects: By constructing an invalid interaction contagion chain graph, previously isolated invalid interaction events are linked into a network with causal propagation relationships. During the interaction intent recognition process, not only are the characteristics of the current interaction analyzed, but its position and propagation path in the contagion chain are also tracked. This allows potential risks to be identified and blocked in the early stages of chain-like accidental touches, solving the problem that existing technologies cannot handle continuous accidental touches. Secondly, it abandons the traditional binary processing logic of positive effective feature matching plus single-point noise filtering, and adopts a technical route of first calculating the initial confidence level and then performing reverse calibration through the contagion chain. Even when there are systematic anomalies in the full-modal data, abnormal interaction patterns can be identified through the propagation law of the contagion chain, which greatly improves the interaction stability in complex environments.

[0015] In addition, newly identified invalid interactions are automatically updated into the infection chain graph as new nodes, continuously improving the nodes and propagation relationships of the infection chain. As users use the system for longer periods, the accuracy of accidental touch recognition will continue to improve. At the same time, the system will dynamically adjust the data processing cycle and interaction trigger threshold based on the propagation characteristics of the infection chain nodes, rationally allocating computing resources and maintaining a smooth and natural interactive experience while ensuring interaction security. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the overall process of the present invention. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0018] Example 1: Taking a VR industrial assembly training scenario for automotive engine cylinder heads as an example, this scenario requires users to assemble precision parts sequentially in a virtual environment. Any accidental touch will reset the assembly progress of the current workstation. Based on this situation and scenario, such as Figure 1 As shown, the specific content is as follows: First, a hardware self-test and port enumeration are performed, scanning all peripheral interfaces. The headset, haptic gloves, optical positioning base station, and condenser microphone are identified using the device's vendor ID and product ID. Then, the corresponding WHQL-certified driver version is matched and loaded. After device plugging and unplugging, the enumeration, driver loading, and clock synchronization are re-executed without requiring an application restart. Next, a time protocol is initiated, using the headset's built-in high-precision TCXO temperature-controlled crystal oscillator as the global master clock source. Synchronization trigger signals are sent to all wired peripherals via wired Ethernet. For haptic gloves, a dynamic clock offset compensation algorithm based on linear regression prediction is used. This algorithm performs master-slave clock offset measurement every 100 milliseconds, recording the offset and transmission delay values ​​of the last 20 measurements. The linear trend of the clock offset is fitted using the least squares method to predict the offset of the next synchronization cycle and adjust the clock timestamp in advance. Ultimately, the time synchronization accuracy in the wireless environment is stably controlled within ±0.8 milliseconds. This is the core foundation for subsequent causal relationship determination in the transmission chain. If the synchronization error exceeds 5 milliseconds, the order of invalid interactions cannot be accurately distinguished, leading to errors in establishing the propagation relationship.

[0019] Each sensor is initialized with an independent lock-free circular data buffer, the size of which is set to store 5 seconds of raw data. A producer-consumer model is used to achieve read-write separation. The write pointer and read pointer are synchronized through atomic operations to ensure that no data race occurs under multi-threaded concurrent access. At the same time, it supports random reading of historical data in any time window.

[0020] After hardware initialization, a pre-built invalid interaction contagion chain graph is loaded and stored using the Neo4j 5.0 embedded graph database. This constructs a causal propagation network for invalid interactions, enabling node queries and depth-first path traversal. The graph employs a two-layer architecture, consisting of a global public graph and a user-private graph. The global public graph contains labeled general invalid interaction samples, covering accidental touch scenarios such as hand tremors, device collisions, strong light interference, and electromagnetic interference. These are primarily generated by manufacturers through standardized laboratory testing and cross-device federated learning. The user-private graph is initially empty and stores user-specific invalid interaction patterns, such as Parkinson's tremors, accidental touches caused by left-handed operation, and eye-tracking deviations caused by wearing glasses.

[0021] Each node in the infection chain contains 12 attributes: a 64-bit unique identifier, a 512-dimensional multimodal feature vector, a node level (1-5), propagation strength, historical trigger count, last trigger timestamp, scenario label (assembly / disassembly / debugging / inspection), user ID, average propagation time, confidence level, feature weight vector, and device model label.

[0022] Each directed edge contains 6 attributes: source node ID, target node ID, propagation probability, average propagation time, historical trigger count, and last propagation timestamp. At the same time, a graph activity decay timer is initialized, and a full graph activity decay calculation is performed every 60 seconds. The current activity of a node is equal to the historical activity multiplied by the natural exponent, which is the negative of (current timestamp - last trigger timestamp) / 300. The half-life is 5 minutes. If the activity after decay is less than 0.1, the node is no longer considered an active node, thus avoiding outdated accidental triggering modes from continuously consuming computing resources.

[0023] The main real-time interactive processing flow is executed cyclically with a minimum scheduling cycle of 10 milliseconds, triggering all sensors to synchronously collect data. The positioning base station collects the three-dimensional position coordinates, quaternion rotation angles, linear velocity, and linear acceleration of the head-mounted display and two controllers; the eye tracker collects the center coordinates of the left and right pupils, the three-dimensional gaze point coordinates, the gaze direction vector, the blink start / end timestamps, and the pupil diameter change; the haptic gloves collect the angles of 27 joints of both hands, the pressure values ​​of 10 fingers, the palm acceleration, and the angular velocity; the microphone collects mono audio signals; the IMU sensor built into the head-mounted display can collect the angular velocity and acceleration of the head, and the ambient light sensor collects the light intensity of the current scene.

[0024] When any interactive trigger event is detected, including collision detection between gestures and virtual objects, endpoint detection of voice commands, and virtual button press events, continuous historical data for the 3 seconds before the interaction, environmental data at the time of the interaction, and continuous subsequent data for the 1 second after the interaction are extracted from each circular buffer to form a complete three-dimensional context-related data packet. The historical data for the first 3 seconds is used to detect whether there is an invalid preceding interaction for the current interaction, the subsequent data for the last 1 second is used to verify whether the current interaction will trigger subsequent chain of accidental touches, and the environmental data is used to distinguish between systemic accidental touches caused by environmental interference and user-initiated operations.

[0025] After data acquisition, the raw data undergoes real-time preprocessing. Extended Kalman filtering is used to smooth the spatial positioning data, establishing a state vector containing position, velocity, and acceleration, and an observation vector containing the observation location. High-frequency jitter noise is removed through prediction and update steps. A 3×3 window median filter is used to process the hand gesture joint angle data, eliminating random glitches from the sensor. The 3σ principle is used to remove outliers from all features, replacing values ​​exceeding the mean ± 3 standard deviations with the historical average of that feature. Finally, all features are linearly normalized to the [0,1] interval to eliminate the influence of different units on subsequent calculations. The impact of computation; after preprocessing, multi-dimensional physical features are extracted and classified. Gesture motion data includes 27 joint angles, three-dimensional motion trajectory of the palm, motion speed and acceleration, and gesture duration; eye tracking data includes gaze point coordinates, gaze direction, blink frequency, pupil diameter change rate, and saccade amplitude; voice command data includes 13-dimensional Mel frequency cepstral coefficients, fundamental frequency, formants, speech intensity, and silence interval; haptic feedback data includes pressure values ​​of 10 fingers, pressure change rate, and contact duration; spatial positioning data includes the position, rotation, speed, acceleration, and motion trajectory curvature of the head-mounted display and controllers.

[0026] Every 100 milliseconds, an active infection chain node detection and differential processing cycle is generated. Nodes that simultaneously meet two conditions are selected from the infection chain graph as active infection chain nodes: they have triggered at least one invalid interaction in the past 10 minutes, and their propagation intensity is greater than 0.2. Propagation intensity is defined as the number of subsequent invalid interactions triggered by the node divided by the total number of triggers of the node. For each active node, its propagation speed is calculated, which is the weighted average of the time intervals of all historical propagation events of the node. The weight of the most recent 100 propagation events is 1, and the weight of earlier events is 0.5. For newly added nodes, since there is no historical propagation data, the average propagation speed of nodes of the same type in the same scenario is used as the initial value. Subsequently, a nested hierarchical cycle adjustment mechanism is executed. First, the basic processing cycle is set to half of the node propagation speed to ensure that at least two data processing and judgments are completed before the node triggers subsequent false triggers. Then, the cycle is shortened level by level according to the node's hierarchy. First-level infection nodes, i.e., source false touches without predecessor nodes, are shortened to half the basic cycle. Second-level infection nodes, false touches directly caused by first-level nodes, are shortened to one-third of the basic cycle. Third-level and above infection nodes are shortened to one-quarter of the basic cycle, achieving an optimal balance between computational resource consumption and timely false touch interception. When multiple active nodes correspond to the same data type, the shortest processing cycle is taken as the final processing cycle for that data type. A smooth cycle transition mechanism is added so that when a node changes from inactive to active, the processing cycle does not suddenly jump, but linearly transitions to the target value within three consecutive processing cycles, avoiding spikes in system load. Upper and lower limits are set for the processing cycle: a minimum of 10 milliseconds, limited by the sensor's highest sampling frequency, and a maximum of [missing value]. The 500-millisecond interval is to ensure the real-time requirements of the interaction. For example, if the currently active node is a primary infection node where a hand tremor accidentally touches the cylinder head bolt, its historical average propagation speed is 200 milliseconds, so the basic processing cycle is 100 milliseconds. The primary node is further shortened to 50 milliseconds. The final processing cycles for each data type are: gesture data 50 milliseconds, eye tracking data 100 milliseconds, voice command data 200 milliseconds, haptic feedback data 150 milliseconds, and spatial positioning data 100 milliseconds. At the same time, the CPU single-core utilization is monitored in real time. When the utilization exceeds 80%, the processing cycle of all data types is automatically extended by no more than 50%, and the priority of non-critical threads is reduced to prioritize the real-time recognition of interactive intents. When the utilization is below 50%, the original processing cycle is gradually restored.

[0027] Next, independent processing threads are allocated to each type of data based on the differentiated processing cycle. The thread priority is inversely proportional to the processing cycle, with shorter processing cycles resulting in higher thread priority, ensuring that data with high real-time requirements can be processed promptly. Corresponding processing parameters are configured for each data processing task. The data processing start time is the current time minus the maximum propagation time of the infection chain node (500 milliseconds). The data processing range covers a complete time window with 1x propagation speed before and after each node, ensuring the complete propagation process of the node is captured. The feature extraction precision is set to 32-bit single-precision floating-point numbers by default, maximizing computational speed while ensuring accuracy. For high-precision scenarios such as medical and aviation applications, this can be switched to 64-bit double-precision floating-point numbers. The original data storage nodes corresponding to each type of data are located according to the processing parameters. A data processing sequence including processing order, number of processing steps, and processing interval is defined for each storage node. The processing order is arranged according to the priority of spatial positioning, eye movement, gesture, voice, and touch. The number of processing steps and processing interval are consistent with the processing cycle of the corresponding data type.

[0028] Based on the data processing sequence, the original data is subjected to batch feature extraction to obtain single-dimensional feature vectors for various data types. Then, the unique infectious chain node-specific features of this invention are further extracted, using similarity to historical invalid interactions as the core feature dimension, specifically including three sub-features: (1) Historical feature similarity: Calculate the cosine similarity between the current feature vector and all historical invalid interaction feature vectors of this type, and take the maximum value; (2) Temporal matching degree: The calculation formula is 1 minus the absolute difference between the current feature timestamp and the active node timestamp divided by the node propagation speed, with a value range of [0,1]; (3) Spatial matching degree: The calculation formula is 1 minus the Euclidean distance between the current feature spatial location and the active node spatial location divided by the maximum interaction distance of 3 meters, with a value range of [0,1]; The final spatiotemporal matching degree is the weighted average of the temporal matching degree and the spatial matching degree, with each accounting for 50% of the weight; After feature extraction is completed, time alignment and spatial association are performed. A linear interpolation algorithm is used to align data with different sampling frequencies to a unified 10-millisecond time axis. For missing timestamps, the time alignment algorithm is used to align the data with different sampling frequencies to a unified 10-millisecond time axis. The positional error after completion is less than 1 cm, achieved by using cubic spline interpolation based on the motion trajectory of the first 3 frames. Spatial association employs Zhang's hand-eye calibration algorithm, which automatically calculates the rotation matrix and translation vector between the gesture sensor coordinate system and the head-mounted display coordinate system by having the user gaze at 9 evenly distributed calibration target points in virtual space. This transforms the coordinates of all sensors to a unified world coordinate system. Dynamic recalibration is triggered when the user moves more than 0.5 meters or uses the device continuously for more than 30 minutes, without requiring manual operation from the user. Finally, all single-dimensional features and contagion chain node features are concatenated into a 512-dimensional multi-dimensional interactive feature set, which is then compressed in real time using the LZ4 lossless compression algorithm. The compression ratio is approximately 2.5:1, and the decompression latency is less than 0.5 milliseconds, significantly reducing memory usage and data transmission bandwidth.

[0029] Dynamic weighted feature fusion based on node hierarchy is performed. First, feature structure information and feature content information are extracted from the feature set. The feature structure consistency evaluation value is calculated using the formula: structured data proportion × 0.6 + semi-structured data proportion × 0.3 + unstructured data proportion × 0.1. Structured data includes spatial positioning data and tactile pressure data; semi-structured data includes speech text and gesture commands; and unstructured data includes raw audio and video frames. Then, five dimensions of content quality indicators are calculated: feature completeness (the number of features actually extracted divided by the number of features to be extracted); feature accuracy (the cosine similarity between the current feature and the standard effective interaction feature); feature timeliness (1 divided by the difference between data processing time and acquisition time, normalized to the [0,1] interval); feature clarity (signal-to-noise ratio divided by the maximum signal-to-noise ratio of the sensor); and feature relevance (the mutual information value between the feature and the current assembly task, normalized to the [0,1] interval). Simultaneously, a data integrity check is performed. If the missing rate of a certain type of data exceeds 20%, the basic weight of that type of data is reduced by 50%. If the percentage exceeds 50%, the influence of this type of data is completely excluded to avoid incorrect judgments caused by missing data. Subsequently, the innovative dynamic weight adjustment mechanism based on the node level of the infection chain is implemented. Existing technologies all use fixed feature weights, which cannot adapt to changes in the risk of accidental touches in different scenarios. This mechanism dynamically adjusts the weights of each dimension according to the matched infection chain node level, reducing the influence of features related to invalid interactions from the source. The default values ​​of the basic weights are: gesture action 0.3, eye tracking 0.25, voice command 0.25, haptic feedback 0.1, and spatial positioning 0.1. All nodes with a feature similarity greater than 60% to the infection chain node features of the current feature set are matched from the infection chain map. The weights of the corresponding dimensions are adjusted according to the node level: features matching first-level infection nodes have their corresponding dimension weights reduced by 80%; features matching second-level infection nodes have their corresponding dimension weights reduced by 50%; and features matching third-level and above infection nodes have their corresponding dimension weights reduced by 20%. At the same time, fine-tuning is performed according to the real-time propagation intensity of the nodes. For every 0.1 increase in propagation intensity, the weight reduction rate increases by 5%.

[0030] For example, the current feature set matches the primary infection node of hand tremors accidentally touching the cylinder head bolt with a degree of 78%. Therefore, the weight of the gesture dimension is reduced from 0.3 to 0.3×(1-80%)=0.06, the weight of the eye tracking dimension is reduced from 0.25 to 0.25×(1-50%)=0.125, and the weights of other dimensions remain unchanged.

[0031] Three feature fusion algorithms are provided for selection: the first is the weighted average method, which has the fastest calculation speed and is suitable for low-performance mobile VR devices; the second is the DS evidence theory fusion method, which can better handle uncertain information and has an accuracy rate about 8% higher than the weighted average method, and is suitable for PCVR devices; the third is the lightweight CNN fusion method, which uses a 3-layer convolutional neural network to automatically learn the optimal weights of each feature and has an accuracy rate about 12% higher than the weighted average method, but has a larger computational load and is suitable for high-performance devices equipped with dedicated graphics cards. We use the default weighted average method to calculate the feature content validity evaluation value, and then perform weighted fusion with the feature structure consistency evaluation value in a 3:7 ratio to obtain the final multi-dimensional interactive feature fusion value.

[0032] By combining standard temporal causal logic of effective interactions to calculate the initial confidence level of interaction intent, the fixed temporal pattern of human interaction is used as an important basis for confidence level calculation. Statistical analysis of the interaction behavior of 1000 volunteers revealed that humans inevitably follow a fixed temporal sequence to complete a conscious interaction action: eye-tracking gaze at the target, hand gesture pointing at the target, and voice confirmation. Specifically, eye-tracking gaze precedes hand gestures by 100-500 milliseconds, and voice confirmation lags behind hand gestures by 100-1000 milliseconds. Interactions that do not conform to this temporal sequence have a 92% probability of failure. It is a false touch; check if the current feature set conforms to the standard timing: if it conforms completely, then add 10% gain to the feature fusion value; if it conforms partially, such as only eye movement and gestures, without speech, then add 5% gain; if it does not conform at all, such as gestures preceding eye gaze, then add 10% attenuation; the initial confidence of the interaction intent is equal to the multi-dimensional interaction feature fusion value multiplied by the timing logic gain or attenuation coefficient. For example, if the feature fusion value is 0.72 and it conforms completely to the standard timing, then the initial confidence is 0.72 × 1.1 = 0.792.

[0033] Next, we come to the core of this invention: the matching of the transmission path of the infection chain and the reverse calibration of the exponential confidence level. Existing technologies all use unidirectional logic that matches effective features in the forward direction to the output confidence level, which cannot identify interactions on the chain of accidental touches. However, we use bidirectional calibration logic that calculates the initial confidence level in the forward direction, matches the reverse transmission chain path, and decays the confidence level exponentially, which fundamentally solves the problem of chain of accidental touches, as detailed below: Extract all complete propagation paths with a length of 3 or more nodes and a propagation probability greater than 0.5 from the infection chain graph. Use a depth-first search algorithm to traverse all first-level nodes until there are no successor nodes. By pre-compiling the feature summary of each path, the path extraction speed is improved by more than 3 times. Then, a multi-scale sliding window with sizes of 3, 4, and 5 is used to match the temporal and spatial features of the feature set to be verified with all complete propagation paths segment by segment. The sliding step size is 1 time unit, and the matching degree is calculated using weighted cosine similarity. Among them, the weight of temporal features accounts for 60%, and the weight of spatial features accounts for 40%. The matching process is accelerated in parallel using the AVX2SIMD instruction set, which can process the similarity calculation of 8 windows at the same time, improving the speed by more than 4 times. If the matching degree of any sliding window reaches 75%, the current interaction is determined to be on the propagation path of an invalid interaction contagion chain. The propagation distance is calculated as the shortest path length between the node corresponding to the current interaction feature and the starting node of the propagation path. The attenuation coefficient is the negative propagation distance of the exponential function e divided by the square. This function is obtained by fitting more than 100,000 chain mis-touch samples and can accurately reflect the attenuation law of the risk of chain mis-touch with the propagation distance. For every increase of 1 in the propagation distance, the probability of mis-touch decreases by about 39%. If the current interaction If multiple propagation paths are matched simultaneously, the largest attenuation coefficient is used for calibration. For example, if the current feature set matches the first two segments of the propagation path (hand tremor causing accidental contact with bolt A, bolt B, and cylinder head) with an accuracy of 82% and the propagation distance is 2, then the attenuation coefficient is e^(-2 / 2) = e^-1 ≈ 0.3679. The final confidence level of the interaction intent is equal to the initial confidence level multiplied by the attenuation coefficient, i.e., 0.792 × 0.3679 ≈ 0.291. If no propagation path is matched, the initial confidence level is directly used as the final confidence level.

[0034] The initial interaction trigger threshold is dynamically adjusted based on the real-time activity of the invalid interaction contagion chain, employing a non-linear dynamic threshold adjustment mechanism. A sliding window is used to count the number of invalid interaction triggers over the past 5 minutes, separately calculating the activity levels of four categories: gesture touch, eye-tracking touch, voice touch, and environmental touch. The sliding window slides once per second, updating the activity level values ​​in real time. The base adjustment coefficient equals 1 plus the total activity divided by 20. When the total activity is greater than 30, a non-linear gain term is added, with the adjustment coefficient equal to 1 plus the total activity divided by 20 plus the square of (total activity - 30) divided by 100, further enhancing the protection level during high-risk periods. When the total activity is less than 5, the threshold is appropriately lowered, with the adjustment coefficient equal to 1 minus (5 - total activity) divided by 20, improving the smoothness during normal interaction periods. It also supports categorized threshold adjustments; for example, when voice touch activity is high, only the trigger threshold for voice commands is increased, without affecting the sensitivity of other interaction methods. Set upper and lower limits for the threshold: the maximum should not exceed 0.9 to avoid the threshold being too high and causing the interaction to completely fail; the minimum should not be lower than 0.4 to avoid the threshold being too low and causing a flood of accidental touches; after each threshold adjustment, it should remain unchanged for 10 seconds to avoid frequent fluctuations in the threshold affecting the user experience. For example, if 3 invalid interactions were triggered in the past 5 minutes, the total activity level is 3, which is less than 5. Therefore, the adjustment coefficient is 1-(5-3) / 20=0.9, the initial trigger threshold is 0.6, and the dynamically adjusted threshold is 0.6×0.9=0.54; when the real-time activity level of the infection chain is 0, the adjustment coefficient is 1, and the dynamic threshold is equal to the initial threshold.

[0035] The final confidence score of 0.291 was compared with the dynamic threshold of 0.54. Since 0.291 is less than 0.54, it was determined to be an invalid interaction intent. The current virtual scene state was maintained, and the bolt tightening operation was not performed, successfully intercepting a chain of false touches. If it is determined to be a valid interaction, the corresponding assembly operation is performed by calling the native API of the Unity engine, including grabbing, moving, rotating, assembling, and tightening bolts of virtual objects. The end-to-end response latency is less than 20 milliseconds, which meets the real-time requirements of VR interaction. If the user believes that the system has misjudged a valid interaction, the user feedback mechanism can be triggered by making a fist with both hands and holding it for 3 seconds. The interaction is marked as a false positive sample, the propagation probability of the corresponding infection chain node is automatically reduced by 0.1, and the confidence gain of this type of interaction is increased by 15% in the following 24 hours until there are no false judgments for 10 consecutive times.

[0036] After the determination is completed, the invalid interaction is updated as a new node in the user's private infection chain graph and optimized. Since the infection chain graph has the ability to continuously self-evolve, the longer it is used, the higher the accuracy of false touch recognition. Therefore, the 512-dimensional feature vector and complete contextual data of the invalid interaction are extracted to generate candidate infection chain nodes. The initial trigger count is 1, the propagation intensity is 0.5, the last trigger time is the current timestamp, the scene tag is assembly, the user ID is the currently logged-in user, and the device model is the currently used head-mounted display model. The feature cosine similarity between the candidate node and all existing nodes in the graph is calculated. If there is an existing node with a similarity greater than 80%, the candidate node is merged into the existing node. The propagation intensity after merging is equal to (original propagation intensity × original trigger count + candidate propagation intensity × 1) / (original trigger count + 1), and the trigger count after merging is equal to the original trigger count plus 1. At the same time, the node is updated. The last trigger time and average propagation time of a node are considered. For example, if the propagation strength of an existing node A is 0.6 and the number of historical triggers is 10, and the similarity between the current candidate node and A is 85%, the propagation strength after merging is approximately 0.59, and the number of triggers is updated to 11. If there is no existing node with a similarity greater than 80%, the candidate node is added to the graph as a first-level infection node. If the time interval between the current invalid interaction and the previous invalid interaction is less than 500 milliseconds and the feature similarity is greater than 70%, a directed edge is established from the first-occurrence node to the current node. The initial propagation probability of the edge is 0.7, the average propagation time is the time interval between two interactions, and the level of the current node is set to the level of the triggering node plus 1. Each time this propagation relationship is triggered subsequently, the propagation probability of the edge will be dynamically adjusted according to the actual results: if it does trigger subsequent false triggers, the propagation probability increases by 0.05; if it does not trigger, the propagation probability decreases by 0.05. Every 24 hours, the system uploads the newly added node feature vectors in the user's private graph to the cloud server after homomorphic encryption. It then aggregates the features uploaded by all devices through horizontal federated learning to update the global public graph. At the same time, it downloads the incremental update part of the global graph. Throughout the process, the user's original interaction data does not leave the local device, thus fully protecting user privacy.

[0037] When multiple users collaborate on assembly in the same virtual space simultaneously, the system automatically activates a multi-user adaptation mode, assigning each user an independent interaction space and a private transmission chain graph. Interaction data from different users is strictly isolated using user IDs to prevent interference. The system comprehensively judges interactions based on the head-mounted display's spatial position, gaze direction, and hand movement trends. When two users' interaction spaces overlap, the system automatically filters out the influence of other users' actions on the current user, prioritizing responses to the user whose gaze direction aligns with their gesture direction, thus avoiding misinterpreting user A's accidental touch as user B's valid interaction. When two users simultaneously grasp the same virtual object, the system integrates their interaction data to jointly determine the interaction intent, weighting the judgment based on the users' gesture strength, gaze duration, and voice commands to ensure the smoothness and accuracy of collaborative operations.

[0038] It also features robust fault tolerance and security mechanisms, monitoring the output status of each sensor in real time. If a sensor fails to output valid data for three consecutive processing cycles, it is considered a sensor malfunction and automatically switches to a backup interaction mode. For example, if the gesture sensor malfunctions, it switches to a voice + eye-tracking interaction mode, reducing the weight of the corresponding data type and issuing a voice prompt to the user. If the data loss rate of a single frame is less than 20%, cubic spline interpolation based on the motion trajectory of the previous three frames is used for completion, with a positional error of less than 1 cm after completion. If the data loss rate is greater than 20%, the current processing is abandoned, and the system waits for the next batch of data. The program uses AES-256 encryption for storage and can be digitally signed for verification to prevent tampering and reverse engineering. System upgrades use an A / B partitioning model. The upgrade package is downloaded and installed to the backup partition. After the upgrade is complete, the system restarts and switches to the new partition. If the upgrade fails, it automatically rolls back to the previous normal version, preventing the device from becoming unusable. All original user interaction data is stored only in the encrypted partition of the local device. Only homomorphically encrypted feature vectors are uploaded to the cloud, and the data cannot be restored to its original form, complying with GDPR and personal information protection laws.

[0039] Example 2: This embodiment is the system of this application, including: a multimodal data acquisition module responsible for global clock synchronization and multi-sensor data acquisition and preprocessing; a differentiated period generation module responsible for active node extraction and nested period adjustment; an interaction feature extraction module responsible for single-dimensional feature and infection chain node feature extraction and spatiotemporal alignment; a feature fusion calculation module responsible for dynamic weight adjustment and multi-dimensional feature fusion; a preliminary confidence calculation module that calculates preliminary confidence based on standard time-series logic; a confidence reverse calibration module responsible for propagation path matching and exponential confidence decay; a trigger threshold adjustment module that dynamically adjusts the interaction trigger threshold according to real-time activity; an interaction intent determination module that compares the final confidence with the dynamic threshold; an interaction response execution module that calls the engine API to execute the corresponding interaction operation; and an infection chain graph update module responsible for adding, merging, and establishing propagation relationships of nodes.

[0040] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0041] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0042] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A VR / AR interaction method, characterized in that, The steps of this method are as follows: Time synchronization is achieved by using a globally unified clock to collect multiple sets of multimodal interaction data in VR / AR interaction scenarios. Based on a pre-constructed invalid interaction contagion chain graph, the propagation characteristics of all active contagion chain nodes in the current scenario are extracted to generate a differentiated interaction processing cycle adapted to the current scenario. Based on the differentiated interaction processing cycle, and taking the complete time window of the infection chain node as the benchmark, a multi-batch, multi-dimensional interaction feature set containing the characteristics of the infection chain node is extracted; A dynamic weight fusion strategy based on the node level of the infection chain is adopted to perform hierarchical integration and analysis on multi-batch multi-dimensional interaction feature sets to obtain multi-dimensional interaction feature fusion values. Combining the standard temporal causal logic of effective interaction, the initial confidence level of interaction intent is calculated based on the fusion value of multi-dimensional interaction features. By matching the complete propagation path of the invalid interaction contagion chain and calculating the intent decay gradient, the confidence of the initial interaction intent is reverse-calibrated to generate the final interaction intent confidence. Based on the real-time activity of the invalid interaction contagion chain, the initial interaction trigger threshold is dynamically adjusted according to the preset threshold adjustment rules; The final interaction intent confidence level is compared with the dynamically adjusted interaction trigger threshold. If the confidence level of the final interaction intent is greater than the dynamically adjusted interaction trigger threshold, it is determined to be a valid interaction intent and the corresponding interaction response operation is executed. Conversely, if the interaction is invalid, it is determined to be an invalid interaction intent and the current VR / AR scene state is maintained. At the same time, the invalid interaction is updated as a new node to the invalid interaction infection chain graph according to preset rules.

2. The VR / AR interaction method according to claim 1, characterized in that, The steps for processing multiple sets of multimodal interaction data in VR / AR interaction scenarios using a globally unified clock are as follows: Identify various multimodal sensor ports in VR / AR interactive scenarios; All sensor ports are controlled by a global unified clock to synchronously collect multiple sets of raw interaction data and corresponding interaction context associated data. The interaction context associated data includes continuous historical interaction data for a preset duration before the interaction occurs, scene environment data when the interaction occurs, and continuous subsequent interaction data for a preset duration after the interaction occurs. Multi-dimensional physical features of each set of original interaction data were extracted and divided into gesture action data, eye tracking data, voice command data, tactile feedback data, and spatial positioning data.

3. The VR / AR interaction method according to claim 1, characterized in that, The steps for generating a differentiated interaction processing cycle adapted to the current scenario are as follows: From the invalid interaction infection chain graph, all nodes that have triggered invalid interactions within a preset statistical period are selected as active infection chain nodes; Calculate the propagation speed of each active infection chain node, where the propagation speed is the average time interval from the node's first appearance to triggering the first subsequent invalid interaction; The basic processing cycle for each active infection chain node is determined based on its corresponding data type. Based on the hierarchical relationship of the infection chain nodes, the basic processing cycle is shortened and adjusted step by step, with different levels of nodes corresponding to different shortening rates; By integrating and adjusting various data processing cycles, a differentiated interactive processing cycle adapted to the current scenario is generated.

4. The VR / AR interaction method according to claim 1, characterized in that, The steps for extracting multi-batch, multi-dimensional interaction feature sets containing features of infection chain nodes are as follows: Based on the differentiated interaction processing cycle, assign independent data processing tasks to each type of multimodal data and configure the corresponding processing parameters; Locate the original data storage nodes corresponding to various types of data according to the processing parameters, and formulate a data processing sequence for each original data storage node, including processing order, number of processing times and processing interval; Based on the data processing sequence, feature extraction is performed on the raw data in batches to obtain single-dimensional interaction features corresponding to various types of data. Furthermore, the infection chain node features corresponding to each single-dimensional interaction feature are extracted. The infection chain node features include the similarity between the feature and historical invalid interaction features, and the spatiotemporal matching degree between the feature and the infection chain node. The single-dimensional interaction features and infection chain node features of various types of data are aligned in time and spatially associated with the timestamp of a globally unified clock to form a multi-batch, multi-dimensional interaction feature set.

5. The VR / AR interaction method according to claim 1, characterized in that, The steps for obtaining the multi-dimensional interactive feature fusion value are as follows: Feature structure information and feature content information are extracted from multiple batches of multi-dimensional interactive feature sets. Specifically, feature structure consistency evaluation value is calculated based on feature structure information, and multi-dimensional content quality indicators are obtained based on feature content information. From the invalid interaction infection chain graph, identify all infection chain nodes whose feature matching degree with the current feature set reaches a preset matching threshold; The weighting coefficients of each dimension of content quality indicators are dynamically adjusted based on the level and propagation intensity of each matching node. By combining the evaluation values ​​of each dimension with the adjusted weight coefficients, the feature content validity evaluation value is calculated; then, the feature structure consistency evaluation value and the feature content validity evaluation value are weighted and fused to obtain the multi-dimensional interactive feature fusion value.

6. The VR / AR interaction method according to claim 1, characterized in that, The steps for generating the final interaction intent confidence score are as follows: Extract all complete transmission paths containing a preset number of nodes from the invalid interaction transmission chain graph; The temporal and spatial features of the interaction feature set to be verified are matched with all complete transmission chains in a step-by-step sliding match. If the matching degree of any segment reaches the preset transmission chain matching threshold, it is determined that the initial interaction intention is on the transmission path of the invalid interaction transmission chain, and the transmission distance of the interaction feature on the transmission path is calculated. The intention decay gradient coefficient is calculated based on the propagation distance. Then, the initial interaction intention confidence is multiplied by the intention decay gradient coefficient to obtain the final interaction intention confidence. If the matching degree of all segments does not reach the preset infection chain matching threshold, the initial interaction intention confidence is directly used as the final interaction intention confidence.

7. The VR / AR interaction method according to claim 1, characterized in that, The steps for dynamically adjusting the initial interaction trigger threshold according to the real-time activity of the invalid interaction contagion chain and the preset threshold adjustment rules are as follows: The total number of times all active infection chain nodes in the invalid interaction infection chain graph are triggered within the preset statistical period is used as the real-time activity of the infection chain. The threshold adjustment coefficient is calculated based on the real-time activity of the infection chain; the initial interaction trigger threshold is multiplied by the threshold adjustment coefficient to obtain the dynamically adjusted interaction trigger threshold. When the real-time activity of the infection chain is 0, the dynamically adjusted interaction trigger threshold is equal to the initial interaction trigger threshold.

8. The VR / AR interaction method according to claim 1, characterized in that, The steps for updating the invalid interaction as a new node to the invalid interaction infection chain graph according to preset rules are as follows: Extract the multi-dimensional features of the invalid interaction and the corresponding interaction context association data, generate candidate infection chain nodes, and calculate the feature similarity between the candidate infection chain nodes and all existing nodes in the invalid interaction infection chain graph. If there are existing nodes whose similarity reaches the preset merging threshold, the candidate node will be merged into the existing node, and the propagation strength and trigger count of the existing node will be updated. If there are no existing nodes whose similarity reaches the preset merging threshold, the candidate node will be added as a first-level infection node to the invalid interaction infection chain graph. If the invalid interaction is caused by other invalid interactions, a propagation association is established between the candidate node and the initiating node, and the level of the candidate node is set to the level of the initiating node plus 1.

9. A VR / AR interactive system, characterized in that, The interactive system includes: The multimodal data acquisition module uses a globally unified clock to perform high-precision time synchronization acquisition of multiple sets of multimodal interaction data in VR / AR interaction scenarios; The differentiated cycle generation module extracts the propagation characteristics of all active infection chain nodes in the current scenario based on a pre-built invalid interaction infection chain graph, and generates a differentiated interaction processing cycle that adapts to the current scenario. The interaction feature extraction module extracts multi-batch, multi-dimensional interaction feature sets containing the features of the infection chain nodes, based on the differentiated interaction processing cycle and the complete time window of the infection chain nodes. The feature fusion calculation module adopts a dynamic weight fusion strategy based on the node level of the infection chain to perform hierarchical integration and analysis on multiple batches of multi-dimensional interaction feature sets to obtain multi-dimensional interaction feature fusion values. The preliminary confidence calculation module combines the standard temporal causal logic of effective interactions and calculates the preliminary interaction intent confidence based on the fusion value of multi-dimensional interaction features. The confidence reverse calibration module performs reverse calibration on the initial interaction intent confidence by matching the complete propagation path of the invalid interaction contagion chain and calculating the intent decay gradient, and generates the final interaction intent confidence. The trigger threshold adjustment module dynamically adjusts the initial interaction trigger threshold according to the real-time activity of the invalid interaction infection chain and the preset threshold adjustment rules. The interaction intent determination module compares the final interaction intent confidence level with the dynamically adjusted interaction trigger threshold to determine the validity of the interaction intent. The interaction response execution module executes the corresponding interaction response operation or maintains the current VR / AR scene state based on the interaction intent determination result. The infection chain graph update module updates the invalid interaction infection chain graph by treating the interaction data determined to be invalid as new nodes according to preset rules.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the VR / AR interaction method according to any one of claims 1 to 8.

Images