A vr multi-terminal collaborative intelligent interaction method and system
By introducing gesture style recognition and meta-intention guidance into the VR system, the problems of inaccurate recognition of user operation intentions and incoherent scene logic are solved, achieving more accurate intention parsing and scene state maintenance, and improving the naturalness and immersion of the interaction.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA STATE SHIPBUILDING CORP LTD RESEARCH INSTITUTE 719
- Filing Date
- 2026-02-13
- Publication Date
- 2026-05-29
AI Technical Summary
Existing VR systems suffer from ambiguous intent in user operation recognition, making it difficult to accurately identify the user's true intentions. This is especially true when there are interactive objects and navigation controls near the touch point, leading to frequent false triggers and chaotic operations, as well as inconsistent scene interaction logic.
By integrating gesture style recognition and meta-intention guidance, and utilizing multi-dimensional physical feature analysis, state network modeling, and meta-intention inference, an intelligent decision-making framework is constructed to accurately analyze user operation intentions and maintain the logical coherence of the scene.
It improves the accuracy and immersion of VR interaction, reduces accidental triggering and operational ambiguity, ensures logical consistency and realism of scene interaction, and provides an adaptive interactive experience.
Smart Images
Figure CN122111228A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular, to a VR multi-terminal collaborative intelligent interaction method and system. Background Art
[0002] With the popularization of virtual reality (VR) technology, applications such as VR panoramic house viewing, car viewing, virtual exhibitions, etc. have become important business and service models. Users switch perspectives, move positions, and interact with virtual objects in panoramic images taken based on real scenes by touching the screen. However, this free touch interaction method brings significant "intention ambiguity" problems: since basic gestures such as swiping and clicking are simultaneously mapped to multiple operations such as "perspective adjustment", "scene stepping", and "object interaction", it is difficult for the system to accurately recognize the user's true intention. Especially when there are interactive objects and navigation controls near the touch point, mis-triggering and operation chaos frequently occur.
[0003] Existing technologies have tried to assist in judgment by introducing user identity information or simple interactive object attributes, but these methods are usually static and isolated. They fail to consider the rich dynamic features contained in the user's operation gestures, nor do they understand a single operation in the macroscopic strategy of the user's entire interaction session. More importantly, the current system treats each interactive object as an independent unit, ignoring the logical associations existing between objects in the real scene, resulting in no联动反应 (I'm not sure what this should be in English, maybe "linked reaction") in other parts of the scene after the user operates on an object, causing a sense of logical break and distortion in the experience.
[0004] Therefore, there is an urgent need for a VR multi-terminal collaborative intelligent interaction method to improve the immersion, interaction efficiency, and user satisfaction. Summary of the Invention
[0005] Aiming at the technical problems existing in the prior art, the present invention realizes the accurate analysis and decision-making of VR touch interaction intentions by integrating gesture style recognition and meta-intention guidance, and intelligently maintains the logical coherence of the scene according to the state network during execution, thereby significantly improving the naturalness, accuracy, and immersion of the interaction.
[0006] The technical solution of the present invention to solve the above technical problems is as follows: A VR multi-terminal collaborative intelligent interaction method, comprising: S101, in response to a touch instruction of a first user received on a target VR panoramic touch interface, obtaining a touch trajectory data stream, extracting trajectory type features and physical feature vectors, and generating a gesture style label corresponding to the touch instruction; S102, determine the attribute information set of at least one interactive object within the preset area covered by the touch point according to the touch point position of the touch instruction, and generate a candidate interaction intent set L, the candidate interaction intents include operation intents and / or global navigation intents related to the interactive object. S103, combining trajectory type features, physical feature vectors and gesture style labels, calculates the matching score of each candidate interaction intent in the set of candidate interaction intents corresponding to the touch command; S104, determine the final target interaction intent from the candidate interaction intents whose matching score is higher than the preset threshold. Preferably, the touch trajectory data stream is a time series. , For timestamps, The screen coordinates of the touch point N represents the touch pressure value, and N represents the number of touch points within the touch duration window from the start to the end of the touch.
[0007] Preferably, the trajectory type feature is set to click or swipe; The physical feature vector includes average touch pressure. instantaneous velocity standard deviation Mean normal acceleration and mean trajectory curvature .
[0008] Preferably, the gesture style label is obtained based on the touch trajectory data stream and a pre-trained spatiotemporal pattern recognition model, and is used to characterize the operation behavior pattern of the first user when completing a touch command. The generation method specifically includes: High-order temporal and spatial distribution features of the touch trajectory data stream are extracted, and these features are combined with the physical feature vector F to form touch features. The features are then input into a pre-trained spatiotemporal pattern recognition neural network model, which outputs the gesture style label corresponding to the touch command.
[0009] Preferably, the pre-trained spatiotemporal pattern recognition neural network model is obtained in the following way: A1. Collect a large number of historical sample sets , For the first Touch characteristics of each sample For the corresponding gesture style tags; A2. Using historical sample sets, the pre-selected neural network structure is trained, and the model parameters are continuously optimized to obtain the final spatiotemporal pattern recognition neural network model.
[0010] Preferably, for each interactive object within the preset area... Query its predefined attribute information , As a unique identifier, For interactive object type, Physical properties This is the set of allowed interaction types for this interactive object.
[0011] Preferably, the calculation of the matching score for each candidate interaction intent in the candidate interaction intent set corresponding to the touch command is specifically as follows: Based on each candidate interaction intent For candidate interaction intents related to interactive objects, the matching score is:
[0012] in, , , These are adjustable weighting coefficients, summing to 1, and are determined based on actual needs and expert experience. The first matching degree is the trajectory type matching degree; The second matching degree, namely the matching degree between physical features and gesture style, depends on the interaction type of the intent under the current gesture style label. Interactive objects Ideal operating characteristics; The third matching degree is the correlation between the first user's historical behavior profile and the type of interactive object.
[0013] Preferably, the second matching degree is calculated according to the following formula:
[0014] in, , Intent interaction type In the physical properties of interactive objects With gesture style tags Ideal eigenvalues under the following conditions As the normalization factor, These are the feature weights.
[0015] Preferably, S104 specifically includes: Select candidates with matching scores higher than a preset threshold from the candidate interaction intent set L to form a high-confidence candidate set C; like If the value is 1, then the candidate interaction intent is directly determined as the target interaction intent; like At that time, for each candidate interaction intent Combining the pre-set state network graph G with the determined meta-intent Calculate its impact assessment value. And a comprehensive matching score Determine the final target interaction intent .
[0016] A VR multi-terminal collaborative intelligent interaction system, characterized in that it executes a VR multi-terminal collaborative intelligent interaction method as described in any one of claims 1-9, characterized in that the system includes: an acquisition module, a filtering module, a matching module, and a determination module; The acquisition module is used to respond to the touch command received from the first user on the target VR panoramic touch interface, acquire the touch trajectory data stream, extract trajectory type features and physical feature vectors, and generate gesture style tags corresponding to the touch command. The filtering module is used to determine the attribute information set of at least one interactive object contained within the preset area covered by the touch point according to the touch point position of the touch instruction, and generate a candidate interactive intent set L, which includes operation intents and / or global navigation intents related to the interactive object. The matching module is used to combine trajectory type features, physical feature vectors and gesture style labels to calculate the matching score of each candidate interaction intent in the set of candidate interaction intents corresponding to the touch command. The determining module is used to determine the final target interaction intent from candidate interaction intents whose matching score is higher than a preset threshold.
[0017] The beneficial effects of this invention are: By introducing multi-dimensional physical feature analysis, gesture style recognition, state network modeling, and meta-intent inference, a hierarchical intelligent decision-making framework is constructed to achieve more accurate interpretation of user operation intentions and more coherent maintenance of scene states.
[0018] Breaking away from the limitations of merely analyzing basic trajectories, this system identifies the "style" of user gestures through neural networks, enabling intent recognition to understand subtle differences in the same operation across different behavioral patterns. By analyzing user interaction sequences to infer macro-strategies (meta-intents), and using these strategies to dynamically adjust the focus of micro-intent decisions, it achieves long-term interaction goals guiding immediate actions. Through explicit modeling of state dependencies between interactive objects, the system can predict and render chain reactions triggered by a single operation, ensuring logical consistency and realism in VR scene interactions. Attached Figure Description
[0019] Figure 1 This is a flowchart illustrating a VR multi-terminal collaborative intelligent interaction method according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of a VR multi-terminal collaborative intelligent interaction system according to an embodiment of the present invention. Detailed Implementation
[0020] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0021] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0022] In the description of this application, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.
[0023] Example 1 Figure 1 This is a flowchart illustrating a VR multi-terminal collaborative intelligent interaction method according to an embodiment of the present invention.
[0024] like Figure 1 As shown, a VR multi-terminal collaborative intelligent interaction method includes the following steps: S101, in response to the touch command received from the first user on the target VR panoramic touch interface, acquire the touch trajectory data stream (the complete trajectory data stream from the start to the end of the touch), extract the trajectory type feature T and the physical feature vector F, and generate the gesture style label corresponding to the touch command. .
[0025] It should be noted that when the target VR panoramic touch interface detects a touch event, the system starts. First, it captures the operator's image through a front imaging device (such as a camera) and distinguishes different users operating at the same time based on facial recognition technology, and associates the touch events belonging to the current main operator (defined as the first user).
[0026] Specifically, the touch trajectory data stream is a time series. ,in, For timestamps, The screen coordinates of the touch point Here, N represents the touch pressure value, and N is the number of touch points within the touch duration window from the start of the touch (finger touching the screen) to the end of the touch (finger leaving the screen). The determination method for the trajectory type feature T is as follows: Calculate the touch duration and maximum displacement, and determine the trajectory type feature T based on the first and second preset thresholds respectively.
[0027] For example, let the touch duration window be... The set of touch points is Calculate the maximum displacement Set the first threshold corresponding to the time. (e.g., 200ms indicates the maximum allowed duration threshold for determining a click) and the second threshold corresponding to the displacement. (For example, 2% of the screen width represents the maximum allowed displacement threshold for a click). If the value is positive, it is considered a click (T=click); otherwise, it is considered a swipe (T=swipe). It should be noted that the determination of the first and second thresholds is based on the actual situation of the target VR panoramic touch interface and historical experience, and is determined by experts. Alternatively, it can be obtained by collecting a large number of user click operation samples and statistically analyzing them. Typical values are 100-300ms and 1%-5% of the screen width, respectively.
[0028] In some embodiments, the physical feature vector includes at least the average touch pressure. instantaneous velocity standard deviation Mean normal acceleration and mean trajectory curvature , can be represented as These correspond to average touch pressure, instantaneous speed standard deviation, average normal acceleration, and average trajectory curvature, respectively.
[0029] Specifically, the average touch pressure is calculated using the following formula: Based on instantaneous velocity sequence The instantaneous velocity standard deviation is calculated using the following formula: ,in, , Its mean value is used to characterize the smoothness of the gesture; the mean normal acceleration is calculated using the following formula: ,in, This is used to depict the urgency of the gesture; the average trajectory curvature is calculated using the following formula: ,in, The derivative is approximated by discrete difference and is used to quantify the curvature of the trajectory.
[0030] In some embodiments, gesture style labels are obtained based on touch trajectory data streams and a pre-trained spatiotemporal pattern recognition model. These labels characterize the operational behavior patterns of the first user when completing touch commands, such as "fine-grained operation," "quick browsing," and "hesitant exploration." The generation method specifically includes: High-order temporal and spatial distribution features of the touch trajectory data stream are extracted, and these features are combined with the physical feature vector F to form touch features. The features are then input into a pre-trained spatiotemporal pattern recognition neural network model, which outputs the gesture style label corresponding to the touch command.
[0031] Specifically, high-order temporal features and spatial distribution features of the touch trajectory data stream are extracted, including: Extracting high-order temporal features from touch trajectory data stream Including instantaneous velocity sequences The first m terms of the discrete cosine transform (DCT) coefficients (take m=5), instantaneous acceleration sequence The peak value of the power spectral density; Extracting spatial distribution features of touch trajectory data stream Including touch point set The ratio of the two eigenvalues of the covariance matrix (the ratio of the larger eigenvalue to the smaller eigenvalue), and the aspect ratio of the trajectory envelope matrix (minimum bounding matrix).
[0032] Specifically, the pre-trained spatiotemporal pattern recognition neural network model is obtained as follows: A1. Collect a large number of historical sample sets , For the first Touch characteristics of each sample This refers to the corresponding gesture style tag.
[0033] For example, the process of labeling historical touch features can be as follows: experts are authorized to watch user operation recordings, and the gesture style is judged in combination with the context (such as whether the user adjusts the perspective at a granular level after the operation, or whether the trajectory is frequently corrected), and divided into three categories: fine operation type (careful operation, slow speed, curved trajectory), fast browsing type (fast speed, straight trajectory, light pressure), and hesitant exploration type (large speed changes, with pauses).
[0034] A2. Using historical sample sets, the pre-selected neural network structure is trained, and the model parameters are continuously optimized to obtain the final spatiotemporal pattern recognition neural network model.
[0035] During training, minimize the cross-entropy loss: P represents the probability distribution of the model output, which will not be elaborated upon in this invention.
[0036] It should be noted that this model is a two-stream network structure: time-series stream processing. Spatial Stream Processing The two-stream outputs are fused with F and then classified through a fully connected layer to output gesture style labels. The model is trained in a supervised manner, with historical sample sets annotated by experts.
[0037] This enables the capture, user separation, and basic quantitative description of raw touch commands, providing standardized input data for subsequent intent analysis. The user separation mechanism solves the problem of command source confusion during concurrent multi-user operations. The composite criteria based on time and displacement improve the robustness of click and swipe classification. Through deep learning of touch dynamics and spatial patterns, the system identifies the user's current operational style, providing important contextual information for subsequent intent matching. It can distinguish operations with similar basic characteristics but potentially different underlying intentions (for example, both are medium-speed swipes, but a "refined operation" style is more likely to rotate a small object, while a "fast browsing" style is more likely to adjust the viewing angle).
[0038] S102, determine the set of attribute information of at least one interactive object within the preset area covered by the touch point based on the touch point position of the touch command. This generates a candidate interaction intent set L, which includes operation intents related to the interactive object and / or global navigation intents. This refers to the attribute information of the Mth interactive object.
[0039] Specifically, for each interactive object within the preset area... Query its predefined attribute information This includes a unique ID, object type, physical properties, and allowed interaction types for each object. Its attribute information is represented as , As a unique identifier, The types of interactive objects (such as doors, windows, lights, switches, etc.) are derived from a preset type set. Physical properties, including moment of inertia. Operating resistance coefficient Current state value (All settings are pre-configured) This is the set of allowed interaction types for the interactive object (such as turning on, turning off, rotating, etc.).
[0040] For each possible operation, a candidate interaction intent is generated, forming a set of candidate interaction intents. Each candidate interaction intent represents a permitted interaction operation to be performed on a specific interactive object. Candidate interaction intents include operation intents related to the interactive object and / or global navigation intents (such as view adjustment and stepping).
[0041] For each interactive object Its allowed set of interaction types is For each Generate a candidate interaction intent , indicating "for interactive objects" Execute operation Global navigation intents are not associated with specific interactive objects, but they are still based on touch trajectories. For example, swipe trajectories are associated with view adjustment commands, and click trajectories are associated with step commands. Each candidate interactive intent I can be represented by a tuple: I = (target interactive object ID, interaction type). For global navigation intents, the target interactive object ID is a null value.
[0042] Therefore, in step S102, generating the candidate interaction intent set L specifically includes: For each interactive object Iterate through the set of allowed interaction types. Each interaction type Generate candidate interaction intents , This is a unique ID identifier for the interactive object; simultaneously, a global navigation intent is added based on the trajectory type characteristic T: if T is a swipe, a view adjustment intent is added. If T is a click, add a step intention. , Indicates null values; yields a set of candidate interaction intents. .
[0043] Specifically, the preset area R is a circular area with a fixed radius r centered on the touch point. The radius can be set according to the screen ratio and the density of interactive objects, for example, the radius... , The minimum distance between interactive objects in the current VR scene. The screen width is specified. If the number of interactive objects within R exceeds the preset maximum number, the maximum number will be determined. (That is, the maximum number of interactive objects considered in a single touch to ensure processing efficiency), then only the closest object to the touch point is retained. Each interactive object, thereby determining The preset upper limit of quantity is determined through expert experience and historical data experiments, which will not be elaborated upon in this invention. It should be noted that the coefficient values of 0.05 and 0.5 corresponding to the screen width and minimum distance, respectively, can also be set according to the actual scenario (distribution of the screen and interactive objects). The purpose of this design is to balance operational error tolerance and target clarity, and guidance is provided for adjusting these coefficients to adapt to different application scenarios (such as using a smaller coefficient in indoor scenarios with dense furniture).
[0044] S103 combines trajectory type feature T, physical feature vector F, and gesture style label. Calculate the matching score for each candidate interaction intent in the set of candidate interaction intents corresponding to the touch command.
[0045] In some embodiments, based on each candidate interaction intent For candidate interaction intents related to interactive objects, the matching score is:
[0046]
[0047] in, , , These are adjustable weighting coefficients, summing to 1, and are determined based on actual needs and expert experience. The first matching degree, i.e., the trajectory type matching degree, ranges from 0 to 1 and is obtained according to a preset mapping table (preset trajectory type - interaction type matching degree lookup value). T is the trajectory type feature. Allowed interaction types; The second matching degree, which is the matching degree between physical features and gesture style, ranges from 0 to 1. This second matching value depends on the interaction type of the intent under the current gesture style label. Interactive objects The ideal operational characteristics; F is the physical feature vector. For gesture style tags, Physical properties; The third matching degree, namely user-type correlation, ranges from 0 to 1. It represents the correlation between the first user's historical behavior profile and the type of interactive object. It is derived based on the historical interaction frequency between the first user and the interactive object type (the higher the frequency, the greater the correlation; the frequency is normalized and used as the profile) or preset preferences. This is the first user's historical behavior profile data, including historical interaction preference information. It is an interactive object type.
[0048] For candidate interaction intents of global navigation intent, their matching score is determined based on the first matching score and the matching score with the gesture style label (the calculation method of the ideal feature value in the calculation of the second matching score can be referred to, but the ideal feature value of the global navigation intent under the gesture style label is substituted, which does not involve the related attributes of the interactive object. The calculation principle is the same, and this invention will not elaborate on it). Each weight is set to 0.5.
[0049] Specifically, the second matching degree is used to measure the closeness between actual physical features and ideal operational features under a specific style, and is calculated according to the following formula:
[0050] in, , Intent interaction type In the physical properties of interactive objects With gesture style tags The ideal eigenvalues are determined by selecting the most frequent eigenvalues from a large number of historical physical eigenvectors, or by analyzing and determining them based on expert experience. As the normalization factor, Feature weights (used to differentiate the importance of different elements in the physical feature vector to the second matching degree; determined by experts based on the actual scenario; sum to 1; larger values indicate greater importance), ideal feature values. Learned from labeled data, different Corresponding to different ideal feature patterns.
[0051] Therefore, this matching score determination method can dynamically adapt to different user operation modes, improving the score's discriminative ability.
[0052] S104, The matching score is higher than the preset threshold. The final target interaction intent is determined from the candidate interaction intents.
[0053] Among them, the preset threshold This is determined based on the experience of historical experts and is used to measure the reliability of the match.
[0054] In some embodiments, S104 specifically includes: From the candidate interaction intent set L, those with a matching score higher than a preset threshold are selected to form a high-confidence candidate set C; if =1, then the candidate interaction intent is directly determined as the target interaction intent; if hour( (representing the number of candidate interaction intents in this subset), for each candidate interaction intent Combining the pre-set state network graph G associated with interactive objects with the determined meta-intent Calculate its impact assessment value. And a comprehensive matching score Determine the final target interaction intent .
[0055] Specifically, when there are multiple candidate interaction intents with matching scores exceeding a preset threshold, an arbitration decision-making stage is entered. This involves: for each candidate interaction intent, simulating the chain of state changes triggered by its execution using a pre-set state network diagram, calculating its execution impact assessment value, and, based on the decision weights dynamically adjusted by the meta-intent, combining the matching score and the normalized impact assessment value, determining the final target interaction intent from multiple candidate interaction intents and executing it. For example, for each candidate interaction intent, the target node it directly operates on is identified. and expected changes in state Using the state network graph G, from Starting from the nodes that are reachable by its influence, we traverse the state transition function of the edges. Calculate the set of all affected nodes. and its predicted state change Next, the execution impact assessment value of the candidate interaction intent is calculated.
[0056] Specifically, the decision-making process considers two aspects: first, the matching score of the candidate interaction intent; and second, the evaluation value of the execution impact of the candidate interaction intent in the state network, which calculates the range and intensity of the chain state changes that may be triggered after the execution of the candidate interaction intent. The trade-off in the decision is dynamically adjusted by the meta-intent: in the "target search" mode, more emphasis is placed on the matching degree, while in the "system exploration" mode, more attention is paid to the scope of impact. Finally, the target interaction intent is selected and executed, and the state of all affected interactive objects is updated synchronously in the state network to render a logically coherent scene feedback.
[0057] In some embodiments, for candidate interaction intents (Operation nodes in the corresponding state network diagram) and state change ), implementation impact assessment value Defined as:
[0058] in, Given the set of affected nodes, the operation nodes are traversed from the state network graph G. The reachable nodes from the starting point are obtained. For nodes The predicted change in state, This is the node importance coefficient, based on the corresponding affected nodes and operating nodes. The strength of the dependency is determined.
[0059] In some embodiments, the final target interaction intent is determined as follows:
[0060] in, , for preset The balancing parameters are used to adjust the weights of the matching score and the impact assessment value, and are determined based on expert experience and the actual scenario. The maximum value of the impact assessment among all candidate interaction intents is used for normalization. The balancing parameter can also be determined offline or online using reinforcement learning or optimization algorithms (such as gradient descent) based on historical interaction data to maximize metrics such as user satisfaction or task completion rate.
[0061] In other embodiments, it is understood that the meta-intent can be used to influence two aspects: Node importance coefficient Under different meta-intents, users focus on different things. For example, under "target search," interactive nodes related to the target are highly important, while under "system exploration," all nodes are of even importance; their values are related to the meta-intent. Relatedly, different meta-intentions have different emphases on the importance of nodes, which are set between 0 and 1.
[0062] Balance parameters Adjust the match score Compared with the impact assessment value The weighting of different factors, for example, in "target search", more emphasis is placed on matching degree (larger α, set to 0.8) to quickly respond to the user's explicit intent, while in "system exploration", more emphasis is placed on impact assessment (smaller α, set to 0.3) to encourage trial and error and discovery.
[0063] Alternatively, a meta-intent-policy configuration table can be pre-defined. For example, when the meta-intent is target search, the balance parameter is set to 0.8, and nodes of type "navigation markers" are assigned a higher importance coefficient.
[0064] Therefore, meta-intention can dynamically adjust decision-making strategies and improve the long-term consistency of interactions.
[0065] In some embodiments, the state network graph G=(V,E) contains a set of nodes. Each node Related Interaction Objects and its state vector In the set of directed edges E, each edge : Represents a node State changes may cause nodes State changes, associated state transition functions Defined from arrive The mapping relationship.
[0066] State network graphs are typically stored in the form of an adjacency matrix A, representing the dependencies between nodes. , , If there exists a slave node arrive The dependent edges, then Indicates the dependency strength; otherwise, it is 0.
[0067] The mapping relationship can be in linear form: The weight parameter matrix of the state transition function By pre-determining physical laws, learning from data, or manually designing, the propagation laws of state changes can be quantified. This is the weight matrix. , They are state vectors , The dimension of dependency strength can be set to [value]. (Frobenius norm).
[0068] For example, physical laws presuppose that, for instance, changes in the opening degree of a window affect the intensity of indoor light. The corresponding elements can be calculated based on the lighting model; data learning: obtained by recording historical state change data and fitting it with linear regression or neural networks; manual design: set by scene designers according to logical relationships.
[0069] In other examples, by recording the sequence of scene state changes after multiple user actions in a VR scene, and using data learning (neural networks or system identification), an approximate transitive relationship between state variables is fitted, thereby quantifying the changes. The estimated value.
[0070] For example, node Corresponding interactive objects Its state vector This includes, but is not limited to, position, orientation, opening degree, brightness, and on / off status, with the specific dimensions determined by the type of interactive object.
[0071] To further clarify, the state network graph G=(V,E) is used to describe the states of all interactants in the scene and their dependencies. Each node... Representing interactive objects and its current state vector Each directed edge Associate a state transition function Used to define nodes State changes How does time affect nodes? state change .
[0072] Specifically, the meta-intention is determined as follows: Based on the first user's historical interaction sequence (including historical intent, physical characteristics, and gesture style) within a preset historical time window, a preset meta-intent recognition model is used to determine the first user's current macro-interaction strategy. For example, meta-intent can be system exploration (broad browsing without a specific goal), target search (finding a specific item or area), functional testing (trying out interactive object functions), etc. It reflects the user's long-term intent and is used to guide micro-intent decisions, making the system more in line with the user's overall goals.
[0073] For example, let the historical interaction sequence be... L is the preset historical time window length. The interactive intent executed at historical moment t. and These are the corresponding physical feature vectors and gesture style labels, respectively.
[0074] For example, the meta-consciousness recognition model is set as a Long Short-Term Memory (LSTM) network. The input is a concatenated feature vector at each time step in H (composed of intent embedding, physical feature vector, and gesture style label embedding). Through sequence learning, the final hidden state of the LSTM is classified through a fully connected layer, outputting the meta-intent probability distribution corresponding to the current time step. The category corresponding to the highest probability is taken as the determined meta-intent. Meta-intents represent the user's macro-level interaction strategy in the current session. During model training, based on a controlled experimental environment or real-world application logs, touch trajectory data and corresponding historical interaction sequences of users when performing known tasks (such as 'Please turn on the living room lights') are collected. Experimenters or those working backward from the task objectives then label the data with corresponding meta-intent tags to generate a training sample set for training a pre-selected model architecture.
[0075] Therefore, based on the initial matching degree screening, an impact assessment based on the scenario logic chain is introduced, and decision preferences are dynamically adjusted using meta-intent (for example, when "target search" is performed, matching degree is emphasized for a quick response, and when "system exploration" is performed, impact is emphasized to encourage discovery), thereby achieving the optimal balance between accurately capturing the user's immediate intent and maintaining the macro-reasonableness of the scenario.
[0076] S105. Execute the final target interaction intent in the target VR panoramic touch interface, update the state of related (all affected) interactive objects according to the state network diagram, and render the corresponding visual effects and interactive feedback.
[0077] Specifically, S105 includes: In the VR panoramic touch interface, execute the final target interaction intent. If it is an interaction object operation intent, drive the corresponding interaction object model to execute the corresponding interaction type. The specified animation, and its state updated to Based on the calculated set of affected nodes and predicted state changes According to the dependency order in the state network, the state updates of the nodes are triggered sequentially or in parallel, and the corresponding visual effects (such as changes in lighting, movement of related objects, etc.) are rendered. If it is a global navigation intent (viewpoint adjustment or stepping), the user's viewpoint or position can be adjusted directly.
[0078] This transforms decision-making into actual visual feedback and scene state changes. Execution is not limited to direct manipulation of objects, but involves chain updates based on the state network, thereby ensuring that the entire scene's response conforms to the physical or logical laws of the real world, greatly enhancing immersion and consistency.
[0079] In summary, this embodiment aims to address the problems in existing technologies, such as inaccurate recognition of interaction intent due to ambiguity in touch operations and broken interaction logic due to the isolation of interactive object states within a scene. By introducing multi-dimensional physical feature analysis, gesture style recognition, state network modeling, and meta-intent inference, a hierarchical intelligent decision-making framework is constructed to achieve more accurate parsing of user operation intent and more coherent maintenance of scene states.
[0080] Breaking away from the limitations of merely analyzing basic trajectories, this system identifies the "style" of user gestures through neural networks, enabling intent recognition to understand subtle differences in the same operation across different behavioral patterns. By analyzing user interaction sequences to infer macro-strategies (meta-intents), and using these strategies to dynamically adjust the focus of micro-intent decisions, it achieves long-term interaction goals guiding immediate actions. Through explicit modeling of state dependencies between interactive objects, the system can predict and render chain reactions triggered by a single operation, ensuring logical consistency and realism in VR scene interactions.
[0081] This embodiment constructs a closed-loop intelligent interaction method that ranges from micro-operation analysis to macro-strategy guidance and scene logic maintenance. It places an isolated touch click or swipe within three contexts—"user's real-time behavior pattern (gesture style)," "user's staged modal goals (meta-intent)," and "the inherent logic of the virtual environment (state network)"—for joint interpretation and decision-making. This significantly improves intent recognition accuracy. Through multi-dimensional feature fusion and contextual understanding, it greatly reduces false triggers and operational ambiguity. It achieves intelligent scene responses; operations are no longer isolated but trigger chain reactions consistent with common sense, greatly enhancing the realism and credibility of the VR world. It provides an adaptive interactive experience, dynamically adjusting response strategies based on different user operation styles and staged goals, making interaction more natural and efficient. It supports complex multi-terminal collaboration, clear intent parsing, and scene state management, providing a technical foundation for multiple users to interact naturally and without conflict in the same scene.
[0082] Example 2 Figure 2 This is a VR multi-terminal collaborative intelligent interaction system according to an embodiment of the present invention.
[0083] like Figure 2 As shown, a VR multi-terminal collaborative intelligent interaction system includes: an acquisition module, a filtering module, a matching module, and a determination module; The acquisition module is used to respond to the touch command received from the first user on the target VR panoramic touch interface, acquire the touch trajectory data stream, extract trajectory type features and physical feature vectors, and generate gesture style tags corresponding to the touch command. The filtering module is used to determine the attribute information set of at least one interactive object contained within the preset area covered by the touch point according to the touch point position of the touch instruction, and generate a candidate interactive intent set L, which includes operation intents and / or global navigation intents related to the interactive object. The matching module is used to combine trajectory type features, physical feature vectors and gesture style labels to calculate the matching score of each candidate interaction intent in the set of candidate interaction intents corresponding to the touch command. The determining module is used to determine the final target interaction intent from candidate interaction intents whose matching score is higher than a preset threshold.
[0084] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0085] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0086] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0087] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0088] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0089] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0090] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A VR multi-terminal collaborative intelligent interaction method, characterized in that, include: S101, in response to the touch command received from the first user on the target VR panoramic touch interface, acquire touch trajectory data stream, extract trajectory type features and physical feature vectors, and generate gesture style labels corresponding to the touch command; S102, determine the attribute information set of at least one interactive object within the preset area covered by the touch point according to the touch point position of the touch instruction, and generate a candidate interaction intent set L, the candidate interaction intents include operation intents and / or global navigation intents related to the interactive object. S103, combining trajectory type features, physical feature vectors and gesture style labels, calculates the matching score of each candidate interaction intent in the set of candidate interaction intents corresponding to the touch command; S104, determine the final target interaction intent from the candidate interaction intents whose matching score is higher than the preset threshold.
2. The VR multi-terminal collaborative intelligent interaction method according to claim 1, characterized in that, The touch trajectory data stream is a time series. , For timestamps, The screen coordinates of the touch point N represents the touch pressure value, and N represents the number of touch points within the touch duration window from the start to the end of the touch.
3. The VR multi-terminal collaborative intelligent interaction method according to claim 2, characterized in that, The trajectory type feature is set to click or swipe; The physical feature vector includes average touch pressure. instantaneous velocity standard deviation Mean normal acceleration and mean trajectory curvature .
4. The VR multi-terminal collaborative intelligent interaction method according to claim 3, characterized in that, The gesture style label is based on touch trajectory data stream and obtained according to a pre-trained spatiotemporal pattern recognition model. It is used to characterize the operation behavior pattern of the first user when completing a touch command. The generation method specifically includes: High-order temporal and spatial distribution features of the touch trajectory data stream are extracted, and these features are combined with the physical feature vector F to form touch features. The features are then input into a pre-trained spatiotemporal pattern recognition neural network model, which outputs the gesture style label corresponding to the touch command.
5. A VR multi-terminal collaborative intelligent interaction method according to claim 4, characterized in that, The pre-trained spatiotemporal pattern recognition neural network model is obtained as follows: A1. Collect a large number of historical sample sets , For the first Touch characteristics of each sample For the corresponding gesture style tags; A2. Using historical sample sets, the pre-selected neural network structure is trained, and the model parameters are continuously optimized to obtain the final spatiotemporal pattern recognition neural network model.
6. The VR multi-terminal collaborative intelligent interaction method according to claim 5, characterized in that, For each interactive object within the preset area Query its predefined attribute information , As a unique identifier, For interactive object type, Physical properties This is the set of allowed interaction types for this interactive object.
7. A VR multi-terminal collaborative intelligent interaction method according to claim 6, characterized in that, The calculation of the matching score for each candidate interaction intent in the set of candidate interaction intents corresponding to the touch command is as follows: Based on each candidate interaction intent For candidate interaction intents related to interactive objects, the matching score is: in, , , These are adjustable weighting coefficients, summing to 1, and are determined based on actual needs and expert experience. The first matching degree is the trajectory type matching degree; The second matching degree, namely the matching degree between physical features and gesture style, depends on the interaction type of the intent under the current gesture style label. Interactive objects Ideal operating characteristics; The third matching degree is the correlation between the first user's historical behavior profile and the type of interactive object.
8. A VR multi-terminal collaborative intelligent interaction method according to claim 7, characterized in that, The second matching degree is calculated according to the following formula: in, , Intent interaction type In the physical properties of interactive objects With gesture style tags Ideal eigenvalues under the following conditions As the normalization factor, These are the feature weights.
9. A VR multi-terminal collaborative intelligent interaction method according to claim 7, characterized in that, S104 specifically includes: Select candidates with matching scores higher than a preset threshold from the candidate interaction intent set L to form a high-confidence candidate set C; like If the value is 1, then the candidate interaction intent is directly determined as the target interaction intent; like At that time, for each candidate interaction intent Combining the pre-set state network graph G with the determined meta-intent Calculate its impact assessment value. And a comprehensive matching score Determine the final target interaction intent .
10. A VR multi-terminal collaborative intelligent interaction system, characterized in that, The VR multi-terminal collaborative intelligent interaction method according to any one of claims 1-9 is characterized in that the system comprises: an acquisition module, a filtering module, a matching module, and a determination module; The acquisition module is used to respond to the touch command received from the first user on the target VR panoramic touch interface, acquire the touch trajectory data stream, extract trajectory type features and physical feature vectors, and generate gesture style tags corresponding to the touch command. The filtering module is used to determine the attribute information set of at least one interactive object contained within the preset area covered by the touch point according to the touch point position of the touch instruction, and generate a candidate interactive intent set L, which includes operation intents and / or global navigation intents related to the interactive object. The matching module is used to combine trajectory type features, physical feature vectors and gesture style labels to calculate the matching score of each candidate interaction intent in the set of candidate interaction intents corresponding to the touch command. The determining module is used to determine the final target interaction intent from candidate interaction intents whose matching score is higher than a preset threshold.