Intelligent interactive control method based on virtual experience scene digital model

Abstract

The invention relates to an intelligent interactive control method based on a virtual experience scene digital model, and belongs to the technical field of digital twinning. The method comprises the following steps: acquiring a digital model of a target virtual experience scene; collecting multi-modal interactive input data of a user in real time; the multi-modal interaction input data comprises limb movement data and voice instruction data of the user; performing scene state analysis according to the digital model and the real-time multi-mode interactive input data, and generating a self-adaptive control strategy; and driving the virtual experience scene digital model to perform state updating based on a self-adaptive control strategy, and generating and outputting a virtual scene state matched with the updated user interaction behavior. High intelligence and self-adaption of interaction between the virtual experience scene and the user are achieved, and the immersion and interaction experience of the user in the virtual environment are greatly improved.

Description

Technical Field

[0001] This invention belongs to the field of digital twin technology, specifically relating to an intelligent interactive control method based on a digital model of a virtual experience scene. Background Technology

[0002] Virtual reality (VR) technology is currently evolving from one-way display to immersive intelligent interaction, with users expecting near-realistic dynamic responses and natural interactive experiences in virtual environments. However, existing technologies face significant bottlenecks in areas such as multimodal interaction, physical simulation accuracy, and real-time synchronization.

[0003] In terms of multimodal interaction, existing technologies struggle to accurately and comprehensively capture and understand the rich variety of user body movements and voice commands, resulting in insufficient accuracy and fluency in the interaction. For example, some complex and subtle body movements may not be accurately recognized, and the parsing of voice commands may also be inaccurate, failing to accurately match operations in virtual scenes.

[0004] In terms of physical simulation accuracy, existing methods cannot accurately simulate the motion transmission and state evolution rules between virtual objects. The motion of objects in virtual scenes may not conform to the laws of real physics, making it difficult for users to obtain a realistic experience during interaction and reducing immersion. For example, physical phenomena such as object collisions and movements are not portrayed realistically.

[0005] Real-time synchronization is also a significant issue. Existing technologies exhibit noticeable delays in translating user interactions into virtual scene state updates. This prevents timely feedback from user actions within the virtual scene, disrupting the real-time nature and continuity of the interaction and impacting the user experience. For example, after a user performs an action, the virtual scene may take some time to change accordingly.

[0006] These bottlenecks severely limit the further development and application of virtual reality technology in various fields, such as education, games, and training. Therefore, there is an urgent need for a new technical approach to solve these problems. The intelligent interactive control method based on the digital model of virtual experience scene proposed in this invention is designed to address these challenges. Summary of the Invention

[0007] To address the aforementioned problems in the existing technology, this invention provides an intelligent interactive control method based on a digital model of a virtual experience scene; The objective of this invention can be achieved through the following technical solutions: An intelligent interactive control method based on a digital model of a virtual experience scenario includes: S1: Obtain a digital model of the target virtual experience scene; the digital model includes spatial structure information, object attribute information, physical rule information, and environmental state information of the virtual experience scene; S2: Real-time acquisition of user's multimodal interactive input data; the multimodal interactive input data includes user's body movement data and voice command data; S3: Analyze the scene state based on the digital model and the real-time multimodal interactive input data, and generate an adaptive control strategy; S4: Based on the adaptive control strategy, drive the digital model of the virtual experience scene to update its state, generate and output the virtual scene state that matches the updated user interaction behavior.

[0008] Furthermore, the spatial structure information is generated by topological mesh data through 3D point cloud reconstruction and BIM modeling, which includes the geometric relationships and spatial constraints of static structures in the scene; the object attribute information is an interactive marker bound to the virtual object; the physical rule information is a mathematical constraint equation used to define the motion transmission and state evolution rules between virtual objects; and the environmental state information is a set of global parameters that evolve over time.

[0009] Furthermore, the multimodal interactive input data is collected by an inertial measurement unit array and a microphone array to acquire the user's joint pose data and raw audio stream in the motion capture area. Based on the joint pose data and raw audio stream collected by the motion state recognition model and the speech recognition model, motion topology recognition and audio conversion are performed to obtain limb motion data and voice command data.

[0010] Specifically, the motion state recognition model converts the Euclidean coordinates of the joints acquired by the IMU array into a rigid body pose matrix through a pose manifold transformation unit, and establishes a continuous and smooth motion trajectory through algebraic mapping; it converts the joint motion into a state transition on the topological structure by constructing a spatiotemporal topological graph with human joints as vertices and skeletal links as edges; it learns and classifies the topological state transitions based on a graph neural network to identify different motion states; the motion state recognition model is trained with labeled motion data to identify the user's motion state from the joint pose data.

[0011] Specifically, the speech recognition model constructs a knowledge graph by combining the object attribute information, generates a vocabulary constraint vector based on a graph attention network, extracts local features and sequence information from the original audio data through a speech recognition network constructed by a convolutional neural network and a recurrent neural network, converts the audio data into corresponding text, calculates the lexical probability in the output layer of the graph attention network according to the vocabulary constraint vector and the knowledge graph, parses the operation predicates and parameters in the text, and outputs structured speech command data according to the state of the interactive marker bits.

[0012] Preferably, the method for scene state analysis includes: The recognized limb movement data is converted into a rigid body transformation matrix, and the effective domain of the movement in the virtual scene is calculated based on the geometric relationships in the spatial structure information. The target object identifier in the voice command data is parsed, and the interactivity flag bit in the object attribute information is queried based on the target object identifier to generate a binding relationship.

[0013] Furthermore, the adaptive control strategy simulates the impact of the physical rule information on limb movement data and voice command data in the virtual experience scene, and generates a control strategy that meets the current scene state and user interaction needs based on the real-time changes of the scene and global parameters, combined with environmental state information.

[0014] Specifically, the method for updating the state is as follows: The motion state vector of the object attribute information is updated according to the adaptive control strategy, and the interactivity flag is dynamically adjusted. When the motion state changes and triggers the topology update condition, local mesh reconstruction is performed based on the spatial constraint condition. At the same time, the environmental parameters are updated according to the global evolution equation in the physical rule information. The updated virtual scene state is rendered and output in real time through a two-way dynamic synchronization mechanism.

[0015] Specifically, the virtual scene state is a real-time snapshot in a four-dimensional state space, including a geometric state layer, a physical property state layer, a physical rule instance layer, and an environment evolution layer; The geometric state layer is used to inherit the reconstructed topological mesh data after state updates; the physical property state layer is used to store dynamically updated object attribute information sets; the physical rule instance layer is used to store application instances of physical rules between virtual objects, and the application instances are dynamically generated and adjusted according to environmental state information and user interaction data; the environment evolution layer is used to record the change process of rendering information over time and is updated through the global evolution equation in the physical rule information.

[0016] Specifically, the implementation method of the bidirectional dynamic synchronization mechanism is as follows: The updated spatial structure information, object attribute information, and environmental state information are encoded into incremental data packets by timestamps through a scene state synchronization queue. These incremental data packets are then transmitted to the rendering engine or distributed processing nodes via a high-speed data communication network. The rendering engine decodes the incremental data packets in timed order and applies the updates to the virtual scene digital model. Simultaneously, it dynamically adjusts environmental parameters based on the global evolution equations in the physical rule information. The incremental data packets employ differential encoding and compression techniques, transmitting only the changes since the last state update, and data security is ensured through encryption mechanisms. At the receiving end, state updates are triggered by an event-driven mechanism. If data packet loss or out-of-order delivery is detected, reordering or retransmission is requested based on the timestamp. The feedback loop compares the rendered virtual scene state with user interaction data, generates control commands, and synchronizes them back to the physical system, achieving real-time bidirectional interaction between the virtual model and the physical system.

[0017] The beneficial effects of this invention are as follows: This invention effectively overcomes the bottlenecks of existing virtual reality technologies in multimodal interaction, physical simulation accuracy, and real-time synchronization by accurately capturing and parsing multimodal interaction data, meticulously analyzing scene states, dynamically generating adaptive control strategies, and achieving real-time state updates and bidirectional synchronization. It can accurately and comprehensively capture and understand the rich variety of user body movements and voice commands, improving the accuracy and fluency of interaction; it can precisely simulate the motion transmission and state evolution rules between virtual objects, making the movement of objects in the virtual scene more consistent with real-world physical laws, enhancing the user's immersion; and it achieves real-time bidirectional interaction between the virtual model and the physical system, greatly reducing the delay between user operations and virtual scene feedback, and improving the real-time nature and coherence of the interaction. This method provides strong support for the further development and application of virtual reality technology in various fields such as education, gaming, and training, bringing users a near-realistic dynamic response and natural interactive experience, and propelling virtual reality technology to a higher level. Attached Figure Description

[0018] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0019] Figure 1 This is a flowchart illustrating an intelligent interactive control method based on a digital model of a virtual experience scene according to the present invention. Figure 2 This is a technical structure block diagram of an intelligent interactive control method based on a digital model of a virtual experience scene in this invention; Figure 3 This is a schematic diagram of the timing structure of an intelligent interactive control method based on a digital model of a virtual experience scene in this invention. Detailed Implementation

[0020] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0021] Please see Figure 1-3 An intelligent interactive control method based on a digital model of a virtual experience scenario includes: S1: Obtain a digital model of the target virtual experience scene; the digital model includes spatial structure information, object attribute information, physical rule information, and environmental state information of the virtual experience scene; S2: Real-time acquisition of user's multimodal interactive input data; the multimodal interactive input data includes user's body movement data and voice command data; S3: Analyze the scene state based on the digital model and the real-time multimodal interactive input data, and generate an adaptive control strategy; S4: Based on the adaptive control strategy, drive the digital model of the virtual experience scene to update its state, generate and output the virtual scene state that matches the updated user interaction behavior.

[0022] Furthermore, the spatial structure information is generated by topological mesh data through 3D point cloud reconstruction and BIM modeling, which includes the geometric relationships and spatial constraints of static structures in the scene; the object attribute information is an interactive marker bound to the virtual object; the physical rule information is a mathematical constraint equation used to define the motion transmission and state evolution rules between virtual objects; and the environmental state information is a set of global parameters that evolve over time.

[0023] Furthermore, the multimodal interactive input data is collected by an inertial measurement unit array and a microphone array to acquire the user's joint pose data and raw audio stream in the motion capture area. Based on the joint pose data and raw audio stream collected by the motion state recognition model and the speech recognition model, motion topology recognition and audio conversion are performed to obtain limb motion data and voice command data.

[0024] Specifically, the motion state recognition model converts the Euclidean coordinates of the joints acquired by the IMU array into a rigid body pose matrix through a pose manifold transformation unit, and establishes a continuous and smooth motion trajectory through algebraic mapping; it converts the joint motion into a state transition on the topological structure by constructing a spatiotemporal topological graph with human joints as vertices and skeletal links as edges; it learns and classifies the topological state transitions based on a graph neural network to identify different motion states; the motion state recognition model is trained with labeled motion data to identify the user's motion state from the joint pose data.

[0025] Specifically, the speech recognition model constructs a knowledge graph by combining the object attribute information, generates a vocabulary constraint vector based on a graph attention network, extracts local features and sequence information from the original audio data through a speech recognition network constructed by a convolutional neural network and a recurrent neural network, converts the audio data into corresponding text, calculates the lexical probability in the output layer of the graph attention network according to the vocabulary constraint vector and the knowledge graph, parses the operation predicates and parameters in the text, and outputs structured speech command data according to the state of the interactive marker bits.

[0026] Preferably, the method for scene state analysis includes: The recognized limb movement data is converted into a rigid body transformation matrix, and the effective domain of the movement in the virtual scene is calculated based on the geometric relationships in the spatial structure information. The target object identifier in the voice command data is parsed, and the interactivity flag bit in the object attribute information is queried based on the target object identifier to generate a binding relationship.

[0027] Furthermore, the adaptive control strategy simulates the impact of the physical rule information on limb movement data and voice command data in the virtual experience scene, and generates a control strategy that meets the current scene state and user interaction needs based on the real-time changes of the scene and global parameters, combined with environmental state information.

[0028] Specifically, the method for updating the state is as follows: The motion state vector of the object attribute information is updated according to the adaptive control strategy, and the interactivity flag is dynamically adjusted. When the motion state changes and triggers the topology update condition, local mesh reconstruction is performed based on the spatial constraint condition. At the same time, the environmental parameters are updated according to the global evolution equation in the physical rule information. The updated virtual scene state is rendered and output in real time through a two-way dynamic synchronization mechanism.

[0029] Specifically, the virtual scene state is a real-time snapshot in a four-dimensional state space, including a geometric state layer, a physical property state layer, a physical rule instance layer, and an environment evolution layer; The geometric state layer is used to inherit the reconstructed topological mesh data after state updates; the physical property state layer is used to store dynamically updated object attribute information sets; the physical rule instance layer is used to store application instances of physical rules between virtual objects, and the application instances are dynamically generated and adjusted according to environmental state information and user interaction data; the environment evolution layer is used to record the change process of rendering information over time and is updated through the global evolution equation in the physical rule information.

[0030] Specifically, the implementation method of the bidirectional dynamic synchronization mechanism is as follows: The updated spatial structure information, object attribute information, and environmental state information are encoded into incremental data packets by timestamps through a scene state synchronization queue. These incremental data packets are then transmitted to the rendering engine or distributed processing nodes via a high-speed data communication network. The rendering engine decodes the incremental data packets in timed order and applies the updates to the virtual scene digital model. Simultaneously, it dynamically adjusts environmental parameters based on the global evolution equations in the physical rule information. The incremental data packets employ differential encoding and compression techniques, transmitting only the changes since the last state update, and data security is ensured through encryption mechanisms. At the receiving end, state updates are triggered by an event-driven mechanism. If data packet loss or out-of-order delivery is detected, reordering or retransmission is requested based on the timestamp. The feedback loop compares the rendered virtual scene state with user interaction data, generates control commands, and synchronizes them back to the physical system, achieving real-time bidirectional interaction between the virtual model and the physical system.

[0031] In this embodiment, taking a museum virtual reality experience as an example, a 3D laser scanner is used to collect point cloud data of the actual museum. A topological mesh is generated using Scan-to-BIM technology. During semantic segmentation, entities such as walls, display cases, and cultural relics are automatically identified. The point cloud is converted into a BIM model with geometric constraints, including physical attributes such as wall thickness and display stand load-bearing capacity. The topological structure is output, defining the visitor walking boundaries (spatial constraints) and the collision volume of display cases. Interactive markers (for rotational observation and audio narration) are added to interactive cultural relics. The rigid body dynamics equations for the rotation of cultural relics and the light reflection model are defined through physical rules.

[0032] In the user behavior capture area, an array of IMU sensors is used to collect the user's joint poses. The user's "reaching out" action is converted into a rigid body transformation matrix, and the Euclidean distance between the user and the artifact model is calculated. If the distance is less than the interaction threshold and the artifact's object attributes are interactive, the action is considered a valid interaction. When the user rotates the artifact, a topology update condition is triggered, and local redrawing is performed based on the BIM mesh. Environmental parameters are updated according to the global evolution equation and through a two-way dynamic synchronization mechanism. The updated virtual museum scene is rendered and output to users in real time, ensuring that users see scene changes that are synchronized with their actions and conform to physical rules. For example, when a user requests to view detailed information about a cultural relic via voice command, the system quickly parses the target object identifier in the voice command data, finds the interactive marker of the corresponding cultural relic, simulates the impact of the operation on the virtual scene based on physical rules, and generates control strategies to update the display state of the cultural relic, such as displaying text descriptions or playing audio narration. At the same time, the system dynamically adjusts the interactive marker and motion state vector of the cultural relic according to the adaptive control strategy. If the cultural relic has environmental parameters such as virtual heat value due to frequent viewing, it will also be updated according to the global evolution equation. Through a two-way dynamic synchronization mechanism, these updated virtual scene states are quickly and accurately transmitted to the user's end for rendering in the form of incremental data packets, allowing users to obtain a coherent, natural, and highly immersive interactive experience in the virtual museum experience.

[0033] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An intelligent interactive control method based on a digital model of a virtual experience scene, characterized in that, include: S1: Obtain a digital model of the target virtual experience scene; the digital model includes spatial structure information, object attribute information, physical rule information, and environmental state information of the virtual experience scene; S2: Real-time acquisition of user's multimodal interactive input data; the multimodal interactive input data includes user's body movement data and voice command data; S3: Analyze the scene state based on the digital model and the real-time multimodal interactive input data, and generate an adaptive control strategy; S4: Based on the adaptive control strategy, drive the digital model of the virtual experience scene to update its state, generate and output the virtual scene state that matches the updated user interaction behavior.

2. The method according to claim 1, characterized in that, The spatial structure information is generated by topological mesh data through 3D point cloud reconstruction and BIM modeling, which includes the geometric relationships and spatial constraints of static structures in the scene; the object attribute information is an interactive marker bound to the virtual object; the physical rule information is a mathematical constraint equation used to define the motion transmission and state evolution rules between virtual objects; and the environmental state information is a set of global parameters that evolve over time.

3. The method according to claim 1, characterized in that, The multimodal interactive input data is collected by an inertial measurement unit array and a microphone array to acquire the user's joint pose data and raw audio stream in the motion capture area. Based on the joint pose data and raw audio stream collected by the motion state recognition model and speech recognition model, motion topology recognition and audio conversion are performed to obtain limb motion data and voice command data.

4. The method according to claim 3, characterized in that, The motion state recognition model converts the Euclidean coordinates of joint points acquired by the IMU array into rigid body pose matrices through a pose manifold transformation unit, and establishes a continuous and smooth motion trajectory through algebraic mapping. It converts joint point motion into state transitions on the topological structure by constructing a spatiotemporal topological graph with human joint points as vertices and skeletal links as edges. It learns and classifies the topological state transitions based on graph neural networks to identify different motion states. The motion state recognition model is trained with labeled motion data to identify the user's motion state from the joint point pose data.

5. The method according to claim 3, characterized in that, The speech recognition model constructs a knowledge graph by combining the object attribute information, generates a vocabulary constraint vector based on a graph attention network, extracts local features and sequence information from the original audio data through a speech recognition network constructed by a convolutional neural network and a recurrent neural network, converts the audio data into corresponding text, calculates the lexical probability in the output layer of the graph attention network according to the vocabulary constraint vector and the knowledge graph, parses the operation predicates and parameters in the text, and outputs structured speech command data according to the state of the interactive marker bits.

6. The method according to claim 1, characterized in that, The method for scene state analysis includes: The recognized limb movement data is converted into a rigid body transformation matrix, and the effective domain of the movement in the virtual scene is calculated based on the geometric relationships in the spatial structure information. The target object identifier in the voice command data is parsed, and the interactivity flag bit in the object attribute information is queried based on the target object identifier to generate a binding relationship.

7. The method according to claim 1, characterized in that, The adaptive control strategy simulates the impact of physical rule information on body movement data and voice command data in the virtual experience scenario. Combined with environmental state information, it generates a control strategy that meets the current scenario state and user interaction needs based on real-time changes in the scenario and global parameters.

8. The method according to claim 1, characterized in that, The method for updating the state is as follows: The motion state vector of the object attribute information is updated according to the adaptive control strategy, and the interactivity flag is dynamically adjusted. When the motion state changes and triggers the topology update condition, local mesh reconstruction is performed based on the spatial constraint condition. At the same time, the environmental parameters are updated according to the global evolution equation in the physical rule information. The updated virtual scene state is rendered and output in real time through a two-way dynamic synchronization mechanism.

9. The method according to claim 1, characterized in that, The virtual scene state is a real-time snapshot in a four-dimensional state space, including a geometric state layer, a physical property state layer, a physical rule instance layer, and an environment evolution layer. The geometric state layer is used to inherit the reconstructed topological mesh data after state updates; the physical property state layer is used to store dynamically updated object attribute information sets; the physical rule instance layer is used to store application instances of physical rules between virtual objects, and the application instances are dynamically generated and adjusted according to environmental state information and user interaction data; the environment evolution layer is used to record the change process of rendering information over time and is updated through the global evolution equation in the physical rule information.

10. The method according to claim 8, characterized in that, The implementation method of the bidirectional dynamic synchronization mechanism is as follows: The updated spatial structure information, object attribute information, and environmental state information are encoded into incremental data packets by timestamp through the scene state synchronization queue; and the incremental data packets are transmitted to the rendering engine or distributed processing nodes through a high-speed data communication network; the rendering engine decodes the incremental data packets in the order of timestamps, applies the updates to the virtual scene digital model, and dynamically adjusts the environmental parameters according to the global evolution equation in the physical rule information. The incremental data packets employ differential coding and compression techniques to transmit only the changes since the last state update, and data security is ensured through an encryption mechanism. At the receiving end, state updates are triggered through an event-driven mechanism. If data packet loss or out-of-order delivery is detected, reordering or retransmission is requested based on the timestamp. The feedback loop compares the rendered virtual scene state with user interaction data, generates control commands, and synchronizes them back to the physical system, enabling real-time bidirectional interaction between the virtual model and the physical system.

Images