A virtual-reality fusion interaction method and system based on digital twins

Abstract

This invention discloses a method and system for virtual-real fusion interaction based on digital twins, specifically relating to the field of digital twins and virtual-real interaction technology. It addresses the problem of difficulty in monitoring semantic drift during virtual-real fusion interaction in existing technologies. This is achieved by constructing a digital twin model in virtual space that is consistent with the current state of the physical entity; using a multimodal interaction algorithm to achieve accurate mapping between user operations and physical entities; analyzing energy changes in abnormal propagation paths using multi-scale dissipative dynamic networks to classify the possibility of semantic drift as global or local; when the possibility of semantic drift is local, assessing the degree of change in semantic entropy caused by user operations using the maximum entropy principle; and analyzing changes in the topology of the semantic graph using graph matching deformation networks to comprehensively evaluate the potential impact of semantic drift. This effectively improves the accuracy and stability of virtual-real interaction, providing technical support for semantic drift monitoring and processing in complex interaction scenarios.

Description

Technical Field

[0001] The present invention relates to the field of digital twin and virtual-reality interaction technology, and more specifically, to a virtual-reality fusion interaction method and system based on digital twin. Background Art

[0002] Virtual-reality fusion interaction based on digital twins refers to synchronizing the state, behavior and environment of physical entities to virtual space in real time through digital twin technology, and building a virtual-reality fusion system that can support real-time user interaction with the help of two-way mapping between virtual models and physical entities. However, when implementing virtual-reality fusion interaction, existing technologies usually lack the ability to accurately evaluate semantic changes in complex interactive scenarios. When the state of physical entities changes dynamically or user operations have nonlinear effects, the interaction between virtual models and physical entities often has semantic inconsistencies.

[0003] In the existing technology, the potential impact of semantic drift in the process of virtual-reality fusion interaction is difficult to quantify and predict, which will cause local or even global deviations in the semantic state of the system, thereby affecting the reliability and accuracy of the interaction and reducing the overall stability and user experience of the digital twin system.

[0004] In order to solve the above problems, a technical solution is now provided. Summary of the invention

[0005] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a virtual-reality fusion interaction method and system based on digital twins to solve the problems raised in the above-mentioned background technology.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] A virtual-reality fusion interaction method based on digital twins includes the following steps:

[0008] Build a digital twin model of the physical entity in the virtual space so that the digital twin model is consistent with the current state of the physical entity;

[0009] Provide an interactive interface between users and digital twin models in virtual space, and use multimodal interaction algorithms to ensure that user operations can be accurately mapped to physical entities;

[0010] The energy changes of abnormal propagation paths are analyzed through a multi-scale dissipative dynamic network, and the possibility of semantic drift is divided into global or local.

[0011] When the possibility of semantic drift is local, the interaction between user operations and virtual model feedback is analyzed by the maximum entropy principle, and the degree of change in semantic entropy caused by user operations is dynamically evaluated; the topological structure of the semantic graph of the digital twin model is analyzed by the graph matching deformation network, and the topological deformation amplitude of the semantic graph is evaluated;

[0012] Based on the degree of change in semantic entropy caused by user operations and the amplitude of topological deformation of the semantic graph, the potential impact of semantic drift in virtual-reality fusion interaction is evaluated.

[0013] In a preferred embodiment, a digital twin model of a physical entity is constructed in a virtual space so that the digital twin model is consistent with the current state of the physical entity, specifically:

[0014] Collect real-time status data of physical entities, including position, posture and speed;

[0015] Preprocess the collected real-time status data;

[0016] According to the shape and size parameters of the physical entity, a virtual geometric model consistent with the physical entity is constructed in the virtual space using 3D modeling software;

[0017] Map the real-time status data of the physical entity into the virtual geometric model;

[0018] Set the initial state of the digital twin model according to the state parameters of the current physical entity.

[0019] In a preferred embodiment, an interactive interface between the user and the digital twin model is provided in the virtual space, and a multimodal interaction algorithm is used to ensure that the user's operation can be accurately mapped to the physical entity, specifically:

[0020] Install and configure interactive equipment for collecting user input, including a voice recognition microphone, a gesture recognition camera, and a touch device;

[0021] Collect user input data through interactive devices, fuse the collected user input data, and generate the user's operation intention expression;

[0022] Generate instructions for controlling the digital twin model based on the operation intention expression, and map the instructions into control operations of the virtual model;

[0023] Transfer the control operations of the virtual model to the physical entity and synchronize the behavior of the physical entity with the user operations in real time.

[0024] In a preferred embodiment, the energy change of the abnormal propagation path is analyzed through a multi-scale dissipative dynamic network, and the possibility of semantic drift is divided into global or local, specifically:

[0025] A multi-scale dissipative dynamic network model is constructed based on the nodes and their connection relationships in the virtual-real fusion interaction. The nodes include sensor nodes, interactive device nodes, virtual model nodes and physical entity nodes.

[0026] Collect real-time energy data of each node, calculate the energy dissipation rate of each node, and identify key nodes where the energy dissipation rate changes significantly;

[0027] Construct anomaly propagation path diagrams based on the connection relationships between key nodes and other nodes to determine the scope and pattern of energy flow;

[0028] The impact area of ​​semantic drift is evaluated according to the scope and pattern of energy flow, and the possibility of semantic drift is divided into global or local.

[0029] In a preferred embodiment, the interaction between user operation and virtual model feedback is analyzed by the maximum entropy principle, and the degree of change of semantic entropy value caused by user operation is dynamically evaluated, specifically:

[0030] Collect user operation input data and virtual model feedback data in real time through interactive devices to form interactive data pairs;

[0031] Based on the maximum entropy principle, a joint probability distribution model of user operation input data and virtual model feedback data is established;

[0032] The interactive semantic entropy of user operations and virtual model feedback is calculated using a joint probability distribution model. Based on the analysis of the interactive semantic entropy, the degree of change in the semantic entropy caused by user operations is evaluated.

[0033] In a preferred embodiment, the interactive semantic entropy value of the user operation and the virtual model feedback is calculated using a joint probability distribution model. Based on the analysis of the interactive semantic entropy value, the degree of change of the semantic entropy value caused by the user operation is evaluated, specifically:

[0034] Using the joint probability distribution model, the interaction semantic entropy value between user operation and virtual model feedback is calculated through the normalization of probability and the law of feature distribution. The expression is:

[0035] ;

[0036] in, is the interactive semantic entropy value, represents the joint probability value of user operation input and virtual model feedback in the joint probability distribution, Represents a specific value in the user input data. Represents the complete set of user input data. Represents a specific value in the virtual model feedback data. Represents the full set of virtual model feedback data;

[0037] Obtain the interaction semantic entropy value at the current moment, obtain the interaction semantic entropy value at the previous moment, subtract the interaction semantic entropy value at the previous moment from the interaction semantic entropy value at the current moment, and obtain the semantic entropy value change value;

[0038] The larger the absolute value of the semantic entropy change, the greater the degree of change in the semantic entropy caused by the user operation.

[0039] In a preferred embodiment, the topological structure of the semantic graph of the digital twin model is analyzed by a graph matching deformation network to evaluate the topological deformation amplitude of the semantic graph, specifically:

[0040] Based on the semantic features of the digital twin model and the association between the features, a semantic graph of the digital twin model is constructed;

[0041] Extract the semantic graph of the current moment from the digital twin model, and call the reference semantic graph in the historical record as a comparison benchmark;

[0042] Use the graph matching deformation network to match the nodes and edges in the current semantic graph with the nodes and edges in the reference semantic graph to identify the structural differences between the two;

[0043] Based on the results of graph matching analysis, the degree of change of the semantic graph in terms of node connection relationships and overall topological structure is quantified.

[0044] In a preferred embodiment, the degree of change in the node connection relationship and the overall topological structure of the semantic graph is quantified as follows:

[0045] According to the difference results of graph matching, the node deformation rate and edge deformation rate are defined;

[0046] The node deformation rate is the ratio of the number of deformed nodes to the total number of nodes; the edge deformation rate is the ratio of the number of deformed edges to the total number of edges;

[0047] The node deformation rate and edge deformation rate are weighted summed to calculate the topological deformation amplitude index.

[0048] In a preferred embodiment, based on the degree of change in semantic entropy value caused by user operation and the topological deformation amplitude of the semantic graph, the potential impact of semantic drift in virtual-reality fusion interaction is evaluated, specifically:

[0049] The preset threshold corresponding to the absolute value of the semantic entropy value change is set, and the absolute value of the semantic entropy value change is compared with the corresponding preset threshold:

[0050] When the absolute value of the semantic entropy change value is greater than the corresponding preset threshold, it is determined that the semantic state change caused by the user operation is significant; otherwise, it is determined that the semantic state change caused by the user operation is normal;

[0051] Set the preset threshold corresponding to the topological deformation amplitude index, and compare the topological deformation amplitude index with its corresponding preset threshold:

[0052] When the topological deformation amplitude index is greater than the corresponding preset threshold, the structural deformation of the semantic graph is determined to be significant; otherwise, the structural deformation of the semantic graph is determined to be normal;

[0053] When the semantic state changes caused by user operations are significant and the structure of the semantic graph is significantly deformed, the potential impact of semantic drift in virtual-reality fusion interaction is judged to be highly significant;

[0054] When only one of the semantic state change caused by user operation and the structural deformation of the semantic graph is significant, the potential impact of semantic drift in virtual-reality fusion interaction is judged to be medium significant;

[0055] When the semantic state changes caused by user operations are normal and the structural deformation of the semantic graph is normal, the potential impact of semantic drift in virtual-reality fusion interaction is judged to be of low significance.

[0056] On the other hand, the present invention provides a virtual-reality fusion interaction system based on digital twins, including a twin model construction module, an interaction interface providing module, a semantic drift division module, an entropy value change evaluation module, a semantic topology analysis module, and a drift impact evaluation module;

[0057] Twin model construction module: Build a digital twin model of the physical entity in the virtual space to make the digital twin model consistent with the current state of the physical entity;

[0058] Interaction interface providing module: provides an interaction interface between users and digital twin models in virtual space, and uses multimodal interaction algorithms to ensure that user operations can be accurately mapped to physical entities;

[0059] Semantic drift classification module: analyzes the energy changes of abnormal propagation paths through a multi-scale dissipative dynamic network and divides the possibility of semantic drift into global or local;

[0060] Entropy change assessment module: When the possibility of semantic drift is local, the interaction between user operations and virtual model feedback is analyzed through the maximum entropy principle to dynamically assess the degree of semantic entropy change caused by user operations;

[0061] Semantic topology analysis module: When the possibility of semantic drift is local, the topological structure of the semantic graph of the digital twin model is analyzed through the graph matching deformation network to evaluate the topological deformation amplitude of the semantic graph;

[0062] Drift impact assessment module: Based on the degree of change in semantic entropy caused by user operations and the amplitude of topological deformation of the semantic graph, the potential impact of semantic drift in virtual-reality fusion interaction is assessed.

[0063] The technical effects and advantages of the virtual-reality fusion interaction method and system based on digital twins of the present invention are as follows:

[0064] 1. The present invention provides a virtual-reality fusion interaction method based on digital twins. By constructing a digital twin model consistent with the state of the physical entity in the virtual space, accurate mapping and dynamic synchronization between virtual and reality are achieved. The method uses a multimodal interaction algorithm to convert the user's operation into accurate control of the digital twin model, ensuring that the user's intention can be reflected in the behavior of the physical entity in real time, effectively improving the accuracy of the interaction and the response efficiency of the system. By introducing a multi-scale dissipative dynamic network, the present invention can comprehensively analyze the energy changes in abnormal propagation paths, divide the possibility of semantic drift into global and local, and significantly enhance the monitoring and classification capabilities of semantic states during virtual-reality interaction.

[0065] 2. When the possibility of semantic drift is local, the present invention further combines the maximum entropy principle and the graph matching deformation network to evaluate the significance of semantic changes from two dimensions: dynamic semantic state and semantic graph topological structure. By comprehensively evaluating the degree of change in semantic entropy values ​​caused by user operations and the amplitude of topological deformation of the semantic graph, the present invention can accurately quantify the potential impact of semantic drift in virtual-reality fusion interaction, and promptly discover and warn of semantic anomalies that may affect system stability. This method overcomes the shortcomings of the prior art that it is difficult to quantify and predict the impact of semantic drift, and provides technical support for improving the interactive reliability, stability and user experience of digital twin technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] Figure 1 A schematic diagram of a virtual-reality fusion interaction method based on digital twins of the present invention;

[0067] Figure 2 This is a structural schematic diagram of a virtual-reality fusion interaction system based on digital twins of the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] Embodiment 1, Figure 1 The present invention provides a virtual-reality fusion interaction method based on digital twins, which includes the following steps:

[0070] Build a digital twin model of the physical entity in the virtual space so that the digital twin model is consistent with the current state of the physical entity.

[0071] An interactive interface is provided between users and digital twin models in the virtual space, and a multimodal interaction algorithm is used to ensure that user operations can be accurately mapped to physical entities.

[0072] The energy changes of abnormal propagation paths are analyzed through a multi-scale dissipative dynamic network, and the possibility of semantic drift is divided into global or local.

[0073] When the possibility of semantic drift is local, the interactive relationship between user operations and virtual model feedback is analyzed by the maximum entropy principle, and the degree of change of semantic entropy value caused by user operations is dynamically evaluated; the topological structure of the semantic graph of the digital twin model is analyzed by the graph matching deformation network, and the topological deformation amplitude of the semantic graph is evaluated.

[0074] Based on the degree of change in semantic entropy caused by user operations and the amplitude of topological deformation of the semantic graph, the potential impact of semantic drift in virtual-reality fusion interaction is evaluated.

[0075] Construct a digital twin model of the physical entity in the virtual space so that the digital twin model is consistent with the current state of the physical entity. Specifically:

[0076] Collect real-time status data of physical entities, including position, posture and speed:

[0077] Various types of sensors are arranged at key locations of physical entities, including accelerometers, gyroscopes, GPS modules, etc., to collect physical parameters such as position, attitude and speed.

[0078] The sensor data is transmitted to the central data processing unit in real time through a communication module (such as Wi-Fi or Bluetooth). The data acquisition frequency can be adjusted according to the dynamic characteristics of the physical entity, such as 10 times per second or more.

[0079] Ensure that sensors cover important dynamic areas of the entire physical entity, such as collecting position information at the joints of a robotic arm and collecting posture data on the main body.

[0080] Preprocess the collected real-time status data:

[0081] Use filtering algorithms (such as low-pass filtering or Kalman filtering) to denoise the collected data and eliminate sensor errors or environmental interference.

[0082] Establish an outlier detection mechanism based on statistical models, such as using the triple standard deviation method to identify and eliminate outliers.

[0083] Standardize heterogeneous data collected by various sensors into a unified format to ensure that subsequent processing can be directly used, such as converting location information into a three-dimensional coordinate format.

[0084] A timestamp calibration algorithm is used to time align the collected multi-source data to ensure that the outputs of different sensors can be processed at the same time point.

[0085] According to the shape and size parameters of the physical entity, a virtual geometric model consistent with the physical entity is constructed in the virtual space using 3D modeling software:

[0086] The 3D shape and size parameters of physical entities are obtained through laser scanners, 3D cameras or existing design data (such as CAD files).

[0087] Use 3D modeling software (such as Autodesk Maya, Blender or Unity) to import the above parameters and generate a virtual geometric model of the physical entity.

[0088] Optimize the model according to actual needs, including:

[0089] Resolution Settings: Adjust the polygon count of the geometry to balance accuracy and computing resources.

[0090] Material definition: Assign material properties to the model so that the appearance characteristics of the physical entity can be intuitively represented in the virtual space.

[0091] The generated virtual model is compared with the original shape and size parameters to verify its geometric consistency and ensure that the model is structurally identical to the physical entity.

[0092] Map the real-time state data of the physical entity into the virtual geometric model:

[0093] Design a state mapping algorithm based on coordinate transformation to map the real-time state data of the physical entity (such as position, posture, speed) into the virtual geometric model. For example:

[0094] The position information is mapped to specific coordinates in three-dimensional space through Euler angle transformation, and the posture information updates the rotation state of the virtual model through quaternion transformation.

[0095] A high-frequency data refresh mechanism is used to ensure that the state update of the virtual geometric model reflects the changes of the physical entity in real time. For example, the data delay is controlled within 10 milliseconds.

[0096] During the state mapping process, the state consistency between the virtual model and the physical entity is monitored in real time. When data loss or abnormality occurs during the mapping process, the data retransmission mechanism is triggered.

[0097] According to the state parameters of the current physical entity, set the initial state of the digital twin model to make it consistent with the current state of the physical entity:

[0098] When the digital twin model is started, the initial state data of the physical entity is collected, including the initial position, posture and speed.

[0099] According to the collected initial state data, the initial position, rotation angle and velocity vector of the virtual model are set through the digital twin modeling platform.

[0100] The consistency between the initial state of the digital twin model and the state of the physical entity is verified, including whether the initial position matches the physical entity, whether there is any deviation in the posture angle, and whether the speed direction is consistent.

[0101] After completing the initial state setting, start the state synchronization mechanism between the virtual model and the physical entity to lay the foundation for subsequent real-time updates.

[0102] Provide an interactive interface between users and digital twin models in the virtual space, and use multimodal interaction algorithms to ensure that user operations can be accurately mapped to physical entities. Specifically:

[0103] Install and configure interactive devices for collecting user input, including voice recognition microphones, gesture recognition cameras, and touch devices:

[0104] According to the user's interaction needs, select interactive devices that support voice, gesture and touch data collection. The voice recognition device uses a microphone array, the gesture recognition uses a depth perception camera, and the touch device is a multi-touch panel.

[0105] When installing the device, make sure it covers the user interaction area. For example, install the gesture recognition camera at eye level, place the microphone array in a quiet area, and install the touch panel in an operable location.

[0106] Collect user input data through interactive devices, fuse the collected user input data, and generate the user's operation intention expression:

[0107] During the interaction process, the voice recognition device collects the user's voice input in real time and extracts the content of the voice command; the gesture recognition device captures the user's movements and gesture trajectories; and the touch device collects the user's touch points and trajectory data.

[0108] Preprocess the collected data, including denoising (such as removing background noise from speech and removing jitter tracks from gestures) and data time synchronization to ensure that speech, gesture, and touch data have unified timestamps.

[0109] Extract key features from each input data, such as extracting keywords and intonation from speech, extracting trajectory points and direction vectors from gesture data, and extracting click positions and gesture trajectories from touch data.

[0110] The speech signal is converted from the time domain to the frequency domain using the short-time Fourier transform (STFT) to extract the characteristic spectrum of the speech. The spectrum is then extracted using the convolutional neural network (CNN) to extract the speech feature vector.

[0111] A high-frame-rate depth camera is used to capture gesture trajectories, and a dynamic trajectory model is established based on the time series data of joint key points. A long short-term memory network (LSTM) is used to extract joint motion pattern features in the time series trajectory.

[0112] By using continuous sampling of touch points, the speed and direction characteristics of the sliding trajectory are extracted, the trajectory data is projected into two-dimensional space, and the user intention area is extracted through cluster analysis.

[0113] The dynamic time warping (DTW) algorithm is used to align the time series of multimodal data to ensure that the voice, gesture and touch data correspond to the same time point. The multi-head attention mechanism is used to weighted embed the features of different modalities to generate a unified cross-modal representation. The feature embedding vector of each modality participates in global weight learning to ensure the coordinated consistency between modalities.

[0114] Construct a multimodal feature graph with modal features as nodes and correlations between modalities as edges; use the graph convolutional network (GCN) to dynamically learn the interactive relationship between multimodal features and extract the fused feature vector.

[0115] The fused features output by GCN are input into the variational autoencoder (VAE) to generate an operation intention expression vector; the operation intention expression vector not only reflects the user's immediate operation, but also contains potential predictions of historical operation behaviors.

[0116] The fused intent vector is parsed using a Softmax-based classifier and labeled into predefined operation categories (such as move, scale, rotate, etc.). Based on the output operation categories, the intent description is further refined through a hierarchical encoding structure, for example, the moving direction, rotation axis, and scaling ratio.

[0117] Generate instructions to control the digital twin model based on the operation intention expression, and map the instructions to the control operations of the virtual model:

[0118] According to the operation intention expression vector, the user's target operation is analyzed, such as moving, rotating, scaling, or triggering a specific function.

[0119] Based on the analysis results, instructions for controlling the digital twin model are generated. The instructions include position adjustment instructions (such as adjusting the coordinates of the model), state control instructions (such as turning certain functions on or off), and view adjustment instructions (such as changing the viewing angle of the virtual model).

[0120] The generated instructions are mapped to the control operations of the virtual model through the digital twin modeling interface. For example, the model position can be adjusted or the model behavior can be triggered by calling the virtual model API.

[0121] Transfer the control operations of the virtual model to the physical entity and synchronize the behavior of the physical entity with the user operation in real time:

[0122] Use wireless communication modules (such as Wi-Fi, Bluetooth or industrial Ethernet) to build a communication channel between the virtual model and the physical entity, and convert the control operations generated by the virtual model into specific instructions executable by the physical entity, such as motion control instructions for a servo motor or path planning instructions for a robotic arm. After the physical entity executes the instructions, its status is fed back to the digital twin model in real time, such as reporting the actual position or status parameters. A real-time synchronization mechanism is used to map the execution results of the physical entity back to the virtual model, and the status of the virtual model is updated to keep it consistent with the physical entity.

[0123] The energy changes of abnormal propagation paths are analyzed through a multi-scale dissipative dynamic network, and the possibility of semantic drift is divided into global or local, specifically:

[0124] A multi-scale dissipative dynamic network model is constructed based on the nodes and their connection relationships in the virtual-real fusion interaction. The nodes include sensor nodes, interactive device nodes, virtual model nodes, and physical entity nodes:

[0125] Sensor node: A hardware device used to collect information about the state of a physical entity, such as a position sensor, velocity sensor, and attitude sensor.

[0126] Interactive device nodes: User input devices such as microphones for speech recognition, cameras for gesture recognition, and touch controls.

[0127] Virtual model node: A dynamic processing module in a digital twin model, such as a model update module or a virtual geometry module.

[0128] Physical entity nodes: These correspond to key components of a physical system, such as the servo motors or hydraulic system of a robotic arm.

[0129] The connection relationship between nodes is defined as the path for data flow or energy transfer, such as the real-time data transmission path from sensors to virtual models, or the control path from user operations to physical entities via interactive devices.

[0130] According to the definition and connection relationship of nodes, a multi-scale dissipative dynamic network model is established using graph representation, in which the weight of the node represents the energy level of the node and the weight of the edge represents the intensity of energy flow.

[0131] For example, the multi-scale dissipative dynamic network model can be represented as a directed weighted graph, which is expressed as:

[0132] ,

[0133] in, represents a multi-scale dissipative dynamic network model, Represents a collection of nodes. Represents the set of connections between nodes. represents the set of connection weights, Represents the energy set of a node.

[0134] Collect real-time energy data of each node, calculate the energy dissipation rate of each node, and identify key nodes where the energy dissipation rate changes significantly:

[0135] For sensor nodes, signal strength and update frequency are collected; for interactive device nodes, processing load and response delay are collected; for virtual model nodes, state change frequency and processing delay are collected; for physical entity nodes, power consumption and movement rate are collected.

[0136] The collected energy data is denoised, a low-pass filtering algorithm is used to eliminate sensor noise, and a time synchronization algorithm is used to align the timestamps of data from different nodes.

[0137] The energy dissipation rate is calculated by first recording the energy level of a node at the start time, then recording the energy level of the node again after a fixed time interval, and calculating the difference between the two energy values ​​to obtain the energy reduction. Then, the energy reduction is divided by the time interval to obtain the energy change per unit time, which is the energy dissipation rate.

[0138] The energy dissipation rate of the node is compared with its historical average dissipation rate, and the nodes with significant changes in dissipation rate are identified as key nodes. A significant change is defined as the absolute value of the deviation between the energy dissipation rate of the node and the historical average dissipation rate of the node being greater than the significance threshold.

[0139] Among them, the significance threshold is calculated by statistically analyzing the standard deviation of the node's historical data, and is used to determine whether the change in the node's energy dissipation rate exceeds the normal range.

[0140] Construct an abnormal propagation path diagram based on the connection relationship between key nodes and other nodes to determine the scope and mode of energy flow:

[0141] Starting from the key nodes, expand along the connection relationship between nodes to build an abnormal propagation path diagram.

[0142] The nodes in the anomaly propagation path graph represent the system units involved in anomaly propagation, and the weight of the edge represents the intensity of energy flow.

[0143] The number of nodes in the anomaly propagation path graph is divided by the total number of nodes in the multi-scale dissipative dynamic network to obtain the coverage ratio of all nodes in the anomaly propagation path graph.

[0144] If the coverage ratio of all nodes in the anomaly propagation path graph is greater than the corresponding preset threshold, the anomaly is determined to have a global impact.

[0145] The preset threshold corresponding to the coverage ratio of all nodes in the anomaly propagation path diagram is set based on historical data analysis. By counting the coverage of anomaly propagation under different semantic drift conditions, the minimum proportion of coverage of global impact is selected, which is generally set to 50% to distinguish between global and local impacts.

[0146] Calculate the directionality of energy flow in the anomaly propagation path diagram:

[0147] ,

[0148] in, Indicates the directionality of energy flow in the abnormal propagation path diagram, Representation Node arrive The energy flow weight, Represents the angle between the connection direction between nodes and the main path direction. represents all the edges in the anomaly propagation path graph, represents the key node, Indicates another specific node that is directly connected to the key node.

[0149] In order to determine whether the directionality of energy flow is significantly concentrated, it is first necessary to analyze the directions of all connected edges in the abnormal propagation path diagram, and define the direction angle of each edge as the angle between its direction and the main path direction. Then, the weighted average of all direction angles is calculated to represent the overall main path direction. Subsequently, the distribution variance of these direction angles is calculated to quantify the degree of discreteness of the direction. Among them, the weight of the distribution variance is determined by the energy flow intensity of each connecting edge. In order to determine the degree of concentration of the direction, a threshold of the direction distribution variance is set in advance. When the calculated distribution variance is lower than the threshold, the direction distribution can be considered to be concentrated. In addition, the total energy flow intensity on the main path needs to be counted and compared with the set minimum intensity standard to ensure that the energy flow intensity in the main path direction reaches a significant level. When the variance of the direction distribution is lower than the threshold and the energy flow intensity of the main path exceeds the minimum standard, the energy flow direction is judged to be significantly concentrated.

[0150] The impact area of ​​semantic drift is evaluated based on the scope and pattern of energy flow, and the possibility of semantic drift is classified as global or local:

[0151] When the coverage ratio of all nodes in the abnormal propagation path diagram is greater than the corresponding preset threshold and the directionality of energy flow is significantly concentrated, the semantic drift is judged to be global; otherwise, the semantic drift is judged to be local.

[0152] When semantic drift is determined to be global, the following measures can be taken:

[0153] Immediately trigger a global warning: notify all system modules and administrators, remind them of the potential impact of semantic drift on the entire system, and provide a real-time analysis report on the scope of impact, causes, and possible risks.

[0154] Suspend some system functions: Suspend high-risk interaction modules and operations, such as complex multimodal interaction operations or real-time synchronization functions, to reduce system complexity and prevent further spread of semantic drift.

[0155] Reinitialize the digital twin model: Use the reference semantic graph and historical data to fully rebuild the digital twin model, correct node attributes, edge relationships, and model status, and ensure that the virtual model is realigned with the physical entity.

[0156] Isolate abnormal paths and modules: In the multi-scale dissipative power network, isolate nodes and edges with significant energy dissipation to avoid abnormal propagation, and mark modules with large changes in semantic entropy values ​​and significant semantic graph deformation as high-risk and handle them separately.

[0157] When the possibility of semantic drift is local, the changes and impacts of the system's semantic state are usually limited to specific areas or modules, but this does not mean that the risk of drift is low or has no impact. Local semantic drift may be an early signal of potential global drift, so it is necessary to further analyze the degree of change in semantic entropy value caused by user operations and the magnitude of topological deformation of the semantic graph. The degree of change in semantic entropy value reflects the impact of user operations on the dynamic changes of interactive semantic state, and the magnitude of topological deformation measures the changes in the connection relationship and overall structure of semantic graph nodes. The combination of the two can more comprehensively evaluate the risk and scope of local drift from the two dimensions of dynamic interaction and structural change, provide a detailed basis for system adjustment, and avoid the spread of local drift to the global. Through this analysis, accurate drift monitoring and efficient response strategies can be achieved.

[0158] The interaction between user operations and virtual model feedback is analyzed through the maximum entropy principle, and the degree of change in semantic entropy caused by user operations is dynamically evaluated. Specifically:

[0159] The interactive device collects user operation input data and virtual model feedback data in real time to form interactive data pairs:

[0160] User operation input data includes the user's voice commands, gesture trajectory and touch position; virtual model feedback data includes real-time change information of the model state, such as rotation angle, scaling ratio and action completion status.

[0161] The collected user operation input data is paired with the virtual model feedback data at the corresponding time point to form an interaction data pair. Each data pair contains the corresponding relationship between user operation and virtual feedback, providing a data basis for subsequent probability distribution modeling.

[0162] Based on the maximum entropy principle, a joint probability distribution model of user operation input data and virtual model feedback data is established:

[0163] The collected user operation input data is used to extract features such as keywords, trajectory direction or position coordinates; the virtual model feedback is used to extract key attributes such as rotation state and scaling factor. These features are used to build a probability distribution model that describes the relationship between user operation and virtual feedback.

[0164] The joint probability distribution represents the possible relationship between user input data and virtual feedback data. This probability distribution is constructed through an optimization method that satisfies the maximum entropy principle and can maximize the uncertainty of the interaction state under the constraints of existing data. The goal is to accurately reflect the matching degree between user operations and virtual feedback through probability distribution.

[0165] The probability model needs to satisfy all observed data distribution constraints, for example, the sum of all joint probabilities should be equal to 1, and the model output should be consistent with the observed data under certain input conditions. These constraints are used to optimize the probability distribution model.

[0166] The interactive semantic entropy value of user operation and virtual model feedback is calculated using the joint probability distribution model. Based on the analysis of the interactive semantic entropy value, the degree of change of the semantic entropy value caused by the user operation is evaluated:

[0167] The joint probability distribution model is used to calculate the interactive semantic entropy value of user operations and virtual model feedback through the normalization of probability and the law of feature distribution.

[0168] The calculation formula of interactive semantic entropy is:

[0169] ;

[0170] in, is the interactive semantic entropy value, which represents the joint probability distribution The entropy value is used to measure the uncertainty between the user operation input data and the virtual model feedback data; Represents the joint probability value of user operation input and virtual model feedback in the joint probability distribution; Represents a specific value in the user's input data, such as a single input feature such as a user's voice command keyword, gesture direction, or touch position; Represents the full set of user operation input data, including all possible operation input features, for example, the set of all possible voice commands, gesture trajectory directions, and touch input positions; Represents a specific value in the feedback data of the virtual model, for example, a single feedback feature such as the rotation angle, scaling ratio, displacement path, etc. of the virtual model; Represents the full set of virtual model feedback data, including all possible feedback features, such as the set of all possible rotation angles, scaling ratios, and displacement paths.

[0171] The interaction semantic entropy value of the current moment is obtained, the interaction semantic entropy value of the previous moment is obtained, and the interaction semantic entropy value of the previous moment is subtracted from the interaction semantic entropy value of the current moment to obtain the semantic entropy value change value.

[0172] The greater the absolute value of the semantic entropy change, the greater the degree of semantic entropy change caused by user operations, indicating that the user operation has a more significant impact on the uncertainty of the system semantic state. When the semantic entropy change is positive, the system uncertainty increases, which may introduce noise or mismatched semantic information and increase the risk of semantic drift; when the change is negative, the system uncertainty decreases, which may make the semantics clearer, but over-concentration may cause the semantic expression to deviate from the user's intention. Therefore, the greater the absolute value of the semantic entropy change, whether positive or negative, indicates that the instability of the semantic state increases, which may be an early signal of potential semantic drift, and its impact on the interaction process needs to be further evaluated.

[0173] The topological structure of the semantic graph of the digital twin model is analyzed through the graph matching deformation network, and the topological deformation amplitude of the semantic graph is evaluated. Specifically:

[0174] Based on the semantic features of the digital twin model and the association between the features, the semantic graph of the digital twin model is constructed:

[0175] Semantic feature definition: The semantic features in the digital twin model include the properties, dynamic states and relationship information of physical entities, such as position, speed, direction and the relationship between modules.

[0176] Semantic graph node construction: Each semantic feature is a node in the semantic graph. For example, the position feature is a node, the speed feature is another node, and the node attributes include numerical value and unit, etc.

[0177] Definition of semantic graph edge: The relationship between semantic features is used as the edge in the semantic graph. For example, the relationship between speed and position is represented by an edge, and the attributes of the edge include the relationship type and weight value.

[0178] Semantic graph construction tool: Use graph database to build semantic graph and define structured relationships between nodes and edges. For example, use Neo4j database to store and manage semantic graph.

[0179] Extract the semantic graph at the current moment from the digital twin model, and call the reference semantic graph in the historical record as a comparison benchmark:

[0180] Extract the semantic features and their relationships at the current moment from the real-time data of the digital twin model, and dynamically generate the current semantic graph.

[0181] From the historical records stored in the graph database, a previously constructed reference semantic graph is called as a comparison benchmark. The reference semantic graph is usually a semantic graph in a stable state of the system.

[0182] Ensure that the node and edge data in the current semantic graph and the reference semantic graph are consistent in the time dimension, such as calibrating the timestamps of node attributes.

[0183] The graph matching deformation network is used to match the nodes and edges in the current semantic graph with the nodes and edges in the reference semantic graph to identify the structural differences between the two:

[0184] Use graph matching deformation network to match two semantic graphs. The matching algorithm is based on node attributes (such as position, state) and edge weights (such as association strength).

[0185] The nodes of the current semantic graph and the reference semantic graph are matched one by one to identify new nodes, missing nodes, and changes in node attributes. For example, when the speed value of a node changes or a new module is added, it is recorded as a node deformation.

[0186] Match the edges in the two semantic graphs and analyze changes in the connection relationship. For example, identify new feature relationships or significant changes in the weight values ​​of the original relationships.

[0187] The differences of the identified nodes and edges are marked on the current semantic graph to form a matching result with difference annotations for subsequent calculation of the deformation amplitude.

[0188] Based on the results of graph matching analysis, quantify the degree of change in the semantic graph in terms of node connectivity and overall topology:

[0189] The topological deformation amplitude of the semantic graph is defined as the comprehensive degree of change between the current semantic graph and the reference semantic graph at the node and edge levels.

[0190] According to the difference results of graph matching, the following deformation indicators are defined:

[0191] Node deformation rate: the ratio of deformed nodes to the total number of nodes; edge deformation rate: the ratio of deformed edges to the total number of edges.

[0192] The topological deformation amplitude index is quantified using the following formula:

[0193] ;

[0194] in, is the topological deformation amplitude index, is the node deformation rate, is the edge deformation rate, are the weight coefficients of node deformation rate and edge deformation rate, Both are greater than 0.

[0195] The weight coefficients of node deformation rate and edge deformation rate are set according to the application scenario of the semantic graph. For example, in scenarios that focus on changes in node characteristics, the weight coefficient of node deformation rate is set to a higher value; when emphasizing the importance of connection relationships, the weight coefficient of edge deformation rate is set to a higher value to ensure that the evaluation results meet system requirements.

[0196] The larger the topological deformation amplitude index is, the more significant the change in the node connection relationship and the overall topological structure of the semantic graph is, and the greater the topological deformation amplitude of the semantic graph is.

[0197] Based on the degree of change in semantic entropy caused by user operations and the topological deformation amplitude of the semantic graph, the potential impact of semantic drift in virtual-reality fusion interaction is evaluated, specifically:

[0198] The topological deformation amplitude of the semantic graph reflects the degree of change in the node connection relationship and overall structure of the digital twin model, while the absolute value of the semantic entropy change value measures the dynamic changes caused by user operations on the uncertainty of the semantic state during the interaction process. The two reveal the potential impact of semantic drift from the perspective of structural stability and interaction consistency respectively. By combining the quantitative indicators of the two, the significance of semantic drift in virtual-reality fusion interaction can be more comprehensively evaluated.

[0199] The preset threshold corresponding to the absolute value of the semantic entropy value change is set, and the absolute value of the semantic entropy value change is compared with the corresponding preset threshold:

[0200] When the absolute value of the semantic entropy change is greater than the corresponding preset threshold, it is determined that the semantic state change caused by the user operation is significant; otherwise, it is determined that the semantic state change caused by the user operation is normal.

[0201] Set the preset threshold corresponding to the topological deformation amplitude index, and compare the topological deformation amplitude index with its corresponding preset threshold:

[0202] When the topological deformation amplitude index is greater than the corresponding preset threshold, the structural deformation of the semantic graph is determined to be significant; otherwise, the structural deformation of the semantic graph is determined to be normal.

[0203] When the semantic state changes caused by user operations are significant and the structure of the semantic graph is significantly deformed, the potential impact of semantic drift in virtual-reality fusion interaction is judged to be highly significant. At this time, both user operations and the structure of the semantic graph have changed significantly, indicating that the impact of semantic drift is wide and the risk is high, and it needs to be handled as a priority.

[0204] When only one of the semantic state changes caused by user operations and the structural deformation of the semantic graph is significant, the potential impact of semantic drift in virtual-reality fusion interaction is judged to be of medium significance. At this time, only one indicator changes significantly, indicating that the impact of semantic drift may be limited to some areas and requires further monitoring and adjustment.

[0205] When the semantic state changes caused by user operations are normal and the structural deformation of the semantic graph is normal, the potential impact of semantic drift in virtual-reality fusion interaction is judged to be of low significance. At this time, both indicators are normal, indicating that the impact of semantic drift on the virtual-reality fusion interaction system is negligible and the system runs stably.

[0206] Among them, the preset threshold corresponding to the absolute value of the semantic entropy change value is set according to the statistical distribution of the semantic entropy value changes in historical data. The 95% quantile of the change value is usually selected as the significance threshold to distinguish normal fluctuations from significant fluctuations. The preset threshold corresponding to the topological deformation amplitude index is set according to the statistical distribution of the deformation amplitude of the semantic graph under normal operating conditions. The 90% quantile of the deformation amplitude is selected as the threshold to distinguish normal topological changes from significant topological deformations.

[0207] When the potential impact of semantic drift in virtual-reality fusion interaction is highly significant, the following measures can be taken:

[0208] Immediately trigger the dynamic early warning mechanism: notify the system administrator and related modules, and indicate the risk of significant semantic drift.

[0209] Suspend related interactive operations: Temporarily freeze user operations related to the affected modules to prevent further expansion of semantic drift.

[0210] Synchronously update the digital twin model: Update the nodes and edges with significant deformation in the semantic graph to make the model consistent with the current physical entity state.

[0211] Start the automatic recovery mechanism: call the historical stable reference semantic graph or adjust the semantic parameters to quickly restore the semantic consistency of the virtual model.

[0212] Record anomalies and optimize strategies: Detailed data of semantic drift is recorded as abnormal events to optimize the semantic entropy value change threshold or topology deformation threshold.

[0213] When the potential impact of semantic drift in virtual-reality fusion interaction is moderately significant, the following measures can be taken:

[0214] Monitor local interaction modules: Strengthen real-time monitoring of affected modules and track further changes in semantic drift.

[0215] Correct the affected single factor: Optimize the specific indicators with significant changes in semantic entropy or topological deformation. For example, adjust the interaction parameters or correct the semantic graph connection relationship.

[0216] Restrict some functional operations: restrict the use of high-risk interactive operations or semantic graph modules with significant deformation to reduce the risk of drift.

[0217] Timed feedback stability assessment: Set a time window to periodically evaluate semantic state changes to ensure that the system gradually returns to normal.

[0218] Delayed update: If the drift risk is still within a controllable range, you can choose to delay the update of the digital twin model and wait for further confirmation.

[0219] Example 2. The difference between Example 2 of the present invention and Example 1 is that this example introduces a virtual-reality fusion interaction system based on digital twins.

[0220] Figure 2 A structural schematic diagram of a virtual-reality fusion interaction system based on digital twins is given in the present invention. A virtual-reality fusion interaction system based on digital twins includes a twin model construction module, an interaction interface providing module, a semantic drift division module, an entropy value change evaluation module, a semantic topology analysis module and a drift impact evaluation module.

[0221] Twin model construction module: Build a digital twin model of the physical entity in the virtual space to make the digital twin model consistent with the current state of the physical entity.

[0222] Interaction interface providing module: provides an interaction interface between users and digital twin models in the virtual space, and uses multimodal interaction algorithms to ensure that user operations can be accurately mapped to physical entities.

[0223] Semantic drift classification module: The energy changes of abnormal propagation paths are analyzed through a multi-scale dissipative dynamic network, and the possibility of semantic drift is divided into global or local.

[0224] Entropy change assessment module: When the possibility of semantic drift is local, the interaction between user operations and virtual model feedback is analyzed through the maximum entropy principle to dynamically assess the degree of change in semantic entropy caused by user operations.

[0225] Semantic topology analysis module: When the possibility of semantic drift is local, the semantic graph topological structure of the digital twin model is analyzed through the graph matching deformation network to evaluate the topological deformation amplitude of the semantic graph.

[0226] Drift impact assessment module: Based on the degree of change in semantic entropy caused by user operations and the amplitude of topological deformation of the semantic graph, the potential impact of semantic drift in virtual-reality fusion interaction is assessed.

[0227] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0228] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination thereof. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or may be transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.

[0229] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0230] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0231] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0232] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, and may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0233] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0234] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0235] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0236] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A virtual-reality fusion interaction method based on digital twins, characterized in that: The steps include: Build a digital twin model of the physical entity in the virtual space so that the digital twin model is consistent with the current state of the physical entity; Provide an interactive interface between users and digital twin models in virtual space, and use multimodal interaction algorithms to ensure that user operations can be accurately mapped to physical entities; The energy changes of abnormal propagation paths are analyzed through a multi-scale dissipative dynamic network, and the possibility of semantic drift is divided into global or local. When the possibility of semantic drift is local, the interaction between user operations and virtual model feedback is analyzed by the maximum entropy principle, and the degree of change in semantic entropy caused by user operations is dynamically evaluated; the topological structure of the semantic graph of the digital twin model is analyzed by the graph matching deformation network, and the topological deformation amplitude of the semantic graph is evaluated; Based on the degree of change in semantic entropy caused by user operations and the magnitude of topological deformation of the semantic graph, the potential impact of semantic drift in virtual-reality fusion interaction is evaluated; The energy changes of abnormal propagation paths are analyzed through a multi-scale dissipative dynamic network, and the possibility of semantic drift is divided into global or local, specifically: A multi-scale dissipative dynamic network model is constructed based on the nodes and their connection relationships in the virtual-real fusion interaction. The nodes include sensor nodes, interactive device nodes, virtual model nodes and physical entity nodes. Collect real-time energy data of each node, calculate the energy dissipation rate of each node, and identify key nodes where the energy dissipation rate changes significantly; Construct anomaly propagation path diagrams based on the connection relationships between key nodes and other nodes to determine the scope and pattern of energy flow; The impact area of ​​semantic drift is evaluated according to the scope and pattern of energy flow, and the possibility of semantic drift is classified as global or local; The interactive semantic entropy value of user operation and virtual model feedback is calculated using the joint probability distribution model. Based on the analysis of the interactive semantic entropy value, the degree of change of the semantic entropy value caused by the user operation is evaluated. Specifically: Using the joint probability distribution model, the interaction semantic entropy value between user operation and virtual model feedback is calculated through the normalization of probability and the law of feature distribution. The expression is: ;in, is the interactive semantic entropy value, represents the joint probability value of user operation input and virtual model feedback in the joint probability distribution, Represents a specific value in the user input data. Represents the complete set of user input data. Represents a specific value in the virtual model feedback data. Represents the full set of virtual model feedback data; Obtain the interaction semantic entropy value at the current moment, obtain the interaction semantic entropy value at the previous moment, subtract the interaction semantic entropy value at the previous moment from the interaction semantic entropy value at the current moment, and obtain the semantic entropy value change value; The larger the absolute value of the semantic entropy change, the greater the degree of semantic entropy change caused by the user operation; The topological structure of the semantic graph of the digital twin model is analyzed through the graph matching deformation network, and the topological deformation amplitude of the semantic graph is evaluated. Specifically: Based on the semantic features of the digital twin model and the association between the features, a semantic graph of the digital twin model is constructed; Extract the semantic graph of the current moment from the digital twin model, and call the reference semantic graph in the historical record as a comparison benchmark; Use the graph matching deformation network to match the nodes and edges in the current semantic graph with the nodes and edges in the reference semantic graph to identify the structural differences between the two; Based on the results of graph matching analysis, quantify the degree of change in the semantic graph in terms of node connectivity and overall topological structure; Quantify the degree of change in the semantic graph in terms of node connectivity and overall topology, specifically: According to the difference results of graph matching, the node deformation rate and edge deformation rate are defined; The node deformation rate is the ratio of the number of deformed nodes to the total number of nodes; the edge deformation rate is the ratio of the number of deformed edges to the total number of edges; The node deformation rate and edge deformation rate are weighted summed to calculate the topological deformation amplitude index.

2. According to a method for virtual-reality fusion interaction based on digital twins according to claim 1, it is characterized in that: Construct a digital twin model of the physical entity in the virtual space so that the digital twin model is consistent with the current state of the physical entity. Specifically: Collect real-time status data of physical entities, including position, posture and speed; Preprocess the collected real-time status data; According to the shape and size parameters of the physical entity, a virtual geometric model consistent with the physical entity is constructed in the virtual space using 3D modeling software; Map the real-time status data of the physical entity into the virtual geometric model; Set the initial state of the digital twin model according to the state parameters of the current physical entity.

3. According to a method for virtual-reality fusion interaction based on digital twins according to claim 2, it is characterized in that: Provide an interactive interface between users and digital twin models in the virtual space, and use multimodal interaction algorithms to ensure that user operations can be accurately mapped to physical entities. Specifically: Install and configure interactive equipment for collecting user input, including a voice recognition microphone, a gesture recognition camera, and a touch device; Collect user input data through interactive devices, fuse the collected user input data, and generate the user's operation intention expression; Generate instructions for controlling the digital twin model based on the operation intention expression, and map the instructions into control operations of the virtual model; Transfer the control operations of the virtual model to the physical entity and synchronize the behavior of the physical entity with the user operations in real time.

4. According to the method of virtual-reality fusion interaction based on digital twins in claim 3, it is characterized in that: The interaction between user operations and virtual model feedback is analyzed through the maximum entropy principle, and the degree of change in semantic entropy caused by user operations is dynamically evaluated. Specifically: Collect user operation input data and virtual model feedback data in real time through interactive devices to form interactive data pairs; Based on the maximum entropy principle, a joint probability distribution model of user operation input data and virtual model feedback data is established; The interactive semantic entropy value of user operation and virtual model feedback is calculated using the joint probability distribution model. Based on the analysis of the interactive semantic entropy value, the degree of change of the semantic entropy value caused by user operation is evaluated.

5. According to the method of virtual-reality fusion interaction based on digital twins according to claim 4, it is characterized in that: Based on the degree of change in semantic entropy caused by user operations and the topological deformation amplitude of the semantic graph, the potential impact of semantic drift in virtual-reality fusion interaction is evaluated, specifically: The preset threshold corresponding to the absolute value of the semantic entropy value change is set, and the absolute value of the semantic entropy value change is compared with the corresponding preset threshold: When the absolute value of the semantic entropy change value is greater than the corresponding preset threshold, it is determined that the semantic state change caused by the user operation is significant; otherwise, it is determined that the semantic state change caused by the user operation is normal; Set the preset threshold corresponding to the topological deformation amplitude index, and compare the topological deformation amplitude index with its corresponding preset threshold: When the topological deformation amplitude index is greater than the corresponding preset threshold, it is determined that the structure deformation of the semantic graph is significant; Otherwise, the structural deformation of the semantic graph is judged to be normal; When the semantic state changes caused by user operations are significant and the structure of the semantic graph is significantly deformed, the potential impact of semantic drift in virtual-reality fusion interaction is judged to be highly significant; When only one of the semantic state change caused by user operation and the structural deformation of the semantic graph is significant, the potential impact of semantic drift in virtual-reality fusion interaction is judged to be medium significant; When the semantic state changes caused by user operations are normal and the structural deformation of the semantic graph is normal, the potential impact of semantic drift in virtual-reality fusion interaction is judged to be of low significance.

6. A virtual-reality fusion interaction system based on digital twins, used to implement a virtual-reality fusion interaction method based on digital twins as described in any one of claims 1 to 5, characterized in that: It includes twin model construction module, interactive interface provision module, semantic drift division module, entropy value change assessment module, semantic topology analysis module and drift impact assessment module; Twin model construction module: Build a digital twin model of the physical entity in the virtual space to make the digital twin model consistent with the current state of the physical entity; Interaction interface providing module: provides an interaction interface between users and digital twin models in virtual space, and uses multimodal interaction algorithms to ensure that user operations can be accurately mapped to physical entities; Semantic drift classification module: analyzes the energy changes of abnormal propagation paths through a multi-scale dissipative dynamic network and divides the possibility of semantic drift into global or local; Entropy change assessment module: When the possibility of semantic drift is local, the interaction between user operations and virtual model feedback is analyzed through the maximum entropy principle to dynamically assess the degree of semantic entropy change caused by user operations; Semantic topology analysis module: When the possibility of semantic drift is local, the topological structure of the semantic graph of the digital twin model is analyzed through the graph matching deformation network to evaluate the topological deformation amplitude of the semantic graph; Drift impact assessment module: Based on the degree of change in semantic entropy caused by user operations and the amplitude of topological deformation of the semantic graph, the potential impact of semantic drift in virtual-reality fusion interaction is assessed.

Images