Digital twin interaction system based on neural radiance fields and cognitive atlas

Abstract

The application is specifically a digital twin interaction system based on neural radiation field and cognitive graph, relates to the technical field of digital twin and human-computer interaction, and comprises the following modules: a data acquisition and key frame screening module, a neural radiation field reconstruction module, a cognitive knowledge graph construction and reasoning module, a posture estimation and intention prediction module, a physical simulation and interaction response module, a multi-agent collaborative decision-making module, and a collaborative computing scheduling module.In the application, multi-resolution hash grid coding is adopted to reduce the spatial query calculation complexity to a constant order of magnitude, compress the neural radiation field training time from hours to minutes, and support continuous rendering from any perspective, overcoming the problem of limited perspective in traditional explicit representation; a four-layer time sequence knowledge graph including a concept layer, an instance layer, a rule layer and a time sequence layer is constructed, a relation graph attention network and a neural symbol fusion reasoning framework are combined, and the system can perform multi-hop causal reasoning on scene entity relationships.

Description

Technical Field

[0001] This invention relates to the field of digital twin and human-computer interaction technology, and in particular to a digital twin interaction system based on neural radiation fields and cognitive maps. Background Technology

[0002] Digital twin technology creates virtual mappings for physical entities, enabling real-time monitoring, simulation analysis, and optimization decision-making of the physical world. It is widely applied in scenarios such as intelligent manufacturing, smart healthcare, and smart cities. Existing digital twin systems primarily rely on explicit representation methods such as pre-built CAD models or 3D scan data (e.g., triangular meshes, point clouds) for 3D scene representation; traditional knowledge graph technology for storing and retrieving static facts for semantic cognition; command-based input methods such as mouse and keyboard or simple voice commands for human-computer interaction; and centralized cloud processing or independent edge processing solutions for computing resources.

[0003] However, the aforementioned existing technologies also have corresponding drawbacks: Explicit 3D representations have high storage overhead and are difficult to update dynamically; pre-rendered lightmaps cannot respond to dynamic lighting changes, and the user's viewpoint is limited. Traditional knowledge graphs struggle to explicitly model time-series evolutionary knowledge and are difficult to integrate with multimodal perception data. Command-based interactions lack the ability to perceive the user's full-body posture in real time and predict potential intentions. Centralized cloud processing suffers from high network latency, while independent edge processing lacks sufficient computing power. The static task offloading strategies of existing collaborative solutions cannot adapt to dynamically changing network bandwidth and computing load, making it difficult for digital twin systems to simultaneously meet the comprehensive requirements of high-fidelity scene representation, deep semantic understanding, natural human-computer interaction, and real-time computing response.

[0004] Therefore, a digital twin interactive system based on neural radiation fields and cognitive maps is proposed to address the aforementioned problems. Summary of the Invention

[0005] The purpose of this invention is to propose a digital twin interactive system based on neural radiation fields and cognitive maps in order to solve the above-mentioned problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A digital twin interactive system based on neural radiation fields and cognitive maps includes: The data acquisition and keyframe filtering module is used to acquire multi-view RGB video streams and depth data through a distributed camera array, and to perform keyframe filtering on the original scene data sequence based on information gain to obtain a set of keyframe images. The neural radiation field reconstruction module is used to train a neural radiation field model based on multi-resolution hash grid coding and multi-scale feature fusion network, obtain the implicit three-dimensional field representation of the target scene, and generate a rendered image by differentiable volume rendering based on the implicit three-dimensional field representation. The cognitive knowledge graph construction and reasoning module is used to construct a temporal knowledge graph based on implicit three-dimensional field representation and domain document data, and to perform semantic reasoning using a relational graph attention network to generate scene semantic understanding results. The posture estimation and intent prediction module is used to fuse inertial measurement data and visual observation data to perform full-body posture estimation, and to predict intent based on action history sequence and task context to generate user interaction intent information. The physical simulation and interactive response module is used to perform physical simulation based on user interaction intent information using a position dynamics solver, update scene state and generate interactive response results. The multi-agent collaborative decision-making module is used to establish digital twin agents for each digital entity in the virtual environment, optimize joint policies through multi-agent reinforcement learning algorithms, and generate and execute collaborative behavior policies. The collaborative computing scheduling module is used to represent computing tasks as a computing graph and perform dynamic segmentation. It uses a deep Q network to dynamically adjust the allocation of computing tasks between the edge and the cloud based on real-time monitoring indicators, and performs adaptive bitrate control on the video stream transmitted to the user interaction terminal.

[0007] Preferably, the keyframe selection based on information gain includes: initializing a keyframe set and adding the first frame of the video stream to it; extracting feature representations for each subsequent frame using a pre-trained image feature extractor, and calculating information gain based on the Euclidean distance between the frame feature vector and the feature vectors of each frame in the selected keyframe set; comparing the information gain with a preset information gain threshold, and adding the current frame to the keyframe set only when the information gain exceeds the threshold; and stopping the selection when the number of frames in the keyframe set reaches a preset number.

[0008] Preferably, radiometric calibration and distortion correction are performed on each frame in the keyframe image set; radiometric calibration is performed by pre-shooting a standard color chart to calibrate the radiometric response coefficient of each camera, and mapping the images acquired by each camera to a unified radiometric space; distortion correction adopts a combined correction of radial distortion and tangential distortion to eliminate geometric deformation caused by lens optical characteristics.

[0009] Preferably, the acquisition of implicit three-dimensional field representation further includes: introducing a deformation field network, which takes spatial position and time parameters as input, outputs spatial displacement offset, maps the observed spatial coordinates to standard spatial coordinates, and performs density and color queries and rendering in the standard space, so that the implicit three-dimensional field representation can express dynamic scenes containing non-rigid moving objects; when the deformation gradient of a certain region is detected to exceed a preset deformation threshold, the region is automatically split into independent neural subfields, and each neural subfield performs density and color queries and rendering respectively, so as to support the representation of dynamic scenes with changes in topology.

[0010] Preferably, the construction of the temporal knowledge graph further includes: detecting the differences between new facts generated in newly collected data and the current temporal knowledge graph; recalculating node embedding representations only for the affected subgraph regions; and propagating the updates to entity nodes associated with the affected subgraphs through message passing of a graph convolutional network, thereby achieving incremental differential updates of the temporal knowledge graph; and generating node state predictions for future time steps based on a temporal graph neural network using the historical state sequence of each node as input, and outputting the trend of device state changes and the probability of abnormal events.

[0011] Preferably, the generation of user interaction intent information further includes: applying a physical information constraint loss term to the estimated full-body posture data, wherein the physical information constraint loss term is a weighted sum of a penalty term for joint angles deviating from the allowable range and a penalty term for joint torques deviating from the Jacobian matrix and the product of external forces, thereby suppressing posture estimation results that do not conform to human kinematics in a soft penalty manner; acquiring RGB images of the user's hand, using a hand keypoint detection network to output the coordinates of hand keypoints, and combining inverse kinematics solution to eliminate depth ambiguity caused by finger self-occlusion, thereby supplementing the full-body posture data with fine finger-level posture data.

[0012] Preferably, the physical simulation further includes: A learnable material constitutive model is introduced, which takes strain, strain rate, temperature and material parameters as inputs and outputs stress through a feedforward neural network to replace the traditional linear constitutive model for calculating material response. Continuous collision detection is performed based on the density field represented by the implicit three-dimensional field. The feedback force is calculated according to the contact depth and material stiffness. The feedback force is applied to the user's hand through a force feedback device, and the texture of the contact surface is simulated through high-frequency vibration.

[0013] Preferably, the establishment of the digital twin agent further includes: extracting policy priors from expert demonstration data through imitation learning to obtain an initial behavioral policy; fine-tuning the initial behavioral policy using a proximal policy optimization algorithm in a virtual simulation environment through reinforcement learning; the proximal policy optimization algorithm constrains the policy update magnitude by pruning the probability ratio of the new and old policies, with the expected value of the pruning probability ratio weighted by the advantage function estimate as the optimization objective; and after optimization, narrowing the gap between the simulation and the real world through domain randomization and residual learning.

[0014] Preferably, the generation computing scheduling strategy and transmission control parameters further include: encoding the user's historical position, velocity, and acceleration sequences based on a long short-term memory network to predict the user's future viewpoint position trajectory; pre-rendering and pre-loading the scene data of the corresponding viewpoint in the edge computing node cache according to the prediction results; the cache adopts a three-level content distribution structure of edge nodes, regional nodes, and central cloud; calculating the difference between the parameters of the old and new neural radiation field models; after sparsifying the difference parameters, only transmitting non-zero difference terms to each edge computing node; and each edge computing node applying patches to update the local model parameters.

[0015] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are: 1. This invention reduces the computational complexity of spatial queries to constant levels by employing multi-resolution hash grid encoding, compresses the training time of neural radiation fields from hours to minutes, and supports continuous rendering from any perspective, overcoming the limitation of traditional explicit representation perspectives; it constructs a four-layer temporal knowledge graph including a concept layer, an instance layer, a rule layer, and a temporal layer, and combines a relational graph attention network and a neural symbol fusion reasoning framework to enable the system to perform multi-hop causal reasoning on scene entity relationships.

[0016] 2. This invention employs an extended Kalman filter scheme that fuses an inertial measurement unit and a visual sensor for full-body pose estimation, and uses a Transformer encoder combined with task context for interactive intent prediction, thereby improving the accuracy of pose estimation and the proactivity of interactive response. It adopts a dynamic computation graph segmentation strategy based on a deep Q-network to dynamically adjust the task allocation between the edge and the cloud according to real-time network bandwidth and computational load. Combined with foveated rendering and scalable video coding, it reduces the end-to-end computational latency of the system and ensures the basic smoothness of the video stream under limited bandwidth. Attached Figure Description

[0017] Further details, features, and advantages of this application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 This is a system structure diagram of the present invention. Detailed Implementation

[0018] Several embodiments of this application will now be described in more detail with reference to the accompanying drawings to enable those skilled in the art to implement this application. This application may be embodied in many different forms and for various purposes and should not be limited to the embodiments set forth herein. These embodiments are provided to make this application thorough and complete, and to fully convey the scope of this application to those skilled in the art. The embodiments described do not limit this application.

[0019] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having a meaning consistent with their meaning in the relevant field and / or the context of this specification, and shall not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0020] Example 1 Its specific implementation method is combined with the appendix Figure 1 Please provide a detailed explanation.

[0021] In this embodiment, it includes: Step 1: Multi-source scene data acquisition and keyframe selection A distributed camera array deployed in the target scene is used to acquire multi-view RGB video streams and depth data of the target scene, obtaining the original scene data sequence. Keyframes are then selected from each frame in the original scene data sequence based on information entropy gain, resulting in a set of keyframe images.

[0022] The keyframe filtering process includes: Initialize keyframe set The first frame of the video stream Add to keyframe set .

[0023] For each subsequent frame image in the original scene data sequence The image of this frame is extracted using a pre-trained image feature extractor. Feature representation based on frame images in feature space With the selected keyframe set The distance to the frame image is calculated. Information gain ,in Indicates a set of known keyframes Conditional frame image The amount of new information provided.

[0024] Information gain Compared with the preset information gain threshold If a comparison is made, Then the frame image Add to keyframe set .

[0025] When keyframe set When the number of frames reaches the preset number, the filtering stops and a set of keyframe images is output. .

[0026] The above information gain threshold This is a constant value pre-set based on scene complexity. The pre-trained image feature extractor mentioned above can be a feature extraction network based on visual self-supervised learning. This feature extraction network takes a single frame image as input and outputs the feature vector corresponding to that frame image. The information gain is approximated by measuring the feature vector and the selected keyframe set. This is accomplished by using the Euclidean distance between the feature vectors of each frame.

[0027] To eliminate the impact of differences in radiometric response among multiple cameras and lens geometric distortion on the accuracy of subsequent reconstruction, the following steps are also included: For the set of keyframe images Each frame of the image was subjected to radiometric calibration and distortion correction. Radiometric calibration employed polynomial radiometric calibration, which involved pre-shooting a standard color chart to calibrate the radiometric response coefficients of each camera, mapping the images acquired by each camera to a unified radiometric space. Distortion correction employed a combined radial and tangential distortion correction method to eliminate geometric distortions caused by lens optical characteristics in each frame of the image, resulting in a calibrated set of keyframe images.

[0028] Step 2: 3D Scene Reconstruction of Neural Radiation Field Based on Hash Grid Encoding Based on a set of keyframe images, a neural radiation field model is trained using multi-resolution hash grid coding and a multi-scale feature fusion network to obtain the implicit three-dimensional field representation of the target scene.

[0029] This step involves the following sub-steps: Step 201: Divide the 3D space of the target scene into a multi-resolution mesh. At each resolution level, assign learnable feature vectors to each mesh vertex and store these feature vectors in a hash table. For any query point in the 3D space... Through query point The query point is obtained by performing trilinear interpolation on the adjacent vertices of the grid, aggregating feature vectors at multiple resolution levels. Spatial coding features .

[0030] Step 202: Encode spatial features Input a multi-scale feature fusion network, output query points Corresponding bulk density and dependent perspective direction RGB color values .

[0031] The multi-scale feature fusion network includes: Coarse networks use spatial encoding features As input, the output is a coarse density distribution, which is used for subsequent importance sampling; The fine network, based on the output of the coarse network, combines the hash-encoded query results to output refined color and density predictions; View-dependent subnetworks based on view direction As input, model the reflection characteristics of the scene.

[0032] The coarse network is a feedforward neural network with four fully connected layers, the fine network is a feedforward neural network with two fully connected layers and additional hash-encoded feature queries, and the view-dependent sub-network is a feedforward neural network with two fully connected layers.

[0033] Step 203: Using the known pixel colors of each frame in the keyframe image set as supervision signals, the feature vectors in the hash table and the parameters of the multi-scale feature fusion network are jointly optimized end-to-end by minimizing the reconstruction error between the rendered image and the real image. The Adam optimization algorithm is adopted, and the loss function is the mean square error between the rendered color and the real color of all training images. ; in, For the index of training images, For the first The true pixel colors of the image For the first The rendered pixel colors corresponding to each image. After training, a neural radiation field model is obtained, which is the implicit 3D field representation of the target scene.

[0034] In the aforementioned multi-resolution hash grid encoding, the hash table is a fixed-size lookup table. When different grid vertices are mapped to the same hash table position, a hash collision occurs, which is automatically resolved during the network training process. This encoding method makes the computational complexity of spatial lookup constant, not increasing with the resolution, and significantly accelerates the training process compared to positional encoding methods.

[0035] To enable the neural radiation field to handle dynamic objects with non-rigid motion in the scene, the following processing is added to step 202: introducing a deformation field network. Deformation field network Based on spatial location and time parameters As input, output spatial displacement offset , will be the moment The observation points are mapped to standard space: ; in, Standard spatial coordinates, For a moment The observed spatial coordinates are below. This is a deformation field network. Hash grid encoding is also used to accelerate queries. Density and color queries and rendering are performed in a standard space, enabling the static implicit 3D field representation to express dynamic scenes.

[0036] To handle potential topological changes in the scene (such as object separation), the following processing is also included in addition to the above deformation field: introducing a neurovoxel splitting process. When the deformation gradient of a certain region exceeds a preset deformation threshold, the region is automatically split into independent neurovoxels. Each neurovoxel is used to query and render density and color separately, thereby supporting the representation of dynamic scenes with changing topology.

[0037] Step 3: Differentiable Volume Rendering and Scene Visualization Based on implicit 3D field representation, light rays are emitted along the user-specified viewpoint, and a layered volumetric rendering strategy is used to perform color integration on the sampling points on the light rays to generate a rendered image from that viewpoint.

[0038] This step involves the following sub-steps: Step 301: Determine the ray equations corresponding to each pixel in the rendered image based on the user-specified viewpoint and position. ,in As the starting point of the light, For the direction of light, These are the parameters for light rays.

[0039] Step 302: Perform coarse sampling on each ray, and sample uniformly along the ray. Each sampling point is input into the coarse network of the trained neural radiation field model to obtain a rough density distribution of each sampling point.

[0040] Step 303: Based on the coarse density distribution, use inverse transformation sampling to concentrate sampling in areas with higher density. Each sampling point is used to input its spatial coordinates and viewpoint direction into a fine network and a viewpoint-dependent sub-network to obtain the fine color value and volume density of each sampling point.

[0041] Step 304: [Regarding...] The volume rendering integral is performed at each sampling point along the direction of the light, the final color value of each pixel is calculated, and the color values ​​of all pixels are combined to generate the rendered image from that viewpoint.

[0042] To eliminate the impact of camera extrinsic parameter calibration errors on reconstruction accuracy during deployment, the following processing is included in step 203 when training the neural radiation field model: the camera pose parameters corresponding to each keyframe image are incorporated into the optimization variables, and the neural radiation field model parameters and camera pose parameters are jointly optimized during training. Lie algebras are used for the camera pose parameters. Parameterization is performed to preserve the manifold constraints of the pose transformation during optimization. The optimization objective is to minimize the mean square error between the rendered colors and the true colors of all training images. ; in, These are parameters for the neural radiation field model. For the first Camera pose of the image For the first The true pixel colors of the image For the first The color of the rendered pixels of an image.

[0043] To support dynamic lighting applications (such as simulating different lighting conditions in a digital twin environment), the following processing is included in addition to step 202: decomposing the color output into albedo components and shading components: ; in, Albedo is a view-independent reflectance that characterizes the color properties of an object's surface itself. For dependent perspective direction and the direction of the light source The shading terms. By introducing spherical harmonic functions as lighting priors, material properties and lighting effects are simultaneously separated during training, allowing the rendering stage to adjust the light source direction. Change the scene lighting.

[0044] Step 4: Cognitive Knowledge Graph Construction and Semantic Reasoning Based on implicit 3D field representation and domain document data, multi-source semantic extraction and temporal knowledge graph construction are performed, and semantic reasoning is carried out using graph neural networks to generate scene semantic understanding results.

[0045] This step involves the following sub-steps: Step 401: Extract information from domain document data (including unstructured documents such as equipment manuals, operating specifications, and historical maintenance records) using a multimodal document understanding model to obtain entity, relationship, and event information. The multimodal document understanding model comprises three sub-encoders: a text encoder, a layout encoder, and an image encoder. The text encoder takes the document's text content as input to extract semantic features; the layout encoder takes the document's two-dimensional positional information as input to extract layout structure features; and the image encoder takes charts and diagrams from the document as input to extract visual features. The output features of the three sub-encoders are fused to output entity names, relationships between entities, and event descriptions from the document.

[0046] Step 402: Embed semantic labels into the implicit 3D field representation. Extend the semantic output branch based on the neural radiation field model, using spatial location coordinates... As input, output the semantic category probability at that location. The semantic output branch is trained under supervision using two-dimensional semantic segmentation labels. Training employs the Adam optimization algorithm, and the loss function is cross-entropy loss. ; in, , For the two-dimensional coordinate index of the pixel, For semantic category indexing, For pixels Corresponding category The true label, Pixels obtained through volume rendering integration Corresponding category The semantic category prediction probability is calculated. After training, the semantic category of any spatial location in the implicit 3D field can be queried, achieving alignment between 3D geometric information and semantic information.

[0047] The aforementioned two-dimensional semantic segmentation labels can be automatically generated by a pre-trained image segmentation model or obtained through manual annotation. The generated labels are the semantic categories to which each pixel in each keyframe image belongs.

[0048] Step 403: Entity, relation, and event information extracted from the domain document, along with spatial entity and semantic category information obtained from the implicit 3D field, are fused to construct a temporal knowledge graph. The basic unit of the temporal knowledge graph is a quadruple. ,in For the head entity, For the relationship, For tail entities, This refers to a timestamp or time interval. The temporal knowledge graph adopts a four-layer architecture: the concept layer defines domain classes and attributes; the instance layer stores specific entities; the rule layer stores domain reasoning rules; and the temporal layer records time constraints.

[0049] Step 404: Based on the constructed temporal knowledge graph, semantic reasoning is performed using a relational graph attention network to generate scene semantic understanding results. The relational graph attention network takes the initial feature vectors of each node in the temporal knowledge graph as input, performs feature propagation and aggregation on each node, and outputs the semantic reasoning results for the query node (e.g., for a specific device malfunction node, inferring its possible fault cause chain). The node feature update method is as follows: ; in, The index of the current node. For the index of neighboring nodes, For network layer indexing, For nodes In the The feature vector of the layer, For nodes The set of neighboring nodes, For a set of relation types, For a set of relation types Relational type index in the middle, For relationship The corresponding learnable weight matrix, These are attention coefficients calculated using a relationship-specific attention mechanism. The activation function. Attention coefficients. Calculated in the following way: ; in, For the index of neighboring nodes, For a set of relation types Relational type index in the middle, For relationship The corresponding attention parameter vector, Indicates transpose. This represents vector concatenation. This represents an exponential function. The relational graph attention network, through multi-layer message passing, can perform reasoning along multi-hop paths in a temporal knowledge graph. The relational graph attention network employs supervised training, using known entity relationship triples as positive samples and randomly generated triples (by replacing the head or tail entity) as negative samples. The loss function is cross-entropy loss, optimized using the Adam optimization algorithm.

[0050] To simultaneously leverage the fuzzy matching capabilities of neural networks and the precise reasoning capabilities of symbolic rules, the following processing is included in addition to step 404: A neural-symbolic fusion reasoning framework is introduced. In this framework, the output of the relational graph attention network serves as the soft constraint input to the symbolic reasoning engine. The symbolic reasoning engine performs precise reasoning based on the domain rules stored in the rule layer, generating a reasoning conclusion. The reasoning conclusion of the symbolic reasoning engine is fed back to the relational graph attention network to correct its feature predictions for relevant nodes. Through alternating iterations between the neural and symbolic parts, the results of fuzzy and precise reasoning are fused to generate a more interpretable semantic understanding result.

[0051] To ensure that the temporal knowledge graph can continuously reflect changes in the physical world without requiring a full reconstruction, the following steps are also included: incremental differential updates to the temporal knowledge graph. Specifically, this process involves: detecting differences between new facts generated in newly acquired data and the current temporal knowledge graph, including added, deleted, and modified facts; recalculating node embedding representations only for the affected subgraph regions; and propagating the updates to entity nodes associated with the affected subgraphs via message passing through a graph convolutional network to obtain the updated temporal knowledge graph.

[0052] To predict the future evolution trend of a temporal knowledge graph (such as equipment failure trends), the following steps are also included: Evolution prediction of the temporal knowledge graph based on a temporal graph neural network. The temporal graph neural network takes the historical state sequence of each node as input, generates node state predictions for future time steps through an autoregressive approach, and outputs predictive semantic understanding results such as equipment state change trends and the probability of abnormal events. The temporal update method for node features is as follows: ; in, The index of the current node. For the index of neighboring nodes, These are the time parameters of a time-series graph neural network. For the edge In time The message being delivered For nodes In time Previous historical status, For message aggregation functions, message aggregation functions The specific form is: to and After concatenation, the message vector is passed through a fully connected layer and an activation function to output the aggregated message vector. Then, the message vectors of the neighboring nodes are summed to obtain the node. In time Update features The temporal graph neural network is trained in a supervised manner on the node state sequence of known historical time steps, and the real node state of future time steps is used as the prediction target. The loss function is the mean squared error, and the Adam optimization algorithm is used for optimization.

[0053] Step 5: User full-body pose estimation and interaction intent prediction Collect user limb movement data, integrate inertial measurement data and visual observation data to estimate whole-body posture, combine user action history and task context to predict intent, and generate user interaction intent information.

[0054] This step involves the following sub-steps: Step 501: Acquire high-frequency acceleration and angular velocity data from inertial measurement units worn on the user's torso and limbs, and simultaneously acquire visual observation data from RGB-D cameras deployed in the scene.

[0055] Step 502: Fuse inertial measurement data and visual observation data using an extended Kalman filter to estimate the user's overall pose. State vector. ,in Represents transpose, state vector Includes attitude quaternions angular velocity ,Location and linear velocity The prediction step predicts the state based on the dynamic equations of the inertial measurement unit, and the update step corrects the prediction results using the absolute position reference provided by visual observation, thereby obtaining the user's full-body posture data at the current moment.

[0056] Step 503: Combine the user's full-body posture data (including joint angles, hand positions, gaze direction, and other posture features) from multiple consecutive moments into a motion history sequence. ,in The length of the action history sequence is used to define the action history sequence. The input is encoded using a Transformer encoder to obtain a contextual feature representation of the action history. The Transformer encoder takes the pose feature vectors at each time step in the action history sequence as the input token sequence, captures the temporal dependencies in the action sequence through a multi-head self-attention mechanism, and outputs a context feature representation that incorporates temporal context information. .

[0057] Step 504: Represent the context features By fusing information with the current task context (including current workflow state, tool state, and environmental constraints), the multi-task learning head outputs the probability distribution of the next operation. The system predicts operational parameters (such as grasping position and force direction) and the remaining time of the current task, and aggregates these predictions to generate user interaction intent information. The multi-task learning head contains three parallel output branches: a classification branch, a fully connected layer followed by Softmax activation, outputs the probability distribution of the next operation category; a regression branch, also a fully connected layer, outputs continuous value predictions of operational parameters; and a duration prediction branch, also a fully connected layer, outputs an estimate of the remaining task time. All three branches share the contextual feature representation output by the Transformer encoder. The total loss function for joint training is a weighted sum of the cross-entropy loss of the classification branch, the mean squared error loss of the regression branch, and the mean squared error loss of the duration prediction branch, and is optimized using the Adam optimization algorithm.

[0058] To ensure that the posture estimation results conform to the laws of human biomechanics, the following processing is included in addition to step 502: A physical information constraint loss term is introduced to impose joint angle and joint moment constraints on the estimated whole-body posture data. The physical information constraint loss is: ; in, Indicates transpose. This is an estimated value for the joint angle. These are the boundary values ​​of the allowable range of joint angles. For joint torque, For Jacobian matrices, As an external force, and , where is the weighting coefficient. This physical information constrains the loss term, using a soft penalty to suppress posture estimation results that do not conform to human kinematics.

[0059] For application scenarios requiring fine hand manipulation, in addition to step 502, the following processing is also included: acquiring RGB images of the user's hand, using a hand keypoint detection network to output the coordinates of 21 key points of the hand (including 4 joints of each finger and the wrist point), and combining inverse kinematics to eliminate depth ambiguity caused by finger self-occlusion, obtaining fine finger-level pose data, and supplementing the fine finger-level pose data into the whole-body pose data.

[0060] The hand keypoint detection network takes the RGB image of the user's hand as input, extracts image features through a convolutional backbone network, and outputs the two-dimensional coordinates of 21 keypoints of the hand through a fully connected output layer. It is trained in a supervised manner using mean squared error loss, with the labeled keypoint coordinates as the supervision target, and is optimized using the Adam optimization algorithm.

[0061] To proactively provide assistance when user hesitation or risk is detected, the process, building upon step 504, includes the following: based on the operation probability distribution and task context in the user interaction intent information, determining whether there is a user hesitation state (e.g., pause time exceeding a preset threshold, or repeated searching of the gaze between multiple targets) or an operation risk state (e.g., the predicted next operation is inconsistent with the correct operation sequence). When the above states are detected, proactive interaction suggestion information is generated, including: visual cues highlighting the target object and operation path in the rendered image, voice guidance describing the next operation, and force guidance information applying guiding force through a haptic feedback device.

[0062] Step 6: Environmental physical simulation and digital twin interactive response generation Based on user interaction intent information, implicit 3D field representation, and scene semantic understanding results, a physical simulation engine is used to simulate physical processes in a virtual environment, generate interactive response results, and update the scene state in the implicit 3D field representation.

[0063] This step involves the following sub-steps: Step 601: Parse the next operation prediction and operation parameter prediction in the user interaction intent information into operation instructions in the virtual environment. The operation instructions include operation type (such as grab, push, cut) and operation parameters (such as position of action, direction of action, magnitude of force).

[0064] Step 602: Based on the operation commands, perform physical simulation of objects affected by the operation in the virtual environment using the position dynamics solver. The processing procedure of the position dynamics solver is as follows: First, execute the prediction step to calculate the predicted position based on the current object position, velocity, and external forces; then execute the constraint projection step to iteratively solve the constraint conditions. And correct the predicted position to a position that satisfies the constraints, where The position of the object in the simulation space is determined by the object type and operation type. For rigid objects, the constraint is to keep the distance between all particles within the object constant. For deformable objects, the constraint is to limit the change in distance between adjacent particles to within the elastic range allowed by the material. Finally, a velocity update step is executed to calculate the updated velocity based on the position difference before and after correction.

[0065] Step 603: Update the physical simulation object state (including position, deformation, velocity, etc.) to the scene state of the virtual environment, and based on the updated scene state, use the differentiable volume rendering in step 3 to generate a new rendered image, and send the new rendered image to the user interaction terminal for display as an interactive response result.

[0066] To improve the realism of material responses in physical simulations, the following processing is included in step 602: a learnable material constitutive model is introduced to replace the traditional fixed-parameter constitutive model. The learnable material constitutive model is a feedforward neural network. In order to respond strain rate ,temperature and material parameters As input, output stress : ; in, In response, For strain rate, For temperature, Material parameters. Feedforward neural network. Pre-trained using real material test data, with stress values ​​from the real test data as the supervision target, and mean square error as the loss function, the Adam optimization algorithm is used for optimization. It can fit the nonlinear stress-strain relationship of different materials such as metals, rubber, and biological tissues, and replace the traditional linear constitutive model to calculate the material response in the physical simulation process.

[0067] To support complex physical phenomena such as anisotropic materials, fracture and cutting, and thermo-mechanical coupling in physical simulations, the following extended processing is included in addition to the position dynamics solver in step 602: the constraint stiffness matrix is ​​used to distinguish tensile, compressive, and shear responses in different directions to simulate the mechanical behavior of anisotropic materials; the constraint topology is dynamically updated during the simulation (by deleting constraint connections at fracture points) to simulate the fracture and cutting process of an object; and the influence of the temperature field on the material stiffness parameters is introduced to simulate thermo-mechanical coupling effects.

[0068] To provide tactile feedback to the user, in addition to step 603, the following processing is also included: continuous collision detection based on the density field represented by an implicit three-dimensional field; calculation of the feedback force according to the contact depth and material stiffness; application of the feedback force to the user's hand through a force feedback device; and simulation of the texture sensation of the contact surface through high-frequency vibration. The collision detection is based on the continuity of the density field, avoiding the object penetration phenomenon that may occur in traditional mesh collision detection.

[0069] Step 7: Multi-agent collaborative decision-making and digital twin behavior generation Based on the semantic understanding results of the scene and the user interaction intent information, multiple digital twins in the virtual environment are driven to make autonomous decisions and generate collaborative behaviors, and output the behavior strategies of the digital twins.

[0070] This step involves the following sub-steps: Step 701: Establish digital twin agents for each digital entity requiring autonomous behavior in the virtual environment. Each digital twin agent comprises three components: a perception component, a cognition component, and an execution component. The perception component obtains current scene state information from implicit 3D field representation and temporal knowledge graph; the cognition component is a domain-adapted large language model that uses scene state information and task descriptions in natural language as input to perform reasoning and output high-level decision commands; the execution component is an end-to-end visual-language-action model that transforms high-level decision commands into low-level action sequences.

[0071] The network architecture of the visual-language-action end-to-end model includes the following components: a visual encoder, which takes the current visually observed image as input and outputs visual feature representations; a language encoder, which takes the high-level decision-making instruction text as input and outputs semantic feature representations; a cross-modal fusion decoder, which adopts a Transformer decoder structure and takes visual and semantic features as input to generate a sequence of action tokens; and an action output layer, a fully connected layer, which converts the sequence of action tokens into continuous action parameter outputs (such as joint torque and end effector pose). The visual-language-action end-to-end model is pre-trained on expert demonstration data using imitation learning, with expert action sequences as the supervision target. The loss function is the mean squared error between the predicted action parameter values ​​and the expert action parameters, and it is optimized using the Adam optimization algorithm.

[0072] In the cognitive component, the domain-adapted large language model is fine-tuned using a low-rank adaptation method. Specifically, the pre-trained weights... Keep frozen, and introduce two low-rank trainable matrices. and This makes the actual weights as Only low-rank trainable matrices need to be trained. and With just a few parameters, a large language model can master the terminology and reasoning patterns of a specific domain.

[0073] Step 702: The cognitive components of each digital twin agent generate their respective behavioral intentions based on the current scene state information and task objectives. For multiple digital twin agents with cooperative or conflicting relationships, joint policy optimization is performed using a multi-agent reinforcement learning algorithm. The joint state space, action space, and reward function of each digital twin agent are determined according to the specific scene task. After solving, the collaborative behavior strategy of each digital twin agent is output.

[0074] Step 703: The execution components of each digital twin agent generate specific action sequences according to the collaborative behavior strategy, input the action sequences into the physical simulation engine in step 6 for execution, drive the corresponding digital twin in the virtual environment to perform actions, and update the scene state.

[0075] For scenarios requiring simulation of large-scale crowd movement, the following processing is added to step 702: A social force model is used to drive the crowd agents. Each pedestrian agent is subject to a driving force pointing towards a target, a repulsive force to avoid collisions with others and obstacles, and an attractive force to attract companions and points of interest. The synthesis of these forces drives the motion updates of the pedestrian agents. The parameters of the social force model are calibrated using real-world crowd movement video data.

[0076] To enable a digital twin to learn from expert demonstrations and continuously optimize its policies, the following steps are also included: First, policy priors are extracted from expert demonstration data through imitation learning to obtain an initial behavioral policy; then, in a virtual simulation environment, the initial behavioral policy is fine-tuned using a proximal policy optimization algorithm, with the optimization objective being: ; in, The policy parameters for the near-end policy optimization algorithm, For the time step index of the near-end policy optimization algorithm, The probability ratio between the old and new strategies. For the current policy in state Select action The probability, For the old strategy in state Select action The probability, This is the estimated value of the dominance function. The pruning range parameters are used. Through large-scale parallel training in a simulation environment, the optimized policy, when deployed to a real environment, narrows the gap between simulation and reality through domain randomization (randomizing physical parameters and visual appearance in the simulation) and residual learning (learning residual corrections between the simulation and the real world).

[0077] Step 8: Edge and Cloud Collaborative Computing Scheduling Based on the current network status and computing load information, the computing tasks in each of the above steps are dynamically segmented and scheduled, and the computing tasks are allocated to edge computing nodes and cloud servers for execution respectively. Adaptive bitrate control is performed on the video stream transmitted to the user interaction terminal to generate computing scheduling strategies and transmission control parameters that meet real-time constraints.

[0078] This step involves the following sub-steps: Step 801: Represent the deep learning models involved in the above steps as computational graphs. , where nodes To compute the operator, edge This represents tensor dependencies. Graph partitioning algorithms are used to compute graph dependencies. Find the set of split points in , calculate graph Divide into subgraphs suitable for execution at edge computing nodes and subgraphs suitable for execution on cloud servers The optimization objective is: ; The constraints are ,in For end-to-end computation delay, For edge power consumption, This represents the amount of data transmitted at the split point. This represents the maximum amount of memory available at the edge. , , These are the weighting coefficients.

[0079] Step 802: Monitor the computing power utilization, network transmission latency, and task queue length of edge computing nodes in real time. Use these monitored metrics as state inputs and dynamically adjust the split point positions using a deep Q-network. The deep Q-network uses metrics such as computing power utilization, network latency, and queue length as state inputs, a discrete split layer selection as its action space, and negative end-to-end latency as a reward. It outputs the Q-value of each available split point in the current state and selects the split point with the largest Q-value as the current scheduling decision. The deep Q-network is continuously updated online and stabilized through experience replay and target network methods.

[0080] Step 803: Perform adaptive bitrate control on the video stream output from the rendering engine to the user interaction terminal. This is based on an estimate of the current network bandwidth. Receiver buffer length The state is determined by the user's gaze point position, and the resolution is the same as the video stream resolution. Frame rate and quantization parameters For the action, a reinforcement learning optimizer is used to determine the combination of encoding parameters at the current moment and generate transmission control parameters.

[0081] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0082] In this document, the use of relational terms such as "first" and "second" is merely for distinguishing one entity or operation from another, and does not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0083] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0084] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0085] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A digital twin interactive system based on neural radiation fields and cognitive maps, characterized in that, include: The data acquisition and keyframe filtering module is used to acquire multi-view RGB video streams and depth data through a distributed camera array, and to perform keyframe filtering on the original scene data sequence based on information gain to obtain a set of keyframe images. The neural radiation field reconstruction module is used to train a neural radiation field model based on multi-resolution hash grid coding and multi-scale feature fusion network, obtain the implicit three-dimensional field representation of the target scene, and generate a rendered image by differentiable volume rendering based on the implicit three-dimensional field representation. The cognitive knowledge graph construction and reasoning module is used to construct a temporal knowledge graph based on implicit three-dimensional field representation and domain document data, and to perform semantic reasoning using a relation graph attention network to generate scene semantic understanding results. The posture estimation and intent prediction module is used to fuse inertial measurement data and visual observation data to perform full-body posture estimation, and to predict intent based on action history sequence and task context to generate user interaction intent information. The physical simulation and interactive response module is used to perform physical simulation based on user interaction intent information using a position dynamics solver, update scene state and generate interactive response results. The multi-agent collaborative decision-making module is used to establish digital twin agents for each digital entity in the virtual environment, optimize joint policies through multi-agent reinforcement learning algorithms, and generate and execute collaborative behavior policies. The collaborative computing scheduling module is used to represent computing tasks as a computing graph and perform dynamic segmentation. It uses a deep Q network to dynamically adjust the allocation of computing tasks between the edge and the cloud based on real-time monitoring indicators, and performs adaptive bitrate control on the video stream transmitted to the user interaction terminal.

2. The digital twin interactive system based on neural radiation fields and cognitive maps according to claim 1, characterized in that, Keyframe selection based on information gain includes: initializing the keyframe set and adding the first frame of the video stream to it; extracting feature representations for each subsequent frame using a pre-trained image feature extractor, and calculating information gain based on the Euclidean distance between the frame feature vector and the feature vectors of each frame in the selected keyframe set; comparing the information gain with a preset information gain threshold, and adding the current frame to the keyframe set only when the information gain exceeds the threshold; and stopping the selection when the number of frames in the keyframe set reaches a preset number.

3. The digital twin interactive system based on neural radiation fields and cognitive maps according to claim 2, characterized in that, Also includes: Perform radiometric calibration and distortion correction on each frame in the keyframe image set; Radiometric calibration involves pre-shooting a standard color chart to calibrate the radiometric response coefficient of each camera, mapping the images acquired by each camera to a unified radiometric space. The distortion correction employs a combination of radial and tangential distortion correction.

4. The digital twin interactive system based on neural radiation fields and cognitive maps according to claim 1, characterized in that, Obtaining implicit three-dimensional field representations also includes the following parts: A deformation field network is introduced, which takes spatial location and time parameters as input and outputs spatial displacement offset. It maps the observed spatial coordinates to standard spatial coordinates and performs density and color queries and rendering in the standard space. When the deformation gradient of a certain region exceeds the preset deformation threshold, the region is automatically split into independent neural subfields, and each neural subfield performs its own density and color query and rendering.

5. The digital twin interactive system based on neural radiation fields and cognitive maps according to claim 1, characterized in that, Building a time-series knowledge graph also includes the following: The system detects the differences between new facts generated in newly collected data and the current temporal knowledge graph, recalculates the node embedding representation for the affected subgraph regions, and propagates the updates to the entity nodes associated with the affected subgraphs through message passing in a graph convolutional network. Based on a time-series graph neural network, the system takes the historical state sequence of each node as input and uses an autoregressive approach to generate node state predictions for future time steps, outputting the trend of device state changes and the probability of abnormal events.

6. The digital twin interactive system based on neural radiation fields and cognitive maps according to claim 1, characterized in that, The generation of user interaction intent information also includes: applying a physical information constraint loss term to the estimated full-body posture data. The physical information constraint loss term is a weighted sum of the penalty term for joint angles deviating from the allowable range and the penalty term for joint torques deviating from the Jacobian matrix and the product of external forces, which suppresses posture estimation results that do not conform to human kinematics in a soft penalty manner; collecting RGB images of the user's hand, using the hand keypoint detection network to output the coordinates of the hand keypoints, and combining inverse kinematics solution to eliminate depth ambiguity caused by finger self-occlusion, supplementing the full-body posture data with fine finger-level posture data.

7. The digital twin interactive system based on neural radiation fields and cognitive maps according to claim 1, characterized in that, Physical simulation also includes: A learnable material constitutive model is introduced, which takes strain, strain rate, temperature and material parameters as inputs and outputs stress through a feedforward neural network to replace the traditional linear constitutive model for calculating material response. Continuous collision detection is performed based on the density field represented by the implicit three-dimensional field. The feedback force is calculated according to the contact depth and material stiffness. The feedback force is applied to the user's hand through a force feedback device, and the texture of the contact surface is simulated through high-frequency vibration.

8. The digital twin interactive system based on neural radiation fields and cognitive maps according to claim 1, characterized in that, The specific logic for establishing a digital twin intelligent agent is as follows: Initial behavioral strategies are obtained by extracting policy priors from expert demonstration data through imitation learning; In a virtual simulation environment, the proximal policy optimization algorithm is used to fine-tune the initial behavior policy through reinforcement learning. The proximal policy optimization algorithm constrains the policy update magnitude by pruning the probability ratio between the old and new policies. The expected value of the pruning probability ratio weighted by the advantage function estimate is used as the optimization objective. After optimization, domain randomization and residual learning are performed.

9. The digital twin interactive system based on neural radiation fields and cognitive maps according to claim 8, characterized in that, The generation of computation scheduling policies and transmission control parameters also includes: The user's historical location, velocity, and acceleration sequences are encoded by a long short-term memory network to predict the user's future viewpoint location trajectory. Based on the prediction results, the scene data of the corresponding viewpoint is pre-rendered and pre-loaded in the edge computing node cache in advance. The cache adopts a three-level content distribution structure of edge nodes, regional nodes, and central cloud. The differences between the parameters of the old and new neural radiation field models are calculated. After the difference parameters are sparsified, only the non-zero difference terms are transmitted to each edge computing node. Each edge computing node applies the patch to update the local model parameters.

Images