Intelligent visual design display interaction system based on virtual reality technology

Abstract

The invention relates to the technical field of virtual reality interaction, in particular to an intelligent visual design display interaction system based on the virtual reality technology, and the system comprises the steps: synchronously collecting space electromagnetic fingerprint data, biomechanical characteristic data and ambient light field distribution data through a multi-mode perception array; generating a primitization scene model with implicit neural representation; inputting the primitization scene model into an adaptive topology sensing network, and generating a delta-dynamic field carrying space-time continuity constraints; constructing an asynchronous space anchor point based on the delta-dynamic field, and outputting a lambda-correction field with a collision avoidance path; and carrying out space-time convolution fusion on the lambda-correction field and real-time biomechanical characteristic data, and generating a three-dimensional visual scene through a physically-guided neural rendering pipeline. According to the method, the element layout and the path in the virtual space are adaptively adjusted, so that potential interactive interference and collision risks are automatically avoided, and the fluency and the safety in the interactive process are ensured.

Description

Technical Field

[0001] The present invention relates to the field of virtual reality interaction technology, and particularly to an intelligent visual design display interaction system based on virtual reality technology. Background Art

[0002] With the rapid development of virtual reality (VR) technology, the application of VR systems in fields such as education has gradually deepened. Virtual reality technology can provide users with an immersive experience as if they were on the scene. However, existing VR systems still have certain limitations in aspects such as scene generation, interaction design, and real-time feedback. Especially in educational and design display applications, traditional interaction methods often lack high precision and high responsiveness, and cannot meet users' needs for immersive and personalized experiences.

[0003] Currently, most interaction systems based on virtual reality technology rely on a single input method (such as a gamepad, touch screen, or mouse and keyboard). Although this method is simple, its interactivity and flexibility are poor. Especially in an environment that requires fine-grained operations and complex dynamic interactions, it is difficult to provide sufficient immersion and realism. Traditional virtual reality systems often cannot real-time perceive users' biomechanical characteristics (such as movement trajectories, postures, hand torques, etc.) and environmental changes, resulting in a lack of precise interaction and feedback between the elements in the virtual world and users' behaviors, which greatly affects the interaction effect and system response speed.

[0004] In addition, collision detection and dynamic scene generation in virtual reality are still technical problems. Traditional collision detection algorithms mostly rely on the boundaries of static objects and predefined paths, and cannot effectively handle real-time changes in complex environments. Especially in a virtual environment with multi-person object interaction or multi-person collaboration, problems such as interaction interference, collision, or occlusion often occur, affecting users' interaction experience. Summary of the Invention

[0005] The present invention provides an intelligent visual design display interaction system based on virtual reality technology.

[0006] An intelligent visual design display interaction system based on virtual reality technology includes:

[0007] Heterogeneous teaching data fusion module: Synchronously collect spatial electromagnetic fingerprint data (including UWB phase difference )), biomechanical characteristic data (including hand joint torque τ), and environmental light field distribution data through a multi-modal perception array, and generate a primitive scene model with implicit neural representation;

[0008] Dynamic topology self-organization module: Input the primitive scene model into an adaptive topology perception network, extract the spatial manifold characteristics of irregular dynamic elements, and generate a δ-dynamic field carrying spatio-temporal continuity constraints;

[0009] Conflict resolution field construction module: Based on the δ-dynamic field, asynchronous spatial anchor points are constructed, and the potential interference area of the teacher-student interaction actions is calculated using the Lie group symmetry constraint algorithm, and a λ-corrected field with a collision avoidance path is output;

[0010] Dynamic scene generation module: Spatiotemporally convolves and fuses the λ-corrected field with real-time biomechanical feature data, and generates a three-dimensional visualization scene through a physically-guided neural rendering pipeline.

[0011] Optionally, the heterogeneous teaching data fusion module includes:

[0012] Deploy a UWB positioning array (operating frequency band 6.5 GHz ± 200 MHz) and a biomechanical sensor (manipulator). Among them, each pair of transceiver units in the UWB positioning array measures the phase difference in real time where is the phase at the receiving end, is the phase at the transmitting end. The phase at the receiving end and the phase at the transmitting end form spatial electromagnetic fingerprint data, reflecting the phase difference between the received and transmitted signals. The biomechanical sensor obtains the torque vector τ=(τ 1 ,τ 2 ,...,τ 21 ) of 21 joints of the hand at a sampling rate of 200 Hz, forming biomechanical feature data;

[0013] Capture ambient light field distribution data through an array of light field cameras, and synchronously obtain the HDR light intensity matrix L(x,y,λ,t), where λ∈[380nm,780nm] is the visible spectral band, and L(x,y,λ,t) describes the light intensity at the spectral wavelength λ and time t at the position (x,y);

[0014] Generate a hardware synchronization trigger pulse signal (frequency 1 kHz ± 50 ppm), time-align the spatial electromagnetic fingerprint data, torque vector and light field data at the millisecond level, and construct a spatio-temporal correlation tensor where N and M represent the dimensions of space and time respectively, and 3 represents the dimensions of electromagnetic fingerprint, torque vector and light field data;

[0015] Input the spatio-temporal correlation tensor into the feature decoupling network, and separate the static scene primitive F static from the dynamic interaction primitive F dynamic through the spatial attention mechanism. Both the static scene primitive and the dynamic interaction primitive are encoded as implicit neural representations and encoded by a multi-layer perceptron (MLP);

[0016] Based on a differentiable renderer, jointly optimize the static scene primitive and the dynamic interaction primitive to generate a primitive scene model including the spatial biomechanical light field coupling relationship, and its implicit representation is:

[0017] Among them, M base is the primitive scene model, representing the primitive representation of the scene, combining static and dynamic primitives, represents the neural field superposition operator, w i is the weight coefficient, dynamically adjusted by the light field distribution L(x, y, λ, t), n is the number of static scene primitives, is the implicit neural representation of the i-th static scene primitive, is the implicit neural representation of the j-th dynamic interaction primitive, t 0 ,t 1 represents the time interval considered in the dynamic primitive optimization process.

[0018] Optionally, the feature decoupling network is based on a deep neural network, separates the spatio-temporal data (i.e., static information and dynamic information) in the spatio-temporal correlation tensor, extracts static and dynamic features according to the input spatio-temporal data. During the decoupling process, the spatial attention mechanism, through a weighting method, enables the feature decoupling network to focus on the feature differences at different spatial positions. In this mechanism, the network assigns different weights to each spatial position according to the input spatio-temporal data, focuses on the regions sensitive to dynamic changes (such as hand movements, object movements, etc.), and automatically identifies and separates the dynamic interaction primitives that change over time and the static scene primitives that do not change over time.

[0019] Optionally, in the generation of the primitive scene model of the spatial biomechanical light field coupling relationship:

[0020] The differentiable renderer is optimized and rendered through backpropagation, automatically adjusts the visual effect of the scene through the optimization process. The differentiable renderer renders the spatial structures of the static scene primitives and the dynamic interaction primitives, calculates and optimizes the performance under the lighting conditions. In the joint optimization, the static scene primitives and the dynamic interaction primitives jointly participate in the rendering and optimization of the scene. Through the differentiable renderer, the implicit neural representations of the static scene primitives and the dynamic interaction primitives are adjusted according to the actual light field data and biomechanical characteristics during the rendering process.

[0021] During the optimization process, spatial information (the position and shape of the primitives), biomechanical information (such as joint torque, posture, etc.) and light field data (ambient light, light field distribution) are coupled.

[0022] Optionally, the adaptive topology perception network in the dynamic topology self-organization module is based on a graph neural network architecture, including a spatial encoder, a manifold feature extractor and a spatio-temporal constraint unit, where:

[0023] The spatial encoder discretizes the primitivized scene model into a supervoxel graph structure, G=(V, E), representing the discretized representation of the primitivized scene model, where V is the set of vertices, and the vertices are the primitives, and E is the set of edges, representing the connection attributes between the vertices. The vertex attributes include the implicit neural representation of the static scene primitives and the activation intensity β of the dynamic interaction primitives;

[0024] The manifold feature extractor adopts four-layer residual graph convolution operations. Each layer of convolutional kernel is defined as a learnable metric space d(v i , v j ), and d(v i , v j ) represents the metric distance between the i-th primitive and the j-th primitive;

[0025] Physical prior knowledge is injected into the spatio-temporal constraint unit to construct dual constraint conditions:

[0026] Spatial continuity constraint;

[0027] Temporal consistency constraint;

[0028] A spatio-temporally continuous δ-dynamic field is generated through a neural differential equation solver.

[0029] Optionally, the conflict resolution field construction module specifically includes:

[0030] Dynamic spatial anchor deployment: Based on the spatio-temporal gradient distribution of the δ-dynamic field, find the hot spots of teacher-student interaction, and deploy a set of dynamic spatial anchors in the hot spots of teacher-student interaction. Each anchor includes position attributes, attitude attributes, and speed attributes:

[0031] Teacher-student action pose representation: Defines the action trajectories of teachers and students, and establishes the action trajectories of teachers and students through Lie algebra, including angular velocity and linear velocity;

[0032] Calculate the potential interference amount: By calculating the adjoint action of the Lie group, obtain the interference amount existing in the teacher-student interaction process, compare the action trajectories of the two, calculate their relative motion in space, and find potential interference. If the interference amount exceeds the preset threshold, an interference warning is issued;

[0033] Collision avoidance path generation: After the interference warning, construct a collision avoidance path. By correcting the δ-dynamic field and considering the potential interference area at the same time, generate a new path weight matrix. The path weight matrix controls the path selection, avoids the interference area and ensures the continuity of the movement.

[0034] Optionally, the position attribute finds the position with the largest change by calculating the change of the δ-dynamic field in the time gradient direction, and compensates for its terminal delay;

[0035] Among the pose attributes, the pose of the anchor point is generated through Lie group symmetry constraints to ensure compliance with the constraints of the inertia tensor in the calculation;

[0036] The velocity attribute calculates the anchor point velocity based on the position change of the δ-dynamic field for motion analysis.

[0037] Optionally, the generation of the collision avoidance path includes

[0038] Using the obtained δ-dynamic field to initialize the correction field. Then, by weighted adjustment of the δ-dynamic field, a decay factor is introduced. The decay factor is 3.2 ± 0.3, which controls the influence range of the interference region. The decay factor exponentially decays the potential interference amount to reduce the influence of the interference region on the correction path;

[0039] Introduce the weight matrix of the path, and weight different path points in the correction path. The weight matrix is a diagonal matrix, including three weight values (corresponding to spatial information, temporal information, and biomechanical data), all calculated through the softmax function. The softmax function adjusts the importance of different points on the path according to the distance or importance of each position.

[0040] Finally, the generated λ-correction field is weighted and integrated by the δ-dynamic field, the influence of the attenuated interference region, and the path weight matrix. Through the λ-correction field, the interaction trajectories of the teacher and students are adjusted to avoid potential collisions while maintaining the smoothness and rationality of the actions.

[0041] Optionally, the dynamic scene generation module specifically includes:

[0042] Fuse the λ-correction field and the real-time biomechanical feature data in space and time. The fusion includes mapping the λ-correction field and the real-time biomechanical feature data into a unified space model, and through the convolution operation of the trained convolutional neural network, synthesizing the mixed data in the space-time dimension so that the spatial state at each moment reflects the influence of the current action and physical constraints;

[0043] The physics-guided neural rendering pipeline generates a natural and realistic rendering effect by combining physical rules with a convolutional neural network.

[0044] Optionally, the convolutional neural network constructs the geometric structure in the scene through the input λ-correction field and the fused biomechanical feature data, and the geometric structure is then corrected by physics guidance to ensure that the movement of objects in the virtual space conforms to physical laws;

[0045] After being guided by physics, the neural rendering pipeline uses the fused information to generate a three-dimensional scene to display the dynamic effect of the interaction between the teacher and the students.

[0046] Advantages of the present invention:

[0047] In the present invention, by synchronously collecting spatial electromagnetic fingerprint data, biomechanical feature data, and environmental light field distribution data through a multimodal perception array, a primitive scene model with implicit neural representation is generated, which can greatly enhance the immersion and interaction experience of the virtual reality system. Especially during the interaction between teachers and students, the fusion of real-time biomechanical data (such as hand joint torque) and environmental light field data enables the virtual scene to accurately respond to the user's actions and environmental changes, thus providing a more natural and intuitive interaction effect.

[0048] In the present invention, through the combination of the δ-dynamic field and the λ-modified field, and by using Lie group symmetry constraints and spatial anchor deployment techniques, dynamic scene generation and conflict resolution in complex environments are achieved. Especially during teacher-student interaction, based on real-time collected biomechanical data and environmental parameters, the element layout and path in the virtual space can be adaptively adjusted, thereby automatically avoiding potential interaction interference and collision risks, ensuring the fluency and safety during the interaction process. The conflict avoidance mechanism has important value in practical applications, especially in virtual teaching and design scenarios that require high-precision interaction, which can effectively improve the user experience and interaction efficiency.

[0049] In the present invention, the proposed physically-guided neural rendering pipeline greatly improves the visual realism in the virtual reality scene through spatio-temporal convolution fusion and physical constraint optimization. After fusing biomechanical data, spatial electromagnetic data, and environmental light field information, the three-dimensional visualization scene generated by the physically-guided encoder can more accurately reflect the object motion, collision, and lighting changes in the virtual world. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only those of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0051] Figure 1 It is a schematic diagram of the system module of an embodiment of the present invention;

[0052] Figure 2 It is a schematic diagram of the dynamic topology self-organization module of an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0053] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it should be noted here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments. For some well-known technologies, those skilled in the art can also adopt other alternative methods for implementation; moreover, the accompanying drawings are only for more specifically describing the embodiments and are not intended to specifically limit the present invention.

[0054] It should be noted that in the specification, when referring to "an embodiment", "embodiment", "exemplary embodiment", "some embodiments", etc., it indicates that the described embodiment may include specific features, structures or characteristics, but not necessarily every embodiment includes such specific features, structures or characteristics. Additionally, when combining embodiments to describe specific features, structures or characteristics, implementing such features, structures or characteristics in combination with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the relevant art.

[0055] Generally, terms can be understood at least in part from their use in the context. For example, at least in part depending on the context, the term "one or more" used herein can be used to describe any feature, structure or characteristic in a singular sense, or can be used to describe a combination of features, structures or characteristics in a plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey a set of exclusive factors, but rather, at least in part depending on the context, can allow for the existence of other factors that may not be explicitly described.

[0056] As Figure 1 - Figure 2 shown, an intelligent visual design display interaction system based on virtual reality technology includes:

[0057] Heterogeneous teaching data fusion module: Synchronously collect spatial electromagnetic fingerprint data (including UWB phase difference )), biomechanical feature data (including hand joint torque τ) and environmental light field distribution data through a multi-modal perception array, and generate a primitive scene model with implicit neural representation;

[0058] Dynamic topology self-organization module: Input the primitive scene model into an adaptive topology perception network, extract the spatial manifold features of irregular dynamic elements, and generate a δ-dynamic field carrying spatio-temporal continuity constraints;

[0059] Conflict resolution field construction module: Construct asynchronous spatial anchor points based on the δ-dynamic field, use the Lie group symmetry constraint algorithm to calculate the potential interference area of the teacher-student interaction actions, and output a λ-corrected field with a collision avoidance path;

[0060] Dynamic scene generation module: Perform spatio-temporal convolutional fusion of the λ-corrected field and real-time biomechanical feature data, and generate a three-dimensional visualization scene through a physically-guided neural rendering pipeline.

[0061] The heterogeneous teaching data fusion module includes:

[0062] Deploy a UWB positioning array (operating frequency band 6.5 GHz ± 200 MHz) and a biomechanical sensor (manipulator). Among them, each pair of transceiver units in the UWB positioning array measures the phase difference in real time Among them is the phase at the receiving end, is the phase at the transmitting end. The phase at the receiving end and the phase at the transmitting end form spatial electromagnetic fingerprint data, reflecting the phase difference between the received and transmitted signals. The biomechanical sensor acquires the torque vector τ=(τ 1 ,τ 2 ,...,τ 21 ) of 21 hand joints at a sampling rate of 200 Hz, forming biomechanical feature data;

[0063] Capture environmental light field distribution data through an optical field camera array, and synchronously obtain the HDR light intensity matrix L(x,y,λ,t), where λ∈[380nm,780nm] is the visible spectral band, and L(x,y,λ,t) describes the light intensity at the spectral wavelength λ and time t in the position (x,y);

[0064] Generate a hardware synchronization trigger pulse signal (frequency 1 kHz ± 50 ppm), perform millisecond-level time alignment on the spatial electromagnetic fingerprint data, torque vector, and light field data, and construct a spatio-temporal correlation tensor where N and M represent the dimensions of space and time respectively, and 3 represents the dimensions of electromagnetic fingerprint, torque vector, and light field data;

[0065] Input the spatio-temporal correlation tensor into the feature decoupling network, and separate the static scene primitive F static and the dynamic interaction primitive F dynamic through the spatial attention mechanism. Both the static scene primitive and the dynamic interaction primitive are encoded as implicit neural representations, and are encoded by a multi-layer perceptron (MLP):

[0066] Dynamic interaction primitive Among them, MLP represents a multi-layer perceptron, τ>0.1 N·m is used to limit the data range, (x,y,z) is the spatial position, θ is the parameter of the MLP. For dynamic interaction primitives (such as hand movements, interactions between people and objects, etc.), in addition to the spatial position, the input also includes the change in the time dimension. By inputting the movement trajectory or interaction behavior of dynamic elements into the MLP, the network can learn the patterns and dynamic characteristics of these elements over time. The MLP not only encodes the spatial information of the object here, but also takes into account the continuity and time-variability of the movement, thereby generating an implicit representation containing time-varying information;

[0067] Implicit Neural Representation of Static Scene Primitives The result of encoding static elements in the scene through an MLP, encoding static elements in the scene (such as buildings, objects, etc.) into an implicit vector representation. Formally, it is expressed as: (Spatial elements that do not change over time), where (x, y, z) is the spatial position in the scene and θ is the network parameter; for static scene primitives (fixed objects, background), the multi-layer perceptron takes the spatial position (three-dimensional coordinates) in the scene as input and encodes these static elements through a multi-layer neural network. The MLP will generate an implicit vector representation according to the spatial distribution and morphological characteristics of the object, and this vector captures the geometric and structural information of the static elements in space;

[0068] Based on a differentiable renderer, jointly optimize the static scene primitives and dynamic interaction primitives to generate a primitive scene model including the coupling relationship of spatial biomechanical light fields, and its implicit representation is:

[0069] where M base is the primitive scene model, representing the primitive representation of the scene, combining static and dynamic primitives, represents the neural field superposition operator, w i is the weight coefficient, dynamically adjusted by the light field distribution L(x, y, λ, t), n is the number of static scene primitives, is the implicit neural representation of the i-th static scene primitive, is the implicit neural representation of the j-th dynamic interaction primitive, t 0 ,t 1 represents the time interval considered in the dynamic primitive optimization process.

[0070] The spatio-temporal correlation tensor includes electromagnetic fingerprint data, biomechanical feature data, and light field data, which are the results of alignment in time and space. This tensor contains information in different dimensions, including spatial position, joint torque, and light intensity. The data at each moment is organized into a high-dimensional tensor, capturing the spatio-temporal relationships between various data sources.

[0071] Based on a deep neural network, the feature decoupling network separates the spatio-temporal data (i.e., static information and dynamic information) in the spatio-temporal correlation tensor, extracts static and dynamic features according to the input spatio-temporal data. During the decoupling process, the spatial attention mechanism, through a weighting method, enables the feature decoupling network to focus on the feature differences at different spatial positions. In this mechanism, the network will assign different weights to each spatial position according to the input spatio-temporal data, focus on areas sensitive to dynamic changes (such as hand movements, object movements, etc.), and automatically identify and separate the dynamic interaction primitives that change over time and the static scene primitives that do not change over time.

[0072] Through the spatial attention mechanism, the static primitives and dynamic primitives will be effectively decoupled in the feature space. The features of the static primitives will be concentrated on the spatially invariant parts (buildings, fixed objects), while the features of the dynamic primitives will be mainly concentrated on the parts that change over time (hand movements, interaction behaviors, etc.). In this way, the implicit neural representations of the static scene primitives and dynamic interaction primitives will be successfully separated.

[0073] In the generation of the primitive scene model of the spatial biomechanical light field coupling relationship:

[0074] First, the static scene primitives and dynamic interaction primitives separated by the feature decoupling network are used as inputs. These primitives respectively contain different implicit neural representations. The static primitives mainly describe objects that do not change over time, and the dynamic primitives describe interaction behaviors that change over time;

[0075] The differentiable renderer is optimized for rendering through backpropagation, automatically adjusting the visual effects of the scene through the optimization process. The differentiable renderer renders the spatial structures of the static scene primitives and dynamic interaction primitives, calculates and optimizes the performance under lighting conditions. In the joint optimization, the static scene primitives and dynamic interaction primitives jointly participate in the rendering and optimization of the scene. Through the differentiable renderer, the implicit neural representations of the static scene primitives and dynamic interaction primitives are adjusted according to the actual light field data and biomechanical characteristics during the rendering process. This means that the appearance of static objects will be affected by lighting changes and spatial relationships, while the dynamic interaction primitives (such as hand movements, posture changes, etc.) will adjust the scene in real time according to the actions and interaction processes.

[0076] During the optimization process, spatial information (the position and shape of the primitives), biomechanical information (such as joint torques, postures, etc.) and light field data (ambient lighting, light field distribution) are coupled. Specifically, the spatial relationships between the static primitives and dynamic primitives affect the rendering effect, while the biomechanical data affects the movement trajectories and interaction behaviors of the dynamic primitives, and the light field data determines the final visual presentation.

[0077] Finally, after joint optimization, the implicit neural representations of the static scene primitives and dynamic interaction primitives are synthesized into a complete primitive scene model. This model not only contains the spatial information of each primitive, but also integrates various information such as biomechanical behaviors (such as hand movements) and environmental light field distributions, thus realizing a highly realistic and interactive virtual scene.

[0078] The backpropagation optimization is as follows:

[0079] Define the loss function: The loss function is used to measure the difference between the generated scene and the actual scene, usually involving spatial consistency, lighting consistency, and biomechanical consistency:

[0080] Wherein:

[0081] represents the loss of spatial consistency, measuring the comparison between the geometry of the scene and the real scene; represents the loss of biomechanical consistency, measuring the accuracy of the biomechanical model based on joint torques, postures, etc.;

[0082] represents the loss of lighting consistency, measuring the difference in the lighting distribution between the ambient light field and the generated scene;

[0083] Gradient calculation: The core of backpropagation is to calculate the gradient of the loss function with respect to the MLP network parameters θ. The gradient of the loss function with respect to each parameter needs to be calculated: This gradient represents the rate of change of the loss function in the parameter space, that is, how the loss will change if the parameter θ is adjusted. Through these gradients, the weights of the network can be adjusted to reduce the loss;

[0084] Parameter update: After calculating the gradients, gradient descent is used to update the parameters of the neural network. The rule for parameter update is: where θ (t) represents the parameter value at the current moment, and θ (t+1) represents the updated parameter value after optimization. η is the learning rate, which controls the step size of each update.

[0085] Optimization objective: The objective is to minimize the loss function through backpropagation optimization, so that the primitive scene model generated by the network can more accurately couple spatial information, biomechanical information, and light field data. Minimize, so that the primitive scene model generated by the network can more accurately couple spatial information, biomechanical information, and light field data.

[0086] The adaptive topology perception network in the dynamic topology self-organization module is based on the graph neural network architecture, including a spatial encoder, a manifold feature extractor, and a spatio-temporal constraint unit. Wherein:

[0087] The spatial encoder discretizes the primitive scene model into a supervoxel graph structure, G=(V, E), representing the discretized representation of the primitive scene model. Wherein, V is the set of vertices, and the vertices are the primitives. E is the set of edges, representing the connection attributes between the vertices. The vertex attributes include the implicit neural representation of the static scene primitive and the activation intensity β of the dynamic interaction primitive;

[0088] The vertex attributes include the implicit neural representation and the dynamic primitive activation intensity, because each vertex represents the information of a primitive (such as an object, a scene element, or an action). Specifically:

[0089] The implicit neural representation part of the static scene primitives is used to represent the spatial information of the static scene primitives. The implicit neural representation can learn complex information such as the geometry and texture of the scene through a neural network, rather than just traditional geometric point coordinates or color values. It allows for more efficient and accurate modeling and reconstruction of scene elements. Especially when dealing with complex scenes, the implicit representation can better capture details. The dynamic primitive activation intensity part describes the dynamic behavior of each primitive, that is, how they change over time. Through the dynamic primitive activation intensity, the motion intensity or interaction activity of the primitives can be captured, which helps the neural network determine which elements in the scene are changing and the intensity of the change, thus generating a more accurate dynamic scene. The implicit neural representation of the static scene primitives helps capture the static spatial information, while the dynamic primitive activation intensity is used to describe the dynamic changes of the primitives in the scene. The combination of the two can comprehensively describe all elements in the scene, whether static or dynamic parts that change over time;

[0090] The manifold feature extractor adopts four - layer residual graph convolution operations, and the convolutional kernel of each layer is defined as a learnable metric space where, v i , v j are both vertices in the supervoxel graph structure, d(v i , v j ) represents the metric distance between the i - th primitive and the j - th primitive, representing the primitives (including dynamic and static) in the primitivized scene model, represents the concatenation operator. Through the differentiable renderer, backpropagation optimization is performed to dynamically adjust the metric weight matrix W of the manifold feature extractor, ‖·‖ 2 is the Euclidean norm, used to measure the distance between vectors. The specific optimization objective is: L total =α·L photo +(1 - α)·L flow , where, L total is the total loss function, which contains the weighted sum of the photometric consistency loss L flow and the optical flow smoothness loss L flow . α is the weighted coefficient of the photometric consistency loss L photo and the optical flow smoothness loss L flow , adjusting the importance of the two;

[0091] Among them, the photometric - consistency loss: L photo =∑‖I rendered -I gt ‖ 2 ;

[0092] The optical flow smoothness loss: α = 0.7±0.05; where, I rendered is the rendered image, representing the result predicted by the model, Igt It is a real image used to calculate the loss by comparing with the rendering result. It represents the optical flow gradient of the dynamic primitive, indicating the spatial variation of the dynamic element, and is used to calculate the optical flow smoothness loss.

[0093] Physical prior knowledge is injected into the spatio-temporal constraint unit to construct a dual constraint condition:

[0094] Spatial continuity constraint: For any adjacent dynamic interaction primitive v i , v j , it is forced to satisfy: σ = 0.3, where σ represents the standard deviation in the spatial continuity constraint, controlling the influence range of the spatial distance between adjacent primitives, and β i , β j is the activation intensity of two adjacent dynamic interaction primitives.

[0095] Temporal consistency constraint: Define the dynamic primitive evolution equation:

[0096] γ = 2.5 ± 0.3 -1 , where γ is the constant in the temporal consistency constraint.

[0097] A spatio-temporally continuous δ-dynamic field is generated through a neural differential equation solver, and its representation is:

[0098] where ⊙ represents the Hadamard product, is the tensor contraction operation, Lap(G) is the graph Laplacian regularization term, and λ = 0.05 ± 0.01, where λ is the regularization coefficient.

[0099] The conflict resolution field construction module specifically includes:

[0100] Dynamic spatial anchor deployment: Based on the spatio-temporal gradient distribution of the δ-dynamic field, find the hot spots of teacher-student interaction, and deploy a set of dynamic spatial anchors in the hot spots of teacher-student interaction. Each anchor includes position attributes, attitude attributes, and velocity attributes:

[0101] Teacher-student action pose representation: Defines the action trajectories of teachers and students. Through Lie algebra, including angular velocity and linear velocity, establish the action trajectories of teachers and students, and the action trajectories are represented by mathematical transformations to ensure compliance with kinematic constraints;

[0102] Calculate the potential interference amount: By calculating the adjoint action of the Lie group, obtain the interference amount existing during the interaction between teachers and students, compare the action trajectories of the two and calculate their relative motion in space, and find potential interference. If the interference amount exceeds the preset threshold, an interference warning is issued;

[0103] Collision avoidance path generation: After interference warning, a collision avoidance path is constructed. By correcting the δ-dynamic field and considering potential interference regions, a new path weight matrix is generated. The path weight matrix controls the path selection, avoids interference regions, and ensures the continuity of motion. The finally generated path can effectively avoid collisions between teachers and students and ensure interactive safety.

[0104] The position attribute finds the position with the largest change by calculating the change of the δ-dynamic field in the time gradient direction and compensates for the terminal delay of it;

[0105] In the attitude attribute, the attitude of the anchor point is generated by the Lie group symmetry constraint to ensure compliance with the constraint conditions of the inertia tensor in the calculation;

[0106] The velocity attribute calculates the anchor point velocity based on the position change of the δ-dynamic field for motion analysis.

[0107] The set of spatial anchor points is represented as: A = {a k |a k =(p k ,R k ,v k )}, where p k is the position coordinate of the dynamic spatial anchor point, R k is the attitude matrix of the dynamic spatial anchor point, vk is the dynamic spatial anchor point velocity, and the attitude matrix R k is generated by the Lie group symmetry constraint and satisfies is the inertia tensor, represents the transpose of R k for calculating the symmetry of the Lie group;

[0108] The Lie algebra representation of the teacher-student action poses is established as:

[0109] The teacher's action trajectory T tea (t):

[0110] The student's action trajectory T stu (t): T 0 is the initial action pose, t is the time point, ξ is the Lie algebra representation, ξ=(angular velocity, linear velocity), Ttea(t) and Tstu(t) respectively represent the action trajectories of the teacher and the student at time t, obtained through the exponential mapping of the Lie group and the Lie algebra, and ξtea and ξstu respectively represent the Lie algebras of the teacher and the student;

[0111] Calculate the potential interference quantity Φ through the adjoint action of the Lie group:

[0112] When Φ > η·tr(J), an interference warning is triggered (η = 0.05), where tr represents the trace of J, which is the sum of its diagonal elements and is used to represent the total inertial characteristics. and respectively represent the adjoint action matrices of the teacher and the student under the Lie group, which are used to transform the Lie algebra into the world coordinate system to calculate the relative interference.

[0113] Collision avoidance path generation includes

[0114] Using the obtained δ-dynamic field to initialize the correction field. Then, by weighted adjustment of the δ-dynamic field, a decay factor is introduced. The decay factor is 3.2 ± 0.3, which controls the influence range of the interference area. The decay factor exponentially decays the potential interference amount to reduce the influence of the interference area on the corrected path.

[0115] Introduce the weight matrix of the path, and weight different path points in the corrected path. The weight matrix is a diagonal matrix, including three weight values (corresponding to spatial information, time information, and biomechanical data), all of which are calculated by the softmax function. The softmax function adjusts the importance of different points on the path according to the distance or importance of each position. The softmax function converts a set of arbitrary real numbers into a probability distribution. Specifically, each value is mapped to the interval [0,1], and the sum of all values is 1. The weight value calculated by the softmax function can be regarded as a "relative importance evaluation" of the input features, and the sum of such weight values is always 1.

[0116] The generation of the corrected field λ- also requires integrating all points on the path, and this path satisfies specific constraints, that is, the rate of change of the position on the path cannot exceed 0.8 times the maximum speed. This limit ensures that the path does not change too violently, avoids overly sharp movements, and maintains the smoothness of the movement.

[0117] Finally, the generated λ-corrected field is weighted and integrated by the δ-dynamic field, the influence of the attenuated interference area, and the path weight matrix. Through the λ-corrected field, the interaction trajectories of the teacher and the student are adjusted to avoid potential collisions while maintaining the smoothness and rationality of the actions.

[0118] The λ-corrected field for constructing the collision avoidance path is specifically expressed as:

[0119]

[0120] Among them, the decay factor β = 3.2 ± 0.3, which controls the influence range of the interference area, and the path weight matrix: W c = softmax(-R k ·e i ),e iRepresents the direction vector in the path, ∫ path is the integration path, ds represents the infinitesimal arc length element along the path path, which is a tiny spatial unit on the path during the integration process and is usually used to describe the bending characteristics of a curve or path. A represents the force on the path, is a constant used to control the rate of exponential decay.

[0121] The dynamic scene generation module specifically includes:

[0122] Fuse the λ-modified field and real-time biomechanical feature data in space-time. The fusion includes mapping the λ-modified field and real-time biomechanical feature data into a unified spatial model to ensure that they can work together to produce realistic and coherent dynamic effects. The λ-modified field represents the influence of spatial position and action changes in the virtual scene, while the biomechanical data provides physical information based on the actions of the teacher and student. Through the convolution operation of the trained convolutional neural network, the mixed data is synthesized in the spatio-temporal dimension, so that the spatial state at each moment reflects the influence of the current action and physical constraints;

[0123] The physics-guided neural rendering pipeline generates natural and realistic rendering effects by combining physical rules with convolutional neural networks, where.

[0124] The convolutional neural network constructs the geometric structure in the scene through the input λ-modified field and the fused biomechanical feature data. The geometric structure is then corrected by physics guidance to ensure that the movement of objects in the virtual space conforms to physical laws, including conservation of momentum and conservation of energy. The physical constraint optimization ensures that the generated three-dimensional scene is more in line with actual physical phenomena (for example, the texture, momentum, and interaction effects of objects);

[0125] After being guided by physics, the neural rendering pipeline uses the fused information to generate a three-dimensional scene, showing the dynamic effects of the interaction between the teacher and the student. This process includes the three-dimensional modeling and rendering of elements such as objects, actions, and interactions in the scene, so that users can intuitively see the effects of actions and interactions in the virtual environment, and finally output a three-dimensional visualization scene that can dynamically reflect the actions, interactions, and physical behaviors of the teacher and the student.

[0126] This invention covers any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To enable the public to have a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments of this invention. However, those skilled in the art can fully understand this invention without the description of these details. Additionally, well-known methods, processes, procedures, elements, and circuits are not described in detail to avoid unnecessary confusion to the essence of this invention.

[0127] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. An intelligent visual design display interactive system based on virtual reality technology, characterized in that: include: Heterogeneous teaching data fusion module: Through the multimodal sensing array, spatial electromagnetic fingerprint data, biomechanical feature data and ambient light field distribution data are synchronously collected to generate a primitive scene model with implicit neural representation; Dynamic topology self-organization module: inputs the primitive scene model into the adaptive topology perception network, extracts the spatial manifold features of irregular dynamic elements, and generates a δ-dynamic field with spatiotemporal continuity constraints; Conflict resolution field construction module: construct asynchronous spatial anchor points based on the δ-dynamic field, use the Lie group symmetry constraint algorithm to calculate the potential interference area of ​​teacher-student interaction, and output the λ-correction field with collision avoidance path; Dynamic scene generation module: The λ-corrected field is fused with real-time biomechanical feature data through spatiotemporal convolution, and a 3D visualization scene is generated through a physically guided neural rendering pipeline.

2. According to claim 1, the intelligent visual design display interactive system based on virtual reality technology is characterized in that: The heterogeneous teaching data fusion module includes: Deploy UWB positioning array and biomechanical sensors, where each pair of transceiver units in the UWB positioning array measures the phase difference in real time in is the phase at the receiving end, is the phase of the transmitter. The phase of the receiver and the phase of the transmitter constitute the spatial electromagnetic fingerprint data. The biomechanical sensor obtains the torque vector of the 21 joints of the hand τ=(τ1,τ2,...,τ 21 ), forming biomechanical characteristic data; The ambient light field distribution data is captured by the light field camera array, and the HDR light intensity matrix L(x, y, λ, t) is obtained synchronously, where λ∈[380nm, 780nm] is the visible spectrum segment, and L(x, y, λ, t) describes the light intensity at the spectral wavelength λ and time t at the position (x, y); Generate hardware synchronization trigger pulse signal, time-align spatial electromagnetic fingerprint data, torque vector and light field data, and construct space-time correlation tensor Where N and M represent the dimensions of space and time, respectively, and 3 represents the dimensions of electromagnetic fingerprint, torque vector, and light field data; The spatiotemporal correlation tensor is input into the feature decoupling network, and the static scene primitives F are separated through the spatial attention mechanism. static Dynamic interaction primitives F dynamic ,Both static scene primitives and dynamic interaction primitives are encoded as implicit neural representations,encoded by a multi-layer perceptron; Based on the differentiable renderer, the static scene primitives and dynamic interaction primitives are jointly optimized to generate a primitive scene model including the spatial biomechanical light field coupling relationship, which is implicitly represented as: Among them, M base It is a primitive scene model, representing a primitive representation of a scene, combining static and dynamic primitives. represents the neural field superposition operator, w i is the weight coefficient, n is the number of static scene primitives, is the implicit neural representation of the i-th static scene primitive, is the implicit neural representation of the jth dynamic interaction primitive, and t0, t1 represent the time intervals considered during the optimization of the dynamic primitive.

3. According to claim 1, the intelligent visual design display interactive system based on virtual reality technology is characterized in that: The feature decoupling network is based on a deep neural network, which separates the spatiotemporal data in the spatiotemporal correlation tensor and extracts static and dynamic features based on the input spatiotemporal data. During the decoupling process, the spatial attention mechanism uses a weighted method to enable the feature decoupling network to focus on the feature differences at different spatial positions, assign different weights to each spatial position based on the input spatiotemporal data, focus on areas that are sensitive to dynamic changes, and automatically identify and separate dynamic interaction primitives that change with time and static scene primitives that do not change with time.

4. According to claim 1, the intelligent visual design display interactive system based on virtual reality technology is characterized in that: In the generation of the primitive scene model of the spatial biomechanical light field coupling relationship: The differentiable renderer performs optimized rendering through back propagation, and automatically adjusts the visual effect of the scene through the optimization process. The differentiable renderer renders the spatial structure of static scene primitives and dynamic interaction primitives, calculates and optimizes the performance under lighting conditions. In the joint optimization, the static scene primitives and the dynamic interaction primitives jointly participate in the rendering and optimization of the scene. Through the differentiable renderer, the implicit neural representation of the static scene primitives and the dynamic interaction primitives is adjusted according to the actual light field data and biomechanical characteristics during the rendering process; During the optimization process, spatial information, biomechanical information and light field data are coupled.

5. According to claim 2, the intelligent visual design display interactive system based on virtual reality technology is characterized in that: The adaptive topology-aware network in the dynamic topology self-organizing module is based on a graph neural network architecture, including a spatial encoder, a manifold feature extractor, and a spatiotemporal constraint unit, wherein: The spatial encoder discretizes the primitive scene model into a supervoxel graph structure, G = (V, E), where V is a set of vertices, which are primitives, and E is a set of edges, which represent the connection properties between vertices. The vertex properties include the implicit neural representation of static scene primitives. and the activation strength of the dynamic interaction primitives, β; The manifold feature extractor uses four layers of residual graph convolution operations, and each convolution kernel is defined as a learnable metric space d(v i ,v j ), d(v i ,v j ) represents the metric distance between the i-th primitive and the j-th primitive; Physical prior knowledge is injected into the space-time constraint unit to construct dual constraint conditions, including space continuity constraint and time consistency constraint; Generate space-time continuous delta-dynamic fields via a neural differential equation solver.

6. The intelligent visual design display interactive system based on virtual reality technology according to claim 1 is characterized in that: The conflict resolution field construction module specifically includes: Dynamic space anchor point deployment: Based on the spatiotemporal gradient distribution of the δ-dynamic field, the hotspots of teacher-student interaction are found, and a set of dynamic space anchor points are deployed in the hotspots of teacher-student interaction. Each anchor point includes position attributes, posture attributes, and speed attributes: Teacher-student motion posture representation: The motion trajectories of teachers and students are defined, and the motion trajectories of teachers and students are established through Lie algebra, including angular velocity and linear velocity; Calculate potential interference: By calculating the Lie group adjoint action, obtain the interference between teachers and students in the interaction process, compare the movement trajectories of the two and calculate their relative motion in space to find out the potential interference. If the interference exceeds the preset threshold, issue an interference warning. Collision avoidance path generation: After the interference warning, a collision avoidance path is constructed. By modifying the δ-dynamic field and taking into account the potential interference area, a new path weight matrix is ​​generated. The path weight matrix controls the choice of path to avoid the interference area and ensure the continuity of motion.

7. The intelligent visual design display interactive system based on virtual reality technology according to claim 6 is characterized in that: The position attribute calculates the change of the delta-dynamic field in the time gradient direction, finds the position with the largest change, and performs terminal delay compensation on it; In the posture attributes, the posture of the anchor point is generated by Lie group symmetry constraints to ensure that the constraints of the inertia tensor are met in the calculation; The velocity attribute is used to calculate the anchor point velocity according to the position change of the delta-dynamic field for motion analysis.

8. The intelligent visual design display interactive system based on virtual reality technology according to claim 6 is characterized in that: The collision avoidance path generation includes: The obtained δ-dynamic field is used to initialize the correction field. Then, the δ-dynamic field is weighted and adjusted to introduce an attenuation factor, which is 3.2±0.

3. The influence range of the interference area is controlled. The attenuation factor exponentially decays the potential interference amount and reduces the influence of the interference area on the correction path. The weight matrix of the path is introduced to weight different path points in the correction path. The weight matrix is ​​a diagonal matrix, including three weight values, all of which are calculated by the softmax function. The softmax function adjusts the importance of different points on the path according to the distance or importance of each position; The final generated λ-correction field is a weighted integration of the δ-dynamic field, the attenuated interference area influence, and the path weight matrix. The interaction trajectory of teachers and students is adjusted through the λ-correction field.

9. The intelligent visual design display interactive system based on virtual reality technology according to claim 1 is characterized in that: The dynamic scene generation module specifically includes: The λ-corrected field and real-time biomechanical feature data are fused in time and space. The fusion includes mapping the λ-corrected field and real-time biomechanical feature data into a unified spatial model, and synthesizing the mixed data in time and space dimensions through the convolution operation of the trained convolutional neural network, so that the spatial state at each moment reflects the influence of the current action and physical constraints. The physically guided neural rendering pipeline generates natural and realistic rendering effects by combining physical rules with convolutional neural networks.

10. The intelligent visual design display interactive system based on virtual reality technology according to claim 1, characterized in that: The convolutional neural network constructs the geometric structure in the scene through the input λ-corrected field and the fused biomechanical feature data, and the geometric structure is then corrected through physical guidance to ensure that the movement of the object in the virtual space complies with the laws of physics; After physical guidance, the neural rendering pipeline uses the fused information to generate a three-dimensional scene to show the dynamic effect of the interaction between teachers and students.

Images