A virtual simulation experiment image data processing method based on digital twinning

Abstract

The application provides a virtual simulation experiment image data processing method based on digital twinning, comprising: obtaining operation input data of a user in a virtual simulation experiment process, and extracting operation semantic features; taking the operation semantic features as time-varying driving conditions, injecting the time-varying driving conditions into a physical simulation model of a digital twinning body, and solving physical field state data sequences generated under continuous superposition and attenuation of operation semantics; generating an image sequence containing causal textures according to the physical field state data sequences, wherein the causal textures are obtained by mapping physical properties in the physical field that cannot be directly observed into visual textures; obtaining actual visual image sequences of a virtual simulation experiment scene in the same space-time range; performing space-time alignment and fusion on the image sequence containing the causal textures and the actual visual image sequences to generate a virtual-real fusion image sequence; and performing space-time feature analysis on the virtual-real fusion image sequence to generate an operation skill evaluation result reflecting a physical cognitive level contained in the user operation.

Description

Technical Field

[0001] This invention relates to the field of virtual simulation technology, and in particular to a method for processing virtual simulation experimental image data based on digital twins. Background Technology

[0002] With the rapid development of virtual reality and digital twin technologies, virtual simulation experiments are increasingly widely used in engineering education, skills training, and scientific visualization. Traditional virtual simulation experiment systems mainly focus on the visual presentation of the experimental process and the realism of the operational interaction, simulating the geometric appearance and basic motion behavior of experimental objects through 3D modeling and physics engines. However, when evaluating users' operational skills, these systems typically adopt a result-oriented evaluation method, such as judging whether the user has completed the preset operation steps, obtained the correct experimental result value, or recording the operation completion time as an efficiency indicator. This evaluation method can only reflect the superficial correctness of the operation and cannot delve into whether the user truly understands the physical laws and causal mechanisms underlying the experiment. On the other hand, some existing virtual simulation systems attempt to indirectly infer the user's knowledge level by recording operation trajectories and analyzing the temporal patterns of operation sequences. However, these methods face two core problems: First, operation trajectories are high-dimensional sparse data, lacking a direct causal relationship with the internal state of the physical system (such as stress fields, temperature fields, flow velocity fields, and other physical properties that cannot be directly observed), making it difficult to establish a quantitative mapping relationship between operational behavior and physical cognition. Second, traditional physical field visualization methods (such as cloud maps and vector maps) usually present physical data in the form of independent windows or overlay layers, separating it from the three-dimensional visual model of the experimental object. This results in users not being able to obtain intuitive spatial references while observing physical field information, which is not conducive to intuitive understanding and analysis of physical phenomena. Therefore, how to integrate the user's operational behavior in virtual simulation experiments, the evolution process of the physical field inside the digital twin, and the final visual presentation at the data level, and extract quantitative features that can effectively characterize the user's physical cognitive level, is a technical problem that urgently needs to be solved in the field of virtual simulation experiment technology. Summary of the Invention

[0003] This invention provides a virtual simulation experiment image data processing method based on digital twins. The method aims to generate physical field state data by injecting user operation semantics into physical simulation, mapping unobservable physical properties into causal textures and fusing them with actual visual images, and finally performing spatiotemporal feature analysis on the fused image sequence to quantitatively evaluate the level of physical cognition implied by user operations.

[0004] This invention provides a method for processing virtual simulation experimental image data based on digital twins, comprising: S1, acquire the user's operation input data during the virtual simulation experiment, and extract the operation semantic features; S2, the operational semantic features are used as time-varying driving conditions and injected into the physical simulation model of the digital twin to solve the physical field state data sequence generated under the continuous superposition and attenuation of operational semantics. S3, Based on the physical field state data sequence, generate an image sequence containing causal textures, wherein the causal textures are obtained by mapping physical properties in the physical field that cannot be directly observed into visual textures; S4, Obtain the actual visual image sequence of the virtual simulation experiment scene within the same spatiotemporal range; S5, the image sequence containing causal texture is spatiotemporally aligned and fused with the actual visual image sequence to generate a virtual-real fused image sequence; S6, perform spatiotemporal feature analysis on the virtual-real fusion image sequence to generate an operational skill assessment result that reflects the user's physical cognitive level.

[0005] Compared with the prior art, the beneficial effects of this application are as follows: 1. This invention directly injects the semantic features of user operations into the physical simulation model of the digital twin, solves the resulting physical field state data, and then maps unobservable physical properties into visual image information through causal textures, establishing a complete causal link from user operation to physical system response to visual feedback, making the subsequent evaluation process clearly interpretable.

[0006] 2. This invention performs spatiotemporal feature analysis on virtual-real fusion image sequences, calculates the physical field evolution entropy and operation-physics causal correlation measure, and quantitatively evaluates the physical cognition level implied by user operations from two independent dimensions: the stability of the physical system evolution and the effectiveness of user operations. This overcomes the shortcomings of traditional result-oriented evaluations that cannot measure deep understanding.

[0007] 3. The evaluation results generated by this invention can be directly fed back to the virtual simulation experiment system, driving adaptive adjustments to the experiment difficulty, visual guidance, or constraints, thereby achieving personalized teaching guidance and improving the intelligence and teaching effectiveness of virtual simulation experiments.

[0008] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0009] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention, but do not constitute a limitation thereof; in the drawings: Figure 1This is a flowchart illustrating a virtual simulation experiment image data processing method based on digital twins provided by the present invention. Figure 2 This is a schematic diagram of the process for generating a virtual-real fusion image sequence according to the present invention; Figure 3 This is a flowchart illustrating the process of generating operational skill assessment results that reflect the user's physical cognitive level in their operations. Detailed Implementation

[0010] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. Example 1:

[0011] To enable those skilled in the art to better understand the present application, the technical solution of the present application will be clearly and completely described below in conjunction with the specific application scenario of the virtual bridge simulation experiment in the embodiments of the present application; it should be understood that the embodiments are merely exemplary, and the protection scope of the present application is not limited thereto. This invention provides a method for processing image data in a virtual simulation experiment based on digital twins. The method can be executed by a computer system integrating a digital twin simulation engine, an image rendering engine, and a data analysis module. The experimental scenario is set as follows: in a virtual environment, a user performs operations such as applying loads, modifying the structure, or adjusting supports on a virtual bridge through interactive devices to test and evaluate their understanding of the bridge's mechanical behavior. Please refer to [link to relevant documentation]. Figure 1 This includes the following steps: S1, acquire the user's operation input data during the virtual simulation experiment, and extract the operation semantic features; S2, using operational semantic features as time-varying driving conditions, injects them into the physical simulation model of the digital twin to solve the physical field state data sequence generated under the continuous superposition and attenuation of operational semantics; S3, Generate an image sequence containing causal textures based on the physical field state data sequence. Causal textures are obtained by mapping physical properties in the physical field that cannot be directly observed to visual textures. S4, Obtain the actual visual image sequence of the virtual simulation experiment scene within the same spatiotemporal range; S5, spatiotemporally align and fuse the image sequence containing causal texture with the actual visual image sequence to generate a virtual-real fused image sequence; S6 performs spatiotemporal feature analysis on the virtual-real fusion image sequence to generate operational skill assessment results that reflect the user's physical cognitive level in their operation.

[0012] Specifically, this embodiment achieves end-to-end processing from user physical operation to physical cognitive level assessment through six stages: Stage S1 is responsible for translating the user's operation into semantic features with physical meaning; Stage S2 calculates the continuous evolution data of physical fields (such as stress fields) by performing dynamic simulations that conform to real physical laws within the digital twin; Stage S3 is used to transform invisible physical field data into visualized causal texture images; Stage S4 is used to simultaneously acquire conventional realistic visual images; Stage S5 is used to perform spatiotemporal alignment and fusion of the above two types of images to generate a virtual-real fusion image that simultaneously contains appearance and internal physical information; Stage S6 is used to perform spatiotemporal feature quantification analysis on the virtual-real fusion image, and finally output an assessment score that reflects the depth of physical cognitive understanding behind the user's operation; the above method transforms the complex problem of physical simulation and cognitive assessment into a computable image data processing problem.

[0013] In one implementation, the process of acquiring user input data during a virtual simulation experiment and extracting semantic features of the operations includes: Real-time acquisition of user operation command stream on multimodal interaction devices. The operation command stream includes at least the operation type, target, location, direction, force, and timestamp. The operation instruction stream is organized into an operation sequence according to the time sequence, and the operation sequence is parsed into an operation semantic feature vector sequence based on a preset physical operation logic rule base. Each feature vector in the sequence of operation semantic feature vectors contains the type of physical quantity that the current operation intends to change, the direction and magnitude of the intended change, and the spatial boundary conditions of the action.

[0014] Specifically, the system collects user interaction data in real time through a data interface. In this embodiment, the multimodal interaction devices are a mouse and a keyboard. When the user operates in the virtual bridge experimental scenario, the system continuously listens for and records mouse and keyboard events, generating a structured operation command stream. Each operation command is a data packet, containing at least the following fields: The operation type field is used to identify the type of action in this operation, such as "click to select", "drag to apply force", "scroll wheel adjustment", "button to modify attributes", etc. Taking "drag to apply force" as an example, it records the process of the user holding down the left mouse button and dragging on the bridge surface model. The target field identifies the target component or region in the digital twin to which the operation is applied. The system emits a ray from the screen coordinates where the mouse cursor is located and shoots into the 3D virtual scene, calculating which object's geometry the ray intersects with first. For example, when the cursor hovers over the central area of ​​the bridge deck, the ray detection returns the collision body as "bridge deck panel", and the system records the target as "bridge deck panel" accordingly. The application location field records the coordinates of the point where the operation is applied in three-dimensional space. These coordinates are the three-dimensional world coordinates (x, y, z) of the intersection of the ray and the bridge surface model. For drag-and-drop operations, this field records the position of the drag start point. The direction of action field is used to record the direction vector of the operating force or displacement. In the mouse dragging scenario, this direction is determined by the direction of the mouse movement on the screen during the dragging process, combined with the current virtual camera's perspective, and projected into the three-dimensional space. For example, when the user drags the mouse downwards, the system maps it to apply a vertically downward direction vector (0,-1,0) to the bridge surface. The force field is used to record the magnitude or displacement of the operating force. In mouse drag scenarios, the force is calculated from the displacement distance of the mouse drag, for example, by multiplying the number of screen pixels dragged by a preset ratio coefficient and converting it into a force value in Newtons (N), such as 80N. For scroll wheel adjustment operations, the force is calculated from the number of scroll wheel increments. The timestamp field is used to record the time when the operation instruction occurred; This series of consecutive operation instructions, arranged in chronological order of their timestamps, constitutes the operation sequence. Parsing the operation sequence into a sequence of operation semantic feature vectors relies on a pre-defined physical operation logic rule base. This rule base is a mapping table that stores the correspondence between user interface operation actions and physical intentions. For example, the rule base pre-defined the following mapping rules: "Object of Action = Bridge Deck Panel, Operation Type = Drag Force Application, Direction of Action = Vertical Downward" is mapped to "Physical Quantity Type to be Changed = Vertical Concentrated Load; Direction to be Changed = Increase; Amplitude = Determined by the Force Field; Spatial Boundary Condition = Circular Area with Preset Radius Centered on the Application Location"; Similarly, "Object of Action = Pier Support Point, Operation Type = Modify Attributes with Key, Pressed Key is F" is mapped to "Physical Quantity Type to be Changed = Displacement Constraint; Direction to be Changed = Apply (from Free to Fixed); Spatial Boundary Condition = Node at the Application Location"; Based on this rule base, the system traverses each instruction in the operation sequence, searches for matching mapping rules, and generates the corresponding operation semantic feature vector. This vector clearly carries the physical purpose of the operation, that is, in which area of ​​the bridge surface (spatial boundary conditions), in what manner (physical quantity type), and by how much physical action (direction and magnitude) is being added or reduced. This vectorized representation enables the computer to understand the physical goal that the user intends to achieve through mouse and keyboard operations.

[0015] Of course, the mapping rules will differ when applied to different virtual simulation scenarios. The mapping rules are preset and are not limited in this embodiment.

[0016] In one implementation, the process of solving the physical field state data sequence generated under the continuous superposition and attenuation of operational semantics includes: The sequence of operational semantic feature vectors is discretized into a series of time-step driving signals, which are then applied to the boundary conditions or source terms of the physical simulation model of the digital twin; the physical simulation model is a system of partial differential equations based on the finite element method or the finite volume method. At each simulation time step, a decay function is first applied to the physical field state data of the previous time step in the entire spatial domain to simulate the natural regression effect of the physical process without operational intervention, and then the driving signal of the current time step is applied. A numerical solver is used to perform transient time-domain solutions, obtaining a snapshot of the physical field state data on the entire physical space grid node at each time step. The snapshots of all time steps constitute a sequence of physical field state data.

[0017] Specifically, a digital twin is a digital copy of a virtual bridge and its physical environment. At its core is a physical simulation model built based on the finite element method (FEM). This model discretizes the bridge geometry into thousands of mesh elements (such as tetrahedrons or hexahedrons) and establishes a set of partial differential equations (such as the Navier-Cauchy equations) describing the elastic mechanical behavior on all mesh nodes. First, the system discretizes the continuous sequence of semantic feature vectors, i.e., with a fixed simulation step size. (like The semantic features are sampled at 0.01s to generate a driving signal for each time step. If the user operation is "add 50N of pressure at a certain point on the bridge deck", the driving signal is to apply a nodal force with a total of 50N and a downward direction to the corresponding multiple mesh nodes. When the user intention is "change the material properties", the driving signal is to modify the material stiffness matrix of the target element, which is a source term acting on the partial differential equation system. When the user intention is "fix a certain place", the driving signal is to set the displacement of the target node to zero, which is a boundary condition acting on the equation system. In short, operations such as applying force or heat sources are usually treated as source terms, while operations such as applying displacement constraints are treated as boundary conditions. At each time step The simulation calculation process is as follows: 1. Applying a decay function: The system first obtains the previous time step. Distribution of physical fields (such as stress fields) The attenuation function is then applied and allowed to decay naturally. In the mechanical scenario of this embodiment, the attenuation function is used to simulate stress relaxation or vibration damping of the material. A simple attenuation model is as follows: ,in The damping coefficient is determined based on the parameters of the bridge material; if the scenario is a heat conduction experiment, the attenuation function is derived from the heat diffusion equation and is in an exponential decay form; different application scenarios require different specific forms of the attenuation function (linear, exponential, or complex models based on physical mechanisms) and parameters (such as...). Although they are different, they have the same function: to simulate the tendency of a physical system to revert to an equilibrium state without human intervention. 2. Applying a driving signal: In the decayed state Apply the driving signal (nodal force, displacement constraint, etc.) obtained from the above discretization to the current time step. 3. Numerical Solution: Call a numerical solver (such as Intel MKL PARDISO, PETSc, or other sparse linear equations solution libraries) to perform transient solution on the partial differential equations with the new load conditions, and calculate the stress, strain, and displacement values ​​on all grid nodes of the entire bridge at this moment; this complete set of physical quantity values ​​distributed on all spatial grid nodes is a snapshot of the physical field state data; save the snapshots of all time steps in sequence to form the physical field state data sequence.

[0018] In one implementation, the process of generating an image sequence containing causal textures based on a sequence of physical field state data includes: Establish causal mapping rules between physical properties that cannot be directly observed in a physical field and visible texture properties. The physical properties that cannot be directly observed include at least one of stress field, strain field or temperature gradient field, and the visible texture properties include at least one of color, brightness or texture direction. Based on the causal mapping rule, a shader program is constructed to render each frame of data snapshot in the physical field state data sequence onto the surface of the three-dimensional geometric model of the digital twin, generating an image sequence containing causal textures; The causal mapping rule is as follows: the scalar value of the physical attribute is mapped to the numerical value or color saturation on the color table, and the direction information of its field is mapped to the directional texture direction of the texture pattern, so that each pixel in the causal texture image has a reversible mapping relationship with its corresponding physical field spatial position and intensity.

[0019] Specifically, this embodiment is used to visualize physical data. First, a causal mapping rule is established in advance. In this embodiment, for the stress field of the bridge, the rule is defined as follows: the scalar value of the Mises equivalent stress, from zero to the material yield strength, is linearly mapped to a preset blue-cyan-green-yellow-red gradient color table; the area with zero stress corresponds to dark blue, and the area with stress close to the yield limit corresponds to bright red; at the same time, the direction of the principal stress is mapped to the direction angle of the fine texture lines superimposed on the bridge surface; for example, when the direction of the principal tensile stress at a certain point is horizontal to the right, the texture lines at that point also show a horizontal direction. In implementation, the system builds a shader program running on the GPU. When generating each frame of causal texture image, the system passes a snapshot of the physical field data at the current time step (stored as a texture buffer or vertex attribute array) to the shader. For each pixel to be rendered on the surface of the bridge 3D model, the shader queries the stress scalar value according to its corresponding mesh node, calculates its color through a color lookup table (LUT), and calculates the rotation amount of the texture coordinates according to the principal stress direction vector, thereby drawing a dynamic texture on the model surface in real time that reflects the magnitude and direction of the physical field. The resulting image frame is a causal texture image. Since the color and texture direction of each pixel are uniquely determined by the physical field of that spatial point, the physical field data can be uniquely deduced from the image, thus forming a reversible mapping relationship. For example, a red area with horizontal texture on the image must correspond to a physical location with high stress and horizontal principal stress direction.

[0020] In one implementation, the process of obtaining the actual visual image sequence of the virtual simulation experiment scene within the same spatiotemporal range includes: Using at least one virtual camera, under the exact same spatiotemporal conditions as when generating causal texture images, the three-dimensional visual model of the digital twin is rendered in real time to generate an actual visual image sequence. The virtual camera's intrinsic parameters, extrinsic parameters, and frame rate are kept synchronized with the rendering parameters used to generate image sequences containing causal textures.

[0021] Specifically, this step is used to generate conventional visual images. Mature real-time rendering technologies in computer graphics (such as Unity) can be used. The system is configured with a virtual camera. The intrinsic parameters (field of view, focal length, resolution such as 1920×1080), extrinsic parameters (position and orientation in world coordinates), and frame rate (such as 60fps) of this camera must be exactly the same as the camera parameters used in step S3 to generate the causal texture image. The rendering process in steps S3 and S4 can be two parallel channels in the same rendering pipeline. One channel uses a physics-driven shader to generate a causal texture map, while the other channel uses a standard PBR (physically based rendering) shader to render a regular visual image with realistic appearance, such as metallic materials, ambient lighting, and shadows. In this way, it is ensured that the perspective and spatiotemporal range of the two image sequences are pixel-level aligned in each frame. For example, if a camera shoots from a 45-degree angle below the side of the bridge, both the causal texture map and the actual visual map record the same part of the bridge from the same perspective.

[0022] In one implementation, please refer to Figure 2The process of spatiotemporally aligning and fusing image sequences containing causal textures with actual visual image sequences to generate a virtual-real fused image sequence includes: Using the three-dimensional geometric model of the digital twin as a common spatial reference benchmark, each frame of the image sequence containing causal texture is spatially registered pixel by pixel with the corresponding frame with the same timestamp in the actual visual image sequence. The two registered images are blended according to a preset blending transparency coefficient to generate a single image that blends reality and virtuality. The virtual-real fusion images at all time steps are arranged in temporal order to form the initial virtual-real fusion image sequence; The initial virtual-real fusion image sequence is submitted to the manual evaluation interface, where the evaluators visually inspect the fusion alignment quality of each frame or key frame to confirm whether the causal texture coincides with the corresponding object edges and feature points in the actual visual image. If the evaluators determine that the alignment quality is unacceptable, the spatial registration parameters of the 3D geometric model or the parameters of the virtual camera will be adjusted and the images will be re-fused; if the evaluators determine that the alignment quality is acceptable, the frame or sequence will be recognized as a virtual-real fusion image sequence that can be used for subsequent analysis.

[0023] Specifically, since the two types of images generated in steps S3 and S4 are both based on the same three-dimensional geometric model of the digital twin and are rendered under the same virtual camera parameters, the three-dimensional geometric model itself is a natural and accurate common spatial reference benchmark; in the causal texture image frame and the actual visual image frame at the same time stamp, the pixel with coordinates (u,v) corresponds to the same spatial point on the surface of the three-dimensional model. Spatial registration is implicitly completed during the rendering process, without the need for additional image registration algorithms (such as SIFT feature matching or optical flow alignment). The fusion process uses alpha blending. For each frame pair, the color value of pixel coordinate (i,j) in the causal texture image is set to... In actual visual images, the pixel color values ​​at the same coordinates are... The resulting pixel color value Calculated using the following formula: ;in, The preset blending transparency coefficient ranges from 0 to 1; in the virtual bridge experiment of this embodiment, A value of 0.4 makes the actual visual appearance dominate the fused image, and the structural details of the bridge, such as the metal material, rivets, and welds, are clearly visible. Meanwhile, the causal texture representing stress distribution is superimposed on the bridge surface in a semi-transparent form, as if an augmented reality information coating is applied to a real object. The above Alpha blending operation is performed on frame pairs at all time steps one by one. All the fused frames are arranged in order of timestamp to form the initial virtual-real fused image sequence. After the initial virtual-real fusion image sequence is generated, the system submits it to the manual evaluation interface. This interface is an interactive image inspection tool running on a display device, which can simultaneously display side-by-side comparison views of the fused image, causal texture image, and actual visual image, and provides interactive functions such as zooming, panning, and frame switching. Evaluators select key frames frame by frame or at preset intervals, focusing on checking locations with significant geometric features in the bridge structure, such as the joints between the bridge deck and piers, the lower edge of the main beam, the area around rivet holes, and support nodes. At these locations, evaluators visually confirm whether the colored stress distribution areas in the causal texture precisely coincide with the corresponding structural edges in the actual visual image. For example, if the red texture area representing high stress in a frame is visibly offset from the geometric outline of the lower edge of the main beam in the actual visual image, the alignment quality of that frame is deemed unqualified. If the number of unqualified frames is too large or the alignment deviation of a key frame exceeds the allowable range, the evaluator can adjust the spatial registration parameters of the 3D geometric model (such as the small translation or rotation of the model in the world coordinate system) or the parameters of the virtual camera (such as the small correction values ​​of the position coordinates and orientation angle in the extrinsic parameters) through the parameter adjustment controls on the interface. The system will then re-execute the fusion operation based on the adjusted parameters. The adjustment and re-fusion process can be repeated until the judges confirm that the alignment quality of all frames or keyframes is up to standard; the image frame sequence that has been manually judged and confirmed to be up to standard is the final virtual-real fusion image sequence used for subsequent spatiotemporal feature analysis. The role of this human evaluation step is to reliably check the key quality indicator of spatial alignment accuracy through the human visual system before entering the computationally intensive spatiotemporal feature analysis, so as to avoid the invalidation of subsequent physical field reconstruction and skill evaluation results due to alignment deviation.

[0024] In one implementation, please refer to Figure 3 The process of performing spatiotemporal feature analysis on virtual-real fusion image sequences includes: From the causal texture components of the virtual-real fusion image sequence, the reversible mapping relationship between pixels in the causal texture image and the spatial position and intensity of the physical field is used to extract the initial values ​​of physical attributes at the pixel level frame by frame, resulting in a sequence of initial value maps of physical attributes; wherein, the sequence of initial value maps of physical attributes is the image reconstruction version of the physical field state data sequence generated in step S2. From the actual visual components of the virtual-real fusion image sequence, extract the visual detail feature map sequence that reflects the surface geometric and material features of the digital twin frame by frame; the visual detail features include at least one of the following: texture complexity features, surface normal variation features, and curvature features; The sequence of initial physical property maps and the sequence of visual detail feature maps are fused pixel by pixel. The visual detail features are used as local correction factors to enhance or suppress the initial physical property values, and the physical field state data sequence after visual geometric detail correction is reconstructed. Based on the corrected physical field state data sequence, the physical field evolution entropy and the operation-physical causal correlation measure are calculated respectively. Together, they constitute the two-dimensional quantitative results of spatiotemporal feature analysis. Among them, the physical field evolution entropy is used to quantify the degree of disorder and instability of the internal state evolution of the physical system during user operation. It consists of two terms: the first term is the information entropy calculated after normalizing the probability density of the physical field intensity over the entire spatial domain, which is used to quantify the degree of disorder in the spatial distribution of the physical field; the second term is the integral of the square of the physical field state space gradient over the entire spatial domain, which is used to quantify the degree of drastic change in the physical field space. The operation-physical causal correlation metric is calculated as follows: within the evaluation time window, the time change rate of the actual physical field state driven by user operation at each moment is subtracted from and integrated point-by-point across the entire spatial domain by the time change rate of the naturally decaying physical field state driven only by the decay function without user intervention, to obtain the instantaneous value of the net causal influence of user operation at that moment; the instantaneous values ​​of the net causal influence at all moments within the evaluation time window are accumulated over time to obtain the operation-physical causal correlation metric, which is used to quantify the cumulative impact of user operation on the evolution path of the physical system.

[0025] Specifically, the causal texture component is first separated from the virtual-real fused image sequence, due to the fusion formula. All parameters are known quantities, and the color value of the causal texture component at each pixel can be obtained by inverse algebraic calculation: After obtaining the causal texture components, the initial values ​​of physical properties are extracted using the reversible mapping relationship established in step S3. The reversible mapping relationship is as follows: the "stress scalar value → color mapping table" defined in step S3 is a monotonic and one-to-one mapping function. Therefore, the stress scalar value of the corresponding spatial position of any pixel in the causal texture image can be uniquely determined by looking up the color mapping table in reverse. Similarly, the principal stress direction vector at the position can be deduced based on the direction angle of the texture lines. The above reverse mapping operation is performed on each pixel of each frame of causal texture component, and the stress scalar value and direction vector obtained constitute the initial physical property map of the frame. The initial physical property maps of all time steps are arranged in order to form a sequence of initial physical property maps. It should be noted that the difference between this sequence of initial physical property values ​​and the original physical field state data sequence output in step S2 is that they are in different coordinate systems. The original data is in the simulation grid space and cannot be aligned with the pixels of visual detail features. If the original physical field state data sequence output in step S2 is used directly, it is impossible to achieve pixel-level precise alignment with the visual detail features. The two are not in the same coordinate system. The reconstruction from the virtual-real fusion image is to put the physical field data and the visual detail features in the same pixel coordinate system to achieve subsequent pixel-by-pixel correction and fusion. Simultaneously, visual detail feature map sequences are extracted from the actual visual components of the virtual-real fusion image sequence; the actual visual components are obtained through... The images are separated; for each actual visual image frame, the following method is used to extract visual detail features: For texture complexity features, the actual visual image is converted into a grayscale image. For each pixel, an M×M neighborhood window centered on it is taken (e.g., M=11 pixels). The contrast statistics of the gray-level co-occurrence matrix in this neighborhood are calculated. The pixel area with the higher contrast value indicates that the texture of the material surface is richer. In the bridge scene, it often corresponds to weld seams, rivet arrangement areas, surface rust, or uneven areas. For the surface normal variation characteristics, the surface normal direction vector at each pixel is estimated by using the distribution information of illumination and shadow in the actual visual image through the single image normal estimation method. Then, the angular deviation between the normal direction at each pixel and the mean normal direction of its surrounding pixels is calculated. The larger the deviation, the more drastic the local geometric curvature change, such as the corner where the bridge deck meets the pier. For curvature features, they are directly obtained from the three-dimensional geometric model data of the digital twin. The average curvature or Gaussian curvature at each vertex of the model surface is calculated. Then, through the texture mapping relationship between the model vertex and the image pixel, the curvature value is projected onto the image space to obtain the curvature feature map. The pixel area with the larger curvature value corresponds to the part with the sharper geometric shape, such as the support edge, the periphery of the opening and other areas prone to stress concentration. After normalizing the texture complexity feature map, surface normal variation feature map, and curvature feature map, they are weighted and summed according to preset weights to obtain the comprehensive visual detail feature map. , where x represents the spatial location of the pixel and t represents the time step; The initial physical property map sequence and the visual detail feature map sequence are fused pixel by pixel, with the visual detail features used as local correction factors to enhance or suppress the initial physical property values; the calculation formula for the correction fusion is as follows: ;in, Let x be the initial value of the physical property extracted from the causal texture component at time t and spatial location x. The intensity of the combined visual details at the same spatiotemporal location has been normalized to the [0,1] interval; The preset correction intensity coefficient is used to control the magnitude of the correction of visual details to the physical field. In this embodiment... The value is 0.3; This formula indicates that in areas with drastic geometric changes and complex textures on the material surface (such as weld roots, rivet hole edges, and abrupt changes in cross-section), due to geometric discontinuities, the actual stress concentration is often higher than the value obtained by the idealized finite element mesh. Therefore, visual detail intensity should be used as an enhancement factor to locally amplify the initial stress value extracted from causal texture. Conversely, in areas with smooth surfaces and uniform textures, geometric continuity is better, stress distribution tends to be uniform, and visual detail intensity is close to zero, so the correction factor... When the value is close to 1, the initial value of the physical property remains basically unchanged; After this correction step, the physical field state data sequence not only retains the macroscopic stress distribution information solved by the physical simulation model in step S2, but also incorporates the local correction of stress distribution by the microscopic geometric and material details reflected in the actual visual image sequence, thereby achieving effective enhancement of the original simulation data.

[0026] Based on the corrected physical field state data sequence The physical field evolution entropy and the operation-physics causal correlation measure are calculated separately, and the two together constitute the two-dimensional quantitative results of spatiotemporal feature analysis. The physical field evolution entropy is used to quantify the degree of disorder and instability in the internal state evolution of a physical system during user operations. Its calculation formula is as follows:

[0027] in, The three-dimensional spatial domain occupied by the bridge structure; The corrected stress scalar value; for Throughout the entire space domain The probability density function on, through the The stress values ​​of all grid nodes within the range of values ​​are statistically analyzed using a histogram (for example, the stress value range is divided into 100 equal intervals, and the proportion of nodes in each interval is counted), and then the histogram is normalized to obtain the result. Let be the spatial gradient vector of the stress field. The finite difference method is used to calculate the first-order differences of the stress value at each node along the x, y, and z directions on a three-dimensional mesh, forming the three components of the gradient vector. The norm of these components is... Take the square root of the sum of the squares of the three components; The first term in the formula is the information entropy term, which measures the uniformity of stress distribution across the entire bridge structure. If high-stress and low-stress areas are randomly and interspersed, the probability density function is relatively dispersed, and the information entropy value is large. If the stress distribution presents a clear and orderly pattern (such as high stress concentrated near the supports and low stress distributed in the middle of the span), the entropy value is small. The second term in the formula is the gradient penalty term, which is used to integrate the square of the magnitude of the stress field spatial gradient over the entire domain. If there is severe stress concentration (such as the stress value at a certain node being much higher than that at its neighboring nodes), the gradient norm is large and the value of this term is high; if the stress change is gradual and continuous, the value of this term is low; 1 / 2 is used to cancel the coefficient 2 generated by the square after differentiation, simplifying the gradient calculation. The sum of the two items It comprehensively reflects the degree of disorder and unevenness of the internal state of the physical system at a certain moment. The lower the value of this index, the more stable and orderly the mechanical state of the bridge at the current moment. The operation-physical causality metric is used to quantify the cumulative impact of user operations on the evolution path of a physical system. Its calculation method is as follows: Within the evaluation time window, the time-varying rate of change of the actual physical field state driven by user operation at each moment is subtracted from and integrated point-by-point across the entire spatial domain by the time-varying rate of change of the naturally decaying physical field state driven only by the decay function without user intervention. This yields the instantaneous value of the net causal influence of the user operation at that moment. The instantaneous values ​​of the net causal influence at all moments within the evaluation time window are accumulated over time to obtain the operation-physical causal correlation metric. Its calculation formula is as follows:

[0028] in, To evaluate the time window, the total duration of the entire experimental process is usually taken; The rate of change of the corrected actual physical field state at time t is calculated using the difference approximation between the preceding and following time steps. The time rate of change of the state of the naturally decaying physical field; Natural decay physical field The method of obtaining it is: In the physical simulation model of step S3, all the driving signals of user operation are set to zero, and only the effect of the decay function is retained. The offline simulation is run once with the same initial conditions, time step Δt and total experimental duration as the main simulation to obtain the physical field state sequence driven only by decay. The same visual detail correction process as the main simulation is also performed on this sequence to obtain the corrected natural decay physical field state sequence, and then its time change rate is calculated. In the formula, the inner spatial integral calculates the integral of the absolute value of the difference between two time rates of change at the same moment over the entire spatial domain. This integral value represents the instantaneous value of the net causal influence of the user's operation on the evolution path of the physical system at the current moment; the outer time integral accumulates this instantaneous value over the entire evaluation time window. The larger the value, the more significant the change in the user's actions, indicating that each action significantly alters the natural evolutionary trajectory of the physical system, demonstrating a clear purpose and effectiveness. Conversely, if the user's actions have almost no impact on the system's evolution, or if the effect of the actions is no different from natural decay, then... Approaching zero; Physical field evolution entropy From the perspective of the stationarity of the evolutionary process and the measurement of operational-physical causality From the perspective of the effectiveness of operational intervention, the two are independent yet complementary, together constituting a two-dimensional quantitative result of the physical cognition level implied in the user's operation; a user with a high level of physical cognition should make the evolution of the physical system both stable and orderly (low evolutionary entropy) and goal-oriented and effective (high causal correlation measure).

[0029] In one implementation, the process of generating operational skill assessment results that reflect the level of physical cognition inherent in a user's actions includes: The cumulative physical field evolution entropy of the entire experimental process is mapped by a negative exponential function and multiplied by the first preset weight coefficient to obtain the first evaluation component. The final accumulated operation-physical causal correlation metric is divided by the standard deviation of the correlation metric during the experiment, and then multiplied by the second preset weighting coefficient to obtain the second evaluation component; The first assessment component and the second assessment component are summed to obtain the comprehensive score of physical cognitive skills, which is used as the result of operational skills assessment.

[0030] Specifically, the formula for calculating the comprehensive score of physical cognitive skills is as follows:

[0031] in, The total duration of the experiment, which is the time elapsed from when the user starts operating until the end of the experiment; To calculate the cumulative physical field evolution entropy throughout the entire experiment, a numerical integration method is used in the discrete implementation for each time step. Approximate the summation of values; divide by The average physical field evolution entropy was then obtained; (-Average Evolutionary Entropy) is a negative exponential function mapping, mapping the average entropy value to the interval (0,1]. When the average entropy value approaches zero, this term approaches 1, representing an extremely stable operation. When the average entropy value increases, this term decays exponentially towards 0, representing a chaotic and disordered operation. The reason for using a negative exponential function instead of a nonlinear mapping is that improvements in the average entropy value at lower levels have a significant impact on the score, while further deterioration at already high levels has a diminishing marginal impact on the score. The first preset weighting coefficient is used to adjust the relative importance of operational stability in the final score; To evaluate the cumulative operational-physical causality measure at the end of the time window (i.e., the end time T of the experiment), for The standard deviation of the sequence over time during the experiment; divide C(T) by The constant 1 in the denominator is used to prevent division by zero errors when σ(C) approaches zero. It also plays a role in adjusting the intensity of the penalty, that is, to impose a penalty on situations where the impact of the operation fluctuates greatly. This is because a high level of physical cognition not only requires the overall effectiveness of the operation, but also requires the operation strategy to have consistency and stability over time. β is the second preset weighting coefficient, which is used to adjust the relative importance of the effectiveness of the operation in the final score. The values ​​of the first preset weight coefficient α and the second preset weight coefficient β are preset by the system administrator according to the emphasis of the specific experimental teaching objectives. First assessment component Second evaluation component After adding them together, we get the final comprehensive score for physical cognitive skills. This score comprehensively reflects the user's operational stability and effectiveness during the virtual simulation experiment, that is, the level of physical cognition implied in the user's operation. The higher the score, the deeper the user's understanding of the physical principles involved in the experiment and the more mature the operation strategy.

[0032] In one implementation, the operational skill assessment results are fed back to the virtual simulation experiment system to drive adaptive adjustments to the experimental scenario. The adaptive adjustments include at least one of adjusting the experimental difficulty, providing visual guidance cues, or changing the experimental constraints.

[0033] Specifically, this process is a manual operation, and will not be described further in this embodiment.

[0034] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing virtual simulation experimental image data based on digital twins, characterized in that, Includes the following steps: S1, acquire the user's operation input data during the virtual simulation experiment, and extract the operation semantic features; S2, the operational semantic features are used as time-varying driving conditions and injected into the physical simulation model of the digital twin to solve the physical field state data sequence generated under the continuous superposition and attenuation of operational semantics. S3, Based on the physical field state data sequence, generate an image sequence containing causal textures, wherein the causal textures are obtained by mapping physical properties in the physical field that cannot be directly observed into visual textures; S4, Obtain the actual visual image sequence of the virtual simulation experiment scene within the same spatiotemporal range; S5, the image sequence containing causal texture is spatiotemporally aligned and fused with the actual visual image sequence to generate a virtual-real fused image sequence; S6, perform spatiotemporal feature analysis on the virtual-real fusion image sequence to generate an operational skill assessment result that reflects the user's physical cognitive level.

2. The method according to claim 1, characterized in that, The process of acquiring user input data during virtual simulation experiments and extracting semantic features of the operations includes: Real-time acquisition of user operation command stream on multimodal interaction devices, the operation command stream including at least operation type, target, location, direction, force and timestamp; The operation instruction stream is organized into an operation sequence according to the time sequence, and the operation sequence is parsed into an operation semantic feature vector sequence based on a preset physical operation logic rule base. Each feature vector in the sequence of operation semantic feature vectors includes the type of physical quantity that the current operation intends to change, the direction and magnitude of the intended change, and the spatial boundary conditions of the action.

3. The method according to claim 1, characterized in that, The process of generating the physical field state data sequence under the continuous superposition and attenuation of the solution operation semantics includes: The sequence of operational semantic feature vectors is discretized into a series of time-step driving signals, which are then applied to the boundary conditions or source terms of the physical simulation model of the digital twin; the physical simulation model is a system of partial differential equations based on the finite element method or the finite volume method. At each simulation time step, a decay function is first applied to the physical field state data of the previous time step in the entire spatial domain to simulate the natural regression effect of the physical process without operational intervention, and then the driving signal of the current time step is applied. A numerical solver is used to perform transient time-domain solutions, obtaining a snapshot of the physical field state data on the entire physical space grid node at each time step. The snapshots of all time steps constitute the physical field state data sequence.

4. The method according to claim 3, characterized in that, The process of generating an image sequence containing causal texture based on the physical field state data sequence includes: Establish a causal mapping rule between physical properties that cannot be directly observed in the physical field and visible texture properties. The physical properties that cannot be directly observed include at least one of stress field, strain field or temperature gradient field, and the visible texture properties include at least one of color, brightness or texture direction. Based on the causal mapping rule, a shader program is constructed to render each frame of data snapshot in the physical field state data sequence onto the surface of the three-dimensional geometric model of the digital twin, generating an image sequence containing causal textures; The causal mapping rule is as follows: the scalar value of the physical attribute is mapped to the numerical value or color saturation on the color table, and the direction information of its field is mapped to the directional texture direction of the texture pattern, so that each pixel in the causal texture image has a reversible mapping relationship with its corresponding physical field spatial position and intensity.

5. The method according to claim 4, characterized in that, The process of obtaining the actual visual image sequence of the virtual simulation experimental scene within the same spatiotemporal range includes: Using at least one virtual camera, under the exact same spatiotemporal conditions as when generating causal texture images, the three-dimensional visual model of the digital twin is rendered in real time to generate an actual visual image sequence. The intrinsic parameters, extrinsic parameters, and frame rate of the virtual camera are kept synchronized with the rendering parameters used when generating the image sequence containing causal textures.

6. The method according to claim 5, characterized in that, The process of spatiotemporally aligning and fusing the image sequence containing causal texture with the actual visual image sequence to generate a virtual-real fused image sequence includes: Using the three-dimensional geometric model of the digital twin as a common spatial reference benchmark, each frame of the image sequence containing causal texture is spatially registered pixel by pixel with the corresponding frame with the same timestamp in the actual visual image sequence. The two registered images are blended according to a preset blending transparency coefficient to generate a single image that blends reality and virtuality. The virtual-real fusion images at all time steps are arranged in temporal order to form the initial virtual-real fusion image sequence; The initial virtual-real fusion image sequence is submitted to the manual evaluation interface, where the evaluators visually inspect the fusion alignment quality of each frame or key frame to confirm whether the causal texture coincides with the corresponding object edges and feature points in the actual visual image. If the evaluator determines that the alignment quality is unqualified, the spatial registration parameters of the three-dimensional geometric model or the parameters of the virtual camera are adjusted and then re-fused; if the evaluator determines that the alignment quality is qualified, the frame or the sequence is confirmed as the virtual-real fused image sequence that can be used for subsequent analysis.

7. The method according to claim 6, characterized in that, The process of performing spatiotemporal feature analysis on the virtual-real fused image sequence includes: From the causal texture components of the virtual-real fusion image sequence, the reversible mapping relationship between pixels in the causal texture image and the spatial position and intensity of the physical field is used to extract the initial values ​​of physical attributes frame by frame, thereby obtaining a sequence of initial physical attribute maps; wherein, the sequence of initial physical attribute maps is the image reconstruction version of the physical field state data sequence generated in step S2. From the actual visual components of the virtual-real fusion image sequence, a sequence of visual detail feature maps reflecting the surface geometric and material features of the digital twin is extracted frame by frame; the visual detail features include at least one of texture complexity features, surface normal variation features, and curvature features; The physical property initial value map sequence and the visual detail feature map sequence are fused pixel by pixel. The visual detail features are used as local correction factors to enhance or suppress the physical property initial values, thereby reconstructing the physical field state data sequence after visual geometric detail correction. Based on the corrected physical field state data sequence, the physical field evolution entropy and the operation-physical causal correlation metric are calculated respectively, and the two together constitute the two-dimensional quantitative result of the spatiotemporal feature analysis. The physical field evolution entropy is used to quantify the degree of disorder and instability of the internal state evolution of the physical system during user operation. It consists of two terms: the first term is the information entropy calculated after normalizing the probability density of the physical field intensity over the entire spatial domain, which is used to quantify the degree of disorder in the spatial distribution of the physical field; the second term is the integral of the square of the physical field state space gradient over the entire spatial domain, which is used to quantify the degree of drastic change in the physical field space. The operation-physical causal correlation metric is calculated as follows: within the evaluation time window, the time change rate of the actual physical field state driven by user operation at each moment is subtracted and integrated point by point over the entire spatial domain from the time change rate of the naturally decaying physical field state driven only by the decay function without user operation intervention, to obtain the instantaneous value of the net causal influence of user operation at that moment; the instantaneous values ​​of the net causal influence at all moments within the evaluation time window are accumulated over time to obtain the operation-physical causal correlation metric, which is used to quantify the cumulative influence of user operation on the evolution path of the physical system.

8. The method according to claim 7, characterized in that, The process of generating operational skill assessment results that reflect the user's physical cognitive level in their operations includes: The cumulative physical field evolution entropy of the entire experimental process is mapped by a negative exponential function and multiplied by the first preset weight coefficient to obtain the first evaluation component. The final accumulated operation-physical causal correlation metric is divided by the standard deviation of the correlation metric during the experiment, and then multiplied by the second preset weighting coefficient to obtain the second evaluation component; The first evaluation component and the second evaluation component are summed to obtain a comprehensive score for physical cognitive skills, which is used as the evaluation result of the operational skills.

9. The method according to claim 8, characterized in that, The operational skill assessment results are fed back to the virtual simulation experiment system to drive the adaptive adjustment of the experimental scenario. The adaptive adjustment includes at least one of adjusting the experimental difficulty, providing visual guidance clues, or changing the experimental constraints.

Images