Product multi-modal dynamic display generation method based on digital twinning

Abstract

The application provides a product multi-modal dynamic display generation method based on digital twinning, comprising: obtaining structured data of a product; constructing a digital twin model of the product according to the structured data; generating multi-dimensional dynamic display content of the corresponding product at different component nodes according to the digital twin model; constructing a unified time axis and an event trigger, and when the event trigger is activated, synchronously controlling the multi-dimensional dynamic display content based on the unified time axis to perform collaborative dynamic display. The timing consistency and real-time response of the multi-modal display are ensured, the overall coordination and interaction efficiency of the display are improved, and the user can intuitively and comprehensively understand the complex dynamic behavior and state change of the product.

Description

Technical Field

[0001] This invention relates to the field of digital twin and visual interaction technology, and in particular to a method for generating multimodal dynamic displays of products based on digital twins. Background Technology

[0002] Currently, in various technical fields such as product marketing, technical training, and industrial design, it is crucial to present complex products clearly, intuitively, and in-depth. Traditional product display methods, such as paper manuals, two-dimensional images, pre-recorded videos, or simple 3D model browsing, have significant limitations. These methods are usually static, one-way, and fragmented; that is, text descriptions, two-dimensional diagrams, 3D models, and verbal explanations are disconnected, making it difficult for users to form a coherent understanding of the product's overall structure, internal working principles, and dynamic relationships between components. Furthermore, updating and maintaining this content is costly and cannot keep pace with product design changes. With the development of digital twin technology, it has become possible to build virtual models that are highly synchronized with physical products. However, in current technologies, digital twins are mainly used for product performance monitoring, predictive maintenance, and process simulation. Their core value lies in data analysis and state mapping. However, the lack of deep integration of 3D visualization, speech synthesis, and graphic generation with digital twins results in the separation of the generated content from the underlying product model data, leading to poor content accuracy and delayed updates. Multimodal content such as 3D animation, voice explanation, and graphic charts are produced independently, making it difficult to achieve precise synchronization in time and space, which affects learning and cognitive effects. Therefore, the present invention provides a method for generating multimodal dynamic displays of products based on digital twins. Summary of the Invention

[0003] This invention provides a method for generating multimodal dynamic product displays based on digital twins. By acquiring structured data of the product, it lays the foundation for building an accurate digital twin model, thereby achieving a true mapping of the product's physical characteristics. Based on the model, it generates multi-dimensional dynamic display content, which can present multi-angle dynamic information of the product from different component nodes. By constructing a unified timeline and event triggers, when the triggers are activated, the unified timeline is used to synchronously control all display content for collaborative dynamic display, ensuring the temporal consistency and real-time response of the multimodal display, improving the overall coordination and interaction efficiency of the display, and helping users intuitively and comprehensively understand the complex dynamic behavior and state changes of the product.

[0004] This invention provides a method for generating multimodal dynamic displays of products based on digital twins, including: Obtain structured data about the product; Build a digital twin model of the product based on structured data; Based on the digital twin model, generate multi-dimensional dynamic display content of the corresponding product at different component nodes; Construct a unified timeline and event triggers, and when the event triggers are activated, synchronously control the multi-dimensional dynamic display content based on the unified timeline to perform collaborative dynamic display.

[0005] Preferably, a method for generating a multimodal dynamic display of a product based on digital twins, comprising acquiring structured data of the product, including: Extract geometric data and assembly relationship data of the product from the product design system; Based on the geometric data and assembly relationship data, the material property data of the corresponding components are matched and obtained from the material library; Based on the assembly relationship data and material property data, the functional parameters and operational constraint data of the components are obtained from the product database.

[0006] Preferably, a method for generating a multimodal dynamic display of a product based on digital twins, comprising constructing a digital twin model of the product based on structured data, including: Identify the structured data and determine the corresponding geometric data, assembly relationship data, material property data, functional parameters, and operational constraint data. Extract point cloud datasets and interface contour datasets that describe the geometric shapes of components in the product from the geometric data. Based on the point cloud dataset and the interface outline dataset, 3D modeling of each component in the product is performed, and the 3D geometric model of each component in the product is output. Based on the assembly relationship data, determine the fit type and degree of freedom of each component in the product, and locate and link the three-dimensional geometric models of each component in virtual space according to coordinate transformation to obtain the assembly model of the product. The material property data is associated and bound with the corresponding components in the product's assembly model to generate a physical model containing the material properties. The functional parameters and operational constraint data are configured and mapped to the internal state variables, external interaction interfaces, and safe operating conditions of each component in the physical model, and a digital twin model of the product is generated based on the configuration and mapping results.

[0007] Preferably, a method for generating multimodal dynamic product displays based on digital twins generates multidimensional dynamic display content of the corresponding product at different component nodes according to the digital twin model, including: The digital twin model is traversed to extract the structured feature description set of each component node in the digital twin model. The structured feature description set includes: functional semantic labels and state variables. Based on the functional semantic tags and state variables, structural description items are generated for each component node, wherein the structural description items include: audio data of voice narration and graphic annotation data; Based on the geometric data and motion constraints in the structured feature description set, and based on the 3D animation engine, a dynamic visualization sequence is generated to display each component node. Obtain the geometric bounding boxes and baseline simulation time from the structured feature description set, establish a spatiotemporal synchronization chain based on the geometric bounding boxes and baseline simulation time, and align and bind the dynamic visualization sequence, audio data of voice narration and graphic annotation data corresponding to the same component node based on the spatiotemporal synchronization chain to generate the visualization display block of the corresponding component node. The simulation process logic and component interaction nodes in the digital twin model are obtained, and the visualization display blocks corresponding to each component node are logically and scene-wise integrated. Based on the integration results, multi-dimensional dynamic display content is generated.

[0008] Preferably, a method for generating a multimodal dynamic display of a product based on digital twins generates structural description items corresponding to each component node according to functional semantic tags and state variables, including: Parse the functional semantic tags and state variables, and determine the content generation instructions for the corresponding component nodes based on the parsing results; Based on the content generation instructions, the text generation model is invoked to output a formatted text fragment describing the structure and function of the corresponding component node. Formatted text fragments are synthesized into speech to generate audio data for narration. At the same time, the system matches and extracts target templates that match the functional semantic tags from the preset graphic template library, and fills the target templates with state variables to generate graphic annotation data.

[0009] Preferably, a method for generating multimodal dynamic displays of products based on digital twins involves acquiring the simulation process logic and component interaction nodes from the digital twin model, and logically and scene-wise integrating the visualization display blocks corresponding to each component node, including: Based on the simulation process logic, the playback sequence and dependencies of each visualization block are determined; Based on the component interaction nodes, establish the spatial layout and switching relationship between various visual display blocks in the scene.

[0010] Preferably, a method for generating multimodal dynamic product displays based on digital twins constructs a unified timeline and event triggers. When an event trigger is activated, it synchronously controls the multidimensional dynamic display content based on the unified timeline for collaborative dynamic display, including: Construct a unified timeline, adapt the unified timeline to a standard time scale based on physical time, and add a global timestamp pointer to the unified timeline based on the adaptation result; The business scenario of the event trigger to be built is obtained based on the management terminal, and the event change characteristics corresponding to the event trigger to be built are determined based on the business scenario. Based on the event change characteristics, determine the corresponding trigger threshold conditions and event identifiers, and construct event triggers based on the event change characteristics, trigger threshold conditions, and event identifiers; Associate event triggers with a unified timeline, and based on the association mapping results, set at least one timestamp anchor point on the unified timeline for each independent dynamic effect element in the multidimensional dynamic display content. A dynamic parsing engine is generated based on the association mapping results and timestamp anchors, and the activation status of event triggers is monitored in real time based on the dynamic parsing engine. When an event trigger is detected to be activated, the target association mapping relationship between the current event trigger and the unified time axis is extracted, and the control command for the current event on the unified time axis is generated based on the target association mapping relationship. The running status of the unified timeline and the position of the global timestamp pointer are updated based on control commands, and all associated timestamp anchors are triggered synchronously based on the update results. Based on the triggered timestamp anchor point, a state update command is sent to the associated dynamic effect element, and the display state of the dynamic effect element at the current time scale is determined based on the state update command. Based on the defined display state, the corresponding component nodes in the digital twin model are rendered, and the state rendering results are synchronously mapped to all corresponding multi-dimensional dynamic display content to complete the multi-modal collaborative dynamic display based on a unified time axis.

[0011] Preferably, a method for generating multimodal dynamic product displays based on digital twins associates and maps event triggers with a unified timeline, including: Get the constructed event triggers and event identifiers, and get the unified timeline and global timestamp pointer; The mapping type of the event trigger to the unified timeline is determined by the business logic corresponding to the trigger threshold condition of the event trigger. The mapping type includes the position that drives the unified timeline to run or updates the global timestamp pointer. If the mapping type is to drive a unified time axis, then the corresponding control instructions are configured for the event trigger. The control instructions include at least one of start, pause, jump, or speed change, and a target time scale is specified as the parameter for the jump or speed change operation. If the mapping type is to update the position of the global timestamp pointer, the event trigger specifies a target time scale and binds the update action of the global timestamp pointer to the activation event of the event trigger; Based on the determined target time scale or control command, establish the calling relationship between the event identifier of the event trigger and the control interface of the unified time axis.

[0012] Preferably, a method for generating multimodal dynamic displays of products based on digital twins, and a multimodal collaborative dynamic display based on a unified timeline, includes: Parallel evaluation timelines are constructed on a unified timeline. Meanwhile, evaluation tasks for multimodal collaborative dynamic display are obtained based on the management terminal, and the evaluation tasks are aligned with the corresponding unified timelines. Based on the evaluation task, quantitative evaluation indicators for each display target in the multimodal collaborative dynamic display are determined, and corresponding data collection interfaces and evaluation rules are configured for each quantitative evaluation indicator. Obtain the timestamp anchor points associated with the dynamic effect elements from the unified timeline, and synchronize the timestamp anchor points to the evaluation timeline as evaluation trigger points; When the multimodal collaborative dynamic display is in progress, the evaluation trigger point on the evaluation timeline drives the corresponding data acquisition interface to start, obtain the real-time status data of the corresponding component node in the digital twin model, and process the real-time status data based on the evaluation rules to obtain the index value of the quantitative evaluation index at the corresponding evaluation trigger point. The indicator values ​​corresponding to all evaluation trigger points on the evaluation timeline are summarized to obtain the evaluation report corresponding to the multimodal collaborative dynamic display.

[0013] Preferably, a method for generating multimodal dynamic displays of products based on digital twins, yielding an evaluation report corresponding to the multimodal collaborative dynamic display, includes: Obtain the evaluation report, and based on the evaluation report, compare the values ​​of the quantitative evaluation indicators with the corresponding expected indicator thresholds to determine the corresponding deviations; Based on the determined deviation location, the position of the evaluation trigger point on the evaluation time axis is generated, and based on the position of the evaluation trigger point on the evaluation time axis, it is back-mapped to the corresponding target time scale on the unified time axis. Determine the control commands and timestamp anchors associated with the target time scale, and use the determined control commands and timestamp anchors as the objects to be adjusted; The parameters of the object to be adjusted are modified. If the object to be adjusted is a control command, the command type or target time scale parameter is modified. If the object to be adjusted is a timestamp anchor point, the display status parameter of the associated dynamic effect element is modified. The corrected parameters are updated to the association mapping configuration item, and the multimodal collaborative dynamic display is re-driven based on the updated association mapping configuration item.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: By acquiring structured product data, a foundation is laid for building an accurate digital twin model, thereby achieving a true mapping of the product's physical characteristics. Based on the model, multi-dimensional dynamic display content is generated, which can present multi-angle dynamic information of the product from different component nodes. By building a unified timeline and event triggers, when the triggers are activated, the unified timeline is used to synchronously control all display content for collaborative dynamic display, ensuring the temporal consistency and real-time response of multi-modal display, improving the overall coordination and interaction efficiency of the display, and helping users intuitively and comprehensively understand the complex dynamic behavior and state changes of the product.

[0015] Other features and advantages of the invention will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained by means of the structures particularly pointed out in this application.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a product multimodal dynamic display generation method based on digital twins in an embodiment of the present invention; Figure 2 This is a flowchart of step 1 in a product multimodal dynamic display generation method based on digital twins according to an embodiment of the present invention; Figure 3 This is a flowchart of step 2 in a product multimodal dynamic display generation method based on digital twins according to an embodiment of the present invention. Detailed Implementation

[0018] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0019] Example 1: This example provides a method for generating multimodal dynamic displays of products based on digital twins, such as... Figure 1 As shown, it includes: Step 1: Obtain structured data for the product; Step 2: Build a digital twin model of the product based on structured data; Step 3: Generate multi-dimensional dynamic display content of the corresponding product at different component nodes based on the digital twin model; Step 4: Construct a unified timeline and event triggers, and when the event triggers are activated, synchronously control the multi-dimensional dynamic display content based on the unified timeline to perform collaborative dynamic display.

[0020] In this embodiment, structured data refers to the product's organized and standardized data, including geometric dimensions, material properties, physical parameters, and component relationships.

[0021] In this embodiment, a digital twin model refers to a virtual model constructed based on structured data and mapped in real time to a physical product, used to simulate the product's state, behavior, and performance.

[0022] In this embodiment, multidimensional dynamic display content refers to dynamic display information generated for different component nodes in the digital twin model, which includes various forms (such as animation, charts, and 3D rendering).

[0023] In this embodiment, the unified timeline refers to a global time reference axis, which is used to synchronize all dynamically displayed content in time and ensure the consistency of the timing of the display actions.

[0024] In this embodiment, an event trigger refers to a triggering mechanism activated according to preset conditions (such as external input or internal logic) to start or adjust the operation of a unified timeline.

[0025] In this embodiment, collaborative dynamic display refers to the dynamic display process in which multiple multi-dimensional dynamic display contents are displayed synchronously and cooperate with each other under the control of a unified time axis.

[0026] The beneficial effects of the above technical solution are as follows: by acquiring structured data of the product, it lays the foundation for building an accurate digital twin model, thereby realizing a true mapping of the product's physical characteristics. Based on the model, multi-dimensional dynamic display content is generated, which can present multi-angle dynamic information of the product from different component nodes. By building a unified timeline and event triggers, when the triggers are activated, the unified timeline is used to synchronously control all display content to perform collaborative dynamic display, ensuring the temporal consistency and real-time response of multi-modal display, improving the overall coordination and interaction efficiency of the display, and helping users to intuitively and comprehensively understand the complex dynamic behavior and state changes of the product.

[0027] Example 2: Based on Example 1, this example provides a method for generating multimodal dynamic displays of products based on digital twins, such as... Figure 2 As shown, the structured data for the product is obtained, including: Step 101: Extract the product's geometric data and assembly relationship data from the product design system; Step 102: Based on the geometric data and assembly relationship data, match and obtain the material property data of the corresponding component from the material library; Step 103: Based on the assembly relationship data and material property data, obtain the functional parameters and operational constraint data of the components from the product database.

[0028] In this embodiment, geometric data refers to vector or three-dimensional model data that describes the physical shape, spatial dimensions, and contours of product components.

[0029] In this embodiment, assembly relationship data refers to topological data that describes the hierarchical structure, connection method, and mating relationship between various components in the product.

[0030] In this embodiment, the material library refers to a database or knowledge base that pre-sets various material property parameters.

[0031] In this embodiment, material property data refers to physical property data corresponding to the component obtained from the material library, such as density, elastic modulus, coefficient of thermal expansion, etc.

[0032] In this embodiment, the product database refers to a database system that stores technical parameters and business data related to the entire product lifecycle.

[0033] In this embodiment, functional parameters refer to quantitative parameters that describe the design performance, performance indicators, or design intent of product components.

[0034] In this embodiment, operational constraint data refers to restrictive data such as physical limits, environmental conditions, or safety rules that product components must comply with under normal operating conditions.

[0035] The beneficial effects of the above technical solution are as follows: By systematically extracting and matching data from multiple sources such as product design systems, material libraries, and product databases, a complete, accurate, and traceable product information chain is established. This ensures the consistency and reliability of multi-dimensional data such as geometry, physics, function, and constraints required for building a digital twin model, laying a solid data foundation for the subsequent generation of high-fidelity dynamic displays. It effectively avoids model distortion caused by missing or contradictory data, and improves the accuracy and credibility of the entire display system.

[0036] Example 3: Based on Example 1, this example provides a method for generating multimodal dynamic displays of products based on digital twins, such as... Figure 3 As shown, a digital twin model of the product is constructed based on structured data, including: Step 201: Identify the structured data and determine the corresponding geometric data, assembly relationship data, material property data, functional parameters, and operational constraint data. Step 202: Extract point cloud datasets and interface contour datasets from the geometric data to describe the geometric shapes of the components in the product. Step 203: Perform 3D modeling of each component in the product based on the point cloud dataset and the interface outline dataset, and output the 3D geometric model of each component in the product; Step 204: Determine the fit type and degree of freedom of each component in the product based on the assembly relationship data, and locate and link the three-dimensional geometric models of each component in the virtual space according to the coordinate transformation to obtain the assembly model of the product. Step 205: Associate and bind the material property data with the corresponding components in the product's assembly model to generate a physical model containing material properties; Step 206: Configure the functional parameters and operational constraint data and map them to the internal state variables, external interaction interfaces and safe operating conditions of each component in the physical model, and generate a digital twin model of the product based on the configuration and mapping results.

[0037] In this embodiment, the point cloud dataset refers to a collection of a large number of discrete points obtained through measurement or design data to describe the three-dimensional spatial position of an object's surface.

[0038] In this embodiment, the interface contour dataset refers to a set of geometric data describing the contact, mating, or assembly boundary shapes between the components of the product.

[0039] In this embodiment, the three-dimensional geometric model refers to a digital model that expresses the three-dimensional shape of an object, generated using computer graphics methods based on point cloud and contour data.

[0040] In this embodiment, the assembly model refers to a combined model that represents the complete structure of a product, formed by locating and connecting multiple independent three-dimensional geometric models in a virtual space according to their assembly relationships.

[0041] In this embodiment, the physical model refers to a computational model that is based on the assembly model, and has associated and bound material property data, thereby possessing basic physical properties (such as mass and inertia).

[0042] In this embodiment, internal state variables refer to parameters used to describe and track the internal states (such as temperature, pressure, and wear) of each component in the digital twin model during operation.

[0043] In this embodiment, the external interaction interface refers to the logical channel or data port defined in the digital twin model, which is used to receive external instructions or output status information to external systems.

[0044] In this embodiment, safe operating conditions refer to the numerical boundaries or logical rules that are preset to ensure the safety of digital twin model simulation or corresponding physical product, and that the state variables of each component are not allowed to exceed.

[0045] The beneficial effects of the above technical solution are as follows: through a systematic data processing and model building process, abstract geometric, material, assembly and functional constraints and other multi-source heterogeneous data are gradually integrated into a digital twin model with high-fidelity geometric shape, accurate physical properties and clear interaction rules. This ensures that the model is not only consistent with the physical product in appearance, but also achieves accurate mapping at the level of internal physical characteristics and functional logic, providing a high-quality model foundation for the subsequent generation of accurate, reliable and interactive multi-dimensional dynamic displays.

[0046] Example 4: Based on Example 1, this example provides a method for generating multimodal dynamic displays of products based on digital twins. It generates multidimensional dynamic display content of the corresponding product at different component nodes according to the digital twin model, including: The digital twin model is traversed to extract the structured feature description set of each component node in the digital twin model. The structured feature description set includes: functional semantic labels and state variables. Based on the functional semantic tags and state variables, structural description items are generated for each component node, wherein the structural description items include: audio data of voice narration and graphic annotation data; Based on the geometric data and motion constraints in the structured feature description set, and based on the 3D animation engine, a dynamic visualization sequence is generated to display each component node. Obtain the geometric bounding boxes and baseline simulation time from the structured feature description set, establish a spatiotemporal synchronization chain based on the geometric bounding boxes and baseline simulation time, and align and bind the dynamic visualization sequence, audio data of voice narration and graphic annotation data corresponding to the same component node based on the spatiotemporal synchronization chain to generate the visualization display block of the corresponding component node. The simulation process logic and component interaction nodes in the digital twin model are obtained, and the visualization display blocks corresponding to each component node are logically and scene-wise integrated. Based on the integration results, multi-dimensional dynamic display content is generated.

[0047] In this embodiment, the structured feature description set refers to a set of feature data extracted from the component nodes of the digital twin model, which includes functional semantic labels (classification information that identifies the functions of the component) and state variables (parameters that describe the simulation dynamics of the component).

[0048] In this embodiment, the structural description item refers to the set of explanatory materials generated for a component node, which includes audio data of voice narration (explanatory audio synthesized from text) and graphic annotation data (explanatory materials that integrate text and illustrations).

[0049] In this embodiment, the spatiotemporal synchronization chain refers to the synchronization rules established based on the geometric bounding box (the extent of the component in three-dimensional space) and the reference simulation time (a unified timing reference) for aligning dynamic visualization sequences, voice and graphic data.

[0050] In this embodiment, the visualization display block refers to a composite display unit that can be independently controlled, formed by binding the dynamic visualization sequence, voice narration data and graphic annotation data of the same component through a spatiotemporal synchronization chain.

[0051] In this embodiment, the component interaction node refers to the control point defined in the digital twin model simulation process, which represents the logical connection or state transmission relationship between components and is used to guide the integration and interaction of multiple visualization blocks.

[0052] In this embodiment, the simulation process logic refers to the programmatic rules extracted from the digital twin model that describe the working order, conditions, and interrelationships of each component of the product. It is the core basis for the dynamic display content logic arrangement.

[0053] The beneficial effects of the above technical solution are as follows: By automatically extracting the structured feature description set of component nodes from the digital twin model, and generating multimodal display materials such as voice, text and graphics, and 3D dynamic sequences based on the structured feature description set, the precise association and efficient derivation of product display content with the underlying twin model are achieved; furthermore, by aligning and binding different modal materials into a unified visual display block through a spatiotemporal synchronization chain, the strict synchronization and consistency of explanations, illustrations, and animations in time and space are ensured; finally, the display blocks are integrated in a scenario-based manner according to simulation logic and interaction relationships, generating multidimensional dynamic display content that is structurally clear, logically coherent, and interactive; effectively improving the generation efficiency and accuracy of complex product display content, ensuring the synergy and immersiveness of multimodal information output, and providing a highly integrated, dynamic, and interactive digital display solution for product design review, operation training, and marketing demonstrations.

[0054] Example 5: Building upon Example 4, this example provides a method for generating multimodal dynamic displays of products based on digital twins. It generates structural description items corresponding to each component node based on functional semantic tags and state variables, including: Parse the functional semantic tags and state variables, and determine the content generation instructions for the corresponding component nodes based on the parsing results; Based on the content generation instructions, the text generation model is invoked to output a formatted text fragment describing the structure and function of the corresponding component node. Formatted text fragments are synthesized into speech to generate audio data for narration. At the same time, the system matches and extracts target templates that match the functional semantic tags from the preset graphic template library, and fills the target templates with state variables to generate graphic annotation data.

[0055] In this embodiment, the content generation instruction refers to a standardized operation request formed by parsing functional semantic tags and state variables, which is used to drive the generation of subsequent text and media content.

[0056] In this embodiment, a formatted text fragment refers to a text paragraph output by a text generation model that describes the structure and function of a component and conforms to predetermined syntax and structural specifications.

[0057] In this embodiment, the graphic template library refers to a pre-set collection of graphic and text combination frames containing various layouts and styles, used to quickly generate standardized graphic and text annotations.

[0058] In this embodiment, the target template refers to a specific graphic template selected from the graphic template library that corresponds to the semantic tag of the current component's function.

[0059] The beneficial effects of the above technical solution are: by parsing functional semantic tags and state variables to automatically form content generation instructions, and linking text generation, speech synthesis and template filling technologies, the automated and standardized generation of product component explanation content (structured text, voice and graphics) is realized; the production efficiency of multimodal display materials is effectively improved, and the high consistency between the explanation content and the twin model data is ensured.

[0060] Example 6: Based on Example 4, this example provides a method for generating multimodal dynamic displays of products based on digital twins. It obtains the simulation process logic and component interaction nodes from the digital twin model, and logically integrates and scene-integrates the visualization display blocks corresponding to each component node, including: Based on the simulation process logic, the playback sequence and dependencies of each visualization block are determined; Based on the component interaction nodes, establish the spatial layout and switching relationship between various visual display blocks in the scene.

[0061] In this embodiment, playback timing and dependency refer to the start time, sequence, and triggering conditions set for each visualization block during the integration process.

[0062] In this embodiment, spatial layout and switching relationship refers to the spatial attributes such as position and angle assigned to each visualization display block in the three-dimensional scene, as well as the screen jump or focus rules triggered by their interaction.

[0063] The beneficial effects of the above technical solution are: by guiding the integration of the visualization display blocks through the simulation process logic and component interaction nodes, the final dynamic display has a clear logical structure, correct temporal relationship and spatial scene that conforms to physical interaction, thereby significantly improving the overall display coherence and user experience.

[0064] Example 7: Based on Example 1, this example provides a method for generating multimodal dynamic displays of products based on digital twins. It constructs a unified timeline and event triggers, and when the event triggers are activated, it synchronously controls the multi-dimensional dynamic display content based on the unified timeline for collaborative dynamic display, including: Construct a unified timeline, adapt the unified timeline to a standard time scale based on physical time, and add a global timestamp pointer to the unified timeline based on the adaptation result; The business scenario of the event trigger to be built is obtained based on the management terminal, and the event change characteristics corresponding to the event trigger to be built are determined based on the business scenario. Based on the event change characteristics, determine the corresponding trigger threshold conditions and event identifiers, and construct event triggers based on the event change characteristics, trigger threshold conditions, and event identifiers; Associate event triggers with a unified timeline, and based on the association mapping results, set at least one timestamp anchor point on the unified timeline for each independent dynamic effect element in the multidimensional dynamic display content. A dynamic parsing engine is generated based on the association mapping results and timestamp anchors, and the activation status of event triggers is monitored in real time based on the dynamic parsing engine. When an event trigger is detected to be activated, the target association mapping relationship between the current event trigger and the unified time axis is extracted, and the control command for the current event on the unified time axis is generated based on the target association mapping relationship. The running status of the unified timeline and the position of the global timestamp pointer are updated based on control commands, and all associated timestamp anchors are triggered synchronously based on the update results. Based on the triggered timestamp anchor point, a state update command is sent to the associated dynamic effect element, and the display state of the dynamic effect element at the current time scale is determined based on the state update command. Based on the defined display state, the corresponding component nodes in the digital twin model are rendered, and the state rendering results are synchronously mapped to all corresponding multi-dimensional dynamic display content to complete the multi-modal collaborative dynamic display based on a unified time axis.

[0065] In this embodiment, associating the event trigger with the unified timeline means that when the event trigger is activated, the global timestamp pointer will be updated to a specified time scale, or the unified timeline will be driven to start, pause, jump, or change speed.

[0066] In this embodiment, each timestamp anchor point is associated with the start, end, or key state frame of the corresponding dynamic effect element.

[0067] In this embodiment, the global timestamp pointer refers to the position marker set on a unified timeline to indicate the current virtual time scale of the collaborative dynamic display in real time.

[0068] In this embodiment, the business scenario refers to the specific application background or operational situation set by the management terminal that requires an event trigger to respond.

[0069] In this embodiment, the event change characteristics refer to the change patterns of external input signals or internal logic data that an event trigger needs to monitor under a specific business scenario.

[0070] In this embodiment, the trigger threshold condition refers to the specific numerical or logical condition set for the event change characteristics to determine and activate the event trigger.

[0071] In this embodiment, the event identifier refers to a specific code or label assigned to uniquely identify and distinguish different event triggers.

[0072] In this embodiment, a timestamp anchor point refers to a marker point set on a specific virtual time scale of a unified time axis, used to associate with and trigger the state update of an independent dynamic effect element.

[0073] In this embodiment, the dynamic parsing engine refers to a software functional module used to parse the activation state of event triggers in real time and generate control instructions for a unified timeline based on the associated mapping relationship and timestamp anchor.

[0074] In this embodiment, the target association mapping relationship refers to the specific control correspondence between the trigger and the unified time axis extracted from the preset association mapping configuration when the event trigger is activated.

[0075] In this embodiment, the control command refers to the command generated by the dynamic parsing engine, which is used to directly change the running state of the unified timeline (such as start, pause, jump) or update the position of the global timestamp pointer.

[0076] In this embodiment, the state update instruction refers to the information containing the current virtual time scale and target state parameters issued by the triggered timestamp anchor to its associated dynamic effect element.

[0077] The beneficial effects of the above technical solution are as follows: by constructing a unified timeline adapted to physical time and business-oriented event triggers, and establishing a precise mapping relationship between the two, a core timing collaborative control mechanism is provided for multimodal dynamic display. Through the dynamic parsing engine, the triggered events are monitored and parsed in real time, and precise timeline control instructions are generated, ensuring that all independent dynamic effect elements can be synchronously updated and rendered according to a unified timing benchmark. This realizes the automation and integration of the display process from event triggering to multimodal content collaborative response, significantly improving the timing consistency, interactive response accuracy and overall coordination of complex product dynamic display.

[0078] Example 8: Based on Example 7, this example provides a method for generating multimodal dynamic displays of products based on digital twins, which associates and maps event triggers with a unified timeline, including: Get the constructed event triggers and event identifiers, and get the unified timeline and global timestamp pointer; The mapping type of the event trigger to the unified timeline is determined by the business logic corresponding to the trigger threshold condition of the event trigger. The mapping type includes the position that drives the unified timeline to run or updates the global timestamp pointer. If the mapping type is to drive a unified time axis, then the corresponding control instructions are configured for the event trigger. The control instructions include at least one of start, pause, jump, or speed change, and a target time scale is specified as the parameter for the jump or speed change operation. If the mapping type is to update the position of the global timestamp pointer, the event trigger specifies a target time scale and binds the update action of the global timestamp pointer to the activation event of the event trigger; Based on the determined target time scale or control command, establish the calling relationship between the event identifier of the event trigger and the control interface of the unified time axis.

[0079] In this embodiment, the mapping type refers to the specific way in which the event trigger applies control to a unified time axis, determined according to the business logic of the event trigger.

[0080] In this embodiment, the target time scale refers to a specific virtual time point pre-specified on a unified time axis, used as a time axis jump target or a global timestamp pointer update destination.

[0081] In this embodiment, the control interface refers to the program interface provided by the unified timeline for receiving external instructions to change its running state or internal pointer position.

[0082] In this embodiment, the calling relationship refers to the program association established between the event identifier and the control interface, which can automatically initiate instruction calls when the event trigger is activated.

[0083] The beneficial effects of the above technical solution are as follows: by classifying and mapping event triggers to different control types of a unified timeline according to business logic, and precisely configuring control instructions or target time scales accordingly, a deterministic logical connection between events and timeline control is established. By establishing the calling relationship between event identifiers and timeline control interfaces, the automatic and precise driving of event triggers on display timing is realized, ensuring the consistency and reliability of multimodal display responses.

[0084] Example 9: Based on Example 7, this example provides a method for generating multimodal dynamic displays of products based on digital twins, and a multimodal collaborative dynamic display based on a unified time axis, including: Parallel evaluation timelines are constructed on a unified timeline. Meanwhile, evaluation tasks for multimodal collaborative dynamic display are obtained based on the management terminal, and the evaluation tasks are aligned with the corresponding unified timelines. Based on the evaluation task, quantitative evaluation indicators for each display target in the multimodal collaborative dynamic display are determined, and corresponding data collection interfaces and evaluation rules are configured for each quantitative evaluation indicator. Obtain the timestamp anchor points associated with the dynamic effect elements from the unified timeline, and synchronize the timestamp anchor points to the evaluation timeline as evaluation trigger points; When the multimodal collaborative dynamic display is in progress, the evaluation trigger point on the evaluation timeline drives the corresponding data acquisition interface to start, obtain the real-time status data of the corresponding component node in the digital twin model, and process the real-time status data based on the evaluation rules to obtain the index value of the quantitative evaluation index at the corresponding evaluation trigger point. The indicator values ​​corresponding to all evaluation trigger points on the evaluation timeline are summarized to obtain the evaluation report corresponding to the multimodal collaborative dynamic display.

[0085] In this embodiment, the evaluation timeline refers to a dedicated timeline built in parallel with the unified timeline for controlling and triggering the effect evaluation process.

[0086] In this embodiment, the evaluation task refers to the effect evaluation work required for a complete or partial multimodal collaborative dynamic display, which is set by the management terminal.

[0087] In this embodiment, the quantitative evaluation index refers to a specific, measurable numerical standard used to measure and evaluate the degree to which a target has been achieved.

[0088] In this embodiment, the evaluation rule refers to the predefined calculation logic and data processing method for processing real-time status data and calculating quantitative evaluation index values.

[0089] The beneficial effects of the above technical solution are as follows: By constructing a parallel evaluation timeline on a unified time axis and aligning it with the display task, synchronous monitoring of the dynamic display process is achieved. By configuring quantitative indicators and data acquisition interfaces directly related to the display objectives, data acquisition and calculation can be automatically triggered during the display, thereby transforming complex display effects into an objective and continuous sequence of indicator values. This process enables automated and accurate evaluation of the multimodal collaborative dynamic display effects.

[0090] Example 10: Based on Example 9, this example provides a method for generating multimodal dynamic displays of products based on digital twins, obtaining an evaluation report corresponding to the multimodal collaborative dynamic display, including: Obtain the evaluation report, and based on the evaluation report, compare the values ​​of the quantitative evaluation indicators with the corresponding expected indicator thresholds to determine the corresponding deviations; Based on the determined deviation location, the position of the evaluation trigger point on the evaluation time axis is generated, and based on the position of the evaluation trigger point on the evaluation time axis, it is back-mapped to the corresponding target time scale on the unified time axis. Determine the control commands and timestamp anchors associated with the target time scale, and use the determined control commands and timestamp anchors as the objects to be adjusted; The parameters of the object to be adjusted are modified. If the object to be adjusted is a control command, the command type or target time scale parameter is modified. If the object to be adjusted is a timestamp anchor point, the display status parameter of the associated dynamic effect element is modified. The corrected parameters are updated to the association mapping configuration item, and the multimodal collaborative dynamic display is re-driven based on the updated association mapping configuration item.

[0091] In this embodiment, deviation refers to the difference between the value of the quantitative evaluation indicator and its corresponding expected threshold.

[0092] In this embodiment, the expected indicator threshold refers to the expected value or acceptable range that is set in advance for the quantitative evaluation indicator to determine whether the display effect meets the standard.

[0093] In this embodiment, the object to be adjusted refers to the control command or timestamp anchor that has been analyzed and determined to cause evaluation deviation and therefore requires correction of its parameters.

[0094] In this embodiment, the association mapping configuration item refers to a configuration file or data set that stores all association mapping relationships and parameters between event triggers and a unified timeline.

[0095] The beneficial effects of the above technical solution are: by accurately mapping the quantitative deviations in the evaluation report back to the specific control links of the unified time axis, a closed-loop link from effect evaluation to display optimization is established. By locating and correcting the time axis control commands or dynamic effect parameters that cause deviations, targeted iteration and precise adjustment of the multimodal collaborative dynamic display process can be achieved, thereby continuously improving the accuracy of the display effect and the target conformity based on objective evaluation data.

[0096] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for generating multimodal dynamic product displays based on digital twins, characterized in that, include: Obtain structured data about the product; Build a digital twin model of the product based on structured data; Based on the digital twin model, generate multi-dimensional dynamic display content of the corresponding product at different component nodes; Construct a unified timeline and event triggers, and when the event triggers are activated, synchronously control the multi-dimensional dynamic display content based on the unified timeline to perform collaborative dynamic display.

2. The method for generating a multimodal dynamic product display based on digital twins according to claim 1, characterized in that, Obtain structured data about the product, including: Extract geometric data and assembly relationship data of the product from the product design system; Based on the geometric data and assembly relationship data, the material property data of the corresponding components are matched and obtained from the material library; Based on the assembly relationship data and material property data, the functional parameters and operational constraint data of the components are obtained from the product database.

3. The method for generating a multimodal dynamic product display based on digital twins according to claim 1, characterized in that, Building a digital twin model of a product based on structured data includes: Identify the structured data and determine the corresponding geometric data, assembly relationship data, material property data, functional parameters, and operational constraint data. Extract point cloud datasets and interface contour datasets that describe the geometric shapes of components in the product from the geometric data. Based on the point cloud dataset and the interface outline dataset, 3D modeling of each component in the product is performed, and the 3D geometric model of each component in the product is output. Based on the assembly relationship data, determine the fit type and degree of freedom of each component in the product, and locate and link the three-dimensional geometric models of each component in virtual space according to coordinate transformation to obtain the assembly model of the product. The material property data is associated and bound with the corresponding components in the product's assembly model to generate a physical model containing the material properties. The functional parameters and operational constraint data are configured and mapped to the internal state variables, external interaction interfaces, and safe operating conditions of each component in the physical model, and a digital twin model of the product is generated based on the configuration and mapping results.

4. The method for generating a multimodal dynamic product display based on digital twins according to claim 1, characterized in that, Based on the digital twin model, multi-dimensional dynamic display content of the corresponding product is generated at different component nodes, including: The digital twin model is traversed to extract the structured feature description set of each component node in the digital twin model. The structured feature description set includes: functional semantic labels and state variables. Based on the functional semantic tags and state variables, structural description items are generated for each component node, wherein the structural description items include: audio data of voice narration and graphic annotation data; Based on the geometric data and motion constraints in the structured feature description set, and based on the 3D animation engine, a dynamic visualization sequence is generated to display each component node. Obtain the geometric bounding boxes and baseline simulation time from the structured feature description set, establish a spatiotemporal synchronization chain based on the geometric bounding boxes and baseline simulation time, and align and bind the dynamic visualization sequence, audio data of voice narration and graphic annotation data corresponding to the same component node based on the spatiotemporal synchronization chain to generate the visualization display block of the corresponding component node. The simulation process logic and component interaction nodes in the digital twin model are obtained, and the visualization display blocks corresponding to each component node are logically and scene-wise integrated. Based on the integration results, multi-dimensional dynamic display content is generated.

5. The method for generating a multimodal dynamic product display based on digital twins according to claim 4, characterized in that, Based on the functional semantic tags and state variables, generate structural description items corresponding to each component node, including: Parse the functional semantic tags and state variables, and determine the content generation instructions for the corresponding component nodes based on the parsing results; Based on the content generation instructions, the text generation model is invoked to output a formatted text fragment describing the structure and function of the corresponding component node. Formatted text fragments are synthesized into speech to generate audio data for narration. At the same time, the system matches and extracts target templates that match the functional semantic tags from the preset graphic template library, and fills the target templates with state variables to generate graphic annotation data.

6. The method for generating a multimodal dynamic product display based on digital twins according to claim 4, characterized in that, Obtain the simulation process logic and component interaction nodes from the digital twin model, and logically and scene-wise integrate the visualization blocks corresponding to each component node, including: Based on the simulation process logic, the playback sequence and dependencies of each visualization block are determined; Based on the component interaction nodes, establish the spatial layout and switching relationship between various visual display blocks in the scene.

7. The method for generating a multimodal dynamic display of a product based on digital twins according to claim 1, characterized in that, Construct a unified timeline and event triggers, and when an event trigger is activated, synchronously control the multi-dimensional dynamic display content based on the unified timeline to perform collaborative dynamic display, including: Construct a unified timeline, adapt the unified timeline to a standard time scale based on physical time, and add a global timestamp pointer to the unified timeline based on the adaptation result; The business scenario of the event trigger to be built is obtained based on the management terminal, and the event change characteristics corresponding to the event trigger to be built are determined based on the business scenario. Based on the event change characteristics, determine the corresponding trigger threshold conditions and event identifiers, and construct event triggers based on the event change characteristics, trigger threshold conditions, and event identifiers; Associate event triggers with a unified timeline, and based on the association mapping results, set at least one timestamp anchor point on the unified timeline for each independent dynamic effect element in the multidimensional dynamic display content. A dynamic parsing engine is generated based on the association mapping results and timestamp anchors, and the activation status of event triggers is monitored in real time based on the dynamic parsing engine. When an event trigger is detected to be activated, the target association mapping relationship between the current event trigger and the unified time axis is extracted, and the control command for the current event on the unified time axis is generated based on the target association mapping relationship. The running status of the unified timeline and the position of the global timestamp pointer are updated based on control commands, and all associated timestamp anchors are triggered synchronously based on the update results. Based on the triggered timestamp anchor point, a state update command is sent to the associated dynamic effect element, and the display state of the dynamic effect element at the current time scale is determined based on the state update command. Based on the defined display state, the corresponding component nodes in the digital twin model are rendered, and the state rendering results are synchronously mapped to all corresponding multi-dimensional dynamic display content to complete the multi-modal collaborative dynamic display based on a unified time axis.

8. The method for generating a multimodal dynamic product display based on digital twins according to claim 7, characterized in that, Associating event triggers with a unified timeline includes: Obtain the constructed event triggers and event identifiers, and obtain the unified timeline and global timestamp pointer; The mapping type of the event trigger to the unified timeline is determined by the business logic corresponding to the trigger threshold condition of the event trigger. The mapping type includes the position that drives the unified timeline to run or updates the global timestamp pointer. If the mapping type is to drive a unified time axis, then the corresponding control instructions are configured for the event trigger. The control instructions include at least one of start, pause, jump, or speed change, and a target time scale is specified as the parameter for the jump or speed change operation. If the mapping type is to update the position of the global timestamp pointer, the event trigger specifies a target time scale and binds the update action of the global timestamp pointer to the activation event of the event trigger; Based on the determined target time scale or control command, establish the calling relationship between the event identifier of the event trigger and the control interface of the unified time axis.

9. A method for generating multimodal dynamic product display based on digital twins according to claim 7, characterized in that, Multimodal collaborative dynamic display based on a unified timeline includes: Parallel evaluation timelines are constructed on a unified timeline. Meanwhile, evaluation tasks for multimodal collaborative dynamic display are obtained based on the management terminal, and the evaluation tasks are aligned with the corresponding unified timelines. Based on the evaluation task, quantitative evaluation indicators for each display target in the multimodal collaborative dynamic display are determined, and corresponding data collection interfaces and evaluation rules are configured for each quantitative evaluation indicator. Obtain the timestamp anchor points associated with the dynamic effect elements from the unified timeline, and synchronize the timestamp anchor points to the evaluation timeline as evaluation trigger points; When the multimodal collaborative dynamic display is in progress, the evaluation trigger point on the evaluation timeline drives the corresponding data acquisition interface to start, obtain the real-time status data of the corresponding component node in the digital twin model, and process the real-time status data based on the evaluation rules to obtain the index value of the quantitative evaluation index at the corresponding evaluation trigger point. The indicator values ​​corresponding to all evaluation trigger points on the evaluation timeline are summarized to obtain the evaluation report corresponding to the multimodal collaborative dynamic display.

10. The method for generating a multimodal dynamic product display based on digital twins according to claim 9, characterized in that, The evaluation report corresponding to the multimodal collaborative dynamic display is obtained, including: Obtain the evaluation report, and based on the evaluation report, compare the values ​​of the quantitative evaluation indicators with the corresponding expected indicator thresholds to determine the corresponding deviations; Based on the determined deviation location, the position of the evaluation trigger point on the evaluation time axis is generated, and based on the position of the evaluation trigger point on the evaluation time axis, it is back-mapped to the corresponding target time scale on the unified time axis. Determine the control commands and timestamp anchors associated with the target time scale, and use the determined control commands and timestamp anchors as the objects to be adjusted; The parameters of the object to be adjusted are modified. If the object to be adjusted is a control command, the command type or target time scale parameter is modified. If the object to be adjusted is a timestamp anchor point, the display status parameter of the associated dynamic effect element is modified. The corrected parameters are updated to the association mapping configuration item, and the multimodal collaborative dynamic display is re-driven based on the updated association mapping configuration item.

Images