Intelligent digital modeling system and method for intangible cultural heritage process based on multi-modal perception

Abstract

The application relates to the technical field of model construction, in particular to a non-heritage craft digital modeling system and method based on multi-modal perception, which comprises the following steps: acquiring action data and environment data of an artisan operation, wherein the action data and the environment data are collected through a sensor system; performing multi-modal data synchronization and feature fusion based on the action data and the environment data to generate a process parameter correlation model; constructing a digital twin model of the non-heritage craft based on the process parameter correlation model and optimizing process parameters; generating real-time operation guidance information based on the digital twin model and performing quality evaluation and feedback optimization on process results. Through a multi-modal data synchronization algorithm optimized by dynamic time warping (DTW), a process parameter correlation model based on a non-Euclidean space graph convolution network and a multi-objective Bayesian optimization framework, the application realizes digital preservation and intelligent optimization of a traditional process.

Description

Technical Field

[0001] This invention relates to the field of digitalization and model building technology of intangible cultural heritage crafts, and more specifically to a digital modeling system and method for intangible cultural heritage crafts based on multimodal perception. Background Technology

[0002] In terms of the technical means required for digital modeling of intangible cultural heritage crafts, existing motion capture equipment is significantly inadequate. Currently, common motion capture equipment mainly includes optical, mechanical, and electromagnetic types. Among them, optical motion capture equipment, such as the Vicon system, while highly accurate and flexible, has stringent requirements for the operating environment, requiring a large, unobstructed space and professional operators and maintenance personnel.

[0003] Mechanical motion capture equipment, such as Ascension's magnetic tracking system, is highly sensitive to electromagnetic interference in the operating environment, prone to signal loss or accuracy degradation, requiring regular calibration and exhibiting high operational complexity. Electromagnetic motion capture equipment, on the other hand, suffers from poor accuracy and stability due to magnetic field interference, making it difficult to meet the demands for precise motion capture in intangible cultural heritage crafts.

[0004] In summary, existing technologies lack a systematic solution that can accurately capture the micro-strain of a craftsman's manual operations, integrate multimodal asynchronous time-series data, and perform correlation modeling and multi-objective optimization of process parameters based on materials physics simulation. Summary of the Invention

[0005] In view of this, the present invention provides a digital modeling system and method for intangible cultural heritage crafts, aiming to solve the above-mentioned technical problems. Through a multimodal data synchronization algorithm optimized by dynamic time warping (DTW), a process parameter association model based on non-Euclidean graph convolutional network, and a multi-objective Bayesian optimization framework, the present invention achieves digital preservation and intelligent optimization of traditional crafts.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A digital modeling method for intangible cultural heritage crafts based on multimodal perception includes:

[0008] The reconfigurable sensor array collects three-dimensional posture data and micro-strain data of the craftsman's hand as motion data, and the environmental sensing module collects environmental temperature, humidity and pressure data as environmental data.

[0009] The three-dimensional posture data, hand micro-strain data, and environmental temperature, humidity, and pressure data are time-aligned using a dynamic time warping algorithm to generate synchronized data.

[0010] The synchronized data is format-converted and standardized based on a lightweight neural network inference engine.

[0011] By introducing a gated attention mechanism into a long short-term memory network, features are extracted from the processed synchronous data to generate fused features;

[0012] The fused features are mapped to a process parameter correlation model that characterizes the nonlinear correlation between process parameters based on a non-Euclidean graph convolutional network.

[0013] Based on the process parameter association model, a digital twin model of intangible cultural heritage process is constructed in a 3D visualization engine, and the physical properties of the process materials in the digital twin model are simulated based on the smooth particle fluid dynamics algorithm.

[0014] Using a multi-objective Bayesian optimization framework with process quality and efficiency as objectives, the system automatically searches for the optimal solution of process parameters and updates the digital twin model based on the optimal solution.

[0015] Based on the digital twin model, real-time operation guidance information is generated through a lightweight semantic segmentation model and displayed through an augmented reality device;

[0016] A defect detection model based on transfer learning is used to evaluate the quality of process results and generate evaluation results.

[0017] The evaluation results are fed back to the digital twin model to adjust the process parameters, and the real-time operation guidance information is updated based on the adjusted process parameters.

[0018] As an embodiment of the present invention, in the method described above, the generation of synchronization data includes:

[0019] Assign timestamps to the action data and environment data to generate a time-stamped data sequence;

[0020] The time difference between the data sequences is calculated based on the dynamic time warping algorithm;

[0021] Synchronous data is generated by adjusting the time difference of the data sequence through interpolation.

[0022] As an embodiment of the present invention, in the method described above, the step of mapping the fused features to a process parameter correlation model characterizing the nonlinear correlation between process parameters based on a non-Euclidean graph convolutional network includes:

[0023] The fused features are input into a non-Euclidean graph convolutional network to generate the correlation between process parameters;

[0024] A process parameter association model is constructed based on the aforementioned relationships;

[0025] When new fusion features are received, the structure and weights of the process parameter association model are dynamically updated;

[0026] Cross-process transfer learning is achieved based on the process parameter association model.

[0027] As an embodiment of the present invention, in the method described above, the step of automatically searching for the optimal solution of process parameters using a multi-objective Bayesian optimization framework with process quality and efficiency as objectives includes:

[0028] A multi-objective optimization problem is established with process quality and process efficiency as optimization objectives.

[0029] A Bayesian optimization algorithm is used to automatically search for Pareto optimal solutions for process parameters in a digital twin model.

[0030] As an embodiment of the present invention, in the method described above, the defect detection model based on transfer learning performs quality assessment of the process results, including:

[0031] The model is pre-trained on various process data as the base model;

[0032] The transfer learning algorithm is used to adapt the basic model to the defect detection task of a specific intangible cultural heritage craft.

[0033] Another objective of this invention is to provide a digital modeling system for intangible cultural heritage crafts based on multimodal perception, comprising:

[0034] A motion capture module with a reconfigurable sensor array is used to acquire the craftsman's three-dimensional posture data and hand micro-strain data;

[0035] The environmental sensing module is used to collect ambient temperature, humidity, and pressure data.

[0036] A multi-source heterogeneous data fusion engine is used to perform multimodal data synchronization and feature fusion based on the action data and environmental data to generate a process parameter correlation model;

[0037] An adaptive edge computing unit is used to construct a digital twin model of the intangible cultural heritage process based on the process parameter association model, and to optimize the process parameters;

[0038] An interactive digital twin modeling platform is used to generate real-time operation guidance information based on the digital twin model, and to perform quality assessment and feedback optimization of process results.

[0039] Another object of the present invention is to provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods described above.

[0040] Another object of the present invention is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the methods described above.

[0041] Compared to existing technologies, the digital modeling system and method for intangible cultural heritage (ICH) crafts based on multimodal perception provided by this invention enables the digital preservation and intelligent optimization of ICH crafts. It collects action data and environmental data from artisans using a reconfigurable sensor array, synchronizes multimodal data based on a dynamic time warping algorithm, extracts and fuses features using a long short-term memory network with a gated attention mechanism, generates a process parameter correlation model using a non-Euclidean graph convolutional network, constructs a digital twin model of the ICH craft, and optimizes the process parameters using a multi-objective Bayesian optimization framework. Simultaneously, it generates real-time operation guidance information based on the digital twin model and performs quality assessment and feedback optimization of the process results, effectively improving the production efficiency and economic benefits of ICH crafts and providing strong technical support for the innovative development of ICH crafts. The system and method have the following effects:

[0042] High-precision sensors and optimized data acquisition circuits are used to ensure the accuracy of environmental perception data and provide reliable environmental parameter support for the digital modeling of intangible cultural heritage crafts.

[0043] In terms of environmental perception, this invention utilizes high-precision sensors and optimizes the data acquisition circuit design to accurately capture key data such as ambient temperature, humidity, and pressure during the production of intangible cultural heritage crafts. These optimized sensor and circuit designs effectively reduce errors and interference during data acquisition, ensuring the accuracy and reliability of the acquired environmental data. Accurate environmental parameters are a crucial foundation for constructing digital models of intangible cultural heritage crafts, providing solid environmental data support for subsequent multimodal data synchronization, feature fusion, and the generation of process parameter correlation models. This allows digital modeling to better reflect actual craft scenarios, enhancing the model's realism and effectiveness.

[0044] By using modular sensor design and domestic chip alternatives, production costs and maintenance difficulties are reduced, making the system more economical and competitive in the market, meeting the actual needs of more enterprises, and promoting the widespread application and popularization of digital technology for intangible cultural heritage crafts.

[0045] In terms of hardware design and cost control, this invention adopts a modular sensor design, integrating sensors with different functions in a modular fashion. This not only simplifies the system assembly and maintenance process and reduces maintenance difficulty, but also facilitates flexible configuration and expansion according to the needs of different intangible cultural heritage crafts. Simultaneously, the use of domestically produced chips to replace imported chips significantly reduces hardware procurement costs while ensuring system performance meets the requirements of digital modeling of intangible cultural heritage crafts. These two improvements significantly reduce the overall cost of the digital modeling system for intangible cultural heritage crafts, making it more economical and competitive in the market. This enables more small and medium-sized traditional intangible cultural heritage enterprises to adopt the system, thereby promoting the application and popularization of digital technology for intangible cultural heritage crafts on a wider scale and facilitating the digital transformation of the intangible cultural heritage industry.

[0046] The AR real-time guidance system is used to intuitively display the operation steps and skills to the trainees. Through multimodal feedback, the trainees can more intuitively feel and understand the key points of the operation, thereby improving learning efficiency and effectiveness.

[0047] To address the challenges of high learning difficulty and long learning cycles in the transmission of intangible cultural heritage crafts, this invention employs an AR real-time guidance system. This system intuitively overlays and displays the operational steps and key techniques of intangible cultural heritage crafts within the learner's field of vision, enabling them to clearly and directly access operational guidance. Simultaneously, combined with a multimodal feedback mechanism, it allows learners to experience and understand the key operational points through visual, auditory, and other means, helping them to grasp the essence of the craft more quickly. This intuitive and diverse learning approach effectively reduces the learning difficulty, shortens the learning cycle, and significantly improves learning efficiency and effectiveness, contributing to the rapid transmission of intangible cultural heritage crafts and the cultivation of talent.

[0048] Digital twin modeling platforms can accurately simulate and intelligently adjust process optimization algorithms, predict potential problems in advance and optimize accordingly, thereby improving product quality and stability.

[0049] The digital twin modeling platform of this invention can accurately simulate the entire process of intangible cultural heritage crafts. By constructing a digital twin model that is highly consistent with the actual craft scenario, it realistically restores the physical properties of the craft materials and the dynamic changes in the craft operations. Simultaneously, the optimization algorithm on the platform can intelligently adjust and optimize the process parameters based on the simulation results and actual process data. Through this accurate simulation and intelligent optimization, potential problems in the process, such as product defects and inefficiencies, can be predicted in advance, and optimization solutions can be provided in a timely manner. This effectively reduces errors and waste in actual production, improves product quality stability and production efficiency, and provides strong support for the large-scale production and quality improvement of intangible cultural heritage crafts. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0052] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0053] Example 1

[0054] Combination Figure 1 This embodiment provides a digital modeling method for intangible cultural heritage crafts based on multimodal perception, including the following steps:

[0055] The system acquires action data and environmental data of the craftsman's operation, which are collected through a sensor system.

[0056] Based on the action data and environmental data, multimodal data synchronization and feature fusion are performed to generate a process parameter association model;

[0057] Based on the process parameter correlation model, a digital twin model of the intangible cultural heritage process is constructed, and the process parameters are optimized.

[0058] Real-time operation guidance information is generated based on the digital twin model, and the process results are evaluated for quality and optimized based on feedback.

[0059] The method of the present invention is described below with reference to specific steps.

[0060] Step 1: Synchronous Acquisition of Multi-Source Data

[0061] In this embodiment, acquiring the craftsman's action data and environmental data includes: collecting the craftsman's three-dimensional posture data and hand micro-strain data through a reconfigurable sensor array; collecting environmental temperature, humidity, and pressure data through an environmental sensing module; filtering the three-dimensional posture data, hand micro-strain data, and environmental data to generate preprocessed data; and transmitting the preprocessed data to an edge computing unit through a low-power communication module.

[0062] Data acquisition and synchronization:

[0063] Sensor data acquisition: A wearable flexible sensor array fits snugly against the craftsman's body, acquiring three-dimensional posture data in real time at a sampling frequency of 200Hz. The built-in 9-axis IMU and flexible strain sensor accurately capture the craftsman's hand micro-strain with an accuracy of ±0.05°. The environmental sensing module's infrared temperature (accuracy ±0.5℃), humidity (accuracy ±2%RH), and pressure (range 0-50N, accuracy ±0.1N) sensors simultaneously record environmental data.

[0064] Data preprocessing and transmission: The collected sensor data is first filtered to remove noise interference and improve data quality. Then, the data is transmitted to the edge computing unit via Bluetooth Low Energy 5.2. During transmission, the data is encrypted to ensure data security.

[0065] Dynamic Timestamp Alignment Algorithm: Due to the different sampling frequencies and transmission delays of motion capture sensors and environmental sensors, multimodal data streams suffer from time asynchrony. This invention employs a Dynamic Time Warping (DTW) algorithm, rather than a simple interpolation or sliding window method, aiming to effectively reduce timing errors in motion capture by using hand micro-strain sensing and the DTW algorithm. This lays the foundation for building an accurate process parameter correlation model. This embodiment achieves high-precision data synchronization, a crucial prerequisite for subsequent effective feature fusion. With FPGA hardware acceleration, a DTW-optimized multimodal data synchronization algorithm is used to align sensor data and environmental data in time. Specifically, aligning the motion data and environmental data using the DTW algorithm includes: assigning timestamps to the motion data and environmental data to generate timestamped data sequences; calculating the time difference between the data sequences based on the DTW algorithm; and adjusting the time difference of the data sequences through interpolation to generate synchronized data. The parallel computing capability of the FPGA enables the algorithm to process data in milliseconds, achieving multi-source data synchronization accuracy of <0.5ms, superior to the existing 1ms accuracy. The specific operation is as follows: assign a timestamp to each sensor data and environmental perception data, then use the DTW algorithm to calculate the time difference between data sequences, and adjust the data through interpolation and other methods to align data from different sources in time.

[0066] The specific calculation method is as follows:

[0067] ① Time difference calculation in Dynamic Time Warping (DTW) algorithm

[0068] First, the action data sequence With environmental data sequence Construct the distance matrix:

[0069]

[0070] in, Represents Euclidean distance

[0071] Subsequently, the cumulative distance matrix is ​​calculated using dynamic programming:

[0072]

[0073] Finally, the optimal time alignment path is obtained: It is used to measure the time difference between different modal data.

[0074] ② Interpolation adjustment method

[0075] To eliminate time differences between different modalities, a piecewise linear interpolation method is used: if a certain action data point With timestamp Its corresponding point on the environmental data timeline is and Then, aligned data is generated through linear interpolation:

[0076]

[0077] This yields a time-synchronized environmental data sequence. This interpolation method, while maintaining computational efficiency, can reduce the multimodal data synchronization error to less than 0.5ms.

[0078] In this example, the multimodal data synchronization and feature fusion based on the action data and environmental data includes: using a dynamic time warping algorithm to align the action data and environmental data in time to generate synchronized data; performing format conversion and standardization processing on the synchronized data based on a lightweight neural network inference engine; extracting features from the synchronized data through a long short-term memory network with a gated attention mechanism to generate fused features; and mapping the fused features to a process parameter association model based on a non-Euclidean graph convolutional network.

[0079] The specific multimodal feature extraction and fusion operations are as follows:

[0080] ① How the gating attention mechanism works

[0081] To highlight key modal features, a gated attention mechanism is introduced. Let the synchronized action features be... Environmental characteristics are The formula for calculating attention weights is:

[0082]

[0083]

[0084] The fusion features are:

[0085]

[0086] This mechanism ensures that the network can adaptively allocate the importance of action modalities and environment modalities under different process scenarios.

[0087] ②Formula for combining LSTM with attention mechanism

[0088] In an LSTM unit, the traditional forget gate, input gate, and output gate are defined as follows:

[0089]

[0090]

[0091]

[0092]

[0093]

[0094] In this patented invention, a gated attention mechanism is embedded into the hidden state update process of LSTM, defining the attention-enhanced hidden state:

[0095]

[0096] in, and These represent the hidden states of the LSTM in the action mode and the environment mode, respectively.

[0097] Final output fused features:

[0098]

[0099] This mechanism combines temporal dynamic modeling (LSTM) with intermodal feature weight adjustment (attention), thereby improving the accuracy and robustness of digital modeling of intangible cultural heritage crafts.

[0100] Step 2: Data Processing and Fusion

[0101] Edge computing unit processing: The edge computing unit is designed based on the RK3588 chip and incorporates a lightweight neural network inference engine. Upon receiving synchronized data, it first performs format conversion and standardization to meet the requirements of subsequent algorithm processing. Then, it utilizes an improved LSTM network (introducing a gated attention mechanism) to extract and fuse features from the multimodal data. The gated attention mechanism addresses the problem of traditional LSTM's insufficient modeling of long-term temporal dependencies in multimodal data by dynamically adjusting the weights of each modality, thus more accurately capturing key features in the data.

[0102] Knowledge Graph Construction and Update: The relationships between process parameters (such as kiln temperature, intensity, and humidity) are often complex, nonlinear, and do not satisfy the Euclidean space assumption. This invention employs a non-Euclidean graph convolutional network (NCN) instead of a traditional fully connected neural network or Euclidean GCN, aiming to better model these complex and unstructured process parameter relationships, thereby constructing a more accurate process knowledge graph. By introducing a NCN, the process parameter relationship model constructed in this invention can more accurately reveal the implicit relationships between complex process parameters, support dynamic model updates and cross-process transfer learning, and improve the system's generalization ability and knowledge reuse rate. The process parameter relationship model is mapped from the fused features using a NCN, including constructing the model based on a NCN, mapping the fused feature data to the knowledge graph, and establishing relationships between different process parameters. This model supports dynamic updates; when new data arrives, it can adjust the graph structure and weights in real time, enabling cross-process transfer learning and applying learned knowledge to other similar processes.

[0103] The specific steps are as follows:

[0104] ①Merge feature mapping to process parameter correlation

[0105] Let the features after fusion be .in, Indicates the number of process parameter nodes. This represents the feature dimension. The update formula for a non-Euclidean graph convolutional network is:

[0106]

[0107] in, Represents a node The neighborhood group, The normalization coefficient is... For the first The weight matrix of the layer, is the activation function. This process maps multimodal fusion features to non-Euclidean geometric relationships between process parameter nodes.

[0108] ② Dynamically update the model structure and weights

[0109] When new data comes in, a graph-structured incremental update rule is used:

[0110] If the new process parameter node If an edge appears, add a node to the graph and create an edge based on the similarity function:

[0111]

[0112] in, This refers to the temperature parameter.

[0113] The weights of the graph convolutional network are updated via online gradient descent:

[0114]

[0115] in, For learning rate, The loss function is defined as . This rule enables the adaptive evolution of the process knowledge graph.

[0116] ③ Cross-process transfer learning

[0117] The source technology field is The target process area is By aligning the distributions of the source and target domains, a transfer mapping function is defined. :

[0118]

[0119]

[0120] in, and These are the low-dimensional embedding representations of the source and target domains, respectively.

[0121] To measure the compatibility between the source and target processes, a joint loss function is defined:

[0122]

[0123] in, This indicates the task loss of the target process (such as the error in process parameter prediction). Maximum Mean Discrepancy is used to measure the difference in feature distributions between the source and target domains.

[0124]

[0125] in, To balance the parameters, a balance is needed between task loss and migration alignment loss.

[0126] By using this loss function, the system can achieve cross-process knowledge transfer and sharing while maintaining the accuracy of the target process prediction.

[0127] Step 3: Digital Twin Model Construction and Optimization

[0128] In this embodiment of the invention, the step of constructing a digital twin model of an intangible cultural heritage craft based on the process parameter association model and optimizing the process parameters includes: initializing the digital twin model of the intangible cultural heritage craft in a 3D visualization engine and setting the physical properties of the process materials; simulating the physical properties of the process materials in the digital twin model based on a smooth particle fluid dynamics algorithm; automatically searching for the optimal solution of the process parameters using a multi-objective Bayesian optimization framework with process quality and efficiency as objectives; and updating the digital twin model based on the optimal solution.

[0129] The physical properties (such as fluidity and deformability) of materials used in intangible cultural heritage crafts (e.g., ceramic clay, embroidered silk) are core factors affecting the quality of finished products. This invention employs the Smooth Particle Hydrodynamics (SPH) algorithm, a meshless Lagrangian simulation method rather than simple rigid body dynamics simulation. It aims to simulate the microscopic physical changes of craft materials during processing with high fidelity, providing a realistic virtual environment for parameter optimization. By combining SPH simulation with multi-objective Bayesian optimization, the digital twin model can perform high-fidelity simulation of the process and automatically seek optimization, improving the efficiency and effectiveness of process parameter optimization and providing a data-driven solution for enhancing the quality of intangible cultural heritage products.

[0130] Process optimization often requires finding a balance among multiple conflicting objectives such as quality, efficiency, and cost. This invention employs a multi-objective Bayesian optimization framework, rather than traditional grid search or single-objective optimization, aiming to efficiently search for Pareto optimal solutions with the fewest possible experiments, providing craftsmen with a variety of feasible and high-quality process parameter selection options.

[0131] The specific model details are as follows:

[0132] ① Model initialization and parameter setting

[0133] In the Unity3D engine, an initial digital twin model of the intangible cultural heritage craft was built. The physical properties and parameters of the model were set according to the actual craft requirements, such as the material properties of ceramics and the stitch parameters of embroidery. Simultaneously, a process material simulation algorithm based on SPH (Smooth Particle Hydrodynamics) was integrated to provide simulations of the physical properties of different materials for the model.

[0134]

[0135]

[0136] in, For particle density, For strength, For kernel function, For smooth length.

[0137] ② Quantitative indicators of process quality and efficiency

[0138] The process quality index (Q) is measured by comparing the error between the digital twin simulation results and the ideal process sample.

[0139]

[0140] in, The feature vector representing the simulation process results, This represents the feature vector of the reference standard process. The closer the value is to 1, the higher the quality of the process.

[0141] Process efficiency (E) is defined by the ratio of processing time to energy consumption.

[0142]

[0143] in, For the process completion time, Energy consumption per unit of process and This is the weighting factor. A higher value indicates a superior manufacturing process.

[0144] ③ Multi-objective Bayesian optimization framework

[0145] The covariance function (kernel function) of the prior model: A Gaussian process (GP) is used as the prior model for Bayesian optimization. Given a process parameter input vector... Its covariance function (kernel function) uses radial basis functions (RBF):

[0146]

[0147] in, For signal variance, This is a length scale hyperparameter.

[0148] The acquisition function for optimal solution search is the Expected Improvement (EI) acquisition function.

[0149]

[0150] Its closed form is:

[0151]

[0152] in, and The mean and standard deviation are predicted by the Gaussian process. Given the current known optimal objective function value, To explore parameters, and These are the cumulative distribution function and probability density function of the standard normal distribution, respectively.

[0153] In multi-objective scenarios, the Pareto Front principle is adopted to integrate different objectives (process quality Q and efficiency E) into the Bayesian optimization framework, thereby achieving joint optimization of multi-objective process parameters.

[0154] Model initialization and parameter settings: An initial digital twin model of the intangible cultural heritage craft is built in the Unity3D engine. The physical properties and parameters of the model are set according to the actual craft requirements, such as the material properties of ceramics and the stitch parameters of embroidery. Simultaneously, a process material simulation algorithm based on SPH (Smooth Particle Hydrodynamics) is integrated to provide simulations of the physical properties of different materials for the model.

[0155] Data-driven model optimization: Data processed by the edge computing unit is transmitted to the digital twin modeling platform. The platform utilizes a multi-objective Bayesian optimization framework to automatically search for Pareto optimal solutions for process parameters within the digital twin environment. Specifically, process parameters are used as optimization variables, with process quality and efficiency as optimization objectives. The Bayesian optimization algorithm continuously adjusts parameter values, simulating the process under different parameter combinations, ultimately yielding a set of optimal process parameter settings to drive the optimization and upgrading of the digital twin model.

[0156] Step 4: Intelligent Assistance Applications and Feedback

[0157] In this embodiment of the invention, the step of generating real-time operation guidance information based on the digital twin model and performing quality assessment and feedback optimization of the process results includes: generating real-time operation guidance information based on a lightweight semantic segmentation model and displaying it through an augmented reality device; performing quality assessment of the process results based on a defect detection model using transfer learning and generating assessment results; feeding the assessment results back to the digital twin model and adjusting the process parameters; and updating the real-time operation guidance information based on the adjusted process parameters. Specifically:

[0158] ① Lightweight semantic segmentation model formula

[0159] The Edge-SegNet employs depthwise separable convolution and channel pruning techniques to achieve lightweight design.

[0160] Depthwise separable convolution: .in, This represents a depthwise convolution operation. This indicates a pointwise convolution operation. and These are depthwise convolution kernels and pointwise convolution kernels, respectively.

[0161] Channel pruning: based on channel weights The sparse regularization constraint preserves channels with high significance: .in, This is the pruning threshold. To preserve the channel set.

[0162] ②AR coordinate transformation formula

[0163] The operation guidance information displayed in AR devices uses a projection transformation from three-dimensional coordinates to two-dimensional screen coordinates:

[0164]

[0165] in, Let be the 3D coordinates of the point in the process scene, and let be the rotation matrix and translation vector of the camera, respectively. For the camera intrinsic parameter matrix, Including focal length and principal point offset, For the final two-dimensional pixel coordinates on the AR display

[0166] This formula ensures that the operational guidance information in the digital twin model is precisely aligned with the craftsman's actual operational scenario, guaranteeing the real-time nature and accuracy of AR guidance.

[0167] Real-time AR guidance: The optimized results of the digital twin model are used to achieve real-time AR annotation through a lightweight semantic segmentation model (Edge-SegNet). Operation steps and guidance information are overlaid onto the craftsman's actual operation scenario with a latency of <20ms. By wearing AR glasses, the craftsman can intuitively see the operation instructions for each step, improving the accuracy and efficiency of their work.

[0168] Quality Assessment and Feedback: A defect detection model based on transfer learning is used to assess the quality of actually produced products. This model is trained using training data with a 70% cross-process reuse rate, enabling it to accurately identify product defects with an accuracy rate of 98.5%. The assessment results are fed back to the digital twin modeling platform and edge computing unit in real time. The platform further optimizes the digital twin model and process parameters based on the feedback information, while the unit adjusts subsequent data processing and analysis, forming a closed-loop intelligent optimization system.

[0169] It should be noted that the advantages of the method of the present invention are as follows:

[0170] Step 2: Feature Extraction and Fusion

[0171] Improved LSTM network: Introducing a gated attention mechanism (Gated Attention-LSTM) to address the problem that traditional LSTM is insufficient for modeling long-term temporal dependencies in multimodal data.

[0172] Knowledge graph construction: A process parameter association model is built based on a non-Euclidean graph convolutional network (Non-Euclidean GCN), which supports dynamic updates and cross-process transfer learning.

[0173] Step 3: Digital Twin Model Construction

[0174] Physics engine optimization: Integrate a process and material simulation algorithm based on SPH (Smooth Particle Hydrodynamics) into Unity3D to support high-precision simulation of different materials such as ceramics and embroidery.

[0175] Reinforcement learning optimization: A multi-objective Bayesian optimization framework is adopted to automatically search for Pareto optimal solutions for process parameters in a digital twin environment.

[0176] Step 4: Intelligent Assistance Applications

[0177] AR Real-time Guidance: Real-time AR annotation of operation steps is achieved through a lightweight semantic segmentation model (Edge-SegNet) with a latency of <20ms.

[0178] Quality assessment: The defect detection model based on transfer learning (with a training data reuse rate of >70% across processes) achieved an accuracy of 98.5%.

[0179] Another embodiment of the present invention provides a digital modeling system for intangible cultural heritage crafts based on multimodal perception, used to implement the method in embodiment 1, including:

[0180] A motion capture module with a reconfigurable sensor array is used to acquire motion data and environmental data of the craftsman's operation;

[0181] The environment perception module of the multi-source heterogeneous data fusion engine is used to perform multimodal data synchronization and feature fusion based on the action data and environmental data to generate a process parameter association model;

[0182] An adaptive edge computing unit is used to construct a digital twin model of the intangible cultural heritage process based on the process parameter association model, and to optimize the process parameters;

[0183] An interactive digital twin modeling platform is used to generate real-time operation guidance information based on the digital twin model, and to perform quality assessment and feedback optimization of process results.

[0184] The following is a detailed analysis of the effectiveness of the method of the present invention.

[0185] 1. Modeling accuracy verification:

[0186] Motion capture error test: A standard 3D motion capture laboratory was set up in the testing environment, using a high-precision optical motion capture system as a benchmark. Ten artisans with different skill levels were selected as test subjects, and they performed representative intangible cultural heritage operations, such as ceramic forming and embroidery. Data was collected simultaneously using the wearable flexible sensor array and optical motion capture system of this invention, with a collection time of 30 minutes per person per session, for a total of 100 sets of data. Using an error calculation method based on rigid body kinematics, the 3D posture data collected by the flexible sensor array was compared and analyzed with the data from the optical motion capture system, and the motion capture error was found to be less than 1.8%. This error mainly comes from the strain compensation algorithm of the flexible sensor, which accurately compensates for the deformation of the sensor during complex operations, and the FPGA time synchronization optimization, which ensures accurate alignment of multi-source data, reduces timing differences during data acquisition and transmission, and thus improves the accuracy of motion capture.

[0187] Environmental sensing error testing: The environmental sensing module was placed in an environmental simulation chamber with known temperature, humidity, and pressure to simulate different process production environments, such as high temperature, high humidity, and low pressure. Each environmental condition was tested 20 times, and the module's measurement data was recorded and compared with data from high-precision standard sensors in the environmental simulation chamber. Mean absolute error (MAE) and root mean square error (RMSE) were used as error evaluation indicators. The calculated temperature measurement error was ±0.3℃, humidity measurement error was ±1%RH, and pressure measurement error was ±0.05N, verifying the high-precision measurement performance of the environmental sensing module. This is attributed to the selection of high-precision sensors and the optimized design of the data acquisition circuit, ensuring the accuracy of the environmental sensing data and providing reliable environmental parameter support for the digital modeling of intangible cultural heritage processes.

[0188] 2. Process optimization effect:

[0189] Case Study on Improved Yield Rate: In the ceramic firing case, a process material simulation algorithm based on SPH (Smooth Particle Hydrodynamics) was integrated into the digital twin modeling platform to perform high-precision simulation of the drying and firing processes of ceramic blanks. Simultaneously, a multi-objective Bayesian optimization framework was used, with ceramic strength, color, density, and other quality indicators as optimization targets, to automatically search for Pareto optimal solutions for the process parameters. Actual production verification showed that the optimized process parameters increased the yield rate of ceramic products from 65%-70% to 75%-78%, an average improvement of 15%-22%. The specific calculation method involved firing 1000 ceramic blanks using both the pre- and post-optimization process parameters, counting the number of qualified finished products, and calculating the improvement in yield rate. This effect mainly stems from the digital twin modeling platform's accurate simulation of the process and intelligent adjustment of the optimization algorithm. By simulating the ceramic firing process under different process parameters, potential problems are predicted and optimized in advance, thereby improving product quality and stability.

[0190] The optimization process and its corresponding technical points: During the process optimization, the high-precision motion capture module and environmental perception module accurately collect the craftsman's operation data and firing environment data, providing rich raw data support for the digital twin modeling platform. The edge computing unit processes this data in real time, ensuring the timeliness and accuracy of the data, and providing reliable input for the optimization algorithm. The multi-objective Bayesian optimization framework, combined with the SPH material simulation algorithm, comprehensively and deeply optimizes the process parameters, ultimately determining the optimal combination of process parameters. These technical points and modules work together to significantly improve the yield rate, not only increasing production efficiency and economic benefits, but also providing strong technical support for the innovative development of intangible cultural heritage crafts.

[0191] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A digital modeling method for intangible cultural heritage crafts based on multimodal perception, characterized in that, include: The reconfigurable sensor array collects three-dimensional posture data and micro-strain data of the craftsman's hand as motion data, and the environmental sensing module collects environmental temperature, humidity and pressure data as environmental data. The three-dimensional posture data, hand micro-strain data, and ambient temperature, humidity, and pressure data are time-aligned using a dynamic time warping algorithm to generate synchronized data. The synchronized data is format-converted and standardized based on a lightweight neural network inference engine. By introducing a gated attention mechanism into a long short-term memory network, features are extracted from the processed synchronous data to generate fused features; The fused features are mapped to a process parameter correlation model that characterizes the nonlinear correlation between process parameters based on a non-Euclidean graph convolutional network. Based on the process parameter association model, a digital twin model of intangible cultural heritage process is constructed in a 3D visualization engine, and the physical properties of the process materials in the digital twin model are simulated based on the smooth particle fluid dynamics algorithm. Using a multi-objective Bayesian optimization framework with process quality and efficiency as objectives, the system automatically searches for the optimal solution of process parameters and updates the digital twin model based on the optimal solution. Based on the digital twin model, real-time operation guidance information is generated through a lightweight semantic segmentation model and displayed through an augmented reality device; A defect detection model based on transfer learning is used to evaluate the quality of process results and generate evaluation results. The evaluation results are fed back to the digital twin model to adjust the process parameters, and the real-time operation guidance information is updated based on the adjusted process parameters.

2. The digital modeling method for intangible cultural heritage crafts based on multimodal perception as described in claim 1, characterized in that, The generation of synchronization data includes: Assign timestamps to the action data and environment data to generate a time-stamped data sequence; The time difference between the data sequences is calculated based on the dynamic time warping algorithm; Synchronous data is generated by adjusting the time difference of the data sequence through interpolation.

3. The digital modeling method for intangible cultural heritage crafts based on multimodal perception as described in claim 1, characterized in that, The non-Euclidean graph convolutional network maps the fused features into a process parameter correlation model representing the nonlinear correlation between process parameters, including: The fused features are input into a non-Euclidean graph convolutional network to generate the correlation between process parameters; A process parameter association model is constructed based on the aforementioned relationships; When new fusion features are received, the structure and weights of the process parameter association model are dynamically updated; Cross-process transfer learning is achieved based on the process parameter association model.

4. The digital modeling method for intangible cultural heritage crafts based on multimodal perception as described in claim 1, characterized in that, The automatic search for optimal solutions to process parameters using a multi-objective Bayesian optimization framework, with process quality and efficiency as objectives, includes: A multi-objective optimization problem is established with process quality and process efficiency as optimization objectives. A Bayesian optimization algorithm is used to automatically search for Pareto optimal solutions for process parameters in a digital twin model.

5. The digital modeling method for intangible cultural heritage crafts based on multimodal perception as described in claim 1, characterized in that, The defect detection model based on transfer learning performs quality assessment of the process results, including: The model is pre-trained on various process data as the base model; The transfer learning algorithm is used to adapt the basic model to the defect detection task of a specific intangible cultural heritage craft.

6. A digital modeling system for intangible cultural heritage crafts based on multimodal perception, used to implement the method as described in any one of claims 1 to 5, characterized in that, include: A motion capture module with a reconfigurable sensor array is used to acquire the craftsman's three-dimensional posture data and hand micro-strain data; The environmental sensing module is used to collect ambient temperature, humidity, and pressure data. A multi-source heterogeneous data fusion engine is used to perform multimodal data synchronization and feature fusion based on the action data and environmental data to generate a process parameter correlation model; An adaptive edge computing unit is used to construct a digital twin model of the intangible cultural heritage process based on the process parameter association model, and to optimize the process parameters; An interactive digital twin modeling platform is used to generate real-time operation guidance information based on the digital twin model, and to perform quality assessment and feedback optimization of process results.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 5.

Images