Intelligent decision-making method and device for cross-modal data, equipment and storage medium

Abstract

The invention provides an intelligent decision-making method and device for cross-modal data, equipment and a storage medium, and the method comprises the steps: carrying out the blockchain storage of collected multi-modal process data, and generating an on-chain hash certificate; performing cross-modal space-time alignment and feature fusion on the multi-modal process data according to the space-time metadata analyzed by the Hash certificate on the chain to generate a process knowledge graph; constructing an interactive teaching model according to the process knowledge graph; determining a multi-dimensional contribution measurement value according to the use data of the interactive teaching model; and based on the multi-dimensional contribution magnitude and the hierarchical smart contract architecture, determining the data access permission through a community voting mechanism. Through implementation of the scheme, the process knowledge graph and the interactive teaching model, the inheritor operation data and the material response data form a causal association model, a traceable decision basis is provided for accurate reproduction of an endangered process, and meanwhile, the intelligent contract can automatically execute a decision to ensure high efficiency and transparency of the decision process.

Description

Technical Field

[0001] This application relates to the field of digital protection of processes, and particularly to an intelligent decision-making method, device, equipment and storage medium for cross-modal data. Background Art

[0002] Currently, the inheritance of endangered handicrafts faces two core contradictions: it is difficult for the physical experience of traditional crafts to be quantifiably preserved without the individual inheritors, and the scattered inheritance behaviors of skills cannot be dynamically bound to the community cultural values. For example, existing technologies for the digital protection of crafts such as bamboo weaving and lacquerware mostly use single video recording or static 3D scanning, which can neither capture the coupling relationships between material deformation (such as the change in bamboo strip toughness), tool trajectories (such as the fine adjustment of carving knife angles), and human mechanics (such as hand pressure distribution) during the process, nor have a skill value evaluation system based on community consensus, resulting in the following problems: 1) The implicit experience of inheritors (such as the perception of firing temperature and the threshold of material strain) is permanently lost with the extinction of individuals because it cannot be quantified; 2) Due to the unclear ownership and lack of incentives for process inheritance data, it is difficult to form a sustainable community collaboration ecosystem. Summary of the Invention

[0003] This application provides an intelligent decision-making method, device, equipment and storage medium for cross-modal data, which is used to solve the problem in related technologies that it is difficult for the physical experience of traditional crafts to be quantifiably preserved without the individual inheritors.

[0004] In the first aspect of this application, an intelligent decision-making method for cross-modal data is provided. The intelligent decision-making method for cross-modal data includes: Generating an on-chain hash certificate by performing blockchain evidence deposit on the collected multi-modal process data; Performing cross-modal spatio-temporal alignment and feature fusion on the multi-modal process data according to the spatio-temporal metadata parsed from the on-chain hash certificate, and generating a process knowledge graph; Constructing an interactive 3D teaching model according to the process knowledge graph; Determining a multi-dimensional contribution metric value according to the usage data of the interactive 3D teaching model; Based on the multi-dimensional contribution metric value and a hierarchical intelligent contract architecture, determining data access permissions through a community voting mechanism.

[0005] Optionally, in the first implementation manner of the first aspect of this application, the step of generating an on-chain hash certificate by performing blockchain evidence deposit on the collected multi-modal process data includes: Performing modal classification and spatio-temporal marking on the multi-modal process data through an edge computing node, and generating a modal data packet carrying coordinates and a timestamp; wherein, the modal data packet includes: video data, audio data, text data, and sensor data; Perform key-frame action capture on the video data and extract the time-frequency feature vectors of the hand movement trajectory; Perform voiceprint noise reduction on the audio data, align the denoised audio data with the text data on the time axis, and generate a synchronized audio-text data pair; Extract the Mel cepstral coefficients of the synchronized audio-text data pair and generate a composite audio fingerprint hash in combination with the keyword hash values of the aligned text; Input the time-frequency feature vectors and the composite audio fingerprint hash into a heterogeneous data fusion module, and construct structured metadata in combination with the dynamic threshold calibration results of the sensor data; Based on the distributed consensus protocol of the consortium chain nodes, perform multi-signature binding on the structured metadata and the IPFS storage paths of each modality data packet to generate a cross-modal hash certificate chain.

[0006] Optionally, in the second implementation manner of the first aspect of this application, the step of performing cross-modal spatio-temporal alignment and feature fusion on the multi-modal process data according to the spatio-temporal metadata parsed from the on-chain hash certificate to generate a process knowledge graph includes: Extract the coordinate deviation compensation coefficient of the video data and the timestamp drift amount of the audio data from the on-chain hash certificate; Perform spatio-temporal calibration on the multi-modal process data according to the coordinate deviation compensation coefficient and the timestamp drift amount to generate a spatio-temporally synchronized data queue; Generate cross-modal topological mapping rules according to the spatio-temporally synchronized data queue and the dynamic association constraints of the multi-modal process data; Construct a cross-modal hypergraph structure according to the cross-modal topological mapping rules; Calculate the process material state transition probability of the nodes of the cross-modal hypergraph structure, and generate a process knowledge graph in combination with the deformation parameter threshold of the sensor data.

[0007] Optionally, in the third implementation manner of the first aspect of this application, the step of constructing an interactive 3D teaching model according to the process knowledge graph includes: Perform kinematic chain analysis on the action-material association nodes in the process knowledge graph, and extract the hand joint angle sequence and the material deformation parameter threshold; Convert the hand joint angle sequence into a rigid body dynamics constraint equation, and determine the drive parameter matrix of the process action decomposition engine in combination with the material deformation parameter threshold; Construct an interactive 3D model corresponding to the drive parameter matrix by calling the real-time deformation simulator of the process action decomposition engine.

[0008] Optionally, in the fourth implementation manner of the first aspect of the present application, the method further includes: According to the critical fracture strain value in the material deformation parameter threshold, the real-time material deformation parameters in the user operation trajectory are segmented into discrete deformation gradient sequences according to a time window; Perform dynamic time warping matching on the deformation gradient sequence and the standard deformation trajectory in the process knowledge graph to determine the deformation deviation integral value within each time window; When the deformation deviation integral value exceeds the tolerance threshold corresponding to the material elastic modulus, generate a pulse frequency modulation parameter for haptic feedback according to the corresponding deformation deviation; Input the pulse frequency modulation parameter into the resonant circuit of the haptic feedback device to trigger the haptic feedback device to correct and control the deformation deviation.

[0009] Optionally, in the fifth implementation manner of the first aspect of the present application, the step of determining the multi-dimensional contribution metric value according to the usage data of the interactive 3D teaching model includes: Determine the spatio-temporal deviation quantification matrix between the user operation trajectory and the standard action nodes of the process knowledge graph through the dynamic time warping algorithm; Input the spatio-temporal deviation quantification matrix into the material deformation prediction model to output the microscopic structure crack propagation probability distribution map; Input the spatio-temporal deviation quantification matrix into the preset material deformation prediction model to generate the material deformation risk probability distribution map; Based on the deviation quantification matrix and the deformation risk probability distribution map, construct a contribution decision tree through a multi-objective optimization algorithm; Perform fault-tolerant consensus verification between the nodes of the consortium chain on the output solution set of the contribution decision tree to generate the multi-dimensional contribution metric value.

[0010] Optionally, in the sixth implementation manner of the first aspect of the present application, the step of determining the data access permission through the community voting mechanism based on the multi-dimensional contribution metric value and the hierarchical smart contract architecture includes: Perform asymmetric zero-knowledge proof verification on the permission change proposal in the hierarchical smart contract architecture to generate a proposal validity identifier; Calculate the Shapley value for community member nodes according to the multi-dimensional contribution metric value, and dynamically allocate voting weight factors; When it is detected that the deviation degree between the voting weight factor and the historical contribution record in the blockchain deposit exceeds the preset threshold, activate the iris biometric verification module against the Sybil attack; Drive the light field camera array of the distributed hardware terminal through the iris verification module to collect live verification data and trigger the smart contract execution engine to update the white list of data access permissions.

[0011] The second aspect of this application provides an intelligent decision-making device for cross-modal data. The intelligent decision-making device for cross-modal data includes: A first generation module for generating an on-chain hash certificate by performing blockchain storage on the collected multi-modal process data; A second generation module for performing cross-modal spatio-temporal alignment and feature fusion on the multi-modal process data according to the spatio-temporal metadata parsed from the on-chain hash certificate to generate a process knowledge graph; A construction module for constructing an interactive 3D teaching model according to the process knowledge graph; A first determination module for determining a multi-dimensional contribution metric value according to the usage data of the interactive 3D teaching model; A second determination module for determining data access permissions through a community voting mechanism based on the multi-dimensional contribution metric value and a hierarchical smart contract architecture.

[0012] The third aspect of the embodiments of this application provides an electronic device, including a memory and a processor. Among them, the processor is used to execute a computer program stored on the memory. When the processor executes the computer program, it implements the steps in the intelligent decision-making method for cross-modal data provided in the first aspect of the embodiments of this application.

[0013] The fourth aspect of the embodiments of this application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps in the intelligent decision-making method for cross-modal data provided in the first aspect of the embodiments of this application.

[0014] In summary, according to an intelligent decision-making method, device, device and storage medium for cross-modal data provided by the solution of this application, an on-chain hash certificate is generated by performing blockchain storage on the collected multi-modal process data; according to the spatio-temporal metadata parsed from the on-chain hash certificate, cross-modal spatio-temporal alignment and feature fusion are performed on the multi-modal process data to generate a process knowledge graph; an interactive 3D teaching model is constructed according to the process knowledge graph; a multi-dimensional contribution metric value is determined according to the usage data of the interactive 3D teaching model; based on the multi-dimensional contribution metric value and a hierarchical smart contract architecture, data access permissions are determined through a community voting mechanism. Through the implementation of the solution of this application, cross-modal spatio-temporal alignment is used to construct a process knowledge graph and an interactive 3D teaching model, so that the operation data of the inheritor and the material response data form a causal association model, providing a traceable decision-making basis for the accurate reproduction of endangered processes. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic flowchart of an intelligent decision-making method for cross-modal data provided by an embodiment of the present application; Figure 2 It is a schematic diagram of program modules of an intelligent decision-making device for cross-modal data provided by an embodiment of the present application; Figure 3 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0016] To make the objectives, features, and advantages of the present application more obvious and understandable, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts shall fall within the protection scope of the present application.

[0017] To solve the problem that it is difficult to quantitively preserve the physical experience of traditional processes in the related art without relying on individual inheritors, an embodiment of the present application provides an intelligent decision-making method for cross-modal data, as Figure 1 It is a schematic flowchart of the intelligent decision-making method for cross-modal data provided by this embodiment. The intelligent decision-making method for cross-modal data includes the following steps: Step 110: Generate an on-chain hash certificate by performing blockchain certification on the collected multi-modal process data.

[0018] Specifically, in the digital protection scenario of endangered handicrafts, multi-modal process data (referring to the full-element information of the process synchronously collected through sensors, imaging devices, and Internet of Things terminals, including multi-source heterogeneous data such as three-dimensional motion trajectories, material deformation parameters, environmental temperature and humidity, and audio guidance information) needs to undergo trusted digital evidence preservation processing. The edge computing nodes deployed in the data collection stage first standardize and encapsulate the original data to form a structured data packet containing timestamps, spatial coordinates, and metadata descriptions. Subsequently, through blockchain digital evidence preservation technology, the data packet is input into a hash function to generate a digital fingerprint, and this fingerprint is jointly written into the distributed ledger of the blockchain network along with the data feature description. During this process, the smart contract automatically triggers multiple verification mechanisms, including but not limited to data integrity verification (verifying whether the data has been tampered with by comparing hash values) and spatio-temporal consistency authentication (verifying the matching degree between the sensor time series and the process action logic). The data packets that pass the verification are packaged into blockchain blocks and permanently stored. The finally generated on-chain hash certificate contains core information such as the data fingerprint, digital evidence preservation time, blockchain height coordinates, and verification node signatures. This certificate serves as the unique identifier of the process data, which can not only prevent the malicious tampering of the inheritor's experience data through the decentralized feature but also provide a traceable ownership certificate for subsequent cross-institutional collaboration.

[0019] Optionally, the multi-modal process data processing flow starts with the modal classification and spatio-temporal marking of the edge computing node. The edge computing node, as a computing unit deployed locally on the process equipment (such as a ceramic throwing machine or an embroidery machine), is equipped with a multi-channel data acquisition card and a real-time processing chip. Facing the hand movement video captured by a 4K industrial camera, the process guidance audio collected by a high-sensitivity microphone array, the operation specification text in an electronic document, and the six-axis inertial sensor data (acceleration, angular velocity), the node first performs modal classification through an improved residual neural network. The input layer of this network uses an adaptive bandwidth filter bank to analyze the mixed signals with different sampling rates (such as 30fps video and 100Hz sensor data), and marks the data as video, audio, text, or sensor modality. Spatio-temporal marking relies on a high-precision clock chip (error ±1μs) and an ultra-wideband positioning module, and the latter uses nanosecond-level pulse signals to measure the device spacing. For example, in the ceramic throwing scenario, the center of the turntable is set as the origin (0, 0, 0) of the three-dimensional coordinate system, and the camera is positioned at (0.5m, 1.2m, 0.8m), finally generating a data packet containing the modal type, spatial coordinates (x, y, z), and UTC timestamp.

[0020] Key frame action capture technology is used for video data processing. Key frames refer to the video frames with sudden state changes during the process operation (such as the moment when the embroidery needle tip pierces the fabric). An algorithm optimized based on the MediaPipe framework (an open-source cross-platform multimedia processing framework) tracks the three-dimensional motion trajectories of 21 hand joints, and the trajectory function is: , Among them, represents the cubic spline interpolation function of the th joint point; is the weight coefficient.

[0021] Using the complex Morlet wavelet transform, calculate the distribution of the motion energy of each joint point in the time-frequency plane and the trajectory curvature characteristics to determine the time-frequency feature vector: , , Among them, is the first derivative of the trajectory, representing the instantaneous velocity; is the second derivative of the trajectory, representing the acceleration; reflects the curvature change of the trajectory, and is used to capture the smoothness and mutation characteristics of hand movements; is the time-frequency feature vector.

[0022] Audio processing uses a deep clustering voiceprint noise reduction model to separate the process guidance speech and environmental noise (such as the sound of the billet casting machine motor) through adversarial training. The denoised audio and the operation text are time-aligned by maximizing mutual information: , Among them, represents the mutual information between the audio frame and the text segment .

[0023] When the mutual information exceeds the threshold When establishing a synchronization relationship (such as matching the timestamp of the "glazing" voice with the text operation instructions). The Mel Frequency Cepstral Coefficients (MFCC) extract 23-dimensional voice features, and through exclusive-or operation with the text keyword hash value (such as the hash of "glaze ratio: SiO2 65%" by SHA-256), a 128-bit composite audio fingerprint is generated, which can uniquely identify a specific process stage (such as the audio-text combination of celadon glaze preparation guidance). The heterogeneous data fusion module receives video time-frequency features, composite audio fingerprints, and sensor data. When receiving sensor data, dynamic threshold calibration of the sensor data is required. For example, during the process, the temperature usually fluctuates to some extent. If the original temperature data is directly used, it may lead to false alarms (such as misjudging abnormal temperature changes). The calibrated temperature adjusts the sensitivity so that the system can more accurately detect real abnormal situations. The fusion process maps the video features (21 joint points, 3 axes, 100 time points), audio fingerprints (128 bits), and calibrated temperature data to 512-dimensional structured metadata through tensor splicing, and outputs a 256-dimensional feature vector after dimensionality reduction by the fully connected layer. Then, the consortium chain nodes adopt an optimized Practical Byzantine Fault Tolerance (PBFT) consensus protocol, and authorized nodes (such as certified craftsmen, quality inspection agencies) perform multi-signature on the structured metadata and the IPFS (InterPlanetary File System, distributed hypermedia distribution protocol) storage path. Ensure that only the holders of legitimate public keys can participate in the evidence storage. The finally generated cross-modal hash certificate chain adopts a three-layer hash binding: the hash of the original data IPFS path, the metadata hash, and the random number Nonce are generated by three SHA-3 operations to generate a unique identifier. For example, in the celadon firing process, the hash values of the kiln temperature data packet, the video frames of the clay shaping action, and the glaze document are anchored in the block header through a Merkle tree (a tree-shaped data structure based on a hash function) to form an anti-tampering process traceability chain.

[0024] Step 120: According to the spatio-temporal metadata parsed from the on-chain hash certificate, perform cross-modal spatio-temporal alignment and feature fusion on the multi-modal process data to generate a process knowledge graph.

[0025] Specifically, achieving spatio-temporal alignment and feature fusion of multi-modal process data based on spatio-temporal metadata extracted from on-chain hash certificates is the core process of constructing a process knowledge graph. Spatio-temporal metadata refers to the time stamps and spatial location information embedded in blockchain evidence storage. Cross-modal spatio-temporal alignment requires adjusting data streams from different sources to a unified spatio-temporal coordinate system. For example, millisecond-level time synchronization is performed between gesture video frames captured by a high-speed camera during the ceramic drawing process and the data stream of the clay deformation sensor, and at the same time, the spatial coordinate offset is calibrated according to the device deployment location. Feature fusion extracts the correlation of each modal data through a deep learning model. For example, the time-series data of the drawing speed is jointly encoded with the image features of the clay stress distribution to generate a feature vector representing the key stages of the process. High-dimensional semantic feature vectors are extracted through a deep neural network, and these vectors are constructed into a graph structure, where the nodes represent process steps or material states, and the edges represent the correlations between them. This graph intuitively reflects the internal connections between various data in the process, supporting process optimization and decision-making analysis.

[0026] In an optional implementation manner of this embodiment, the steps of performing cross-modal spatio-temporal alignment and feature fusion on multi-modal process data according to the spatio-temporal metadata parsed from the on-chain hash certificate to generate a process knowledge graph include: extracting the coordinate deviation compensation coefficient of video data and the timestamp drift amount of audio data from the on-chain hash certificate; performing spatio-temporal calibration on the multi-modal process data according to the coordinate deviation compensation coefficient and the timestamp drift amount to generate a spatio-temporal synchronization data queue; generating a cross-modal topological mapping rule according to the spatio-temporal synchronization data queue and the dynamic association constraint of the multi-modal process data; constructing a cross-modal hypergraph structure according to the cross-modal topological mapping rule; calculating the process material state transition probability of the nodes of the cross-modal hypergraph structure, and combining the deformation parameter threshold of the sensor data to generate a process knowledge graph.

[0027] Specifically, the video data coordinate deviation compensation coefficient and the audio data timestamp drift amount extracted from the on-chain hash certificate can be used to correct the spatial and temporal errors generated during the data acquisition process. The coordinate deviation compensation coefficient is used to correct the spatial coordinate offset caused by device positioning errors, while the timestamp drift amount is used to correct the synchronization error of audio-visual modality data during the acquisition process. Based on these compensation parameters, the multi-modal process data is spatio-temporally calibrated so that all modal data is aligned to the same time reference, forming a spatio-temporally synchronized data queue. This queue ensures the time consistency between different data sources and provides a basis for subsequent data association. After the spatio-temporally synchronized data queue is established, cross-modal topological mapping rules are generated through dynamic association constraints. These rules are used to describe the relationships between different modal data, such as the association between hand movements and audio instructions, the correspondence between sensor data and video key frames, etc. The establishment of cross-modal topological mapping rules depends on the similarity calculation of features between modalities, including time series matching, dynamic time warping (DTW), and content-based feature comparison. The optimization of the mapping rules uses the maximum weight matching algorithm to ensure the best correspondence between different modalities. Based on the cross-modal topological mapping rules, a cross-modal hypergraph structure is constructed. Different from the traditional graph structure, the hypergraph structure allows a single hyperedge to connect multiple modal data nodes, thus more accurately depicting the multi-modal associations in complex process flows. For example, in the ceramic drawing process, hand movements, turntable speed, glazing instruction text, and temperature sensor data may all be associated through hyperedges to jointly describe a specific process step. After the hypergraph structure is constructed, the transition relationship between different process stages is calculated and analyzed based on the state transition probability of the process materials. Define the state transition matrix as: , where represents the association weight between process features, represents the deformation parameter distance between two process states, and the exponential term controls the sensitivity of the state transition, and the parameter is used to adjust the smoothness of the state transition to ensure compliance with the real physical process, is the normalization factor to ensure that the sum of each row of the state transition probability matrix is 1. This term represents the sum of the transition probabilities from state i to all possible states k. When calculating , by dividing by this normalization factor, it is ensured that all transition probabilities add up to 1, which conforms to the standard definition in probability theory.

[0028] In addition, in the cross-modal hypergraph structure, the interaction between different modal data nodes needs to be modeled through dynamic association constraints. Let the hypergraph H=(V, E) consist of a node set V and a hyperedge set E, and each hyperedge e( connects multiple modal data points, then the cross-modal feature aggregation can be expressed as: , wherein, is the feature representation of node v, represents the association weight between hyperedge e and node v, is the number of nodes connected by the hyperedge, is the feature aggregation function, is the non - linear activation function. This formula ensures the retention of key features during the cross - modal data fusion process and enhances the association between different modal data.

[0029] Based on the cross - modal hypergraph structure, the evolution paths between different process states can be further deduced. Combining material properties, process parameters, and environmental variables, a process knowledge graph is formed. The process knowledge graph is used to describe the process flow, material properties, process states, and the relationships between them, ensuring that multi - modal data can be organized into an interpretable structured knowledge framework. In the hypergraph, each node can correspond to a process state, such as "wet blank forming", "semi - drying", "pretreatment before firing" in the process of ceramic drawing, and the hyperedge represents the transition relationship between states. By calculating the state transition probability, the optimal evolution path of different process steps can be determined, and the optimal evolution direction of a certain state can be deduced using conditional probability. In addition, the key parameters of the material (such as temperature, humidity, chemical composition, etc.) are stored as node attributes, and the change of the process state is restricted by the deformation parameters. For example, in the glaze formulation stage, different oxide ratios affect the final glaze color, and the knowledge graph can record the influence relationship of different components and provide suggestions for the optimal ratio.

[0030] Finally, combined with the deformation parameter threshold of the sensor data , when , abnormal state transitions are suppressed to ensure that the generated process knowledge graph conforms to the real process logic.

[0031] Step 130: Construct an interactive 3D teaching model according to the process knowledge graph.

[0032] Specifically, in this embodiment, the process of constructing an interactive 3D teaching model based on a process knowledge graph is realized by mapping the entity relationships in the graph into three-dimensional visualization operation logics. The nodes (such as material states, tool parameters) and edges (such as causal relationships) defined in the process knowledge graph are converted into dynamic parameters in a 3D scene. For example, in a ceramic firing teaching model, the kiln temperature node corresponds to the thermal radiation visualization effect of a three-dimensional kiln, and the causal relationship between temperature and glaze cracking is presented through real-time physical engine simulation. The physical engine is a calculation module based on material mechanics equations, which can simulate physical behaviors in process such as clay deformation and liquid flow. For example, when the user adjusts the rotation speed of a virtual pottery wheel, the engine will calculate the deformation amount of the blank in real time according to the "rotation speed - centrifugal force" relationship in the knowledge graph. The interactive operation is realized through a motion capture device and a haptic feedback device. For example, when the user's gesture deviates from the standard embroidery stitch trajectory, the haptic glove will generate a vibration prompt, and at the same time, the 3D interface will automatically call the remedial measure node in the knowledge graph to generate a correction animation.

[0033] In an optional implementation manner of this embodiment, the steps of constructing an interactive 3D teaching model according to a process knowledge graph include: performing kinematic chain analysis on the action-material association nodes in the process knowledge graph, extracting the hand joint angle sequence and the material deformation parameter threshold; converting the hand joint angle sequence into a rigid body dynamics constraint equation, and determining the driving parameter matrix of the process action decomposition engine in combination with the material deformation parameter threshold; constructing an interactive 3D model corresponding to the driving parameter matrix by calling the real-time deformation simulator of the process action decomposition engine.

[0034] Specifically, in this embodiment, by analyzing the association information between actions and materials recorded in the process knowledge graph, integrating the captured human motion data and material response information, and using kinematic chain analysis technology to obtain the angle change sequence of each hand joint during movement, while using high-precision sensors to obtain the critical parameter values when the material deforms under external force. This parameter describes the boundary at which the material changes from an elastic state to a plastic state under a certain stress condition. During this process, the hand joint angle sequence is transformed into a mathematical model through a set of innovative rigid body dynamics constraint conversion formulas, which are defined in a non-linear integral form as: , where, represents the function of the change of each hand joint angle with time, and are the angular velocity and angular acceleration respectively, T represents the duration of the movement, is the coefficient describing the rigid body dynamics characteristics, and It corresponds to the material deformation parameter threshold. This formula integrates acceleration and velocity information in a non-linear manner. By capturing the subtle changes during the movement, a mathematical expression reflecting the action constraints is constructed, thereby providing a basis for the generation of the subsequent parameter matrix. Based on this conversion result, the driving parameter matrix of the process action decomposition engine can fuse the hand movement characteristics and the critical deformation information of the material, provide dynamic input for the engine, and call the finite element analysis and adaptive mesh algorithm in the real-time deformation simulator to accurately simulate the deformation behavior of the material under various working conditions during the process. For example, in assembly or welding operations, by adjusting the mapping relationship between the hand movement and the material force response in real time, the interaction nodes in the 3D model can intuitively reflect the impact of the action on the material state, thus realizing a highly interactive and accurately feedback virtual 3D environment. This environment not only intuitively shows the coupling effect between each movement joint and the material deformation, but also provides reliable data support for the process flow optimization and intelligent decision-making.

[0035] In an optional implementation manner of this embodiment, according to the critical fracture strain value in the material deformation parameter threshold, the real-time material deformation parameters in the user operation trajectory are segmented into discrete deformation gradient sequences according to time windows; the dynamic time warping matching is performed between the deformation gradient sequences and the standard deformation trajectories in the process knowledge graph to determine the integral value of the deformation deviation degree within each time window; when the integral value of the deformation deviation degree exceeds the tolerance threshold corresponding to the material elastic modulus, the pulse frequency modulation parameter for the tactile feedback is generated according to the corresponding deformation deviation degree; the pulse frequency modulation parameter is input into the resonant circuit of the tactile feedback device to trigger the corrective control of the tactile feedback device for the deformation deviation degree.

[0036] Specifically, in this embodiment, during the real-time monitoring of the process, by extracting the critical fracture strain value in the material deformation parameter threshold, the critical point at which the material changes from the elastic state to the fracture state under the action of external force is obtained. This critical fracture strain value is a key index determined according to the physical properties of the material. Then, the real-time material deformation parameters collected in the user operation trajectory are segmented according to the preset time windows, thereby forming discrete deformation gradient sequences. For example, the tissue or structure strain data obtained by using a high-precision strain gauge in remote control can be cut into multiple time periods per second after preprocessing. These discrete sequences can fully reflect the change trend of the material deformation over time. Subsequently, the dynamic time warping matching technology is used to compare the discrete deformation gradient sequences with the standard deformation trajectories in the process knowledge graph. The dynamic time warping matching is a non-linear algorithm that optimally aligns the two sequences by adjusting the time axis, thereby calculating the integral value of the deformation deviation degree within each time window. The integral formula of the deformation deviation degree integral value is expressed as: , where, Represents the actual strain of the material during the user's operation, represents the strain value at the corresponding time point of the standard deformation trajectory in the process knowledge graph, is the time decay coefficient, which is used to adjust the influence of the matching difference between different time points on the integration result, represents the time difference between the actual operation time t of the user and the time t' of the standard deformation trajectory, represents the attenuation factor, which attenuates the time difference through an exponential function. This formula aims to accurately capture the change in the deformation rate and output an integral value of the deformation deviation. When this integral value exceeds the tolerance threshold corresponding to the elastic modulus of the material, the elastic modulus of the material, as an important physical quantity describing the material's resistance to deformation, its tolerance threshold determines the acceptable error range. At this time, the corresponding tactile feedback pulse frequency modulation parameter is generated based on the deformation deviation. After precise calculation, this parameter reflects the degree of deviation. Subsequently, the modulation parameter is input into the resonant circuit of the tactile feedback device, and the resonant circuit can amplify and resonate with a specific frequency signal, thereby driving the device to output the corresponding mechanical vibration or pressure pulse to achieve real-time correction control of the deformation deviation of the material during the user's operation.

[0037] Step 140: Determine the multi-dimensional contribution metric value according to the usage data of the interactive 3D teaching model.

[0038] Specifically, in this embodiment, calculating the multi-dimensional contribution metric value based on the usage data of the interactive 3D teaching model is achieved by analyzing the correlation between the user's operation behavior and the process knowledge graph. The multi-dimensional contribution metric value includes indicators such as operation normativity, experience dissemination degree, and innovation adoption rate. Among them, operation normativity refers to the matching degree between the user's operation trajectory and the standard process parameters in the knowledge graph. The experience dissemination degree is calculated by the number of times the user corrects wrong operations and the invocation frequency of the remedial strategies in the knowledge graph. The innovation adoption rate is evaluated based on the similarity between the new data pattern generated after the user independently adjusts the process parameters and the evidence stored on the historical chain. For example, if the innovation of wood carving techniques is repeatedly reproduced by other students, its contribution value will increase with the number of verifications. Data collection relies on the built-in spatio-temporal logging function of the model to record raw information such as the user's gesture trajectory, tool parameter adjustment, and three-dimensional scene interaction duration. After feature extraction, it is input into the contribution evaluation model, which dynamically assigns weights in combination with the process constraint conditions in the knowledge graph. For example, the operation risk coefficient of the high-temperature kiln change experiment is incorporated into the contribution value attenuation factor, and the calculation result is finally written into the blockchain through a smart contract to form a traceable contribution certificate.

[0039] In an alternative implementation of this embodiment, the steps of determining the multi-dimensional contribution metric value based on the usage data of the interactive 3D teaching model include: determining the spatio-temporal deviation degree quantization matrix between the user operation trajectory and the standard action nodes of the process knowledge graph through the dynamic time warping algorithm; inputting the spatio-temporal deviation degree quantization matrix into the material deformation prediction model to output the probability distribution map of microstructural crack propagation; inputting the spatio-temporal deviation degree quantization matrix into the preset material deformation prediction model to generate the probability distribution map of material deformation risk; constructing a contribution degree decision tree through a multi-objective optimization algorithm based on the deviation degree quantization matrix and the deformation risk probability distribution map; performing fault-tolerant consensus verification between the alliance chain nodes on the output solution set of the contribution degree decision tree to generate the multi-dimensional contribution metric value.

[0040] Specifically, in this embodiment, through the dynamic time warping (DTW) algorithm, the user operation trajectory can be spatio-temporally aligned with the standard action nodes in the process knowledge graph to construct the spatio-temporal deviation degree quantization matrix. In this process, the operation trajectory data of the user (such as the hand movement trajectory in ceramic throwing) is matched with the time series data of the standard process action nodes, and the DTW algorithm calculates the cost matrix to find the optimal bending path, thereby quantifying the deviation degree. The definition of the spatio-temporal deviation degree quantization matrix is: , where, represents the spatio-temporal feature vector of the i-th frame of the user trajectory, including coordinates (x, y, z), speed v, and curvature κ, while is the feature vector of the j-th frame of the standard node. The element in the matrix represents the spatio-temporal deviation degree between the user operation and the standard operation, is the covariance matrix, represents calculating the distance between the user's current action and the standard action using the Mahalanobis distance. Through this matrix, the deviation of the user operation can be effectively captured, and thus a basis for the prediction and risk assessment of material deformation can be provided. Next, the obtained spatio-temporal deviation degree quantization matrix is input into the material deformation prediction model. Through methods such as finite element analysis, this model can calculate the propagation probability of micro-cracks and evaluate the initiation and propagation probability of cracks based on the change of the user operation deviation. The calculation of the crack propagation probability can be quantified through the following integral model: , where, is the deviation degree at time t, is the critical deformation threshold, α is the material sensitivity coefficient, βis the Weibull shape parameter. This model can quantify the impact of deviation on crack propagation and provide data support for subsequent risk assessment. At the same time, the spatio-temporal deviation quantification matrix is also input into the material deformation risk prediction model, and the risk probability distribution map of material deformation is generated through multi-physical field coupling analysis. The definition of risk probability takes into account the influence of temperature gradient to provide more accurate prediction of material deformation. The calculation formula of risk index R(x, y, z) is: , where, is the local temperature gradient, is the safety threshold, is the thermal stress coupling coefficient. Through this model, the deformation risk of materials can be evaluated according to different deviations and temperature gradients, and the risk probability distribution map of deformation can be generated. Based on the deviation quantification matrix and the risk probability distribution map of deformation, a contribution decision tree is constructed by using a multi-objective optimization algorithm. The objective function optimizes the decision path by minimizing the operation deviation and risk suppression, where: ， where, represents the deviation minimization objective function, and the goal is to reduce the deviation between user operations and the standard process by adjusting the operation parameters in the process. represents the weight of the k-th action node. The weight of each action node can be set according to its contribution to the process. For example, the weight of "casting force" may be larger, while the weight of "glazing speed" may be smaller. represents the deviation of the k-th action node, and K represents the total number of action nodes, covering all action nodes related to the process. represents the risk suppression objective function, and the goal is to optimize the risk caused by material deformation in the process. Its purpose is to reduce the deformation risk caused by deviating from the standard path. represents the value of the m-th material deformation risk, and M represents the total number of deformation risks. This formula represents a multi-objective optimization problem, aiming to minimize the operation deviation and deformation risk simultaneously. The objective function measures the deviation and risk in the process through two parts ( and ), and the goal is to adjust the process parameters to achieve the optimal balance.

[0041] Finally, a fault-tolerant consensus verification among consortium chain nodes is performed on the output solution set of the contribution degree decision tree. Through the threshold signature mechanism, multiple verification nodes jointly verify the optimized solution set to ensure the correctness and consistency of the decision results. The generated multi-dimensional contribution measurement values will be written into the smart contract and broadcast through the blockchain network to ensure that all nodes synchronously update the process data. It effectively realizes the quantification of the spatio-temporal deviation degree between the user operation trajectory and the process knowledge graph, and further provides intelligent support for process optimization, risk assessment and decision-making in traditional handicraft operations such as ceramics.

[0042] Step 150: Based on the multi-dimensional contribution measurement values and the hierarchical smart contract architecture, determine the data access permissions through the community voting mechanism.

[0043] Specifically, in this embodiment, the hierarchical smart contract architecture divides the permission logic into a basic layer and an extended layer. The basic layer contract processes the core verification rules of data ownership, and the extended layer contract dynamically adjusts the user permission level according to the contribution value. After calculating the operation standardization, innovation adoption rate and other indicators in the multi-dimensional contribution measurement values through the weight factor, they are converted into voting weight values. For example, users with a high degree of experience dissemination have a higher voting influence on the proposal for the scope of process data opening. The community voting mechanism adopts an on-chain governance model. When a user applies to access key process data, it triggers the extended layer contract to generate a voting proposal and writes the proposal content and the permission grading standard into the blockchain event log. During the voting process, the system non-linearly maps the user voting weight according to the contribution value. After the voting result is verified, the basic layer contract updates the data access permission list. For example, the celadon glaze formula data that passes the proposal will be open to users who meet the contribution value for the microstructure scan map.

[0044] In an alternative implementation manner of this embodiment, the step of determining the data access permissions through the community voting mechanism based on the multi-dimensional contribution measurement values and the hierarchical smart contract architecture includes: performing an asymmetric zero-knowledge proof verification on the permission change proposal in the hierarchical smart contract architecture to generate a proposal validity identifier; calculating the Shapley value for community member nodes according to the multi-dimensional contribution measurement values and dynamically allocating voting weight factors; when it is detected that the deviation degree between the voting weight factor and the historical contribution record in the blockchain deposit exceeds the preset threshold, activating the iris biometric verification module against the Sybil attack; driving the light field camera array of the distributed hardware terminal through the iris verification module to collect live verification data and trigger the smart contract execution engine to update the data access permission whitelist.

[0045] Specifically, in this embodiment, in the hierarchical smart contract architecture, the verification of permission change proposals is implemented through the asymmetric zero-knowledge proof technology. Zero Knowledge Proof (ZKP) is a cryptographic method used to verify the truth of a proposition without revealing any other information. Asymmetric zero-knowledge proof refers to the zero-knowledge proof generated based on asymmetric encryption algorithms (such as public-key encryption), which can ensure the validity of the proposal during the verification process while protecting the privacy of users. When a permission change proposal is generated, the relevant information is verified through zero-knowledge proof to ensure whether the proposal complies with the contract regulations and the actual situation. The validity identifier generated after verification can be used as the basis for the credibility of the proposal to ensure that the permission change will not be maliciously tampered with or deceive the system.

[0046] In terms of community management, the system dynamically calculates the voting weight factor of community members according to the multi-dimensional contribution measurement value. To fairly allocate the voting weight, the Shapley value calculation method is adopted. The Shapley value originally originated from the cooperative game theory and aims to measure the marginal contribution of an individual in cooperation. In this scenario, the Shapley value fairly allocates the voting weight by considering the impact of each member on the overall result in different cooperation situations. The calculation formula of the Shapley value is to perform a weighted sum of all possible participation combinations to obtain the contribution degree of each member. During this process, the Shapley value ensures that the voting weight of community members reflects their true contributions to the community. If a deviation from the historical contribution records stored in the blockchain is found in the calculation of the voting weight factor and exceeds the preset threshold, the system will activate the iris biometric verification module. Iris verification is a highly secure and unique identity verification method that identifies based on the uniqueness of an individual's iris. Through the light field camera array of the distributed hardware terminal, the system can collect live iris images for real-time comparison and verification to ensure that the identity of the voter is consistent with the historical records. This biometric verification process effectively prevents the risks of impersonation and identity fraud. Once the iris verification is passed, the smart contract execution engine will update the data access permission whitelist to ensure that only verified users can modify or operate the key content in the contract. This process dynamically adjusts the permissions to ensure the security of the system in terms of identity verification and data access control. This mechanism not only prevents interference from malicious users but also provides more detailed access control through whitelist management, further enhancing the security of the smart contract system.

[0047] An intelligent decision-making method for cross-modal data provided by the solution of the present application generates an on-chain hash certificate by performing blockchain certification on the collected multi-modal process data; performs cross-modal spatio-temporal alignment and feature fusion on the multi-modal process data according to the spatio-temporal metadata parsed from the on-chain hash certificate to generate a process knowledge graph; constructs an interactive 3D teaching model according to the process knowledge graph; determines a multi-dimensional contribution metric value according to the usage data of the interactive 3D teaching model; and determines data access rights through a community voting mechanism based on the multi-dimensional contribution metric value and a hierarchical smart contract architecture. Through the implementation of the solution of the present application, a process knowledge graph and an interactive 3D teaching model are constructed through cross-modal spatio-temporal alignment, enabling a causal association model to be formed between the data operated by the inheritor and the material response data, and providing a traceable decision-making basis for the accurate reproduction of endangered processes.

[0048] Figure 2 An intelligent decision-making device for cross-modal data provided by an embodiment of the present application can be used to implement the intelligent decision-making method for cross-modal data in the foregoing embodiment. As Figure 2 shown, the intelligent decision-making device for cross-modal data mainly includes: A first generation module 10 for generating an on-chain hash certificate by performing blockchain certification on the collected multi-modal process data; A second generation module 20 for performing cross-modal spatio-temporal alignment and feature fusion on the multi-modal process data according to the spatio-temporal metadata parsed from the on-chain hash certificate to generate a process knowledge graph; A construction module 30 for constructing an interactive 3D teaching model according to the process knowledge graph; A first determination module 40 for determining a multi-dimensional contribution metric value according to the usage data of the interactive 3D teaching model; A second determination module 50 for determining data access rights through a community voting mechanism based on the multi-dimensional contribution metric value and a hierarchical smart contract architecture.

[0049] In an alternative implementation of this embodiment, the first generation module is specifically configured to: perform modal classification and spatio-temporal marking on multi-modal process data through edge computing nodes to generate modal data packets carrying coordinates and timestamps; wherein, the modal data packets include: video data, audio data, text data, and sensor data; perform key-frame action capture on the video data and extract the time-frequency feature vector of the hand movement trajectory; perform voiceprint noise reduction on the audio data, and align the noise-reduced audio data with the text data on the time axis to generate a synchronized audio-text data pair; extract the Mel cepstral coefficients of the synchronized audio-text data pair, and generate a composite audio fingerprint hash in combination with the keyword hash value of the aligned text; input the time-frequency feature vector and the composite audio fingerprint hash into the heterogeneous data fusion module, and construct structured metadata in combination with the dynamic threshold calibration result of the sensor data; based on the distributed consensus protocol of the consortium chain nodes, perform multi-signature binding on the structured metadata and the IPFS storage path of each modal data packet to generate a cross-modal hash certificate chain.

[0050] In an alternative implementation of this embodiment, the second generation module is specifically configured to: extract the coordinate deviation compensation coefficient of the video data and the timestamp drift amount of the audio data from the on-chain hash certificate; perform spatio-temporal calibration on the multi-modal process data according to the coordinate deviation compensation coefficient and the timestamp drift amount to generate a spatio-temporally synchronized data queue; generate a cross-modal topology mapping rule according to the spatio-temporally synchronized data queue and the dynamic association constraint of the multi-modal process data; construct a cross-modal hypergraph structure according to the cross-modal topology mapping rule; calculate the process material state transition probability of the nodes of the cross-modal hypergraph structure, and generate a process knowledge graph in combination with the deformation parameter threshold of the sensor data.

[0051] In an alternative implementation of this embodiment, the construction module is specifically configured to: perform kinematic chain analysis on the action-material association nodes in the process knowledge graph, and extract the hand joint angle sequence and the material deformation parameter threshold; convert the hand joint angle sequence into a rigid body dynamics constraint equation, and determine the drive parameter matrix of the process action decomposition engine in combination with the material deformation parameter threshold; construct an interactive 3D model corresponding to the drive parameter matrix by calling the real-time deformation simulator of the process action decomposition engine.

[0052] In an alternative implementation of this embodiment, the intelligent decision-making device for cross-modal data further includes: a control module. The control module is configured to: divide the real-time material deformation parameters in the user operation trajectory into discrete deformation gradient sequences according to the critical fracture strain value in the material deformation parameter threshold by time window; perform dynamic time warping matching on the deformation gradient sequence and the standard deformation trajectory in the process knowledge graph to determine the integral value of deformation deviation degree within each time window; when the integral value of deformation deviation degree exceeds the tolerance threshold corresponding to the material elastic modulus, generate the pulse frequency modulation parameter for haptic feedback according to the corresponding deformation deviation degree; input the pulse frequency modulation parameter into the resonant circuit of the haptic feedback device to trigger the correction control of the haptic feedback device for the deformation deviation degree.

[0053] In an alternative implementation of this embodiment, the first determination module is specifically configured to: determine the spatio-temporal deviation degree quantization matrix between the user operation trajectory and the standard action nodes of the process knowledge graph through the dynamic time warping algorithm; input the spatio-temporal deviation degree quantization matrix into the material deformation prediction model to output the microscopic structure crack propagation probability distribution map; input the spatio-temporal deviation degree quantization matrix into the preset material deformation prediction model to generate the material deformation risk probability distribution map; construct a contribution decision tree through a multi-objective optimization algorithm based on the deviation degree quantization matrix and the deformation risk probability distribution map; perform fault-tolerant consensus verification between the output solution sets of the contribution decision tree for the consortium chain nodes to generate a multi-dimensional contribution measurement value.

[0054] In an alternative implementation of this embodiment, the second determination module is specifically configured to: perform asymmetric zero-knowledge proof verification on the permission change proposal in the hierarchical intelligent contract architecture to generate a proposal validity identifier; calculate the Shapley value for community member nodes according to the multi-dimensional contribution measurement value to dynamically allocate the voting weight factor; when it is detected that the deviation degree between the voting weight factor and the historical contribution record in the blockchain deposit exceeds the preset threshold, activate the iris biometric verification module against the Sybil attack; drive the light field camera array of the distributed hardware terminal through the iris verification module to collect live verification data and trigger the intelligent contract execution engine to update the data access permission whitelist.

[0055] An intelligent decision-making device for cross-modal data provided by the solution of the present application generates an on-chain hash certificate by performing blockchain certification on the collected multi-modal process data; performs cross-modal spatio-temporal alignment and feature fusion on the multi-modal process data according to the spatio-temporal metadata parsed from the on-chain hash certificate to generate a process knowledge graph; constructs an interactive 3D teaching model according to the process knowledge graph; determines a multi-dimensional contribution metric value according to the usage data of the interactive 3D teaching model; and determines the data access permission through a community voting mechanism based on the multi-dimensional contribution metric value and a hierarchical smart contract architecture. Through the implementation of the solution of the present application, a process knowledge graph and an interactive 3D teaching model are constructed through cross-modal spatio-temporal alignment, enabling a causal association model to be formed between the data operated by the inheritor and the material response data, and providing a traceable decision-making basis for the precise reproduction of endangered processes.

[0056] The one provided according to the solution of the present application Figure 3 An electronic device provided in an embodiment of the present application. This electronic device can be used to implement the intelligent decision-making method for cross-modal data in the foregoing embodiment, and mainly includes: A memory 301, a processor 302, and a computer program 303 stored on the memory 301 and executable on the processor 302. The memory 301 and the processor 302 are communicatively connected. When the processor 302 executes the computer program 303, the intelligent decision-making method for cross-modal data in the foregoing embodiment is implemented. Among them, the number of processors can be one or more.

[0057] The memory 301 can be a high-speed random access memory (RAM, Random Access Memory) or a non-volatile memory, such as a disk memory. The memory 301 is used to store executable program code, and the processor 302 is coupled to the memory 301.

[0058] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, which can be disposed in the electronic device in the foregoing embodiments. The computer-readable storage medium can be the memory in the foregoing Figure 3 illustrated embodiment.

[0059] A computer program is stored on the computer-readable storage medium, and when the program is executed by a processor, the intelligent decision-making method for cross-modal data in the foregoing embodiment is implemented. Furthermore, the computer-readable storage medium can also be various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a RAM, a magnetic disk, or an optical disc that can store program code.

[0060] Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0061] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: various media that can store program codes such as USB flash drives, mobile hard disks, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical discs.

[0062] As described above, the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or equivalently replace some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the various embodiments of the present application.

Claims

1. An intelligent decision-making method for cross-modal data, characterized in that, Including: Generate an on-chain hash certificate by conducting blockchain evidence preservation on the collected multi-modal process data; Perform cross-modal spatio-temporal alignment and feature fusion on the multi-modal process data according to the spatio-temporal metadata parsed from the on-chain hash certificate to generate a process knowledge graph; Construct an interactive 3D teaching model based on the process knowledge graph; Determine a multi-dimensional contribution metric value according to the usage data of the interactive 3D teaching model; Based on the multi-dimensional contribution metric value and a hierarchical smart contract architecture, determine data access permissions through a community voting mechanism.

2. The intelligent decision-making method for cross-modal data according to claim 1, wherein The step of generating an on-chain hash certificate by conducting blockchain evidence preservation on the collected multi-modal process data includes: Conduct modal classification and spatio-temporal marking on the multi-modal process data through an edge computing node to generate a modal data packet carrying coordinates and timestamps; wherein, the modal data packet includes: video data, audio data, text data, and sensor data; Perform key-frame action capture on the video data and extract the time-frequency feature vector of the hand movement trajectory; Perform voiceprint noise reduction on the audio data, and align the denoised audio data with the text data on the time axis to generate a synchronized audio-text data pair; Extract the Mel cepstral coefficients of the synchronized audio-text data pair, and generate a composite audio fingerprint hash in combination with the keyword hash value of the aligned text; Input the time-frequency feature vector and the composite audio fingerprint hash into a heterogeneous data fusion module, and construct structured metadata in combination with the dynamic threshold calibration result of the sensor data; Based on the distributed consensus protocol of the consortium chain node, perform multi-signature binding on the structured metadata and the IPFS storage path of each modal data packet to generate a cross-modal hash certificate chain.

3. The intelligent decision-making method for cross-modal data according to claim 2, wherein The step of performing cross-modal spatio-temporal alignment and feature fusion on the multi-modal process data according to the spatio-temporal metadata parsed from the on-chain hash certificate to generate a process knowledge graph includes: Extract the coordinate deviation compensation coefficient of the video data and the timestamp drift amount of the audio data from the on-chain hash certificate; Perform spatio-temporal calibration on the multi-modal process data according to the coordinate deviation compensation coefficient and the timestamp drift amount to generate a spatio-temporally synchronized data queue; Generate a cross-modal topology mapping rule according to the spatio-temporally synchronized data queue and the dynamic association constraint of the multi-modal process data; Construct a cross-modal hypergraph structure according to the cross-modal topology mapping rule; Calculate the process material state transition probability of the nodes of the cross-modal hypergraph structure, and generate a process knowledge graph in combination with the deformation parameter threshold of the sensor data.

4. The intelligent decision-making method for cross-modal data according to claim 1, characterized in that, The step of constructing an interactive 3D teaching model based on the process knowledge graph includes: Perform kinematic chain analysis on the action-material association nodes in the process knowledge graph, and extract the hand joint angle sequence and the material deformation parameter threshold; Convert the hand joint angle sequence into a rigid body dynamics constraint equation, and determine the driving parameter matrix of the process action decomposition engine in combination with the material deformation parameter threshold; Construct an interactive 3D model corresponding to the drive parameter matrix by invoking the real-time deformation simulator of the process action decomposition engine.

5. The intelligent decision-making method for cross-modal data according to claim 4, wherein The method further includes: According to the critical fracture strain value in the material deformation parameter threshold, segment the real-time material deformation parameters in the user operation trajectory into discrete deformation gradient sequences according to time windows; Perform dynamic time warping matching on the deformation gradient sequence and the standard deformation trajectory in the process knowledge graph to determine the integral value of the deformation deviation degree within each time window; When the integral value of the deformation deviation degree exceeds the tolerance threshold corresponding to the material elastic modulus, generate a pulse frequency modulation parameter for haptic feedback according to the corresponding deformation deviation degree; Input the pulse frequency modulation parameter into the resonant circuit of the haptic feedback device to trigger the haptic feedback device to correct and control the deformation deviation degree.

6. The intelligent decision-making method for cross-modal data according to claim 1, wherein The step of determining the multi-dimensional contribution metric value according to the usage data of the interactive 3D teaching model includes: Determine the spatio-temporal deviation degree quantization matrix between the user operation trajectory and the standard action nodes of the process knowledge graph through the dynamic time warping algorithm; Input the spatio-temporal deviation degree quantization matrix into the material deformation prediction model to output the probability distribution map of microstructural crack propagation; Input the spatio-temporal deviation degree quantization matrix into a preset material deformation prediction model to generate a probability distribution map of material deformation risk; Based on the deviation degree quantization matrix and the deformation risk probability distribution map, construct a contribution decision tree through a multi-objective optimization algorithm; Perform fault-tolerant consensus verification between the output solution sets of the contribution decision tree among the consortium chain nodes to generate the multi-dimensional contribution metric value.

7. The intelligent decision-making method for cross-modal data according to claim 1, characterized in that The step of determining the data access permission through the community voting mechanism based on the multi-dimensional contribution metric value and the hierarchical smart contract architecture includes: Verify the asymmetric zero-knowledge proof of the permission change proposal in the hierarchical smart contract architecture to generate a proposal validity identifier; Calculate the Shapley value for community member nodes according to the multi-dimensional contribution metric value to dynamically allocate voting weight factors; When it is detected that the deviation degree between the voting weight factor and the historical contribution record in the blockchain deposit exceeds the preset threshold, activate the iris biometric verification module against the Sybil attack; Drive the light field camera array of the distributed hardware terminal through the iris verification module to collect live verification data and trigger the smart contract execution engine to update the data access permission whitelist.

8. An intelligent decision-making device for cross-modal data, characterized in that, The intelligent decision-making device for cross-modal data includes: A first generation module for generating an on-chain hash certificate by performing blockchain deposit on the collected multi-modal process data; A second generation module for performing cross-modal spatio-temporal alignment and feature fusion on the multi-modal process data according to the spatio-temporal metadata parsed from the on-chain hash certificate to generate a process knowledge graph; A construction module for constructing an interactive 3D teaching model according to the process knowledge graph; A first determination module for determining the multi-dimensional contribution metric value according to the usage data of the interactive 3D teaching model; A second determination module for determining the data access permission through the community voting mechanism based on the multi-dimensional contribution metric value and the hierarchical smart contract architecture.

9. An electronic device, characterized in that, It includes a memory and a processor, wherein: The processor is used to execute a computer program stored on the memory; When the processor executes the computer program, it implements the steps in the intelligent decision-making method for cross-modal data according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps in the intelligent decision-making method for cross-modal data according to any one of claims 1 to 7.

Images