Integrated multi-mode culture resource intelligent data governance and management system

Abstract

The invention relates to the technical field of data processing, in particular to an integrated multi-mode culture resource intelligent data governance and management system. The system comprises a data acquisition module, an intelligent processing module, a data management module, a knowledge organization module and an interactive display module. Synchronous acquisition of videos, audios, images and three-dimensional point clouds is realized through camera equipment, audio acquisition equipment and a laser scanner, and cultural elements are extracted through action recognition, speech recognition model processing dialect transcription, image semantic segmentation and named entity recognition by adopting a convolutional neural network. The system realizes structured description and semantic mapping of data on the basis of non-abandoned domain ontology, and constructs a knowledge graph by means of a graph database and a graph neural network. And finally, realizing three-dimensional visual display of the cultural resources by combining a virtual reality technology. According to the method, the problems of non-abandoned multi-modal data processing splitting, insufficient semantic organization and weak display interactivity in the prior art are solved, and the digital governance level and the propagation capability of cultural resources are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an integrated multimodal cultural resource intelligent data governance and management system. Background Technology

[0002] Intangible cultural heritage is an important cultural treasure accumulated throughout human history, possessing extremely high historical, artistic, and social value. With the development of digital technology, the digital protection and transmission of cultural heritage has become a key focus of widespread attention in academia and industry. Traditional methods of digitizing intangible cultural heritage often rely on single-modal data recording (such as video or text), making it difficult to fully present the multi-dimensional complexity of intangible cultural heritage projects in terms of techniques, semantic context, and spatial environment.

[0003] Currently, some technologies have attempted to process intangible cultural heritage resources using image recognition, speech-to-text transcription, and 3D reconstruction, but the following problems still exist. First, intangible cultural heritage projects typically involve multimodal data such as video, audio, text, images, and point clouds. Existing technologies generally use independent tools to process these separately, resulting in data fragmentation and a lack of a unified semantic system, making it difficult to support cross-modal fusion and reasoning analysis. Second, most digitized intangible cultural heritage projects only complete data storage and display, failing to establish a complete ontology structure and knowledge graph, lacking semantic organization and connections, and failing to meet the diverse needs of public inquiries, researcher reasoning, and developer access. In addition, although some achievements have attempted to present intangible cultural heritage in 3D or VR, they mostly remain at the stage of static model rendering, unable to achieve deep interaction with structured knowledge, resulting in limited user experience and insufficient cultural dissemination. Summary of the Invention

[0004] To overcome the above shortcomings, this invention provides an integrated multimodal cultural resource intelligent data governance and management system, which aims to improve the problems of fragmented multimodal data, weak semantic organization, difficulty in linking knowledge, and poor interactive display in the existing technology, and realize the unified collection, intelligent analysis, structured governance, knowledge integration and immersive display of intangible cultural heritage resources.

[0005] To achieve the above objectives, the present invention provides the following technical solution.

[0006] An integrated, multimodal intelligent data governance and management system for cultural resources includes: The data acquisition module is used to synchronously record the process of inheritors demonstrating their skills using video equipment, acquire audio data using environmental sound acquisition equipment, generate three-dimensional point cloud data of cultural sites and traditional artifacts using laser scanners, and perform time synchronization across multiple devices. The intelligent processing module is used to input video data into a convolutional neural network for action recognition, perform deep learning speech recognition transcription on audio data, run semantic segmentation algorithms on image data to annotate cultural elements, and extract cultural knowledge entities from text through named entity recognition algorithms. The data governance module is used to perform ontology mapping on the parsed data based on the ontology library in the field of intangible cultural heritage, generate a structured description that conforms to the data model, and perform integrity checks on the generated data and hierarchical access control based on the identity authentication of the inheritors. The knowledge organization module is used to store standardized data in the form of RDF triples in a graph database, add spatial coordinates to cultural resources through geocoding, establish a timestamp historical evolution index, and use graph neural networks to discover semantic relationships between cultural resources. The interactive display module is used to provide semantic retrieval based on knowledge graphs through SPARQL queries, generate personalized recommendations using collaborative filtering algorithms, and render 3D data into display content using the Unity3D engine.

[0007] Furthermore, the steps for multi-device time synchronization are as follows: At the start of data collection, a unified timestamp is generated for each data collection device based on the global clock. The time deviation between devices is calculated by detecting characteristic events in the collected data. A sliding window alignment algorithm is used to perform dynamic time correction on the multimodal data stream, and the order of the acquired frames is rearranged in the data buffer to ensure the time consistency of video frames, audio frames and point cloud frames.

[0008] Furthermore, the steps for action recognition using a convolutional neural network are as follows: The sequence of pose key points from the inheritor's action video and the RGB frame features are input in parallel into a multi-branch convolutional neural network; The network was trained using transfer learning and few-sample data augmentation strategies, including mirror transformation, temporal slicing, and action interpolation augmentation, to adapt to the scarcity of data samples of intangible cultural heritage skills. Based on the output action category sequence, the skill process is further decomposed and skill step labels are established to form skill entity information that can be used for ontology mapping.

[0009] Furthermore, the steps for performing deep learning speech recognition transcription are as follows: An acoustic model is constructed based on sub-word units, and the language model is expanded by combining a multi-dialect phoneme dictionary. An adaptive acoustic modeling method is adopted, and the Mandarin corpus model is adapted to a specific dialect corpus through transfer learning. During the recognition process, dialect language models are dynamically switched, and recognition results with low confidence are optimized through a context rescoring mechanism; The results of named entity recognition are used to correct specific cultural terms.

[0010] Furthermore, the steps for ontology mapping of the parsed data are as follows: Construct a multi-level ontology structure consisting of a concept layer, an attribute layer, and an instance layer; The mapping relationship between multimodal data tags and ontology entities is analyzed through a rule engine. The rules include a composite logic based on regular expressions, semantic similarity calculation, and cross-modal feature comparison. For entities that cannot be automatically mapped, a manual verification interface is invoked for semi-supervised correction. The final mapping result is converted into RDF triples and written into the knowledge graph.

[0011] Furthermore, the steps for data integrity checking are as follows: After the multimodal data collection is completed, field integrity verification rules are established based on the predefined intangible cultural heritage resource data model to detect missing fields, primary keys and related attributes; A cross-modal consistency verification algorithm is used to perform logical alignment detection on video, audio, point cloud and text data, including collection timestamp comparison, event semantic cross-validation and content context matching; The anomaly detection model identifies noisy, duplicate, and conflicting entities in the data and classifies and labels them based on confidence scores. The inspection results are recorded in the quality report, and non-compliant data is automatically re-collected or manually reviewed to ensure the structural integrity and data reliability of the knowledge graph.

[0012] Furthermore, the steps for graph neural networks to discover semantic associations are as follows: Construct a heterogeneous graph with inheritors, skills, artifacts, places, and time nodes as multiple types of nodes; The weighted information transfer between nodes is calculated based on graph attention network, and the node feature representation is updated by combining the spatiotemporal attributes of the nodes and the relationship type of the edges. Calculate the potential association probability between nodes using a multi-hop neighbor aggregation strategy; Incremental learning updates are performed on node relationships with association confidence scores below a threshold to support the semantic expansion of cultural resources.

[0013] Furthermore, the steps for rendering 3D data into display content are as follows: A multi-level detail model is constructed for laser scanning point cloud data, and the rendering accuracy is dynamically selected based on device performance; By binding image textures with semantic tags, hot information about cultural elements is overlaid onto a 3D scene in real time; Optimize VR / AR scene interaction latency through spatial chunking loading and asynchronous rendering; It supports WebXR interfaces and cross-terminal adaptation, enabling immersive browsing and interaction of cultural resources.

[0014] The present invention has the following beneficial effects: 1. In this invention, based on a unified timestamp and sliding window alignment mechanism, high-precision synchronization of multimodal acquisition data such as video, audio and 3D point cloud is achieved, which effectively avoids the data misalignment and time deviation problems in the traditional intangible cultural heritage digitization process, ensures the consistency of skill movements, language expression and spatial information, and provides timing guarantee for subsequent multimodal analysis and semantic mapping.

[0015] 2. In this invention, by constructing an intangible cultural heritage ontology structure covering concepts, attributes, and instances, and combining it with a rule engine to complete the automatic mapping between multimodal data tags and ontology entities, and by introducing a manual verification interface to improve accuracy, the structured expression and semantic organization of intangible cultural heritage knowledge are realized, laying the foundation for knowledge graph construction and deep reasoning, and breaking through the limitations of fragmented knowledge organization in existing systems.

[0016] 3. In this invention, a heterogeneous graph model is constructed using graph neural networks to construct elements such as inheritors, skills, and artifacts, and potential semantic relationships are discovered. At the same time, Unity3D and WebXR are combined to realize the three-dimensional visualization and immersive display of cultural resources, which significantly enhances the dissemination and interactivity of intangible cultural heritage resources and meets the diverse needs of public experience and professional research. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the structure of an integrated multimodal cultural resource intelligent data governance and management system proposed in this invention; Figure 2 This is a flowchart of an integrated multimodal cultural resource intelligent data governance and management method proposed in this invention. Detailed Implementation

[0018] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] Example 1 In a first embodiment of the present invention, the present invention provides an integrated multimodal cultural resource intelligent data governance and management system, such as... Figure 1 As shown, it includes: The data acquisition module is used to synchronously record the process of inheritors demonstrating their skills using video equipment, acquire audio data using environmental sound acquisition equipment, generate three-dimensional point cloud data of cultural sites and traditional artifacts using laser scanners, and perform time synchronization across multiple devices. Furthermore, the steps for multi-device time synchronization are as follows: At the start of data collection, a unified timestamp is generated for each data collection device based on the global clock. The time deviation between devices is calculated by detecting characteristic events in the collected data. A sliding window alignment algorithm is used to perform dynamic time correction on the multimodal data stream, and the order of the acquired frames is rearranged in the data buffer to ensure the time consistency of video frames, audio frames and point cloud frames.

[0020] Specifically, the system activates the global clock module before data acquisition begins. This clock module can be provided by a high-precision clock chip in the main control computer. Upon startup, the camera equipment, ambient sound acquisition equipment, and laser scanner synchronize with the main control computer's global clock via a network time synchronization protocol or a precise time protocol to ensure that the initial time base of each device is consistent. For devices that do not support network time synchronization, a synchronization pulse can be sent via a trigger signal line at the moment of acquisition startup to write the current timestamp into the local cache of each device.

[0021] During the acquisition process, each device's data stream contains detectable feature events. For example, in a video stream, feature events might include brightness abrupt change frames, the start frame of a performer's action, or a cursor blinking signal frame; in an audio stream, feature events might include volume spikes (such as drumbeats or clapping sounds); and in a point cloud stream, feature events might include large-scale displacement boundaries or external synchronization signal reflection points. The system extracts the frame number and timestamp of the feature events from each data stream using feature detection algorithms (such as sliding window-based threshold detection or template matching algorithms). Feature events occurring at the same time (or adjacent sampling points) are matched, and the deviation between the current time of each device and the global clock is calculated.

[0022] Establish a sliding time window of length N (e.g., 5s~10s) and continuously accumulate the time deviation of each device within the window. Use weighted averaging or Kalman filtering to smooth the deviation curve to reduce jitter caused by sudden noise. Dynamically adjust the acquisition time reference of each device based on the corrected time deviation. For example: for video streams, frame sequence time axis can be fine-tuned by frame interpolation or frame dropping; for audio streams, time alignment can be achieved through resampling or buffer delay compensation; for point cloud data, index rearrangement can be used to achieve correspondence with video frames.

[0023] The system sets up a unified data buffer, where all multimodal data is first cached. Based on the global clock timestamp and correction results, the video frames, audio frames, and point cloud frames in the buffer are uniformly sorted. Only when the time alignment error of a data frame in the buffer is less than a set threshold (e.g., ±5ms) will it be output to the subsequent intelligent processing module, ensuring strict synchronization of the multimodal data.

[0024] Using the above method, a high degree of consistency can be maintained on the time axis of video, audio and point cloud data under different sampling frequencies and transmission delays, with the error range controllable at the millisecond level, ensuring the accuracy of subsequent multimodal data fusion and analysis.

[0025] The intelligent processing module is used to input video data into a convolutional neural network for action recognition, perform deep learning speech recognition transcription on audio data, run semantic segmentation algorithms on image data to label cultural elements, and extract cultural knowledge entities from text through named entity recognition algorithms. Furthermore, the steps for action recognition using a convolutional neural network are as follows: The sequence of pose key points from the inheritor's action video and the RGB frame features are input in parallel into a multi-branch convolutional neural network; The network was trained using transfer learning and few-sample data augmentation strategies, including mirror transformation, temporal slicing, and action interpolation augmentation, to adapt to the scarcity of data samples of intangible cultural heritage skills. Based on the output action category sequence, the skill process is further decomposed and skill step labels are established to form skill entity information that can be used for ontology mapping.

[0026] Specifically, RGB frame sequences are extracted from video data synchronized across multiple devices. These sequences are sampled at a fixed frame rate (e.g., 25fps) and each frame is normalized and its resolution is standardized (e.g., 224×224 pixels). Open-source or self-developed pose estimation algorithms (e.g., OpenPose, HRNet) are used to extract the sequence of key points on the full-body skeleton of the inheritor (e.g., coordinates of 17 or 33 joints) from the video frames, and missing key points are interpolated to complete the sequence. The RGB frame features and the temporal sequence of the skeleton key points are used as two types of input data, fed in parallel into a multi-branch convolutional neural network.

[0027] A multi-branch convolutional neural network consists of spatial branches, temporal branches, and fusion branches: The spatial branch (Branch A) is used to extract spatial features of RGB frames, using ResNet-50 or MobileNet as the backbone CNN.

[0028] Temporal branch (Branch B) is used to analyze the temporal dynamic features of key points in the skeleton. It can be implemented using 1D temporal convolution (TCN) or Temporal-ResNet.

[0029] Branch C: The feature vectors of the spatial branch and the temporal branch are fused through concatenation, weighting or attention mechanisms, and then the action category is output through a fully connected layer.

[0030] In one implementation, the structural parameters of a multi-branch convolutional neural network are as follows: Spatial branch: ResNet-50, input is 224×224×3 RGB frames, output feature dimension is 2048.

[0031] Temporal branch: TCN structure, input is a sequence of keypoint coordinates (T×N×2), where T is the time step and N is the number of keypoints. The convolutional kernel size is set to {3,5} multi-scale, and the output feature dimension is 256.

[0032] Fusion branch: After feature concatenation, classification is performed through a self-attention module + two fully connected layers (256 → 64 → number of action categories).

[0033] The spatial branch can load ImageNet pre-trained weights, while the temporal branch can use skeleton sequence weights from the Kinetics or NTU RGB+D datasets as initialization parameters to shorten the training cycle. During data preprocessing, mirror transformation, temporal slicing, action interpolation augmentation, and multi-view sampling can be used for data augmentation, and cross-entropy loss or Focal Loss is preferred to handle class imbalance. A cosine annealing learning rate is used, with 300 training iterations.

[0034] During the inference phase, the network outputs a sequence of action categories arranged in time series. The system aggregates continuous action categories into skill step segments through state transition detection (such as Dynamic Time Warping (DTW) or Conditional Random Field (CRF)). A unique skill entity ID and label are generated for each skill step, recording the associated action category, action duration, required objects, and key operational elements, which serve as input data for subsequent ontology mapping.

[0035] The aforementioned network structure and training method can achieve high-precision action recognition under the condition of small sample intangible cultural heritage datasets, and can output skill step labels with semantic associations, supporting the physical construction of knowledge graphs.

[0036] Furthermore, the steps for performing deep learning speech recognition transcription are as follows: An acoustic model is constructed based on sub-word units, and the language model is expanded by combining a multi-dialect phoneme dictionary. An adaptive acoustic modeling method is adopted to adapt the Mandarin large corpus model to a specific dialect corpus through transfer learning; During the recognition process, the dialect language model is dynamically switched, and the recognition results with low confidence are optimized through a context re-scoring mechanism; Specific cultural terms are corrected by combining the named entity recognition results.

[0037] Specifically, to solve the problem of diverse dialects and sparse vocabulary in intangible cultural heritage, the corpus is first segmented into sub-word units by the Byte-Pair Encoding or SentencePiece method, and the unit length is between phonemes and words. For example, "gonggu" is segmented into "gong + gu", and the dialect variant "luo-gu" can be uniformly mapped to the same sub-word unit. The acoustic model can adopt an end-to-end deep neural network structure, such as the Conformer or Transformer-CTC model, whose input is acoustic features (such as 80-dimensional log-Mel filter bank features log-Mel FBank), and the output is a sequence of sub-word units. The number of model layers can be selected as 12 encoder layers + 12 decoder layers, the hidden dimension is 512 - 768, and the activation function uses Swish or ReLU.

[0038] Based on the known Mandarin phoneme set, the dialect-specific phonemes are extended and a unified phoneme dictionary for multiple dialects is established. For example, in Sichuan dialect, the phenomenon of indistinguishable "n" and "l" can add the mapping relationship of such phonemes in the dictionary; the unique entering tone endings [-p / -t / -k] in Cantonese are used as new phonemes. Construct an n-gram or Transformer language model that supports multiple dialects and train it in combination with dialect corpora (such as folk oral accounts, local scripts). Different dialects can train language models independently and support dynamic loading during system inference. The acoustic model trained with a large Mandarin corpus (>1000h) is used as the base model, and the underlying general feature extraction layer is frozen, and only the high-level acoustic modeling layer is fine-tuned to adapt to the target dialect data (less than 100h). During the fine-tuning process, multi-task learning is introduced, that is, predicting phoneme / sub-word labels and dialect category labels simultaneously to enhance the dialect discrimination ability. Data augmentation methods are used to expand the dialect corpus, such as spectrogram occlusion, speed perturbation, and background noise superposition, to solve the problem of insufficient dialect data volume.

[0039] During inference, according to the preset identity information of the inheritor or the dialect category probability output by the real-time acoustic model, the corresponding dialect language model is dynamically switched. If the dialect cannot be determined, multi-language model parallel decoding is adopted, and the candidate with the highest probability is selected. The candidate transcription results generated by the speech recognition technology are re-scored using context information. The methods include calculating the sentence probability based on a bidirectional language model (BERT or RoBERTa); re-ordering using a conditional random field constrained by context; and focusing on re-scoring the entries with confidence lower than the threshold to reduce recognition errors.

[0040] Perform named entity recognition on the preliminary transcription text to extract terms related to the intangible cultural heritage field (such as names of artifacts, names of skills, names of places). The named entity recognition model can adopt BiLSTM-CRF or BERT-CRF. For the entries with low recognition confidence and matching the named entity recognition lexicon, replace them with known terms in the ontology library. For example, correct the result of speech recognition technology "luo gu" to "gongs and drums" according to the NER result, and bind the unique entity ID in the knowledge graph.

[0041] Through the above method, the system can achieve high-precision speech recognition under the conditions of multiple dialects and low-resource corpora, and output accurate transcription texts with cultural term annotations, providing a reliable data basis for subsequent ontology mapping and knowledge graph construction.

[0042] The data governance module is used to perform ontology mapping on the parsed data based on the intangible cultural heritage field ontology library, generate a structured description that conforms to the data model, and perform integrity checking on the generated data and hierarchical access permission management based on the identity authentication of inheritors; Furthermore, the steps for performing ontology mapping on the parsed data are as follows: Construct a multi-level ontology structure of the concept layer, attribute layer and instance layer; Parse the mapping relationship between multi-modal data tags and ontology entities through a rule engine, and the rules include a composite logic based on regular expressions, semantic similarity calculation and cross-modal feature comparison; Call the manual verification interface to perform semi-supervised correction on the entities that cannot be automatically mapped; Convert the final mapping result into RDF triples and write them into the knowledge graph.

[0043] Specifically, construct concept class nodes according to the intangible cultural heritage field classification system (such as skill categories, genres, artifacts, inheritors, regions, etc.). For example, "ceramics production" is used as a concept node of the skill category, and "blue and white porcelain firing technique" is used as a subordinate concept. Define attributes and relationships for each concept, such as "used artifacts", "performance venue", "inheritor", "origin era", etc. The attribute value types can be text, time, geographical coordinates, image URIs, etc. Generate instance nodes based on the multi-modal parsing results (video action tags, speech transcription texts, image recognition results, etc.). For example, "Li Moumou" is an instance node of the "inheritor" category, and "Longquan celadon cup" is an instance node of the "artifact" category.

[0044] The system integrates a rule engine (such as Drools or Jess) to manage the mapping rule base. Each rule describes the mapping logic between tags and entities. These rules include: regular expression rules: keyword matching between speech recognition text and OCR image tags, for example, mapping ".*qingci.shaozhi." to the concept node "qinghuaci shaozhi yujian" (blue and white porcelain firing technique); semantic similarity rules: calculating semantic similarity between tags and entity names (e.g., using Word2Vec or BERT vector cosine similarity), and mapping when the similarity is higher than a threshold; cross-modal feature comparison rules: comparing video action tags with a pose database (e.g., "rotating throwing action" → "throwing technique"); and comparing image recognition results with an object appearance feature vector database (e.g., detecting "long-necked bottle + blue glaze" → "Longquan celadon bottle"). The system executes these rules in priority order and generates a confidence score for each mapping result. When a single entity matches multiple candidate entities, the candidate entity with the highest confidence score is selected as the mapping object.

[0045] When the confidence level of the mapping result is lower than the threshold (e.g., 0.6) or there are multiple candidate conflicts, the manual verification interface is triggered. The manually verified mapping result is added to the training set as a new sample, and the semantic similarity model and rule weights are dynamically updated through online learning. For example, if the manual annotation "celadon covered bowl" should be mapped to "Longquan celadon ware", the system automatically updates the semantic word vector library to improve the accuracy of subsequent mappings.

[0046] The mapped entity-attribute relationships are converted into RDF triples. Examples: (Longquan celadon firing technique, utensils used, Longquan celadon cup); (Li Moumou, inherited technique, Longquan celadon firing technique); (Longquan celadon firing technique, origin era, Song Dynasty). The triples are then written to a graph database (such as Neo4j, GraphDB, or JanusGraph). The system automatically creates timestamp and geographic coordinate indexes, supporting subsequent SPARQL queries and graph neural network semantic association analysis.

[0047] Using the methods described above, multimodal parsing results (video, audio, image, text, etc.) can be structurally aligned with the knowledge ontology in the intangible cultural heritage field, ensuring data consistency and semantic accuracy of the knowledge graph. The mapping accuracy is significantly improved by combining automatic mapping rules with semi-supervised manual correction. The multi-level ontology structure and triplet storage enable the system to perform semantic retrieval, knowledge reasoning, and historical tracing capabilities.

[0048] Furthermore, the steps for data integrity checking are as follows: After the multimodal data collection is completed, field integrity verification rules are established based on the predefined intangible cultural heritage resource data model to detect missing fields, primary keys and related attributes; A cross-modal consistency verification algorithm is used to perform logical alignment detection on video, audio, point cloud and text data, including collection timestamp comparison, event semantic cross-validation and content context matching; The anomaly detection model identifies noisy, duplicate, and conflicting entities in the data and classifies and labels them based on confidence scores. The inspection results are recorded in the quality report, and non-compliant data is automatically re-collected or manually reviewed to ensure the structural integrity and data reliability of the knowledge graph.

[0049] Specifically, the system first loads a predefined intangible cultural heritage resource data model. This model includes required fields, primary keys, attribute relationships, and data type constraints corresponding to multimodal data. For example, the entity "Inheritor" must include the attributes: name, gender, place of origin, and inherited skill ID; the entity "Object" must include the attributes: name, material, and associated skill ID. The system then iterates through various data instances collected to detect issues such as missing required fields, primary key conflicts (duplicate IDs), and incorrect attribute types. The detection results generate a preliminary field integrity report, marking the locations of fields with anomalies.

[0050] The timestamps of the acquired video, audio, point cloud, and text data are aligned and verified. If the time stamp of the same event (e.g., "start throwing") differs from that of the point cloud stream by more than a set threshold (e.g., ±100ms), it is marked as temporal inconsistency. Event labels (e.g., video action recognition results, speech transcription keywords, and text descriptions) are extracted from the multimodal data and compared using the event mapping table in the knowledge base. For example, if the video recognition result is "tapping the clay blank," but the speech transcription text is "throwing," it is marked as semantic inconsistency. The speech transcription text is compared with the contextual information (e.g., the order of the craft process) in the knowledge graph using natural language processing algorithms. If there are abnormal process order or contextual conflicts, it is marked as logical inconsistency.

[0051] Anomaly detection models based on machine learning (such as Isolation Forest or Autoencoder) are used to calculate anomaly scores on feature vectors of various data types. Examples of anomaly categories include: Noisy data (low signal-to-noise ratio or excessive missing frame features); Duplicate data (repeated collection of multimodal data or repeated labeling of the same event); Conflicting entities (conflicting entity attributes in different data sources, such as inconsistent birth dates of the same heir). Based on the confidence scores output by the anomaly detection model, the system categorizes data into high-risk, low-risk, and normal categories, and labels them on the data management interface.

[0052] The system integrates field integrity verification, cross-modal consistency verification, and anomaly detection results to generate a quality report. The report includes a list of anomaly instances, anomaly types, severity levels, location markers, and handling recommendations. For high-severity anomalies, an automatic data re-collection process is triggered (e.g., re-capturing missing video clips or rescanning point cloud data). For some low-confidence anomalies or logically conflicting data, a manual review interface is triggered for expert confirmation and correction.

[0053] Through the above integrity check mechanism, the integrity, temporal consistency and semantic consistency of multimodal data can be guaranteed. Data missing, duplication, conflict and noise problems can be detected and handled in advance, and quality reports can be generated and the supplementary collection / review process can be automatically triggered to ensure the integrity and reliability of the data structure written into the knowledge graph.

[0054] The knowledge organization module is used to store standardized data in the form of RDF triples in a graph database, add spatial coordinates to cultural resources through geocoding, establish a timestamp historical evolution index, and use graph neural networks to discover semantic relationships between cultural resources. Furthermore, the steps for graph neural networks to discover semantic associations are as follows: Construct a heterogeneous graph with inheritors, skills, artifacts, places, and time nodes as multiple types of nodes; The weighted information transfer between nodes is calculated based on graph attention network, and the node feature representation is updated by combining the spatiotemporal attributes of the nodes and the relationship type of the edges. Calculate the potential association probability between nodes using a multi-hop neighbor aggregation strategy; Incremental learning updates are performed on node relationships with association confidence scores below a threshold to support the semantic expansion of cultural resources.

[0055] Specifically, based on the data entity types in the knowledge graph, inheritors, skills, artifacts, locations, and time nodes are defined as different types of nodes. Each type of node includes its ontological attributes and multimodal features. For example, inheritor nodes include: name, region, skill category, and associated audio / video features; skill nodes include: process flow, category label, and cultural value score; artifact nodes include: material and morphological feature vectors (extracted from image point clouds); location nodes include: geographic coordinates and spatial layout features; and time nodes include: time of occurrence and historical stage features. Semantic relationships are established between nodes, such as inheritor-skill: inheritance relationship; skill-artifact: usage relationship; skill-location: performance / production location relationship; and node-time: time index relationship. The heterogeneous graph is stored using adjacency lists or sparse matrices, compatible with graph neural network frameworks.

[0056] Node features are updated by weighting the features of neighboring nodes, with the weights determined by the attention coefficients. The decision, in which the attention coefficient is calculated, is as follows: ; in, , This represents the feature vector of two connected nodes. Represents a linear transformation matrix. This represents the attention parameters. When calculating the attention weights, the node's timestamp, geographic location embedding, and edge relation type embedding are incorporated to enhance the modeling ability for the spatiotemporal dependence of cultural resources and multi-relation semantics. The final updated node features are: ; in, This represents the activation function. Represents nodes A set of neighboring nodes connected by edges. During node feature updates, a multi-hop neighbor aggregation strategy is employed: the first hop aggregates direct neighbor information (e.g., skills directly associated with a successor); the second hop aggregates indirect neighbor information (e.g., objects, places, etc. associated with the skill). Each hop aggregation result is combined with the features of the previous hop through residual connections to preserve multi-level semantics. For each pair of candidate nodes... Calculate the association probability based on the updated node features: ; in, This represents the learnable parameter matrix. When... If the value exceeds a set threshold (e.g., 0.7), it is determined that there is a potential semantic association.

[0057] Candidate edges with a correlation confidence level below a threshold are marked as "unconfirmed relationships" and are not added to the core graph structure. When new data is input into the system (e.g., new cultural events or expert annotation results), incremental updates are performed only on the affected subgraphs: updating node features, edge weights, and potential relationship probabilities; manually confirmed new edges are added to the training samples to fine-tune the graph attention network online. Through incremental learning, the potential relationship network between cultural resources is continuously improved, enabling dynamic expansion of the knowledge graph and the ability to "discover unknown relationships."

[0058] This method can automatically identify implicit semantic relationships between cultural resources (such as apprenticeship relationships between inheritors, and historical connections between skills and locations) and dynamically expand the knowledge graph. Combining multi-relationship modeling with spatiotemporal attribute fusion makes the graph reasoning results more accurate, supporting subsequent semantic retrieval, recommendation, and visualization.

[0059] The interactive display module is used to provide semantic retrieval based on knowledge graphs through SPARQL queries, generate personalized recommendations using collaborative filtering algorithms, and render 3D data into display content using the Unity3D engine.

[0060] Furthermore, the steps for rendering 3D data into display content are as follows: A multi-level detail model is constructed for laser scanning point cloud data, and the rendering accuracy is dynamically selected based on device performance; By binding image textures with semantic tags, hot information about cultural elements is overlaid onto a 3D scene in real time; Optimize VR / AR scene interaction latency through spatial chunking loading and asynchronous rendering; It supports WebXR interfaces and cross-terminal adaptation, enabling immersive browsing and interaction of cultural resources.

[0061] Specifically, a laser scanner is used to collect 3D point cloud data of cultural sites and traditional artifacts. The original point clouds are then denoised, registered, and reconstructed into a mesh to generate a basic 3D mesh model. A Level of Detail (LOD) algorithm is used to generate multi-level precision models, for example: LOD0: high-precision, fully detailed model (for close-up display); LOD1: medium-precision model (reducing polygon count by 20%–50%); LOD2: low-precision model (reducing polygon count by more than 70%). The simplification algorithm can employ quadtree / octree hierarchical structures or a progressive mesh simplification algorithm based on edge folding. The appropriate LOD model is dynamically selected based on the user terminal's performance metrics (such as video memory size and GPU performance) and real-time rendering frame rate.

[0062] Semantic tags (such as technique steps, artifact names, and historical events) for cultural elements are obtained from the knowledge graph. The corresponding 3D coordinates or region boundaries of entities are then marked on the model. High-resolution image textures are mapped onto the LOD model surface and located using UV coordinates. Using the method of binding semantic tags to model regions, "hotspot information" is dynamically overlaid in the scene. For example, a historical background description is displayed floating above the "celadon kiln" location; technique steps are labeled in the area showing the inheritor's actions. Hotspot information can be presented in the form of transparent information windows or interactive buttons.

[0063] Large-scale 3D scenes are spatially segmented, for example using octrees or meshes, dividing the entire cultural venue into several sub-blocks. Based on the user's field of view and location, only sub-block data within the current field of view is loaded to reduce memory usage. Multi-threading or task queues are used to load data for distant or undisplayed sub-blocks in the background. Data within the user's field of view is rendered first in the foreground to ensure low latency (<50ms) during VR / AR interaction. Sub-blocks that may enter the user's field of view are pre-loaded through gaze prediction and motion trajectory prediction to achieve a smooth transition.

[0064] Integrating WebXR interfaces based on Unity3D or Unreal Engine, the system outputs rendering results to browsers that support the WebXR standard (such as Chrome and Edge). VR / AR scenes can be experienced on the web without installing a separate app. The system automatically detects the terminal type (such as PC, mobile phone, VR headset) and performance metrics, dynamically selecting the rendering pipeline, LOD precision, and effects on / off. On mobile devices, some high-cost effects (such as real-time shadows and complex particle effects) are disabled to ensure a stable frame rate above 30fps.

[0065] Leveraging LOD (Level of Detail) and spatial chunking loading technologies, the 3D displays of large-scale cultural venues or artifacts can run smoothly across multiple terminal devices. Through semantic tag binding and hotspot overlay, users can gain an immersive cultural knowledge experience through VR / AR interaction. Support for WebXR interfaces and cross-platform adaptation significantly lowers the barrier to entry and enables widespread dissemination.

[0066] Example 2: The silk weaving technique in a certain region is a typical intangible cultural heritage. Its weaving process involves multiple steps, including setting up the loom, threading heddles, beveling, and finishing, and is complex and relies heavily on highly skilled manual operation. Traditional digitization methods suffer from the following problems: the process is lengthy, making it difficult to record details completely; the weaving site is noisy, with dialects mixed with environmental noise, resulting in low accuracy of speech transcription; multimodal data is scattered and lacks structured knowledge graph support, failing to provide reliable data for subsequent immersive VR / AR scene displays.

[0067] To address the aforementioned issues, a system incorporating an integrated multimodal intelligent data governance and management method for cultural resources was adopted. The process of this method is as follows: Figure 2 As shown. The specific implementation process of this method is as follows: A multimodal data acquisition environment was established in the silk weaving workshop: multiple high-definition cameras were used to simultaneously capture the entire silk weaving process of the inheritor from different angles, including steps such as threading, mounting on the loom, weaving, and finishing; directional microphones and ambient sound acquisition devices were configured to capture the inheritor's dialect explanations, the sound of the loom operating, and the surrounding ambient sounds; a laser scanner was used to acquire 3D point cloud data of the loom equipment, finished silk products, and the working environment. All acquisition devices were activated through a global clock, and a sliding window algorithm was used to dynamically align the video, audio, and point cloud data in time to ensure data consistency.

[0068] Video data is input into a multi-branch convolutional neural network (such as a combination of ResNet and TCN) to automatically identify technical actions such as "threading heddles, weft insertion, and shuttle changing," and to generate skill labels for each step. Audio data is input into a deep learning speech recognition model that supports dialect adaptation to automatically transcribe the content explained by the inheritors, and to optimize low-confidence recognition results through a contextual rescoring mechanism. Semantic segmentation is performed on image frames and point cloud data to automatically identify cultural elements such as "loom pedals," "heddles and reeds," and "silk weaving patterns."

[0069] Based on the silk weaving technique ontology database, the parsed data is structured and managed to construct a multi-level ontology structure of "technique process - tools and implements - inheritors - locations - finished products". A rule engine is used to perform semantic similarity matching and cross-modal feature comparison, accurately mapping video action tags and audio text to corresponding technique steps, tools and implements, or finished product entities. Low-confidence mapping results trigger manual review, and manually corrected samples are fed back to the model for semi-supervised optimization. Completeness checks are performed on the mapped structured data to detect missing required fields, cross-modal semantic conflicts, and duplicate entities, generating quality reports and triggering automatic re-collection or manual confirmation.

[0070] The processed data is transformed into RDF triples and stored in a graph database, with time-series and geographic coordinate indexes automatically created. Graph attention networks are used to analyze the knowledge graph, uncovering potential skill-tool-finished product relationships, such as specific loom parts required for a particular silk weaving pattern; and inferring possible master-apprentice chains between different inheritors to complete missing relationships in the graph.

[0071] The system uses Level-of-Depth (LOD) models to generate 3D models of silk weaving workshops and looms, dynamically selecting the rendering precision based on the user's terminal performance. When a user browses a section of the loom, the system displays a functional introduction to that component and explanations of related techniques; clicking the "weft insertion" step plays the corresponding technique demonstration video and expert audio explanations. Users can access an immersive interactive experience of silk weaving techniques via a WebXR browser, mobile phone, or VR headset.

[0072] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An integrated multimodal intelligent data governance and management system for cultural resources, characterized in that, include: The data acquisition module is used to synchronously record the process of inheritors demonstrating their skills using video equipment, acquire audio data using environmental sound acquisition equipment, generate three-dimensional point cloud data of cultural sites and traditional artifacts using laser scanners, and perform time synchronization across multiple devices. The intelligent processing module is used to input video data into a convolutional neural network for action recognition, perform deep learning speech recognition transcription on audio data, run semantic segmentation algorithms on image data to label cultural elements, and extract cultural knowledge entities from text through named entity recognition algorithms. The data governance module is used to perform ontology mapping on the parsed data based on the ontology library in the field of intangible cultural heritage, generate a structured description that conforms to the data model, and perform integrity checks on the generated data and hierarchical access control based on the identity authentication of the inheritors. The knowledge organization module is used to store standardized data in the form of RDF triples in a graph database, add spatial coordinates to cultural resources through geocoding, establish a timestamp historical evolution index, and use graph neural networks to discover semantic relationships between cultural resources. The interactive display module is used to provide semantic retrieval based on knowledge graphs through SPARQL queries, generate personalized recommendations using collaborative filtering algorithms, and render 3D data into display content using the Unity3D engine.

2. The integrated multimodal cultural resource intelligent data governance and management system according to claim 1, characterized in that, The steps for time synchronization across multiple devices are as follows: At the start of data collection, a unified timestamp is generated for each data collection device based on the global clock. The time deviation between devices is calculated by detecting characteristic events in the collected data. A sliding window alignment algorithm is used to perform dynamic time correction on the multimodal data stream, and the order of the acquired frames is rearranged in the data buffer to ensure the time consistency of video frames, audio frames and point cloud frames.

3. The integrated multimodal cultural resource intelligent data governance and management system according to claim 1, characterized in that, The steps for action recognition using a convolutional neural network are as follows: The sequence of pose key points from the inheritor's action video and the RGB frame features are input in parallel into a multi-branch convolutional neural network; The network was trained using transfer learning and few-sample data augmentation strategies, including mirror transformation, temporal slicing, and action interpolation augmentation, to adapt to the scarcity of data samples of intangible cultural heritage skills. Based on the output action category sequence, the skill process is further decomposed and skill step labels are established to form skill entity information that can be used for ontology mapping.

4. The integrated multimodal cultural resource intelligent data governance and management system according to claim 1, characterized in that, The steps for performing deep learning speech recognition transcription are as follows: An acoustic model is constructed based on sub-word units, and the language model is expanded by combining a multi-dialect phoneme dictionary. An adaptive acoustic modeling method is adopted, and the Mandarin corpus model is adapted to a specific dialect corpus through transfer learning. During the recognition process, dialect language models are dynamically switched, and recognition results with low confidence are optimized through a context rescoring mechanism; The results of named entity recognition are used to correct specific cultural terms.

5. The integrated multimodal cultural resource intelligent data governance and management system according to claim 1, characterized in that, The steps for ontology mapping of parsed data are as follows: Construct a multi-level ontology structure consisting of a concept layer, an attribute layer, and an instance layer; The mapping relationship between multimodal data tags and ontology entities is analyzed through a rule engine. The rules include a composite logic based on regular expressions, semantic similarity calculation, and cross-modal feature comparison. For entities that cannot be automatically mapped, a manual verification interface is invoked for semi-supervised correction. The final mapping result is converted into RDF triples and written into the knowledge graph.

6. The integrated multimodal cultural resource intelligent data governance and management system according to claim 1, characterized in that, The steps for data integrity checking are as follows: After the multimodal data collection is completed, field integrity verification rules are established based on the predefined intangible cultural heritage resource data model to detect missing fields, primary keys and related attributes; A cross-modal consistency verification algorithm is used to perform logical alignment detection on video, audio, point cloud and text data, including collection timestamp comparison, event semantic cross-validation and content context matching; The anomaly detection model identifies noisy, duplicate, and conflicting entities in the data and classifies and labels them based on confidence scores. The inspection results are recorded in the quality report, and non-compliant data is automatically re-collected or manually reviewed to ensure the structural integrity and data reliability of the knowledge graph.

7. The integrated multimodal cultural resource intelligent data governance and management system according to claim 1, characterized in that, The steps for graph neural networks to discover semantic associations are as follows: Construct a heterogeneous graph with inheritors, skills, artifacts, places, and time nodes as multiple types of nodes; The weighted information transfer between nodes is calculated based on graph attention network, and the node feature representation is updated by combining the spatiotemporal attributes of the nodes and the relationship type of the edges. Calculate the potential association probability between nodes using a multi-hop neighbor aggregation strategy; Incremental learning updates are performed on node relationships with association confidence scores below a threshold to support the semantic expansion of cultural resources.

8. The integrated multimodal cultural resource intelligent data governance and management system according to claim 1, characterized in that, The steps to render 3D data into display content are as follows: A multi-level detail model is constructed for laser scanning point cloud data, and the rendering accuracy is dynamically selected based on device performance; By binding image textures with semantic tags, hot information about cultural elements is overlaid onto a 3D scene in real time; Optimize VR / AR scene interaction latency through spatial chunking loading and asynchronous rendering; It supports WebXR interfaces and cross-terminal adaptation, enabling immersive browsing and interaction of cultural resources.

Images