Fragmented data cross-modal label generation system and method based on deep transfer learning

Abstract

The invention provides a fragment data cross-modal label generation system and method based on deep transfer learning. The method comprises the following steps: extracting first high-dimensional feature vectors in different modes; mapping the first high-dimensional feature vectors of different modals into the same semantic space through a cross-modal comparison loss function to realize multi-modal alignment and fusion to obtain second high-dimensional feature vectors; labeling semantic tags corresponding to the second high-dimensional feature vectors based on the fragmented data components by adopting a small sample transfer learning algorithm; a multi-channel Hash encoder is adopted, a self-adaptive encoding strategy is called according to different modal data combinations, and the second high-dimensional feature vector is encoded into a multi-channel binary Hash code; in combination with an incremental graph neural network, the binary hash codes and the corresponding semantic tags are dynamically expanded into the historical knowledge graph; matched fine-grained tags are established for semantic differentiation features of different entity combinations in the target knowledge graph, and a cross-modal tag tree is obtained by combining three-matrix hierarchical construction.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence. More specifically, the embodiments of the present application relate to a system and method for generating cross-modal labels for fragmented data based on deep transfer learning. Background Art

[0002] The rapid development of information technology has ushered in the full arrival of the digital age. The amount of data generated across various industries is growing exponentially, and the widespread presence of fragmented data has become a key characteristic of the data landscape. This fragmented data encompasses multiple modalities, including text, images, audio, and video. Each modality expresses semantic information in a unique way. For example, text data conveys information through vocabulary, grammar, and semantic structure, while images convey content through color, shape, and spatial layout. Audio conveys meaning through waveform, frequency, and timbre. Differences between modalities are reflected not only in the physical storage format of the data but also in its semantic expression logic. This diversity increases the complexity of data processing and analysis and poses challenges to cross-modal data integration.

[0003] In related technologies, fragmented data exhibits weak semantic relevance due to its inherently incomplete information and loose structure. In practical applications, data often becomes fragmented due to limitations in collection conditions and transmission losses. For example, user reviews, product images, and sales records on e-commerce platforms are stored independently. There is no natural connection between the textual descriptions in the reviews, the visual features of the product images, or the time-series data of the sales records, making it difficult to directly establish semantic mappings. Traditional data processing methods, such as rule-based association analysis or simple feature concatenation, are unable to effectively mine these potential semantic connections. Consequently, subsequent label generation fails to accurately reflect the core content of the data, limiting the depth and breadth of data analysis. Furthermore, the rapid expansion of data volumes has made processing fragmented data on the scale of tens of millions or even hundreds of millions commonplace. Consequently, various technical fields require rapid analysis and labeling to support business decision-making and service optimization. However, traditional label generation methods struggle to meet practical requirements in terms of efficiency, accuracy, and semantic relevance mining when dealing with massive amounts of data. Therefore, a more efficient technical solution is urgently needed to address at least one of the aforementioned technical issues. Summary of the Invention

[0004] In this context, the embodiments of the present application hope to provide a fragmented data cross-modal label generation system and method based on deep transfer learning, which can realize the automatic construction of cross-modal label trees, solve the technical problem of poor semantic relevance caused by the fragmentation of multimodal data, improve the precision of semantic description, and improve the efficiency of label generation.

[0005] In a first aspect of the embodiments of the present application, a method for generating cross-modal labels for fragmented data based on deep transfer learning is provided, comprising:

[0006] Obtaining raw multimodal data to be processed; using a multi-branch deep feature extraction network, and applying different deep learning models based on the characteristics of different modal data, extracting first high-dimensional feature vectors under different modalities from the raw multimodal data;

[0007] Through the cross-modal contrast loss function, the first high-dimensional feature vectors under different modalities are mapped to the same semantic space, achieving multimodal alignment and fusion processing of the first high-dimensional feature vectors, and obtaining the second high-dimensional feature vector with unified semantic measurement;

[0008] Using a small sample transfer learning algorithm, based on the fragmented data components in the original multimodal data, annotate the semantic labels corresponding to the second high-dimensional feature vector;

[0009] Using a multi-channel hash encoder, calling different adaptive encoding strategies for different modal data combinations, and encoding the second high-dimensional feature vector into a multi-channel binary hash code;

[0010] Combined with the incremental graph neural network IncGNN, the binary hash code and the corresponding semantic label are dynamically expanded into the historical knowledge graph to obtain an updated target knowledge graph, including: converting the binary hash code corresponding to the newly added semantic label into a real-valued vector through a hyperbolic tangent function; utilizing the incremental update mechanism of IncGNN to aggregate the converted real-valued vector with the semantically similar node embeddings in the historical entities of the historical knowledge graph to obtain the newly added entity; calculating the semantic distance between the newly embedded new entity and the historical entity through cosine similarity, measuring the proximity between the new entity and the historical entity in the feature space, to obtain the semantic similarity between the new entity and the historical entity; if the semantic similarity exceeds a set threshold, establishing a relationship edge between the new entity and the historical entity according to the semantic association relationship of the cross-modal features, and setting the relationship weight between the new entity and the historical entity according to the Hamming similarity of the hash code and the semantic similarity, so as to complete the update of the historical knowledge graph and obtain the target knowledge graph;

[0011] A graph convolutional network (GCN) including a local structure enhancement contrast module is used to establish matching fine-grained labels for the semantic differentiation features of different entity combinations in the target knowledge graph. Combined with the first label association matrix, the second label association matrix, and the third label association matrix, the fine-grained labels in the target knowledge graph are hierarchically constructed to obtain a cross-modal label tree corresponding to the target knowledge graph.

[0012] In a second aspect of the embodiments of the present application, a system for generating cross-modal labels for fragmented data based on deep transfer learning is provided, comprising:

[0013] An acquisition module is used to acquire the original multimodal data to be processed; a multi-branch deep feature extraction network is used to extract the first high-dimensional feature vectors under different modalities from the original multimodal data using different deep learning models according to the characteristics of different modal data;

[0014] A unification module is used to map the first high-dimensional feature vectors under different modalities into the same semantic space through a cross-modal contrast loss function, realize multimodal alignment and fusion processing of the first high-dimensional feature vectors, and obtain a second high-dimensional feature vector with a unified semantic metric;

[0015] a labeling module, configured to label semantic labels corresponding to the second high-dimensional feature vector based on the fragmented data components in the original multimodal data using a small sample transfer learning algorithm;

[0016] an encoding module, configured to employ a multi-channel hash encoder, invoke different adaptive encoding strategies for different modal data combinations, and encode the second high-dimensional feature vector into a multi-channel binary hash code;

[0017] An update module is used to dynamically expand the binary hash codes and corresponding semantic labels into the historical knowledge graph in combination with the incremental graph neural network IncGNN to obtain an updated target knowledge graph; the binary hash codes corresponding to the newly added semantic labels are converted into real-valued vectors through the hyperbolic tangent function; the incremental update mechanism of IncGNN is used to aggregate the converted real-valued vectors with the semantically similar node embeddings in the historical entities of the historical knowledge graph to obtain newly added entities; the semantic distance between the newly embedded newly added entities and the historical entities is calculated by cosine similarity to measure the proximity between the newly added entities and the historical entities in the feature space to obtain the semantic similarity between the newly added entities and the historical entities; if the semantic similarity exceeds a set threshold, a relationship edge is established between the newly added entity and the historical entity according to the semantic association relationship of the cross-modal features, and the relationship weight between the newly added entity and the historical entity is set according to the Hamming similarity of the hash code and the semantic similarity to complete the update of the historical knowledge graph and obtain the target knowledge graph;

[0018] The labeling module is used to use a graph convolutional network (GCN) including a local structure enhancement contrast module to establish matching fine-grained labels for the semantically differentiated features of different entity combinations in the target knowledge graph, and to hierarchically construct the fine-grained labels in the target knowledge graph by combining the first label association matrix, the second label association matrix, and the third label association matrix to obtain a cross-modal label tree corresponding to the target knowledge graph.

[0019] In a third aspect of the embodiments of the present application, a terminal device is provided, comprising: at least one processor, a memory, and an input / output unit; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the method for generating cross-modal labels for fragmented data based on deep transfer learning as described in any one of the first aspects.

[0020] In a fourth aspect of the embodiments of the present application, a computer-readable storage medium is provided, which includes instructions that, when executed on a computer, enable the computer to execute the method for generating cross-modal labels for fragmented data based on deep transfer learning as described in any one of the first aspects.

[0021] In a fifth aspect of the embodiments of the present application, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method for generating cross-modal labels for fragmented data based on deep transfer learning as described in any one of the first aspects.

[0022] According to the implementation method of the present application, the fragmented data cross-modal label generation system and method based on deep transfer learning have achieved systematic breakthroughs in many aspects such as multimodal data processing and small sample learning, providing an effective path for intelligent data governance. In the feature extraction and fusion stage, for different modal data such as images, text, and audio, a multi-branch deep feature extraction network is used to adapt a dedicated deep learning model to fully mine the unique information of the data. Using the cross-modal contrast loss function, the feature vectors of each modality are mapped to a unified semantic space to achieve deep alignment and fusion of multi-source data, and improve data consistency and interpretability. In the semantic label generation link, a small sample transfer learning algorithm is introduced. With the help of the prior knowledge of the pre-trained model, it can be efficiently generalized with a small number of labeled samples, reducing the dependence on manual labeling and shortening the model training cycle. In the face of the real-time processing needs of massive data, the multi-channel hash encoder and the incremental graph neural network work together. The former dynamically encodes according to the modal combination, converts high-dimensional features into lightweight binary hash codes, and optimizes storage and computing efficiency; the latter realizes the dynamic update of the knowledge graph, supporting the real-time processing and knowledge precipitation of tens of millions of data. When constructing a knowledge graph, a graph convolutional network with a local structure enhancement contrast module is used to deeply model entity relationships, identify semantically differentiated features, avoid label confusion, and construct a hierarchical cross-modal label tree to enhance the sophistication of semantic descriptions. This technical solution systematically addresses the challenges of data fragmentation and multi-source heterogeneity, enabling automated, efficient, and low-cost data processing. It achieves high standards in feature extraction, semantic alignment, knowledge accumulation, and label generation, providing reliable technical support for multimodal data governance.

[0023] In terms of technical effects, firstly, this application addresses the fragmentation of multimodal data due to its dispersed sources and heterogeneous structures by achieving modality-adaptive feature decoupling and reorganization through a multi-branch deep feature extraction network. It dynamically selects the optimal deep learning model (e.g., CNN for visual texture processing, Transformer for text semantics analysis) for different modal characteristics, such as image, text, and audio, breaking through the traditional single-model forced alignment of multimodal data. Combined with a cross-modal contrastive loss function, it maps fragmented features into a unified semantic space, effectively eliminating the semantic gap between modalities and enabling fine-grained feature alignment of data from different sources. Secondly, through a hierarchical feature interaction mechanism and a dynamic weight allocation strategy, it achieves deep synergy of multimodal features. During the feature fusion stage, a multi-channel hash encoder adaptively selects encoding paths for different modal combinations (e.g., image-text, audio-video). A cross-attention mechanism is used to strengthen associations between highly correlated modalities (e.g., product images and text), while adversarial training is used to improve robustness for less correlated modalities (e.g., ambient audio and product parameters). Finally, this application integrates incremental graph neural networks (IncGNN) with multi-channel hash coding technology to build a dynamically evolving knowledge ecosystem. Faced with massive fragmented data streams, feature dimensionality reduction is achieved through hash coding, and at the same time, the local incremental update mechanism of IncGNN is utilized to complete the dynamic expansion of the knowledge graph with only local node calculations. Combined with the local structure enhancement comparison module, the entity relationship recognition accuracy of the knowledge graph is improved, and the fine-grained expansion capability of the label tree is improved. In short, this application can realize the automated construction of cross-modal label trees, solve the technical problem of poor semantic relevance caused by the fragmentation of multimodal data, improve the precision of semantic description, and improve the efficiency of label generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A flowchart of a method for generating cross-modal labels for fragmented data based on deep transfer learning provided in one embodiment of the present application;

[0025] Figure 2 A schematic diagram of the structure of a cross-modal label generation system for fragmented data based on deep transfer learning provided in one embodiment of the present application;

[0026] Figure 3 The structural diagram of a medium in an embodiment of the present application is schematically shown. DETAILED DESCRIPTION

[0027] Reference below Figure 1 , Figure 1 This is a flow chart of a method for generating cross-modal labels for fragmented data based on deep transfer learning provided in one embodiment of the present application. It should be noted that the implementation methods of the present application can be applied to any applicable labeling scenario.

[0028] Figure 1The process of the method for generating cross-modal labels for fragmented data based on deep transfer learning provided in an embodiment of the present application includes:

[0029] Step S101: obtaining original multimodal data to be processed.

[0030] In step S102 , a multi-branch deep feature extraction network is used to extract first high-dimensional feature vectors under different modalities from the original multimodal data using different deep learning models according to the characteristics of different modal data.

[0031] In step S103, the first high-dimensional feature vectors under different modalities are mapped into the same semantic space through a cross-modal contrast loss function, so as to realize multimodal alignment and fusion processing of the first high-dimensional feature vectors and obtain a second high-dimensional feature vector with a unified semantic metric.

[0032] Step S104: Using a small sample transfer learning algorithm, based on the fragmented data components in the original multimodal data, annotate the semantic label corresponding to the second high-dimensional feature vector.

[0033] Step S105 : Using a multi-channel hash encoder, calling different adaptive encoding strategies for different modal data combinations, and encoding the second high-dimensional feature vector into a multi-channel binary hash code.

[0034] In step S106, combined with the incremental graph neural network IncGNN, the binary hash code and the corresponding semantic label are dynamically expanded into the historical knowledge graph to obtain an updated target knowledge graph; the KL divergence between the newly added entities and the historical entities in the target knowledge graph is calculated. If the KL divergence is greater than the set threshold, the semantic drift is determined and local re-clustering is triggered, and the target knowledge graph is dynamically updated.

[0035] Step S107: Use a graph convolutional network (GCN) including a local structure enhancement contrast module to establish matching fine-grained labels for the semantically differentiated features of different entity combinations in the target knowledge graph. Combined with the first label association matrix, the second label association matrix, and the third label association matrix, the fine-grained labels in the target knowledge graph are hierarchically constructed to obtain a cross-modal label tree corresponding to the target knowledge graph.

[0036] Through the above steps S101 to 107, first, a multi-branch deep feature extraction network is used to adapt the most appropriate deep learning model for feature extraction based on the data characteristics of different modalities such as images, text, and audio. This specialized processing method can fully tap the unique information of each modal data and lay a solid foundation for subsequent analysis. After extracting high-dimensional features, an innovative cross-modal contrast loss function is used to map the feature vectors from different modalities into a unified semantic space. This mapping mechanism effectively bridges the semantic gap between modalities, achieves deep alignment and organic fusion of multi-source data, and enables the fused feature vectors to have a unified and comparable semantic metric standard, thereby improving the consistency and interpretability of cross-modal data. In the semantic label generation link, a small sample transfer learning algorithm is introduced to make full use of the rich semantic prior knowledge accumulated by the pre-trained model. This strategy is particularly suitable for the actual situation where annotation resources are scarce in fragmented data environments. Only a small number of annotated samples are required to efficiently generalize to new data scenarios. This learning mechanism greatly reduces the reliance on large-scale manual annotation, which not only reduces the workload of manual annotation, but also shortens the model training cycle and improves annotation efficiency. Furthermore, to meet the real-time processing needs of massive amounts of fragmented data, a multi-channel hash encoder and an incremental graph neural network collaborate to dynamically select the optimal encoding strategy based on different modal combinations, converting high-dimensional feature vectors into lightweight binary hash codes. This representation preserves key semantic information while optimizing storage space and computational efficiency. The introduction of an incremental graph neural network (IncGNN) addresses the key challenge of real-time updates. This technical solution dynamically integrates newly generated hash codes and labels into the existing knowledge graph without complex global reconstruction, enabling seamless online expansion of the knowledge base. This architecture provides breakthrough support for real-time processing and knowledge accumulation of tens of millions of data points, enabling the continuous evolution of the data governance system. During the knowledge graph and labeling system construction phase, a graph convolutional network with a local structure enhancement contrast module is introduced to deeply model the complex entity relationships within the knowledge graph. This module accurately identifies the semantically distinct features between different entity combinations, and through a contrastive learning mechanism, strengthens the representation of key differential information, effectively avoiding label confusion. Based on this sophisticated semantic modeling, the system constructs a hierarchical, cross-modal label tree structure, forming a semantic progression from macro-concepts to micro-attributes. This tree-like labeling architecture significantly improves the sophistication and coverage of semantic descriptions, bringing the recall of key features to new heights. The resulting knowledge graph possesses extremely rigorous structure and comprehensive knowledge, providing a solid foundation for semantic understanding for downstream tasks.Overall, it has demonstrated systematic breakthroughs in multimodal data processing, small sample learning, real-time knowledge updating, and fine-grained label construction. It can effectively solve the problems of data fragmentation and governance of multi-source heterogeneous data, and achieve automated, efficient, and low-cost large-scale data processing. It can also cover the refinement of feature extraction, the precision of semantic alignment, the real-time knowledge precipitation, and the granularity of label generation, providing a practical technical path for the intelligent governance of multimodal data.

[0037] Step S101: obtaining original multimodal data to be processed.

[0038] In the embodiments of the present application, original multimodal data, i.e., fragmented data or fragmented data, is a heterogeneous data set containing multiple forms of expression such as text, images, audio, video, and time series signals. These data come from a wide range of sources, each carrying information of different dimensions, and exist in a fragmented state, making it difficult to directly perform unified analysis. For example, in the field of e-commerce, original multimodal data covers text descriptions of product introductions, actual product photos, text content of user reviews, and product display videos. In industrial scenarios, sensor time series data generated by equipment operation, audio signals of equipment fault alarms, and image data of equipment appearance or internal structure all fall into the category of original multimodal data.

[0039] In the embodiments of the present application, raw multimodal data, also known as fragmented data, refers to a heterogeneous data set composed of multiple modalities that has not been preprocessed or structured. The modalities here may include but are not limited to: text (such as product titles, descriptions, user reviews, etc.), images (such as actual photos of products, photos of the appearance or internal structure of equipment, etc.), audio (such as equipment failure alarm sounds, user voice comments, etc.), video (such as product displays, production process monitoring, etc.), time series signals (such as continuous data such as temperature, current, vibration, etc. collected by sensors). These data usually come from multiple sources, have different load information dimensions, and are stored in the form of multiple time periods and discrete fragments, making it difficult to directly perform unified correlation analysis or feature extraction.

[0040] Taking e-commerce scenarios as an example, raw multimodal data includes, but is not limited to, product descriptions, product images, unboxing or review videos, and user reviews with accompanying images. Taking industrial scenarios as an example, raw multimodal data also includes, but is not limited to, multi-channel sensor time series data generated during equipment operation, alarm audio signals, and images or videos of the device's appearance and internal structure. Unified modeling, alignment, and fusion of this raw multimodal data provide fundamental data support for subsequent intelligent analysis and decision-making.

[0041] In the embodiments of this application, multimodality refers to the multiple data types or structures of data information. By integrating multiple types of data (i.e., multimodal data) such as visual, auditory, and textual inputs, the system's ability to understand complex information is enhanced. For example, autonomous driving systems combine camera images, radar data, and GPS information for environmental perception, or medical diagnostics integrate imaging, medical records, and laboratory data to improve accuracy.

[0042] The methods for acquiring raw multimodal data vary depending on the data source. In internet applications, web crawler technology is used to extract text, images, videos, and other data from data sources such as web pages and social media platforms according to established rules. For example, news information platforms use crawlers to retrieve news articles (text), accompanying images (images), and related video content from the internet, enabling real-time collection of massive amounts of news information. In IoT scenarios, data collection is carried out using various sensors. Devices such as temperature sensors, pressure sensors, and cameras collect real-time data such as environmental parameters and device status information from the physical world. For example, temperature and humidity sensors in smart buildings continuously collect indoor temperature and humidity data (time series data), while cameras capture images of people's movements (image data), providing basic data for intelligent building management. Within enterprise systems, APIs are used to obtain structured or semi-structured multimodal data from databases and business systems, such as customer profile text and customer service call recordings (audio), enabling the integration and circulation of enterprise data.

[0043] Therefore, on the one hand, the diversity of multimodal data can more comprehensively characterize the characteristics of objects, avoiding the limitations of single-modal data. For example, in the field of intelligent security, combining surveillance video (images, time series) with abnormal sound detection (audio) data can more accurately identify security events such as intrusions and fires, significantly reducing the error rate compared to relying solely on video data. On the other hand, the acquisition of raw multimodal data enables the system to handle complex and real-world application scenarios, meeting the needs of different industries for in-depth data analysis. Taking medical diagnosis as an example, doctors can combine multimodal data such as patient medical records, medical imaging (images), and electrocardiogram signals (time series) to make more accurate diagnostic decisions, thereby improving the accuracy and reliability of medical diagnoses.

[0044] It can be further understood that multimodality refers to the technical scope of collaborative processing and fusion of any two or more data types in the above-mentioned raw multimodal data. By integrating information from different modalities at the feature layer or decision layer, the system can obtain more complete semantics and spatiotemporal context, thereby improving the perception, understanding, and reasoning capabilities of complex scenarios. For example, in autonomous driving scenarios, camera images, laser or millimeter-wave radar point clouds, and positioning signals (GPS / IMU) are combined for environmental perception and vehicle positioning. Taking medical diagnosis as an example, the integration of medical images, electronic medical records, and laboratory test results can improve the accuracy of disease screening and diagnosis.

[0045] In step S102 , a multi-branch deep feature extraction network is used to extract first high-dimensional feature vectors under different modalities from the original multimodal data using different deep learning models according to the characteristics of different modal data.

[0046] As an optional embodiment, in step S102, the modality to which each raw data in the raw multimodal data belongs is identified, and the raw data of different modalities are preprocessed; the preprocessed raw text sequence is input into the BERT and BiLSTM joint model, the BERT pre-training model is used to extract the contextual semantic features of the raw text sequence described in the text, and BiLSTM is used to capture the implicit bidirectional dependency of the fragmented data in the raw text sequence through sequence modeling to obtain a text semantic vector under the text modality; the preprocessed raw image matrix is ​​passed through the ResNet-50 network to extract image visual features, and then the global semantic features are extracted by the Transformer model to obtain a visual semantic vector under the image modality; the preprocessed raw time series data is passed through the LSTM autoencoder to extract dynamic pattern features, and feature reconstruction is implemented through the LSTM autoencoder to retain time series dependencies to obtain a time series pattern vector under the image modality.

[0047] Specifically, a multi-branch deep feature extraction network is fundamental for cross-modal data feature mining. Its core lies in customizing a combination of deep learning models based on the characteristics of text, images, and time series data to extract precise, high-dimensional semantic features. Through modality recognition and preprocessing, the network standardizes the raw multimodal data, paving the way for subsequent deep feature extraction.

[0048] Understandably, cross-modality refers to a data mining method that establishes semantic associations and information interactions between different modalities (such as text, images, audio, and video). Its core is to bridge the semantic gap between modalities through algorithmic models, enabling information conversion, fusion, and collaborative processing across sensory channels, thereby enhancing AI systems' ability to understand and generate complex information.

[0049] The combined BERT and BiLSTM model plays a key role in feature extraction from text modalities. The BERT pre-trained model learns linguistic knowledge from massive amounts of text data and can understand the contextual semantics of text. For example, when processing the sentence "This phone has powerful performance, but poor battery life," BERT can accurately identify the semantic contrast between "performance" and "battery life." BiLSTM, on the other hand, uses bidirectional sequence modeling to capture the dependencies between fragmented information in text. For example, when analyzing the development of events in news reports, BiLSTM can combine contextual information to clarify the causal logic of events. Combined, the two transform raw text sequences into text semantic vectors rich in semantic and structural information, providing support for understanding text content and generating labels.

[0050] For example, newer pre-trained models such as the GPT series, ERNIE, or RoBERTa can be used. Their optimized training strategies and larger corpora can extract richer semantic information. Alternatively, dynamic masking techniques (such as SpanBERT) can be combined to enhance the ability to model the semantics of text fragments. Furthermore, text can be modeled as a graph structure (such as a semantic dependency graph) using graph convolutional networks (GCNs) or graph attention networks (GATs) to capture the complex relationships between words, thus overcoming the limitations of traditional sequence models.

[0051] Feature extraction from image modalities relies on the collaborative work of ResNet-50 and Transformer. As a convolutional neural network, ResNet-50 excels at extracting local visual features from images. For example, when recognizing car images, it can capture visual information about essential components like lights and tires. Transformer, building on this strength, utilizes a self-attention mechanism to strengthen its understanding of the global semantics of an image. When analyzing a city panorama, Transformer can integrate elements such as buildings, roads, and crowds to construct a complete scene semantics. The two work together to transform the original image matrix into a visual semantic vector that combines local details with global semantics, enabling semantic alignment of image information with data from other modalities.

[0052] In addition to the above models, you can also use Swin Transformer, Vision Pro Transformer

[0053] mer (ViT), whose windowed attention mechanism and layered architecture enable more efficient processing of local and global image information, is particularly suitable for feature extraction in complex scene images. Furthermore, lightweight models such as MobileViT and ShuffleNet V3 can be used to significantly reduce computational complexity while maintaining accuracy, making them suitable for edge devices or real-time processing scenarios.

[0054] LSTM autoencoders are used to extract features from time series data. LSTM's unique gating mechanism makes it suitable for processing time series and can memorize long-term dependencies. For example, when analyzing stock price trends, LSTM can record correlations between price changes at different points in time. Through an encoding-decoding process, the autoencoder compresses raw time series data into dynamic pattern feature vectors and then reconstructs the restored data, ensuring that key time series patterns and dependencies are preserved. For example, using industrial sensor data, the LSTM autoencoder can extract pattern features of abnormal fluctuations from time series data such as temperature and pressure during equipment operation, providing data support for fault warnings.

[0055] In addition to the aforementioned models, self-supervised contrastive learning frameworks such as SimCLRv2 and MoCo v3 can also be used to introduce dynamic dictionaries and memory mechanisms into cross-modal feature alignment to improve the semantic consistency of features across different modalities. Furthermore, the architectures of multimodal pre-training models such as CLIP (Contrastive Language Image Pretraining) and ALIGN can be used to learn cross-modal mapping relationships through large-scale image-text pair training, thereby enhancing feature fusion.

[0056] From a technical perspective, the multi-branch deep feature extraction network achieves specialized processing of data from different modalities. Through customized model combinations, the data characteristics of each modality are fully exploited, avoiding information loss when general models process multimodal data. The extracted high-dimensional feature vectors lay a solid foundation for subsequent cross-modal fusion and label generation, improving the semantic expression capabilities of multimodal data. This enables the system to more accurately capture the essential characteristics of data when processing complex and fragmented data, providing a guarantee for the accurate generation of multimodal labels.

[0057] It can be understood that, further optionally, in a multi-branch deep feature extraction network, different deep learning models work together through the standardization of data preprocessing, division of labor and cooperation in feature extraction, and the unified goal of cross-modal fusion, forming a complete link from heterogeneous data to unified semantic features.

[0058] During the data input phase, raw multimodal data (text, images, time series, etc.) first undergoes modality identification and standardization preprocessing, converting data of varying formats into a form suitable for processing by each branch network. For example, text data is unified into a sequence of word vectors, image data is resized into a fixed-size pixel matrix, and time series data is normalized into a sequence of equally spaced samples. This preprocessing provides a foundation for the stable operation of subsequent branch models, ensuring comparability of data from different modalities at the same scale.

[0059] When it comes to feature extraction, each branch network performs specialized tasks based on the data modality. In the text branch, the BERT pre-trained model leverages general semantic knowledge learned from a massive corpus to capture contextual information, while BiLSTM complements the long-range dependencies of the text through bidirectional sequence modeling. The combination of the two enables a progression from "semantic understanding" to "structural modeling." In the image branch, ResNet-50 first uses convolutional layers to extract local image features (such as edges and textures). The Transformer then integrates global semantics (such as scene layout and object relationships) through a self-attention mechanism, forming a collaborative "local perception-global cognition" model. In the time series branch, the LSTM autoencoder's encoding process compresses the dynamic patterns of the time series data, while the decoding process verifies and strengthens the temporal dependencies of the features, ensuring that the extracted features both capture the core patterns and preserve the essential properties of the time series. While these branch networks appear to operate independently, they complement each other through division of labor, avoiding information omission or semantic bias when a single model processes multimodal data.

[0060] The high-dimensional feature vectors extracted by each branch are ultimately synergistically fused through cross-modal contrastive learning. The cross-modal contrastive loss function acts as a connecting bridge, forcing the feature vectors of different modalities to be aligned in the semantic space. For example, in an e-commerce scenario, the semantic vectors extracted from the product description text and the visual vectors extracted from the product image are mapped to the same semantic space through contrastive learning, so that text keywords such as "fashion" and "durable" are associated with the design style and material details in the image. This synergy not only depends on the accuracy of feature extraction by each branch network, but also on the constraints of the feature space imposed by the cross-modal loss function, ensuring that information from different modalities can complement and verify each other, ultimately forming a fused feature with a unified semantic metric, providing comprehensive and comparable input for subsequent label generation.

[0061] Through the above-mentioned collaborative mechanism, the multi-branch deep feature extraction network breaks the modal barriers and transforms fragmented and heterogeneous raw data into structured and semantic feature expressions, providing core technical support for the in-depth analysis and intelligent label generation of multimodal data.

[0062] In step S103, the first high-dimensional feature vectors under different modalities are mapped into the same semantic space through a cross-modal contrast loss function, so as to realize multimodal alignment and fusion processing of the first high-dimensional feature vectors and obtain a second high-dimensional feature vector with a unified semantic metric.

[0063] As an optional embodiment, in step S103, the first high-dimensional feature vectors are converted into a unified feature dimension through independent encoders, and then the re-encoded first high-dimensional feature vectors are projected into a shared semantic space; feature vectors under different modalities are selected from the shared semantic space as anchor samples; feature vectors with the same semantics as the anchor samples are selected from the shared semantic space as positive samples, and feature vectors with different semantics from the anchor samples are selected as negative samples to obtain cross-modal sample pairs; a pre-trained CLIP model is used to generate continuous similarity weights for cross-modal sample pairs, and a cross-modal contrast loss function is constructed based on the continuous similarity weights; the continuous similarity weights are used to reflect the semantic association strength between different first high-dimensional feature vectors in the cross-modal sample pairs; the cross-modal contrast loss function is used to The loss function calculates the similarity distribution between cross-modal sample pairs, sets the dynamic boundary threshold with a ternary fuzzy boundary constraint, and adjusts the relative positions between cross-modal feature vectors in the shared semantic space with reference to the similarity distribution and the similarity distribution heat map constructed by the dynamic boundary threshold, so as to bring semantically similar samples closer and push semantically irrelevant samples further away; calculates the association matrix of cross-modal feature vectors in the shared semantic space through the attention cross mechanism; identifies cross-modal semantically related regions in the association matrix in combination with the similarity distribution heat map to achieve cross-modal semantic alignment; uses a learnable gating coefficient to weightedly fuse the cross-modal related feature vectors in the cross-modal semantically related regions and the original first high-dimensional feature vector, and performs mixed extraction on the weighted fusion results through multiple granularity layers to obtain the second high-dimensional feature vector.

[0064] In the embodiments of the present application, high-dimensional feature vectors, independent encoders, shared semantic space, anchor samples, positive samples, negative samples, pre-trained CLIP models, continuous similarity weights, cross-modal contrast loss functions, cross-modal sample pairs, ternary fuzzy boundary constraints, dynamic boundary thresholds, similarity distribution heat maps, correlation matrices, and learnable gating coefficients are provided.

[0065] Specifically, in step S103, the core of the cross-modal contrast loss function is to break the semantic barriers of different modal data such as text, image, and time series, and map heterogeneous features to a unified space by constructing a computable semantic association measurement system.

[0066] Taking the cross-modal alignment of product text descriptions and physical product images in e-commerce scenarios as an example, in step S103, independent encoders first perform dimensionality unification and semantic projection on features from different modalities. For example, the BERT features (768 dimensions) of the product text and the ResNet features (2048 dimensions) of the image are each converted to a 512-dimensional vector using a fully connected layer and then projected into a shared semantic space. In this case, the text features of "red dress" and the image features of the red skirt are considered potentially related samples, while the image features of "blue jeans" are considered semantically unrelated samples.

[0067] High-dimensional feature vectors are numerical vectors extracted from raw data that contain rich semantic information. In security scenarios, image feature vectors extracted from surveillance video frames may contain information such as the shape, color, and motion trajectory of the target object, and their dimensions can reach thousands of dimensions. Furthermore, text feature vectors derived from natural language processing of alarm texts encode semantic information such as the event description, time of occurrence, and location, and their dimensions can also reach hundreds of dimensions. These high-dimensional feature vectors serve as the foundation for subsequent processing, preserving the core information of the original data. However, due to their dimensionality and modal characteristics, direct correlation is difficult.

[0068] In the conversion module for data from different modalities, independent encoders are used to map high-dimensional feature vectors to a uniform dimension for subsequent processing. For example, the high-dimensional visual features of video images are compressed and converted into 512-dimensional vectors using a multi-layer neural network. Similarly, the semantic features of warning text are mapped to 512 dimensions. This approach aligns features from different modalities, much like machining parts of varying specifications to a uniform size, facilitating cross-modal fusion.

[0069] In the embodiment of the present application, the shared semantic space is an abstract mathematical space to which the feature vectors of different modalities are mapped after encoding to achieve a unified expression at the semantic level. In the security scenario, both the video image features and the alarm text features are projected into the shared semantic space after being processed by independent encoders. In this space, the video image features of the "break-in event" and the text features describing the event can be associated based on semantics, so that data originally belonging to different modalities can be compared and integrated in the same language environment.

[0070] Anchor samples are reference feature vectors selected from the shared semantic space and used to construct sample pairs and calculate similarity. For example, when processing security data at a specific moment, a text feature describing an alert "someone climbing over a wall" is selected as an anchor sample. This anchor sample serves as a benchmark for finding semantically related or unrelated samples.

[0071] Positive samples are feature vectors with the same or similar semantics as the anchor sample. In the above example, the surveillance video frame corresponding to the text description "Someone climbing over the wall" contains an image feature vector of a person climbing over the wall, which is a positive sample. The selection of positive samples aims to strengthen the model's ability to recognize semantic consistency, allowing the model to learn data features that express the same semantics across different modalities.

[0072] Negative samples are feature vectors with semantically different semantics from the anchor sample. For example, the textual features of a warning describing "vehicles passing normally" or the image features of a person walking normally in a surveillance video can serve as negative samples for the anchor sample "someone climbing over a fence." The introduction of negative samples helps the model distinguish different semantics and avoids incorrectly associating irrelevant information.

[0073] The selection of anchor samples and positive and negative samples follows the principle of semantic consistency. If the text feature of "red dress" is used as the anchor, the corresponding image feature of the red dress is the positive sample, while the image feature of blue jeans is the negative sample, forming a cross-modal triplet. The pre-trained CLIP model plays a key role in this. Its cross-modal understanding capabilities, acquired through large-scale image-text pair training, can calculate the similarity weights between the text of "red dress" and its image (e.g., 0.92) and the similarity weights between the text of "red dress" and its image (e.g., 0.21). These weights directly reflect the strength of semantic association.

[0074] It is understood that in the embodiments of this application, the cross-modal model, trained on large-scale image and text data, can understand the semantic connections between images and text. In security applications, the pre-trained CLIP model can calculate the semantic similarity between the text features of "someone climbing over the wall" and the corresponding surveillance footage features, and can also determine the differences between them and other unrelated images or text. Based on the knowledge learned from pre-training, similarity scores are assigned to feature pairs from different modalities.

[0075] Continuous similarity weights, generated by the pre-trained CLIP model, quantify the strength of semantic associations between pairs of cross-modal samples. For example, for the text "someone climbed over a wall" and the corresponding video, the CLIP model might assign a similarity weight of 0.9, indicating a high degree of semantic correlation. However, for the text "vehicles passing normally" and the video of "someone climbed over a wall," the similarity weight might be only 0.2, clearly reflecting the semantic difference.

[0076] The construction of the cross-modal contrastive loss function introduces a three-dimensional fuzzy boundary constraint. Taking the similarity distribution of anchor points, positive samples, and negative samples as an example, a dynamic boundary threshold is set to the positive sample similarity minus 0.3. When the negative sample similarity exceeds this threshold, the loss calculation is triggered. Visualizing the similarity distribution through a heat map, we can intuitively see areas of high similarity (warm colors) between the "red dress" text and the image, and areas of low similarity (cool colors) with the blue jeans image. The model adjusts feature positions accordingly, bringing similar semantic samples closer together (for example, reducing the distance between text and corresponding image features) and pushing out dissimilar samples further apart.

[0077] The cross-modal contrast loss function is a mathematical function constructed based on continuous similarity weights. It measures the degree of semantic match between pairs of cross-modal samples and guides model parameter adjustment. In security scenarios, this function calculates the similarity between the anchor sample (alarm text) and the positive sample (corresponding video footage), as well as the difference with the negative sample. By minimizing the loss, the model encourages semantically similar samples to be brought closer together in the shared semantic space, while pushing semantically unrelated samples further away.

[0078] A cross-modal sample pair is a combination of an anchor sample and either a positive or negative sample. For example, a positive sample pair consists of (text features of "someone climbing over a wall" and video features of climbing over a wall) and a negative sample pair consists of (text features of "someone climbing over a wall" and text features of a vehicle passing normally). These sample pairs serve as the basic units for model learning and optimization. By comparing the similarities between different sample pairs, the model gradually grasps the rules for cross-modal semantic alignment. A similarity distribution heatmap visualizes the similarity between feature vectors in a shared semantic space. In the security data analysis interface, the similarity distribution heatmap visually illustrates the correlation between different alarm text and surveillance video features, with darker colors indicating higher similarity. The similarity distribution heatmap allows operators to quickly identify potential connections in the data and provides a visual basis for model optimization.

[0079] Furthermore, the dynamic boundary threshold is a specific threshold derived from the ternary fuzzy boundary constraint and adjusts as sample data changes. During the ongoing operation of the security system, when new event types emerge or data distribution changes, the dynamic boundary threshold can be adaptively adjusted to ensure that the model is always semantically aligned with the appropriate standard, avoiding false positives or missed positives caused by fixed thresholds.

[0080] Furthermore, the ternary fuzzy boundary constraint is used as a mechanism to set similarity thresholds, defining a fuzzy boundary range based on the relationship between anchor samples, positive samples, and negative samples. In security data processing, this constraint dynamically determines a threshold range based on the similarity distribution of sample pairs. For example, if the similarity between a positive sample and an anchor sample is 0.9, and the similarity between a negative sample and an anchor sample is 0.2, the ternary fuzzy boundary constraint might set a threshold range of 0.6-0.8 to determine whether the semantic association between the sample pairs meets expectations.

[0081] The cross-attention mechanism further enhances semantic association mining. By calculating the correlation matrix of all feature vectors in the shared space, we can find high-weight connections between the "red" text feature and the red pixel region in the image, and the "dress" text feature and the skirt outline in the image. Combined with similarity heat maps, these high-weight regions are identified as cross-modal semantically related regions, such as the correspondence between text keywords and image visual focal points.

[0082] The correlation matrix, calculated through the attention cross-checking mechanism, is used to quantify the strength of semantic associations between all feature vectors in a shared semantic space. In security scenarios, the correlation matrix can demonstrate the correlation between text features of different alarm events and various surveillance image features, such as the correlation between the text "fire alarm" and features of flames and smoke. This provides a precise quantitative basis for subsequent semantic alignment and information fusion.

[0083] The final feature fusion utilizes a learnable gating coefficient. Specifically, the learnable gating coefficient, an adjustable parameter introduced during the feature fusion stage, controls the weighting of different modal features during the fusion process. For example, when fusing the textual features of "someone climbed over the wall" and the video features, the learnable gating coefficient automatically adjusts the weighting based on the strength of their association in the shared semantic space. If the textual description is clear and the video features are prominent, the gating coefficient might assign a weight of 0.6 to the textual features and 0.4 to the video features. This allows for the proper fusion of multimodal information, generating a more representative high-dimensional feature vector and improving the accuracy of security event detection and analysis.

[0084] For example, the gating coefficient automatically assigns a high weight (such as 0.8) to cross-modal features in semantically related areas (such as the "red" text feature and the red pixel image feature), and a low weight (such as 0.2) to irrelevant features. Then, semantic information of different scales (such as the global "dress" concept and the local "red" attribute) is extracted through multi-granularity layer mixing to generate a second high-dimensional feature vector containing multimodal semantics.

[0085] In terms of technical effectiveness, this mechanism improves the accuracy of semantic alignment between images and text on e-commerce platforms. When a user searches for "red dress," the system accurately matches images containing red hems, reducing mismatches of blue or trouser-related items. In medical scenarios, the cross-modal alignment error rate between pathology text descriptions and medical images is reduced, enabling precise association of "lung nodule" text with nodule areas in CT images, providing a solid foundation for semantic unification for intelligent analysis of multimodal data.

[0086] Step S104: Using a small sample transfer learning algorithm, based on the fragmented data components in the original multimodal data, annotate the semantic label corresponding to the second high-dimensional feature vector.

[0087] As an optional embodiment, in step S104, the fragmented data components in the original multimodal data are identified, and the domain relevance index of each fragmented data component is calculated; the domain relevance index represents the semantic overlap between each fragmented data component and the target domain; through parameter fine-tuning and knowledge distillation, the general pre-trained model is adaptively migrated to the target domain where the fragmented data components are located, and a small sample annotation model under the target domain is constructed; the parameter fine-tuning includes adding a fragmented domain adaptive adapter on the top layer of the general pre-trained model; the parameter sparsity of the fragmented domain adaptive adapter is dynamically adjusted based on the domain relevance index, and the lower the relevance, the higher the sparsity; the second high-dimensional feature vector is semantically annotated through the small sample annotation model to obtain the semantic label corresponding to the second high-dimensional feature vector.

[0088] In the multimodal data processing flow, step S104 uses a small-sample transfer learning algorithm to efficiently annotate semantic labels, effectively addressing the challenge of scarce annotated samples in fragmented data scenarios. The technical logic and application value of this process are fully demonstrated using the combined analysis of medical images and diagnostic text as an example.

[0089] The first step is to identify fragmented data components within the raw multimodal data. In the medical field, raw data includes image data such as patient X-rays and CT scans, as well as text data such as diagnostic records and medical history descriptions written by doctors. These data often exhibit fragmented characteristics: X-ray images may only record local lesions at a certain moment, and diagnostic texts are relatively brief in describing the condition. Identifying fragmented data components means sorting out the scattered but critical information within these data, such as locating the position and morphology of lung nodules from CT images, and extracting the preliminary judgment of "suspected pneumonia" from diagnostic records, providing raw material for subsequent model training.

[0090] Next, the general pre-trained model is transferred to the target domain through parameter fine-tuning and knowledge distillation. For example, the visual language model CLIP, pre-trained on large-scale image-text datasets, learns general image-text semantic associations. In medical scenarios, due to the limited number of annotated medical images and diagnostic text samples, directly training a new model is prone to overfitting. In this case, a parameter fine-tuning strategy is adopted. A domain adapter is added to the top layer of the CLIP model, freezing the underlying parameters. Only the adaptation layer is fine-tuned for medical data, adapting it to the specialized terminology and image features of the medical field. Simultaneously, knowledge distillation is used to construct a teacher-student model architecture. The teacher model is CLIP, trained on a general dataset, while the student model is learned on a small number of medical image and text pairs (e.g., 100 sets of pneumonia images and diagnostic descriptions). Soft labels output by the teacher model (e.g., the probability distribution of various diseases) guide the training of the student model. Knowledge is transferred through loss functions such as KL divergence, allowing the student model to quickly grasp the semantic association patterns in the medical field.

[0091] Finally, the constructed small-sample annotation model semantically annotates the second-highest-dimensional feature vector. When fed the second-highest-dimensional feature vector, which is the result of fusion of a patient's chest X-ray image and the associated medical record text in step S103, the small-sample annotation model can output precise semantic labels, such as "pneumonia (89% confidence)" and "thickened lung texture." These labels not only include the disease diagnosis but also describe the image features, providing doctors with auxiliary diagnostic information.

[0092] From a technical perspective, the small-sample transfer learning algorithm improves annotation efficiency and accuracy. In medical data annotation, compared to traditional full-data training models, this method reduces the sample size required for annotation, shortens the training cycle from weeks to days, and improves annotation accuracy. It also effectively avoids the data scarcity problem caused by the high privacy and high annotation costs of medical data, providing a viable technical solution for applications such as medical image-assisted diagnosis and disease prediction. It also provides a reference for semantic annotation in other small-sample scenarios (such as rare disease research and new material analysis), achieving efficient transfer of general knowledge to domain-specific knowledge.

[0093] Further optionally, in the above steps, the general pre-trained model is adaptively migrated to the target domain where the fragmented data component is located through parameter fine-tuning and knowledge distillation, and a small sample annotation model under the target domain is constructed, including: adding a fragmented domain adaptive adapter on the top layer of the general pre-trained model, freezing the underlying parameters, and fine-tuning the adaptation layer; the fragmented domain adaptive adapter includes a semantic completion sublayer for text fragments and a visual restoration sublayer for image fragments, and the activation conditions of each sublayer are set based on the modal type of the corresponding fragmented component; constructing a teacher-student model; using multiple differently initialized teacher models to generate integrated soft labels on a general dataset, when the student model learns in the fragmented dataset in the target domain where the fragmented data component is located, the output distribution of multiple teacher models is synchronously fitted through the KL divergence loss function; the output distribution of the teacher model is softened through the temperature parameter; wherein, when the student model is trained in the fragmented dataset, the temperature parameter is dynamically adjusted to balance the generalization and specificity of knowledge transfer; when the sample size of the fragmented dataset is less than a preset threshold, the temperature parameter is increased to enhance the smoothness of the soft label and reduce the risk of overfitting.

[0094] Specifically, in small-sample transfer learning scenarios, adapting general pre-trained models to the target domain through parameter fine-tuning and knowledge distillation is the core technical path for achieving efficient semantic annotation. For example, in industrial equipment fault diagnosis, equipment operating data is often fragmented and labeled samples are scarce. This series of technical approaches can effectively address the model training challenges caused by insufficient data.

[0095] Adding a fragmented domain-adaptive adapter on top of a general pre-trained model, freezing the underlying parameters, and fine-tuning the adaptation layer are fundamental operations for achieving domain migration. Pre-trained models such as BERT learn basic semantic knowledge of the language from massive amounts of general text. However, in the industrial field, specialized terms such as "bearing clearance" and "gear meshing frequency" require the model to be re-adapted. In this case, a lightweight domain adapter is added on top of BERT to specifically convert general semantics into industrial domain semantics. Freezing BERT's underlying parameters avoids changing its learned general knowledge, and only fine-tuning the adapter's parameters is performed. For example, when processing equipment fault diagnosis text, the adapter learns to associate "vibration anomalies" with equipment failure modes through training on a small number of industrial text samples, allowing the model to quickly adapt to the semantic expressions of the industrial field.

[0096] The Fragmented Domain Adaptive Adapter is a core adapter module for processing multimodal fragmented data. Its core design principle is to achieve efficient repair and adaptation of different types of fragmented data through precise sub-modal processing and conditional dynamic activation. The semantic completion sublayer for text fragments primarily addresses semantic gaps caused by transmission interruptions or incomplete truncation. This sublayer first analyzes the fragment's grammatical structure and contextual associations using a text modality feature extractor, such as a pre-trained model based on BERT. It then uses a context-aware attention mechanism to fill in semantic gaps. For example, for truncated text fragments such as "The 2024 Global Artificial Intelligence Conference will be held in location A, with an expected attendance of over 50,000," the sublayer can integrate domain corpora to complete key information such as "50,000 participants, focusing on the implementation of generative AI industries," ensuring the text's semantic integrity and logical coherence.

[0097] The visual restoration sublayer for image fragments focuses on resolving incomplete visual information caused by pixel loss and regional damage. Furthermore, a visual completion architecture based on a generative adversarial network (GAN) is employed to first perform feature matching on the edge contours and texture features of image fragments, and then reconstruct the missing areas through a generator module. For example, if a vehicle license plate fragment is obscured in surveillance footage, the sublayer can restore the complete license plate image based on the vehicle's overall outline and surrounding pixel textures, while simultaneously ensuring the authenticity of the restoration result through visual consistency checks.

[0098] It is worth noting that the activation conditions of the above two sub-layers can be dynamically determined based on the modal type of the input data. When the data is detected to contain text feature identifiers (such as UTF-8 encoding format, part-of-speech tagging information), the text semantic completion sub-layer is automatically activated; when the data is detected to contain image feature identifiers (such as RGB channel structure, pixel matrix dimension), the image visual repair sub-layer is automatically started. If mixed image and text fragments are encountered (such as a damaged poster with text annotations), both sub-layers will be activated simultaneously, and the overall repair will be completed through a cross-modal collaborative mechanism to ensure the accuracy and efficiency of the adaptation process.

[0099] A teacher-student model is constructed, and multiple teacher models with different initializations are used to generate integrated soft labels, further improving the stability of knowledge transfer. In industrial fault diagnosis, multiple BERT models with different initializations are used as teacher models. They each learn different semantic understanding emphases on a common text dataset. These teacher models generate different predicted probability distributions (soft labels) for the same equipment fault description text; some may focus on semantic logic, while others may focus on keyword matching. The student model learns on a fragmented dataset of a small number of industrial equipment fault texts and corresponding fault type labels (e.g., "motor overheating - cooling system failure"). Using the KL divergence loss function, the student model simultaneously fits the output distributions of multiple teacher models, much like learning different problem-solving approaches from multiple experienced "teachers," thereby absorbing more comprehensive knowledge. For example, when learning to diagnose "abnormal equipment noise," the student model can more accurately associate fault causes such as "bearing wear" or "loose gear" by integrating the soft labels from multiple teacher models.

[0100] Softening the output distribution of the teacher model using the temperature parameter is key to balancing knowledge generalization and specificity. The temperature parameter acts like a regulator, controlling the sharpness of soft labels. In the early stages of industrial fault diagnosis training, when the fragmented dataset is small (e.g., only 20 sets of fault data have been collected), increasing the temperature parameter smoothes the output distribution of the teacher model. For example, the teacher model originally identified a device fault as "motor overheating" with a 90% probability. After temperature softening, the probability distribution of related fault types, such as "motor overheating" and "circuit short circuit," becomes more uniform, preventing the student model from prematurely relying on local features in a limited sample and reducing the risk of overfitting. As more training data is collected, the temperature parameter is gradually reduced to make the distribution of soft labels more focused, enhancing the model's ability to recognize specific industrial fault patterns and achieving a dynamic balance between general knowledge and domain-specific knowledge.

[0101] Compared to knowledge distillation methods using a single teacher model, the use of multiple teachers with integrated soft labels and dynamic temperature adjustment improves the model's accuracy in identifying new fault types and enhances its generalization capabilities. Furthermore, the model achieves similar diagnostic results with less than one-fifth the sample size of traditional methods, significantly reducing the industry's reliance on large-scale labeled data and shortening the model development cycle. This provides efficient and reliable technical support for real-time monitoring and intelligent maintenance of industrial equipment.

[0102] In step S105, a multi-channel hash encoder is used to encode the second high-dimensional feature vector into a multi-channel binary hash code, invoking different adaptive encoding strategies for different modal data combinations. The multi-channel hash encoder is a feature encoding technology designed for multimodal data. It processes different modal data (e.g., text, image) through independent branches, dynamically selects the optimal encoding strategy, and maps high-dimensional features into compact binary hash codes.

[0103] As an optional embodiment, in step S105, for the text-image combination channel, a deep hashing network is used. The text semantic vector and the visual semantic vector are jointly input and mapped to shared feature pairs of the same dimension through a fully connected layer. The shared feature pairs are then mapped to hash codes using an activation function, with each bit representing a semantic feature dimension. The InfoNCE loss function is used to construct hash codes with close Hamming distances for semantically similar shared feature pairs, while hash codes with long Hamming distances for semantically unrelated shared feature pairs.

[0104] Deep hashing networks play a key role in the text-image combination pipeline. In e-commerce scenarios, the textual semantic vector extracted from a product description (e.g., "thin and light laptop, long battery life") using BERT+BiLSTM and the visual semantic vector extracted from the product image using ResNet-50+Transformer are jointly input into the deep hashing network. The network uses fully connected layers to map the features of both modalities to the same dimension, forming shared feature pairs, much like integrating the text description and image information on a single information card. An activation function then converts the shared feature pairs into binary hash codes, where each bit in the code corresponds to a semantic feature dimension. For example, a bit of 1 might indicate the "thin and light" feature, while a bit of 0 indicates the absence of that feature. The InfoNCE loss function plays a key role in this process, constraining shared feature pairs of semantically similar products (e.g., thin and light laptops from different brands) to have hash codes with close Hamming distances, placing them adjacent in the code space. Semantically unrelated products (e.g., laptops and sneakers) maintain a larger Hamming distance between their hash codes. As a result, the original high-dimensional image and text features are compressed into compact binary codes, which greatly reduces storage and computing costs while ensuring semantic consistency.

[0105] In step S105, for the temporal text combination channel, a graph attention hash network is used to construct the text semantic vector and the temporal pattern vector into a temporal text heterogeneous association graph. The association weights between the graph nodes in the temporal text heterogeneous association graph are calculated through the graph attention mechanism, so that the hash code captures the dependency relationship between the temporal pattern and the text semantics, and obtains a hash code containing structural information.

[0106] In the time-series-text combination channel, the graph attention hashing network is suitable for processing data with time series characteristics. For example, in industrial equipment monitoring, the time-series pattern vectors extracted from sensor data (such as time-series changes in temperature and speed) are combined with text semantic vectors extracted from text data such as equipment maintenance logs to construct a heterogeneous time-series-text association graph. The nodes in the graph represent different time-series data segments or text keywords, while the edges represent the associations between them. The graph attention mechanism calculates association weights between nodes to highlight the importance of key information. For example, as the device temperature continues to rise, the association weight between the "temperature sensor data" node and the "overheating warning" text node increases. The hash code generated based on these weights captures the dependency between the time-series pattern and the text semantics, for example, closely associating the time-series feature of "abnormal temperature rise" with the text description of "cooling system failure." The resulting hash code not only captures the data features but also preserves the structural information between the time-series text, supporting subsequent fault diagnosis and prediction.

[0107] Finally, in step S105, semantically consistent binary hash codes are generated through comparative learning constraints, wherein for the same semantic entity, the Hamming distance of the hash codes generated by different channels is less than a preset threshold.

[0108] Specifically, achieving cross-channel semantic consistency through contrastive learning constraints is a key feature of multi-channel hash coding. In e-commerce and industrial scenarios, for the same semantic entity (such as "hot-selling products" or "normal operation of equipment"), although the hash codes generated by different channels (images, text, and time-series text) come from different modal data, the contrastive learning mechanism forces their Hamming distance to be less than a preset threshold. This means that whether the codes are generated from product images and descriptions or from equipment operation data and maintenance records, as long as the semantics are the same, the codes will be close to each other in the feature space. This constraint ensures that multimodal data can maintain semantic consistency and relatability after compression encoding, facilitating subsequent rapid retrieval and analysis based on hash coding.

[0109] As a result, the multi-channel hash encoder improves the processing efficiency of multimodal data. In the e-commerce field, the storage cost of product image and text data is reduced. When users search for "thin and light laptops", the system can quickly match similar products through hash coding, improving retrieval speed. In industrial scenarios, the encoding and compression of equipment time series and text data enables fault warning systems to process massive amounts of monitoring data in real time, shortening fault identification response time to seconds. At the same time, comparative learning constraints ensure semantic consistency across different data sources, reducing misjudgments due to differences in data modalities, and effectively improving the effectiveness of data governance and intelligent decision-making.

[0110] For example, the adaptive encoding strategy of a multi-channel hash encoder dynamically adjusts based on the characteristics of different modal combinations. The following describes other implementation methods based on specific scenarios and technical logic:

[0111] For example, a temporal association hashing network is deployed in the audio-video combination channel. In short video content understanding scenarios, audio (such as background music and vocals) and video frames (visual images) need to be encoded into semantically consistent hash codes. A temporal association hashing network can be used in this scenario: a CNN extracts visual features from video frames (such as character movements and scene colors), while an LSTM extracts temporal features from the audio (such as melody and rhythm spectrum). Both are fed into a bidirectional attention mechanism to calculate the cross-modal association weights between the audio and video frames. For example, when a person speaks in a video, the association weight between the audio speech segment and the corresponding video frame increases. Subsequently, a temporal alignment loss function (such as the CTC loss) is used to constrain the hash codes, minimizing the Hamming distance between hash codes for audio and video segments with the same semantic meaning (such as a "person speaking" scene). Technically, this strategy enables fast short video retrieval. For example, when a user searches for "piano performance videos," hash codes can be used to match audio with piano sounds and videos with performance images, improving retrieval accuracy.

[0112] For example, a graph semantic hashing network is deployed in the text knowledge graph combination channel.

[0113] In intelligent question-answering systems, user query text (such as "Einstein's contribution to the theory of relativity") needs to be encoded in conjunction with entity relationships in the knowledge graph (such as "Einstein-proposed-relativity"). A graph semantic hash network can be constructed in this case: BERT is first used to extract the text semantic vector, and then the subgraph structure of the knowledge graph (such as entity nodes and relationship edges) is converted into a structured feature vector using a graph convolutional network (GCN). Then, through a cross-modal graph attention mechanism, keywords in the text (such as "relativity") are mapped to corresponding entity nodes in the knowledge graph. For example, the features of "relativity" and the "physical theory" category node in the knowledge graph are weightedly fused. When generating the hash code, a triplet graph loss function is used, requiring that the Hamming distance between the hash code of the text-knowledge graph pair and the positive sample (semantically related text-graph pairs) is smaller than that of the negative sample (irrelevant pairs). This strategy allows the question-answering system to more accurately associate text semantics with the knowledge graph structure. For example, when a user asks "the founder of quantum mechanics", the system can quickly locate entities such as "Bohr" and "Einstein" in the knowledge graph through hash coding, thereby improving the accuracy of the answer.

[0114] For example, a stereo semantic hashing network is deployed in the 3D point cloud image combination channel.

[0115] In autonomous driving scenarios, the 3D point cloud from the lidar (e.g., the spatial coordinates of obstacles) and the camera image (e.g., the visual appearance of objects) need to be encoded into a unified hash code. This can be achieved using a stereo semantic hashing network: PointNet extracts geometric features from the 3D point cloud (e.g., the 3D outline of a vehicle), while ResNet extracts visual features from the image (e.g., the vehicle's color and shape). Both are then fed into a cross-modal feature fusion module. This module includes a spatial alignment layer that projects the point cloud onto the image plane using the camera's intrinsic parameter matrix, aligning the 3D coordinates of the same object with the 2D pixel locations. For example, the point cloud outline of the vehicle ahead is forced to align with the vehicle's bounding box in the image, generating a corresponding feature vector. During hash encoding, a multi-view contrast loss function is used, requiring that the hash codes of the 3D-2D feature pairs of the same object remain consistent across different viewpoints (e.g., the Hamming distance of the hash codes is less than 0.1 when the vehicle is viewed from the front and side). Technically, this strategy can improve the obstacle recognition efficiency of autonomous driving systems. The compressed hash code reduces the amount of data required, while improving obstacle classification accuracy. It also supports real-time retrieval of similar obstacle forms in historical scenes.

[0116] For example, a dynamic gated hash network is deployed in the sensor data text instruction combination channel.

[0117] In smart home scenarios, sensor data (such as time-series data from temperature and humidity sensors) and user text commands (such as "adjust the living room temperature to 25°C") need to be encoded into hash codes for control signals. A dynamic gated hashing network can be used for this purpose: a Transformer extracts the temporal patterns of sensor data (such as temperature curves), and an LSTM extracts the semantic features of the text command (such as the keywords "temperature" and "25°C"). A learnable gating unit dynamically adjusts the weighting of these two. When sensor data fluctuates significantly (such as a sudden temperature rise), the gating unit increases the weight of the sensor features; otherwise, it prioritizes the semantics of the text command. For example, when a user issues the "adjust temperature" command, if the current temperature is close to 25°C, the hash code will rely more heavily on the sensor data, reducing unnecessary adjustments. A task-oriented loss function is used to optimize the semantic relevance of the hash code to the actual control action (such as the air conditioner on / off status). This strategy improves the command response accuracy of smart home systems and reduces misoperations. It also reduces the storage of sensor data and facilitates rapid retrieval of historical control scenarios in the cloud.

[0118] In step S106, the binary hash codes and corresponding semantic labels are dynamically added to the historical knowledge graph using an incremental graph neural network (IncGNN), resulting in an updated target knowledge graph. The Incremental Graph Neural Network (IncGNN) is a graph learning model that supports dynamic updates. It gradually integrates newly generated data (such as real-time annotated labels or newly added modal features) into the existing knowledge graph, avoiding global reconstruction. For example, in a continuously annotated cross-modal data stream, only the local nodes and edges affected by the newly added data are updated, maintaining the real-time and consistency of the knowledge graph, making it suitable for large-scale dynamic scenarios.

[0119] As an optional embodiment, in step S106, the binary hash code corresponding to the newly added semantic label is converted into a real-valued vector by a hyperbolic tangent function. Then, the incremental update mechanism of IncGNN is used to aggregate the converted real-valued vector with the semantically similar node embeddings in the historical entities of the historical knowledge graph to obtain the newly added entity. Then, the semantic distance between the newly embedded new entity and the historical entity is calculated by cosine similarity, and the degree of proximity between the new entity and the historical entity in the feature space is measured to obtain the semantic similarity between the new entity and the historical entity. If the semantic similarity exceeds the set threshold, a relationship edge between the new entity and the historical entity is established according to the semantic association relationship of the cross-modal features, and the relationship weight between the new entity and the historical entity is set according to the Hamming similarity of the hash code and the semantic similarity to complete the update of the historical knowledge graph and obtain the target knowledge graph.

[0120] Specifically, in the scenario where the knowledge graph is dynamically updated, step S106 realizes the efficient fusion of binary hash coding and semantic labels through the incremental graph neural network (IncGNN).

[0121] Taking the construction of a knowledge graph in the field of intelligent education as an example, the binary hash codes corresponding to the newly added semantic tags are first converted into real-valued vectors using the hyperbolic tangent function. In the intelligent education system, when a student's learning record of "Basic Principles of Quantum Mechanics" is detected, the binary hash code of this knowledge point (e.g., generated through a text-video multi-channel approach) is converted into a real-valued vector. This conversion maps the discrete binary code to a continuous space, facilitating subsequent similarity calculations with historical entities in the knowledge graph. For example, after conversion, the binary code of "Basic Principles of Quantum Mechanics" becomes a real-valued vector containing 512 dimensions, each dimension representing the feature strength of this knowledge point in different semantic directions.

[0122] Next, IncGNN's incremental update mechanism is used to aggregate similar entity embeddings. Historical entity nodes such as "wave-particle duality" and "Schrödinger's equation" already exist in the knowledge graph. IncGNN's incremental update module aggregates the real-valued vector of "Basic Principles of Quantum Mechanics" with the embeddings of these historical entities. For example, the system recognizes that "Basic Principles of Quantum Mechanics" and "Wave-Particle Duality" are similar in semantic space, and then weights and fuses their embedding vectors to generate a new entity representation. This aggregation is not a simple superposition, but rather, through the message passing mechanism of the graph neural network, allows the new entity to absorb the semantic knowledge of the historical entity while retaining its own characteristics, just like students establishing connections with existing cognition when learning new knowledge.

[0123] The semantic distance between newly added entities and historical entities is calculated using cosine similarity. Taking "Basic Principles of Quantum Mechanics" and "Wave-Particle Duality" as examples, the system calculates the cosine similarity of their real-valued vectors, obtaining a similarity value of 0.85, indicating a high degree of proximity in semantic space. If the similarity exceeds a set threshold (e.g., 0.7), a relationship edge is established between the two entities. In the knowledge graph, this is represented by a directed edge from the "Basic Principles of Quantum Mechanics" node to the "Wave-Particle Duality" node, symbolizing that the former contains knowledge elements of the latter.

[0124] Relationship weights are set based on hash-coded Hamming similarity and semantic similarity. In addition to cosine similarity, the system also calculates hash-coded Hamming similarity between "Basic Principles of Quantum Mechanics" and "Wave-Particle Duality." If 70% of the bits in their binary encodings are identical, the Hamming similarity is 0.7. Combining the cosine similarity (0.85) and the Hamming similarity (0.7), the relationship edge weight is set to 0.8 (e.g., taking a weighted average). This weight reflects the strength of the knowledge association; high-weighted edges are prioritized in subsequent knowledge reasoning and recommendations.

[0125] From a technical perspective, this incremental update mechanism improves the dynamic expansion capabilities of the knowledge graph. In intelligent education systems, the update delay of the knowledge graph is shortened from several hours in traditional methods to seconds, and the efficiency of integrating new knowledge points is improved. At the same time, through the dual constraints of semantic similarity and hash similarity, the accuracy of the relationship between entities in the knowledge graph is improved, enabling the recommendation system to provide students with more accurate learning paths. For example, based on the association between the "Basic Principles of Quantum Mechanics" and historical knowledge points, it recommends related content such as "Wave Function Collapse" and "Uncertainty Principle". The matching accuracy of students' learning paths is improved, effectively enhancing the adaptability of the knowledge graph in dynamic scenarios.

[0126] As an optional embodiment, after step S106, graph convolution operations can also be used to iterate the semantic information of the newly added entity based on the embedding and relationship weights of the neighboring nodes, so that the semantic information of the newly added entity is integrated with the neighboring entities to ensure the overall semantic association of the knowledge graph. The structural similarity of each relationship edge is calculated to determine whether it is redundant, so as to detect whether the newly added entity causes semantically similar relationship edges to appear in the target historical graph. If the structural similarity exceeds the set structural similarity threshold, the redundant relationship edges are merged and the relationship edge weights are updated to avoid structural confusion of the target knowledge graph due to repeated information and improve knowledge retrieval efficiency.

[0127] After the knowledge graph is dynamically updated, graph convolution operations and redundancy detection mechanisms further optimize the graph structure and semantic associations. For example, in the construction of a product knowledge graph in the e-commerce sector, graph convolution operations are used to fuse the semantic information of newly added entities with that of neighboring entities. Essentially, this process involves iteratively aggregating the features of neighboring nodes to update the entity's semantics. When an e-commerce platform introduces a new product, "Foldable Tablet," it is added to the knowledge graph as a new entity. Through graph convolution operations, this product incorporates embedded information from neighboring nodes (such as "Tablet," "Foldable Screen Technology," and "Electronic Products"). For example, the "Foldable Screen Tablet" node updates its semantic representation based on its own characteristics, referencing the functional attributes of "Tablet" and the feature descriptions of "Foldable Screen Technology." This process is similar to a new employee integrating into a team, acquiring industry knowledge and work experience through interaction with colleagues, and making its description more relevant to real-world business scenarios. Iterative operations continuously refine the semantics of the newly added entity, strengthening its semantic associations with neighboring entities and ensuring semantic consistency and coherence across all entities in the knowledge graph.

[0128] Detecting redundancy by calculating the structural similarity of relationship edges is intended to maintain the simplicity of the knowledge graph. In an e-commerce knowledge graph, "foldable tablet" may establish "equipped" relationships with "high-performance processor" and "long-lasting battery," respectively, while "high-end tablet" also has similar relationships with these two components. In this case, calculating the structural similarity of the two relationship edges "foldable tablet-equipped-high-performance processor" and "high-end tablet-equipped-high-performance processor" not only considers the connection between the nodes but also compares the relationship attributes (such as the equipped processor model and performance parameters). If the similarity exceeds a set threshold (such as 0.8), it is determined to be a redundant relationship, indicating that the two edges are highly semantically redundant.

[0129] When redundant relationship edges are detected, the system performs a merge operation and updates the relationship weight. For example, the two relationship edges pointing from "foldable tablet" and "high-end tablet" to "high-performance processor" are merged, and the relationship weight is updated based on the information from both. If the processor performance of the "foldable tablet" is stronger, the weight of the merged relationship will be increased accordingly, highlighting its advantage in the attribute of "equipped with a high-performance processor." This operation prevents the knowledge graph from becoming bloated and cluttered due to duplicate information, just like when organizing a warehouse, storing similar goods together and marking them with priority to facilitate quick and easy subsequent search and use.

[0130] From a technical perspective, graph convolution operations and redundancy detection mechanisms improve the quality of the knowledge graph. In e-commerce scenarios, the structural complexity of the knowledge graph is reduced, and retrieval efficiency is improved. When users search for "foldable tablet," the system can more quickly and accurately present the core attributes and related information of products. Furthermore, semantic fusion makes the semantic expression of the knowledge graph more precise. Based on the optimized graph, the recommendation system can more accurately recommend related products to users, effectively improving the user shopping experience and platform operational efficiency.

[0131] Step S107: Use a graph convolutional network (GCN) including a local structure enhancement contrast module to establish matching fine-grained labels for the semantically differentiated features of different entity combinations in the target knowledge graph. Combined with the first label association matrix, the second label association matrix, and the third label association matrix, the fine-grained labels in the target knowledge graph are hierarchically constructed to obtain a cross-modal label tree corresponding to the target knowledge graph.

[0132] In the embodiment of the present application, the cross-modal tag tree is a hierarchical semantic tag structure that contains multiple types of data. The entity relationships in the cross-modal knowledge graph are modeled through a graph neural network to form a progressive tag system from macro concepts to micro attributes. For example, in the image-text joint analysis, the root node is "animal", the child nodes are refined into categories such as "cat" and "dog", and the leaf nodes describe specific features (such as "fur color" and "body shape"), supporting fine-grained semantic retrieval and reasoning.

[0133] As an optional embodiment, in step S107, the target knowledge graph is forward-calculated using GCN, and the embedding vector of each entity is integrated with the local structural features of the entity itself; wherein the local structural features include the attribute features of the entity itself and the relationship edge features between entities. The embedding vectors are clustered using DBSCAN or K-means algorithms to obtain semantic cluster groups. For newly added entities, the probability distribution difference between the embedding vector of the newly added entity and the historical cluster center is calculated; if the probability distribution difference exceeds the preset drift condition, the newly added node is determined to be semantic drift, triggering the corresponding semantic cluster reclassification to obtain dynamically updated semantic cluster groups. The semantic features of the entities in the semantic cluster are aggregated through the attention mechanism, and high-frequency keywords are extracted as fine-grained labels. The compactness of the node embedding in the semantic cluster is calculated by confidence, and the partitioning method of the fine-grained labels is dynamically configured based on the compactness to obtain a fine-grained label set; wherein, the higher the compactness, the higher the confidence, and the higher the semantic consistency of the node embedding in the semantic cluster. The first label association matrix is ​​obtained by counting the number of co-occurrences of each fine-grained label with other labels in the historical data. The shortest path length of the fine-grained tags in the target knowledge graph is calculated, and the topological structure of the target knowledge graph is converted into a numerical correlation by graph Laplacian matrix decomposition, combining the relationship edge weights and the shortest path lengths of each fine-grained tag to obtain a second tag correlation matrix. The semantic similarity of the feature vectors corresponding to the fine-grained tags is calculated to obtain a third tag correlation matrix for identifying the semantic correlation. Based on the first tag correlation matrix, the second tag correlation matrix, and the third tag correlation matrix, the fine-grained tags in the target knowledge graph are hierarchically constructed to obtain the cross-modal tag tree.

[0134] Specifically, in constructing a multimodal knowledge graph, step S107 dynamically generates a fine-grained label tree using a graph convolutional network and a clustering algorithm. Taking the smart grid equipment monitoring scenario as an example, when processing the knowledge graph for power grid equipment, the embedding vector of each device node (such as a transformer or circuit breaker) not only includes its own attributes (model, rated power) but also incorporates neighboring node and edge features through GCN. For example, the embedding of the "transformer" node incorporates edge information such as "connected to - substation" and "temperature anomaly detected," making the node semantics more complete. This is similar to understanding device functionality through a comprehensive understanding of the device's specifications (its own attributes) and its installation environment (its neighboring relationships).

[0135] In this embodiment, a graph convolutional network (GCN) generates an embedding vector that incorporates local structure by hierarchically aggregating neighborhood information of nodes in a knowledge graph. Each entity's embedding not only includes its own attribute features (such as text description and type), but also integrates the features of its directly connected relationships (such as relationship type and connection strength).

[0136] From a computational perspective, each layer of GCN operates in two steps: collecting feature information from each node's immediate neighbors, combining its own features with the aggregated results, and generating a new embedding through nonlinear transformations. After stacking multiple layers, higher-order neighbor information is gradually incorporated, forming a representation that encompasses global topological relationships. For example, in a knowledge graph, if entities A and B have a "cooperation" relationship, GCN will transfer B's feature weights to A's embedding.

[0137] This embedding vector simultaneously encodes the entity's semantics and its structural role in the graph, resolving the problem of traditional methods ignoring higher-order neighborhoods. For example, in a medical knowledge graph, a disease entity can more accurately express its complex semantics by aggregating features of its related symptoms and drug neighbors.

[0138] In equipment fault detection scenarios, the DBSCAN algorithm groups nodes whose embedding vectors are densely connected into clusters by setting a neighborhood radius and a minimum number of samples. For example, if the temperature and vibration feature vectors of a group of equipment form irregular clusters in a high-dimensional space, this may correspond to a new fault mode (such as "insulation aging with partial discharge"). This ability to detect clusters of arbitrary shapes enables the system to identify abnormal patterns without prior labels, similar to how doctors discover new disease types by clustering patient symptoms. If subtypes within the broad category of "equipment fault" (such as "short circuit fault" and "overload fault") are known, K-means can be used to partition node embeddings into K clusters. For example, embedding vectors of historical fault data can be clustered into K=5 subtypes, with each cluster center representing a typical fault mode. When a new fault occurs, the fault type can be quickly determined by calculating its distance from the cluster center, enabling automated classification of known faults, similar to how a library organizes books by category for easy retrieval. Using the density clustering algorithm DBSCAN or the partitioning algorithm K-means, the embedding vectors generated by GCN are partitioned into semantic clusters. New entities detect semantic drift through differences in probability distribution, triggering dynamic re-clustering. From a computational perspective, DBSCAN clusters based on density reachability, requiring core points to meet a minimum number of samples (minPts) within a neighborhood radius (ε), making it suitable for non-convex distributions. K-means partitions by minimizing the intra-cluster squared error. Drift detection involves calculating the KL divergence or JS distance between the embedding of the new entity and the historical cluster center. If the difference exceeds a threshold (e.g., KL divergence > 0.3), semantic drift is detected, and local re-clustering is initiated. This allows for adaptation to the dynamic evolution of the knowledge graph. For example, a newly added "Blockchain Healthcare" entity may trigger re-clustering to generate a new cluster, "Digital Healthcare," due to its feature distribution deviating from the original "Medical Technology" cluster, thus avoiding semantic dilution of the old cluster.

[0139] Continuing with the above example, when a new "Smart Meter Communication Interruption" node is added, if the probability distribution difference between its embedding vector and the existing cluster center exceeds a threshold (e.g., KL divergence > 0.5), semantic drift is detected. The system then triggers cluster reclassification, assigning the node to a new cluster (e.g., "Communication Failure") or adjusting the boundaries of existing clusters. For example, the original "Sensor Failure" cluster may be subdivided into "Data Acquisition Failure" and "Communication Transmission Failure" due to the addition of the new node, similar to adjusting the classification system based on the characteristics of new species in biological taxonomy. An attention mechanism extracts high-frequency keywords within a semantic cluster as labels, and dynamically adjusts the label granularity based on the compactness of the cluster embeddings. In terms of computational principle, attention weights are assigned to key features of entities within the cluster (e.g., "CT Scan" and "AI Assisted" for the "Imaging Diagnosis" entity), and weighted labels are generated. When calculating compactness, the inverse variance of the node embeddings within the cluster is used as a compactness metric. High compactness (low variance) results in high confidence, and coarse-grained labels are directly output. Low compactness requires subdivision of labels (e.g., "CT Scan" is broken down into "Low-Dose CT" and "3D Reconstruction"). This improves the semantic accuracy of labels. For example, when the medical cluster is compact, the label "cancer treatment" is generated, while when it is loose, it is refined to "targeted therapy" or "immunotherapy" to avoid label ambiguity.

[0140] Continuing with the above example, within the "temperature anomaly" semantic cluster, the attention mechanism focuses on high-frequency keywords such as "overheating," "poor heat dissipation," and "cooling system," extracting these words as fine-grained labels. By calculating the compactness of the node embeddings within the cluster (e.g., mean cosine similarity), the labeling method is dynamically configured. If the compactness is high (e.g., 0.9), indicating strong node semantic consistency, a more refined labeling method (e.g., "radiator clogged," "fan failure") can be used. If the compactness is low (e.g., 0.6), a more generalized label (e.g., "temperature anomaly") is used, similar to adjusting the detail of teaching content based on the knowledge level of the students in the class.

[0141] In step S107, the associations between labels are quantified using three matrices. The first matrix represents statistical associations based on historical co-occurrence counts (e.g., the high co-occurrence of "diabetes" and "insulin"); the second matrix represents shortest path associations based on graph topology (e.g., the weighted correlation between the "doctor-hospital" path); and the third matrix represents semantic associations based on embedding similarity (e.g., the high cosine similarity between "surgery" and "minimally invasive"). The second matrix is ​​constructed using graph Laplacian matrix decomposition, combining the shortest path lengths between nodes with edge weights to convert them into numerical associations. Shorter paths and higher edge weights indicate greater association values. The third matrix is ​​constructed by calculating the cosine similarity of the entity embeddings corresponding to the labels to capture the underlying semantics. This integrates statistical, structural, and semantic information. For example, "heart stent" and "interventional therapy" are strongly associated due to their high co-occurrence rate (first matrix), close paths (second matrix), and similar embeddings (third matrix).

[0142] In the above steps, further optionally, the shortest path correlation matrix based on the graph topology obtained by combining the graph Laplacian matrix normalization process is obtained. It is expressed as follows: Asp [ i , j ] = e − γ ⋅ D [ i , j ] ⋅ w avg （ e ij ） .in, is the shortest path matrix, which is calculated by Dijkstra or Floyd-Warshall algorithm to calculate the shortest path between nodes (labels) in the graph, and the edge weight Indicates the strength of the relationship (e.g., the weight of the "doctor-hospital" edge reflects the closeness of the connection). With label The number of co-occurrences in historical data. For label With label Embedding vector of co-occurrence counts in historical data. Scaling factor that controls how quickly the path length affects the correlation. From the label To label The average weight of all edges on the metric function. Map the path length to the interval between 0 and 1. The shorter the path, the greater the association value. Multiply by the edge weight Strengthen the contribution of high-weight paths.

[0143] The three association matrices are integrated and weighted fusion (e.g., matrix weighted averaging) is used to generate a cross-modal label tree, achieving hierarchical organization of labels (e.g., "Medical Technology → Diagnostic Imaging → MRI"). Each matrix is ​​normalized and assigned a weight (e.g., 0.4 for the co-occurrence matrix, 0.3 for the path matrix, and 0.3 for the semantic matrix), and then fused using matrix addition. Hierarchical clustering is performed based on the fusion results, generating a tree structure: the root node is the coarse-grained category, and the leaf nodes are the detailed labels. The label tree takes into account multi-dimensional associations to support precise retrieval and reasoning. For example, in a medical knowledge base, a user querying "cancer treatment" can locate "chemotherapy side effect management" along the subtree, shortening the search path.

[0144] The first matrix is ​​generated by counting co-occurrences (e.g., "short circuit" and "trip" frequently co-occur). The second matrix is ​​calculated by calculating the shortest path length between labels (e.g., the path length from "transformer failure" to "power outage"). The third matrix is ​​calculated by calculating the semantic similarity of labels (e.g., "overload" and "overcurrent" have similar semantics). Combining these three matrices, the fine-grained labels are hierarchically constructed into a cross-modal label tree. For example, the root node is "equipment failure," which is further divided into "electrical failure" and "mechanical failure." "Electrical failure" is further subdivided into "short circuit" and "overload." This creates a hierarchical system similar to biological taxonomy, facilitating fault tracing and diagnosis.

[0145] In smart grid scenarios, step S107 improves fault diagnosis accuracy, accelerates the discovery of new faults, and enhances fault knowledge retrieval efficiency through the hierarchical structure of the label tree. Furthermore, the semantic drift detection mechanism enables the system to adapt to knowledge changes brought about by device upgrades. For example, after smart meters are connected, the communication fault classification is automatically expanded, ensuring the timeliness and accuracy of the knowledge graph and providing strong support for the intelligent operation and maintenance of power systems.

[0146] In the embodiment of the present application, the automatic construction of a cross-modal tag tree is achieved, which solves the technical problem of poor semantic relevance caused by the fragmentation of multimodal data, improves the precision of semantic description, and improves the efficiency of tag generation.

[0147] After introducing the method of the exemplary embodiment of the present application, next, refer to Figure 2 A fragmented data cross-modal label generation system based on deep transfer learning in an exemplary embodiment of the present application is described. The device includes: an acquisition module for acquiring original multimodal data to be processed; a multi-branch deep feature extraction network is used to extract first high-dimensional feature vectors under different modalities from the original multimodal data using different deep learning modes according to the characteristics of different modal data; a unification module is used to map the first high-dimensional feature vectors under different modalities to the same semantic space through a cross-modal contrast loss function, thereby realizing multimodal alignment and fusion processing of the first high-dimensional feature vectors, and obtaining a second high-dimensional feature vector with a unified semantic metric; a labeling module is used to label the semantic labels corresponding to the second high-dimensional feature vectors based on the fragmented data components in the original multimodal data using a small sample transfer learning algorithm. ; An encoding module, which is used to use a multi-channel hash encoder to call different adaptive encoding strategies for different modal data combinations, and encode the second high-dimensional feature vector into a multi-channel binary hash code; an update module, which is used to combine the incremental graph neural network IncGNN to dynamically expand the binary hash code and the corresponding semantic label into the historical knowledge graph to obtain an updated target knowledge graph; a label module, which is used to use a graph convolutional network GCN containing a local structure enhancement contrast module to establish matching fine-grained labels for the semantic differentiation features of different entity combinations in the target knowledge graph, and to construct a hierarchical structure of the fine-grained labels in the target knowledge graph in combination with the first label association matrix, the second label association matrix, and the third label association matrix to obtain a cross-modal label tree corresponding to the target knowledge graph. The above system can implement each step described in the above method implementation method, and the specific implementation method of each step will not be repeated here.

[0148] After introducing the method and system of the exemplary embodiment of the present application, a terminal device of the exemplary embodiment of the present application is described below. The terminal device can implement each step described in the above method implementation, and the specific implementation of each step will not be repeated here. Figure 3 For a description of the computer-readable storage medium of the exemplary embodiment of the present application, please refer to Figure 3 The computer-readable storage medium shown is a CD 30, which stores a computer program (i.e., a program product). When executed by a processor, the computer program implements each step described in the above method implementation. The specific implementation of each step is not repeated here.

[0149] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which will not be described in detail here. The above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-mentioned embodiments, ordinary technicians in this field should understand that any technician familiar with this technical field can still modify the technical solutions described in the above-mentioned embodiments within the technical scope disclosed in this application, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not deviate from the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating cross-modal labels for fragmented data based on deep transfer learning, characterized in that: The method comprises: Obtaining raw multimodal data to be processed; using a multi-branch deep feature extraction network, and applying different deep learning models based on the characteristics of different modal data, extracting first high-dimensional feature vectors under different modalities from the raw multimodal data; Through the cross-modal contrast loss function, the first high-dimensional feature vectors under different modalities are mapped to the same semantic space, achieving multimodal alignment and fusion processing of the first high-dimensional feature vectors, and obtaining the second high-dimensional feature vector with unified semantic measurement; Using a small sample transfer learning algorithm, based on the fragmented data components in the original multimodal data, annotate the semantic labels corresponding to the second high-dimensional feature vector; Using a multi-channel hash encoder, calling different adaptive encoding strategies for different modal data combinations, and encoding the second high-dimensional feature vector into a multi-channel binary hash code; Combined with the incremental graph neural network IncGNN, the binary hash code and the corresponding semantic label are dynamically expanded into the historical knowledge graph to obtain an updated target knowledge graph, including: converting the binary hash code corresponding to the newly added semantic label into a real-valued vector through a hyperbolic tangent function; utilizing the incremental update mechanism of IncGNN to aggregate the converted real-valued vector with the semantically similar node embeddings in the historical entities of the historical knowledge graph to obtain the newly added entity; calculating the semantic distance between the newly embedded new entity and the historical entity through cosine similarity, measuring the proximity between the new entity and the historical entity in the feature space, to obtain the semantic similarity between the new entity and the historical entity; if the semantic similarity exceeds a set threshold, establishing a relationship edge between the new entity and the historical entity according to the semantic association relationship of the cross-modal features, and setting the relationship weight between the new entity and the historical entity according to the Hamming similarity of the hash code and the semantic similarity, so as to complete the update of the historical knowledge graph and obtain the target knowledge graph; A graph convolutional network (GCN) including a local structure enhancement contrast module is used to establish matching fine-grained labels for the semantic differentiation features of different entity combinations in the target knowledge graph. Combined with the first label association matrix, the second label association matrix, and the third label association matrix, the fine-grained labels in the target knowledge graph are hierarchically constructed to obtain a cross-modal label tree corresponding to the target knowledge graph.

2. The method for generating cross-modal labels for fragmented data based on deep transfer learning according to claim 1 is characterized in that: The multi-branch deep feature extraction network is used to extract first high-dimensional feature vectors under different modalities from the original multimodal data using different deep learning models according to different modal data characteristics, including: Identifying the modality to which each raw data in the raw multimodal data belongs, and preprocessing the raw data of different modalities; The preprocessed original text sequence is input into the BERT and BiLSTM joint model. The BERT pre-trained model is used to extract the contextual semantic features of the original text sequence described in the article. The BiLSTM is used to capture the implicit bidirectional dependencies of the fragmented data in the original text sequence through sequence modeling to obtain the text semantic vector in the text modality. The preprocessed original image matrix is ​​passed through the ResNet-50 network to extract the image visual features, and then the Transformer model is used to enhance the extraction of global semantic features to obtain the visual semantic vector under the image modality; The preprocessed raw time series data is used to extract dynamic pattern features through the LSTM autoencoder, and feature reconstruction is achieved through the LSTM autoencoder to retain the time series dependency and obtain the time series pattern vector under the image modality.

3. The method for generating cross-modal labels for fragmented data based on deep transfer learning according to claim 1 is characterized in that: The method maps the first high-dimensional feature vectors under different modalities into the same semantic space through the cross-modal contrast loss function, realizes multimodal alignment and fusion processing of the first high-dimensional feature vectors, and obtains the second high-dimensional feature vector with unified semantic measurement, including: The first high-dimensional feature vectors are converted into a unified feature dimension through independent encoders, and then the re-encoded first high-dimensional feature vectors are projected into the shared semantic space; Select feature vectors from different modalities as anchor samples from the shared semantic space; Selecting a feature vector with the same semantics as the anchor sample from the shared semantic space as a positive sample, and a feature vector with different semantics from the anchor sample as a negative sample, to obtain a cross-modal sample pair; A pre-trained CLIP model is used to generate continuous similarity weights for cross-modal sample pairs, and a cross-modal contrast loss function is constructed based on the continuous similarity weights; the continuous similarity weights are used to reflect the semantic association strength between different first high-dimensional feature vectors in the cross-modal sample pairs; The cross-modal contrast loss function is used to calculate the similarity distribution between cross-modal sample pairs, a dynamic boundary threshold is set with a ternary fuzzy boundary constraint, and a similarity distribution heat map constructed with reference to the similarity distribution and the dynamic boundary threshold is used to adjust the relative positions between cross-modal feature vectors in the shared semantic space to bring semantically similar samples closer together and push semantically unrelated samples further away. Through the attention cross-mechanism, the correlation matrix of cross-modal feature vectors in the shared semantic space is calculated; Combined with the similarity distribution heat map, cross-modal semantic related regions in the association matrix are identified to achieve cross-modal semantic alignment; A learnable gating coefficient is used to perform weighted fusion on the cross-modal correlation feature vector in the cross-modal semantic related area and the original first high-dimensional feature vector, and the weighted fusion result is mixed and extracted through a multi-granularity layer to obtain the second high-dimensional feature vector.

4. The method for generating cross-modal labels for fragmented data based on deep transfer learning according to claim 1 is characterized in that: The method of using a small sample transfer learning algorithm to label the semantic label corresponding to the second high-dimensional feature vector based on the fragmented data components in the original multimodal data includes: Identifying fragmented data components in the original multimodal data and calculating a domain relevance index for each fragmented data component; the domain relevance index represents a semantic overlap between each fragmented data component and a target domain; Through parameter fine-tuning and knowledge distillation, the general pre-trained model is adaptively migrated to the target domain where the fragmented data components are located, and a small sample annotation model for the target domain is constructed. The parameter fine-tuning includes adding a fragmented domain adaptive adapter on top of the general pre-trained model; the parameter sparsity of the fragmented domain adaptive adapter is dynamically adjusted based on the domain relevance index, where lower relevance indicates higher sparsity. The second high-dimensional feature vector is semantically annotated using the small sample annotation model to obtain a semantic label corresponding to the second high-dimensional feature vector.

5. The method for generating cross-modal labels for fragmented data based on deep transfer learning according to claim 4 is characterized in that: The general pre-training model is adaptively migrated to the target domain where the fragmented data components are located through parameter fine-tuning and knowledge distillation, and a small sample annotation model in the target domain is constructed, including: Add a fragmented domain adaptive adapter on top of the general pre-trained model, freeze the underlying parameters, and fine-tune the adaptation layer. The fragmented domain adaptive adapter includes a semantic completion sublayer for text fragments and a visual restoration sublayer for image fragments. The activation conditions of each sublayer are set based on the modality type of the corresponding fragmented component. Constructing a teacher-student model; using multiple differently initialized teacher models to generate integrated soft labels on a common dataset; when the student model learns on the fragmented dataset in the target domain where the fragmented data components are located, synchronously fitting the output distributions of the multiple teacher models using a KL divergence loss function; The output distribution of the teacher model is softened by the temperature parameter. When the student model is trained on a fragmented dataset, the temperature parameter is dynamically adjusted to balance the generalization and specificity of knowledge transfer. When the sample size of the fragmented dataset is less than the preset threshold, the temperature parameter is increased to enhance the smoothness of the soft label and reduce the risk of overfitting.

6. The method for generating cross-modal labels for fragmented data based on deep transfer learning according to claim 1, characterized in that: The multi-channel hash encoder is used to call different adaptive encoding strategies for different modal data combinations to encode the second high-dimensional feature vector into a multi-channel binary hash code, including: For the text-image combination channel, a deep hashing network is used. The text semantic vector and the visual semantic vector are jointly input and mapped to shared feature pairs of the same dimension through a fully connected layer. The shared feature pairs are then mapped into hash codes through an activation function, with each bit representing a semantic feature dimension. The InfoNCE loss function is used to construct hash codes with close Hamming distances for semantically similar shared feature pairs, and hash codes with long Hamming distances for semantically unrelated shared feature pairs. For the temporal text combination channel, a graph attention hash network is used to construct a temporal text heterogeneous association graph by combining text semantic vectors and temporal pattern vectors. The graph attention mechanism is used to calculate the association weights between graph nodes in the temporal text heterogeneous association graph, so that the hash code captures the dependency between temporal patterns and text semantics, obtaining a hash code containing structural information. Semantically consistent binary hash codes are generated through contrastive learning constraints; for the same semantic entity, the Hamming distance of the hash codes generated by different channels is less than a preset threshold.

7. The method for generating cross-modal labels for fragmented data based on deep transfer learning according to claim 1, characterized in that: After dynamically expanding the binary hash code and the corresponding semantic label into the historical knowledge graph by combining the incremental graph neural network IncGNN to obtain the updated target knowledge graph, the method further includes: Using graph convolution operations, the semantic information of newly added entities is iteratively added based on the embedding and relationship weights of neighboring nodes, so that the semantic information of newly added entities is integrated with that of neighboring entities, ensuring the semantic relevance of the entire knowledge graph; By calculating the structural similarity of each relationship edge, we can determine whether it is redundant and detect whether the newly added entity leads to semantically similar relationship edges in the target history graph. If the structural similarity exceeds the set structural similarity threshold, the redundant relationship edges are merged and the relationship edge weights are updated to avoid structural confusion in the target knowledge graph due to repeated information and improve knowledge retrieval efficiency.

8. The method for generating cross-modal labels for fragmented data based on deep transfer learning according to claim 1, characterized in that: The graph convolutional network (GCN) including the local structure enhancement contrast module is used to establish matching fine-grained labels for the semantically differentiated features of different entity combinations in the target knowledge graph. The fine-grained labels in the target knowledge graph are hierarchically constructed by combining the first label association matrix, the second label association matrix, and the third label association matrix to obtain a cross-modal label tree corresponding to the target knowledge graph, including: The target knowledge graph is forward-calculated by GCN, and the embedding vector of each entity is integrated with the local structural features of the entity itself; wherein the local structural features include the attribute features of the entity itself and the relationship edge features between entities; Use DBSCAN or K-means algorithm to cluster the embedded vectors to obtain semantic cluster groups; For newly added entities, the probability distribution difference between the embedding vector of the newly added entity and the historical cluster center is calculated; if the probability distribution difference exceeds the preset drift condition, the newly added node is judged to be semantic drift, triggering the corresponding semantic cluster reclassification to obtain dynamically updated semantic cluster grouping; The attention mechanism aggregates the semantic features of entities within a semantic cluster, extracts high-frequency keywords as fine-grained labels, calculates the compactness of node embeddings within the semantic cluster using confidence, and dynamically configures the partitioning method of fine-grained labels based on the compactness to obtain a fine-grained label set. The higher the compactness, the higher the confidence, and the higher the semantic consistency of node embeddings within the semantic cluster. By counting the number of co-occurrences of each fine-grained tag with other tags in the historical data, a first tag association matrix is ​​obtained; Calculate the shortest path length of the fine-grained label in the target knowledge graph, and convert the topological structure of the target knowledge graph into a numerical correlation by combining the relationship edge weights and the shortest path lengths of each fine-grained label through graph Laplacian matrix decomposition to obtain a second label correlation matrix; Calculate the semantic similarity of the feature vectors corresponding to the fine-grained labels to obtain a third label association matrix for identifying the semantic association; Based on the first label association matrix, the second label association matrix, and the third label association matrix, the fine-grained labels in the target knowledge graph are hierarchically constructed to obtain the cross-modal label tree.

9. A fragmented data cross-modal label generation system based on deep transfer learning, characterized by: The system comprises: An acquisition module is used to acquire the original multimodal data to be processed; a multi-branch deep feature extraction network is used to extract the first high-dimensional feature vectors under different modalities from the original multimodal data using different deep learning models according to the characteristics of different modal data; A unification module is used to map the first high-dimensional feature vectors under different modalities into the same semantic space through a cross-modal contrast loss function, realize multimodal alignment and fusion processing of the first high-dimensional feature vectors, and obtain a second high-dimensional feature vector with a unified semantic metric; a labeling module, configured to label semantic labels corresponding to the second high-dimensional feature vector based on the fragmented data components in the original multimodal data using a small sample transfer learning algorithm; an encoding module, configured to employ a multi-channel hash encoder, invoke different adaptive encoding strategies for different modal data combinations, and encode the second high-dimensional feature vector into a multi-channel binary hash code; An update module is used to dynamically expand the binary hash code and the corresponding semantic label into the historical knowledge graph in combination with the incremental graph neural network IncGNN to obtain an updated target knowledge graph; The update module is specifically used to convert the binary hash code corresponding to the newly added semantic label into a real-valued vector through the hyperbolic tangent function; the incremental update mechanism of IncGNN is used to aggregate the converted real-valued vector with the semantically similar node embeddings in the historical entities of the historical knowledge graph to obtain the newly added entity; the semantic distance between the newly embedded new entity and the historical entity is calculated by cosine similarity, and the proximity between the new entity and the historical entity in the feature space is measured to obtain the semantic similarity between the new entity and the historical entity; if the semantic similarity exceeds the set threshold, a relationship edge is established between the new entity and the historical entity according to the semantic association relationship of the cross-modal features, and the relationship weight between the new entity and the historical entity is set according to the Hamming similarity of the hash code and the semantic similarity to complete the update of the historical knowledge graph and obtain the target knowledge graph; The labeling module is used to use a graph convolutional network (GCN) including a local structure enhancement contrast module to establish matching fine-grained labels for the semantically differentiated features of different entity combinations in the target knowledge graph, and to hierarchically construct the fine-grained labels in the target knowledge graph by combining the first label association matrix, the second label association matrix, and the third label association matrix to obtain a cross-modal label tree corresponding to the target knowledge graph.

Images