Fragment data cross-modal label generation system and method based on deep transfer learning

Abstract

The application provides a kind of based on deep migration learning's fragment data cross-modal label generation system and method.The method includes: extracting first high-dimensional feature vector under different modalities;By cross-modal contrast loss function, the first high-dimensional feature vector of different modalities is mapped into the same semantic space to realize multi-modal alignment and fusion to obtain second high-dimensional feature vector;Using small sample transfer learning algorithm, based on fragment data component, the semantic label corresponding to the second high-dimensional feature vector is labeled;Using multi-channel hash encoder, for different modal data combination, call adaptive coding strategy, encode the second high-dimensional feature vector into multi-channel binary hash code;Combined with incremental graph neural network, dynamically expand binary hash code and corresponding semantic label into historical knowledge graph;For the semantic differentiation features of different entity combinations in the target knowledge graph, establish matching fine-grained label, and obtain cross-modal label tree by combining three matrix hierarchical construction.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence, and more particularly, embodiments of the present application relate to a deep transfer learning based fragmented data cross-modal label generation system and method. BACKGROUND

[0002] At present, the rapid development of information technology has promoted the comprehensive arrival of the digital era, and the amount of data generated by various industries has increased exponentially, and the widespread existence of fragmented data has become a key feature in the field of data. These fragmented data cover text, images, audio, video and other modalities, and different modal data express semantic information in unique forms. For example, text data transmits information through vocabulary, grammar and semantic structure, images express content by relying on color, shape and spatial layout, and audio carries meaning through waveform, frequency and timbre. The differences between modalities not only reflect in the physical storage form of data, but also reflect in the semantic expression logic, which increases the complexity of data processing and analysis, and brings challenges to the fusion of cross-modal data.

[0003] In related technologies, the semantic association of fragmented data is weak, which is due to the characteristics of incomplete information and loose structure of the data itself. In actual application scenarios, data often becomes scattered and fragmented due to limitations of collection conditions, loss in transmission process, etc. For example, user comments, product pictures, sales records and other data in e-commerce platforms are stored independently, and there is a lack of natural connection clues between the textual description in the comments and the visual features of the product pictures, the time series data of the sales records, making it difficult to directly establish a semantic mapping relationship. Traditional data processing methods, such as rule-based association analysis or simple feature splicing, cannot effectively mine this potential semantic association, resulting in that the subsequent label generation is difficult to accurately reflect the core content of the data, affecting the depth and breadth of data analysis. In addition, the sharp expansion of data size makes it a common practice to process fragmented data of tens of millions or even hundreds of millions of sizes in a short period of time, so each technical field needs to complete analysis and label annotation in a short period of time to support business decision-making and service optimization. However, the traditional label generation method is difficult to meet the actual needs in terms of efficiency and accuracy, and the mining of semantic association, so it is urgent to design a more efficient technical solution to solve at least one of the above technical problems. SUMMARY

[0004] In this context, embodiments of the present application aim to provide a deep transfer learning based fragmented data cross-modal label generation system and method, which can realize the automatic construction of cross-modal label trees, solve the technical problem of poor semantic association caused by the fragmentation of multi-modal data, improve the semantic description precision, and improve the label generation efficiency.

[0005] In a first aspect of the embodiments of the present application, a fragment data cross-modal label generation method based on deep transfer learning is provided, comprising:

[0006] Obtaining original multi-modal data to be processed; using a multi-branch deep feature extraction network, different deep learning modes are used to extract first high-dimensional feature vectors under different modalities from the original multi-modal data according to different modal data characteristics;

[0007] Mapping the first high-dimensional feature vectors under different modalities into the same semantic space through a cross-modal contrast loss function, realizing multi-modal alignment and fusion processing of the first high-dimensional feature vectors, and obtaining second high-dimensional feature vectors with unified semantic metrics;

[0008] Using a small sample transfer learning algorithm, labeling the semantic labels corresponding to the second high-dimensional feature vectors based on the fragmented data components in the original multi-modal data;

[0009] Using a multi-channel hash encoder, different adaptive encoding strategies are called for different modal data combinations to encode the second high-dimensional feature vectors into multi-channel binary hash codes;

[0010] Combining an incremental graph neural network IncGNN, the binary hash codes and the corresponding semantic labels are dynamically expanded into a historical knowledge graph to obtain an updated target knowledge graph, including: converting the binary hash codes corresponding to the new semantic labels into real value vectors through a hyperbolic tangent function; using the incremental update mechanism of IncGNN, aggregating the converted real value vectors with node embeddings with similar semantics in the historical entities of the historical knowledge graph to obtain new entities; calculating the semantic distance between the new entities and the historical entities through cosine similarity to measure the closeness of the new entities and the historical entities in the feature space to obtain the semantic similarity between the new entities and the historical entities; if the semantic similarity exceeds a set threshold, a relationship edge is established between the new entities and the historical entities according to the semantic correlation of cross-modal features, and the relationship weight between the new entities and the historical entities is set according to the Hamming similarity of the hash codes and the semantic similarity, to complete the update of the historical knowledge graph and obtain the target knowledge graph;

[0011] Using a graph convolution network GCN containing a local structure enhancement contrast module to establish matching fine-grained labels for the semantic differentiated features of different entity combinations in the target knowledge graph, combining a first label association matrix, a second label association matrix, and a third label association matrix to hierarchically construct the fine-grained labels in the target knowledge graph to obtain a cross-modal label tree corresponding to the target knowledge graph.

[0012] In a second aspect of the embodiments of the present application, a fragment data cross-modal label generation system based on deep transfer learning is provided, comprising:

[0013] An acquisition module is configured to acquire original multi-modal data to be processed; a multi-branch deep feature extraction network is used to extract first high-dimensional feature vectors under different modalities from the original multi-modal data by using different deep learning modes according to different modal data characteristics;

[0014] A unification module is configured to map the first high-dimensional feature vectors under different modalities into the same semantic space by using a cross-modal contrast loss function, to realize multi-modal alignment and fusion processing of the first high-dimensional feature vectors, and to obtain second high-dimensional feature vectors with unified semantic metrics;

[0015] A labeling module is configured to label semantic labels corresponding to the second high-dimensional feature vectors based on fragment data components in the original multi-modal data by using a small sample transfer learning algorithm;

[0016] An encoding module is configured to encode the second high-dimensional feature vectors into multi-channel binary hash codes by using a multi-channel hash encoder and calling different adaptive encoding strategies for different modal data combinations;

[0017] An updating module is configured to dynamically expand the binary hash codes and corresponding semantic labels into a historical knowledge graph by using an incremental graph neural network (IncGNN) to obtain an updated target knowledge graph; convert the binary hash codes corresponding to the new semantic labels into real value vectors by using a hyperbolic tangent function; aggregate the converted real value vectors with node embeddings with similar semantics in the historical entities of the historical knowledge graph by using an incremental update mechanism of the IncGNN to obtain new entities; calculate the semantic distance between the new entities and the historical entities in the feature space by using cosine similarity to measure the closeness of the new entities and the historical entities; if the semantic similarity exceeds a set threshold, establish a relationship edge between the new entities and the historical entities according to the semantic correlation of cross-modal features, and set the relationship weight between the new entities and the historical entities according to the Hamming similarity of the hash codes and the semantic similarity, to complete the updating of the historical knowledge graph and obtain the target knowledge graph;

[0018] A label module is configured to use a graph convolutional network (GCN) containing a local structure enhanced contrast module to establish matching fine-grained labels for the semantic differentiated features of different entity combinations in the target knowledge graph, and to hierarchically construct the fine-grained labels in the target knowledge graph by combining a first label association matrix, a second label association matrix, and a third label association matrix to obtain a cross-modal label tree corresponding to the target knowledge graph.

[0019] In a third aspect of embodiments of the present application, a terminal device is provided, comprising at least one processor, a memory and an input-output unit; wherein the memory is configured to store a computer program, and the processor is configured to invoke the computer program stored in the memory to execute the deep transfer learning based fragmented data cross-modal label generation method of any one of the first aspect.

[0020] In a fourth aspect of embodiments of the present application, a computer readable storage medium is provided, comprising instructions which, when executed on a computer, cause the computer to execute the deep transfer learning based fragmented data cross-modal label generation method of any one of the first aspect.

[0021] In a fifth aspect of embodiments of the present application, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the deep transfer learning based fragmented data cross-modal label generation method of any one of the first aspect.

[0022] According to the deep transfer learning based fragmented data cross-modal label generation system and method of embodiments of the present application, systematic breakthroughs are achieved in multi-modal data processing, small sample learning and other aspects, providing an effective path for data intelligent governance. In the feature extraction and fusion stage, for different modal data such as images, texts and audios, a multi-branch deep feature extraction network is used to adapt to a special deep learning model, and unique data information is fully mined. Using a cross-modal contrast loss function, each modal feature vector is mapped to a unified semantic space, realizing deep alignment and fusion of multi-source data, and improving data consistency and explainability. In the semantic label generation link, a small sample transfer learning algorithm is introduced, and with the prior knowledge of the pre-trained model, efficient generalization can be achieved under a small number of labeled samples, reducing the dependence on manual labeling and shortening the model training cycle. In the face of real-time processing needs of massive data, a multi-channel hash encoder and an incremental graph neural network work together, the former dynamically encodes according to the modal combination, converts high-dimensional features into lightweight binary hash codes, and optimizes storage and computing efficiency; the latter realizes dynamic updating of the knowledge graph, supports real-time processing and knowledge sedimentation of ten million data. In the construction of the knowledge graph, through the graph convolutional network with local structure enhancement contrast module, the entity relationship is deeply modeled, the semantic difference features are identified, the label confusion is avoided, the hierarchical cross-modal label tree is constructed, and the semantic description precision is improved. This technical scheme systematically solves the problems of data fragmentation and multi-source heterogeneity, realizes automatic, efficient and low-cost data processing, and achieves high standards in feature extraction, semantic alignment, knowledge sedimentation and label generation, providing reliable technical support for multi-modal data governance.

[0023] In terms of technical effectiveness, firstly, this application addresses the fragmentation problem of dispersed and heterogeneous multimodal data sources by employing a multi-branch deep feature extraction network to achieve modality-adaptive feature decoupling and recombination. For different modal characteristics such as images, text, and audio, the optimal deep learning model is dynamically selected (e.g., CNN for visual texture processing, Transformer for text semantic parsing), breaking through the traditional single-model's forced alignment limitation on multimodal data. Combined with a cross-modal contrastive loss function, fragmented features are mapped to a unified semantic space, effectively eliminating the semantic gap between modalities and enabling fine-grained alignment of data from different sources at the feature level. Secondly, through a hierarchical feature interaction mechanism and a dynamic weight allocation strategy, deep collaboration of multimodal features is achieved. In the feature fusion stage, a multi-channel hash encoder is used to adaptively select encoding paths for different modal combinations (e.g., image-text, audio-video): a cross-attention mechanism is used to strengthen the association for highly relevant modalities (e.g., product images and text), while adversarial training is used to improve robustness for low-relevance modalities (e.g., environmental audio and product parameters). Finally, this application integrates Incremental Graph Neural Network (IncGNN) with multi-channel hash encoding technology to construct a dynamically evolving knowledge ecosystem. Faced with massive fragmented data streams, feature dimensionality reduction is achieved through hash encoding, while the local incremental update mechanism of IncGNN allows for dynamic expansion of the knowledge graph with only local node computation. Combined with a local structure enhancement and contrast module, the accuracy of entity relationship recognition in the knowledge graph is improved, and the fine-grained expansion capability of the label tree is enhanced. In summary, this application enables automated construction of cross-modal label trees, solves the technical problem of poor semantic relevance caused by multimodal data fragmentation, improves the granularity of semantic description, and increases label generation efficiency. Attached Figure Description

[0024] Figure 1 A flowchart illustrating a method for generating cross-modal labels for fragmented data based on deep transfer learning, provided in an embodiment of this application;

[0025] Figure 2 A schematic diagram of the structure of a fragmented data cross-modal label generation system based on deep transfer learning provided in an embodiment of this application;

[0026] Figure 3 A schematic diagram of the structure of a medium according to an embodiment of this application is shown. Detailed Implementation

[0027] The following is for reference. Figure 1 , Figure 1 This is a flowchart illustrating a method for generating cross-modal labels for fragmented data based on deep transfer learning, provided in one embodiment of this application. It should be noted that the implementation methods of this application can be applied to any applicable annotation scenario.

[0028] Figure 1The embodiment of the application shown provides a flow of a fragment data cross-modal label generation method based on deep transfer learning, which comprises the following steps:

[0029] In step S101, original multi-modal data to be processed is acquired.

[0030] In step S102, a multi-branch deep feature extraction network is used to extract first high-dimensional feature vectors under different modes from the original multi-modal data by using different deep learning modes according to different modal data characteristics.

[0031] In step S103, the first high-dimensional feature vectors under different modes are mapped into the same semantic space by using a cross-modal contrast loss function, so as to realize multi-modal alignment and fusion processing of the first high-dimensional feature vectors and obtain second high-dimensional feature vectors with unified semantic metrics.

[0032] In step S104, a small sample transfer learning algorithm is used to label semantic labels corresponding to the second high-dimensional feature vectors based on fragment data components in the original multi-modal data.

[0033] In step S105, a multi-channel hash encoder is used to call different adaptive encoding strategies for different modal data combinations, so as to encode the second high-dimensional feature vectors into multi-channel binary hash codes.

[0034] In step S106, the binary hash codes and corresponding semantic labels are dynamically expanded into a historical knowledge graph by using an incremental graph neural network IncGNN, so as to obtain an updated target knowledge graph; the KL divergence of new entities and historical entities in the target knowledge graph is calculated, and if the KL divergence is greater than a set threshold, it is determined that semantic drift occurs and local re-clustering is triggered, and the target knowledge graph is dynamically updated.

[0035] In step S107, a graph convolution network GCN containing a local structure enhancement contrast module is used to establish matching fine-grained labels for semantic differentiated features of different entity combinations in the target knowledge graph, and the fine-grained labels in the target knowledge graph are hierarchically constructed by combining a first label association matrix, a second label association matrix and a third label association matrix, so as to obtain a cross-modal label tree corresponding to the target knowledge graph.

[0036] Through the above steps S101 to S107, first, through the multi-branch deep feature extraction network, the most suitable deep learning model is adapted for feature extraction according to the data characteristics of different modalities such as images, texts and audios. This specialized processing method can fully exploit the unique information of each modality data, laying a solid foundation for subsequent analysis. After extracting high-dimensional features, a cross-modal contrast loss function is innovatively used to map feature vectors from different modalities into a unified semantic space. This mapping mechanism effectively bridges the semantic gap between modalities, achieving deep alignment and organic fusion of multi-source data, so that the fused feature vectors have unified and comparable semantic measurement standards, improving the consistency and interpretability of cross-modal data. In the semantic label generation stage, a small sample transfer learning algorithm is introduced to fully utilize the rich semantic prior knowledge accumulated by the pre-trained model. This strategy is particularly suitable for the actual situation of scarce annotation resources in a fragmented data environment, and only a small amount of labeled samples are needed to efficiently generalize to new data scenarios. This learning mechanism greatly reduces the dependence on large-scale manual annotation, not only reducing the workload of manual annotation, but also shortening the model training period and improving annotation efficiency. Furthermore, to meet the real-time processing needs of massive fragmented data, a multi-channel hash encoder and an incremental graph neural network are used in a collaborative architecture. The multi-channel hash encoder can dynamically select the optimal encoding strategy according to different modality combinations, converting high-dimensional feature vectors into lightweight binary hash codes. This representation form not only preserves key semantic information but also optimizes storage space and computing efficiency. The introduction of the incremental graph neural network (IncGNN) solves the key challenge of real-time updating. This technical solution can dynamically integrate newly generated hash codes and labels into the existing knowledge graph without complex global reconstruction, realizing seamless online expansion of the knowledge base. This architecture has broken through the real-time processing and knowledge sedimentation of data at the scale of millions, enabling the data governance system to have the ability to continuously evolve. In the knowledge graph and label system construction stage, a graph convolutional network with a local structure enhancement contrast module is introduced to deeply model complex entity relationships in the knowledge graph. This module can accurately identify the semantic difference features between different entity combinations, and through a contrast learning mechanism, it can strengthen the expression of key differentiated information, effectively avoiding label confusion. Based on this fine semantic modeling, the system constructs a hierarchical cross-modal label tree structure, forming a semantic progressive system from macro concepts to micro attributes. This tree-shaped label architecture greatly improves the fineness and coverage of semantic description, enabling the recall ability of key features to reach a new level. The finally constructed knowledge graph has high structural rigor and knowledge completeness, providing a solid semantic understanding foundation for downstream tasks.Overall, it shows systematic breakthroughs in multi-modal data processing, small sample learning, real-time knowledge updating, and fine-grained label construction, effectively solving the problem of fragmented data and multi-source heterogeneous data management, achieving automated, efficient, and low-cost large-scale data processing, and covering the refinement of feature extraction, the precision of semantic alignment, the real-time of knowledge sedimentation, and the granularity of label generation, providing a feasible technical path for the intelligent management of multi-modal data.

[0037] In step S101, the original multi-modal data to be processed is obtained.

[0038] In the embodiments of the present application, the original multi-modal data, i.e. fragmented data or fragmented data, is a heterogeneous data set containing text, image, audio, video, time series signal and other forms of expression. These data come from a wide range of sources and carry different dimensions of information, and exist in a fragmented state, making it difficult to directly analyze them uniformly. For example, in the e-commerce field, the original multi-modal data includes text descriptions of product introductions, product real photo pictures, user review text content, and product display videos. In the industrial scenario, sensor time series data generated by device operation, audio signals of device fault alarms, image data of device appearance or internal structure, all belong to the category of original multi-modal data.

[0039] In the embodiments of the present application, the original multi-modal data, also known as fragmented data, refers to a heterogeneous data set composed of multiple modalities that has not been preprocessed or structured. The modalities here can include but are not limited to: text (such as product title, description, user reviews, etc.), images (such as product real photo, device appearance or internal structure photo, etc.), audio (such as device fault alarm sound, user voice evaluation, etc.), video (such as product display, production process monitoring, etc.), time series signal (such as temperature, current, vibration, etc. continuous data collected by sensors). These data usually come from multiple sources and have different dimensions of information, and are stored in multiple time periods and discrete fragments, making it difficult to directly perform unified correlation analysis or feature extraction.

[0040] Taking the e-commerce scenario as an example, the original multi-modal data includes but is not limited to product description text, product pictures, unboxing or review videos, user review text and pictures, etc. which all constitute original multi-modal data. Taking the industrial scenario as an example, the original multi-modal data includes but is not limited to multi-channel sensor time series data generated during device operation, alarm audio signals, images or videos of device appearance and internal structure, which also belong to the category of original multi-modal data. By uniformly modeling, aligning and fusing the above-mentioned original multi-modal data, it can provide basic data support for subsequent intelligent analysis and decision-making.

[0041] In the embodiments of the present application, multi-modal refers to multiple data types or data structures of data information. By fusing the inputs of multiple types of data (i.e., multi-modal data) such as vision, hearing, and text, the understanding ability of the system for complex information is improved. For example, an autonomous driving system combines camera images, radar data, and GPS information for environment perception, or in medical diagnosis, image, medical record, and laboratory data are fused to improve accuracy.

[0042] The acquisition methods of original multi-modal data vary due to different data sources. In the Internet application scenario, through web crawler technology, text, images, videos, and other data are captured from data sources such as web pages and social media platforms according to predetermined rules. For example, a news information platform uses a crawler to obtain news articles (text), pictures (images), and related video content on the network, realizing real-time collection of massive news information. In the Internet of Things scenario, data is collected by various sensors, and devices such as temperature sensors, pressure sensors, and cameras collect environmental parameters, device state information, and other data in the physical world in real time. For example, temperature and humidity sensors in a smart building continuously collect indoor temperature and humidity data (time series data), and cameras collect personnel activity pictures (image data), providing basic data for intelligent building management. In enterprise internal systems, structured or semi-structured multi-modal data is obtained from databases and business systems through API interfaces, such as authorized data such as customer information text and customer service call recordings (audio) in an enterprise customer management system, realizing integration and circulation of enterprise data.

[0043] Therefore, on the one hand, the diversity of multi-modal data can more comprehensively depict the characteristics of things, avoiding the limitations of single modal data. For example, in the field of intelligent security, combining monitoring video (image, time series) and abnormal sound detection (audio) data can more accurately identify security events such as intrusion and fire, and the misjudgment rate is greatly reduced compared to relying only on video data. On the other hand, the acquisition of original multi-modal data enables the system to handle complex and real application scenarios and meet the needs of different industries for deep data analysis. For example, in medical diagnosis, doctors can integrate multi-modal data such as patient medical record text, medical images (images), and electrocardiogram signals (time series) to make more accurate diagnosis decisions and improve the accuracy and reliability of medical diagnosis.

[0044] It can be further understood that multi-modal refers to the technical category of collaborative processing and fusion of any two or more data types in the above-mentioned original multi-modal data. By integrating different modal information at the feature layer or the decision layer, the system can obtain more complete semantics and spatial and temporal context, thereby improving the perception, understanding and reasoning ability of complex scenes. For example, in the automatic driving scene, camera images, laser or millimeter wave radar point clouds and positioning signals (GPS / IMU) are combined for environment perception and vehicle positioning. For example, in medical diagnosis, medical images, electronic medical record texts and laboratory test results are integrated to improve the accuracy of disease screening and diagnosis.

[0045] In step S102, a multi-branch deep feature extraction network is used to extract first high-dimensional feature vectors under different modalities from the original multi-modal data according to the characteristics of different modal data using different deep learning modes.

[0046] As an optional embodiment, in step S102, the modal to which each original data in the original multi-modal data belongs is identified, and the original data of different modalities is preprocessed; the preprocessed original text sequence is input into a BERT and BiLSTM combined model, a BERT pre-training model is used to extract the context semantic features of the original text sequence, and a BiLSTM is used to capture the bidirectional dependency relationship implied in the fragmented data in the original text sequence through sequence modeling to obtain a text semantic vector under the text modality; the preprocessed original image matrix is extracted through a ResNet-50 network to extract image visual features, and then a Transformer model is used to strengthen the extraction of global semantic features to obtain a visual semantic vector under the image modality; the preprocessed original time series data is extracted through an LSTM autoencoder to extract dynamic pattern features, and the LSTM autoencoder is used to realize feature reconstruction to retain the time series dependency to obtain a time series pattern vector under the image modality.

[0047] Specifically, the multi-branch deep feature extraction network is a basic link for cross-modal data feature mining, and its core is to customize the combination of deep learning models according to the characteristics of text, image and time series data to extract accurate high-dimensional semantic features. The network standardizes the original multi-modal data through modal recognition and preprocessing to create conditions for subsequent deep feature extraction.

[0048] It can be understood that cross-modal refers to a data mining method that establishes semantic association and information interaction between different modalities (such as text, image, audio, video, etc.). Its core is to break down the semantic gap between different modalities through algorithm models to realize information conversion, fusion and collaborative processing across sensory channels, thereby improving the understanding and generation ability of artificial intelligence systems for complex information.

[0049] In text modality feature extraction, the joint BERT and BiLSTM model plays a crucial role. The BERT pre-trained model learns language knowledge based on massive amounts of text data, enabling it to understand the contextual semantics of text. For example, when processing the sentence "This phone has powerful performance but poor battery life," BERT can accurately identify the semantic comparison between "performance" and "battery life." BiLSTM, on the other hand, captures the dependencies between fragmented information in text through bidirectional sequence modeling. For instance, when analyzing the development of events in news reports, BiLSTM can combine contextual information to clarify the causal logic of events. The combination of both can transform the original text sequence into a text semantic vector containing rich semantic and structural information, providing support for text content understanding and tag generation.

[0050] For example, employing newer pre-trained models such as the GPT series, ERNIE, or RoBERTa, with their optimized training strategies and larger corpora, can extract richer semantic information. Alternatively, combining dynamic masking techniques (such as SpanBERT) can enhance the ability to model the semantics of text fragments. Furthermore, modeling text as a graph structure (such as a semantic dependency graph) and utilizing graph convolutional networks (GCNs) or graph attention networks (GATs) to capture complex relationships between words can overcome the limitations of traditional sequence models.

[0051] Feature extraction of image modalities relies on the collaborative work of ResNet-50 and Transformer. ResNet-50, as a convolutional neural network, excels at extracting local visual features from images; for example, when recognizing car images, it can capture visual information about basic components such as headlights and tires. Transformer, building on this, utilizes a self-attention mechanism to enhance the understanding of global image semantics. When analyzing a panoramic city image, Transformer can integrate elements such as buildings, roads, and crowds to construct a complete scene semantics. Together, they transform the original image matrix into a visual semantic vector containing both local details and global semantics, enabling semantic alignment of image information with other modal data.

[0052] In addition to the models mentioned above, Swing Transformer, Vision Pro Transformer, and others can also be used.

[0053] Models like MobileViT (ViT) and others, with their window attention mechanism and hierarchical architecture, can process local and global information in images more efficiently, making them particularly suitable for feature extraction from complex scene images. Furthermore, lightweight models such as MobileViT and ShuffleNet V3 can be used to significantly reduce computational load while maintaining accuracy, making them suitable for edge devices or real-time processing scenarios.

[0054] The feature extraction of time series data adopts an LSTM autoencoder. The unique gating mechanism of LSTM makes it suitable for processing time series and can remember long-term dependency information, such as recording the correlation of price changes at different time points when analyzing stock price trends. The autoencoder compresses the original time series data into a dynamic pattern feature vector through the encoding-decoding process, and then reconstructs the data to ensure that the key time series patterns and dependencies are preserved. For example, an LSTM autoencoder can extract pattern features of abnormal fluctuations from time series data such as temperature and pressure of a device in operation, providing data support for fault warning.

[0055] In addition to the above models, self-supervised contrast learning frameworks such as SimCLRv2, MoCo v3, etc. can be used to introduce dynamic dictionaries and memory mechanisms in cross-modal feature alignment, improving the semantic consistency of different modal features. Further, reference can be made to the architecture of multi-modal pre-training models such as CLIP (Contrastive Language Image Pretraining) and ALIGN, which can learn cross-modal mapping relationships through large-scale image-text pairs, enhancing feature fusion effects.

[0056] From a technical effect point of view, the multi-branch deep feature extraction network realizes the specialized processing of different modal data. Through customized model combination, the data features of each modality are fully explored, avoiding information loss when a general model processes multi-modal data. The extracted high-dimensional feature vectors lay a solid foundation for subsequent cross-modal fusion and label generation, improving the semantic expression ability of multi-modal data, enabling the system to more accurately capture the essential features of complex fragmented data when processing complex fragmented data, and providing a guarantee for accurate generation of multi-modal labels.

[0057] It can be understood that further optionally, in the multi-branch deep feature extraction network, different deep learning modes work collaboratively through standardized data preprocessing, feature extraction division of labor, and unified cross-modal fusion goals, forming a complete link from heterogeneous data to unified semantic features.

[0058] In the data input stage, the original multi-modal data (text, image, time series, etc.) is first subjected to modal recognition and standardized preprocessing, converting different formats of data into forms suitable for processing by each branch network. For example, text data is unified into word vector sequences, image data is adjusted to fixed-size pixel matrices, and time series data is standardized into equally spaced sampling sequences. This preprocessing provides a foundation for stable operation of subsequent branch models, ensuring that different modal data are comparable at the same scale.

[0059] In the text branch, the BERT pre-training model captures the context information of the text by virtue of the general semantic knowledge learned from a large amount of corpus, and the BiLSTM supplements the long-distance dependency relationship of the text through bidirectional sequence modeling, and the combination of the two realizes the progression from "semantic understanding" to "structure modeling". In the image branch, the ResNet-50 first extracts local features (such as edges and textures) of the image using convolutional layers, and the Transformer integrates global semantics (such as scene layout and object relationship) based on this through a self-attention mechanism, forming a collaborative mode of "local perception-global cognition". In the time sequence branch, the encoding process of the LSTM autoencoder compresses the dynamic mode of the time sequence data, and the decoding process verifies and strengthens the time sequence dependency of the features, ensuring that the extracted features contain core patterns and retain the essential properties of time sequences. These branch networks, although seemingly independent, complement each other through division of labor, avoiding information omission or semantic bias when a single model processes multi-modal data.

[0060] The high-dimensional feature vectors extracted by each branch are finally fused through cross-modal contrast learning. The cross-modal contrast loss function serves as a connecting bridge, forcing the feature vectors of different modalities to align in the semantic space. For example, in the e-commerce scenario, the semantic vectors extracted from the product description text and the visual vectors extracted from the product picture are mapped to the same semantic space through contrast learning, establishing a connection between text keywords such as "fashion" and "durable" and design styles and material details in the picture. This synergy not only depends on the accuracy of the feature extraction by each branch network, but also relies on the constraint of the cross-modal loss function on the feature space, ensuring that the information of different modalities can complement and verify each other, and finally forming a fused feature with unified semantic metrics, providing comprehensive and comparable input for subsequent label generation.

[0061] Through the above collaborative mechanism, the multi-branch deep feature extraction network breaks down the modal barriers and transforms fragmented and heterogeneous raw data into structured and semantic feature representations, providing core technical support for deep analysis and intelligent label generation of multi-modal data.

[0062] In step S103, the first high-dimensional feature vectors under different modalities are mapped into the same semantic space through a cross-modal contrast loss function, realizing multi-modal alignment and fusion processing of the first high-dimensional feature vectors, and obtaining a second high-dimensional feature vector with unified semantic metrics.

[0063] As an optional embodiment, in step S103, the first high-dimensional feature vectors are respectively converted into a unified feature dimension by independent encoders, and then the re-encoded first high-dimensional feature vectors are projected into a shared semantic space; a feature vector under different modalities is selected from the shared semantic space as an anchor sample; a feature vector with the same semantics as the anchor sample is selected from the shared semantic space as a positive sample, and a feature vector with different semantics from the anchor sample is selected as a negative sample to obtain a cross-modal sample pair; a pre-trained CLIP model is used to generate a continuous similarity weight of the cross-modal sample pair, and a cross-modal contrast loss function is constructed based on the continuous similarity weight; the continuous similarity weight is used to reflect the semantic correlation strength between different first high-dimensional feature vectors in the cross-modal sample pair; the similarity distribution between the cross-modal sample pairs is calculated by using the cross-modal contrast loss function, a dynamic boundary threshold is set by using a ternary fuzzy boundary constraint, a similarity distribution heat map is constructed by referring to the similarity distribution and the dynamic boundary threshold, the relative positions between the cross-modal feature vectors in the shared semantic space are adjusted to narrow the semantic similar samples and widen the semantic irrelevant samples; an association matrix of the cross-modal feature vectors in the shared semantic space is calculated through an attention cross mechanism; in combination with the similarity distribution heat map, a cross-modal semantic related area in the association matrix is identified to realize cross-modal semantic alignment; a learnable gating coefficient is used to weight and fuse the cross-modal associated feature vectors in the cross-modal semantic related area and the original first high-dimensional feature vectors, and a multi-granularity layer is used to mix and extract the weighted fusion results to obtain the second high-dimensional feature vectors.

[0064] In the embodiments of the present application, high-dimensional feature vectors, independent encoders, shared semantic spaces, anchor samples, positive samples, negative samples, pre-trained CLIP models, continuous similarity weights, cross-modal contrast loss functions, cross-modal sample pairs, ternary fuzzy boundary constraints, dynamic boundary thresholds, similarity distribution heat maps, association matrices, and learnable gating coefficients

[0065] Specifically, in step S103, the core of the cross-modal contrast loss function is to break through the semantic barriers of different modalities such as text, image, and time sequence, and map heterogeneous features to a unified space by constructing a computable semantic correlation measurement system.

[0066] Taking the cross-modal alignment of product text description and real image in an e-commerce scenario as an example, in step S103, first, the independent encoders unify the dimensions and project the semantics of different modalities. For example, the BERT feature (768 dimensions) of the product text and the ResNet feature (2048 dimensions) of the image are respectively converted into 512-dimensional vectors through a fully connected layer, and then projected into a shared semantic space. At this time, the text feature of "red dress" and the image feature of red skirt will be regarded as potential associated samples, while the image feature of "blue jeans" will be regarded as a semantic irrelevant sample.

[0067] High-dimensional feature vectors refer to high-dimensional numerical vectors containing rich semantic information extracted from raw data. In the security scenario, image feature vectors extracted from monitoring video frames may contain information such as the shape, color, and motion trajectory of the target object, with dimensions up to thousands of dimensions; and text feature vectors obtained after natural language processing of alarm texts encode semantics such as event description, time of occurrence, and location, with dimensions up to hundreds of dimensions. These high-dimensional feature vectors are the basis for subsequent processing and retain the core information of the original data, but are difficult to directly correlate due to differences in dimensions and modal characteristics.

[0068] The conversion module for different modal data uses an independent encoder to map high-dimensional feature vectors to a unified dimension for subsequent processing. For example, for high-dimensional visual features of video images, the independent encoder compresses and converts them into 512-dimensional vectors through a multi-layer neural network; for semantic features of alarm texts, the encoder also maps them to 512 dimensions. In this way, the features of different modalities are consistent in dimension, just like machining different specifications of parts to a unified size, providing a prerequisite for cross-modal fusion.

[0069] In the embodiments of the present application, the shared semantic space is an abstract mathematical space, and the feature vectors of different modalities are mapped to it after encoding to achieve unified expression at the semantic level. In the security scenario, whether it is a video image feature or an alarm text feature, after being processed by the independent encoder, it is projected into the shared semantic space. In this space, the video feature of the "intrusion event" and the text feature describing the event can be associated based on semantics, so that data originally belonging to different modalities can be compared and fused in the same language environment.

[0070] The reference feature vector selected from the shared semantic space is an anchor sample used to construct sample pairs and calculate similarity. For example, when processing security data at a certain time, an alarm text feature describing "someone climbing over the fence" is selected as an anchor sample. This anchor sample will serve as a reference for finding other samples that are semantically related or unrelated.

[0071] The positive sample is a feature vector with the same or similar semantics as the anchor sample. In the above example, the image feature vector in the monitoring video frame that contains the action of a person climbing over the fence corresponding to the text description "someone climbing over the fence" is the positive sample. The selection of the positive sample aims to strengthen the model's ability to recognize semantic consistency, so that the model learns the data features that express the same semantics under different modalities.

[0072] The negative sample is a feature vector with different semantics from the anchor sample. For example, the alarm text feature describing "vehicle normal passing" or the image feature showing a person walking normally in the monitoring video can be used as a negative sample of the anchor sample "someone climbing over the wall". The introduction of negative samples helps the model to distinguish different semantics and avoid incorrect association of irrelevant information.

[0073] The selection of anchor samples and positive and negative samples follows the principle of semantic consistency. If the text feature of "red dress" is used as an anchor, the corresponding image feature of the red dress is a positive sample, and the image feature of blue jeans is a negative sample, forming a cross-modal triple. The pre-trained CLIP model plays a key role in this regard. Its cross-modal understanding ability obtained through large-scale image-text pair training can calculate the similarity weight (such as 0.92) between the text and image of "red dress" and the similarity weight (such as 0.21) between the text and the image of blue jeans. These weights directly reflect the strength of semantic association.

[0074] It can be understood that in the embodiments of the present application, the cross-modal model trained by large-scale image-text data can understand the semantic association between images and texts. In security applications, the pre-trained CLIP model can calculate the semantic similarity between the text feature of "someone climbing over the wall" and the corresponding monitoring screen feature, and can also judge the difference between it and other unrelated pictures or texts. Based on the knowledge learned by pre-training, similarity scores are assigned to feature pairs of different modalities.

[0075] The numerical values generated by the pre-trained CLIP model are continuous similarity weights used to quantify the strength of semantic association between cross-modal sample pairs. For example, for the text of "someone climbing over the wall" and the corresponding video screen, the CLIP model may give a similarity weight of 0.9, indicating that the two are highly related in semantics; while for the text of "vehicle normal passing" and the video screen of "someone climbing over the wall", the similarity weight may be only 0.2, which intuitively reflects the semantic difference.

[0076] The construction of the cross-modal contrast loss function introduces a three-element fuzzy boundary constraint. Taking the similarity distribution of anchor, positive sample and negative sample as an example, the dynamic boundary threshold is set to be the similarity of the positive sample minus 0.3, and the loss calculation is triggered when the similarity of the negative sample exceeds the threshold. The similarity distribution can be visualized by a heat map, and the high similarity area (warm color) of the text and image of "red dress" and the low similarity area (cool color) of the image of blue jeans can be directly observed. The model adjusts the feature position accordingly, so that the distance between samples of the same type is reduced (such as the distance between the text and the corresponding image feature is reduced), and the distance between different samples is increased.

[0077] The cross-modal contrast loss function is a mathematical function based on a continuous similarity weight, used to measure the degree of semantic matching between cross-modal sample pairs and guide the model to adjust parameters. In the security scenario, the function calculates the similarity between the anchor sample (alarm text) and the positive sample (corresponding video frame), as well as the difference with the negative sample, by minimizing the loss value, prompting the model to bring similar samples closer together in the shared semantic space, and push away samples that are not semantically related.

[0078] The cross-modal sample pair is a combination of the anchor sample and the positive sample or the negative sample. For example, the text feature "someone climbing the fence" and the video frame feature "climbing the fence" constitute a positive sample pair, and the text feature "someone climbing the fence" and the text feature "vehicle passing normally" constitute a negative sample pair. These sample pairs are the basic units for model learning and optimization, and by comparing the similarity of different sample pairs, the model gradually masters the rules of cross-modal semantic alignment. The similarity distribution heat map is a chart that visually presents the similarity between feature vectors in the shared semantic space. In the security data analysis interface, the similarity distribution heat map can intuitively show the degree of association between different alarm texts and monitoring picture features, with darker colors indicating higher similarity. Through the similarity distribution heat map, the operator can quickly find potential associations in the data, and also provides a visual basis for model optimization.

[0079] Further, the dynamic boundary threshold is a specific threshold value based on the ternary fuzzy boundary constraint, which will adjust with the changes in sample data. In the process of continuous operation of the security system, when new event types appear or the data distribution changes, the dynamic boundary threshold can be adjusted adaptively to ensure that the model always performs semantic alignment with appropriate standards, avoiding false positives or false negatives caused by fixed thresholds.

[0080] Further, the ternary fuzzy boundary constraint is used to set the mechanism of the similarity threshold, based on the relationship between the anchor sample, the positive sample and the negative sample, to define a fuzzy boundary range. In security data processing, the constraint dynamically determines a threshold range based on the similarity distribution of the sample pairs. For example, if the similarity between the positive sample and the anchor sample is 0.9, and the similarity between the negative sample and the anchor sample is 0.2, the ternary fuzzy boundary constraint may set the threshold range to 0.6-0.8, to determine whether the semantic association of the sample pair meets the expectations.

[0081] The attention cross mechanism further strengthens the semantic association mining. By calculating the association matrix of all feature vectors in the shared space, it can be found that the "red" text feature has a high weight connection with the red pixel area in the image, and the "dress" text feature has a high weight connection with the skirt contour in the image. Combined with the similarity heat map, these high weight areas are identified as cross-modal semantic related areas, such as the correspondence between text keywords and image visual focus.

[0082] wherein the matrix calculated by the attention cross mechanism is a correlation matrix used to quantify the semantic correlation strength between all feature vectors in the shared semantic space. In the security scenario, the correlation matrix can show the correlation between different alarm event text features and various monitoring picture features, such as the correlation degree of the “fire alarm” text with the flame picture feature, the correlation degree with the smoke picture feature, etc., to provide accurate quantitative basis for subsequent semantic alignment and information fusion.

[0083] The final feature fusion adopts a learnable gating coefficient. Specifically, the learnable gating coefficient introduced in the feature fusion stage is used to control the weight of different modal features in the fusion process. For example, when fusing the text feature and the video picture feature of “someone climbing over the wall”, the learnable gating coefficient will automatically adjust the weight according to the correlation strength of the two in the shared semantic space. If the text description is clear and the video picture feature is obvious, the gating coefficient may give the text feature a weight of 0.6 and the video picture feature a weight of 0.4, so as to realize reasonable fusion of multi-modal information, generate a more representative high-dimensional feature vector, and improve the accuracy of security event detection and analysis.

[0084] For example, for cross-modal features related to semantic regions (such as “red” text features and red pixel image features), the gating coefficient automatically gives a high weight (such as 0.8) to irrelevant features (such as 0.2), and then extracts semantic information of different scales (such as global “dress” concepts and local “red” attributes) through multi-granularity layer mixing, to generate a second high-dimensional feature vector containing multi-modal semantics.

[0085] From the technical effect, this mechanism improves the accuracy of semantic alignment of text and images on e-commerce platforms. When a user searches for “red dress”, the system can accurately match images containing a red skirt, reducing the mis-matching of blue or pants-type goods. In the medical scenario, the cross-modal alignment error rate of pathological text description and medical images is reduced, realizing accurate correlation between “lung nodule” text and nodule area in CT images, and providing a solid semantic unification foundation for intelligent analysis of multi-modal data.

[0086] Step S104: using a small sample transfer learning algorithm, based on the fragmented data components in the original multi-modal data, labeling the semantic labels corresponding to the second high-dimensional feature vector.

[0087] As an optional embodiment, in step S104, the fragmented data components in the original multi-modal data are identified, and the field relevance indexes of the fragmented data components are calculated; the field relevance index represents the semantic overlap of each fragmented data component with the target field; the general pre-training model is adaptively migrated to the target field of the fragmented data components through parameter fine-tuning and knowledge distillation, and a small sample labeling model in the target field is constructed; the parameter fine-tuning includes adding a fragmented field self-adaptive Adapter to the top layer of the general pre-training model; the parameter sparsity of the fragmented field self-adaptive Adapter is dynamically adjusted based on the field relevance index, and the lower the relevance is, the higher the sparsity is; the second high-dimensional feature vector is semantically labeled by the small sample labeling model, and the semantic label corresponding to the second high-dimensional feature vector is obtained.

[0088] In the multi-modal data processing flow, step S104 realizes efficient labeling of semantic labels through a small sample transfer learning algorithm, effectively solving the problem of insufficient labeled samples in the fragmented data scenario. Taking the analysis of medical images and diagnostic texts as an example, the technical logic and application value of this process can be fully demonstrated.

[0089] First, identify the fragmented data components in the original multi-modal data. In the medical field, the original data includes image data such as X-ray images and CT scan images of patients, and text data such as diagnostic records and medical record descriptions written by doctors. These data often exhibit fragmented characteristics: X-ray images may only record local lesions at a certain time, and diagnostic texts may only describe the disease condition briefly. Identifying fragmented data components is to sort out the scattered but key information in these data, such as locating the position and shape of lung nodules from CT images, and extracting the preliminary judgment of "suspected pneumonia" from diagnostic records, to provide raw materials for subsequent model training.

[0090] Next, the general pre-trained model is migrated to the target field through parameter fine-tuning and knowledge distillation. Taking the visual language model CLIP as an example, it is pre-trained on large-scale image-text data and learns general image-text semantic association knowledge. In the medical scene, due to the limited labeled medical images and diagnosis text samples, directly training a new model is prone to overfitting. At this time, the parameter fine-tuning strategy is adopted, a domain adapter is added to the top layer of the CLIP model, the bottom layer parameters are frozen, and only the adaptation layer is fine-tuned for medical data, so that it adapts to the professional terms and image features of the medical field; At the same time, knowledge distillation is used to build a teacher-student model architecture, the teacher model is the CLIP trained on the general data set, and the student model learns on a small number of medical image and text pairs (such as 100 groups of pneumonia images and diagnosis descriptions). The soft label output by the teacher model (such as the probability distribution of each disease) guides the student model training, and through the loss function such as KL divergence, knowledge is transferred to let the student model quickly master the semantic association rules in the medical field.

[0091] Finally, the second high-dimensional feature vector is semantically annotated by the constructed small sample annotation model. When the second high-dimensional feature vector of a patient's chest X-ray image and related medical record text fused in step S103 is input, the small sample annotation model can output accurate semantic labels such as "pneumonia (confidence 89%) " and "pulmonary texture thickening". These labels not only include disease diagnosis results, but also cover image feature descriptions, providing auxiliary diagnosis information for doctors.

[0092] From the technical effect, the small sample transfer learning algorithm improves the annotation efficiency and accuracy. In medical data annotation, compared with traditional full-data training model, this method reduces the sample size required for annotation, shortens the training cycle from several weeks to several days, and improves the annotation accuracy. At the same time, it effectively avoids the problem of data scarcity caused by strong medical data privacy and high annotation cost, provides a feasible technical solution for medical image auxiliary diagnosis, disease prediction and other applications, and provides a reference for semantic annotation in other small sample scenarios (such as rare disease research and new material analysis), achieving efficient transfer from general knowledge to field-specific knowledge.

[0093] Further optionally, in the above step, the general pre-training model is adapted and migrated to the target field of the fragmented data component through parameter fine-tuning and knowledge distillation to construct a small sample labeling model in the target field, comprising: adding a fragmented field adaptation Adapter on the top layer of the general pre-training model, freezing the bottom layer parameters, and fine-tuning the adaptation layer; the fragmented field adaptation Adapter includes a semantic completion sublayer for text fragments and a visual repair sublayer for image fragments, and the enabling conditions of each sublayer are obtained based on the modal type of the corresponding fragmented component; a teacher-student model is constructed; a plurality of different initialized teacher models are used to generate integrated soft labels on a general data set, and the student model learns in the fragmented data set in the target field of the fragmented data component, and the output distribution of the plurality of teacher models is fitted through a KL divergence loss function; the output distribution of the teacher model is softened through a temperature parameter; wherein the temperature parameter is dynamically adjusted to balance the generalization and specificity of knowledge transfer when the student model is trained in the fragmented data set; when the sample size of the fragmented data set is less than a preset threshold, the temperature parameter is increased to increase the smoothness of the soft label and reduce the risk of overfitting.

[0094] Specifically, in the small sample transfer learning scenario, adapting the general pre-training model to the target field through parameter fine-tuning and knowledge distillation is the core technical path to achieve efficient semantic labeling. Taking industrial equipment fault diagnosis as an example, the device operation data often presents fragmented characteristics, and the labeled samples are scarce, and this series of technical means can effectively solve the model training problem caused by insufficient data.

[0095] Adding a fragmented field adaptation Adapter on the top layer of the general pre-training model and freezing the bottom layer parameters and fine-tuning the adaptation layer is the basic operation to realize field migration. The pre-training model such as BERT has learned the basic semantic knowledge of language on a large amount of general text, but in the industrial field, professional terms such as "bearing clearance" and "gear meshing frequency" need to be adapted by the model. At this time, a lightweight field adapter is added on the top layer of BERT, which is responsible for converting general semantics to industrial field semantics. Freeze the bottom layer parameters of BERT to avoid changing its learned general knowledge, and only fine-tune the parameters of the adapter. For example, when processing device fault diagnosis text, the adapter learns to associate "vibration anomaly" with device fault patterns through training of a small amount of industrial text samples, so that the model quickly adapts to the semantic expression of the industrial field.

[0096] The Fragmentation Domain Adaptive Adapter is a core adaptation module for multimodal fragmented data processing. Its core design philosophy is to achieve efficient repair and adaptation of different types of fragmented data through precise modality-specific processing and conditional dynamic activation. The semantic completion sublayer for text fragments primarily addresses semantic breaks caused by transmission interruptions or incomplete truncation. This sublayer first uses a text modality feature extractor, such as a BERT-based pre-trained model, to analyze the grammatical structure and contextual relationships of the fragments. Then, it uses a context-aware attention mechanism to fill in the semantic gaps. For example, for truncated text fragments like "The 2024 Global Artificial Intelligence Conference will be held in location A, with an expected attendance exceeding [number missing]," the sublayer can combine a domain corpus to complete key information such as "50,000 attendees, focusing on the application of generative AI industries," ensuring the semantic integrity and logical coherence of the text.

[0097] The visual inpainting sublayer for image fragments focuses on addressing the problem of incomplete visual information caused by pixel loss and regional damage. Furthermore, it employs a visual completion architecture based on Generative Adversarial Networks (GANs). First, it performs feature matching on the edge contours and texture features of the image fragments, and then reconstructs the missing areas through a generator module. For example, in surveillance footage, for vehicle license plate fragments caused by occlusion, the sublayer can repair a complete license plate image based on the overall vehicle outline and surrounding pixel textures, while simultaneously ensuring the authenticity of the repaired result through visual consistency verification.

[0098] It is worth noting that the activation conditions for the above two sub-layers can be dynamically determined based on the modality type of the input data. When the data contains text feature identifiers (such as UTF-8 encoding format, part-of-speech tagging information), the text semantic completion sub-layer is automatically activated; when the data contains image feature identifiers (such as RGB channel structure, pixel matrix dimensions), the image visual restoration sub-layer is automatically activated. If mixed text and image fragments are encountered (such as a damaged poster with text annotations), both sub-layers will be activated simultaneously, and the overall restoration will be completed through a cross-modal collaborative mechanism, ensuring the accuracy and efficiency of the adaptation process.

[0099] A teacher-student model is constructed, and multiple different initialized teacher models are used to generate integrated soft labels to further improve the stability of knowledge transfer. In industrial fault diagnosis, multiple BERT models with different initializations are used as teacher models, which learn different semantic understanding focuses on general text datasets. The prediction probability distribution (soft label) generated by these teacher models for the same device fault description text is different, some may focus on semantic logic, and some may focus on keyword matching. The student model is then learned on a small amount of industrial device fault text and corresponding fault type label (such as "motor overheating-cooling system failure") fragment dataset. Through the KL divergence loss function, the student model synchronously fits the output distribution of multiple teacher models, like learning different problem-solving ideas from multiple experienced "teachers", thereby absorbing more comprehensive knowledge. For example, when the student model learns to diagnose "device abnormal noise", it can more accurately associate it with "bearing wear" or "loose gear" and other fault causes by integrating the soft labels of multiple teacher models.

[0100] Softening the output distribution of the teacher model through the temperature parameter is the key to balancing the generality and specificity of knowledge. The temperature parameter acts like a regulator, controlling the sharpness of the soft label. In the early stages of industrial fault diagnosis training, when the sample size of the fragment dataset is small (such as only 20 groups of fault data are collected), the temperature parameter is increased to make the output distribution of the teacher model more smooth. For example, the teacher model originally judges that a certain device fault is "motor overheating" with a probability of 90%, after temperature softening, the probability distribution of related fault types such as "motor overheating" and "circuit short circuit" is more uniform, avoiding the student model from prematurely relying too much on local features in limited samples, and reducing the risk of overfitting. As the training data increases, the temperature parameter is gradually reduced, making the distribution of soft labels more focused, enhancing the model's ability to recognize specific fault patterns in the industrial field, and achieving a dynamic balance between general knowledge and field-specific knowledge.

[0101] Therefore, compared with the knowledge distillation method of a single teacher model, after using multiple teacher integrated soft labels and dynamic temperature adjustment, the model's accuracy in identifying new fault types is improved, and its generalization ability is enhanced. At the same time, the model can achieve similar diagnostic results with only 1 / 5 of the sample size required by traditional methods, greatly reducing the dependence on large-scale labeled data in the industrial field, shortening the model development cycle, and providing efficient and reliable technical support for real-time monitoring and intelligent maintenance of industrial equipment.

[0102] In step S105, a multi-channel hash encoder is used to call different adaptive encoding strategies for different modal data combinations, and the second high-dimensional feature vector is encoded into a multi-channel binary hash code. The multi-channel hash encoder is a feature encoding technology established for multi-modal data. Different modal data (such as text, image) are processed by independent branches, and the optimal encoding strategy is dynamically selected to map high-dimensional features to compact binary hash codes.

[0103] As an optional embodiment, in step S105, for the text-image combination channel, a deep hash network is used to take the text semantic vector and the visual semantic vector as joint input, map them to a shared feature pair of the same dimension through a fully connected layer, and map the shared feature pair to a hash code through an activation function. Each bit identifies a semantic feature dimension. Through the InfoNCE loss function, the shared feature pairs with similar semantics are constructed into hash codes with close Hamming distances, and the shared feature pairs with irrelevant semantics are constructed into hash codes with far Hamming distances.

[0104] For the text-image combination channel, the deep hash network plays a core role. In the e-commerce scenario, the text semantic vector extracted by BERT+BiLSTM from the written description of the goods (such as “light and thin notebook computer, long battery life”) and the visual semantic vector extracted by ResNet-50+Transformer from the real photo of the goods are taken as joint input into the deep hash network. The network maps the features of the two modalities to the same dimension through a fully connected layer to form a shared feature pair, just like integrating the written description and the picture information on the same information card. Then, the activation function converts the shared feature pair into a binary hash code, and each bit of the code corresponds to a semantic feature dimension. For example, a bit of 1 may represent the “light and thin” feature, and a bit of 0 may represent the absence of the feature. The InfoNCE loss function plays a key constraint role in this process. It constructs the shared feature pairs of goods with similar semantics (such as different brands but the same light and thin notebook computers) into hash codes with close Hamming distances, so that these goods are adjacent in the encoding space. The hash codes of goods with irrelevant semantics (such as notebook computers and sports shoes) maintain a large Hamming distance. As a result, the originally high-dimensional text-image features are compressed into compact binary codes, which greatly reduces the storage and computing costs while ensuring semantic consistency.

[0105] In step S105, for the time-series text combination channel, a graph attention hash network is used to construct a time-series text heterogeneous association graph from the text semantic vector and the time-series mode vector, calculate the association weights between the graph nodes in the time-series text heterogeneous association graph through the graph attention mechanism, make the hash code capture the dependency relationship between the time-series mode and the text semantics, and obtain a hash code containing structural information.

[0106] In the time-series text combination channel, the graph attention hash network is suitable for processing data with time series characteristics. Taking industrial equipment monitoring as an example, the time-series pattern vectors extracted by the LSTM autoencoder from the sensor data of the equipment running (such as the time-series changes of temperature and speed) and the text semantic vectors extracted from the equipment maintenance log and other text data are combined to form a time-series text heterogeneous association graph. The nodes in the graph represent different time-series data segments or text keywords, and the edges represent the association between them. The graph attention mechanism highlights the importance of key information by calculating the association weights between nodes. For example, when the temperature of the equipment continues to rise, the association weight between the "temperature sensor data" node and the "overheating warning" text node will increase. The hash code generated based on these weights can capture the dependency between time-series patterns and text semantics, such as closely associating the time-series feature of "abnormal temperature rise" with the text description of "heat dissipation system failure". The final hash code not only contains data features but also retains the structural information between time-series texts, providing support for subsequent fault diagnosis and prediction.

[0107] Finally, in step S105, a semantic-consistent binary hash code is generated by contrastive learning constraint. For the same semantic entity, the Hamming distance between the hash codes generated by different channels is less than a preset threshold.

[0108] Specifically, the realization of cross-channel semantic consistency through contrastive learning constraint is the key feature of multi-channel hash coding. In e-commerce and industrial scenarios, for the same semantic entity (such as "hot-selling goods" or "normal operation of equipment"), although the hash codes generated by different channels (text and image) come from different modal data, the contrastive learning mechanism forces their Hamming distance to be less than a preset threshold. This means that no matter whether the encoding is generated from product images and descriptions or from equipment operation data and maintenance records, as long as the semantics are the same, the encodings will be close to each other in the feature space. This constraint ensures that multi-modal data can maintain semantic consistency and associability after compression encoding, facilitating subsequent fast retrieval and analysis based on hash codes.

[0109] Therefore, the multi-channel hash encoder improves the processing efficiency of multi-modal data. In the e-commerce field, the storage cost of product image and text data is reduced, and when a user searches for "light and thin laptop", the system can quickly match similar products through hash coding, improving the retrieval speed. In the industrial scenario, the encoding and compression of equipment time-series and text data enable the fault warning system to process massive monitoring data in real time, shortening the fault identification response time to seconds, while the contrastive learning constraint ensures the semantic consistency of different data sources, reducing false positives caused by data modality differences, and effectively improving the efficiency of data governance and intelligent decision-making.

[0110] For example, the adaptive coding strategy of the multi-channel hash encoder dynamically adjusts according to the characteristics of different modal combinations. The following describes other implementation modes in combination with specific scenarios and technical logic:

[0111] For example, a time sequence correlation hash network is deployed in an audio-video combination channel. In a short video content understanding scenario, audio (such as background music and human voice) and video frames (visual pictures) need to be encoded into hash codes with consistent semantics. At this time, a time sequence correlation hash network can be used: visual features (such as human actions and scene colors) of video frames are extracted by a CNN, and time sequence features (such as melodies and rhythm spectrums) of audio are extracted by an LSTM, both of which are input into a bidirectional attention mechanism to calculate cross-modal correlation weights of audio-video frames. For example, when a picture of a person speaking appears in the video, the correlation weight of the voice segment in the audio and the corresponding video frame is enhanced. Subsequently, the hash encoding is constrained by a time sequence alignment loss function (such as CTC loss) to reduce the Hamming distance of hash encodings of audio-video segments (such as “person speaking” scenes) with the same semantics. Technically, this strategy can realize fast retrieval of short videos. For example, when a user searches for “piano performance video”, the hash encoding can match the content of the audio containing the sound of a piano and the video containing the performance picture, and the retrieval accuracy is improved.

[0112] For example, a graph semantic hash network is deployed in a text knowledge graph combination channel.

[0113] In an intelligent question-answering system, user query text (such as “Einstein's contribution to the theory of relativity”) needs to be combined with entity relationships (such as “Einstein-proposed-theory of relativity”) in a knowledge graph for encoding. At this time, a graph semantic hash network can be constructed: a text semantic vector is extracted by BERT, and a subgraph structure (such as entity nodes and relationship edges) of a knowledge graph is converted into a structured feature vector by a graph convolution network (GCN). Then, a cross-modal graph attention mechanism is used to map keywords (such as “theory of relativity”) in the text to corresponding entity nodes in the knowledge graph, for example, to weight and fuse the features of “theory of relativity” and the “physical theory” category node in the knowledge graph. When generating hash encoding, a triple graph loss function is used to require the Hamming distance between the hash encoding of the text-knowledge graph pair and the positive sample (semantically related text-graph pair) to be smaller than that of the negative sample (irrelevant pair). This strategy can more accurately associate text semantics and knowledge graph structures in a question-answering system, such as when a user asks “founder of quantum mechanics”, the system can quickly locate entities such as “Bohr” and “Einstein” in the knowledge graph through hash encoding, and the accuracy of the answer is improved.

[0114] For example, a stereo semantic hash network is deployed in a 3D point cloud image combination channel.

[0115] In the automatic driving scenario, the 3D point cloud of the laser radar (such as the spatial coordinates of the obstacles) and the camera image (such as the visual appearance of the object) need to be encoded into a unified hash code. A stereo semantic hash network can be used: the geometric features of the 3D point cloud (such as the stereo contour of the vehicle) are extracted by PointNet, the visual features of the image (such as the color and shape of the vehicle) are extracted by ResNet, and then both are input into a cross-modal feature fusion module. The module includes a spatial alignment layer, which projects the point cloud onto the image plane through the camera intrinsic matrix, so that the 3D coordinates and 2D pixel positions of the same object are corresponded. For example, the point cloud contour of the vehicle in front and the vehicle frame in the image will be forced to align, generating a related feature vector. When hashing, a multi-view contrast loss function is used, which requires the hash codes of the 3D-2D feature pairs of the same object to remain consistent under different views (such as when the vehicle is photographed from the front and the side, the Hamming distance of the hash codes is less than 0.1). Technically, this strategy can improve the obstacle recognition efficiency of the automatic driving system, reduce the data volume after hash encoding, and improve the obstacle classification accuracy, while supporting real-time retrieval of similar obstacle patterns in historical scenarios.

[0116] For example, a dynamic gated hash network is deployed in the sensor data text instruction combination channel.

[0117] In the smart home scenario, sensor data (such as time series data of temperature and humidity sensors) and user text instructions (such as "set the temperature in the living room to 25°C") need to be encoded into hash codes of control signals. At this time, a dynamic gated hash network can be used: the time series pattern of the sensor data (such as the temperature change curve) is extracted by Transformer, and the semantic features of the text instruction (such as the "temperature" and "25°C" keywords) are extracted by LSTM. The weight of the two is dynamically adjusted by a learnable gating unit. When the sensor data fluctuates greatly (such as a sudden temperature rise), the gating unit will increase the weight of the sensor features, and vice versa. For example, when the user sends a "temperature adjustment" instruction, if the current temperature is close to 25°C, the hash encoding will rely more on the sensor data, reducing unnecessary adjustments. When generating hash codes, a task-oriented loss function is used, which associates the semantic correlation between the encoding result and the actual control action (such as the air conditioner switch status) as the optimization target. This strategy can improve the instruction response accuracy of the smart home system, reduce misoperations, and reduce the storage volume of sensor data, facilitating fast retrieval of historical control scenarios on the cloud.

[0118] At step S106, the binary hash code and the corresponding semantic label are dynamically expanded into the historical knowledge graph by combining the incremental graph neural network, to obtain an updated target knowledge graph. The incremental graph neural network (IncGNN) is a graph learning model that supports dynamic updating. By gradually integrating newly generated data (such as real-time labeled labels or newly added modal features) into the existing knowledge graph, global reconstruction is avoided. For example, in a continuous labeling cross-modal data stream, only the local nodes and edges affected by the newly added data are updated to maintain the real-time and consistency of the knowledge graph, which is suitable for large-scale dynamic scenarios.

[0119] As an optional embodiment, in step S106, the binary hash code corresponding to the newly added semantic label is converted into a real-valued vector by a hyperbolic tangent function. Then, by using the incremental updating mechanism of IncGNN, the converted real-valued vector is aggregated with the node embedding of the historical entity with similar semantics in the historical knowledge graph to obtain a new entity. Then, the semantic distance between the new entity and the historical entity in the feature space is calculated by the cosine similarity to measure the closeness of the new entity and the historical entity in the feature space, so as to obtain the semantic similarity between the new entity and the historical entity. If the semantic similarity exceeds a set threshold, a relationship edge is established between the new entity and the historical entity according to the semantic association relationship of the cross-modal features, and the relationship weight between the new entity and the historical entity is set according to the Hamming similarity of the hash code and the semantic similarity, to complete the updating of the historical knowledge graph and obtain the target knowledge graph.

[0120] Specifically, in the scenario of dynamic updating of the knowledge graph, step S106 realizes efficient fusion of the binary hash code and the semantic label by using the incremental graph neural network (IncGNN).

[0121] Taking the construction of a knowledge graph in the field of intelligent education as an example, first, the binary hash code corresponding to the newly added semantic label is converted into a real-valued vector by a hyperbolic tangent function. In an intelligent education system, when a student's learning record of "basic principles of quantum mechanics" is detected, the binary hash code (such as generated by text-video multi-channel) of this knowledge point is converted into a real-valued vector. This conversion maps discrete binary codes to continuous space, facilitating subsequent similarity calculation with historical entities in the knowledge graph. For example, the binary code of "basic principles of quantum mechanics" is converted into a real-valued vector containing 512 dimensions, and each dimension represents the feature intensity of the knowledge point in different semantic directions.

[0122] Next, the incremental update mechanism of IncGNN is used to aggregate similar entity embeddings. In the knowledge graph, there are historical entity nodes such as "wave-particle duality" and "Schrödinger equation". Through the incremental update module of IncGNN, the real-valued vector of "Basic Principles of Quantum Mechanics" is aggregated with the embeddings of these historical entities. For example, the system identifies that "Basic Principles of Quantum Mechanics" is similar to "Wave-Particle Duality" in the semantic space, and then the embedding vectors of the two are weighted and fused to generate a new entity representation. This aggregation is not simply superimposed, but through the message passing mechanism of the graph neural network, the new entity absorbs the semantic knowledge of the historical entity while retaining its own characteristics, just like a student learning new knowledge while establishing connections with existing knowledge.

[0123] The semantic distance between the new entity and the historical entity is calculated by cosine similarity. Taking "Basic Principles of Quantum Mechanics" and "Wave-Particle Duality" as an example, the system calculates the cosine similarity of their real-valued vectors and gets a similarity value of 0.85, indicating that they are highly similar in the semantic space. If the similarity exceeds a certain threshold (such as 0.7), a relationship edge between them is established. In the knowledge graph, this is represented as a directed edge from the "Basic Principles of Quantum Mechanics" node to the "Wave-Particle Duality" node, symbolizing that the former contains the knowledge elements of the latter.

[0124] According to the relationship between the Hamming similarity of the hash code and the semantic similarity, the relationship weight is set. In addition to cosine similarity, the system also calculates the Hamming similarity of the hash code of "Basic Principles of Quantum Mechanics" and "Wave-Particle Duality". If 70% of the bit positions in their binary codes are the same, the Hamming similarity is 0.7. By combining the cosine similarity (0.85) and the Hamming similarity (0.7), the weight of the relationship edge is set to 0.8 (for example, take the weighted average). This weight value reflects the strength of the knowledge association, and in subsequent knowledge reasoning and recommendation, high-weight edges will be given priority.

[0125] From the technical effect, this incremental update mechanism improves the dynamic expansion capability of the knowledge graph. In the intelligent education system, the update delay of the knowledge graph is shortened from several hours in traditional methods to seconds, and the integration efficiency of new knowledge points is improved. At the same time, through the dual constraints of semantic similarity and hash similarity, the accuracy of the relationship between entities in the knowledge graph is improved, enabling the recommendation system to more accurately provide learning paths for students, such as recommending "wave function collapse" and "uncertainty principle" based on the association between "Basic Principles of Quantum Mechanics" and historical knowledge points. The matching accuracy of the student's learning path is improved, effectively enhancing the adaptability of the knowledge graph in dynamic scenarios.

[0126] As an optional embodiment, after step S106, the semantic information of the newly added entity can also be iteratively updated according to the embedding and relationship weight of the neighbor nodes by using graph convolution operation, so that the semantic information of the newly added entity and the neighbor entities are mutually fused, and the overall semantic association of the knowledge graph is ensured. Whether the redundancy is caused by the newly added entity is determined by calculating the structural similarity of each relationship edge, so as to detect whether the newly added entity causes the appearance of a relationship edge with similar semantics in the target historical graph. If the structural similarity exceeds a set structural similarity threshold, the redundant relationship edge is merged and the relationship edge weight is updated, so as to avoid the structure confusion of the target knowledge graph caused by repeated information and improve the knowledge retrieval efficiency.

[0127] After the dynamic update of the knowledge graph, the graph convolution operation and the redundancy detection mechanism further optimize the graph structure and semantic association. Taking the construction of a commodity knowledge graph in the e-commerce field as an example, the semantic information of the newly added entity and the neighbor entities is fused by using the graph convolution operation, which essentially updates the semantic information by iteratively aggregating the features of the neighbor nodes. When a new product, “foldable screen tablet computer”, is introduced into the e-commerce platform, the product is added to the knowledge graph as a newly added entity. Through graph convolution operation, it absorbs the embedding information of the neighbor nodes (such as “tablet computer”, “foldable screen technology”, and “electronic product”). For example, the “foldable screen tablet computer” node will refer to the functional attributes of “tablet computer” and the characteristic description of “foldable screen technology”, and update the semantic expression in combination with its own characteristics. This process is like a new employee integrating into a team by communicating with colleagues to acquire industry knowledge and work experience, so that the self-description is more in line with the actual business scenario. Iterative operation enables the semantic of the newly added entity to be continuously improved, and the semantic association with the neighbor entities is more closely, ensuring the semantic consistency and coherence of all entities in the knowledge graph.

[0128] The redundancy is detected by calculating the structural similarity of the relationship edge, in order to maintain the simplicity of the knowledge graph. In the e-commerce knowledge graph, the “foldable screen tablet computer” may establish a “equipped with” relationship with “high-performance processor” and “long-lasting battery”, respectively, while “high-end tablet computer” also has similar relationships with these two components. At this time, the structural similarity of the two relationship edges “foldable screen tablet computer-equip-high-performance processor” and “high-end tablet computer-equip-high-performance processor” is calculated, which not only considers the connection relationship between the nodes, but also compares the attributes of the relationship (such as the processor model and performance parameters equipped). If the similarity exceeds a set threshold (such as 0.8), it is determined that the relationship is redundant, indicating that the two edges are highly repeated in semantics.

[0129] When a redundant relationship edge is detected, the system performs a merging operation and updates the relationship weight. For example, the two relationship edges of "foldable tablet computer" and "high-end tablet computer" pointing to "high-performance processor" are merged, and the relationship weight is updated by combining the information of the two. If the processor performance of the "foldable tablet computer" is stronger, the relationship weight after merging will be adjusted accordingly, highlighting its advantage in the attribute of "equipped with high-performance processor". This operation avoids the knowledge graph becoming bloated and chaotic due to repeated information, just like organizing the warehouse by concentrating similar goods and labeling the priority, which facilitates subsequent quick search and use.

[0130] From the technical effect, the graph convolution operation and the redundancy detection mechanism improve the quality of the knowledge graph. In the e-commerce scenario, the structural complexity of the knowledge graph is reduced, and the retrieval efficiency is improved. When the user searches for "foldable tablet computer", the system can quickly and accurately present the core attributes and associated information of the goods. At the same time, semantic fusion makes the semantic expression of the knowledge graph more accurate, and the recommendation system based on the optimized graph improves the accuracy of recommending associated goods, effectively improving the user shopping experience and platform operation efficiency.

[0131] In step S107, a graph convolution network GCN containing a local structure enhancement contrast module is used to establish matching fine-grained labels for the semantic differentiated features of different entity combinations in the target knowledge graph. The fine-grained labels in the target knowledge graph are hierarchically constructed by combining the first label association matrix, the second label association matrix, and the third label association matrix, to obtain the cross-modal label tree corresponding to the target knowledge graph.

[0132] In the embodiments of the present application, the cross-modal label tree is a hierarchical semantic label structure containing multiple types of data. By modeling the entity relationships in the cross-modal knowledge graph through a graph neural network, a progressive label system from macro-concept to micro-attribute is formed. For example, in image-text joint analysis, the root node is "animal", the child nodes are refined into "cat" and "dog", and the leaf nodes describe specific features (such as "fur color" and "body size"), supporting fine-grained semantic retrieval and reasoning.

[0133] As an optional embodiment, in step S107, the target knowledge graph is calculated forwardly by the GCN, and the embedding vector of each entity fuses the local structure features of the entity itself; wherein the local structure features include the attribute features of the entity itself and the relationship edge features between entities. The embedding vectors are clustered by using the DBSCAN or K-means algorithm to obtain semantic cluster grouping. For a newly added entity, the probability distribution difference between the embedding vector of the newly added entity and the historical cluster center is calculated; if the probability distribution difference exceeds the preset drift condition, it is determined that the newly added node is a semantic drift, and the corresponding semantic cluster is reclassified to obtain dynamically updated semantic cluster grouping. The semantic features of the entities in the semantic cluster are aggregated by using the attention mechanism, high-frequency keywords are extracted as fine-grained labels, the compactness of the node embedding in the semantic cluster is calculated by using the confidence, and the division mode of the fine-grained labels is dynamically configured based on the compactness to obtain a set of fine-grained labels; wherein the higher the compactness, the higher the confidence, and the higher the semantic consistency of the node embedding in the semantic cluster. The first label association matrix is obtained by counting the co-occurrence times of each fine-grained label and other labels in the historical data. The shortest path length of the fine-grained label in the target knowledge graph is calculated, the topology structure of the target knowledge graph is converted into a numerical association degree by using Laplacian matrix decomposition combined with the relationship edge weight and the shortest path length of each fine-grained label, and the second label association matrix is obtained. The semantic similarity of the feature vectors corresponding to the fine-grained labels is calculated to obtain the third label association matrix for identifying the semantic association degree. Based on the first label association matrix, the second label association matrix and the third label association matrix, the fine-grained labels in the target knowledge graph are hierarchically constructed to obtain the cross-modal label tree.

[0134] Specifically, in the construction of the multi-modal knowledge graph, step S107 realizes the dynamic generation of the fine-grained label tree by using the graph convolution network and the clustering algorithm. Taking the intelligent power grid device monitoring scene as an example, when processing the power grid device knowledge graph, the embedding vector of each device node (such as a transformer or a circuit breaker) not only contains its own attributes (such as model and rated power), but also fuses the neighbor node and relationship edge features through the GCN. For example, the embedding of the “transformer” node aggregates the relationship edge information such as “connected to - substation” and “monitored to - temperature anomaly”, so that the node semantics are more complete. This is similar to comprehensively understanding the function of the device through the device manual (its own attributes) and the installation environment (neighbor relationship).

[0135] In the embodiments of the present application, the graph convolution network (GCN) generates an embedding vector that fuses local structure by hierarchically aggregating the neighborhood information of the nodes in the knowledge graph. The embedding of each entity not only contains its own attribute features (such as text description and type), but also integrates the relationship edge features (such as relationship type and connection strength) directly connected to it.

[0136] From the computational perspective, each layer of GCN operates in two steps: collecting feature information from the direct neighbors of each node, combining its own features with the aggregation results, and generating new embeddings through nonlinear transformation. After stacking multiple layers, high-order neighbor information is gradually integrated, forming a representation that contains global topological relationships. For example, in a knowledge graph, if entity A has a "cooperation" relationship with entity B, GCN will transfer the features of B to the embedding of A.

[0137] In this way, the embedding vector encodes both the semantic of the entity and its structural role in the graph, solving the problem of ignoring high-order neighbors that traditional methods ignore. For example, in a medical knowledge graph, a disease entity can express its complex semantics more accurately by aggregating the features of related symptoms and drug neighbors.

[0138] In the device fault detection scenario, the DBSCAN algorithm divides nodes with similar embedding vectors into clusters by setting the neighborhood radius and minimum sample size. For example, when detecting a group of devices with temperature and vibration feature vectors forming an irregular cluster in high-dimensional space, it may correspond to a new fault mode (such as "insulation aging and concurrent partial discharge"). This ability to discover clusters of arbitrary shapes allows the system to identify abnormal patterns without prior labels, just like a doctor discovering new disease types from patient symptom clusters. If the subtypes under the "device fault" category are known (such as "short circuit fault" and "overload fault"), K-means can be used to divide the node embeddings into K clusters. For example, clustering the embedding vectors of historical fault data into K=5 subtypes, each cluster center represents a typical fault mode. When a new fault occurs, its distance to the cluster center is calculated to quickly determine the fault type, achieving automatic classification of known faults, just like a library storing books by category for easy retrieval. Using the density clustering algorithm DBSCAN or the partitioning algorithm K-means, the embedding vectors generated by GCN are divided into semantic clusters. New entities are detected for semantic drift by comparing their probability distributions, triggering dynamic re-clustering. From a computational perspective, DBSCAN divides clusters based on density reachability, requiring core points to meet the minimum sample size (minPts) within the neighborhood radius (ε), suitable for non-convex distribution; K-means divides clusters by minimizing the sum of squared errors within clusters. In drift detection, the KL divergence or JS distance between the embedding of a new entity and the historical cluster center is calculated. If the difference exceeds the threshold (such as KL divergence > 0.3), it is determined to be a semantic drift, triggering local re-clustering. In this way, the knowledge graph adapts to dynamic evolution. For example, a new "blockchain medical" entity may deviate from the original "medical technology" cluster due to feature distribution, triggering re-clustering to generate a new cluster "digital medicine", avoiding semantic dilution of the old cluster.

[0139] Continuing the above example, when a new "smart meter communication interruption" node is added, if its embedding vector differs from the probability distribution of existing cluster centers by more than a threshold (e.g., KL divergence > 0.5), it is determined that there is semantic drift. At this point, the system triggers cluster reclassification, assigning the node to a new cluster (e.g., "communication failure") or adjusting the boundaries of existing clusters. For example, the original "sensor failure" cluster may be subdivided into "data acquisition failure" and "communication transmission failure" due to the addition of the new node, similar to adjusting the classification system in biological taxonomy based on the characteristics of new species. Based on the attention mechanism, high-frequency keywords within the semantic cluster are extracted as labels, and the label granularity is dynamically adjusted based on the compactness of the embedding within the cluster. From a computational perspective, attention weights are assigned to the key features of entities within the cluster (e.g., "CT scan" and "AI assistance" for the "image diagnosis" entity), and weighted labels are generated. In the compactness calculation, the inverse variance of the node embedding within the cluster is used as the compactness indicator. When the compactness is high (small variance), the confidence is high, and the coarse-grained label is directly output. When the compactness is low, the label needs to be subdivided (e.g., "low-dose CT" and "three-dimensional reconstruction" are extracted from "CT scan"). In this way, the semantic accuracy of the label is improved. For example, when the medical cluster has high compactness, the "tumor treatment" label is generated, and when it is loose, it is refined into "targeted therapy" and "immunotherapy", avoiding label ambiguity.

[0140] Continuing the above example, within the "temperature anomaly" semantic cluster, the attention mechanism focuses on high-frequency keywords such as "overheating", "poor heat dissipation", and "cooling system". These words are extracted as fine-grained labels. By calculating the compactness of the node embedding within the cluster (e.g., the average cosine similarity), the label division method is dynamically configured. If the compactness is high (e.g., 0.9), it indicates that the semantic consistency of the nodes is strong, and more detailed label division can be used (e.g., "heatsink blockage" and "fan failure"). If the compactness is low (e.g., 0.6), a more generalized label (e.g., "temperature anomaly") is used, similar to adjusting the level of detail in teaching content based on the knowledge mastery of students in a class.

[0141] In step S107, the relevance between labels is quantified through three groups of matrices. The first matrix is based on statistical association of historical co-occurrence frequency (e.g., "diabetes" and "insulin" have high co-occurrence frequency); the second matrix is based on the shortest path association of graph topology (e.g., the path weight between "doctor" and "hospital" is converted into association degree); the third matrix is based on semantic association of embedding similarity (e.g., the cosine similarity between "surgery" and "minimally invasive" is high). When constructing the second matrix, the graph Laplacian matrix decomposition is used to combine the shortest path length between nodes and edge weights into a numerical association degree. The shorter the path and the higher the edge weight, the larger the association value. When constructing the third matrix, the cosine similarity of the embedding of the corresponding entities of the labels is calculated to capture the underlying semantics. In this way, statistical, structural, and semantic information are integrated. For example, "heart stent" and "interventional therapy" are strongly associated due to high co-occurrence rate (first matrix), short path (second matrix), and similar embedding (third matrix).

[0142] In the above steps, further optionally, a shortest path association matrix based on graph topology is obtained in combination with the graph Laplacian normalization process is expressed as follows: Asp [ i , j ] = e − γ ⋅ D [ i , j ] ⋅ w avg （ e ij ） wherein, is the shortest path matrix, which is calculated by Dijkstra or Floyd-Warshall algorithm to calculate the shortest path between nodes (labels) in the graph, and the edge weight represents the relationship strength (such as the weight of the "doctor-hospital" edge reflecting the connection tightness). The label and the label co-occurrence times in historical data. is the embedding vector of the co-occurrence times of the label and the label in historical data. is a scaling factor that controls the decay rate of path length on the correlation degree. is the average weight of all edges from the label to the label . The indicator function maps the path length to the interval of 0 to 1, and the shorter the path, the greater the correlation value. Multiply the edge weight to strengthen the contribution of high-weight paths.

[0143] Integrate the three association matrices to generate a cross-modal label tree through weighted fusion (such as matrix weighted average) to realize the hierarchical organization of labels (such as "medical technology → image diagnosis → MRI"). After normalizing each matrix, assign weights (such as co-occurrence matrix weight 0.4, path matrix 0.3, semantic matrix 0.3), and fuse through matrix addition. Based on the fusion result, hierarchical clustering is performed to generate a tree structure: the root node is a coarse-grained category, and the leaf node is a subdivided label. The label tree takes into account multi-dimensional correlation and supports accurate retrieval and reasoning. For example, in a medical knowledge base, a user query "tumor treatment" can locate "chemotherapy side effect management" along the sub-tree, shortening the retrieval path.

[0144] The first matrix is obtained by counting the co-occurrence times (such as "short circuit" and "trip" frequently co-occurring), the second matrix is obtained by calculating the shortest path length between labels (such as the path length from "transformer fault" to "power failure accident"), and the third matrix is obtained by calculating the semantic similarity between labels (such as "overload" and "overcurrent" having similar semantics). Integrate the three matrices to hierarchically construct a cross-modal label tree from fine-grained labels. For example, the root node is "equipment failure", which is further divided into "electrical failure" and "mechanical failure", and "electrical failure" is further divided into "short circuit", "overload", etc., forming a hierarchical system similar to biological taxonomy, which is convenient for fault tracing and diagnosis.

[0145] In the smart grid scenario, the fault diagnosis accuracy is improved by step S107, the new fault discovery speed is accelerated, and the label tree hierarchical structure improves the efficiency of fault knowledge retrieval. At the same time, the semantic drift detection mechanism enables the system to adapt to the knowledge changes brought by device upgrades, such as the access of smart meters, automatically expanding communication fault classification, ensuring the timeliness and accuracy of the knowledge graph, and providing strong support for the intelligent operation and maintenance of the power system.

[0146] In the embodiments of the present application, the automatic construction of the cross-modal label tree is realized, the technical problem of poor semantic association caused by multi-modal data fragmentation is solved, the semantic description precision is improved, and the label generation efficiency is improved.

[0147] After introducing the method of the example embodiment of the present application, next, with reference to Figure 2 A cross-modal label generation system for fragmented data based on deep transfer learning is described for the example embodiment of the present application. The device includes: an acquisition module for acquiring original multi-modal data to be processed; a multi-branch deep feature extraction network is used to extract first high-dimensional feature vectors under different modalities from the original multi-modal data according to different modal data characteristics using different deep learning modes; a unification module is used to map the first high-dimensional feature vectors under different modalities into the same semantic space by a cross-modal contrast loss function, to realize multi-modal alignment and fusion processing of the first high-dimensional feature vectors, and to obtain second high-dimensional feature vectors with unified semantic metrics; a labeling module is used to label semantic labels corresponding to the second high-dimensional feature vectors based on fragmented data components in the original multi-modal data using a small sample transfer learning algorithm; an encoding module is used to encode the second high-dimensional feature vectors into multi-channel binary hash codes by calling different adaptive encoding strategies for different modal data combinations using a multi-channel hash encoder; an updating module is used to dynamically expand the binary hash codes and corresponding semantic labels into a historical knowledge graph using an incremental graph neural network IncGNN to obtain an updated target knowledge graph; a label module is used to establish matching fine-grained labels for semantic differentiated features of different entity combinations in the target knowledge graph using a graph convolutional network GCN containing a local structure enhanced contrast module, and to hierarchically construct fine-grained labels in the target knowledge graph to obtain a cross-modal label tree corresponding to the target knowledge graph in combination with a first label association matrix, a second label association matrix, and a third label association matrix. The above system can realize each step described in the method embodiments, and the specific implementation of each step is not repeated here.

[0148] After introducing the method and system of the exemplary embodiments of the present application, next, a terminal device of the exemplary embodiments of the present application is described. The terminal device can implement each step described in the method embodiments, and the specific implementation of each step is not repeated here. Refer to Figure 3 The computer readable storage medium of the exemplary embodiments of the present application is described, please refer to Figure 3 The computer readable storage medium shown is an optical disc 30, and a computer program (i.e. program product) is stored on the optical disc 30, which, when run by a processor, will implement each step described in the method embodiments. The specific implementation of each step is not repeated here.

[0149] It should be noted that the examples of the computer readable storage medium can also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other optical, magnetic storage medium, which will not be repeated here. The above described embodiments are only specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit it, the protection scope of the present application is not limited to this, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art within the technical scope disclosed by the present application, they can still modify or easily think of changes to the technical solutions described in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and all should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for cross-modal label generation of fragmented data based on deep transfer learning, characterized in that, The method comprises: acquiring original multi-modal data to be processed; using a multi-branch deep feature extraction network, different deep learning modes are used to extract first high-dimensional feature vectors under different modalities from the original multi-modal data according to different modal data characteristics; mapping the first high-dimensional feature vectors under different modalities into the same semantic space through a cross-modal contrast loss function, realizing multi-modal alignment and fusion processing of the first high-dimensional feature vectors, and obtaining second high-dimensional feature vectors with unified semantic metrics; using a small sample transfer learning algorithm, labeling semantic labels corresponding to the second high-dimensional feature vectors based on fragmented data components in the original multi-modal data; using a multi-channel hash encoder, different adaptive encoding strategies are called for different modal data combinations, and the second high-dimensional feature vectors are encoded into multi-channel binary hash codes; combining an incremental graph neural network IncGNN, the binary hash codes and the corresponding semantic labels are dynamically expanded into a historical knowledge graph to obtain an updated target knowledge graph, including: converting the binary hash codes corresponding to the new semantic labels into real value vectors through a hyperbolic tangent function; using the incremental update mechanism of the IncGNN, the converted real value vectors are aggregated with node embeddings with similar semantics in the historical entities of the historical knowledge graph to obtain new entities; the semantic distance between the new entities and the historical entities is calculated by cosine similarity to measure the closeness of the new entities and the historical entities in the feature space to obtain the semantic similarity between the new entities and the historical entities; if the semantic similarity exceeds a set threshold, a relationship edge is established between the new entities and the historical entities according to the semantic correlation of the cross-modal features, and the relationship weight between the new entities and the historical entities is set according to the Hamming similarity of the hash codes and the semantic similarity, to complete the update of the historical knowledge graph and obtain the target knowledge graph; using a graph convolution network GCN containing a local structure enhancement contrast module, matching fine-grained labels are established for the semantic differentiated features of different entity combinations in the target knowledge graph, and the fine-grained labels in the target knowledge graph are hierarchically constructed in combination with a first label association matrix, a second label association matrix and a third label association matrix, to obtain a cross-modal label tree corresponding to the target knowledge graph.

2. The method of claim 1, wherein the method is based on deep transfer learning. The multi-branch deep feature extraction network used to extract first high-dimensional feature vectors under different modalities from the original multi-modal data according to different modal data characteristics comprises: identifying the modalities to which each original data in the original multi-modal data belongs, and preprocessing the original data of different modalities; inputting the preprocessed original text sequence into a BERT and BiLSTM combined model, using a BERT pre-training model to extract the context semantic features of the original text sequence, and using BiLSTM to capture the bidirectional dependency relationship of fragmented data implied in the original text sequence through sequence modeling to obtain a text semantic vector under the text modality; The pre-processed original image matrix is extracted by a ResNet-50 network to obtain visual semantic vectors in the image mode, and a Transformer model is used to strengthen the extraction of global semantic features. The pre-processed original time series data is extracted by an LSTM autoencoder to obtain time series mode vectors in the image mode.

3. The method of claim 1, wherein the method is based on deep transfer learning. The first high-dimensional feature vectors in different modes are mapped into the same semantic space by the cross-modal contrast loss function, realizing multi-modal alignment and fusion processing of the first high-dimensional feature vectors, and obtaining the second high-dimensional feature vectors with unified semantic metrics, including: The first high-dimensional feature vectors are converted into unified feature dimensions by independent encoders, and the re-encoded first high-dimensional feature vectors are projected into a shared semantic space. Feature vectors in different modes are selected as anchor samples from the shared semantic space. Feature vectors with the same semantics as the anchor samples are selected as positive samples, and feature vectors with different semantics from the anchor samples are selected as negative samples, to obtain a cross-modal sample pair. A pre-trained CLIP model is used to generate a continuous similarity weight of the cross-modal sample pair, and a cross-modal contrast loss function is constructed based on the continuous similarity weight; the continuous similarity weight is used to reflect the semantic correlation strength between different first high-dimensional feature vectors in the cross-modal sample pair. The similarity distribution between the cross-modal sample pairs is calculated by using the cross-modal contrast loss function, a dynamic boundary threshold is set by a ternary fuzzy boundary constraint, and a similarity distribution heat map is constructed by referring to the similarity distribution and the dynamic boundary threshold, so as to adjust the relative positions between the cross-modal feature vectors in the shared semantic space, to pull closer the semantically similar samples and push away the semantically irrelevant samples. An association matrix of the cross-modal feature vectors in the shared semantic space is calculated by an attention cross mechanism. The cross-modal semantic correlation area in the association matrix is identified in combination with the similarity distribution heat map, realizing cross-modal semantic alignment. Learnable gating coefficients are used to weight and fuse the cross-modal associated feature vectors in the cross-modal semantic correlation area and the original first high-dimensional feature vectors, and a multi-granularity layer is used to mix and extract the weighted fusion results, to obtain the second high-dimensional feature vectors.

4. The method of claim 1, wherein the method is based on deep transfer learning. The small sample transfer learning algorithm is used to label the semantic labels corresponding to the second high-dimensional feature vectors based on the fragmented data components in the original multi-modal data, including: The fragmented data components in the original multi-modal data are identified, and the domain relevance indexes of the fragmented data components are calculated; the domain relevance indexes represent the semantic overlap degrees of the fragmented data components and the target domain. The fragmented data components in the original multi-modal data are identified, and the domain relevance indexes of the fragmented data components are calculated; the domain relevance indexes represent the semantic overlap degrees of the fragmented data components and the target domain. The general pre-training model is adaptively migrated to the target field of the fragmented data component through parameter fine-tuning and knowledge distillation, and a small sample labeling model in the target field is constructed; the parameter fine-tuning comprises adding a fragmented field adaptation Adapter to the top layer of the general pre-training model; the parameter sparsity of the fragmented field adaptation Adapter is dynamically adjusted based on the field correlation index, and the lower the correlation is, the higher the sparsity is; The second high-dimensional feature vector is semantically labeled by the small sample labeling model to obtain a semantic label corresponding to the second high-dimensional feature vector.

5. The method of claim 4, wherein the method is based on deep transfer learning. The general pre-training model is adaptively migrated to the target field of the fragmented data component through parameter fine-tuning and knowledge distillation, and a small sample labeling model in the target field is constructed, comprising: A fragmented field adaptation Adapter is added to the top layer of the general pre-training model, the bottom layer parameters are frozen, and the adaptation layer is fine-tuned; the fragmented field adaptation Adapter comprises a semantic completion sublayer for text fragments and a visual repair sublayer for image fragments, and the enabling conditions of each sublayer are obtained based on the modal type of the corresponding fragmented component; A teacher-student model is constructed; a plurality of different initialized teacher models generate integrated soft labels on a general data set, and the student model learns in a fragmented data set in the target field of the fragmented data component, and the output distribution of the plurality of teacher models is fitted synchronously through a KL divergence loss function; The output distribution of the teacher model is softened through a temperature parameter; wherein the temperature parameter is dynamically adjusted to balance the generalization and specificity of knowledge transfer when the student model is trained in the fragmented data set; when the sample size of the fragmented data set is less than a preset threshold, the temperature parameter is increased to enhance the smoothness of the soft label and reduce the risk of overfitting.

6. The method of claim 1, wherein the method is based on deep transfer learning. The second high-dimensional feature vector is encoded into a multi-channel binary hash code by using a multi-channel hash encoder and calling different adaptive encoding strategies for different modal data combinations, comprising: For a text-image combination channel, a deep hash network is used, the text semantic vector and the visual semantic vector are taken as joint inputs, and the shared feature pairs of the same dimension are mapped to the shared feature pairs of the same dimension through a full connection layer, and the shared feature pairs are mapped to hash codes through an activation function, and each bit identifies a semantic feature dimension; wherein, through an InfoNCE loss function, the shared feature pairs with similar semantics are constructed as hash codes with close Hamming distances, and the shared feature pairs with irrelevant semantics are constructed as hash codes with far Hamming distances; For a time-series text combination channel, a graph attention hash network is used, a time-series text heterogeneous association graph is constructed from the text semantic vector and the time-series mode vector, the correlation weights between the graph nodes in the time-series text heterogeneous association graph are calculated through a graph attention mechanism, the hash code is made to capture the dependency relationship between the time-series mode and the text semantics, and a hash code containing structural information is obtained; Binary hash codes with consistent semantics are generated through contrastive learning constraints; wherein, for the same semantic entity, the Hamming distances of the hash codes generated by different channels are less than a preset threshold.

7. The method of claim 1, wherein the method is based on deep transfer learning. The combination incremental graph neural network IncGNN dynamically expands the binary hash code and the corresponding semantic label into the historical knowledge graph to obtain an updated target knowledge graph, and further includes: The semantic information of the newly added entity is iteratively updated according to the embedding of the neighbor nodes and the relationship weight by using the graph convolution operation, so that the semantic information of the newly added entity and the neighbor entities are mutually fused, and the semantic association of the whole knowledge graph is ensured; Whether the newly added entity causes the appearance of a semantic similar relationship edge in the target historical graph is detected by calculating the structural similarity of each relationship edge; If the structural similarity exceeds a set structural similarity threshold, the redundant relationship edge is merged and the relationship edge weight is updated, so as to avoid the structure confusion of the target knowledge graph caused by repeated information and improve the knowledge retrieval efficiency.

8. The method of claim 1, wherein the method is based on deep transfer learning. The graph convolution network GCN containing a local structure enhancement comparison module is used to establish a matching fine-grained label for the semantic differentiated features of different entity combinations in the target knowledge graph, and the fine-grained labels in the target knowledge graph are hierarchically constructed by combining the first label association matrix, the second label association matrix and the third label association matrix, so as to obtain the cross-modal label tree corresponding to the target knowledge graph, including: The embedding vector of each entity is fused with the local structure features of the entity itself by performing forward calculation on the target knowledge graph by using the GCN; wherein the local structure features include the attribute features of the entity itself and the relationship edge features between entities; The embedding vectors are clustered by using the DBSCAN or K-means algorithm to obtain semantic cluster groups; For the newly added entity, the probability distribution difference between the embedding vector of the newly added entity and the historical cluster center is calculated; if the probability distribution difference exceeds a preset drift condition, the newly added node is determined as a semantic drift, and the corresponding semantic cluster reclassification is triggered to obtain dynamically updated semantic cluster groups; The semantic features of the entities in the semantic cluster are aggregated by using the attention mechanism, high-frequency keywords are extracted as fine-grained labels, the compactness of the node embedding in the semantic cluster is calculated by using the confidence, and the division mode of the fine-grained label is dynamically configured based on the compactness, so as to obtain a fine-grained label set; wherein the higher the compactness, the higher the confidence, and the higher the semantic consistency of the node embedding in the semantic cluster; The first label association matrix is obtained by counting the co-occurrence times of each fine-grained label and other labels in the historical data. The shortest path length of the fine-grained label in the target knowledge graph is calculated, the topological structure of the target knowledge graph is converted into a numerical correlation degree by using the graph Laplacian matrix decomposition and combining the relationship edge weight and the shortest path length of each fine-grained label, and the second label association matrix is obtained. The semantic similarity of the feature vectors corresponding to the fine-grained labels is calculated to obtain the third label association matrix for identifying the semantic correlation degree. The fine-grained labels in the target knowledge graph are hierarchically constructed based on the first label association matrix, the second label association matrix and the third label association matrix, and the cross-modal label tree is obtained. 9.A system for cross-modal label generation of fragmented data based on deep transfer learning, characterized in that, The system includes: The acquisition module is configured to acquire original multi-modal data to be processed; a multi-branch deep feature extraction network is adopted to extract first high-dimensional feature vectors under different modalities from the original multi-modal data by using different deep learning modes according to different modal data characteristics; The unification module is configured to map the first high-dimensional feature vectors under different modalities into the same semantic space by using a cross-modal contrast loss function, to realize multi-modal alignment and fusion processing of the first high-dimensional feature vectors, and to obtain second high-dimensional feature vectors with unified semantic metrics; The labeling module is configured to label semantic labels corresponding to the second high-dimensional feature vectors based on fragmented data components in the original multi-modal data by using a small sample transfer learning algorithm; The encoding module is configured to encode the second high-dimensional feature vectors into multi-channel binary hash codes by using a multi-channel hash encoder and calling different adaptive encoding strategies for different modal data combinations; The updating module is configured to dynamically expand the binary hash codes and corresponding semantic labels into a historical knowledge graph by using an incremental graph neural network (IncGNN) to obtain an updated target knowledge graph; The updating module is specifically configured to convert the binary hash codes corresponding to the new semantic labels into real value vectors by using a hyperbolic tangent function; aggregate the converted real value vectors and node embeddings with similar semantics in the historical entity of the historical knowledge graph by using an incremental update mechanism of the IncGNN, to obtain a new entity; calculate the semantic distance between the new entity and the historical entity in the feature space by using a cosine similarity, to measure the closeness of the new entity and the historical entity in the feature space, to obtain the semantic similarity between the new entity and the historical entity; if the semantic similarity exceeds a set threshold, establish a relationship edge between the new entity and the historical entity according to the semantic correlation relationship of the cross-modal features, and set the relationship weight between the new entity and the historical entity according to the Hamming similarity of the hash codes and the semantic similarity, to complete the updating of the historical knowledge graph and obtain the target knowledge graph; The label module is configured to establish matching fine-grained labels for semantic differentiated features of different entity combinations in the target knowledge graph by using a graph convolution network (GCN) containing a local structure enhanced contrast module, and hierarchically construct the fine-grained labels in the target knowledge graph by combining a first label association matrix, a second label association matrix and a third label association matrix, to obtain a cross-modal label tree corresponding to the target knowledge graph.

Images