Multi-modal large model collaborative learning method and device based on knowledge graph enhancement

Abstract

The invention relates to the field of artificial intelligence, discloses a multi-modal large model collaborative learning method and device based on knowledge graph enhancement, and aims to solve the problems of inconsistent representation, calculation redundancy, low efficiency of cross-modal association and difficulty in high-precision space-time alignment in multi-modal data fusion. The method comprises the following steps: acquiring preprocessed multi-modal data to generate a feature vector; constructing and embedding a knowledge graph into a semantic space; and a knowledge perception attention mechanism is utilized, so that the large model collaboratively fuses knowledge and feature vectors, and unified semantic representation is generated to execute downstream tasks. The device comprises a data acquisition module, a preprocessing module, a knowledge graph construction and updating module, a knowledge embedding module, a collaborative learning module and a task decoding module. According to the technical scheme, the defects that a traditional method is not uniform in representation, redundant in calculation, low in cross-modal association efficiency and limited in high-precision space-time alignment are overcome, the model interpretability is enhanced, and the method is also suitable for digital twinborn scenes of factories / production lines / stations / equipment / warehousing and safety ring management.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and more specifically to a method and apparatus for collaborative learning of multimodal large models based on knowledge graph enhancement. Background Technology

[0002] With the rapid development of artificial intelligence technology, data processing and knowledge discovery have become core drivers of innovation across various industries. In increasingly complex real-world application scenarios, effectively integrating, analyzing, and understanding heterogeneous information from different modalities to achieve more comprehensive and accurate situational awareness and decision support is a crucial direction for current technological research. Among these, multimodal data processing technology aims to improve machines' cognitive abilities and application efficiency by integrating features from multiple data sources such as images, text, voice, video, and point clouds to construct a unified semantic understanding framework.

[0003] Against this backdrop, large deep learning-based models have demonstrated unprecedented performance in processing specific single-modal data, and their superior feature learning and complex pattern recognition capabilities have significantly propelled the intelligentization process in many fields. However, when these large models need to work collaboratively to handle complex reasoning tasks involving the fusion of multimodal information, establishing efficient association mechanisms between different modalities, overcoming semantic differences and representation gaps between modalities, and achieving true collaborative learning have become key challenges facing the field of multimodal artificial intelligence.

[0004] Existing technologies for multimodal data fusion still have significant shortcomings, mainly due to two major bottlenecks: inconsistent representations and redundant computational resources. For example, while mainstream cascaded fusion architectures can achieve joint analysis of data such as visible light and synthetic aperture radar (SAR) data, they often consume up to 40% of computational resources during feature alignment, severely restricting real-time performance and scalability. Even worse, when faced with hyperspectral data, which typically contains hundreds of bands, and joint modeling tasks with 3D point clouds, traditional methods suffer from a lack of a unified cross-modal representation space, leading to a sharp decline in the efficiency of information association between different modalities, typically exceeding 60%. Furthermore, some recent attempts, such as platforms employing neural radiation field technology, also have accuracy limitations in spatiotemporal alignment, with errors potentially reaching the 1.5 pixel level, far from meeting the requirements of high-precision applications such as millimeter-level deformation monitoring. Therefore, there is an urgent need for a multimodal large-scale model collaborative learning scheme that can effectively address the aforementioned problems of inconsistent representations, computational redundancy, and insufficient alignment accuracy. Summary of the Invention

[0005] In view of this, the present invention provides a multimodal large model collaborative learning method and apparatus based on knowledge graph enhancement, aiming to solve the above-mentioned technical problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A knowledge graph-enhanced multimodal large model collaborative learning method includes: Acquire multimodal raw data, which includes perception data from at least two different modalities; The multimodal raw data is preprocessed to generate a sequence of modality-specific feature vectors. The preprocessing operations include data cleaning, denoising, calibration, and spatiotemporal registration. Construct and dynamically update a domain knowledge graph, which includes entities, relations and attributes related to the multimodal raw data, and represents knowledge in the form of triples; The knowledge of the domain knowledge graph is embedded into a semantic space compatible with the modality-specific feature vector sequence. The knowledge embedding is achieved by adjusting the parameters of the knowledge graph embedding model so that the knowledge embedding vector and the modality-specific feature vector have similar distribution characteristics. The multimodal large model is subjected to knowledge graph-enhanced collaborative learning. The collaborative learning uses a knowledge-aware attention mechanism to deeply fuse the knowledge embedding of the knowledge graph with the modality-specific feature vector sequence to generate a unified semantic representation. Based on the unified semantic representation, downstream task decoding is performed. The downstream task includes at least one of millimeter-level deformation monitoring, hyperspectral land cover classification, urban change detection, or natural language question answering, and the results of the downstream task are output.

[0007] Furthermore, the acquisition of multimodal raw data includes: Acquire visible light image data using a visible light camera; Synthetic aperture radar image data is acquired using a synthetic aperture radar sensor; Acquire hyperspectral image data using a hyperspectral imager; Acquire 3D point cloud data using a LiDAR scanner; The preprocessing operations for the multimodal raw data include: The visible light image data and synthetic aperture radar image data are subjected to timestamp synchronization, spatial registration, radiometric correction and geometric correction. The timestamp synchronization accuracy is at the microsecond level and the spatial registration error is controlled within 0.5 pixels. Band selection and dimensionality reduction are performed on the hyperspectral image data to extract key spectral features; and ground filtering and normal vector estimation are performed on the three-dimensional point cloud data to obtain accurate geometric structure information.

[0008] Furthermore, the generation of the modality-specific feature vector sequence includes: For the visible light image data and hyperspectral image data, spatial and spectral features are extracted using a deep convolutional neural network or a visual transformer network to generate visible light feature vector sequences and hyperspectral feature vector sequences. For the synthetic aperture radar (SAR) image data, texture, structure, and scattering features are extracted using a complex convolutional network or a SAR specialization transformer network to generate a SAR feature vector sequence; and For the three-dimensional point cloud data, local geometric features and global shape features are extracted using a point cloud processing network to generate a three-dimensional point cloud feature vector sequence.

[0009] Furthermore, the construction and dynamic updating of a domain knowledge graph includes: Extract entities, relations, and attributes from domain-related structured databases, unstructured text descriptions, and ontology; The entities, relationships, and attributes are embedded into a high-dimensional vector space; The knowledge graph is stored in a graph database; Implicit relationships in the knowledge graph are discovered using path-based reasoning, rule-based reasoning, or neural symbolic reasoning methods. The structure or entity attributes of the knowledge graph are adjusted using the new discoveries of the multimodal large model, thereby achieving the dynamic update.

[0010] Furthermore, the embedding of the knowledge from the domain knowledge graph into a semantic space compatible with the modality-specific feature vector sequence is specifically achieved by adjusting the parameters of the knowledge graph embedding model so that the knowledge embedding vector and the modality-specific feature vector have the same feature dimension and similar distribution characteristics.

[0011] Furthermore, the multimodal large model collaborative learning with knowledge graph enhancement includes: The entity embedding vectors and relation embedding vectors related to the current scene in the knowledge graph are used as queries or keys to be input into the multi-head self-attention mechanism; The multi-head self-attention mechanism enables knowledge-guided cross-modal attention computation. Based on the semantic association information provided by the knowledge graph, the knowledge-guided cross-modal attention computation dynamically adjusts the importance weights of different modal features and focuses on feature regions related to the concepts in the knowledge graph, thereby achieving semantic alignment of features.

[0012] Furthermore, the multimodal large model collaborative learning with knowledge graph enhancement also includes: Based on the unified semantic space provided by the knowledge graph, the features of the same semantic concepts identified in different modalities are associated and normalized through the corresponding entities in the knowledge graph to achieve implicit feature alignment. In the collaborative learning process, a collaborative loss function is adopted, which includes multimodal task loss, knowledge graph loss and collaborative consistency loss. The collaborative consistency loss is used to measure the consistency between multimodal features and knowledge embeddings, and is achieved through contrastive learning or metric learning.

[0013] Furthermore, the generation of unified semantic representations includes: The fusion features output by the collaborative learning module are aggregated through a global pooling layer or an additional transformer encoder to generate a unified representation vector with fixed dimensions and high semantic density. The downstream task decoding includes: For the millimeter-level deformation monitoring task, the unified semantic representation is decoded into a three-dimensional displacement vector or deformation field, with an output accuracy of 0.1 millimeters. For the hyperspectral land cover classification task, the unified semantic representation is decoded into pixel-level land cover category labels, and the corresponding confidence scores are output. For the urban change detection task, the unified semantic representation is decoded into bounding boxes of the changed regions and change types; For the natural language question answering task, the unified semantic representation is decoded into a natural language answer.

[0014] A knowledge graph-enhanced multimodal large model collaborative learning device includes: A multimodal data acquisition module is used to acquire multimodal raw data, which includes sensing data from at least two different modalities. The data preprocessing module is used to perform preprocessing operations on the multimodal raw data to generate a sequence of modality-specific feature vectors. The preprocessing operations include data cleaning, noise reduction, calibration, and spatiotemporal registration. The knowledge graph construction and update module is used to construct and dynamically update a domain knowledge graph, which includes entities, relations and attributes related to the multimodal original data and represents knowledge in the form of triples. The knowledge embedding module is used to embed the knowledge of the domain knowledge graph into a semantic space compatible with the modality-specific feature vector sequence. The knowledge embedding is achieved by adjusting the parameters of the knowledge graph embedding model so that the knowledge embedding vector and the modality-specific feature vector have similar distribution characteristics. The collaborative learning module is used to perform multimodal large model collaborative learning with knowledge graph enhancement. The collaborative learning uses a knowledge-aware attention mechanism to deeply fuse the knowledge embedding of the knowledge graph with the modality-specific feature vector sequence to generate a unified semantic representation. The task decoding module is used to perform downstream task decoding based on the unified semantic representation. The downstream task includes at least one of millimeter-level deformation monitoring, hyperspectral land cover classification, urban change detection, or natural language question answering, and outputs the results of the downstream task. The multimodal data acquisition module includes at least one of a visible light camera, a synthetic aperture radar sensor, a hyperspectral imager, and a lidar scanner. The data preprocessing module includes at least one of the following: timestamp synchronization unit, spatial registration unit, radiometric correction unit, geometric correction unit, band selection unit, dimension reduction unit, ground filtering unit, and normal vector estimation unit.

[0015] Furthermore, downstream tasks also include equipment health monitoring, production line cycle time, work-in-process visibility, safety and environmental compliance, and energy consumption analysis.

[0016] Compared to existing technologies, this invention provides a unified semantic framework by introducing a domain knowledge graph, guiding multimodal feature mapping to a consistent representation space, thus fundamentally solving the technical problem of inconsistent representations in traditional multimodal data fusion. It employs a knowledge-aware attention mechanism, integrating the structured semantic information of the knowledge graph into the feature fusion process of the multimodal large model, precisely focusing on task-related feature regions, reducing unnecessary feature alignment calculations, and lowering the computational resource consumption of the feature alignment process to less than 10%, significantly reducing computational redundancy. By combining the prior geographical and physical relationships of the knowledge graph with the fine feature extraction capabilities of the multimodal large model, high-precision spatiotemporal alignment is achieved, reducing the spatiotemporal alignment error to 0.1 pixels, meeting the stringent accuracy requirements of millimeter-level deformation monitoring. By integrating the semantic information of the knowledge graph into the large model, traceable semantic evidence is provided for the model's decision-making, significantly enhancing the model's interpretability. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the overall technical architecture of a multimodal large model collaborative learning method based on knowledge graph enhancement according to the present invention. Detailed Implementation

[0019] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] See appendix Figure 1 This embodiment provides a knowledge graph-enhanced multimodal large-scale model collaborative learning method to address the technical problems of representation inconsistency and computational redundancy in multimodal data fusion. This method aims to improve cross-modal association efficiency and reduce computational overhead during feature alignment by constructing a unified knowledge graph to achieve deep fusion and collaborative learning of multimodal information.

[0021] In its implementation, this method includes the following steps: S101, acquire and preprocess multimodal data.

[0022] As one embodiment of the present invention, acquiring and preprocessing multimodal data specifically includes: Multimodal raw data is acquired in parallel using multiple sensors, such as visible light sensors, synthetic aperture radar (SAR) sensors, hyperspectral sensors, and 3D point cloud sensors. Visible light sensors acquire two-dimensional image information of the target area; their data format is typically a pixel matrix with red, green, and blue channels, achieving spatial resolution at the decimeter level and temporal resolution at the second level depending on the acquisition frequency. SAR sensors acquire surface deformation information and penetration characteristics; their data is represented in complex form, including amplitude and phase information, with spatial resolution typically at the meter to centimeter level and strong penetration capability through clouds and fog. Hyperspectral sensors acquire spectral reflectance data from hundreds of narrow bands, forming a 3D data cube. Each pixel contains detailed spectral features for precise ground feature identification and material analysis; the number of bands is typically two to four hundred, with spectral resolution reaching the nanometer level. 3D point cloud sensors acquire spatially discrete point sets and their geometric attributes through laser scanning; each point contains 3D coordinates and reflectance intensity information for accurate 3D modeling and structural analysis, with a point density reaching hundreds of points per square meter.

[0023] After acquisition, the raw data is sent to the data preprocessing module. This module performs data cleaning, denoising, calibration, and spatiotemporal registration. Data cleaning includes imputing missing values, such as through linear interpolation or nearest neighbor interpolation; and detecting and removing outliers, such as using the statistically based three-sigma criterion or the local anomaly factor algorithm to identify and remove them. Denoising is performed according to the characteristics of different modalities of the data: Gaussian filtering or median filtering can be used to remove random noise from visible light images; speckle noise suppression algorithms, such as Lieberthal filtering or frost filtering, can be used for synthetic aperture radar data; minimum noise fraction transform or singular value decomposition can be used to remove banded noise from hyperspectral data; and statistical filtering or radius filtering can be used to remove outliers from 3D point cloud data.

[0024] Calibration operations aim to correct inherent sensor biases and environmental influences. Visible light data requires radiometric calibration and atmospheric correction, converting digital grayscale values ​​into surface reflectance. Synthetic Aperture Radar (SAR) data requires geometric and radiometric correction to eliminate radar side-looking projection distortion and convert it into backscattering coefficients. Hyperspectral data requires spectral correction and dark current compensation to ensure the accuracy of the spectral curves. 3D point cloud data requires attitude correction and ground point filtering to eliminate errors caused by platform motion and separate ground and non-ground points.

[0025] Spatiotemporal registration is a crucial step in multimodal data fusion. Transformation relationships between modalities are established by matching corresponding points or feature regions in different modalities, for example, using scale-invariant feature transformations or accelerated robust feature algorithms. For image modalities, affine or projective transformations are used to align images from different perspectives to a unified coordinate system. For hyperspectral and visible light data, geometric correction based on image registration is employed. For point cloud data, an iterative nearest-point algorithm is used for registration. After registration, all data is uniformly converted to a standard geographic coordinate system or a local Cartesian coordinate system, ensuring timestamp consistency. The data is integrated into multidimensional tensor form and stored in a multimodal data repository, awaiting further processing. For example, visible light images are stored as height, width, and channel tensors; synthetic aperture radar data is stored as height, width, real, and imaginary part tensors; hyperspectral data is stored as height, width, and band tensors; and 3D point cloud data is stored as a matrix containing the number of points and the number of dimensions.

[0026] S102, Constructing a knowledge graph based on multimodal data.

[0027] As one embodiment of the present invention, constructing a knowledge graph based on multimodal data specifically includes: Entities, relationships, and events are extracted from preprocessed multimodal data and structured into a knowledge graph. First, entity extraction is performed: identifying land cover categories (e.g., buildings, roads, vegetation, water bodies) from visible light images; identifying surface deformation areas (e.g., subsidence areas, uplift areas) from synthetic aperture radar data; identifying material composition entities (e.g., minerals, soil types) from hyperspectral data; and identifying structural components (e.g., roofs, walls, trees) from 3D point cloud data. Entity extraction can utilize deep learning models, such as named entity recognition models based on convolutional neural networks or recurrent neural networks. Extracted entities are assigned unique identifiers and stored as nodes in the knowledge graph.

[0028] Next, relation extraction is performed to identify semantic associations between entities. For example, in visible light images, the relation "a building is located next to a road" can be identified. In synthetic aperture radar data, the causal relationship "deformation areas cause building damage" can be identified. In hyperspectral data, the compositional relationship "a certain mineral constitutes a surface component" can be identified. In 3D point cloud data, the spatial relationship "a roof covers the main body of a building" can be identified. Relation extraction can employ methods based on pattern matching, machine learning, or deep learning, such as using graph neural networks or attention mechanism models to identify dependencies between entity pairs. The extracted binary or multi-relationships are stored as edges in a knowledge graph.

[0029] Next, event extraction is performed to identify complex events with temporal or spatial attributes from multimodal data. For example, by comparing synthetic aperture radar data at different time points, "surface deformation events" can be identified, which include attributes such as time, location, deformation magnitude, and possible causes. "Vegetation growth cycle events" can be identified by analyzing changes in hyperspectral data. "Building construction progress events" can be identified by integrating visible light and point cloud data. Event extraction requires combining temporal analysis with multimodal evidence fusion.

[0030] Finally, knowledge fusion and graph storage are performed. Entities, relations, and events extracted from different modalities are fused to eliminate ambiguity in entity referencing and redundancy in relations. For example, if visible light images and point cloud data both identify the same "building" entity, they are merged into a unique node in the knowledge graph. Ontology aligns entity and relation categories from different sources to ensure the structural consistency of the knowledge graph. Ultimately, a unified knowledge graph is constructed, representing knowledge internally in the form of triples (subject, predicate, object), and stored in a graph database, such as using an attribute graph or resource description framework model. Nodes and edges in the graph can be attached with attributes, such as entity location coordinates, recognition confidence, timestamps, and other metadata.

[0031] S103 utilizes knowledge graphs to enhance the representation of multimodal data.

[0032] As one embodiment of the present invention, enhancing multimodal data representation using knowledge graphs specifically includes: Deep learning models are used to integrate the semantic information of knowledge graphs into the feature representations of multimodal data, thereby enhancing the semantic richness and relevance of the representations. First, knowledge graph embedding is generated. Graph neural networks are used to learn from the knowledge graph, generating low-dimensional dense vector representations of each entity and relation within the knowledge graph—this is the knowledge graph embedding. For example, graph convolutional networks, graph attention networks, or knowledge representation learning models (such as TransE and RotatE) can be employed. These embedding vectors can capture the semantic similarity, hierarchical structure, and relational patterns between entities.

[0033] Next, multimodal feature encoding is performed. The preprocessed multimodal data are input into their respective encoders for feature extraction. Visible light data is processed using a convolutional neural network to extract visual features; synthetic aperture radar data is processed using a specially designed radar feature extraction network to extract texture and deformation features; hyperspectral data is processed using a 3D convolutional neural network or a spectral-spatial attention network to extract spectral and spatial features; and 3D point cloud data is processed using a point network or a graph neural network to extract geometric and topological features. These encoders map the raw data to a unified feature space, generating modality-specific feature vectors.

[0034] Next, graph-feature fusion is performed. The generated knowledge graph embeddings are fused with the multimodal feature vectors. One fusion strategy is to find the corresponding entity embedding in the knowledge graph for each region or entity in the multimodal features. Then, the knowledge graph embeddings are combined with the corresponding multimodal feature vectors through concatenation, element-wise addition, or attention weighting mechanisms. For example, for a building region in an image, its visual feature vector can be concatenated with the embedding vector of the "building" entity in the knowledge graph to form an enhanced feature vector.

[0035] Another fusion strategy is to design a cross-modal attention mechanism. This mechanism allows multimodal features to interact and be enhanced under the guidance of a knowledge graph. Relational information in the knowledge graph can serve as queries or keys for the attention mechanism, guiding features from different modalities to pay attention to each other. For example, when processing an image containing "river" and "bridge," the relationship "bridge spans river" in the knowledge graph can increase the association weight between these two entity features in the image.

[0036] Through the aforementioned fusion process, the representation of each multimodal data sample not only includes the physical features of its original modality but also incorporates rich semantic association information provided by the knowledge graph, such as geographical location, functional attributes, and interactions. This enhanced representation has higher semantic density and cross-modal consistency, providing higher-quality input for subsequent collaborative learning of large multimodal models.

[0037] The map-feature fusion process can be represented by the following formula: in, This indicates the enhanced representation after fusion. This represents the feature encoding vector of multimodal data. Representing the knowledge graph embedding vector, the Fusion function can be concatenation, element-wise addition, or weighted fusion based on attention mechanisms.

[0038] S104 performs multimodal large model collaborative learning.

[0039] As one embodiment of the present invention, multimodal large model collaborative learning specifically includes: The enhanced multimodal representations from the knowledge graph are input into a large multimodal model for end-to-end collaborative learning to achieve deep understanding and reasoning across modal information. First, a unified representation learning module is constructed. This module typically employs a transformer encoder-decoder architecture, capable of handling long sequences and capturing global dependencies. The enhanced multimodal representations are linearly projected into the high-dimensional embedding space of the large model, with positional encoding added to preserve spatiotemporal information. The encoder receives enhanced feature sequences from different modalities and captures complex dependencies within each modality through a multi-head self-attention mechanism. For example, for visible light features, the self-attention mechanism can capture spatial relationships between different regions in an image; for hyperspectral features, it can capture spectral relationships between different bands.

[0040] Secondly, cross-modal alignment learning is performed. Within the encoder, cross-modal attention layers facilitate the interaction and alignment of features from different modalities. Features from each modality can serve as queries, while features from other modalities act as keys and values, enabling the model to learn semantic correspondences between modalities. For example, visible light features can query synthetic aperture radar features to find complementary information when describing the same ground feature. This mechanism helps eliminate representational inconsistencies between modalities, mapping heterogeneous information to a shared semantic space.

[0041] Then, task-specific fine-tuning is performed. After pre-training on a general dataset, the large multimodal model needs to be fine-tuned for specific downstream tasks. For example, for a multimodal object detection task, the output layer of the large model can be connected to a detection head to predict the object's class and bounding box. For a multimodal scene understanding task, the large model can output semantic labels or descriptive text for the scene. During fine-tuning, the loss function is calculated using task-specific labeled data, and the model parameters are updated using an optimizer such as the Adam optimizer.

[0042] The core of collaborative learning lies in the ability of large models to simultaneously process and integrate information from multiple modalities. Guided by knowledge graphs, and through self-attention and cross-modal attention mechanisms, they dynamically adjust the weights of features from different modalities, thereby achieving a comprehensive understanding of complex scenes. This learning paradigm allows models to benefit from complementary information. For example, when visible light images are of poor quality, the penetration information from synthetic aperture radar can compensate for the deficiencies; when hyperspectral data provides detailed material information, point cloud data provides accurate 3D structures. By continuously learning and optimizing its internal knowledge representation and intermodal relationships, the model ultimately improves its performance on multimodal understanding and reasoning tasks.

[0043] The loss function for multimodal large model collaborative learning can be defined as: in, It is the total loss function. It is a cross-modal alignment loss, for example, a contrastive learning loss can be used to ensure that features of different modalities are similar in the shared space; These are task-specific losses, such as cross-entropy loss for classification tasks or mean squared error for regression tasks. It is a hyperparameter that balances alignment loss and task loss.

[0044] S105, output and apply the learning results.

[0045] As one embodiment of the present invention, outputting and applying the learning results specifically includes: The enhanced representations or predictions obtained from the collaborative learning of multimodal large models are decoded and adapted for use in various downstream applications. First, the results are decoded. After completing collaborative learning, the multimodal large model generates outputs specific to a particular task. For example, in object detection, the model might output bounding box coordinates, object category labels, and confidence scores. In scene understanding, the model might generate a natural language description of the input scene. In change detection, the model might output a mask map of the changed regions. The decoder is responsible for converting these abstract features or encodings into understandable and actionable results. For sequence generation tasks, the decoder can use an autoregressive approach to progressively generate text or label sequences.

[0046] Next, the application interface is encapsulated. The decoded results are encapsulated through a standardized application programming interface (API) to facilitate integration and invocation by different downstream systems or users. The interface design takes into account data format compatibility, communication protocol uniformity, and the need for real-time response. For example, a unified representation layer can be provided to standardize the semantic results of multimodal fusion (e.g., high-precision land cover classification maps, 3D deformation models, event reports).

[0047] Next, performance evaluation and feedback are conducted. The output results are evaluated quantitatively and qualitatively to verify the effectiveness and reliability of the method. Quantitative evaluation can be performed by calculating metrics such as precision, recall, F1 score, and mean precision. Qualitative evaluation involves expert interpretation or comparative analysis with existing technologies. The evaluation results can serve as feedback for further optimization of the model's training strategy or adjustment of the knowledge graph construction rules.

[0048] Finally, the applications are deployed. The learning results are applied to practical decision support systems or intelligent applications. For example, in smart city management, they can be used to monitor changes in urban infrastructure in real time and identify abnormal events. In natural resource monitoring, they can be used to accurately identify land cover types and assess the ecological environment. In disaster emergency response, they can be used to quickly assess the disaster situation and plan rescue routes. This application enables the system to provide more accurate and comprehensive information based on a deep understanding of multimodal data, thereby improving the scientific nature and real-time performance of decision-making. For example, in the field of millimeter-level deformation monitoring, large models can accurately identify and predict minute deformations based on fused hyperspectral and point cloud data, combined with regional geological background information in knowledge graphs. This significantly reduces the spatiotemporal alignment error of traditional methods from 1.5 pixels to the sub-pixel level, or even the millimeter level, meeting the requirements for high-precision monitoring.

[0049] This embodiment also provides a knowledge graph-enhanced multimodal large-scale model collaborative learning system, which is used to implement the aforementioned knowledge graph-enhanced multimodal large-scale model collaborative learning method. The system includes multiple functional units that work collaboratively to complete the acquisition of multimodal data, construction of the knowledge graph, feature enhancement, collaborative learning of the large model, and application of the results.

[0050] A knowledge graph-enhanced multimodal large-scale model collaborative learning system includes: The system includes a multimodal data acquisition unit, a data preprocessing unit, a knowledge graph construction unit, a knowledge graph enhancement and representation unit, a multimodal large-scale model collaborative learning unit, and a result output and application unit.

[0051] The multimodal data acquisition unit is used to acquire multimodal raw data in parallel using various sensors such as visible light sensors, synthetic aperture radar sensors, hyperspectral sensors, and 3D point cloud sensors, and then transmits the acquired raw data to the data preprocessing unit. This unit ensures the synchronous acquisition of data from different modalities, has high-frequency data acquisition capabilities, and is compatible with multiple data formats. Internally, it contains the drive and control modules for each sensor.

[0052] The data preprocessing unit, electrically connected to the multimodal data acquisition unit, is used to clean, denoise, calibrate, and spatiotemporally register the received raw multimodal data. This unit includes a data cleaning module, a denoising module, a calibration module, and a spatiotemporal registration module. It uses specialized algorithms to process noise and biases in various modal data and integrates the processed data into a unified format multidimensional tensor, storing it in an internal cache or shared storage, awaiting access from the knowledge graph construction unit.

[0053] The knowledge graph construction unit, electrically connected to the data preprocessing unit, is used to extract entities, relations, and events from preprocessed multimodal data and structure them into a knowledge graph. This unit includes an entity extraction module, a relation extraction module, an event extraction module, and a knowledge fusion module. It utilizes deep learning and natural language processing techniques to extract semantic information from multimodal data and integrates it into a graph database based on triples, maintaining the ontology and instances of the knowledge graph.

[0054] The knowledge graph enhancement representation unit, electrically connected to the data preprocessing unit and the knowledge graph construction unit, is used to enhance the feature representation of multimodal data by leveraging the semantic information of the knowledge graph. This unit includes a knowledge graph embedding generation module, a multimodal feature encoding module, and a graph-feature fusion module. This unit performs embedding learning on the knowledge graph using a graph neural network to generate low-dimensional vectors of entities and relationships. These embedded vectors are then fused with the multimodal feature encoding vectors through an attention mechanism or concatenation operation, thereby producing a semantically rich enhanced representation.

[0055] The multimodal large-scale model collaborative learning unit, electrically connected to the knowledge graph augmented representation unit, is used to input the knowledge graph-enhanced multimodal representations into the multimodal large-scale model for end-to-end collaborative learning. This unit includes a unified representation learning module, a cross-modal alignment learning module, and a task-specific fine-tuning module. It employs a transformer-based large-scale model, capturing intramodal dependencies and intermodal associations through multi-head self-attention mechanisms and cross-modal attention layers. After pre-training on a general dataset, it is fine-tuned for specific downstream tasks to achieve deep understanding and reasoning of cross-modal information. This unit utilizes a distributed computing framework and a high-performance graphics processing unit for model training and inference.

[0056] The results output and application unit is electrically connected to the multimodal large-scale model collaborative learning unit. It decodes and adapts the enhanced representations or predictions obtained from the multimodal large-scale model collaborative learning for use in various downstream application scenarios. This unit includes a results decoding module, an application interface encapsulation module, and a performance evaluation and feedback module. This unit is responsible for converting the abstract features or encodings output by the model into understandable practical results, such as object detection boxes, semantic segmentation maps, or natural language descriptions. It also provides standardized application programming interfaces to support seamless integration of these results into practical application systems such as smart city management, natural resource monitoring, or disaster emergency response. This unit is also responsible for continuous monitoring and evaluation of system performance.

[0057] The system in this embodiment achieves a complete link from low-level perception to high-level semantic understanding of multimodal data through the close collaboration of the aforementioned units. The data flow begins at the multimodal data acquisition unit, undergoes purification and alignment by the data preprocessing unit, and then forms structured knowledge in the knowledge graph construction unit. The knowledge graph enhancement and representation unit injects the semantics of the knowledge graph into the multimodal features, forming a more expressive joint representation. The multimodal large-model collaborative learning unit performs deep learning and inference on this basis, and finally, through the result output and application unit, provides high-value decision support and information services for practical applications.

[0058] In this embodiment, all communication between components and modules is conducted through predefined data interfaces and protocols, ensuring the integrity and consistency of data transmission. For example, the data preprocessing unit and the knowledge graph construction unit exchange data via shared memory or message queues. The knowledge graph enhancement representation unit and the multimodal large model collaborative learning unit transfer feature data through a standardized tensor interface. This modular design not only improves the system's scalability and maintainability but also allows for independent optimization and upgrades of each unit.

[0059] This application also provides a specific implementation method for industrial scenarios. By reusing the core technology framework of "multimodal data fusion - knowledge graph enhancement - collaborative learning" and combining it with the unique data attributes and business needs of industrial scenarios, it enables high-precision implementation of downstream tasks such as equipment health monitoring, production line cycle optimization, and safety and environmental compliance supervision.

[0060] 1. Industrial multimodal data source acquisition and preprocessing adaptation 1.1 Data Source Access and Synchronization The multimodal data acquisition module connects to typical data sources in industrial scenarios, and uses a "direct hardware connection + protocol adaptation" approach to achieve real-time data acquisition. The specific access method is as follows: Factory camera data: The workstation monitoring camera (1920×1080 resolution, 25fps frame rate) and the machine vision camera (line scan frame rate 5000Hz, used for workpiece size detection) are connected via gigabit industrial Ethernet to collect images of equipment operation status (such as the appearance of motor bearings and the position of materials on the conveyor belt) and visual feature data of workpieces. The data format is RGB / YUV raw image stream, which is transmitted to the data preprocessing module via RTSP protocol. PLC / SCADA Data: The system connects to the factory PLC (such as Siemens S7-1500) and SCADA system via the OPC UA protocol to collect real-time operating parameters of the equipment (such as motor speed 1500r / min, pump pressure 0.8MPa, conveyor belt speed 1.2m / s) and process status (such as valve opening and closing status, process execution progress). The data sampling frequency is 50ms / time, and the data is pushed to the data preprocessing module in JSON format. MES / ERP Data: Through RESTful API, connect to the factory's MES system (such as SAP MES) and ERP system to obtain production work order information (such as work order number, bill of materials, planned output), material flow records (such as material batch, warehousing time, workstation requisition records) and energy consumption statistics (such as daily power consumption of 1200kWh in the workshop, equipment energy consumption ratio). The data is updated every 1 minute and synchronized to the knowledge graph construction module in XML format. Industrial sensor data: Distributed sensors, including vibration sensors (sampling frequency 2kHz, measurement range 0-50g), temperature sensors (accuracy ±0.1℃, measurement range -40~125℃), gas sensors (detecting CO / NO2, resolution 0.1ppm), and weight sensors (accuracy 0.1kg), are connected through a LoRaWAN industrial gateway. The gateway collects data on equipment vibration spectrum, bearing temperature, workshop air quality, and material weight. The raw data is preprocessed by the gateway (removing spike noise) and then transmitted to the data preprocessing module via the MQTT protocol.

[0061] 1.2 Industrial Data Preprocessing Optimization The data preprocessing module addresses the characteristics of industrial data, such as high real-time requirements, complex noise sources (e.g., electromagnetic interference from motors), and large modal differences. It adds an industrial adaptation operation to the original "cleaning-denoising-calibration-spatiotemporal registration" process. Noise suppression: Wavelet threshold denoising is used on vibration sensor data (db4 wavelet basis, decomposition level 5) to remove high-frequency noise caused by motor electromagnetic interference; sliding window filtering (window size 5) is used on PLC data to smooth instantaneous fluctuations (such as parameter jumps caused by voltage drops). Spatiotemporal registration: Cross-modal registration is achieved through "timestamp alignment + device coordinate mapping"—using the PLC system timestamp (accuracy 1ms) as a benchmark, the time deviation of the camera and sensor is corrected (controlled within ±5ms); based on the device coordinate system of the factory digital twin model (such as workstation number + 3D coordinates), visual data (pixel coordinates) and sensor data (device ID associated coordinates) are mapped to a unified workshop coordinate system, and the registration error is controlled within ±1cm (meeting the accuracy requirements of industrial workstations); Feature vector generation: Targeting the modal characteristics of industrial data, the original feature extraction logic is reused and the network selection is optimized—a lightweight vision transformer (MobileViT) is used to extract the device appearance feature vector (dimension 512) from visual data; a temporal convolutional network (TCN) is used to extract the operating trend feature vector (dimension 256) from PLC time-series data; and a complex convolutional network is used to extract the vibration feature vector (dimension 512) from sensor spectrum data, ultimately generating a modal-specific feature vector sequence unique to industrial scenarios.

[0062] 2. Construction and dynamic updating of knowledge graphs in the industrial sector 2.1 Initialization of Industrial Knowledge Graph The knowledge graph construction and updating module uses "equipment-workstation-material-work order-safety and environmental indicators" as the core entities, extracts knowledge from structured / unstructured data and constructs triples. The specific process is as follows: Entity extraction: Extract "equipment entities" (e.g., motor M101, pump P203, attributes include equipment model, rated parameters, installation location), "workstation entities" (e.g., assembly workstation A01, inspection workstation B02, attributes include workstation function, associated equipment), and "work order entities" (e.g., work order W305, attributes include planned output, deadline); extract "fault entities" (e.g., motor bearing wear, attributes include fault characteristics, repair plan) from equipment manuals (unstructured text); extract "compliance indicator entities" (e.g., workshop CO concentration ≤30ppm, attributes include threshold range, warning level) from safety and environmental regulations. Relationship definition: Define the relationships between entities through rule-based reasoning to form triples, such as (motor M101, affiliated workstation, assembly workstation A01), (work order W305, associated material, material B702), (bearing wear, associated feature, vibration frequency 1200Hz), (CO concentration, compliance threshold, 30ppm). Knowledge graph storage: The knowledge graph is stored using the graph database Neo4j. Entity nodes contain unique IDs and attribute dictionaries, and relationship edges contain weights (e.g., the "workstation" relationship has a weight of 1.0, and the "related features" relationship has a weight of 0.8, representing the strength of the relationship). This supports efficient entity querying and relationship traversal.

[0063] 2.2 Dynamic Updates of Industrial Knowledge Graph Based on the characteristics of industrial scenarios, such as "real-time data changes and dynamic business adjustments," dynamic updates to the data graph are achieved through "event triggering + model inference." Triggering conditions: When the MES system pushes a new work order (such as the creation of work order W306), the PLC detects abnormal equipment parameters (such as the vibration of motor M101 exceeding the threshold), or the sensor detects fluctuations in safety and environmental indicators (such as the CO concentration rising to 28ppm), the spectrum update is triggered. Update logic: For new work orders, automatically add a "work order entity" and its associated relationships (e.g., (work order W306, associated workstation, inspection workstation B02)); for abnormal equipment parameters, add an "abnormal event entity" and its associated relationships through neural symbol reasoning (combined with the associated characteristics of the fault entity) (e.g., (motor M101, abnormality occurred, vibration exceeding standard event E401)); for fluctuations in safety and environmental indicators, update the real-time attributes of the "compliance indicator entity" (e.g., (CO concentration, current value, 28ppm)). Knowledge verification: After the update, the consistency of the knowledge graph is verified through the "domain rule base" (such as "the same workstation cannot be associated with two incomplete work orders at the same time"), and conflicting relationships (such as incorrectly associated work order-workstation relationships) are eliminated to ensure the accuracy of knowledge.

[0064] 3. Knowledge-enhanced collaborative learning in the execution of downstream industrial tasks 3.1 Industrial Knowledge Embedding and Feature Fusion The knowledge embedding module adjusts the parameters of the original knowledge embedding model to address the entity / relationship characteristics of industrial knowledge graphs, ensuring that the knowledge embedding vectors are compatible with the industrial modal feature vectors. Embedding model optimization: The TransR model (adapted to the "multi-attribute, multi-relationship" characteristics of industrial entities) is adopted. The embedding vector dimension of equipment entities and fault entities is adjusted to 512 (consistent with the dimension of visual / sensor feature vectors). By minimizing the loss of "entity-relationship-entity" triples (such as the distance loss of (motor M101, abnormal event, vibration exceeding standard event E401)), the distribution difference between knowledge embedding vectors and industrial feature vectors is controlled within ±10%. Knowledge-aware attention fusion: The collaborative learning module uses industrial knowledge embedding vectors (such as equipment rated parameter embedding and fault feature embedding) as "knowledge query keys" and inputs them into a multi-head self-attention mechanism. For example, in the equipment health monitoring task, the attention mechanism dynamically increases the weight of the vibration sensor feature vector (the weight ratio increases from 30% to 60%) based on the "rated vibration value of motor M101 (knowledge embedding)", focuses on the feature regions related to equipment faults, and achieves "knowledge-guided cross-modal feature alignment".

[0065] 3.2 Decoding and Output of Downstream Industrial Tasks Based on a unified semantic representation (the fusion of knowledge embedding and industrial multimodal features), the task decoding module achieves accurate decoding for downstream industrial tasks, with the following specific output: Equipment health monitoring: The unified semantic representation is input into the fully connected decoding layer and decoded into "equipment health score (0-100 points)" and "fault warning type" - for example, by fusing vibration characteristics and fault knowledge, the health score of motor M101 is decoded to be 65 points, and the warning is "slight bearing wear", with an output accuracy of 90% (fault false alarm rate ≤5%), and associated with "maintenance plan" in the knowledge graph (e.g. (bearing wear, recommended maintenance, replace 6205 model bearing)). Production line cycle time optimization: PLC timing characteristics (such as conveyor belt speed), MES work order characteristics (such as planned output) are integrated with the "workstation association relationship" in the knowledge graph and decoded into "workstation cycle time adjustment suggestions" - for example, the optimal cycle time of assembly workstation A01 is decoded as 30s / piece, and the control instruction "increase the conveyor belt speed to 1.5m / s" is output and pushed to the PLC system synchronously; Safety and environmental compliance supervision: Gas sensor data, workshop camera image features and "safety and environmental compliance indicators" knowledge are embedded and integrated to decode into "compliance status labels (compliance / warning / exceeding standards)" and "risk area location" - for example, decoding the CO concentration in the southwest corner of the workshop as 29ppm, outputting a "warning" label, locating the risk area (coordinates X=15m, Y=8m), and triggering an audible and visual alarm; Work-in-process visibility: Machine vision data (workpiece images), MES material data, and "material-work order association" are integrated and decoded into "work-in-process traceability information"—for example, by matching the appearance features of the workpiece, the batch (B702), work order attribution (W305), and process progress (60% completed) of work-in-process S801 are output and displayed visually on the digital twin interface.

[0066] 4. Implementation of Intranet / Dedicated Network Deployment To ensure the security and real-time performance of industrial data, this solution adopts a deployment architecture of "edge computing + internal network closed loop," which does not rely on public network transmission. Deployment Layers: In the factory intranet environment, it is divided into "Data Acquisition Layer - Edge Processing Layer - Cloud Collaboration Layer"—The data acquisition layer (workshop level) deploys industrial gateways, directly connecting cameras, sensors, and PLCs to achieve local data access; the edge processing layer (factory level) deploys GPU servers (such as NVIDIA A10) to run data preprocessing, knowledge graph construction, and collaborative learning inference (latency controlled within 50ms to meet industrial real-time requirements); the cloud collaboration layer (enterprise level) deploys a knowledge graph backup and task monitoring system, interacting with the edge layer only through an intranet dedicated line (such as a 10Gbps industrial Ethernet), without accessing the public network; Security Protection: Security is ensured by "data encryption + access control" - data at the acquisition layer is transmitted with AES-256 encryption; the edge layer and the cloud layer communicate through a VPN tunnel; knowledge graphs and model parameters are only accessible to authorized accounts (such as equipment maintenance personnel and production line administrators), and the scope of operation is controlled based on the RBAC permission model (such as maintenance personnel can only view equipment health data and cannot modify the graph). Compatibility and Adaptation: Supports commonly used industrial protocols (Profinet, Modbus, OPC UA) and system interfaces (MES / ERP standard API), eliminating the need to modify existing factory hardware and allowing direct reuse of existing data acquisition links, thus reducing deployment costs.

[0067] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0068] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimodal large-scale model collaborative learning method based on knowledge graph enhancement, characterized in that, include: Acquire multimodal raw data, which includes perception data from at least two different modalities; The multimodal raw data is preprocessed to generate a sequence of modality-specific feature vectors. The preprocessing operations include data cleaning, denoising, calibration, and spatiotemporal registration. Construct and dynamically update a domain knowledge graph, which includes entities, relations and attributes related to the multimodal raw data, and represents knowledge in the form of triples; The knowledge of the domain knowledge graph is embedded into a semantic space compatible with the modality-specific feature vector sequence. The knowledge embedding is achieved by adjusting the parameters of the knowledge graph embedding model so that the knowledge embedding vector and the modality-specific feature vector have similar distribution characteristics. The multimodal large model is subjected to knowledge graph-enhanced collaborative learning. The collaborative learning uses a knowledge-aware attention mechanism to deeply fuse the knowledge embedding of the knowledge graph with the modality-specific feature vector sequence to generate a unified semantic representation. Based on the unified semantic representation, downstream task decoding is performed. The downstream task includes at least one of millimeter-level deformation monitoring, hyperspectral land cover classification, urban change detection, or natural language question answering, and the results of the downstream task are output.

2. The multimodal large-scale collaborative learning method based on knowledge graph enhancement according to claim 1, characterized in that, The acquisition of multimodal raw data includes: Acquire visible light image data using a visible light camera; Synthetic aperture radar image data is acquired using a synthetic aperture radar sensor; Acquire hyperspectral image data using a hyperspectral imager; Acquire 3D point cloud data using a LiDAR scanner; The preprocessing operations for the multimodal raw data include: The visible light image data and synthetic aperture radar image data are subjected to timestamp synchronization, spatial registration, radiometric correction and geometric correction. The timestamp synchronization accuracy is at the microsecond level and the spatial registration error is controlled within 0.5 pixels. Band selection and dimensionality reduction are performed on the hyperspectral image data to extract key spectral features; and ground filtering and normal vector estimation are performed on the three-dimensional point cloud data to obtain accurate geometric structure information.

3. The multimodal large-scale collaborative learning method based on knowledge graph enhancement according to claim 1, characterized in that, The generation of the modality-specific feature vector sequence includes: For the visible light image data and hyperspectral image data, spatial and spectral features are extracted using a deep convolutional neural network or a visual transformer network to generate visible light feature vector sequences and hyperspectral feature vector sequences. For the synthetic aperture radar (SAR) image data, texture, structure, and scattering features are extracted using a complex convolutional network or a SAR specialization transformer network to generate a SAR feature vector sequence; and For the three-dimensional point cloud data, local geometric features and global shape features are extracted using a point cloud processing network to generate a three-dimensional point cloud feature vector sequence.

4. The multimodal large-scale collaborative learning method based on knowledge graph enhancement according to claim 1, characterized in that, The construction and dynamic updating of a domain knowledge graph includes: Extract entities, relations, and attributes from domain-related structured databases, unstructured text descriptions, and ontology; The entities, relationships, and attributes are embedded into a high-dimensional vector space; The knowledge graph is stored in a graph database; Implicit relationships in the knowledge graph are discovered using path-based reasoning, rule-based reasoning, or neural symbolic reasoning methods. The structure or entity attributes of the knowledge graph are adjusted using the new discoveries of the multimodal large model, thereby achieving the dynamic update.

5. The multimodal large-scale collaborative learning method based on knowledge graph enhancement according to claim 1, characterized in that, The process of embedding the knowledge from the domain knowledge graph into a semantic space compatible with the modality-specific feature vector sequence is specifically achieved by adjusting the parameters of the knowledge graph embedding model so that the knowledge embedding vector and the modality-specific feature vector have the same feature dimension and similar distribution characteristics.

6. The multimodal large model collaborative learning method based on knowledge graph enhancement according to claim 1, characterized in that, The multimodal large model collaborative learning method that performs knowledge graph enhancement includes: The entity embedding vectors and relation embedding vectors related to the current scene in the knowledge graph are used as queries or keys to be input into the multi-head self-attention mechanism; The multi-head self-attention mechanism enables knowledge-guided cross-modal attention computation. Based on the semantic association information provided by the knowledge graph, the knowledge-guided cross-modal attention computation dynamically adjusts the importance weights of different modal features and focuses on feature regions related to the concepts in the knowledge graph, thereby achieving semantic alignment of features.

7. The multimodal large-scale collaborative learning method based on knowledge graph enhancement according to claim 1, characterized in that, The multimodal large model collaborative learning method that performs knowledge graph enhancement also includes: Based on the unified semantic space provided by the knowledge graph, the features of the same semantic concepts identified in different modalities are associated and normalized through the corresponding entities in the knowledge graph to achieve implicit feature alignment. In the collaborative learning process, a collaborative loss function is adopted, which includes multimodal task loss, knowledge graph loss and collaborative consistency loss. The collaborative consistency loss is used to measure the consistency between multimodal features and knowledge embeddings, and is achieved through contrastive learning or metric learning.

8. The multimodal large model collaborative learning method based on knowledge graph enhancement according to claim 1, characterized in that, The generation of unified semantic representations includes: The fusion features output by the collaborative learning module are aggregated through a global pooling layer or an additional transformer encoder to generate a unified representation vector with fixed dimensions and high semantic density. The downstream task decoding includes: For the millimeter-level deformation monitoring task, the unified semantic representation is decoded into a three-dimensional displacement vector or deformation field, with an output accuracy of 0.1 millimeters. For the hyperspectral land cover classification task, the unified semantic representation is decoded into pixel-level land cover category labels, and the corresponding confidence scores are output. For the urban change detection task, the unified semantic representation is decoded into bounding boxes of the changed regions and change types; For the natural language question answering task, the unified semantic representation is decoded into a natural language answer.

9. A knowledge graph-enhanced multimodal large model collaborative learning device, applicable to any of the knowledge graph-enhanced multimodal large model collaborative learning methods described in claims 1-8, characterized in that, include: A multimodal data acquisition module is used to acquire multimodal raw data, which includes sensing data from at least two different modalities. The data preprocessing module is used to perform preprocessing operations on the multimodal raw data to generate a sequence of modality-specific feature vectors. The preprocessing operations include data cleaning, noise reduction, calibration, and spatiotemporal registration. The knowledge graph construction and update module is used to construct and dynamically update a domain knowledge graph, which includes entities, relations and attributes related to the multimodal original data and represents knowledge in the form of triples. The knowledge embedding module is used to embed the knowledge of the domain knowledge graph into a semantic space compatible with the modality-specific feature vector sequence. The knowledge embedding is achieved by adjusting the parameters of the knowledge graph embedding model so that the knowledge embedding vector and the modality-specific feature vector have similar distribution characteristics. The collaborative learning module is used to perform multimodal large model collaborative learning with knowledge graph enhancement. The collaborative learning uses a knowledge-aware attention mechanism to deeply fuse the knowledge embedding of the knowledge graph with the modality-specific feature vector sequence to generate a unified semantic representation. The task decoding module is used to perform downstream task decoding based on the unified semantic representation. The downstream task includes at least one of millimeter-level deformation monitoring, hyperspectral land cover classification, urban change detection, or natural language question answering, and outputs the results of the downstream task. The multimodal data acquisition module includes at least one of a visible light camera, a synthetic aperture radar sensor, a hyperspectral imager, and a lidar scanner. The data preprocessing module includes at least one of the following: timestamp synchronization unit, spatial registration unit, radiometric correction unit, geometric correction unit, band selection unit, dimension reduction unit, ground filtering unit, and normal vector estimation unit.

10. The multimodal large model collaborative learning method based on knowledge graph enhancement according to claim 9, characterized in that, Downstream tasks also include equipment health monitoring, production line cycle time, work-in-process visibility, safety and environmental compliance, and energy consumption analysis.

Images