Cross-modal collaborative methods based on temporal large models and knowledge-enhanced large models

By constructing a dual-path large model and a cross-modal dynamic alignment method, the problems of insufficient knowledge injection and single fusion method in cross-modal temporal tasks are solved, realizing multi-granularity feature extraction and semantic unification, and improving the generalization ability and prediction accuracy of the task.

CN123117344APending Publication Date: 2026-11-17大唐山东发电有限公司新能源分公司
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611332125.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-31
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing technologies lack multi-level and multi-granular knowledge injection strategies when dealing with complex cross-modal temporal tasks, and cannot effectively convey the deep semantic information of structured knowledge. Furthermore, the cross-modal fusion method is singular and the fusion weights are fixed, resulting in poor generalization ability.

Method used

A dual-path large model is constructed to extract multi-granularity temporal features and knowledge-enhanced semantic features respectively. Through cross-modal dynamic alignment and hierarchical collaborative fusion, a unified cross-modal fusion representation is generated to perform collaborative reasoning tasks.

Benefits of technology

It achieves semantic unification of temporal and knowledge modalities, improves the accuracy of cross-modal information interaction and the stability of the final output, ensures that the prediction results conform to domain knowledge rules, and provides interpretable knowledge traceability.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123117344A_ABST
    Figure CN123117344A_ABST
Patent Text Reader

Abstract

This invention relates to the field of artificial intelligence technology, specifically to a cross-modal collaborative method based on a temporal-series large model and a knowledge-enhanced large model, comprising the following steps: S1, dual-path large model construction; S2, cross-modal dynamic alignment; S3, hierarchical collaborative fusion; S4, collaborative inference execution. This invention establishes a temporal-knowledge joint representation space and employs a contrastive learning method to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs, effectively narrowing the feature space gap between temporal and knowledge modalities. Furthermore, S24 introduces dynamic alignment weights: these weights adaptively adjust the alignment strength based on the modal quality and semantic relevance of the input data, improving the accuracy of cross-modal information interaction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, specifically to a cross-modal collaborative method based on a time-series large model and a knowledge-enhanced large model. Background Technology

[0002] In recent years, with the rapid development of large model technology, temporal large models and knowledge-enhanced large models have made significant progress in temporal data analysis and knowledge-intensive tasks, respectively. Temporal large models, through large-scale pre-training, can capture complex temporal dependencies and perform excellently in tasks such as prediction and anomaly detection; knowledge-enhanced large models, by incorporating external knowledge bases, improve the interpretability and domain adaptability of the model. However, existing technologies still have the following problems when dealing with complex cross-modal temporal tasks: Existing knowledge augmentation methods mostly remain at the shallow knowledge embedding and splicing stage, directly splicing knowledge graph embeddings with text embeddings and inputting them into the model, lacking multi-level and multi-granular knowledge injection strategies; deep semantic information such as entity relationships and attribute constraints in structured knowledge cannot be effectively transferred to the temporal reasoning process, making it difficult to fully leverage the guiding role of domain knowledge in temporal analysis.

[0003] Existing cross-modal fusion methods mostly employ a single-level approach, either fusing only at the feature layer or only at the decision layer. This fails to achieve synergistic complementarity between two large-scale models throughout the entire process from feature extraction and semantic interaction to final decision-making. Furthermore, the fusion weights are typically pre-set fixed values, unable to adaptively adjust based on the modal quality and semantic relevance of the input data, resulting in poor generalization ability across different scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a cross-modal collaborative method based on a time-series large model and a knowledge-enhanced large model, so as to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A cross-modal collaborative method based on a time-series large model and a knowledge-enhanced large model includes the following steps: S1. Dual-path large model construction: Construct a time-series large model and a knowledge-enhanced large model respectively. The time-series large model is used to extract multi-granular time-series features from the input time-series data, and the knowledge-enhanced large model is used to integrate structured knowledge from an external knowledge base with textual semantic information. S2. Cross-modal dynamic alignment: Establish a temporal-knowledge joint representation space, and achieve semantic alignment between temporal features and knowledge features through contrastive learning to generate cross-modal aligned representations; S3, Hierarchical Collaborative Fusion: Perform multi-dimensional fusion operations at the feature layer, attention layer, and decision layer respectively to generate a unified cross-modal fusion representation; S4. Collaborative Reasoning Execution: Based on cross-modal fusion representation, a collaborative reasoning engine is built to execute downstream time series prediction, anomaly detection, and intelligent decision-making tasks, and output the final reasoning results.

[0006] Preferably, the specific process of step S1, the construction of the dual-path large model, is as follows: S11. Constructing a large-scale temporal model: Adopting the temporal Transformer architecture, multi-scale temporal convolution and positional encoding are introduced to encode the input temporal data and output multi-granularity temporal feature tensors; S12. Construct a knowledge-enhanced large model: Adopt an architecture that combines knowledge graph embedding and pre-trained language models to vectorize the entities, relations, and attributes of the external knowledge base and fuse them with text semantics to output knowledge-enhanced semantic features. S13. Independent pre-training: Perform independent pre-training on the time series large model and the knowledge-enhanced large model, and complete parameter initialization on the large-scale time series dataset and knowledge corpus respectively.

[0007] Preferably, the specific process of S11, constructing the large-scale time series model, is as follows: For input timing data Standardization processing is performed, among which For time step, For feature dimensions; A multi-scale temporal convolutional network is used to extract local features at different time scales and generate multi-scale feature maps. Relative position encoding and timestamp encoding are introduced to inject position information and absolute time information into each time step; Global temporal dependency modeling is performed using a multi-layer temporal Transformer encoder, outputting multi-granular temporal features. ,in For the hidden layer dimension; Global average pooling and max pooling are fused on the original temporal features to obtain dimension-aligned temporal features. The calculation formula is:

[0008] in, For pooling and merging weights, For global average pooling, This is for global max pooling.

[0009] Preferably, the specific process of S12, constructing the knowledge-enhanced large model, is as follows: Construct a domain knowledge graph containing entity sets. Relation sets triple set ; The TransH knowledge graph embedding algorithm is used to vectorize entities and relations, generating an entity embedding matrix. With relation embedding matrix ,in Embedding dimensions for knowledge; The knowledge embedding is concatenated with the text embedding of a pre-trained language model, and then fused through a knowledge-aware attention mechanism to output knowledge-enhanced semantic features. ,in For the number of knowledge entities; Entity-level attention weighting is applied to the original knowledge features to obtain dimension-aligned knowledge features. The calculation formula is:

[0010]

[0011]

[0012] in, For the first Attention weights for each knowledge entity Assign an importance score to the entity. It is a two-layer perceptron.

[0013] Preferably, in step S2, cross-modal dynamic alignment, the process of establishing a temporal-knowledge joint representation space and achieving semantic alignment between temporal features and knowledge features through contrastive learning is as follows: S21. Construct a cross-modal projection network: Construct a temporal projection head and a knowledge projection head respectively, and align the temporal features with dimensions. With knowledge characteristics Mapped to a unified joint representation space, the projected feature dimensions are all... The calculation formula is:

[0014]

[0015] in, , The projection weight matrix is... , For bias terms; S22. Constructing positive and negative sample pairs: Sample pairs are constructed based on timestamp-event association rules. The specific rules are as follows: Positive sample pairs: temporal features within the same time window and knowledge entity features corresponding to events occurring within that time window; Negative sample pairs: Temporal features within the same time window and knowledge entity features within other time windows, as well as randomly sampled irrelevant knowledge entity features, with a positive-to-negative sample ratio of 1:10; S23. Design a cross-modal contrastive learning loss function: maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The formula for calculating the cross-modal contrastive learning loss function is as follows:

[0016] in, The cosine similarity function is used. For temperature coefficient, For the first One time-series feature, These are the corresponding positive sample knowledge features. This represents negative sample knowledge features; S24. Introducing dynamic alignment weights: Adaptively adjusting the alignment strength based on the semantic relevance between modalities, using dynamic alignment weights. The calculation formula is:

[0017] in, It is the Sigmoid activation function. This is the weight matrix. The bias term is denoted by , and the variance of the time series features (reflecting the quality of the time series data) is denoted by . The entropy of the attention weights of knowledge entities (reflecting knowledge relevance); S25. Generate cross-modal aligned representations:

[0018]

[0019]

[0020] Achieve semantic unification between temporal sequences and knowledge modalities.

[0021] Preferably, in step S3, hierarchical collaborative fusion, the process of performing multi-dimensional fusion operations at the feature layer, attention layer, and decision layer is as follows: S31. Feature Layer Fusion: A gated fusion mechanism is adopted to adaptively learn the fusion weights of temporal features and knowledge features, generating a feature layer fusion representation. The calculation formula for the gated fusion mechanism is as follows:

[0022] in, It is the Sigmoid activation function. For learnable weight matrix, For bias terms, For element-wise multiplication, To merge the gate values; S32. Attention Layer Fusion: Design a cross-modal cross-attention mechanism to achieve bidirectional attention interaction between temporal features and knowledge features, generating an attention layer fusion representation. The calculation formula for the cross-modal cross-attention mechanism is as follows: , ,

[0023] , ,

[0024]

[0025] in, , , , , , , For query, key, and value projection matrix, The dimension of the key; S33. Decision-level fusion: An ensemble learning strategy is used to weight and fuse the preliminary prediction results of the dual-path model to generate the final decision-level output. The calculation formula is as follows: ,

[0026] in, , These are the task output headers for the time-series model and the knowledge model, respectively. The decision fusion weights are adaptively adjusted using the validation set.

[0027] S34. Generate a unified cross-modal fusion representation: Concatenate the feature layer fusion representation, attention layer fusion representation, and decision layer fusion representation, and generate the final unified cross-modal fusion representation through linear transformation. .

[0028] Preferably, in step S4, during collaborative reasoning execution, the process of constructing a collaborative reasoning engine based on cross-modal fusion representations and executing downstream tasks is as follows: S41. Preliminary prediction: Cross-modal fusion representation Input a task-specific output head network to generate preliminary prediction results; S42. Knowledge Constraint Regularization: A knowledge constraint regularization term based on first-order logic is introduced to ensure that the prediction results conform to the domain knowledge rules. The regularization loss calculation formula is as follows:

[0029] in, For the first The satisfaction function of a knowledge rule, which determines the prediction result when it satisfies the rule. ,otherwise ; S43. Uncertainty Estimation: The Monte Carlo dropout method is used to quantify the confidence level of the prediction results. Specifically, the dropout layer is kept on during the inference phase. The next forward propagation yields the set of prediction results. Calculate the predicted mean and variance:

[0030]

[0031] Confidence interval is ; S44. Knowledge Origin: Based on the gradient contribution method, locate the knowledge entity that contributes the most to the prediction. The contribution calculation formula is as follows:

[0032] in, For the task loss function, For the first Characteristics and contribution of each knowledge entity The larger the value, the greater the influence of the entity on the prediction result. The top 3 knowledge entities with the highest contribution are output as the source tracing results. S45. Multi-task joint optimization: The total loss function is a weighted sum of the contrastive learning loss, task loss, and knowledge constraint regularization loss.

[0033] in, , To reduce the weights, the AdamW optimizer was used for training with a learning rate of 1e-4, a weight decay of 1e-5, 100 iterations, and a batch size of 32. S46. Output the final inference result: including the predicted value. Confidence interval Explainable knowledge traceability information (the top 3 knowledge entities in terms of contribution and their contribution values).

[0034] Cross-modal collaborative systems based on temporal large-scale models and knowledge-enhanced large-scale models include: The dual-path large model construction module is used to build a temporal large model and a knowledge-enhanced large model respectively. The temporal large model is used to extract multi-granular temporal features from the input temporal data, and the knowledge-enhanced large model is used to integrate structured knowledge from an external knowledge base with textual semantic information. The cross-modal dynamic alignment module is used to establish a temporal-knowledge joint representation space. It achieves semantic alignment between temporal features and knowledge features through contrastive learning, and generates cross-modal aligned representations. The hierarchical collaborative fusion module is used to perform multi-dimensional fusion operations at the feature layer, attention layer, and decision layer respectively, to generate a unified cross-modal fusion representation; The hierarchical collaborative fusion module includes: The feature layer fusion unit is used to adopt a gated fusion mechanism to adaptively learn the fusion weights of temporal features and knowledge features, and generate a feature layer fusion representation. Attention layer fusion unit is used to design cross-modal cross-attention mechanism to realize bidirectional attention interaction between temporal features and knowledge features, and generate attention layer fusion representation; The decision-level fusion unit is used to employ an ensemble learning strategy to weight and fuse the preliminary prediction results of the dual-path model to generate the final decision-level output. The collaborative reasoning execution module is used to build a collaborative reasoning engine based on cross-modal fusion representation, execute downstream time series prediction, anomaly detection, and intelligent decision-making tasks, and output the final reasoning results. The collaborative reasoning execution module includes: The task output head unit is used to fuse cross-modal representations into a task-specific output head network to generate preliminary prediction results. The knowledge constraint unit is used to introduce knowledge constraint regularization to ensure that the prediction results conform to the domain knowledge rules. Uncertainty estimation unit, used to perform uncertainty estimation and quantify the confidence level of the prediction results; The result output unit is used to output the final inference result, including the predicted value, confidence interval, and interpretability knowledge tracing information.

[0035] Preferably, the dual-path large model construction module includes: The temporal large model unit adopts the temporal Transformer architecture, introduces multi-scale temporal convolution and positional encoding to encode the input temporal data and output multi-granularity temporal feature tensors; The knowledge-enhanced large model unit adopts an architecture that combines knowledge graph embedding with a pre-trained language model. It vectorizes the entities, relations, and attributes of the external knowledge base and integrates them with text semantics to output knowledge-enhanced semantic features. The pre-training unit is used to independently pre-train the large-scale temporal model and the large-scale knowledge-enhanced model, and to initialize the parameters on the large-scale temporal dataset and the knowledge corpus, respectively.

[0036] Preferably, the cross-modal dynamic alignment module includes: The joint representation space construction unit is used to construct a cross-modal projection network, which maps temporal features and knowledge features to a unified joint representation space respectively. The contrastive learning training unit is used to design a cross-modal contrastive learning loss function to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. Dynamic alignment weight unit is used to introduce dynamic alignment weights and adaptively adjust the alignment strength according to the semantic relevance between modalities; Alignment representation output unit is used to generate cross-modal aligned representations to achieve semantic unification of temporal and knowledge modalities.

[0037] Compared with the prior art, the beneficial effects of the present invention are: This invention constructs a dual-path large model to extract multi-granularity temporal features and knowledge-enhanced semantic features, fully leveraging the complementary advantages of the temporal large model in temporal dependency modeling and the knowledge-enhanced large model in semantic understanding. It establishes a temporal-knowledge joint representation space and uses a contrastive learning method to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs, effectively narrowing the feature space gap between temporal and knowledge modalities. At the same time, the introduction of dynamic alignment weights can adaptively adjust the alignment strength according to the modal quality and semantic relevance of the input data, improving the accuracy of cross-modal information interaction.

[0038] This invention performs multi-dimensional fusion operations at the feature layer, attention layer, and decision layer, achieving end-to-end collaboration from feature extraction and semantic interaction to final decision-making. The feature layer employs gated fusion and adaptive learning fusion weights, which can dynamically adjust the contribution ratio of temporal and knowledge features according to different inputs. The attention layer achieves bidirectional information interaction through cross-modal cross-attention, enabling temporal features to focus on relevant knowledge entities, and knowledge features to focus on corresponding temporal patterns. The decision layer uses an ensemble learning strategy to fuse the prediction results of the dual-path model, further improving the stability and accuracy of the final output.

[0039] This invention introduces a knowledge constraint regularization term to ensure that the prediction results conform to the domain knowledge rules and avoid producing erroneous results that violate common sense. At the same time, it performs knowledge tracing, which can output the knowledge entities that contribute the most to the prediction, thus realizing the interpretability of the reasoning process. In addition, it quantifies the confidence of the prediction results through uncertainty estimation, providing a more reliable basis for decision-making.

[0040] This invention employs a dual-path large model that can be independently pre-trained and updated. The cross-modal alignment and fusion module can be flexibly adjusted according to different tasks. The system is applicable to various cross-modal time series tasks such as time series prediction, anomaly detection, and intelligent decision-making, and has broad application prospects. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the process steps of the present invention; Figure 2 This is a schematic diagram of the system architecture of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0043] Please see Figure 1 A cross-modal collaborative method based on a time-series large model and a knowledge-enhanced large model includes the following steps: S1. Dual-path large model construction: A time-series large model and a knowledge-enhanced large model are constructed separately. The time-series large model is used to extract multi-granularity time-series features from the input time-series data, while the knowledge-enhanced large model is used to fuse structured knowledge from an external knowledge base with textual semantic information. The specific process is as follows: S11. Constructing a large-scale temporal model: The specific process of using a temporal Transformer architecture, introducing multi-scale temporal convolution and positional encoding to encode the input temporal data and output multi-granularity temporal feature tensors is as follows: For input timing data Standardization processing is performed, among which For time step, For feature dimensions; A multi-scale temporal convolutional network is used to extract local features at different time scales and generate multi-scale feature maps. Relative position encoding and timestamp encoding are introduced to inject position information and absolute time information into each time step; Global temporal dependency modeling is performed using a multi-layer temporal Transformer encoder, outputting multi-granular temporal features. ,in For the hidden layer dimension; Global average pooling and max pooling are fused on the original temporal features to obtain dimension-aligned temporal features. The calculation formula is:

[0044] in, For pooling and merging weights, For global average pooling, This is for global max pooling.

[0045] S12. Constructing a large-scale knowledge-enhanced model: This involves employing an architecture combining knowledge graph embedding and a pre-trained language model to vectorize the entities, relationships, and attributes of the external knowledge base and fuse them with textual semantics to output knowledge-enhanced semantic features. The specific process is as follows: Construct a domain knowledge graph containing entity sets. Relation sets triple set ; The TransH knowledge graph embedding algorithm is used to vectorize entities and relations, generating an entity embedding matrix. With relation embedding matrix ,in Embedding dimensions for knowledge; The knowledge embedding is concatenated with the text embedding of a pre-trained language model, and then fused through a knowledge-aware attention mechanism to output knowledge-enhanced semantic features. ,in For the number of knowledge entities; Entity-level attention weighting is applied to the original knowledge features to obtain dimension-aligned knowledge features. The calculation formula is:

[0046]

[0047]

[0048] in, For the first Attention weights for each knowledge entity Assign an importance score to the entity. It is a two-layer perceptron.

[0049] S13. Independent pre-training: Perform independent pre-training on the time series large model and the knowledge-enhanced large model, and complete parameter initialization on the large-scale time series dataset and knowledge corpus respectively.

[0050] S2. Cross-modal dynamic alignment: Establish a temporal-knowledge joint representation space, and achieve semantic alignment between temporal features and knowledge features through contrastive learning. The specific process of generating cross-modal aligned representations is as follows: S21. Construct a cross-modal projection network: Construct a temporal projection head and a knowledge projection head respectively, and align the temporal features with dimensions. With knowledge characteristics Mapped to a unified joint representation space, the projected feature dimensions are all... The calculation formula is:

[0051]

[0052] in, , The projection weight matrix is... , For bias terms; S22. Constructing positive and negative sample pairs: Sample pairs are constructed based on timestamp-event association rules. The specific rules are as follows: Positive sample pairs: temporal features within the same time window and knowledge entity features corresponding to events occurring within that time window; Negative sample pairs: Temporal features within the same time window and knowledge entity features within other time windows, as well as randomly sampled irrelevant knowledge entity features, with a positive-to-negative sample ratio of 1:10; S23. Design a cross-modal contrastive learning loss function: maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The formula for calculating the cross-modal contrastive learning loss function is as follows:

[0053] in, The cosine similarity function is used. For temperature coefficient, For the first One time-series feature, These are the corresponding positive sample knowledge features. This represents negative sample knowledge features; S24. Introducing dynamic alignment weights: Adaptively adjusting the alignment strength based on the semantic relevance between modalities, using dynamic alignment weights. The calculation formula is:

[0054] in, It is the Sigmoid activation function. This is the weight matrix. The bias term is denoted by , and the variance of the time series features (reflecting the quality of the time series data) is denoted by . The entropy of the attention weights of knowledge entities (reflecting knowledge relevance); S25. Generate cross-modal aligned representations:

[0055]

[0056]

[0057] Achieve semantic unification between temporal sequences and knowledge modalities.

[0058] S3. Hierarchical Collaborative Fusion: The specific process of performing multi-dimensional fusion operations at the feature layer, attention layer, and decision layer to generate a unified cross-modal fusion representation is as follows: S31. Feature Layer Fusion: A gated fusion mechanism is adopted to adaptively learn the fusion weights of temporal features and knowledge features, generating a feature layer fusion representation. The calculation formula for the gated fusion mechanism is as follows:

[0059] in, It is the Sigmoid activation function. For learnable weight matrix, For bias terms, For element-wise multiplication, To merge the gate values; S32. Attention Layer Fusion: Design a cross-modal cross-attention mechanism to achieve bidirectional attention interaction between temporal features and knowledge features, generating an attention layer fusion representation. The calculation formula for the cross-modal cross-attention mechanism is as follows: , ,

[0060] , ,

[0061]

[0062] in, , , , , , , For query, key, and value projection matrix, The dimension of the key; S33. Decision-level fusion: An ensemble learning strategy is used to weight and fuse the preliminary prediction results of the dual-path model to generate the final decision-level output. The calculation formula is as follows: ,

[0063] in, , These are the task output headers for the time-series model and the knowledge model, respectively. The decision fusion weights are adaptively adjusted using the validation set.

[0064] S34. Generate a unified cross-modal fusion representation: Concatenate the feature layer fusion representation, attention layer fusion representation, and decision layer fusion representation, and generate the final unified cross-modal fusion representation through linear transformation. .

[0065] S4. Collaborative Reasoning Execution: The specific process of constructing a collaborative reasoning engine based on cross-modal fusion representation, executing downstream time series prediction, anomaly detection, and intelligent decision-making tasks, and outputting the final reasoning result is as follows: S41. Preliminary prediction: Cross-modal fusion representation Input a task-specific output head network to generate preliminary prediction results; S42. Knowledge Constraint Regularization: A knowledge constraint regularization term based on first-order logic is introduced to ensure that the prediction results conform to the domain knowledge rules. The regularization loss calculation formula is as follows:

[0066] in, For the first The satisfaction function of a knowledge rule, which determines the prediction result when it satisfies the rule. ,otherwise ; S43. Uncertainty Estimation: The Monte Carlo dropout method is used to quantify the confidence level of the prediction results. Specifically, the dropout layer is kept on during the inference phase. The next forward propagation yields the set of prediction results. Calculate the predicted mean and variance:

[0067]

[0068] Confidence interval is ; S44. Knowledge Origin: Based on the gradient contribution method, locate the knowledge entity that contributes the most to the prediction. The contribution calculation formula is as follows:

[0069] in, For the task loss function, For the first Characteristics and contribution of each knowledge entity The larger the value, the greater the influence of the entity on the prediction result. The top 3 knowledge entities with the highest contribution are output as the source tracing results. S45. Multi-task joint optimization: The total loss function is a weighted sum of the contrastive learning loss, task loss, and knowledge constraint regularization loss.

[0070] in, , To reduce the weights, the AdamW optimizer was used for training with a learning rate of 1e-4, a weight decay of 1e-5, 100 iterations, and a batch size of 32. S46. Output the final inference result: including the predicted value. Confidence interval Explainable knowledge traceability information (the top 3 knowledge entities in terms of contribution and their contribution values).

[0071] like Figure 2 As shown, the cross-modal collaborative system based on a temporal large model and a knowledge-enhanced large model includes: The dual-path large model construction module is used to construct a temporal-series large model and a knowledge-enhanced large model respectively. The temporal-series large model is used to extract multi-granular temporal features from the input temporal-series data, while the knowledge-enhanced large model is used to fuse structured knowledge from an external knowledge base with textual semantic information, including: The temporal large model unit adopts the temporal Transformer architecture, introduces multi-scale temporal convolution and positional encoding to encode the input temporal data and output multi-granularity temporal feature tensors; The knowledge-enhanced large model unit adopts an architecture that combines knowledge graph embedding with a pre-trained language model. It vectorizes the entities, relations, and attributes of the external knowledge base and integrates them with text semantics to output knowledge-enhanced semantic features. The pre-training unit is used to independently pre-train the large-scale temporal model and the large-scale knowledge-enhanced model, and to initialize the parameters on the large-scale temporal dataset and the knowledge corpus, respectively.

[0072] The cross-modal dynamic alignment module is used to establish a temporal-knowledge joint representation space. Through contrastive learning, it achieves semantic alignment between temporal features and knowledge features, generating cross-modal aligned representations, including: The joint representation space construction unit is used to construct a cross-modal projection network, which maps temporal features and knowledge features to a unified joint representation space respectively. The contrastive learning training unit is used to design a cross-modal contrastive learning loss function to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. Dynamic alignment weight unit is used to introduce dynamic alignment weights and adaptively adjust the alignment strength according to the semantic relevance between modalities; Alignment representation output unit is used to generate cross-modal aligned representations to achieve semantic unification of temporal and knowledge modalities.

[0073] The hierarchical collaborative fusion module is used to perform multi-dimensional fusion operations at the feature layer, attention layer, and decision layer respectively, to generate a unified cross-modal fusion representation; The hierarchical collaborative fusion module includes: The feature layer fusion unit is used to adopt a gated fusion mechanism to adaptively learn the fusion weights of temporal features and knowledge features, and generate a feature layer fusion representation. Attention layer fusion unit is used to design cross-modal cross-attention mechanism to realize bidirectional attention interaction between temporal features and knowledge features, and generate attention layer fusion representation; The decision-level fusion unit is used to employ an ensemble learning strategy to weight and fuse the preliminary prediction results of the dual-path model to generate the final decision-level output.

[0074] The collaborative reasoning execution module is used to build a collaborative reasoning engine based on cross-modal fusion representations, execute downstream time series prediction, anomaly detection, and intelligent decision-making tasks, and output the final reasoning results, including: The task output head unit is used to fuse cross-modal representations into a task-specific output head network to generate preliminary prediction results. The knowledge constraint unit is used to introduce knowledge constraint regularization to ensure that the prediction results conform to the domain knowledge rules. Uncertainty estimation unit, used to perform uncertainty estimation and quantify the confidence level of the prediction results; The result output unit is used to output the final inference result, including the predicted value, confidence interval, and interpretability knowledge tracing information.

[0075] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A cross-modal collaborative method based on a time-series large model and a knowledge-enhanced large model, characterized in that, Includes the following steps: S1. Dual-path large model construction: Construct a time-series large model and a knowledge-enhanced large model respectively. The time-series large model is used to extract multi-granular time-series features from the input time-series data, and the knowledge-enhanced large model is used to integrate structured knowledge from an external knowledge base with textual semantic information. S2. Cross-modal dynamic alignment: Establish a temporal-knowledge joint representation space, and achieve semantic alignment between temporal features and knowledge features through contrastive learning to generate cross-modal aligned representations; S3, Hierarchical Collaborative Fusion: Perform multi-dimensional fusion operations at the feature layer, attention layer, and decision layer respectively to generate a unified cross-modal fusion representation; S4. Collaborative Reasoning Execution: Based on cross-modal fusion representation, a collaborative reasoning engine is built to execute downstream time series prediction, anomaly detection, and intelligent decision-making tasks, and output the final reasoning results.

2. The cross-modal collaborative method based on a time-series large model and a knowledge-enhanced large model according to claim 1, characterized in that, The specific process of step S1, the construction of the dual-path large model, is as follows: S11. Constructing a large-scale temporal model: Adopting the temporal Transformer architecture, multi-scale temporal convolution and positional encoding are introduced to encode the input temporal data and output multi-granularity temporal feature tensors; S12. Construct a knowledge-enhanced large model: Adopt an architecture that combines knowledge graph embedding and pre-trained language models to vectorize the entities, relations, and attributes of the external knowledge base and fuse them with text semantics to output knowledge-enhanced semantic features. S13. Independent pre-training: Perform independent pre-training on the time series large model and the knowledge-enhanced large model, and complete parameter initialization on the large-scale time series dataset and knowledge corpus respectively.

3. The method of claim 2, wherein the method further comprises: The specific process of constructing the large-scale time series model in S11 is as follows: standardizing the input time series data wherein is a time step, is a feature dimension; A multi-scale temporal convolutional network is used to extract local features at different time scales and generate multi-scale feature maps. Relative position encoding and timestamp encoding are introduced to inject position information and absolute time information into each time step; Global temporal dependency modeling is performed using a multi-layer temporal Transformer encoder, outputting multi-granular temporal features. ,in For the hidden layer dimension; Global average pooling and max pooling are fused on the original temporal features to obtain dimension-aligned temporal features. The calculation formula is: in, For pooling and merging weights, For global average pooling, This is for global max pooling.

4. The cross-modal collaborative method based on a time-series large model and a knowledge-enhanced large model according to claim 2, characterized in that, The specific process of constructing the knowledge-enhancing large model in S12 is as follows: Construct a domain knowledge graph containing entity sets. Relation sets triple set ; The TransH knowledge graph embedding algorithm is used to vectorize entities and relations, generating an entity embedding matrix. With relation embedding matrix ,in Embedding dimensions for knowledge; The knowledge embedding is concatenated with the text embedding of a pre-trained language model, and then fused through a knowledge-aware attention mechanism to output knowledge-enhanced semantic features. ,in For the number of knowledge entities; Entity-level attention weighting is applied to the original knowledge features to obtain dimension-aligned knowledge features. The calculation formula is: in, For the first Attention weights for each knowledge entity Assign an importance score to the entity. It is a two-layer perceptron.

5. The cross-modal collaborative method based on a time-series large model and a knowledge-enhanced large model according to claim 1, characterized in that, In step S2, cross-modal dynamic alignment, the process of establishing a temporal-knowledge joint representation space and achieving semantic alignment between temporal features and knowledge features through contrastive learning is as follows: S21. Construct a cross-modal projection network: Construct a temporal projection head and a knowledge projection head respectively, and align the temporal features with dimensions. With knowledge characteristics Mapped to a unified joint representation space, the projected feature dimensions are all... The calculation formula is: in, , The projection weight matrix is... , For bias terms; S22. Constructing positive and negative sample pairs: Sample pairs are constructed based on timestamp-event association rules. The specific rules are as follows: Positive sample pairs: temporal features within the same time window and knowledge entity features corresponding to events occurring within that time window; Negative sample pairs: Temporal features within the same time window and knowledge entity features within other time windows, as well as randomly sampled irrelevant knowledge entity features, with a positive-to-negative sample ratio of 1:10; S23. Design a cross-modal contrastive learning loss function: maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The formula for calculating the cross-modal contrastive learning loss function is as follows: in, The cosine similarity function is used. For temperature coefficient, For the first One time-series feature, These are the corresponding positive sample knowledge features. This represents negative sample knowledge features; S24. Introducing dynamic alignment weights: Adaptively adjusting the alignment strength based on the semantic relevance between modalities, using dynamic alignment weights. The calculation formula is: in, It is the Sigmoid activation function. This is the weight matrix. Here, represents the bias term, and represents the variance of the time-series features. The entropy of the attention weights of knowledge entities; S25. Generate cross-modal aligned representations: Achieve semantic unification between temporal sequences and knowledge modalities.

6. The cross-modal collaborative method based on a temporal series large model and a knowledge-enhanced large model according to claim 1, characterized in that, In step S3, hierarchical collaborative fusion, the process of performing multi-dimensional fusion operations at the feature layer, attention layer, and decision layer is as follows: S31. Feature Layer Fusion: A gated fusion mechanism is adopted to adaptively learn the fusion weights of temporal features and knowledge features, generating a feature layer fusion representation. The calculation formula for the gated fusion mechanism is as follows: in, It is the Sigmoid activation function. For learnable weight matrix, For bias terms, For element-wise multiplication, To merge the gate values; S32. Attention Layer Fusion: Design a cross-modal cross-attention mechanism to achieve bidirectional attention interaction between temporal features and knowledge features, generating an attention layer fusion representation. The calculation formula for the cross-modal cross-attention mechanism is as follows: , , , , in, , , , , , , For query, key, and value projection matrix, The dimension of the key; S33. Decision-level fusion: An ensemble learning strategy is used to weight and fuse the preliminary prediction results of the dual-path model to generate the final decision-level output. The calculation formula is as follows: , in, , These are the task output headers for the time-series model and the knowledge model, respectively. For decision fusion weights; S34. Generate a unified cross-modal fusion representation: Concatenate the feature layer fusion representation, attention layer fusion representation, and decision layer fusion representation, and generate the final unified cross-modal fusion representation through linear transformation. .

7. The cross-modal collaborative method based on a time-series large model and a knowledge-enhanced large model according to claim 1, characterized in that, In step S4, during collaborative reasoning execution, the process of constructing a collaborative reasoning engine based on cross-modal fusion representation and executing downstream tasks is as follows: S41. Preliminary prediction: Cross-modal fusion representation Input a task-specific output head network to generate preliminary prediction results; S42. Knowledge Constraint Regularization: A knowledge constraint regularization term based on first-order logic is introduced to ensure that the prediction results conform to the domain knowledge rules. The regularization loss calculation formula is as follows: in, For the first The satisfaction function of a knowledge rule, which determines the prediction result when it satisfies the rule. ,otherwise ; S43. Uncertainty Estimation: The Monte Carlo dropout method is used to quantify the confidence level of the prediction results. Specifically, the dropout layer is kept on during the inference phase. The next forward propagation yields the set of prediction results. Calculate the predicted mean and variance: Confidence interval is ; S44. Knowledge Origin: Based on the gradient contribution method, locate the knowledge entity that contributes the most to the prediction. The contribution calculation formula is as follows: in, For the task loss function, For the first Characteristics and contribution of each knowledge entity The larger the value, the greater the influence of the entity on the prediction result. The top 3 knowledge entities with the highest contribution are output as the source tracing results. S45. Multi-task joint optimization: The total loss function is a weighted sum of the contrastive learning loss, task loss, and knowledge constraint regularization loss. in, , To compensate for the loss weights, the AdamW optimizer is used for training; S46. Output the final inference result: including the predicted value. Confidence interval Explainable knowledge traceability information.

8. A cross-modal collaborative system based on a time-series large-scale model and a knowledge-enhanced large-scale model, characterized in that, include: The dual-path large model construction module is used to build a temporal large model and a knowledge-enhanced large model respectively. The temporal large model is used to extract multi-granular temporal features from the input temporal data, and the knowledge-enhanced large model is used to integrate structured knowledge from an external knowledge base with textual semantic information. The cross-modal dynamic alignment module is used to establish a temporal-knowledge joint representation space. It achieves semantic alignment between temporal features and knowledge features through contrastive learning, and generates cross-modal aligned representations. The hierarchical collaborative fusion module is used to perform multi-dimensional fusion operations at the feature layer, attention layer, and decision layer respectively, to generate a unified cross-modal fusion representation; The hierarchical collaborative fusion module includes: The feature layer fusion unit is used to adopt a gated fusion mechanism to adaptively learn the fusion weights of temporal features and knowledge features, and generate a feature layer fusion representation. Attention layer fusion unit is used to design cross-modal cross-attention mechanism to realize bidirectional attention interaction between temporal features and knowledge features, and generate attention layer fusion representation; The decision-level fusion unit is used to employ an ensemble learning strategy to weight and fuse the preliminary prediction results of the dual-path model to generate the final decision-level output. The collaborative reasoning execution module is used to build a collaborative reasoning engine based on cross-modal fusion representation, execute downstream time series prediction, anomaly detection, and intelligent decision-making tasks, and output the final reasoning results. The collaborative reasoning execution module includes: The task output head unit is used to fuse cross-modal representations into a task-specific output head network to generate preliminary prediction results. The knowledge constraint unit is used to introduce knowledge constraint regularization to ensure that the prediction results conform to the domain knowledge rules. Uncertainty estimation unit, used to perform uncertainty estimation and quantify the confidence level of the prediction results; The result output unit is used to output the final inference result, including the predicted value, confidence interval, and interpretability knowledge tracing information.

9. The cross-modal collaborative system based on a temporal large model and a knowledge-enhanced large model according to claim 1, characterized in that, The dual-path large model construction module includes: The temporal large model unit adopts the temporal Transformer architecture, introduces multi-scale temporal convolution and positional encoding to encode the input temporal data and output multi-granularity temporal feature tensors; The knowledge-enhanced large model unit adopts an architecture that combines knowledge graph embedding with a pre-trained language model. It vectorizes the entities, relations, and attributes of the external knowledge base and integrates them with text semantics to output knowledge-enhanced semantic features. The pre-training unit is used to independently pre-train the large-scale temporal model and the large-scale knowledge-enhanced model, and to initialize the parameters on the large-scale temporal dataset and the knowledge corpus, respectively.

10. The cross-modal collaborative system based on a temporal large model and a knowledge-enhanced large model according to claim 1, characterized in that, The cross-modal dynamic alignment module includes: The joint representation space construction unit is used to construct a cross-modal projection network, which maps temporal features and knowledge features to a unified joint representation space respectively. The contrastive learning training unit is used to design a cross-modal contrastive learning loss function to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. Dynamic alignment weight unit is used to introduce dynamic alignment weights and adaptively adjust the alignment strength according to the semantic relevance between modalities; Alignment representation output unit is used to generate cross-modal aligned representations to achieve semantic unification of temporal and knowledge modalities.