Multimodal Causal Reasoning Method and Device Based on Large Language Model

Abstract

This invention provides a multimodal causal inference method and apparatus based on a large language model, relating to the field of data processing technology. It maps multimodal data to a vector representation space using a pre-set multimodal encoder; constructs a multimodal vector sequence based on the vector representation space; inputs the multimodal vector sequence into a pre-set large language model for recognition, obtaining causal inference results; the multimodal encoder is pre-trained based on multimodal sample data and a causal contrastive loss function; the causal contrastive loss function is jointly determined by a contrastive learning loss function and a causal regularization term; the large language model is pre-trained based on the sample multimodal vector sequence. This invention solves the embedding bias problem by introducing a multimodal decoder to construct the vector representation space, effectively fusing features from different modalities. Furthermore, combining causal inference with a large language model enables the model to derive causal relationships from cross-modal data, improving inference performance.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a multimodal causal reasoning method and apparatus based on a large language model. Background Technology

[0002] In current applications of multimodal artificial intelligence, modal alignment and cross-modal causal reasoning are two key issues. Existing technologies, such as Contrastive Language-Image Pretraining (CLIP) models, primarily focus on bimodal alignment between text and images, using contrastive learning methods to calculate similarity between text and images. However, CLIP models only support comparisons within the same modality and cannot perform cross-modal causal reasoning, thus limiting their ability to handle more complex cross-modal causal relationships.

[0003] Further extending to the audio modality, the Audio Contrastive Language-Image Pretraining (AudioCLIP) model, while successfully incorporating the audio modality into the contrastive learning framework, remains fundamentally a contrastive learning framework, lacking the ability to model and reason about causal relationships. This prevents AudioCLIP from effectively performing causal reasoning across multiple modalities, limiting its application in higher-level intelligent tasks. Furthermore, while Large Language Models (LLMs) excel in processing text data, they still suffer from modality fragmentation in cross-modal causal reasoning. Even when these models can handle data from different modalities, their reasoning abilities regarding intermodal correlations and causal relationships remain insufficient, preventing them from performing cross-modal reasoning as naturally as humans.

[0004] Therefore, the main problem with existing technologies is the lack of in-depth understanding and modeling ability of causal relationships between different modalities, which limits the effectiveness of multimodal systems in practical applications. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a multimodal causal reasoning method and apparatus based on a large language model to solve the problem of modal fragmentation in current cross-modal causal reasoning.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of this invention discloses a multimodal causal reasoning method based on a large language model, the method comprising:

[0008] The received multimodal data is mapped to a vector representation space through a pre-set multimodal encoder; the multimodal encoder is pre-trained based on multimodal sample data and a causal contrastive loss function; the causal contrastive loss function is jointly determined by a contrastive learning loss function and a causal regularization term.

[0009] Construct a multimodal vector sequence corresponding to the multimodal data based on the vector representation space;

[0010] The multimodal vector sequence is input into a preset large language model for recognition to obtain causal inference results; the large language model is pre-trained based on the sample multimodal vector sequence.

[0011] Preferably, the preset multimodal encoder includes a dynamic low-rank projection matrix, a preset alignment matrix, and a preset layer normalization.

[0012] The step of mapping the received multimodal data to a vector representation space using a preset multimodal encoder includes:

[0013] When multimodal data is received, feature extraction is performed on the data of each modality in the multimodal data to obtain the feature vector of each modality;

[0014] The eigenvectors of each mode are projected into intermediate vectors using a dynamic low-rank projection matrix.

[0015] Based on a preset alignment matrix and preset layer normalization, the intermediate vector of each modality is mapped to the vector representation space.

[0016] Preferably, the step of inputting the multimodal vector sequence into a preset large language model for recognition to obtain causal inference results includes:

[0017] The multimodal vector sequence is input into a preset large language model;

[0018] Determine the similarity between the key vectors in the knowledge base of the large language model and the multimodal vector sequence;

[0019] The causal reasoning result is determined based on the similarity.

[0020] Preferably, the process of pre-training a multimodal encoder based on multimodal sample data and a causal contrastive loss function includes:

[0021] The initial multimodal encoder is used to extract features from the multimodal sample data to obtain the multimodal sample feature vector;

[0022] The sample feature vectors of the multimodal encoder are projected into sample intermediate vectors using the dynamic low-rank projection matrix in the initial multimodal encoder.

[0023] Based on the preset alignment matrix and preset layer normalization in the initial multimodal encoder, the sample intermediate vector is processed, and a causal correlation matrix is ​​created based on the processing result.

[0024] Determine whether the causal contrast loss function of the initial multimodal encoder has converged based on the causal correlation matrix;

[0025] If convergence is achieved, the initial multimodal encoder is determined to be a well-trained multimodal encoder.

[0026] If convergence is not achieved, the dynamic low-rank projection matrix and the alignment matrix are adjusted, and the process returns to the step of extracting features from the multimodal sample data using the initial multimodal encoder to obtain the multimodal sample feature vector.

[0027] Preferably, the process of pre-training a large language model based on sample multimodal vector sequences includes:

[0028] The sample multimodal vector sequence is input into the initial large language model to determine the similarity between the key vector and the sample multimodal vector sequence in the knowledge base of the initial large language model.

[0029] The causal reasoning result is determined based on the similarity.

[0030] Based on the causal inference results, determine whether the loss function of the initial large language model has converged;

[0031] If the loss function of the initial large language model converges, then the initial large language model is determined as the trained large language model.

[0032] If the loss function of the initial large language model does not converge, the parameters of the initial large language model are adjusted, and the process returns to the step of inputting the sample multimodal vector sequence into the initial large language model and determining the similarity between the key vector and the sample multimodal vector sequence in the knowledge base of the initial large language model.

[0033] A second aspect of this invention discloses a multimodal causal reasoning device based on a large language model, the device comprising:

[0034] A mapping unit is used to map the received multimodal data to a vector representation space through a preset multimodal encoder; the multimodal encoder is pre-trained based on multimodal sample data and a causal contrastive loss function; the causal contrastive loss function is jointly determined by a contrastive learning loss function and a causal regularization term.

[0035] A construction unit is used to construct a multimodal vector sequence corresponding to the multimodal data based on the vector representation space;

[0036] The recognition unit is used to input the multimodal vector sequence into a preset large language model for recognition and to obtain causal inference results; the large language model is trained in advance based on the sample multimodal vector sequence.

[0037] Preferably, the preset multimodal encoder includes a dynamic low-rank projection matrix, a preset alignment matrix, and a preset layer normalization.

[0038] The mapping unit includes:

[0039] The feature extraction module is used to extract features from the data of each modality in the multimodal data when multimodal data is received, so as to obtain the feature vector of each modality.

[0040] The projection module is used to project the feature vector of each mode into an intermediate vector using a dynamic low-rank projection matrix.

[0041] The mapping module is used to map the intermediate vector of each modality to the vector representation space based on a preset alignment matrix and a preset layer normalization.

[0042] Preferably, the identification unit includes:

[0043] The input module is used to input the multimodal vector sequence into a preset large language model;

[0044] The first determining module is used to determine the similarity between the key vectors in the knowledge base of the large language model and the multimodal vector sequence;

[0045] The second determining module is used to determine the causal reasoning result based on the similarity.

[0046] Preferred options also include:

[0047] The feature extraction unit is used to extract features from the multimodal sample data using the initial multimodal encoder to obtain multimodal sample feature vectors.

[0048] The projection unit is used to project the multimodal sample feature vectors into sample intermediate vectors through the dynamic low-rank projection matrix in the initial multimodal encoder.

[0049] A creation unit is used to process the sample intermediate vector based on the preset alignment matrix and preset layer normalization in the initial multimodal encoder, and to create a causal correlation matrix based on the processing result.

[0050] The first judgment unit is used to determine whether the causal contrast loss function of the initial multimodal encoder has converged based on the causal correlation matrix.

[0051] The first determining unit is used to determine the initial multimodal encoder as a trained multimodal encoder if convergence is achieved.

[0052] The first adjustment unit is used to adjust the dynamic low-rank projection matrix and the alignment matrix if convergence is not achieved, and then return to execute the feature extraction unit.

[0053] Preferred options also include:

[0054] The input unit is used to input the sample multimodal vector sequence into the initial large language model and determine the similarity between the key vector and the sample multimodal vector sequence in the knowledge base of the initial large language model.

[0055] The second determining unit is used to determine the causal reasoning result based on the similarity.

[0056] The second judgment unit is used to determine whether the loss function of the initial large language model has converged based on the causal inference result;

[0057] The third determining unit is used to determine the initial large language model as a trained large language model if the loss function of the initial large language model converges.

[0058] The second adjustment unit is used to adjust the parameters of the initial large language model and return to the input unit if the loss function of the initial large language model fails to converge.

[0059] The present invention provides a multimodal causal inference method and apparatus based on a large language model. The method maps multimodal data to a vector representation space using a pre-set multimodal encoder; constructs a multimodal vector sequence based on the vector representation space; inputs the multimodal vector sequence into a pre-set large language model for recognition, obtaining causal inference results. The multimodal encoder is pre-trained based on multimodal sample data and a causal contrast loss function. The causal contrast loss function is jointly determined by a contrastive learning loss function and a causal regularization term. The large language model is pre-trained based on the sample multimodal vector sequence. This invention solves the embedding bias problem caused by modal heterogeneity by introducing a multimodal decoder to construct the vector representation space, effectively fusing features from different modalities. Furthermore, combining causal inference with a large language model enables the model to derive causal relationships from cross-modal data, improving inference performance. Attached Figure Description

[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0061] Figure 1 A flowchart illustrating a multimodal causal reasoning method based on a large language model, provided in an embodiment of the present invention;

[0062] Figure 2 This is a schematic diagram illustrating the process of training a multimodal encoder according to an embodiment of the present invention;

[0063] Figure 3 This is a schematic diagram illustrating the process of training a large language model according to an embodiment of the present invention;

[0064] Figure 4 This is a structural block diagram of a multimodal causal reasoning device based on a large language model, provided in an embodiment of the present invention. Detailed Implementation

[0065] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0066] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0067] As the background technology shows, although AudioCLIP successfully incorporates audio modal contrastive learning, it lacks causal reasoning capabilities. Large language models still suffer from modal fragmentation in cross-modal causal reasoning, which limits their application in higher-level intelligent tasks.

[0068] Therefore, embodiments of the present invention provide a multimodal causal reasoning method and apparatus based on a large language model. The method maps multimodal data to a vector representation space using a pre-set multimodal encoder; constructs a multimodal vector sequence based on the vector representation space; inputs the multimodal vector sequence into a pre-set large language model for recognition, obtaining causal reasoning results; the multimodal encoder is pre-trained based on multimodal sample data and a causal contrast loss function; the causal contrast loss function is jointly determined by a contrastive learning loss function and a causal regularization term; the large language model is pre-trained based on the sample multimodal vector sequence. This invention solves the embedding bias problem caused by modal heterogeneity by introducing a multimodal decoder to construct the vector representation space, effectively fusing features from different modalities. Furthermore, combining causal reasoning with a large language model enables the model to derive causal relationships from cross-modal data, improving reasoning performance and making the model more intelligent in complex causal reasoning tasks, extracting meaningful causal connections from multimodal information and performing more accurate correlation analysis.

[0069] See Figure 1 The diagram illustrates a flowchart of a multimodal causal reasoning method based on a large language model, provided by an embodiment of the present invention. The method includes:

[0070] Step S101: Map the received multimodal data to the vector representation space using a preset multimodal encoder.

[0071] It should be noted that multimodal data can be constructed based on WebVid-10M (video), AudioSet (audio), and Wikipedia (text).

[0072] In the specific implementation step S101, when multimodal data (such as data in three modalities: text, image, and audio) is received, the multimodal data is mapped to a vector representation space through a preset multimodal encoder.

[0073] This vector representation space is a unified vector representation space. That is, text, image, and audio data are all mapped to vectors within the same space (e.g., v_text, v_image, v_audio). Within this vector representation space, semantically similar or causally related sample points, regardless of their modality, should have similar vector distances.

[0074] It should be noted that the preset multimodal encoder is pre-trained based on multimodal sample data (such as text, images, and audio) and a causal contrastive loss function. For details of the training process, please refer to the embodiments of this invention. Figure 2 The content of that article will not be discussed here.

[0075] Understandably, the pre-defined multimodal encoder employs an improved AudioCLIP architecture, including a dynamic low-rank projection matrix, a pre-defined alignment matrix, and pre-defined layer normalization. The pre-defined multimodal encoder pre-constructs a vector representation space by jointly training multimodal sample data (such as text, images, and audio).

[0076] Specifically, in step S101, the process of mapping the received multimodal data to the vector representation space using a preset multimodal encoder is as follows (processes A1 to A3):

[0077] Process A1: When multimodal data is received, feature extraction is performed on the data of each modality in the multimodal data to obtain the feature vector of each modality.

[0078] In implementing process A1, when multimodal data is received, feature extraction is performed on the data of each modality through the corresponding encoder to obtain the feature vector of each modality.

[0079] Specifically, in the preset multimodal encoder, the text encoder adopts a Transformer-based structure, with input being lowercase byte pair encoding (BPE encoding), a maximum sequence length of 76, and output being a fixed-dimensional (e.g., 1024-dimensional) text feature vector h_text.

[0080] For example, the text data processing procedure is as follows: the text data is encoded using BPE (the vocabulary used contains 49,408 words) and truncated to a 76-word sequence, and then a 1024-dimensional text feature vector h_text is generated by the Transformer encoder.

[0081] The image encoder is a contrastive language-image pretraining (CLIP) model based on a residual network (ResNet). It is used to receive a preprocessed image (e.g., 224×224 pixels) and output a fixed-dimensional (e.g., 1024-dimensional) image feature vector h_image.

[0082] For example, the image data processing procedure involves extracting keyframes from the image data, cropping them to 224×224 pixels, and then generating a 1024-dimensional image feature vector h_image after extraction by ResNet-50.

[0083] The audio encoder uses a cardinality-based residual network (ESResNeXT) model, replacing its classification layer with randomly initialized fully connected layers. It receives a preprocessed audio spectrogram (e.g., Mel spectrum) and outputs a fixed-dimensional (e.g., 1024-dimensional) audio feature vector, h_audio.

[0084] For example, the audio data processing procedure is as follows: the audio data is first processed by frame segmentation (frame length 25ms, step size 10ms), and spectral features are generated by an 80-dimensional Mel filter bank. The spectral features are then input into the ESResNeXT encoder and output as a 1024-dimensional audio feature vector h_audio.

[0085] Understandably, in process A1, the three encoders operate independently, mapping the raw data of their respective modalities to their specific feature spaces. These h-feature vectors contain rich information about their respective modalities, but they are not yet aligned. Due to the significant differences in statistical properties (i.e., heterogeneity) between the raw features of text, image, and audio (h_text, h_image, h_audio), direct contrastive learning is ineffective. Directly using fully connected layers for projection would incur substantial computational overhead. Therefore, further processing of the raw features (h_text, h_image, h_audio) is necessary.

[0086] Process A2: Project the eigenvectors of each mode into intermediate vectors using a dynamic low-rank projection matrix.

[0087] In implementing process A2, a dynamic low-rank projection matrix W_m (where m is the mode) is introduced for each mode. The eigenvectors of each mode are projected into intermediate vectors using the dynamic low-rank projection matrix.

[0088] The dynamic low-rank projection matrix is ​​shown in formula (1).

[0089] p_m=W_m·h_m+b_m(1)

[0090] In formula (1), W_m∈ r_m is the rank of the dynamic low-rank projection matrix, d is the dimension; p_m is the intermediate vector; b_m is a vector with the same dimension as the intermediate vector p_m, that is, b_m is the bias term. The role of b_m is to perform a global translation or offset on the result of matrix multiplication W_m·h_m.

[0091] In some specific embodiments, the rank of the audio, r_audio, can be 1.5 times the rank of the image, r_image, and the rank of the audio is typically maintained between 136 and 142.

[0092] Using formula (1), the feature vector of each modality is projected into an intermediate vector: h_text→p_text, h_image→p_image, h_audio→p_audio.

[0093] It should be noted that the rank r_m of the dynamic low-rank projection matrix is ​​not fixed, but is adaptively initialized according to the modal complexity (e.g., r_text=64, r_image=96, r_audio=128), and dynamically adjusted during training according to its singular value distribution. For specific adjustment strategies, please refer to the embodiments of this invention. Figure 2 The content of that article will not be discussed here.

[0094] It is understandable that in process A2, the three heterogeneous modal features h_m are projected onto three intermediate vectors p_text, p_image, and p_audio, which have the same dimensions and similar distributions.

[0095] In this process, the rank of the dynamic low-rank projection matrix is ​​flexibly adjusted according to the characteristics of the input data and the task requirements, thereby avoiding the problems of resource waste or insufficient representation ability caused by a fixed rank.

[0096] The dynamic rank mechanism of the dynamic low-rank projection matrix can not only improve the model's performance when processing data of different modalities, but also significantly improve computational efficiency and reduce unnecessary computational burden, thereby enhancing the model's ability to handle complex tasks while ensuring high efficiency.

[0097] Process A3: Based on the preset alignment matrix and preset layer normalization, the intermediate vector of each modality is mapped to the vector representation space.

[0098] It should be noted that although the intermediate vectors of each mode obtained after projection through the dynamic low-rank projection matrix have the same dimension, they still need to be mapped to the final unified vector representation space.

[0099] Therefore, in process A3, a shared, pre-defined alignment matrix U_align (e.g., U_align∈) is used. A pre-defined layer normalization (i.e., LayerNorm layer) maps the intermediate vector of each modality to a d-dimensional (e.g., 512-dimensional) vector representation space. Specifically, as shown in Equation (2).

[0100] v_m=LayerNorm(p_m·U_align)(2)

[0101] In formula (2), p_m is the intermediate vector of each mode; v_m is the vector in the vector representation space.

[0102] Furthermore, d-dimensionality refers to the dimension of the final unified vector representation space, which is the length of the vector v_m. d is a flexibly configurable hyperparameter (e.g., 512 dimensions). Its purpose is to provide a standardized, compact, and comparable vector space for all modalities.

[0103] Formula (2) maps the intermediate vector of each modality to the vector representation space. That is, p_text→v_text, p_image→v_image, p_audio→v_audio.

[0104] Through process A3, text, image, and audio data are all converted into vectors in a vector representation space (such as v_text, v_image, v_audio). In this vector representation space, sample points that are semantically similar or causally related should have similar vector distances, regardless of which modality they come from.

[0105] Step S102: Construct a multimodal vector sequence corresponding to the multimodal data based on the vector representation space.

[0106] In the specific implementation of step S102, the vectors (v_text, v_image, v_audio) corresponding to the multimodal data (i.e., the multimodal data received in step S101) are extracted from the vector representation space. Based on the semantic content of the triple and the preset causal logic, the triple is constructed into a new triple and marked as a multimodal vector sequence, such as Q=[v_audio; v_image; v_text].

[0107] It should be noted that the predefined causal logic refers to the predefined or learned causal relationship structure. This is to construct a causal sequence of vectors (v_text, v_image, v_audio) from different modalities into a causally meaningful multimodal vector sequence, such as Q=[v_audio; v_image; v_text].

[0108] Step S103: Input the multimodal vector sequence into the preset large language model for recognition to obtain the causal inference result.

[0109] In the specific implementation step S103, the multimodal vector sequence is input into the preset large language model; the similarity between the key vector and the multimodal vector sequence in the knowledge base of the large language model is determined; and then the causal inference result is determined based on the similarity.

[0110] It should be noted that in the pre-defined large language model, a cross-modal attention gate is used to calculate the similarity between (K,V) in the large language model's own knowledge base and the multimodal vector sequence Q. Based on the similarity, the knowledge most relevant to the multimodal vector sequence Q is dynamically selected and focused, ultimately determining the causal inference result. Here, K in (K,V) is the Key index, and V is the Value, which is the value vector within the large language model.

[0111] In some practical applications, the pre-defined large language model can also generate text output based on the results of causal reasoning, such as disposal instructions.

[0112] It should be noted that the large language model is pre-trained based on the multimodal vector sequence of the samples; for details of the training process, please refer to the embodiments of this invention. Figure 3 The content of that article will not be discussed here.

[0113] In some specific embodiments, the preset large language model can also update its knowledge base based on the user's evaluation of the causal reasoning results. For specific update methods, please refer to the embodiments of the present invention. Figure 3 The content of that article will not be discussed here.

[0114] In this embodiment of the invention, a unified multimodal vector representation space is constructed by introducing a pre-defined multimodal decoder, thus solving the embedding bias problem caused by modal heterogeneity. The multimodal decoder can effectively fuse features from different modalities (such as text, images, and audio), eliminating their differences and enabling comparison and correlation of representations from different modalities within the same space. Furthermore, combining this with a pre-defined large language model for causal inference allows the model to derive causal relationships from cross-modal data, rather than simply associating them. This method significantly improves inference performance, making the model more intelligent in complex causal inference tasks, capable of extracting meaningful causal connections from multimodal information, and performing more accurate correlation analysis.

[0115] The above Figure 1 For details regarding the pre-training process of the mentioned multimodal encoder, please refer to [link / reference]. Figure 2 The diagram illustrates the process of training a multimodal encoder according to an embodiment of the present invention, including the following steps:

[0116] Step S201: Use the initial multimodal encoder to extract features from the multimodal sample data to obtain the multimodal sample feature vector.

[0117] It should be noted that the initial multimodal encoder E was constructed using an improved AudioCLIP architecture. unified Initial multimodal encoder E unifiedIt includes at least a text encoder, an image encoder, and an audio encoder. For detailed descriptions of the text encoder, image encoder, and audio encoder, please refer to the embodiments of this invention. Figure 1 The content described in process A1 will not be repeated here.

[0118] In the specific implementation step S201, the initial multimodal encoder is used to extract independent features from the multimodal sample data, that is, the triplet sample data containing text, image and audio (e.g., "a dog is barking", a picture of a dog, and the audio of the dog barking), to obtain the sample feature vector of each modality. Specifically, as shown in formula (3).

[0119] (3)

[0120] In formula (3), This is the feature vector of the sample.

[0121] Step S202: Project the multimodal sample feature vectors into sample intermediate vectors using the dynamic low-rank projection matrix in the initial multimodal encoder.

[0122] It is understandable that the process of projecting the multimodal sample feature vectors into intermediate sample vectors using the dynamic low-rank projection matrix in the initial multimodal encoder is as described in the embodiments of this invention. Figure 1 The process described in process A2 is similar and will not be repeated here.

[0123] It should be noted that the rank adjustment strategy for the dynamic low-rank projection matrix is ​​as follows:

[0124] Understandably, the rank is adaptively adjusted through modal complexity. Specifically, the singular values ​​of the dynamic low-rank projection matrix W_m are periodically calculated (e.g., every 5 epochs), and the ratio of the intermediate singular value to the largest singular value is calculated. As shown in formula (4).

[0125] (4)

[0126] In formula (4), These are intermediate singular values; is the maximum singular value; r_m is the rank of the dynamic low-rank projection matrix.

[0127] Calculate the ratio according to formula (4). ,like If the rank is greater than 0.7, then the principal components in the feature set are relatively few, and the rank can be reduced (e.g., r_m = r_m - 8) to further compress the information and reduce the computational load. If the value is less than 0.1, the indicator features are scattered and require more dimensions for representation. Increasing the rank (e.g., r_m = r_m + 16) can help retain more information. If 0.1 < If the value is less than 0.7, then the hyperparameter combination in the indicator feature set is appropriate, and there is no need to adjust the rank.

[0128] For example, as can be seen from formula (5), the adaptive mechanism of the rank of the dynamic low-rank projection matrix is ​​to determine whether to increase or decrease the rank by analyzing the "principal component concentration" of the matrix, so as to achieve a balance between computational efficiency and representational ability.

[0129] (5)

[0130] In formula (5), It is the new rank of the projection matrix W_m of mode m after the (k+1)th adjustment. It is the current rank of the projection matrix W_m of mode m at the k-th adjustment. The increment or decrement of the rank (i.e., the adjustment step size) is determined by the condition judgment on the right.

[0131] In formula (5), It is the rm / 2th singular value (i.e., the middle singular value) of the projection matrix W_m. rm is the current rank. It is a dynamic low-rank projection matrix. ) is the largest singular value (i.e., the first largest singular value) of the projection matrix W_m.

[0132] In formula (5), It is a ratio , Known as the "singular value decay ratio," it is a key indicator for measuring the degree of information concentration in a matrix.

[0133] Understandably, if the ratio A value greater than 0.7 indicates that the first few principal components of the matrix (the eigendirections corresponding to the largest singular values) already contain most of the information, while the information in the remaining dimensions is relatively less and redundant. In this case, it is safe to reduce the rank (Δr=-8) to reduce computational overhead.

[0134] Step S203: Based on the preset alignment matrix and preset layer normalization in the initial multimodal encoder, process the sample intermediate vectors and create a causal correlation matrix based on the processing results.

[0135] Understandably, based on the preset alignment matrix and preset layer normalization in the initial multimodal encoder, after processing the sample intermediate vectors, the sample intermediate vectors are transformed into vectors in the same vector representation space. Causal triples are constructed based on these vectors, and then causal correlation matrices are created based on the causal triples.

[0136] Specifically, starting from a set of multimodal data (i.e., a triple of text, image, and audio) in the same vector representation space, based on its semantic content and pre-defined causal logic, it is reorganized into a new causal triple T = (m_cause, m_effect, m_noise) (e.g., m...). cause =Thunder audio, m effect =Lightning image, m noise =Irrelevant text), and then determine the causal correlation matrix M based on the causal triple T. causal .

[0137] It is important to note that the causal relationship matrix M... causal Activation via cosine similarity thresholding, such as cos(v) i ,v j When M > 0.85, causal [i,j]=1;cos(v i ,v j When M <= 0.85, causal [i,j]=0.

[0138] Among them, cos(v i ,v j That is, the vector representation of all sample pairs (i,j) in the vector space. i and v j Cosine similarity between them.

[0139] Step S204: Determine whether the causal contrast loss function of the initial multimodal encoder has converged based on the causal correlation matrix.

[0140] Specifically, the causal contrastive loss function is jointly determined by the contrastive learning loss function and the causal regularization term.

[0141] Contrastive learning loss (L_contrastive) is such as the Information Noise Contrastive Loss (InfoNCE Loss). It brings positive sample pairs (such as v_text and its corresponding v_image) closer together and pushes negative sample pairs (such as v_text and v_image of other images in multimodal data) further apart. This ensures cross-modal semantic alignment.

[0142] The causal regularization term (L_causal) is crucial for achieving causal inference. It involves constructing causal triples T = (m_cause, m_effect, m_noise), where m_cause is the cause modality, m_effect is the effect modality, and m_noise is the noise modality, and designing a loss term to maximize the mutual information between v_cause and v_effect while minimizing the mutual information between v_cause and v_noise. This process does not require manual causal labeling; instead, it implicitly guides the model to learn causal relationships through the structure of the data itself (constructing the triples).

[0143] Causal contrastive loss function As shown in Equation (6). Equation (6) defines the joint training objective of the entire multimodal encoder, which combines the traditional contrastive learning loss with a new causal regularization term to guide the multimodal encoder to implicitly learn causal relationships while aligning cross-modal semantics.

[0144] (6)

[0145] In formula (6), The vector is the dot product (inner product) of p_i and p_j, used to calculate the similarity between them. It is a temperature hyperparameter that controls the smoothness of the similarity distribution. It calculates the similarity index of positive sample pairs (i,j).

[0146] In formula (6), It calculates the sum of the similarity indices of all negative sample pairs (i,k).

[0147] In formula (6), λ is the regularization coefficient, which controls the weight of the causal constraint term relative to the contrastive loss. A set of embedding vectors representing the "cause" modality (e.g., the sound of thunder). A set of embedding vectors representing the “outcome” modality (e.g., a lightning image). It is a causal correlation matrix. This is a binary matrix, where M_causal[i,j]=1 indicates that in the training data, there is a causal relationship between the i-th "cause" sample and the j-th "result" sample.

[0148] In the specific implementation of step S204, it is determined whether the causal contrast loss function of the initial multimodal encoder has converged based on the causal correlation matrix. If the causal contrast loss function of the initial multimodal encoder has converged based on the causal correlation matrix, then step S205 is executed; if the causal contrast loss function of the initial multimodal encoder has not converged based on the causal correlation matrix, then step S206 is executed.

[0149] Step S205: If convergence is achieved, the initial multimodal encoder is determined as the trained multimodal encoder.

[0150] Step S206: If convergence is not achieved, adjust the dynamic low-rank projection matrix and alignment matrix, and return to step S201.

[0151] In this embodiment of the invention, the causal contrastive loss function, through the synergistic effect of contrastive loss and causal regularization term, drives the multimodal encoder to learn a unified semantic representation space with a deeper level and richer structure. It not only requires the multimodal encoder to learn basic cross-modal associations, but also, by introducing causal constraints, prompts the multimodal encoder to capture the inherent causal mechanisms between things.

[0152] The above Figure 1 The detailed content mentioned regarding the pre-training of a large language model based on sample multimodal vector sequences will be discussed in more detail. Figure 3 The diagram illustrates the process of training a large language model according to an embodiment of the present invention, including the following steps:

[0153] Step S301: Input the sample multimodal vector sequence into the initial large language model and determine the similarity between the key vector and the sample multimodal vector sequence in the knowledge base of the initial large language model.

[0154] In the specific implementation step S301, the sample multimodal vector sequence is input into the initial large language model (e.g., Large Language Model Meta AI (LLM LLaMA)) and passed through a cross-modal attention gate. The similarity between key vector K and sample multimodal vector sequence Q in the knowledge base of the large language model is calculated, and an attention weight matrix is ​​output. The knowledge base of the large language model is, for example, a large-scale multimodal science question answering benchmark dataset (ScienceQA).

[0155] It should be noted that in cross-modal attention gates In this context, σ is the Sigmoid function.

[0156] Step S302: Determine the causal reasoning result based on similarity.

[0157] In the specific implementation step S302, based on the attention weight matrix corresponding to the similarity, the corresponding Values ​​(i.e., the specific knowledge content) are obtained and weighted and summed to obtain a dynamically combined causal inference result that is most relevant to the multimodal vector sequence of the sample. Specifically, as shown in formula (7).

[0158] (7)

[0159] In formula (7), Indicates the result of causal reasoning; V represents the attention weight matrix; V represents the Values ​​(i.e., the specific knowledge content) corresponding to the attention weight matrix. This represents the set of embedding vectors representing the "cause" modality (e.g., the audio of thunder). It is used as an offset term in the output to ensure that the final causal inference result is directly related to the triggering event itself.

[0160] Step S303: Determine whether the loss function of the initial large language model has converged based on the causal inference results.

[0161] In the specific implementation of step S303, it is determined whether the loss function of the initial large language model has converged based on the causal inference results. If the loss function of the initial large language model has converged based on the causal inference results, then step S304 is executed; if the loss function of the initial large language model has not converged based on the causal inference results, then step S305 is executed.

[0162] Step S304: If the loss function of the initial large language model converges, then the initial large language model is determined as the trained large language model.

[0163] Step S305: If the loss function of the initial large language model does not converge, adjust the parameters of the initial large language model and return to step S301.

[0164] Furthermore, embodiments of the present invention Figure 1 The pre-defined large language model involved can also update its knowledge base based on users' evaluations of the causal reasoning results. The specific update methods are as follows:

[0165] When the pre-defined large language model receives a positive evaluation from the user on the causal reasoning result, the weight of the causal chain is strengthened, as shown in formula (8).

[0166] ΔMccausal←ΔMccausal+η(vcause veffect) (η=0.05) (8)

[0167] In formula (8), ΔMcausal is the increment (or change) of the causal chain weights. It is a matrix that records the strength of new causal relationships learned by the system from successful handling events within the current processing cycle. This value accumulates continuously.

[0168] η is the learning rate, a hyperparameter that controls the strength of the influence of newly learned causal information on existing weights. A typical value is, for example, η = 0.05.

[0169] Vcause is the vector of the "cause" modality in the currently successfully handled event in the unified representation space. For example, in the input "glass breaking sound + fire alarm sound", the vector of "glass breaking sound".

[0170] A Veffect is a vector in a unified representation space representing the "outcome" modality of a currently successfully handled event. For example, in the input "Infrared camera captures window flash," the vector is for "flash."

[0171] It's an outer product operation. v_cause The result of v_effect is a matrix that combines the information from two vectors to form a pattern that represents the specific causal relationship of "v_cause causing v_effect".

[0172] Then, for example, the knowledge base key-value pairs are updated incrementally every 24 hours, as shown in formula (9).

[0173] K(new)=K(old)⊕0.2ΔKcausal(9)

[0174] In formula (9), K(new) is the new state of the "key" vector in the knowledge base of the LLaMA model.

[0175] K(old) is the old state of the "key" vector in the knowledge base of the LLaMA model.

[0176] ⊕ indicates that the new causal knowledge ΔK_causal is merged into the existing knowledge base K(old).

[0177] ΔKcausal is the increment of the causal knowledge base key values. It is obtained by processing ΔM_causal (such as aggregation or dimensionality reduction) and is used to update the knowledge base inside the LLaMA model.

[0178] In this embodiment of the invention, the reasoning ability of the large language model has achieved a qualitative leap, enabling it to deeply integrate multimodal information such as vision, language, and audio. It not only understands surface correlations but also reveals the complex causal mechanisms behind the data. Because judgments based on causal relationships reduce reliance on surface statistical associations of the data, the output results are more robust, fair, and logical.

[0179] Corresponding to the multimodal causal reasoning method based on a large language model provided in the above embodiments of the present invention, see also... Figure 4 The diagram shows a structural block diagram of a multimodal causal reasoning device based on a large language model provided by an embodiment of the present invention.

[0180] The device includes a mapping unit 401, a construction unit 402, and an identification unit 403.

[0181] The mapping unit 401 is used to map the received multimodal data to the vector representation space through a preset multimodal encoder; the multimodal encoder is pre-trained based on multimodal sample data and a causal contrast loss function; the causal contrast loss function is jointly determined by the contrastive learning loss function and the causal regularization term.

[0182] Construction unit 402 is used to construct a multimodal vector sequence corresponding to multimodal data based on the vector representation space.

[0183] The recognition unit 403 is used to input the multimodal vector sequence into a preset large language model for recognition and to obtain causal inference results; the large language model is trained in advance based on the sample multimodal vector sequence.

[0184] In this embodiment of the invention, a unified multimodal vector representation space is constructed by introducing a pre-defined multimodal decoder, thus solving the embedding bias problem caused by modal heterogeneity. The multimodal decoder can effectively fuse features from different modalities (such as text, images, and audio), eliminating their differences and enabling comparison and correlation of representations from different modalities within the same space. Furthermore, combining this with a pre-defined large language model for causal inference allows the model to derive causal relationships from cross-modal data, rather than simply associating them. This method significantly improves inference performance, making the model more intelligent in complex causal inference tasks, capable of extracting meaningful causal connections from multimodal information, and performing more accurate correlation analysis.

[0185] Combination Figure 4 The content shown includes a dynamic low-rank projection matrix, a preset alignment matrix, and a preset layer normalization in the preset multimodal encoder.

[0186] The mapping unit 401 includes a feature extraction module, a projection module, and a mapping module.

[0187] The feature extraction module is used to extract features from the data of each modality in the multimodal data when multimodal data is received, so as to obtain the feature vector of each modality.

[0188] The projection module is used to project the eigenvectors of each modality into intermediate vectors using a dynamic low-rank projection matrix.

[0189] The mapping module is used to map the intermediate vector of each modality to the vector representation space based on a preset alignment matrix and a preset layer normalization.

[0190] Combination Figure 4The content shown, the recognition unit 403, includes: an input module, a first determination module and a second determination module.

[0191] The input module is used to input multimodal vector sequences into a pre-defined large language model.

[0192] The first determination module is used to determine the similarity between key vectors and multimodal vector sequences in the knowledge base of the large language model.

[0193] The second determining module is used to determine the causal reasoning result based on similarity.

[0194] Combination Figure 4 The device, as shown, also includes: a feature extraction unit, a projection unit, a creation unit, a first judgment unit, a first determination unit, and a first adjustment unit.

[0195] The feature extraction unit is used to extract features from the multimodal sample data using the initial multimodal encoder to obtain multimodal sample feature vectors.

[0196] The projection unit is used to project the multimodal sample feature vectors into sample intermediate vectors using the dynamic low-rank projection matrix in the initial multimodal encoder.

[0197] A creation unit is used to process the intermediate vectors of samples based on the preset alignment matrix and preset layer normalization in the initial multimodal encoder, and to create a causal correlation matrix based on the processing results.

[0198] The first judgment unit is used to determine whether the causal contrast loss function of the initial multimodal encoder has converged based on the causal correlation matrix.

[0199] The first determining unit is used to determine the initial multimodal encoder as a trained multimodal encoder if convergence is achieved.

[0200] The first adjustment unit is used to adjust the dynamic low-rank projection matrix and alignment matrix if convergence is not achieved, and then return to the feature extraction unit.

[0201] Combination Figure 4 The device, as shown, also includes: an input unit, a second determining unit, a second judging unit, a third determining unit, and a second adjusting unit.

[0202] The input unit is used to input the sample multimodal vector sequence into the initial large language model and determine the similarity between the key vector and the sample multimodal vector sequence in the knowledge base of the initial large language model.

[0203] The second determining unit is used to determine the causal reasoning result based on similarity.

[0204] The second judgment unit is used to determine whether the loss function of the initial large language model has converged based on the causal inference results.

[0205] The third determining unit is used to determine the initial large language model as the trained large language model if the loss function of the initial large language model converges.

[0206] The second adjustment unit is used to adjust the parameters of the initial large language model if the loss function of the initial large language model has not converged, and then return to the execution input unit.

[0207] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0208] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0209] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for multi-modal causal reasoning based on a large language model, characterized in that, The method comprises: mapping the received multi-modal data to a vector representation space through a preset multi-modal encoder; the multi-modal encoder is trained in advance based on sample data of multi-modal and a causal contrast loss function; the causal contrast loss function is determined by a contrast learning loss function and a causal regularization term; constructing a multi-modal vector sequence corresponding to the multi-modal data based on the vector representation space; inputting the multi-modal vector sequence into a preset large language model for recognition to obtain a causal reasoning result; the large language model is trained in advance according to sample multi-modal vector sequences; wherein the process of training the multi-modal encoder in advance based on sample data of multi-modal and a causal contrast loss function comprises: extracting features of the sample data of multi-modal by using an initial multi-modal encoder to obtain sample feature vectors of multi-modal; projecting the sample feature vectors of multi-modal into sample intermediate vectors through a dynamic low-rank projection matrix in the initial multi-modal encoder; processing the sample intermediate vectors based on a preset alignment matrix and a preset layer normalization in the initial multi-modal encoder, and creating a causal correlation matrix based on the processing result; determining whether the causal contrast loss function of the initial multi-modal encoder converges according to the causal correlation matrix; if the initial multi-modal encoder converges, the initial multi-modal encoder is determined as a trained multi-modal encoder; if the initial multi-modal encoder does not converge, adjusting the dynamic low-rank projection matrix and the alignment matrix, and returning to execute the step of extracting features of the sample data of multi-modal by using an initial multi-modal encoder to obtain sample feature vectors of multi-modal.

2. The method of claim 1, wherein, The preset multi-modal encoder includes a dynamic low-rank projection matrix, a preset alignment matrix and a preset layer normalization; the process of mapping the received multi-modal data to a vector representation space through a preset multi-modal encoder comprises: when receiving multi-modal data, extracting features of the data of each modality in the multi-modal data to obtain feature vectors of each modality; projecting the feature vectors of each modality into intermediate vectors by using a dynamic low-rank projection matrix; mapping the intermediate vectors of each modality to a vector representation space based on a preset alignment matrix and a preset layer normalization.

3. The method of claim 1, wherein, the process of inputting the multi-modal vector sequence into a preset large language model for recognition to obtain a causal reasoning result comprises: inputting the multi-modal vector sequence into a preset large language model; determining the similarity between the key vectors in the knowledge base of the large language model and the multi-modal vector sequence; determining the causal reasoning result based on the similarity.

4. The method of claim 1, wherein, the process of training the large language model in advance according to sample multi-modal vector sequences comprises: inputting sample multi-modal vector sequences into an initial large language model to determine the similarity between the key vectors in the knowledge base of the initial large language model and the sample multi-modal vector sequences; determining the causal reasoning result based on the similarity; determining whether the loss function of the initial large language model converges according to the causal reasoning result; if the loss function of the initial large language model converges, the initial large language model is determined as a trained large language model; If the loss function of the initial large language model does not converge, the parameters of the initial large language model are adjusted, and the step of inputting the sample multi-modal vector sequence into the initial large language model, determining the similarity between the key vector in the knowledge base of the initial large language model and the sample multi-modal vector sequence is executed again.

5. A large language model-based multi-modal causal reasoning apparatus, characterized by, The device comprises: The mapping unit is configured to map the received multi-modal data to a vector representation space through a preset multi-modal encoder; the multi-modal encoder is trained in advance based on sample multi-modal data and a causal contrast loss function; the causal contrast loss function is determined by a contrast learning loss function and a causal regularization term; The construction unit is configured to construct a multi-modal vector sequence corresponding to the multi-modal data based on the vector representation space; The recognition unit is configured to input the multi-modal vector sequence into a preset large language model for recognition to obtain a causal reasoning result; the large language model is trained in advance according to sample multi-modal vector sequences; The feature extraction unit is configured to extract features of sample multi-modal data using an initial multi-modal encoder to obtain sample feature vectors of the multi-modal data; The projection unit is configured to project the sample feature vectors of the multi-modal data into sample intermediate vectors through a dynamic low-rank projection matrix in the initial multi-modal encoder; The creation unit is configured to process the sample intermediate vectors based on a preset alignment matrix and a preset layer normalization in the initial multi-modal encoder, and create a causal correlation matrix based on the processing result; The first determination unit is configured to, if the causal contrast loss function converges, determine the initial multi-modal encoder as a trained multi-modal encoder; The first adjustment unit is configured to, if the causal contrast loss function does not converge, adjust the dynamic low-rank projection matrix and the alignment matrix, and return to execute the feature extraction unit. The preset multi-modal encoder comprises a dynamic low-rank projection matrix, a preset alignment matrix, and a preset layer normalization; 6. The apparatus of claim 5, wherein, The mapping unit comprises: The feature extraction module is configured to, when receiving multi-modal data, extract features of data of each modality in the multi-modal data to obtain feature vectors of each modality; The projection module is configured to project the feature vectors of each modality into intermediate vectors using a dynamic low-rank projection matrix; The mapping module is configured to map the intermediate vectors of each modality to a vector representation space based on a preset alignment matrix and a preset layer normalization. The recognition unit comprises:

7. The apparatus of claim 5, wherein, The input module is configured to input the multi-modal vector sequence into a preset large language model; The first determination module is configured to determine the similarity between the key vector in the knowledge base of the large language model and the multi-modal vector sequence; The second determination module is configured to determine a causal reasoning result based on the similarity. Further comprising:

8. The apparatus of claim 5, wherein, An input unit is configured to input a sample multi-modal vector sequence into an initial large language model, and determine the similarity between a key vector in a knowledge base of the initial large language model and the sample multi-modal vector sequence. ​ A second determining unit is configured to determine a causal inference result based on the similarity; A second judging unit is configured to judge whether a loss function of the initial large language model converges according to the causal inference result; A third determining unit is configured to determine the initial large language model as a trained large language model if the loss function of the initial large language model converges. A second adjusting unit is configured to adjust parameters of the initial large language model and return to execute the input unit if the loss function of the initial large language model does not converge.

Images