A method and system for multimodal interactive generation pre-training model

Abstract

The application belongs to the technical field of model pre-training, and particularly relates to a multi-modal interactive generation pre-training model method and system, which comprises the following steps: inputting input data into a multi-modal model for forward operation within a training time sequence, extracting a target neuron activation state to form a sequence, solving a mode-dependent factor, fusing to obtain a neuron historical time sequence encoding containing a short-term context vector and a long-term activation baseline, aggregating neighborhood time sequence encoding through a graph aggregation network to obtain neighborhood correction encoding, combining long and short-term feature differences to obtain a modulation coefficient to complete encoding scaling, respectively correcting original connection weights and activation functions of neurons, calculating a main pre-training loss and an auxiliary loss based on an optimized model output, taking a weighted sum of the two as an optimization target, and completing multi-modal model parameter iterative updating through back propagation. The application can realize double dynamic fine-tuning of neuron weights and activation functions.

Description

Technical Field

[0001] This invention relates to the field of model pre-training technology. More specifically, this invention relates to a method and system for generating pre-trained models based on multimodal interactive methods. Background Technology

[0002] Multimodal models can achieve deep alignment and semantic understanding of cross-modal features through joint learning on massive amounts of heterogeneous data such as images, text, and audio. However, traditional pre-training methods generally rely on static connection weight update rules and fixed activation functions, lacking the ability to track and utilize the activation changes of neurons over historical time steps. This neglect of the historical information of micro-neurons makes the model prone to feature response saturation or catastrophic forgetting of specific modalities when faced with high-frequency alternating inputs from different modalities. At the same time, existing multimodal feature fusion methods highly rely on macro-level attention mechanisms or fusion modules, neglecting the collaborative ability of underlying neurons in processing multimodal inputs over time. This makes it difficult for the model to maintain stable and consistent multimodal feature representations during long and continuous training, severely limiting the pre-training effect and the upper limit of generalization.

[0003] Currently, some improved methods attempt to record the activation values ​​of neurons in the preceding time steps during the training phase to calculate local contextual information, or use tools such as graph neural networks to detect the internal topological structure of the model, aiming to realize information interaction between local neurons and enhance the temporal adaptability of the model by imposing parameter constraints.

[0004] However, most existing temporal coding methods only perform simple moving averages on historical states, failing to precisely identify the essential differences between short-term fluctuations and long-term stable baselines of neurons. Furthermore, they lack quantitative representations of the specific contributions of different input modalities during activation, resulting in a lack of multimodal perception capabilities in the generated historical features. Simultaneously, existing topological interaction methods mostly remain at the level of coarse-grained global parameter regularization, lacking a modulation mechanism that organically combines the aggregation information of the neuron's neighborhood with the differences in the target neuron's own short- and long-term states. This makes it difficult to achieve dual fine-tuning of the target neuron's connection weights and activation functions at specific training time steps. Moreover, existing modulation strategies are often detached from the overall loss optimization system of the model, lacking collaborative supervision of primary and secondary losses. This prevents the full activation of the representational potential of micro-neurons while ensuring convergence stability, thus limiting further breakthroughs in the overall performance of multimodal models. Summary of the Invention

[0005] To address the technical problem of the inability to precisely perceive modal contributions and the dynamic adjustment of neurons, this invention proposes a method and system for generating pre-trained models based on multimodal interactive generation, which can achieve dual dynamic fine-tuning of neuron weights and activation functions.

[0006] In a first aspect, the present invention provides a method for generating a pre-trained model based on multimodal interactive generation, comprising: S1, acquiring input data in consecutive training time steps and feeding it into the forward propagation of the multimodal model, employing a forward calculation mechanism twice in each training time step to extract activation states of target neurons within the model to construct an activation state sequence, calculating modality dependency factors representing contributions from different input modalities for the activation states in each training time step, and generating historical temporal codes for target neurons; S2, delineating neuron neighborhoods based on the network topology of the multimodal model, aggregating information from the historical temporal codes of each neuron within the neuron neighborhood to generate neighborhood correction codes, and generating neighborhood correction codes based on short-term context vectors and long-term activation bases. The modulation coefficients are calculated based on the relative differences of the lines. The modulation coefficients are used to scale the neighborhood correction code, and weight adjustment vectors and activation adjustment parameters are generated by linear projection. The weight adjustment vectors are superimposed on the original connection weights of the target neuron. The cache of the first exploratory forward propagation is cleared so that the updated parameters can be applied to forward inference. S3, the same batch of input data from the current time step is used to perform a second formal forward propagation on the multimodal model after the above parameter and activation adjustment. The main pre-training loss is calculated based on the output of this formal forward propagation, and the auxiliary loss is calculated. The parameters of the multimodal model are updated by backpropagation using the weighted sum of the main pre-training loss and the auxiliary loss.

[0007] By employing the above technical solutions, a historical temporal encoding containing short-term context and long-term activation baseline is constructed to accurately represent the contribution of multimodal temporal features. Simultaneously, by combining the neighborhood correction encoding generated by the graph aggregation network with the adaptive modulation coefficients based on the difference between long and short time series, weight adjustment vectors and activation adjustment parameters are dynamically generated. Within a single training time step, dual fine-tuning of neuron connection weights and activation functions is completed. Parameter updates are performed under joint supervision of master and auxiliary losses, significantly enhancing the model's dynamic adaptability to multimodal inputs, accelerating pre-training convergence, and improving overall generalization performance and representation quality.

[0008] Preferably, generating the modality dependency factor includes: obtaining the feature representation vectors corresponding to each input modality in the feature embedding layer; mapping the state vector formed by concatenating the hidden states of the target neuron at each training time step within a preset time window and the feature representation vectors of each input modality to the same feature dimension through a linear projection layer; using the mapped state vector as the query vector and the mapped feature representation vector as the key vector to calculate the multi-head attention weights; calculating the inner product of the feature representation vectors of each input modality and the weight vector of the target neuron to obtain the feature projection value; performing weighted processing on the feature projection value to generate the original feature contribution scalars for each input modality; concatenating all the original feature contribution scalars at the same training time step into a multi-dimensional input vector, inputting it into a hyperbolic tangent function for nonlinear mapping to generate a feature vector containing the contribution direction and intensity of each input modality, which serves as the modality dependency factor.

[0009] By adopting the above technical solution, a multi-head attention mechanism is introduced to calculate the correlation weight between each input modality and the neuron state. The modality dependency factor is generated by combining the feature projection value and the hyperbolic tangent function nonlinear mapping. This can accurately characterize the contribution direction and intensity of different modalities to the activation of the target neuron, enabling historical time-series encoding to have fine-grained multimodal perception capabilities and providing a reliable modality contribution basis for subsequent dynamic adjustment.

[0010] Preferably, the generation of the historical temporal encoding of the target neuron includes: multiplying the activation states of the target neuron at each training time step within a preset time window before the current training time step as weights with the modality dependency factor of the corresponding training time step, and concatenating them to obtain a modality-weighted activation sequence; inputting the modality-weighted activation sequence into a long short-term memory network in chronological order to obtain the hidden state vector of the last training time step, as the short-term context vector; using an exponential moving average algorithm to iteratively update the mathematical expectation and variance of the activation state values ​​of the target neuron at the current training time step, and concatenating the updated mathematical expectation and variance into a vector, as the long-term activation baseline; and in the tensor concatenation layer, concatenating the short-term context vector and the long-term activation baseline along the feature dimension to generate the historical temporal encoding of the target neuron.

[0011] By adopting the above technical solution, LSTM is used to perform temporal modeling of modality-weighted activation sequences to extract short-term context vectors. At the same time, exponential moving average is used to iteratively update the mathematical expectation and variance of activation to construct a long-term activation baseline. The two are then concatenated into a historical temporal code, which effectively distinguishes the transient fluctuations and long-term stable characteristics of neuronal activity, and provides rich temporal feature expressions for characterizing the dynamic changes of neurons.

[0012] Preferably, generating the neighborhood correction code includes: constructing a graph adjacency matrix based on the neuron connection relationships of the hidden layer of the multimodal model; determining the set of first-order connected neurons of the target neuron according to the graph adjacency matrix as the neuron's neighborhood; calculating the inner product of the historical temporal code of the target neuron and the historical temporal codes of each neuron in the neuron's neighborhood, and obtaining the edge connection weights of each adjacent neuron after normalization; calculating the product of the historical temporal code of each neuron in the neuron's neighborhood and the corresponding edge connection weight, and summing and aggregating them to obtain an aggregated feature vector; and performing a residual connection and linear mapping between the aggregated feature vector and the historical temporal code of the target neuron to generate the neighborhood correction code.

[0013] By adopting the above technical solution, the neighborhood of a neuron is delineated based on the graph adjacency matrix, and the attention aggregation of neighborhood information is realized by calculating the edge connection weight through normalized inner product. Then, the neighborhood correction code is generated through residual connection and linear mapping. This can effectively integrate the group temporal state in the local topology, enhance the target neuron's ability to capture and correct neighborhood context information, and improve the stability and consistency of feature representation.

[0014] Preferably, the calculation of modulation coefficients includes: aligning the short-term context vector and the long-term activation baseline to the same dimension through corresponding linear mapping layers, and then subtracting them to obtain a relative difference vector; inputting the relative difference vector into a multilayer perceptron for feature projection and outputting a difference scalar; inputting the difference scalar into a sigmoid activation function for nonlinear mapping, and scaling the mapping result to a preset range to obtain the modulation coefficients.

[0015] By adopting the above technical solution, the relative difference vector is obtained after aligning the short-term context vector with the long-term activation baseline, and the modulation coefficients are generated using a multilayer perceptron and a sigmoid activation function. These coefficients are then constrained to a preset range, which can adaptively characterize the degree to which the current state of the neuron deviates from the historical baseline. This provides a dynamic adjustment signal for scaling the subsequent neighborhood correction coding, thereby achieving fine-grained modulation amplitude control.

[0016] Preferably, the step of generating the weight adjustment vector and activation adjustment parameters through linear projection includes: inputting the scaled neighborhood correction code into a first fully connected layer for dimensionality reduction compression, inputting it into a second fully connected layer for dimensionality increase mapping, and generating the weight adjustment vector with the same dimension as the connection weights of the target neuron; inputting the scaled neighborhood correction code into a parameter prediction network containing a multilayer perceptron, and outputting a first mapping scalar and a second mapping scalar; using the first mapping scalar as an activation bias term, the second mapping scalar as an activation gain term, and combining the activation bias term and the activation gain term to form the activation adjustment parameters used to adjust the activation function of the target neuron.

[0017] Preferably, the calculation of the master pre-training loss includes: randomly selecting a fixed proportion of image patches and text words in the input data for masking; feeding the masked input data into the multimodal model to generate predicted modal features; inputting the original unmasked input data into a parameter-frozen teacher model to extract target high-dimensional semantic features corresponding to the mask positions; calculating the mean square error between the predicted modal features and the target high-dimensional semantic features; averaging the mean square errors of all mask positions to obtain the reconstruction loss; and feeding the reconstruction loss back to the multimodal model as the master pre-training loss at the current training time step.

[0018] Preferably, the calculation of the auxiliary loss includes: extracting the second historical temporal code generated by the target neuron in the multimodal model after data augmentation of the same input data, as a positive sample; projecting the historical temporal code of the target neuron and the positive sample to the same dimension through a linear mapping layer, and calculating the cosine similarity between the projected historical temporal code and the projected positive sample as the consistency score of the positive sample pair; randomly selecting the historical temporal code generated by the target neuron during forward propagation of other input data from the current training batch as a negative sample, projecting the negative sample through the corresponding linear mapping layer, and calculating the cosine similarity between the projected historical temporal code and the projected negative sample as the consistency score of the negative sample pair.

[0019] Secondly, this invention provides a multimodal interactive pre-trained model generation system, comprising the following modules: a generation module, which acquires input data in consecutive training time steps and sends it into the forward propagation of the multimodal model; employs a forward calculation mechanism twice in each training time step to extract activation states of target neurons within the model and construct an activation state sequence; calculates modality dependency factors representing the contributions of different input modalities for the activation states in each training time step; and generates historical temporal codes for target neurons; and an adjustment module, which delineates neuron neighborhoods based on the network topology of the multimodal model; aggregates information from the historical temporal codes of each neuron within the neuron neighborhood to generate neighborhood correction codes; and adjusts the codes based on short-term context vectors and long-term activation vectors. The modulation coefficients are calculated based on the relative differences of the baseline. The modulation coefficients are used to scale the neighborhood correction code, and weight adjustment vectors and activation adjustment parameters are generated by linear projection. The weight adjustment vectors are superimposed on the original connection weights of the target neuron. The cache of the first exploratory forward propagation is cleared so that the updated parameters can be applied to forward inference. The calculation module uses the same batch of original input data from the current time step and feeds it into the multimodal model after the above parameter and activation adjustment to perform a second formal forward propagation. Based on the output of this formal forward propagation, the main pre-training loss is calculated, the auxiliary loss is calculated, and the parameters of the multimodal model are updated by backpropagation using the weighted sum of the main pre-training loss and the auxiliary loss.

[0020] By organically combining the generation module, adjustment module, and calculation module, the activation state recording of exploratory forward propagation, modality dependency factor calculation, neighborhood correction encoding generation, parameter fine-tuning, and joint optimization of primary and secondary losses are integrated into a unified system. This system can systematically perform dynamic adjustment of neurons and collaborative loss updates during multimodal pre-training, thereby improving model training efficiency, representation accuracy, and generalization ability.

[0021] The present invention has the following beneficial effects: This invention generates a historical temporal code that integrates short-term context and long-term baseline by constructing a sequence of neuronal activation states and calculating modality dependence factors. This effectively represents and preserves the feature contributions of different modalities in the temporal dimension, thereby improving the perception accuracy of multimodal information interaction. Furthermore, this invention combines network topology with graph aggregation networks to integrate the temporal codes of neighboring neurons to generate neighborhood correction codes, and calculates modulation coefficients based on the differences in long and short time-course characteristics for scaling, thereby achieving the fusion and fine calibration of state information among local neuron groups; then, the scaled codes are converted into weight adjustment vectors and activation adjustment parameters to dynamically optimize neuron connection weights and activation functions, and updated through backpropagation under joint supervision of master and auxiliary losses, thereby accelerating pre-training convergence and enhancing the model's generalization performance and multimodal representation quality. Attached Figure Description

[0022] Figure 1 This is a flowchart of a method for generating pre-trained models based on multimodal interactive methods; Figure 2 This is a schematic diagram illustrating the changes in modality dependency factors at different training time steps; Figure 3 This is a schematic diagram showing the distribution of edge connection weights among adjacent neurons within the neighborhood of a neuron. Detailed Implementation

[0023] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0024] Example 1 In Embodiment 1 of the present invention, as Figure 1 As shown, a method for generating pre-trained models based on multimodal interactive methods includes the following steps: S1. Input data is acquired in consecutive training time steps and fed into the forward propagation of the multimodal model. Two forward calculation mechanisms are used in each training time step, including the first exploratory forward propagation and the second formal forward propagation. First, the exploratory forward propagation is performed. At each training time step, the activation state of the target neuron in the model is extracted to construct the activation state sequence. The modality dependency factor representing the contribution of different input modalities is calculated for the activation state of each training time step. Based on the activation state sequence and the modality dependency factor, the historical temporal code of the target neuron is generated. The historical temporal code includes the short-term context vector and the long-term activation baseline.

[0025] A multimodal pre-training dataset containing images and text is loaded. In consecutive training time steps, each batch of data is input into a Transformer-based multimodal model for forward propagation. A forward hook function `register_forward_hook` is registered in the multilayer perceptron layer of the multimodal model. At each training time step, the one-dimensional tensor output of the target neuron is intercepted and extracted as the current activation state, and the activation states are stored in a pre-initialized double-ended queue in chronological order. When the queue length reaches a preset time window size, the oldest activation state is removed to construct a fixed-length activation state sequence. Modality dependency factors are calculated, and the attention weight matrix generated by the cross-modal attention module during forward propagation in the multimodal model is extracted. For both image and text modalities, the attention weights for the corresponding modal feature labels are averaged along the sequence length dimension using average pooling.

[0026] The average pooling feature values ​​of the two modalities are normalized using the Softmax function to obtain scalar values ​​between 0 and 1 that sum to one. These scalar values ​​serve as modality dependency factors representing the contribution of each input modality. For the generation of historical temporal codes, the activation state tensor of each time step in the activation state sequence is multiplied element-wise with the corresponding modality dependency factor to perform feature weighting. The weighted activation state sequence is then input into a Long Short-Term Memory (LSTM) network in time step order. The hidden state vector output by the LSTM at the last time step is extracted as the short-term context vector. An exponential moving average algorithm is used to continuously and smoothly update the weighted activation states of all historical time steps, with the smoothing decay hyperparameter set to 0.99. The current time step's weighted activation state is multiplied by 0.01 and then added to the previous time step's long-term activation baseline multiplied by 0.99 to iteratively calculate the current long-term activation baseline.

[0027] In some embodiments, calculating a modality dependency factor representing the contribution of different input modalities for the activation state at each training time step includes: Obtain the feature representation vectors of each input modality in the feature embedding layer at the current training time step; The state vector, which is formed by concatenating the hidden states of the target neuron at each training time step within a preset time window, and the feature representation vectors of each input modality are respectively mapped to the same feature dimension through a linear projection layer. The mapped state vector is used as the query vector, and the mapped feature representation vector is used as the key vector to calculate the multi-head attention weights. Calculate the inner product of the feature representation vector of each input modality and the weight vector of the target neuron to obtain the feature projection value; The feature projection values ​​are weighted according to the multi-head attention weights to generate the original feature contribution scalars for each input modality; All original feature contribution scalars extracted at the same training time step are concatenated into a multidimensional input vector, and then input into the hyperbolic tangent function for nonlinear mapping to generate feature vectors containing the contribution direction and intensity of each input mode. The feature vectors are used as mode dependence factors.

[0028] The multimodal model's input includes at least two modalities: visual and text. At the feature embedding layer, visual and text feature representation vectors with a dimension of 1024 are generated respectively. The preset time window length T is set to 16. The hidden states of the target neurons over the past 16 time steps are extracted and concatenated along the time dimension to form a state vector. This vector is then reduced to 512 dimensions through a first linear projection layer containing a 1024×512 dimension weight matrix. The 1024-dimensional feature representation vectors of each modality are also linearly mapped to 512 dimensions through independent fully connected layers. The mapped state vectors are used as query vectors, and the mapped modal feature representation vectors are used as key vectors, inputting into a multi-head attention network. The multi-head attention network structure includes multiple parallel scaling dot product calculation branches and a concatenating and fusing output layer. The inputs are the query vector and the key vector, and the output is the multi-head attention weight distribution.

[0029] Preferably, the network is configured with 8 attention heads, and attention weights of length 8 are calculated. For each input modality, the inner product of the 1024-dimensional feature representation vector and the synaptic connection weight vector of the target neuron of length 1024 is calculated to obtain the feature projection value. The feature projection values ​​are weighted and summed according to the 8-head attention weights to generate the original feature contribution scalars for each input modality. When the input is configured for joint learning of image, text, and speech modalities, the three extracted original feature contribution scalars are concatenated at a single training time step to form a multi-dimensional input vector of length 3. This multi-dimensional input vector is input into the hyperbolic tangent activation function for nonlinear mapping, so that the value of each dimension of the output vector is precisely within the continuous closed interval [-1, 1], generating a normalized feature vector containing the contribution direction and intensity of each input modality, which serves as a modality dependency factor, such as... Figure 2 As shown, this illustrates the changing patterns of different modal contribution weights during training, reflecting the target neuron's response bias to various modal information and supporting the rationality of the modal contribution representation method.

[0030] In some embodiments, a historical temporal encoding of the target neuron is generated based on the activation state sequence and modality-dependent factors. The historical temporal encoding includes a short-term context vector and a long-term activation baseline, including: The activation states of the target neuron in each training time step within a preset time window before the current training time step are used as weights and multiplied with the modality dependency factor of the corresponding training time step, and then concatenated to obtain the modality-weighted activation sequence. The modality-weighted activation sequence is input into the Long Short-Term Memory network in chronological order to obtain the hidden state vector of the last training time step, which is used as the short-term context vector. The exponential moving average algorithm is used to iteratively update the expected value and variance of the target neuron activation state value at the current training time step. The updated expected value and variance are concatenated into a vector and used as the long-term activation baseline. In the tensor concatenation layer, the short-term context vector and the long-term activation baseline are concatenated along the feature dimension to generate the historical temporal encoding of the target neuron.

[0031] Using a preset time window length of 16, the activation state values ​​of the target neuron are extracted sequentially over the past 16 consecutive time steps. For each time step, the scalar activation state value of that time step is used as a weight and multiplied by a scalar-vector multiplication operation with the 3D modality dependency factor obtained at the same time step. These 16 modulated 3D vectors are then sequentially combined in chronological order to generate a 16×3 matrix, which serves as the modality-weighted activation sequence. This entire 2D sequence is then input into a Long Short-Term Memory (LSTM) network. The LTM network structure consists of a sequentially connected input gate, a forget gate, a cell state update module, and an output gate. The input is the modality-weighted activation sequence matrix of each time step, and the output is the hidden state vector extracted along the temporal sequence.

[0032] The network is configured with a two-layer structure, with 512 hidden neurons. The 512-dimensional hidden state vector output at the 16th time step is used as the short-term context vector. An exponential moving average algorithm is used to iteratively update the expected value and variance of the target neuron's activation state values, with a decay constant factor set. =0.999, update expected value is and update the variance to ,in This represents the current activation state value. The updated mathematical expectation scalar and variance scalar are concatenated to obtain a 2-dimensional long-term activation baseline. In the tensor concatenation layer, the generated 1×512-dimensional short-term context vector and the 1×2-dimensional long-term activation baseline are concatenated in parallel along the feature dimensions to generate a 514-dimensional historical temporal code.

[0033] S2. Based on the network topology structure of the multimodal model, the neighborhood of a neuron is defined. The historical temporal codes of each neuron in the neighborhood are aggregated using a graph aggregation network to generate a neighborhood correction code. The modulation coefficient is calculated based on the relative difference between the short-term context vector and the long-term activation baseline. The neighborhood correction code is scaled using the modulation coefficient. The scaled neighborhood correction code is then used to generate a weight adjustment vector and an activation adjustment parameter through linear projection. The weight adjustment vector is superimposed on the original connection weights of the target neuron, and the activation adjustment parameter is used to adjust the activation function of the target neuron. The cache of the first probing forward propagation is cleared so that the updated parameters can be applied to forward inference.

[0034] Graph data objects are constructed based on the fully connected network topology of a multilayer perceptron in a multimodal model. Each neuron within the same layer is considered a graph node, and a neuronal neighborhood of the target neuron is constructed based on the neuron connections. A graph aggregation network is used as the information aggregation component. The historical temporal encoding of each neuron within the neighborhood (i.e., the concatenated tensor of the short-term context vector and the long-term activation baseline) is input into the graph aggregation network as the initial feature of the node. The correlation score between the features of the target node and its neighbors is calculated. Based on this score, a weighted sum is performed on the historical temporal encodings of all neighboring nodes, and the result is processed through a LeakyReLU nonlinear activation function to generate a high-dimensional neighborhood correction encoding tensor. To calculate the modulation coefficients, the short-term context vector and the long-term activation baseline are aligned to the same dimension through corresponding linear mapping layers and then subtracted to obtain a relative difference vector. This relative difference vector is input into the multilayer perceptron for feature projection, outputting a difference scalar.

[0035] The calculated Euclidean distance value, after subtracting a preset smoothing bias threshold, is input into the Sigmoid activation function. The output is mapped to the 0-1 interval as modulation coefficients. Tensor multiplication is used to multiply the calculated modulation coefficient scalar with the high-dimensional neighborhood correction encoding tensor to achieve proportional scaling of feature intensity. The scaled neighborhood correction encoding is fed in parallel into the corresponding network branches to perform mapping operations. One side of the network layer outputs a weight adjustment vector with the same dimension as the target neuron's connection weights. The other side's parameter prediction network outputs activation adjustment parameters composed of two scalars: an activation bias term and an activation gain term. During the model parameter adjustment stage, after calculation based on probing forward propagation, matrix addition is used to add the weight adjustment vector element-wise and overlay it onto the original connection weight tensor corresponding to the target neuron in the multilayer perceptron network. The activation bias term and activation gain term are combined to construct an activation adjustment expression to replace the original activation function input, thereby changing the form of the activation function.

[0036] In some embodiments, the neural neighborhood is delineated based on the network topology of a multimodal model, and a graph aggregation network is used to aggregate the historical temporal codes of each neuron in the neighborhood to generate a neighborhood correction code. This includes: constructing a graph adjacency matrix based on the neuron connection relationships of the hidden layer of the multimodal model; determining the set of first-order connected neurons of the target neuron as the neuron neighborhood based on the graph adjacency matrix; calculating the inner product of the historical temporal code of the target neuron and the historical temporal codes of each neuron in the neighborhood, and obtaining the edge connection weights of each adjacent neuron after normalization; calculating the product of the historical temporal code of each neuron in the neighborhood and the corresponding edge connection weight, and summing and aggregating the products to obtain an aggregated feature vector; and performing a residual connection and linear mapping between the aggregated feature vector and the historical temporal code of the target neuron to generate a neighborhood correction code.

[0037] The graph aggregation network structure includes an adjacency calculation module for extracting node feature information, a feature aggregation layer, and a residual normalization output layer. The input is the topological graph relationships of the neural nodes in the multimodal model and the historical temporal codes of all nodes; the output is a neighborhood correction code. The connection weights of the hidden fully connected layers of the multimodal model are extracted, and undirected connections with absolute values ​​greater than 0.05 are extracted to generate a graph adjacency matrix. Based on this graph adjacency matrix, topological nodes with first-order associations with the target neuron are determined, and 40 neighboring neurons are extracted to form the neuron's neighborhood. The 514-dimensional historical temporal code of the target neuron and the 514-dimensional historical temporal codes of each of the 40 neighboring neurons are extracted. The inner product of the historical temporal code of the target neuron and the historical temporal codes of each neighboring neuron is calculated to obtain a similarity score of length 40.

[0038] The 40 scores are processed using a normalization classifier such as the Softmax function to generate 40 coefficients that sum to 1 and whose values ​​are in the range [0,1]. These coefficients are then used as edge connection weights. Figure 3 As shown, this illustrates the differences in the correlation strength between neurons in the topological structure. Based on the obtained weight coefficients, the 514-dimensional historical temporal codes of the 40 corresponding neighboring neurons are weighted, and the weighted vectors are directly summed to obtain a 514-dimensional aggregated feature vector. This summation is used to establish a residual connection between the aggregated feature vector and the historical temporal code of the target neuron. The summed residuals are then input into a 514×514 linear mapping layer for linear transformation and smoothing calculations, generating a 514-dimensional neighborhood correction code.

[0039] In some embodiments, calculating the modulation coefficients based on the relative difference between the short-term context vector and the long-term activation baseline includes: aligning the short-term context vector and the long-term activation baseline to the same dimension through corresponding linear mapping layers and then subtracting them to obtain a relative difference vector; inputting the relative difference vector into a multilayer perceptron for feature projection and outputting a difference scalar; inputting the difference scalar into a sigmoid activation function for nonlinear mapping, and scaling the mapping result to a preset range to obtain the modulation coefficients.

[0040] For the 512-dimensional short-term context vector and the 2-dimensional long-term activation baseline, corresponding linear mapping layers are established. The short-term context vector is compressed by dimensionality reduction through the 512×128-dimensional linear mapping layer; the long-term activation baseline is mapped by dimensionality increase through the 2×128-dimensional linear mapping layer, thus aligning them to a unified 128-dimensional space. The aligned short-term context vector and the long-term activation baseline are subtracted element-wise to extract a 128-dimensional relative difference vector representing the relative difference shift intensity. The 128-dimensional relative difference vector is then input into a multilayer perceptron for feature compression. The multilayer perceptron structure includes a feedforward fully connected input layer, a nonlinear activation hidden layer, and a linear fully connected output layer. The input is a one-dimensional relative difference vector, and the output is a difference scalar. The hidden layer of this multilayer perceptron has 64 neurons and uses the error linear unit activation function. The output layer is a fully connected layer with a unique size of 1, which outputs a single-element difference scalar through forward propagation. The difference scalar input logic curve activation function is nonlinearly mapped to the interval [0,1]. The mapping result is then linearly scaled using the equation Y=0.4X+0.8 to scale the boundary constraints to a preset range of [0.8,1.2], thus obtaining the modulation coefficient used to adjust the global fine-tuning amplitude.

[0041] In some embodiments, the scaled neighborhood correction code is used to generate a weight adjustment vector and an activation adjustment parameter through linear projection. The weight adjustment vector is superimposed on the original connection weights of the target neuron, and the activation adjustment parameter is used to adjust the activation function of the target neuron. The cache of the first probing forward propagation is cleared so that the updated parameters can be applied to forward inference. This includes: inputting the scaled neighborhood correction code into a first fully connected layer for dimensionality reduction compression, inputting it into a second fully connected layer for dimensionality increase mapping, generating a weight adjustment vector with the same dimension as the connection weights of the target neuron; inputting the scaled neighborhood correction code into a parameter prediction network containing a multilayer perceptron, outputting a first mapping scalar and a second mapping scalar; using the first mapping scalar as an activation bias term, the second mapping scalar as an activation gain term, and combining the activation bias term and the activation gain term to form the activation adjustment parameter used to adjust the activation function of the target neuron.

[0042] The scaled 514-dimensional neighborhood correction code is input into the first fully connected layer, whose dimension is set to 514×128, to compress the correction code. It is then sequentially input into the second fully connected layer for dimension upscaling, with the dimension set to 128×1024, ensuring the output weights match the 1024-dimensional target neuron connections. This generates a 1024-dimensional weight adjustment vector sequence. In parallel processing, the scaled neighborhood correction code is input into the parameter prediction network, which is a multilayer perceptron with 128 hidden nodes and independent branches. The output is deconstructed into two scalar values: the first mapping scalar and the second mapping scalar. To limit the adjustment amplitude, a numerical truncation mechanism is implemented so that the output first mapping scalar is confined to the closed interval [-0.1, 0.1], serving as the activation bias term. The second mapping scalar obtained from the corresponding prediction is constrained to the range [0.9, 1.1] using a smoothing operator and used as the activation gain term. The combined activation adjustment parameters of the synthetic regulation closed-loop formula are used to replace the original activation function input. The activation adjustment expression is: ,in The original activation value. This is the newly activated state value.

[0043] S3. Using the same batch of original input data from the current time step, feed it into the multimodal model after adjusting the above parameters and activation to perform the second formal forward propagation. Calculate the main pre-training loss and the auxiliary loss based on the output of this formal forward propagation. Update the parameters of the multimodal model through backpropagation using the weighted sum of the main pre-training loss and the auxiliary loss.

[0044] After the multimodal model, adjusted for parameters and activation, completes its second formal forward propagation, the prediction results of various features output by the model are obtained. For the masked language modeling task in the text branch, the cross-entropy loss function is used to calculate the cross-entropy loss between the probability distribution of the predicted words in the text and the true word labels. For the image-text contrastive learning task, the InfoNCE loss function is used to calculate the contrastive loss between global image features and global text features in the joint feature space. The cross-entropy loss and the contrastive loss are directly added to form the main pre-training loss. For the calculation of the auxiliary loss, the weight adjustment vector and activation adjustment parameters generated by all target neurons in the network at the current time step are extracted. The mean squared error function is used to calculate the L2 norm regularization penalty term of the weight adjustment vector, i.e., the mean of the squares of the weight adjustment vector. The mean of the absolute values ​​of the activation adjustment parameters is calculated. The weight adjustment penalty term and the activation adjustment penalty term are added to form the auxiliary loss used to constrain excessive fluctuations of the model neurons. A constant floating-point number of 0.01 is set as the regularization hyperparameter. The auxiliary loss is multiplied by the regularization hyperparameter and then added to the main pre-training loss to obtain the total loss scalar used to guide gradient descent. The gradient matrix of the total loss scalar with respect to all learnable parameters within the multimodal model is calculated using the backpropagation function. The AdamW optimizer engine is then invoked to perform a weight update operation, and the overall parameter weights of the multimodal model are iteratively updated using the calculated gradient matrix and the learning rate scheduled by the cosine annealing algorithm.

[0045] In some alternative embodiments, the same batch of original input data from the current time step is fed into the multimodal model adjusted by the above parameters and activation to perform a second formal forward propagation. The specific method for calculating the master pre-training loss based on the output of this formal forward propagation may also include: randomly selecting a fixed proportion of image patches and text words in the input data for masking; feeding the masked input data into the multimodal model to generate predicted modal features; inputting the original unmasked input data into the teacher model with frozen parameters to extract the target high-dimensional semantic features corresponding to the mask positions; calculating the mean square error between the predicted modal features and the target high-dimensional semantic features; averaging the mean square errors of all mask positions to obtain the reconstruction loss; and feeding the reconstruction loss back to the multimodal model as the master pre-training loss at the current training time step.

[0046] When preparing the training batch data, each 256×256 pixel original image within the sample is segmented into non-overlapping 16×16 pixel grids as image patches. Based on the Bernoulli distribution masking method, 75% of the image patches are randomly selected for masking, and the original pixel information is replaced by non-learnable grayscale blocks. For text data, after word segmentation, 15% of the text words are randomly selected for masking and replacement. The masked input data is fed into the forward propagation of the multimodal model to be fine-tuned to complete the computation processing, and the 1024-dimensional predicted features corresponding to the mask positions are output to generate predicted modal features. A teacher model with gradient update canceled and network parameters locked is adopted. The network structure of the teacher model includes a dual-stream image and text input embedding layer, a self-attention feature encoding extraction layer, and a multimodal alignment mapping output module.

[0047] The original, unmasked input data is fed into the teacher model, and 1024-dimensional clean features corresponding to the mask positions are extracted as the target high-dimensional semantic features. Based on the difference between the predicted modal features and the target high-dimensional semantic features, the array error is calculated using the mean squared error formula. The arithmetic mean of the individual error results for all mask positions is calculated to obtain the reconstruction loss. This value is used as the gradient error propagation for the current training batch and fed back into the multimodal model to be fine-tuned as the main pre-training loss.

[0048] In some embodiments, calculating the auxiliary loss includes: extracting the second historical temporal code generated by the target neuron in the multimodal model after data augmentation of the same input data, as a positive sample; projecting the historical temporal code of the target neuron and the positive sample to the same dimension through a linear mapping layer, and calculating the cosine similarity between the projected historical temporal code and the projected positive sample, as the consistency score of the positive sample pair; randomly selecting the historical temporal code generated by the target neuron during forward propagation of other input data from the current training batch as a negative sample, projecting the negative sample through the corresponding linear mapping layer, and calculating the cosine similarity between the projected historical temporal code and the projected negative sample, as the consistency score of the negative sample pair; and calculating the contrastive learning loss using the InfoNCE loss function based on the consistency scores of the positive sample pair and the negative sample pair, as the auxiliary loss.

[0049] After data augmentation, the same input data is extracted and re-input into the multimodal model for forward propagation. The 514-dimensional second historical temporal code generated by the target neuron is extracted and used as the positive sample to be tested. A fully connected linear matrix transformation layer with a configuration dimension of 514×128 is called to project the original historical temporal code of the target neuron and the newly generated positive sample onto this 128-dimensional space. The cosine similarity between the two equal-width vectors is calculated, and the output value bounded in the interval [-1,1] is used as the consistency score of the positive sample pair. With the total batch data volume configured to be equal to 256, 255 completely different input data from the current training batch are selected after excluding samples from the same source. The historical temporal code generated by the target neuron during the forward propagation of the input data in the multimodal model is extracted and used as the negative sample group.

[0050] After projecting the aforementioned 255 negative samples using the same 514×128 linear matrix transformation layer, the cosine similarity between the projected historical time-series code and each projected negative sample is calculated, serving as the set of consistency scores for negative sample pairs. Based on the temperature coefficient... =0.07 to calculate the contrastive learning loss. Specifically, the consistency scores of each item are scaled using the temperature coefficient. The scaled positive sample score index is placed in the numerator, and the scaled positive sample score index is summed with the scores of all 255 negative samples and placed in the denominator to obtain the quotient. The negative logarithm of the result is then taken to obtain the contrastive learning loss. The learning loss is used as an auxiliary loss for a single training session to guide the adjustment of multimodal model parameters.

[0051] The datasets selected were the multimodal image-text datasets COCO and Flickr30k, which were used for pre-training and fine-tuning testing. The baseline scheme used a standard multimodal converter model with 86 million parameters, and the optimizer adopted the AdamW algorithm with an initial learning rate set to 1 / 100,000. The evaluation metrics used were image retrieval accuracy to text and text retrieval accuracy to image, with the top-ranked matching accuracy selected as the core evaluation data.

[0052] The baseline model, without any proposed innovative modules, achieved an image retrieval accuracy of 64.5% for text and 51.2% for image retrieval. Variant 1, building upon the baseline by adding only modality dependency factor and historical time-series coding modules, achieved an image retrieval accuracy of 68.3% for text and improved text retrieval accuracy to 55.4%. Variant 2, building upon Variant 1 by adding a graph aggregation network to generate neighborhood correction coding and a modulation coefficient calculation mechanism, further improved the image retrieval accuracy to 71.6% for text and 59.8% for image retrieval. The complete technical solution, building upon Variant 2, utilizes an auxiliary loss module incorporating contrastive learning loss, achieving a peak image retrieval accuracy of 75.2% for text and improving text retrieval accuracy to 63.1% for image retrieval.

[0053] Example 2 Embodiment 2 of the present invention proposes a pre-trained model generation system based on multimodal interactive generation, comprising the following modules: The generation module is used to acquire input data in consecutive training time steps and feed it into the forward propagation of the multimodal model. It employs a forward calculation mechanism twice in each training time step. First, it performs a probing forward propagation. Then, it extracts the activation states of the target neurons in the model at each training time step to construct an activation state sequence. It calculates the modality dependency factor representing the contribution of different input modalities for the activation states at each training time step. Based on the activation state sequence and the modality dependency factor, it generates the historical temporal code of the target neurons. The historical temporal code includes a short-term context vector and a long-term activation baseline. The adjustment module is used to delineate the neighborhood of neurons based on the network topology of the multimodal model. It uses a graph aggregation network to aggregate the historical temporal codes of each neuron in the neighborhood to generate neighborhood correction codes. It calculates the modulation coefficients based on the relative difference between the short-term context vector and the long-term activation baseline. It uses the modulation coefficients to scale the neighborhood correction codes. The scaled neighborhood correction codes are then used to generate weight adjustment vectors and activation adjustment parameters through linear projection. The weight adjustment vectors are superimposed on the original connection weights of the target neuron, and the activation adjustment parameters are used to adjust the activation function of the target neuron. The cache of the first probing forward propagation is cleared so that the updated parameters can be applied to forward inference. The calculation module is used to feed the original batch of input data from the current time step into the multimodal model after adjusting the above parameters and activation to perform a second formal forward propagation. Based on the output of this formal forward propagation, the main pre-training loss is calculated, the auxiliary loss is calculated, and the parameters of the multimodal model are updated through backpropagation using the weighted sum of the main pre-training loss and the auxiliary loss.

[0054] In the description of this specification, "multiple" or "several" means at least two, such as two, three or more, unless otherwise expressly and specifically defined.

Claims

1. A method for generating pre-trained models based on multimodal interactive generation, characterized in that, include: S1. Input data is acquired in consecutive training time steps and fed into the forward propagation of the multimodal model. The forward calculation mechanism is used twice in each training time step to extract the activation state of the target neuron in the model and construct the activation state sequence. The modality dependency factor representing the contribution of different input modalities is calculated for the activation state of each training time step to generate the historical temporal code of the target neuron. S2, based on the network topology structure of the multimodal model, the neighborhood of neurons is delineated, the historical temporal codes of each neuron in the neighborhood are aggregated to generate neighborhood correction codes, the modulation coefficient is calculated based on the relative difference between the short-term context vector and the long-term activation baseline, the neighborhood correction codes are scaled using the modulation coefficients, and weight adjustment vectors and activation adjustment parameters are generated by linear projection, respectively. The weight adjustment vectors are superimposed on the original connection weights of the target neuron, the cache of the first exploratory forward propagation is cleared, so that the updated parameters can be applied to forward inference; S3. Using the same batch of original input data from the current time step, feed it into the multimodal model after adjusting the above parameters and activation to perform a second formal forward propagation. Calculate the main pre-training loss and the auxiliary loss based on the output of this formal forward propagation. Update the parameters of the multimodal model through backpropagation using the weighted sum of the main pre-training loss and the auxiliary loss.

2. The method according to claim 1, characterized in that, Generating the modality dependency factor includes: obtaining the feature representation vectors corresponding to each input modality in the feature embedding layer; mapping the state vector formed by concatenating the hidden states of the target neuron at each training time step within a preset time window and the feature representation vectors of each input modality to the same feature dimension through a linear projection layer; using the mapped state vector as the query vector and the mapped feature representation vector as the key vector to calculate the multi-head attention weights; calculating the inner product of the feature representation vectors of each input modality and the weight vector of the target neuron to obtain the feature projection value; performing weighted processing on the feature projection value to generate the original feature contribution scalars for each input modality; concatenating all the original feature contribution scalars at the same training time step into a multi-dimensional input vector, inputting it into a hyperbolic tangent function for nonlinear mapping to generate a feature vector containing the contribution direction and intensity of each input modality, which serves as the modality dependency factor.

3. The method according to claim 1, characterized in that, The generation of the historical temporal encoding of the target neuron includes: multiplying the activation states of the target neuron at each training time step within a preset time window before the current training time step as weights with the modality dependency factor of the corresponding training time step, and concatenating them to obtain a modality-weighted activation sequence; inputting the modality-weighted activation sequence into a long short-term memory network in chronological order to obtain the hidden state vector of the last training time step, as the short-term context vector; using an exponential moving average algorithm to iteratively update the mathematical expectation and variance of the activation state values ​​of the target neuron at the current training time step, and concatenating the updated mathematical expectation and variance into a vector, as the long-term activation baseline; and in the tensor concatenation layer, concatenating the short-term context vector and the long-term activation baseline along the feature dimension to generate the historical temporal encoding of the target neuron.

4. The method according to claim 1, characterized in that, Generating the neighborhood correction code includes: constructing a graph adjacency matrix based on the neuron connection relationships of the hidden layer of the multimodal model; determining the set of first-order connected neurons of the target neuron as the neuron's neighborhood according to the graph adjacency matrix; calculating the inner product of the historical temporal code of the target neuron and the historical temporal codes of each neuron in the neuron's neighborhood, and obtaining the edge connection weights of each adjacent neuron after normalization; calculating the product of the historical temporal code of each neuron in the neuron's neighborhood and the corresponding edge connection weight, and summing and aggregating them to obtain an aggregated feature vector; and performing a residual connection and linear mapping between the aggregated feature vector and the historical temporal code of the target neuron to generate the neighborhood correction code.

5. The method according to claim 1, characterized in that, The calculation of the modulation coefficient includes: The short-term context vector and the long-term activation baseline are aligned to the same dimension through the corresponding linear mapping layer, and then the difference is calculated to obtain the relative difference vector. The relative difference vector is input into a multilayer perceptron for feature projection, and the difference scalar is output. The difference scalar is input into the Sigmoid activation function for nonlinear mapping, and the mapping result is scaled to a preset range to obtain the modulation coefficient.

6. The method according to claim 1, characterized in that, The step of generating the weight adjustment vector and activation adjustment parameters through linear projection includes: The scaled neighborhood correction code is input into the first fully connected layer for dimensionality reduction and compression, and then input into the second fully connected layer for dimensionality increase mapping to generate the weight adjustment vector with the same dimension as the connection weight of the target neuron. The scaled neighborhood correction code is input into a parameter prediction network containing a multilayer perceptron, and the output is a first mapping scalar and a second mapping scalar. The first mapping scalar is used as the activation bias term, and the second mapping scalar is used as the activation gain term. The activation bias term and the activation gain term are combined to form the activation regulation parameter used to adjust the activation function of the target neuron.

7. The method according to claim 1, characterized in that, The calculation of the master pre-training loss includes: A fixed proportion of image blocks and text words are randomly selected from the input data for masking, and the masked input data is then fed into the multimodal model to generate predicted modal features. The original, unmasked input data is input into the teacher model with frozen parameters, and the target high-dimensional semantic features corresponding to the mask position are extracted. Calculate the mean square error between the predicted modal features and the target high-dimensional semantic features, and average the mean square errors of all mask positions to obtain the reconstruction loss; The reconstruction loss is fed back as the main pre-training loss of the multimodal model at the current training time step.

8. The method according to claim 1, characterized in that, The calculation of the auxiliary loss includes: extracting the second historical temporal code generated by the target neuron in the multimodal model after data augmentation of the same input data, as a positive sample; projecting the historical temporal code of the target neuron and the positive sample to the same dimension through a linear mapping layer, and calculating the cosine similarity between the projected historical temporal code and the projected positive sample as the consistency score of the positive sample pair; randomly selecting the historical temporal code generated by the target neuron during forward propagation of other input data from the current training batch as a negative sample, projecting the negative sample through the corresponding linear mapping layer, and calculating the cosine similarity between the projected historical temporal code and the projected negative sample as the consistency score of the negative sample pair.

9. A system for generating pre-trained models based on multimodal interactive generation, characterized in that, The module includes the following components: a generation module, which acquires input data in consecutive training time steps and feeds it into the forward propagation of the multimodal model. It employs a two-step forward calculation mechanism in each training time step to extract activation states from the target neurons within the model, constructing an activation state sequence. It calculates the modality dependency factor representing the contribution of different input modalities for the activation states in each training time step, generating the historical temporal code of the target neurons. An adjustment module, based on the network topology of the multimodal model, delineates the neuron neighborhood. It aggregates the historical temporal codes of each neuron within the neighborhood to generate a neighborhood correction code. It calculates the modulation coefficient based on the relative difference between the short-term context vector and the long-term activation baseline, scales the neighborhood correction code using the modulation coefficient, and generates a weight adjustment vector and activation adjustment parameters through linear projection. It then superimposes the weight adjustment vector onto the original connection weights of the target neurons, clears the cache from the initial exploratory forward propagation, and allows the updated parameters to be applied to forward inference. The calculation module uses the same batch of original input data from the current time step and feeds it into the multimodal model after adjusting the above parameters and activation to perform a second formal forward propagation. Based on the output of this formal forward propagation, it calculates the main pre-training loss and the auxiliary loss. It then uses the weighted sum of the main pre-training loss and the auxiliary loss to update the parameters of the multimodal model through backpropagation.

10. The system according to claim 9, characterized in that, Generating the modality dependency factor includes: obtaining the feature representation vectors corresponding to each input modality in the feature embedding layer; mapping the state vector formed by concatenating the hidden states of the target neuron at each training time step within a preset time window and the feature representation vectors of each input modality to the same feature dimension through a linear projection layer; using the mapped state vector as the query vector and the mapped feature representation vector as the key vector to calculate the multi-head attention weights; calculating the inner product of the feature representation vectors of each input modality and the weight vector of the target neuron to obtain the feature projection value; performing weighted processing on the feature projection value to generate the original feature contribution scalars for each input modality; concatenating all the original feature contribution scalars at the same training time step into a multi-dimensional input vector, inputting it into a hyperbolic tangent function for nonlinear mapping to generate a feature vector containing the contribution direction and intensity of each input modality, which serves as the modality dependency factor.

Images