A method and device for dynamically generating a multimodal hybrid expert model

Abstract

The present invention relates to the field of multimodal models, and specifically provides a method and device for dynamically generating a multimodal hybrid expert model, including the following: S1.1, preprocessing data; S1.2, a vectorization module converts input data into vectors; S1.3, extracting features through a multi-layer Transformer; S1.4, calculating the selection probability of each module in each module pool; S1.5, a dynamic expert module assembles them into an expert in sequence according to the calculated probability; S1.6, an output module generates output through a decoder; S2.1, a pre-training stage; S2.2, a module pool initialization stage; S2.3, a full parameter training stage. Compared with the prior art, the present invention can effectively process complex multimodal data through innovative technologies such as dynamic expert generation, shared layer pools and dynamic routing, while maintaining good adaptability to single-modal tasks.

Description

Technical Field

[0001] The present invention relates to the field of multimodal models, and specifically provides a method and device for dynamically generating a multimodal hybrid expert model. Background Art

[0002] With the rapid development of artificial intelligence technology, multimodal learning has become one of the hot topics in current research. Multimodal learning aims to simultaneously process and fuse information from different modalities (such as text, images, audio, etc.) to achieve more comprehensive and accurate understanding and decision-making. However, traditional multimodal learning methods face many challenges. First, many traditional models often use large neural network structures when processing multimodal data, resulting in a large number of parameters not being fully utilized under specific inputs, causing a waste of computing resources and inefficient parameter utilization. Secondly, although the existing Mixture of Experts (MoE) model can improve the flexibility of the model to a certain extent, the number of experts is usually fixed, which is difficult to dynamically adjust according to the complexity of the input data, and the model lacks flexibility.

[0003] In addition, when processing multimodal data, information from different modalities often requires different processing methods, but existing models are difficult to effectively dynamically adjust their structures for different modalities, resulting in limited modality-specific performance. Finally, many models designed for multimodality do not work well when processing unimodal data, lack versatility, and have poor adaptability to unimodal scenarios.

[0004] In view of the above problems, there is an urgent need for a new model structure that can flexibly process multimodal data while maintaining high efficiency and adapt to task requirements of different complexities. Summary of the invention

[0005] The present invention aims at solving the above-mentioned deficiencies of the prior art and provides a method for dynamically generating a multi-modal hybrid expert model with strong practicability.

[0006] A further technical task of the present invention is to provide a device for dynamically generating a multi-modal hybrid expert model that is reasonably designed, safe and applicable.

[0007] The technical solution adopted by the present invention to solve its technical problem is:

[0008] A method for dynamically generating a multimodal hybrid expert model, when applied to processing multimodal data, includes the following:

[0009] (1) Model construction;

[0010] S1.1, the multimodal input module receives multimodal input data and preprocesses the data;

[0011] For text, word segmentation and stop word removal will be performed; for images, resizing and color space conversion preprocessing will be performed;

[0012] S1.2, the vectorization module converts the input data into vectors;

[0013] The vectorization module uses the multi-modal compatible bge-visualized-m3 to vectorize text and images respectively. The use of the bge-visualized-m3 model allows text and images to be represented in the same vector space.

[0014] After vectorization, the model concatenates the converted vectors and adds separators so that the model can distinguish information from different modalities;

[0015] S1.3, the feature extraction module extracts features from the concatenated vectors through a multi-layer Transformer;

[0016] An 8-layer Transformer layer is used for feature extraction. After feature extraction, the model enters the dynamic routing module and the dynamic expert module;

[0017] S1.4, the dynamic routing module calculates the selection probability of each module in each module pool;

[0018] There are three module pools, corresponding to the three layers in the dynamic expert module. The modules in the first and third module pools are linear layers, and the modules in the second module pool are SwiGLU activation function layers.

[0019] Assuming that there are M modules in each pool, the selection probability of the mth module in the lth pool (l∈1,2,3) is:

[0020]

[0021] where w l，m and b l，m is a learnable parameter, h is the input feature, T refers to the transpose of the matrix, and the softmax function is used to calculate the selection probability of each module. The function of the softmax function is to convert a set of numerical values ​​into a probability distribution, ensuring that all output values ​​are between 0 and 1 and the sum is 1. In this way, the most suitable dynamic expert module is selected based on these probability values;

[0022] S1.5, the dynamic expert module selects the module with the highest probability from each module pool according to the calculated probability, and assembles the selected modules with the highest probability into an expert in sequence;

[0023] Specifically, the structure of the expert is: linear layer → SwiGLU → linear layer. Two groups of modules are selected to assemble two experts E1 and E2. Assume that the selected modules are m1,i , m 2，j , m 3，k and m 1，p , m 2，q , m 3，r , where m 1,i , m 2，j , m 3，k For E1, m 1，p , m 2，q , m 3， r is for E2, then the expert processing process can be expressed as:

[0024] For E1: h1 = m 3，k (SwiGLU(m 2，j (m 1，i (h))))

[0025] For E2: h2 = m 3，r (SwiGLU(m 2，q (m 1，p (h))))

[0026] Where SwiGLU is an activation function, defined as: SwiGLU(x) = x*σ(β*x), σ is the sigmoid function, β is a learnable parameter, and h is the result of the expert operation;

[0027] Then, combine the outputs of the two experts:

[0028] h expert =αh1+(1-α)h2

[0029] Where α is a learnable parameter used to balance the contributions of the two experts;

[0030] Normalization layer: Use layer normalization to process expert output:

[0031] h norm1 =LayerNorm(h expert )

[0032] Attention layer: Use a multi-head self-attention mechanism to process the normalized output:

[0033] h attn =MultiHeadAttention(h norm1 )

[0034] Second Normalization Layer: Layer Normalization is used again to process the output of the attention layer:

[0035] h norm2 =LayerNorm(hattn )

[0036] This structure is repeated n times, and the output of each time is used as the input of the next time. This is repeated 8 times, and the output of the last repetition is used as the final feature representation of the entire model.

[0037] S1.6, the output module generates output through the decoder;

[0038] (2) Model training;

[0039] S2.1, pre-training stage;

[0040] First, two sets of initial modules are fixed for training, and the loss function of the pre-training stage is:

[0041]

[0042] Where: N is the number of training samples, T is the maximum length of the generated text, y i，t is the true word of the ith sample at time step t, y i，＜t represents all generated words before time step t, x i is the input of the module, θ represents the model parameters, including all layer parameters of the two fixed experts; P(y i,t |y i，＜t , x i ; θ) is the probability of the model predicting the next word given the previous context and input, and λ is the regularization coefficient;

[0043] This loss function consists of two parts:

[0044] One part is the negative log-likelihood loss:

[0045] This part encourages the model to accurately predict each word in the sequence, taking into account previous context and input;

[0046] The other part is the L2 regularization term: λ|θ|2;

[0047] This part prevents the model from overfitting by penalizing large parameter values;

[0048] By minimizing this loss function, the parameters of each layer of the two fixed experts are optimized to effectively handle the text generation task;

[0049] S2.2, module pool initialization phase;

[0050] After pre-training is completed, the module pool initialization phase begins. In this phase, different degrees of noise are added to the pre-trained modules and multiple copies are made to form a larger module pool. This process is expressed as:

[0051]

[0052] Where m pre is a pre-trained module, ∈ is a module with a mean of 0 and a variance of σ 2 Noise sampled from a normal distribution;

[0053] S2.3, full parameter training stage;

[0054] In the final full-parameter training phase, the initialized module pool is used for end-to-end training;

[0055] At this stage, all parameters are involved in the training, including the parameters of the dynamic routing, the parameters of the dynamic expert module, and the parameters of the attention layer and the normalization layer.

[0056] A device for dynamically generating a multimodal hybrid expert model, comprising: at least one memory and at least one processor;

[0057] The at least one memory is used to store a machine-readable program;

[0058] The at least one processor is used to call the machine-readable program to execute a method for dynamically generating a multimodal hybrid expert model.

[0059] Compared with the prior art, the method and device for dynamically generating a multimodal hybrid expert model of the present invention have the following outstanding beneficial effects:

[0060] The method of the present invention can be applied to a variety of multimodal learning tasks. In the image-text matching task, the model can learn the semantic consistency between text and image, which is used in the recommendation system. In the visual question-answering task, the model can understand the image content and answer related questions, which is very useful for intelligent assistants or image understanding systems.

[0061] In the context of sentiment analysis, the model can comprehensively consider the sentiment information of text and images to provide more accurate sentiment judgment, which has important applications in social media analysis or user feedback processing. For cross-modal retrieval tasks, such as retrieving images based on text or retrieving text based on images, the multimodal processing capability of this model can effectively learn the semantic correspondence between different modalities.

[0062] At the same time, due to its flexible structure, the method is also applicable to single-modal scenarios, such as text classification, image recognition and other tasks. In these tasks, the model can achieve better performance by dynamically selecting the most suitable experts to handle inputs of different types or difficulties. This wide applicability makes the present invention of great value in various practical applications, which can greatly reduce the cost of redesigning and training models when switching between different tasks.

[0063] In general, the present invention provides a novel, flexible and efficient method for constructing a multimodal hybrid expert model. Through innovative technologies such as dynamic expert generation, shared layer pooling and dynamic routing, this method can effectively process complex multimodal data while maintaining good adaptability to single-modal tasks. This method not only solves many problems in the prior art, but also provides new ideas and directions for the development of multimodal learning and hybrid expert models, and is expected to play an important role in many application fields of artificial intelligence. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the solution of the present invention, the present invention is further described in detail below in conjunction with specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] A best embodiment is given below:

[0066] A method for dynamically generating a multimodal hybrid expert model in this embodiment includes the following steps:

[0067] (1) Model construction;

[0068] S1.1, the multimodal input module receives multimodal input data and preprocesses the data;

[0069] The design of the multimodal input module takes into account the characteristics of different types of data and can flexibly handle various input formats. For example, users input text and images separately. The text can be original natural language sentences or paragraphs, and the image can be a picture file in various common formats such as JPEG, PNG, etc.

[0070] The system will perform preliminary processing on these inputs: for text, it will perform operations such as word segmentation and stop word removal; for images, it will perform preprocessing such as resizing and color space conversion. This flexible input processing mechanism enables the model to adapt to a variety of application scenarios and data preprocessing strategies, preparing for the subsequent vectorization process.

[0071] S1.2, the vectorization module converts the input data into vectors;

[0072] Advanced vectorization techniques are used, such as the multimodal compatible bge-visualized-m3, to vectorize text and images respectively. This pre-trained model can provide rich semantic information and visual features, laying a good foundation for subsequent processing. The use of the bge-visualized-m3 model enables us to represent text and images in the same vector space, which greatly simplifies the subsequent multimodal fusion process and improves the model's ability to process multimodal data.

[0073] After vectorization, the model concatenates the converted vectors and adds separators. It enables the model to distinguish information from different modalities, which helps with subsequent cross-modal learning.

[0074] S1.3, the feature extraction module extracts features from the concatenated vectors through a multi-layer Transformer;

[0075] The concatenated vectors are passed through multiple layers of Transformer for feature extraction. The use of Transformer enables the model to capture the complex interactions between different modalities, which is crucial for multimodal learning tasks. In this example, 8 layers of Transformer are used for feature extraction.

[0076] After feature extraction, the model enters the dynamic routing module and the dynamic expert module.

[0077] S1.4, the dynamic routing module calculates the selection probability of each module in each module pool;

[0078] In this example, there are three module pools, corresponding to the three layers of experts. The modules in the first and third module pools are linear layers, and the modules in the second module pool are SwiGLU activation function layers. Assuming that there are M modules in each pool, the selection probability of the mth module in the lth pool (l∈1,2,3) is:

[0079]

[0080] where w l，m and b l，m is a learnable parameter, h is the input feature, and T refers to the transpose of the matrix. The softmax function is used to calculate the selection probability of each module. The function of the softmax function is to convert a set of numerical values ​​into a probability distribution, ensuring that all output values ​​are between 0 and 1 and the sum is 1. In this way, we can select the most suitable module composition expert based on these probability values.

[0081] S1.5, the dynamic expert module selects the module with the highest probability from each module pool according to the calculated probability, and assembles them into an expert in sequence;

[0082] According to the calculated probability, the module with the highest probability is selected from each module pool and assembled into an expert in sequence. Specifically, the structure of the expert is: linear layer → SwiGLU → linear layer. We select two groups of modules to assemble two experts E1 and E2. Assume that the selected modules are m 1，i , m 2，j , N 3，k (for E1) and m 1，p , m 2，q , N 3，r (For E2), the expert processing process can be expressed as:

[0083] For E1: h1 = m 3，k (SwiGLU(m 2，j (m 1，i (h))))

[0084] For E2h2=m 3，r (SwiGLU(m 2，q (m 1，p (h))))

[0085] SwiGLU is an activation function, defined as: SwiGLU(x) = x*σ(β*x), σ is the sigmoid function, β is a learnable parameter, and h is the result of the expert operation.

[0086] We then merge the outputs of the two experts:

[0087] h expert =αh1+(1-α)h2

[0088] where α is a learnable parameter used to balance the contributions of the two experts.

[0089] Normalization layer: Use layer normalization to process the expert output:

[0090] h norm1 =LayerNorm(h expert )

[0091] Attention layer: Use a multi-head self-attention mechanism to process the normalized output:

[0092] h attn =MultiHeadAttention(hn orm1 )

[0093] Second Normalization Layer: Layer Normalization is used again to process the output of the attention layer:

[0094] h norm2 =LayerNorm(hattn )

[0095] This structure is repeated n times, and the output of each time is used as the input of the next time. In this case, it is repeated 8 times. The output of the last repetition is used as the final feature representation of the entire model.

[0096] S1.6, the output module generates output through the decoder;

[0097] The design of the decoder depends on the specific task and can be a simple linear layer or a more complex structure such as a Transformer decoder.

[0098] (2) Model training;

[0099] S2.1, pre-training stage;

[0100] In this stage, we first fix two sets of initial modules for training. The purpose of this is to allow the model to learn basic feature representation capabilities while avoiding introducing too much uncertainty in the initial stage. The loss function of the pre-training stage is:

[0101]

[0102] Where: N is the number of training samples, T is the maximum length of the generated text, y i，t is the true word of the ith sample at time step t, y i，＜t represents all generated words before time step t, x i is the input of the module, θ represents the model parameters (including all layer parameters of the two fixed experts), P(y i，t |y i，＜t , x i ; θ) is the probability of the model predicting the next word given the previous context and input, and λ is the regularization coefficient.

[0103] This loss function consists of two parts:

[0104] One part is the negative log-likelihood loss:

[0105] This part encourages the model to accurately predict each word in the sequence, taking into account previous context and input.

[0106] The other part is the L2 regularization term: λ|θ| 2

[0107] This part prevents the model from overfitting by penalizing large parameter values.

[0108] By minimizing this loss function, we optimize the parameters of each layer of the two fixed experts so that they can effectively handle the text generation task. This pre-training stage allows the model to learn basic language generation capabilities and multimodal information processing capabilities.

[0109] This pre-training stage lays the foundation for subsequent dynamic expert generation and ensures that the basic module has good text generation capabilities.

[0110] S2.2, module pool initialization phase;

[0111] After pre-training is completed, we enter the module pool initialization phase. In this phase, we add different degrees of noise to the pre-trained modules and replicate them to form a larger module pool. This process can be expressed as:

[0112]

[0113] Where m pre is a pre-trained module, ∈ is a module with a mean of 0 and a variance of σ 2 The advantage of this initialization method is that it increases the diversity and exploration ability of the model by introducing noise while retaining the useful features learned by the pre-training module.

[0114] S2.3, full parameter training stage;

[0115] In the final full parameter training phase, we use the initialized module pool for end-to-end training. In this phase, all parameters are trained, including the parameters of dynamic routing, the parameters of the expert module, and the parameters of the attention mechanism and normalization layer.

[0116] This progressive training strategy has several significant advantages:

[0117] 1. Stability: By pre-training and gradually introducing complexity, instability during training is avoided.

[0118] 2. Efficiency: A smaller model is used in the pre-training phase, which is computationally efficient and speeds up the overall training.

[0119] 3. Generalization ability: Module pool initialization increases the diversity of the model and helps improve the generalization ability of the model.

[0120] 4. Flexibility: The full parameter training phase allows the model to be fine-tuned according to specific tasks, improving the task adaptability of the model.

[0121] In practical applications, the training parameters at each stage can be adjusted according to the characteristics of specific tasks and data sets, such as learning rate, batch size, number of training rounds, etc. At the same time, some advanced training techniques, such as learning rate decay and gradient clipping, can be used to further improve the effect and stability of training.

[0122] The method of the present invention can be applied to a variety of multimodal learning tasks. In the image-text matching task, the model can learn the semantic consistency between text and image for use in the recommendation system. In the visual question-answering task, the model can understand the image content and answer related questions, which is very useful for intelligent assistants or image understanding systems. In multimodal sentiment analysis, the model can comprehensively consider the sentiment information of text and image to provide more accurate sentiment judgment, which has important applications in social media analysis or user feedback processing. For cross-modal retrieval tasks, such as retrieving images based on text or retrieving text based on images, the multimodal processing capability of this model can effectively learn the semantic correspondence between different modalities.

[0123] At the same time, due to its flexible structure, the method is also applicable to single-modal scenarios, such as text classification, image recognition and other tasks. In these tasks, the model can achieve better performance by dynamically selecting the most suitable experts to handle inputs of different types or difficulties. This wide applicability makes the present invention of great value in various practical applications, which can greatly reduce the cost of redesigning and training models when switching between different tasks.

[0124] In general, the present invention provides a novel, flexible and efficient method for constructing a multimodal hybrid expert model. Through innovative technologies such as dynamic expert generation, shared layer pooling and dynamic routing, this method can effectively process complex multimodal data while maintaining good adaptability to single-modal tasks. This method not only solves many problems in the prior art, but also provides new ideas and directions for the development of multimodal learning and hybrid expert models, and is expected to play an important role in many application fields of artificial intelligence.

[0125] Based on the above method, a device for dynamically generating a multimodal hybrid expert model in this embodiment includes: at least one memory and at least one processor;

[0126] at least one memory for storing a machine-readable program;

[0127] At least one processor is used to call the machine-readable program to execute a method for dynamically generating a multimodal hybrid expert model.

[0128] The above-mentioned specific implementations are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above-mentioned specific implementations. Any technical solutions that conform to the above-mentioned specific implementations of the present invention and any appropriate changes or substitutions made by ordinary technicians in the relevant technical field shall fall within the patent protection scope of the present invention.

[0129] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for dynamically generating a multimodal hybrid expert model, characterized in that: When applied to processing multimodal data, it includes the following: (1) Model construction; S1.1, the multimodal input module receives multimodal input data and preprocesses the data; For text, word segmentation and stop word removal will be performed; for images, resizing and color space conversion preprocessing will be performed; S1.2, the vectorization module converts the input data into vectors; The vectorization module uses the multi-modal compatible bge-visualized-m3 to vectorize text and images respectively. The use of the bge-visualized-m3 model allows text and images to be represented in the same vector space. After vectorization, the model concatenates the converted vectors and adds separators so that the model can distinguish information from different modalities; S1.3, the feature extraction module extracts features from the concatenated vectors through a multi-layer Transformer; An 8-layer Transformer layer is used for feature extraction. After feature extraction, the model enters the dynamic routing module and the dynamic expert module; S1.4, the dynamic routing module calculates the selection probability of each module in each module pool; There are three module pools, corresponding to the three layers in the dynamic expert module. The modules in the first and third module pools are linear layers, and the modules in the second module pool are SwiGLU activation function layers. Assuming that there are M modules in each pool, the selection probability of the mth module in the lth pool (l∈1,2,3) is: where w l,m and b l,m is a learnable parameter, h is the input feature, T refers to the transpose of the matrix, and the softmax function is used to calculate the selection probability of each module. The function of the softmax function is to convert a set of numerical values ​​into a probability distribution, ensuring that all output values ​​are between 0 and 1 and the sum is 1. In this way, the most suitable dynamic expert module is selected based on these probability values; S1.5, the dynamic expert module selects the module with the highest probability from each module pool according to the calculated probability, and assembles the selected modules with the highest probability into an expert in sequence; Specifically, the structure of the expert is: linear layer → SwiGLU → linear layer. Two groups of modules are selected to assemble two experts E1 and E2. Assume that the selected modules are m 1,i ,m 2,j ,m 3,k and m 1,p ,m 2,q ,m 3,r , where m 1,i ,m 2,j ,m 3,k For E1, m 1,p ,m 2,q ,m 3,r For E2, the expert processing process can be expressed as: For E1: h1 = m 3,k (SwiGLU(m 2,j (m 1,i (h)))) For E2: h2 = m 3,r (SwiGLUm 2,q (m 1,p (h)))) Where SwiGLU is an activation function, defined as: SwiGLU(x) = x*σ(β*x), σ is the sigmoid function, β is a learnable parameter, and h is the result of the expert operation; Then, combine the outputs of the two experts: h expert =αh1+(1-α)h2 Where α is a learnable parameter used to balance the contributions of the two experts; Normalization layer: Use layer normalization to process expert output: h norm1 =LayerNorm(h expert ) Attention layer: Use a multi-head self-attention mechanism to process the normalized output: h attn =MultiHeadAttention(h norm1 ) Second Normalization Layer: Layer Normalization is used again to process the output of the attention layer: h norm2 =LayerNorm(h attn ) This structure is repeated n times, and the output of each time is used as the input of the next time. This is repeated 8 times, and the output of the last repetition is used as the final feature representation of the entire model. S1.6, the output module generates output through the decoder; (2) Model training; S2.1, pre-training stage; First, two sets of initial modules are fixed for training, and the loss function of the pre-training stage is: Where: N is the number of training samples, T is the maximum length of the generated text, y i,t is the true word of the ith sample at time step t, y i,<t represents all generated words before time step t, x i is the input of the module, θ represents the model parameters, including all layer parameters of the two fixed experts; P(y i,t |y i,<t ,x i ; θ) is the probability of the model predicting the next word given the previous context and input, and λ is the regularization coefficient; This loss function consists of two parts: One part is the negative log-likelihood loss: This part encourages the model to accurately predict each word in the sequence, taking into account previous context and input; The other part is the L2 regularization term: λ|θ| 2 ; This part prevents the model from overfitting by penalizing large parameter values; By minimizing this loss function, the parameters of each layer of the two fixed experts are optimized to effectively handle the text generation task; S2.2, module pool initialization phase; After pre-training is completed, the module pool initialization phase begins. In this phase, different degrees of noise are added to the pre-trained modules and multiple copies are made to form a larger module pool. This process is expressed as: Where m pre is a pre-trained module, ∈ is a module with a mean of 0 and a variance of σ 2 Noise sampled from a normal distribution; S2.3, full parameter training stage; In the final full-parameter training phase, the initialized module pool is used for end-to-end training; At this stage, all parameters are involved in the training, including the parameters of the dynamic routing, the parameters of the dynamic expert module, and the parameters of the attention layer and the normalization layer.

2. A device for dynamically generating a multimodal hybrid expert model, characterized in that: include: at least one memory and at least one processor; The at least one memory is used to store a machine-readable program; The at least one processor is configured to call the machine-readable program to execute the method of claim 1.

Images