Hybrid expert multi-modal large model feature fusion method and system

Abstract

The invention provides a hybrid expert multi-modal large model feature fusion method and system, and the method comprises the steps: carrying out the autoregression pre-training of a large model, carrying out the token processing of text, image and voice data, and predicting a next element; in the distillation process from the large model to the small model, the knowledge of the large model is transmitted to the small model through a soft label; and the small models are expanded, and more diversified tasks are processed by adding the multi-modal small models and adjusting the gating network. In the aspect of feature fusion, a token-embedded adaptive gating strategy is adopted, and effective information integration among different modes is realized. According to the method, various multi-modal small models are designed, the tasks are completed by comprehensively utilizing multi-modal large model experts, and the model performance, the processing efficiency and the task universality are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of multimodal machine learning, and in particular to a hybrid expert multimodal large model feature fusion method and system. Background Art

[0002] In the field of natural language processing and multimodal machine learning, with the rapid development of deep learning technology, large models based on the Transformer architecture have become the main means of dealing with complex tasks. However, existing large models often fail to fully capture and fuse information between different modalities when dealing with multimodal tasks. This limitation leads to poor information fusion when the model handles cross-modal tasks, which in turn affects the overall performance of the task. For example, when processing image, text, and voice information at the same time, existing models usually find it difficult to effectively combine the information of these modalities, resulting in unsatisfactory performance of the model in complex scenarios.

[0003] In order to improve the performance of the model in multimodal tasks, researchers have proposed a variety of methods, such as directly splicing multimodal data into the model or using a simple attention mechanism to interact information between modalities. However, these methods often have difficulty dealing with the problem of information misalignment between modalities. Specifically, data of different modalities may have significant differences in timing, scale, and even semantics. Direct information fusion may lead to the loss of important information or the introduction of noise, thereby affecting the accuracy and generalization ability of the model. In addition, existing methods also face challenges in computational efficiency and storage requirements when processing large-scale multimodal data, making it difficult to widely promote them in practical applications.

[0004] Another problem with existing large models is their huge parameter size and consumption of computing resources. Although large models have shown powerful capabilities in processing complex tasks, their huge computing overhead and inference time make them difficult to apply in resource-limited environments. In addition, since large models need to process a large number of parameters, the training and inference processes are often difficult to perform real-time performance optimization and cannot quickly respond to task requirements. Therefore, how to reduce the consumption of computing resources while maintaining model performance has become a major challenge for existing technologies.

[0005] In order to solve these problems, knowledge distillation methods have been proposed to transfer the knowledge of large models to small models, thereby reducing the parameter scale while retaining the performance of the model. However, traditional knowledge distillation methods usually only focus on the overall performance of the model, while ignoring the specific performance of the small model in multimodal tasks. Due to the lack of distillation strategies for multimodal tasks, it is difficult for small models to achieve the expected results in practical applications, especially when dealing with complex information interactions between different modalities. The performance of the small model after distillation is still limited. These problems make the existing multimodal large model feature fusion methods have certain limitations in application.

[0006] like Figure 1 As shown in Figure 1, the current multimodal feature fusion method based on multi-model mixing faces serious challenges in terms of expansion. Although simply increasing model parameters or stacking model structures can improve performance to a certain extent, it brings about a huge waste of computing resources and performance bottlenecks. In addition, as the number of experts increases, the difficulty of managing and allocating data also increases significantly, resulting in a lack of flexibility and practicality in the model expansion strategy.

[0007] Even if small models can be successfully expanded in some cases, their application is still difficult. This is mainly because small models have difficulty capturing detailed and critical feature information when facing multimodal tasks. At the same time, the existing fixed gating mechanism and simple fusion strategy often limit the performance of the model and cannot fully adapt to the task requirements, resulting in poor multimodal feature fusion effect of small models.

[0008] The challenge of multimodal feature fusion lies not only in the expansion of small models, but also in the design of fusion strategies. Simple fusion strategies often fail to cope with the complex requirements of multimodal tasks, resulting in poor performance of the model in these tasks. Current methods are too fixed in their gating mechanisms, lack flexibility, and are difficult to adapt to dynamic task changes, which ultimately results in poor multimodal fusion results. Summary of the invention

[0009] In view of this, an embodiment of the present invention provides a hybrid expert multimodal large model feature fusion method and system to solve or partially solve the above-mentioned problems, significantly improving model performance, processing efficiency and task universality.

[0010] The technical solution to achieve the purpose of the present invention is:

[0011] A hybrid expert multimodal large model feature fusion method comprises the following steps:

[0012] S1, construct datasets under multiple modalities, perform autoregressive pre-training on the large model respectively, and obtain the large model corresponding to each field;

[0013] S2, performs the distillation process from the large model to the small model, transfers the knowledge of the large model to the small model through soft labels, and obtains the small model under each modality;

[0014] S3, expanding the small model under each mode to obtain a small model group under each mode;

[0015] S4, the small model groups under multiple modalities are fused through the adaptive gating network with soft label embedding to obtain a model that adapts to multi-modality.

[0016] Furthermore, a distillation process from a large model to a small model is performed to obtain a small model under each mode, which specifically includes the following steps:

[0017] S2.1, configure the teacher model and student model. The teacher model is the large model trained in step S1. Let the teacher model be The student model is A variety of student models to choose from

[0018] S2.2, adjust the temperature parameters and obtain the soft labels of the teacher model through the normalized exponential function softmax;

[0019] S2.3, the student model calculates the distillation loss through softprediction and soft labels;

[0020] S2.4, the student model calculates the loss through hardprediction and ground truth;

[0021] S2.5, based on the losses in steps 2.3 and S2.4, calculate the total loss function through the weight coefficient;

[0022] S2.6, based on the loss function, distill the teacher model to the student model to obtain a student model that meets the requirements, i.e., a small model.

[0023] Furthermore, the soft labels of the teacher model are:

[0024]

[0025] Where T is the temperature parameter, z T is the output of the teacher model.

[0026] Further, the distillation loss in step S2.3 is:

[0027]

[0028] Among them, p s is the softprediction of the student model, z S is the output of the student model, and KL(·) is the KL divergence function.

[0029] Furthermore, the loss of step S2.4 is:

[0030]

[0031] in, is the mean square error function.

[0032] Furthermore, the total loss function in step 2.5 is:

[0033]

[0034] Among them, α is the weight coefficient.

[0035] Furthermore, the small model groups under the multiple modalities are fused through an adaptive gating network, specifically including:

[0036] Each small model receives data of different modalities and generates feature tokens;

[0037] Through the adaptive gating network, the fusion is performed, and the corresponding expert network, i.e., the small model, is adaptively activated according to the characteristics and context information of each token.

[0038] Furthermore, constructing datasets under multiple modes specifically includes:

[0039] S1.1, segment and tokenize text data using subword segmentation algorithm;

[0040] S1.2, divide the image data into multiple fixed-size patches and encode them into tokens;

[0041] S1.3, segment the speech data and convert it into feature vectors, and then tokenize it;

[0042] S1.4, process the data from step S1.1 to step S1.3 by random masking and sequence completion to obtain a data set.

[0043] A hybrid expert multi-modal large model feature fusion system, comprising:

[0044] A data construction unit, used to construct data sets under multiple modes;

[0045] The large model training unit performs autoregressive pre-training on the large model through the data set, and transfers the knowledge of the large model to the small model through soft labels to obtain the large model corresponding to each field;

[0046] The distillation unit performs the distillation process from the large model to the small model to obtain the small model under each mode;

[0047] An expansion unit expands the small model under each mode to obtain a small model group under each mode;

[0048] The fusion unit fuses the small model groups under multiple modalities through an adaptive gating network embedded with soft labels to obtain a model that adapts to multiple modalities.

[0049] The embodiments of the present invention have at least the following technical effects:

[0050] (1) Improve the ability to accurately process multimodal data. Through autoregressive pre-training, the present invention can more accurately predict and generate sequence information when processing multimodal data such as text, images, and speech; through joint training of multimodal data, the model can capture the deep correlation between different modalities, thereby providing more accurate results in practical applications; this precise feature extraction and sequence prediction significantly improves the processing effect of multimodal tasks, making applications in complex environments more robust.

[0051] (2) Improve model reasoning efficiency. By distilling a large model into a small model, the present invention significantly improves the reasoning speed of the model. While maintaining the knowledge of the large model, the small model greatly reduces the number of parameters and significantly improves the reasoning efficiency; this efficient reasoning capability enables the model to provide high-quality output in a resource-constrained environment, thereby expanding the possibility of application scenarios.

[0052] (3) Enhance the adaptability and scalability of the model. The present invention improves the model's ability to cope with diverse tasks by expanding the capacity of a small model. By increasing the number of expert networks and adopting an adaptive gating strategy, the model can flexibly process different types of multimodal data and ensure that the key information of each modality can be effectively integrated; this scalability not only improves the adaptability of the model, but also demonstrates excellent performance in a wide range of tasks.

[0053] (4) Optimizing the efficiency of multimodal feature fusion. The adaptive gating strategy effectively activates the appropriate expert network during the feature fusion process, allowing data from different modalities to be fused at a more detailed level; through this mechanism, the present invention achieves more efficient information integration in multimodal data processing, ensuring that the important features of each modality are fully utilized, thereby providing more comprehensive and accurate support in the decision-making process.

[0054] (5) Dynamic optimization of model performance. During the model expansion phase, the present invention measures the expansion effect in real time through multiple sets of benchmark tasks, and automatically selects the best experts to optimize the model. Through this dynamic experimental adjustment, the model can automatically adapt when processing different tasks to ensure that it is always in the best performance state. This automated optimization process not only improves the overall quality of the model, but also reduces the need for manual intervention, greatly improving development efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0056] Figure 1 A problem description diagram for the multimodal feature fusion method based on multi-model mixture.

[0057] Figure 2 This is a flowchart of the steps of a hybrid expert multimodal large model feature fusion method in Example 1.

[0058] Figure 3 This is the large model autoregressive pre-training flow chart in Example 1.

[0059] Figure 4 This is the large model distillation flow chart in Example 1.

[0060] Figure 5 This is a flowchart of small model expansion and multi-modal adaptive gating feature fusion in Example 1. DETAILED DESCRIPTION

[0061] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0062] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0063] The embodiment of the present invention provides a hybrid expert multi-modal large model feature fusion method, comprising the following steps:

[0064] S1, autoregressive pre-training of large models;

[0065] S2, distillation process from large model to small model;

[0066] S3, small model expansion;

[0067] S4, Multimodal small model adaptive gating feature fusion.

[0068] According to a specific implementation method of the embodiment of the present invention, combined with Figure 3 , the step S1 is specifically as follows:

[0069] S1.1, segment and tokenize the text data using BPE (Byte Pair Encoding) subword segmentation algorithm;

[0070] Define the input text sequence as x=(x1,x2,...,x n ); Use the BPE algorithm to segment the text sequence and obtain the segmented sequence x′=(x′1,x′2,...,x′ m) ; tokenize the sequence after word segmentation to obtain the token sequence t x =(t x1 ,t x2 ,...,t xm ).

[0071] S1.2, divide the image data into multiple fixed-size patches and encode them into tokens;

[0072] Define the input image as I, whose size is H×W×C (height, width, number of channels); divide the image into multiple fixed-size patches (blocks), each patch size is P×P×C, and the number of patches obtained is Encode each patch to get a token sequence

[0073] S1.3, segment the speech data and convert it into feature vectors, and then tokenize it;

[0074] Define the input speech sequence as a=(a1,a2,…,a T ); Divide the speech data into segments, each segment has a length of L, and obtain the segment sequence a′=(a′1, a′2, …, a′ K ), where a′ i =(a (i-1)L+1 ,...,a iL ); Convert each segment into a feature vector and obtain a feature vector sequence f a =(f a1 ,f a2 ,...,f aK ) ; tokenize the feature vector to get the token sequence t a =(t a1 ,t a2 ,...,t aK ).

[0075] S1.4, through random mask and sequence completion;

[0076] Define the input token sequence as t=(t1, t2, ..., t N ); Randomly mask some tokens and get the masked sequence in is the token after masking; define the loss function L for the sequence completion task and minimize the loss function:

[0077]

[0078] S1.5, configure the pre-training dataset, validation dataset, and benchmark dataset for downstream tasks;

[0079] Define the pre-training dataset D train , validation set D val , and the benchmark dataset D for downstream tasks benchmark ; Use D train Perform model pre-training with D val To verify the model, use D benchmark Evaluate the model's performance on downstream tasks.

[0080] S1.6, trains large models through data sets, monitors and records operator gradients and losses during training, and supports one-click early termination of training.

[0081] Define the loss function during training as L, and the gradient as Monitor and record the loss function L and gradient Supports one-click early termination of training. Define the stopping condition. If the stopping condition is met, the training will be terminated early.

[0082] According to a specific implementation method of the embodiment of the present invention, combined with Figure 4 , the step S2 is specifically as follows:

[0083] S2.1, configure the teacher model (large model) and the student model (small model), and provide a variety of small model options;

[0084] Define the teacher model as The student model is Provide a variety of student models to choose from

[0085] S2.2, design distillation strategy, including adjusting temperature parameters and selecting loss function;

[0086] Define the temperature parameter as T and select the loss function For distillation:

[0087]

[0088] S2.3, obtain the soft label of the teacher model through softmax;

[0089] Define the output of the teacher model as z T , get soft labels through softmax:

[0090]

[0091] S2.4, the student model calculates the distillation loss through softprediction and soft labels;

[0092] Define the output of the student model as z S , the soft prediction of the student model is:

[0093]

[0094] Calculate distillation loss:

[0095]

[0096] S2.5, the student model calculates the loss through hardprediction and ground truth;

[0097] Define ground truth as y, hardprediction of the student model:

[0098]

[0099] Calculate the loss:

[0100]

[0101] S2.6, evaluate the distillation effect, calculate the soft and hard losses and the performance of the validation dataset;

[0102] Calculate the total loss:

[0103]

[0104] Using the validation dataset D val Evaluate the performance of the student model, the loss on the validation set is

[0105] According to a specific implementation of the embodiment of the present invention, step S3 is specifically:

[0106] Increase the number of small models: define a new set of small models as Increasing the number of small models helps to enhance the diversity of the models, thereby improving the overall performance.

[0107] According to a specific implementation method of the embodiment of the present invention, combined with Figure 5 , the step S4 is specifically as follows:

[0108] S4.1, each small model receives data of different modalities and generates corresponding feature tokens;

[0109] Define the new modal data as q k, which represents the data of the kth modal input, q k It is passed as input to the k-th small model to generate feature tokens. The feature token generated by the k-th small model is where u k It is the key feature representing the modal data and will be used as input in subsequent model processing for further feature fusion.

[0110] S4.2, the gating network adaptively decides which tokens should activate which expert networks based on the features and contextual information of each token;

[0111] Define the new context information as h, which represents the background information of the sequence currently processed by the model. h will be input into the gating network together with each feature token to determine how to process the token. The activation probability output by the gating network is s k =h(u k ,h), where s k It will be used to select and activate the appropriate expert network to further process the feature token. The activated expert network is where m = argmax(s k ).

[0112] S4.3, through the adaptive gating strategy, the information between different modalities is effectively fused; the fused feature representation is defined as in is the activated expert network. r integrates information from different modalities and realizes feature fusion between modalities through adaptive gating strategy. r will be used as the final feature representation for downstream tasks of the model, such as classification or regression. This fusion process ensures that the model can effectively integrate and utilize data from different modalities.

[0113] S4.4, optimize the training process so that the model can capture detailed and key feature information in multimodal data;

[0114] Define the training optimization objective as It represents the loss function to be minimized during the training process. The complexity of multimodal data is taken into account to ensure that the model can capture the detailed and key features in each modality. The model is able to gradually improve its performance in multimodal tasks, which includes extracting key information from different modalities and fusing it in the final model output.

[0115] S4.5, continue training on the expanded MOE model to improve its performance on new tasks.

[0116] The MOE model after expansion is defined as It contains more expert networks and more complex gating mechanisms. The ability to handle multimodal data has been enhanced through expansion. Continue training to improve its performance in new task sets. This training will help the model adapt to new data distribution and task requirements, thereby improving its generalization ability.

[0117] S4.6, using multiple sets of benchmark tasks to measure the expansion effect;

[0118] Define multiple groups of benchmark tasks as Each task Used to evaluate the specific capabilities of the model, these benchmark tasks cover different task scenarios, so as to comprehensively measure the expansion effect of the model. The performance on each group of tasks is {R1, R2, ..., R L},in Represents the evaluation result of the model on the i-th task. By comparing these results, we can determine whether the expanded model has been improved in each task.

[0119] This embodiment provides a hybrid expert multi-modal large model feature fusion system, including:

[0120] A data construction unit, used to construct data sets under multiple modes;

[0121] The large model training unit performs autoregressive pre-training on the large model through the data set, and transfers the knowledge of the large model to the small model through soft labels to obtain the large model corresponding to each field;

[0122] The distillation unit performs the distillation process from the large model to the small model to obtain the small model under each mode;

[0123] An expansion unit expands the small model under each mode to obtain a small model group under each mode;

[0124] The fusion unit fuses the small model groups under multiple modalities through an adaptive gating network embedded with soft labels to obtain a model that adapts to multiple modalities.

[0125] This embodiment also provides a computer storage medium, characterized in that the computer storage medium stores an executable program, and the executable program is executed by a processor to implement the steps of the hybrid expert multimodal large model feature fusion method.

[0126] Example 1

[0127] Figure 2This is a flowchart of the steps for a feature fusion method of a MOE (Mixture of Experts) multi-modal large model provided by an embodiment of the present invention. The steps included in this method will be introduced with the following embodiment scenarios.

[0128] To better understand the implementation of this patent, we will use the multi-modal data processing of an autonomous vehicle as a unified scenario to explain each operation step. This scenario includes instructions in text data, roads and traffic signs in image data, and navigation prompts in voice data.

[0129] S1. Autoregressive pre-training of the large model, specifically including:

[0130] S1.1. Perform BPE (Byte Pair Encoding) word segmentation and tokenization on the text data

[0131] In the autonomous driving scenario, the input text sequence can be a driving instruction received by the vehicle, such as "Turn left at the next intersection." First, use the BPE algorithm to segment this text sequence, splitting the sentence into smaller sub-word units, such as "at", "the", "next", "intersection", "turn", "left"; then, tokenize these sub-word units to generate a series of tokens, which will be used for further model input processing.

[0132] S1.2. Divide the image data into multiple patches of a fixed size and encode them into tokens

[0133] The input image data can be a road image captured by an in-vehicle camera, with a size of (1280, 720, 3). Divide the image into multiple patches of a fixed size of (32, 32) to generate a patch matrix of (40, 22). Then, encode each patch to generate a token sequence of the image so that the model can recognize and process road signs and other important visual information.

[0134] S1.3. Segment the voice data and convert it into feature vectors, and then perform tokenization

[0135] The input voice data can be a voice prompt provided by the navigation system, such as "Please turn right 300 meters ahead." Segment the voice data, with each segment having a length of 0.5 seconds, to obtain multiple segmented voice data. Then, convert each segment into a feature vector, and tokenize these feature vectors to generate corresponding token sequences for further processing of voice information.

[0136] S1.4. Perform training through random masking and sequence completion tasks

[0137] During the training process, the input token sequence may come from the token sequence of a text instruction, the token sequence of an image, or the token sequence of speech. Randomly mask some tokens, such as masking the token corresponding to the character "turn" in the text sequence. Then, define a loss function and minimize this loss to complete the sequence completion task, such as predicting the masked "turn" character token.

[0138] S1.5, Configure the pre-training dataset, validation set, and benchmark dataset for downstream tasks

[0139] For the autonomous driving scenario, the pre-training dataset may include large-scale driving instruction texts, road images, and voice navigation data; the validation set is used to evaluate the performance of the model in multi-modal data fusion; the downstream task benchmark dataset can contain real-world autonomous driving tasks, such as path planning, traffic sign recognition, etc., to evaluate the final performance of the model.

[0140] S1.6, Monitor and record the operator gradients and loss during the training process, and support one-key early termination of training

[0141] During the training process, the loss function can be used to monitor the performance of the model on multi-modal tasks, such as the error in predicting the driving path. The gradient is used to reflect the update direction and magnitude of the model parameters. Support for one-key early termination of training, for example, when the performance of the validation set no longer improves, terminate the training early to save computing resources.

[0142] S2:, The distillation process from a large model to a small model, specifically including:

[0143] S2.1, Configure the teacher model and the student model, and provide multiple small model alternatives

[0144] In this scenario, the teacher model can be a pre-trained multi-modal large model for handling autonomous driving tasks. The student model is a smaller and more efficient model that can be used for deployment on resource-constrained vehicles. Provide multiple student model alternatives to allow selection of a suitable model according to the computing power of different vehicles.

[0145] S2.2, Design the distillation strategy, including adjusting the temperature parameter and selecting the loss function

[0146] During the distillation process, the temperature parameter is used to adjust the softness of the output of the teacher model to better guide the learning of the student model. The selected loss function is used to optimize the performance of the student model. For example, the KL divergence loss function is used to measure the difference between the outputs of the teacher model and the student model.

[0147] S2.3, Obtain the soft label of the teacher model through softmax

[0148] During training, the teacher model may generate soft labels for navigation instructions such as "turn right 300 meters ahead". The output is normalized using the softmax function to generate soft labels, which will serve as a reference to guide the student model's learning.

[0149] S2.4, the student model calculates the distillation loss through softprediction and soft labels

[0150] The student model generates its own prediction output, such as "Please turn right ahead". Soft prediction is calculated and compared with the soft label provided by the teacher model, and the distillation loss is calculated to gradually narrow the gap between the output of the student model and the teacher model.

[0151] S2.5, the student model calculates the loss through hardprediction and ground truth

[0152] When the student model generates a final prediction, such as "turn right", it is compared with the ground truth (e.g., the actual driving path) and the corresponding loss is calculated to ensure the accuracy of the model in practical applications.

[0153] S2.6, evaluate the distillation effect, calculate the soft and hard losses and the performance of the validation dataset

[0154] Evaluate the distillation effect of the student model and calculate the loss between the soft label and the soft prediction during the distillation process, as well as the loss between the hard label and the ground truth. Use the validation dataset to evaluate the performance of the student model in different driving scenarios to determine its reliability and effectiveness in practical applications.

[0155] S3, small model expansion, including:

[0156] Each small model added can process different modal data, such as one dedicated to voice data and another to image data. The gating mechanism is adjusted to flexibly activate the most suitable model in a specific driving task, thereby improving the overall performance and adaptability of the model.

[0157] S4, multi-modal small model adaptive gating feature fusion, specifically including:

[0158] S4.1, each small model receives data of different modalities and generates corresponding feature tokens.

[0159] New modality data may include new sensor inputs (such as infrared cameras), which are passed to the corresponding small models. For example, model A receives text data, model B receives image data, and model C receives speech data. Each small model converts these inputs into feature tokens and generates new feature tokens. This process ensures that in different driving tasks, the model can capture more key information and process it effectively to form comprehensive driving decisions.

[0160] S4.2, the gating network adaptively decides which tokens should activate which expert networks based on the features and contextual information of each token.

[0161] For example, when a vehicle is at a complex intersection, the gating network decides which expert networks to activate to process information such as traffic lights and pedestrians based on the current road conditions and contextual information to ensure that the vehicle can pass safely.

[0162] S4.3, through adaptive gating strategy, effective fusion of information between different modalities is achieved.

[0163] Through adaptive gating strategies, data from cameras, radars, and navigation systems are fused into a unified feature representation for driving decisions. For example, by combining visual and voice data, current traffic conditions and driving routes can be accurately determined.

[0164] S4.4, optimize the training process so that the model can capture detailed and key feature information in multimodal data.

[0165] Optimize the training process so that the model can recognize and utilize key feature information in different modalities, such as detecting subtle changes in traffic signs or identifying important information in voice commands, to ensure the model's performance in complex driving environments.

[0166] S4.5, continue training on the expanded MOE model to improve its performance on new tasks.

[0167] The expanded MOE model has more expert networks and complex gating mechanisms. Through continued training, the model can adapt to new driving scenarios, such as night driving or bad weather, and improve the performance and stability of the model in these new tasks.

[0168] The performance of the expanded small model combination is evaluated using multiple sets of autonomous driving benchmark tasks, such as urban road driving and highway lane changing. These tasks will comprehensively measure the performance of the model in different driving environments and determine whether the expanded model has improved in various tasks.

[0169] The present invention uses autoregressive pre-training of large models to tokenize text, image and speech data and predict the next element; the distillation process from large models to small models transfers the knowledge of large models to small models through soft labels; the small model is expanded by adding multimodal small models and adjusting the gating network to handle more diverse tasks. In terms of feature fusion, an adaptive gating strategy of token embedding is adopted to achieve effective information integration between different modalities. The present invention designs a variety of multimodal small models, comprehensively utilizes large model experts of multiple modalities to complete tasks, and significantly improves model performance, processing efficiency and task universality.

[0170] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A hybrid expert multimodal large model feature fusion method, characterized in that: The following steps are involved: S1, construct datasets under multiple modalities, perform autoregressive pre-training on the large model respectively, and obtain the large model corresponding to each field; S2, performs the distillation process from the large model to the small model, transfers the knowledge of the large model to the small model through soft labels, and obtains the small model under each modality; S3, expanding the small model under each mode to obtain a small model group under each mode; S4, the small model groups under multiple modalities are fused through the adaptive gating network with soft label embedding to obtain a model that adapts to multi-modality.

2. According to claim 1, a hybrid expert multimodal large model feature fusion method is characterized in that: The distillation process from the large model to the small model is carried out to obtain the small model under each mode, which specifically includes the following steps: S2.1, configure the teacher model and student model. The teacher model is the large model trained in step S1. Let the teacher model be The student model is A variety of student models to choose from S2.2, adjust the temperature parameters and obtain the soft labels of the teacher model through the normalized exponential function softmax; S2.3, the student model calculates the distillation loss through softprediction and soft labels; S2.4, the student model calculates the loss through hardprediction and ground truth; S2.5, based on the losses in steps 2.3 and S2.4, calculate the total loss function through the weight coefficient; S2.6, based on the loss function, distill the teacher model to the student model to obtain a student model that meets the requirements, i.e., a small model.

3. The hybrid expert multimodal large model feature fusion method according to claim 2 is characterized in that: The soft labels of the teacher model are: Where T is the temperature parameter, z T is the output of the teacher model.

4. The hybrid expert multimodal large model feature fusion method according to claim 3 is characterized in that: The distillation loss of step S2.3 is: Among them, p s is the softprediction of the student model, z S is the output of the student model, and KL(·) is the KL divergence function.

5. The hybrid expert multimodal large model feature fusion method according to claim 4 is characterized in that: The loss of step S2.4 is: in, is the mean square error function.

6. The hybrid expert multimodal large model feature fusion method according to claim 1, characterized in that: The total loss function in step 2.5 is: Among them, α is the weight coefficient.

7. The hybrid expert multimodal large model feature fusion method according to claim 1, characterized in that: The small model groups under the multiple modalities are fused through an adaptive gating network, specifically including: Each small model receives data of different modalities and generates feature tokens; Through the adaptive gating network, the fusion is performed, and the corresponding expert network, i.e., the small model, is adaptively activated according to the characteristics and context information of each token.

8. The hybrid expert multimodal large model feature fusion method according to claim 1, characterized in that: Constructing data sets under multiple modes specifically includes: S1.1, segment and tokenize text data using subword segmentation algorithm; S1.2, divide the image data into multiple fixed-size patches and encode them into tokens; S1.3, segment the speech data and convert it into feature vectors, and then tokenize it; S1.4, process the data from step S1.1 to step S1.3 by random masking and sequence completion to obtain a data set.

9. A hybrid expert multi-modal large model feature fusion system for implementing any of the methods described in claims 1-8, characterized in that: include: A data construction unit, used to construct data sets under multiple modes; The large model training unit performs autoregressive pre-training on the large model through the data set, and transfers the knowledge of the large model to the small model through soft labels to obtain the large model corresponding to each field; The distillation unit performs the distillation process from the large model to the small model to obtain the small model under each mode; An expansion unit expands the small model under each mode to obtain a small model group under each mode; The fusion unit fuses the small model groups under multiple modalities through an adaptive gating network embedded with soft labels to obtain a model that adapts to multiple modalities.

10. A computer storage medium, characterized in that: The computer storage medium stores an executable program, and the executable program is executed by a processor to implement the steps of the hybrid expert multimodal large model feature fusion method described in any one of claims 1-8.

Images