Multi-modal large model training strategy determination method
By comparing training strategies and replacing connectors for the state of multimodal large models, the problem of strategy selection and optimization in multimodal large models training is solved, and the model performance and resource utilization efficiency are improved.
Patent Information
- Application Number
- CN202411265080.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-10
- Publication Date
- 2024-12-31
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the training process of multimodal large models, how to reasonably select and optimize each part of the model and the training strategies lead to large differences in performance and lack of effective strategy determination methods.
By obtaining each state of the initial multimodal large model, train it using one- and two-stage training strategies, obtain performance parameters, compare performance metrics, determine the best training strategy, and replace the initial connector to optimize the model.
The optimal model training strategy is determined, the performance of multimodal large models is improved, and the efficient utilization of resources is ensured.
Smart Images
Figure CN119227740A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of multimodal large model training strategy determination, and in particular to a multimodal large model training strategy determination method. Background Art
[0002] With the development of deep learning technology, large multimodal models have achieved remarkable results in various tasks, including image classification, target detection, semantic segmentation, natural language processing, etc.; however, large multimodal models need to be trained before use. There are many training strategies. Different training strategies consume different resources when training large multimodal models, and the performance of the multimodal models finally trained is also different; therefore, in the training process of large multimodal models, how to reasonably select and optimize the various parts of the model, and how to select and optimize the training strategy have become urgent problems to be solved. Summary of the invention
[0003] 1. Technical issues to be resolved
[0004] In view of the above technical problems, the present invention provides a method for determining a multimodal large model training strategy to solve the problems in the prior art of how to reasonably select and optimize the various parts of the model and how to select and optimize the training strategy during the training process of the multimodal large model.
[0005] (II) Technical solution
[0006] The present invention provides a method for determining a multimodal large model training strategy, the method is applied to training an initial multimodal large model, the initial multimodal large model comprising a target large language model W1, a target image encoder W2 and an initial connector W3; wherein the target large language model is used to process input text information, the target image encoder is used to process input image information, and the initial connector is used to align text information output by the target large language model with image information output by the target image encoder;
[0007] The method comprises the following steps:
[0008] S100, obtaining each state corresponding to the initial multimodal large model to obtain a state list A=(A1, A2, A3); wherein A1, A2 and A3 are respectively the first state, the second state and the third state corresponding to the initial multimodal large model; when the state corresponding to the initial multimodal large model is A1, W1, W2 and W3 are all in the thawed state; when the state corresponding to the initial multimodal large model is A2, W1 and W3 are both in the thawed state, and W2 is in the frozen state; when the state corresponding to the initial multimodal large model is A3, W2 and W3 are both in the thawed state, and W1 is in the frozen state;
[0009] S200, using a preset one-stage training strategy, respectively train the initial multimodal large model in each state in A to obtain a first intermediate multimodal large model list B = (B1, B2, B3); wherein B1 is an intermediate multimodal large model obtained by training the initial multimodal large model in A1, B2 is an intermediate multimodal large model obtained by training the initial multimodal large model in A2, and B3 is an intermediate multimodal large model obtained by training the initial multimodal large model in A3;
[0010] S300, using a preset two-stage training strategy, respectively train the initial multimodal large model in each state in A to obtain a second intermediate multimodal large model list C = (C1, C2, C3); wherein C1 is an intermediate multimodal large model obtained by training the initial multimodal large model in A1, C2 is an intermediate multimodal large model obtained by training the initial multimodal large model in A2, and C3 is an intermediate multimodal large model obtained by training the initial multimodal large model in A3;
[0011] S400, obtaining the performance parameters of each intermediate multimodal large model in B to obtain a first performance parameter list α=(α1, α2, α3); and obtaining the performance parameters of each intermediate multimodal large model in C to obtain a second performance parameter list β=(β1, β2, β3); wherein α1 is the performance parameter of B1, α2 is the performance parameter of B2, α3 is the performance parameter of B3, β1 is the performance parameter of C1, β2 is the performance parameter of C2, and β3 is the performance parameter of C3;
[0012] S500: If α1<β1, α2<β2, and α3<β3, the preset two-stage training strategy is determined as the target training strategy corresponding to the initial multimodal large model.
[0013] In some embodiments of the present invention, step S200 includes the following steps:
[0014] S210, if the state of the initial multimodal large model is A1, then use the preset first training set to train W1, W2 and W3 simultaneously to obtain B1;
[0015] S220, if the state of the initial multimodal large model is A2, then use the preset first training set to train W1 and W3 simultaneously, and do not train W2, to obtain B2;
[0016] S230, if the state of the initial multimodal large model is A3, then use the preset first training set to train W2 and W3 simultaneously, without training W1, to obtain B3.
[0017] In some embodiments of the present invention, step S300 includes the following steps:
[0018] S310, if the state of the initial multimodal large model is A1, adjust W1 and W2 to a frozen state, and keep W3 in an unfrozen state;
[0019] S320, training W3 using a preset second training set;
[0020] S330, if the training of W3 meets the preset training conditions, W1 and W2 are adjusted to the unfrozen state, and W1, W2 and W3 are trained simultaneously using the preset first training set to obtain C1;
[0021] S340, if the state of the initial multimodal large model is A2, adjust W1 to a frozen state;
[0022] S350, training W3 using a preset second training set;
[0023] S360, if the training of W3 meets the preset training condition, W1 is adjusted to an unfrozen state, and W1 and W3 are trained simultaneously using the preset first training set to obtain C2;
[0024] S370, if the state of the initial multimodal large model is A3, adjust W2 to a frozen state;
[0025] S380, training W3 using a preset second training set;
[0026] S390, if the training of W3 meets the preset training condition, W2 is adjusted to an unfrozen state, and W2 and W3 are trained simultaneously using the preset first training set to obtain C3.
[0027] In some embodiments of the present invention, α is obtained by the following steps:
[0028] S410, obtaining the training duration corresponding to each intermediate multimodal large model in B to obtain a first training duration list T=(T1, T2, T3); wherein T1 is the training duration corresponding to B1, T2 is the training duration corresponding to B2, and T3 is the training duration corresponding to B3;
[0029] S420, using each intermediate multimodal large model in B to predict the results of the preset standard sample set, and obtaining the prediction time used by each intermediate multimodal large model in B to predict the results of the preset standard sample set, so as to obtain a prediction result list U = (U1, U2, U3) and a prediction time list H = (H1, H2, H3); wherein U1 is the prediction result corresponding to B1, U2 is the prediction result corresponding to B2, and U3 is the prediction result corresponding to B3; H1 is the prediction time corresponding to B1, H2 is the prediction time corresponding to B2, and H3 is the prediction time corresponding to B3;
[0030] S430, according to U and the actual results corresponding to the preset standard sample set, determine the similarity between the predicted results corresponding to each intermediate multimodal large model in B and the actual results, so as to obtain a similarity list F = (F1, F2, F3); wherein F1 is the similarity between the predicted results corresponding to B1 and the actual results, F2 is the similarity between the predicted results corresponding to B2 and the actual results, and F3 is the similarity between the predicted results corresponding to B3 and the actual results;
[0031] S440, according to T, H and F, determine α=(α1, α2, α3); wherein,
[0032] α1=F1-λ×T1 / MAX(T)-ω×H1 / MAX(H);
[0033] α2=F2-λ×T2 / MAX(T)-ω×H2 / MAX(H);
[0034] α3=F3-λ×T3 / / MAX(T)-ω×H3 / MAX(H);
[0035] λ is the preset training duration weight, ω is the preset prediction duration weight; MAX() is the preset maximum value function; 0<λ<1, 0<ω<1.
[0036] In some embodiments of the present invention, after step S500, the method further includes the following steps:
[0037] S600, obtaining each known type of connector to obtain a connector list L = (L1, L2, ..., Li, ..., Ln), i = 1, 2, ... n; wherein Li is the known i-th type of connector, and n is the number of known connectors;
[0038] S610, using each known connector in L to replace the initial connector corresponding to the initial multimodal large model, to obtain a candidate multimodal large model list QT = (QT1, QT2, ..., QTi, ..., QTn); wherein QTi is a candidate multimodal large model obtained by replacing the initial connector corresponding to the initial multimodal large model with Li; wherein the state of each candidate multimodal large model is the same;
[0039] S620, training each candidate multimodal large model in QT using the target training strategy to obtain a list of spare multimodal large models BT = (BT1, BT2, ..., BTi, ..., BTn); wherein BTi is a spare multimodal large model obtained by training QTi using the target training strategy;
[0040] S630, obtaining the performance parameters of each standby multimodal large model in BT to obtain a third performance parameter list η=(η1, η2, ..., ηi, ..., ηn); wherein ηi is the performance parameter corresponding to BTi;
[0041] S640, obtaining a target performance parameter QU=MAX(η); wherein MAX() is a preset maximum value function;
[0042] S650: Determine the connector corresponding to the standby multimodal large model corresponding to QU as the target connector.
[0043] (III) Beneficial effects
[0044] It can be seen from the above technical solution that the present invention has at least the following beneficial effects:
[0045] The method for determining a multimodal large model training strategy of the present invention obtains each state of the initial multimodal large model, and uses a one-stage training strategy and a two-stage training strategy to train the initial multimodal large model in each state, respectively, to obtain a first intermediate multimodal large model list B and a second intermediate multimodal large model list C; obtains the performance parameters of each intermediate multimodal large model in the first intermediate multimodal large model list B and the performance parameters of each intermediate multimodal large model in the second intermediate multimodal large model list C, to obtain a first performance parameter list α and a second performance parameter list β; if each performance parameter in α is less than the corresponding performance parameter in β, it means that the performance of the intermediate multimodal large model trained using the preset two-stage training strategy is better, therefore, the preset two-stage training strategy is determined as the target training strategy corresponding to the initial multimodal large model; thereby achieving the purpose of determining the best model training strategy. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 A flowchart of a method for determining a multimodal large model training strategy provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0047] The present invention provides a method for determining a multimodal large model training strategy. In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below in combination with specific embodiments and with reference to the accompanying drawings. Specific embodiments
[0049] like Figure 1As shown, the present invention provides a method for determining a multimodal large model training strategy, the method is applied to training an initial multimodal large model, the initial multimodal large model includes a target large language model W1, a target image encoder W2 and an initial connector W3; wherein the target large language model is used to process input text information, the target image encoder is used to process input image information, and the initial connector is used to align text information output by the target large language model with image information output by the target image encoder;
[0050] The method comprises the following steps:
[0051] S100, obtaining each state corresponding to the initial multimodal large model to obtain a state list A=(A1, A2, A3); wherein A1, A2 and A3 are respectively the first state, the second state and the third state corresponding to the initial multimodal large model; when the state corresponding to the initial multimodal large model is A1, W1, W2 and W3 are all in the thawed state; when the state corresponding to the initial multimodal large model is A2, W1 and W3 are both in the thawed state, and W2 is in the frozen state; when the state corresponding to the initial multimodal large model is A3, W2 and W3 are both in the thawed state, and W1 is in the frozen state;
[0052] S200, using a preset one-stage training strategy, respectively train the initial multimodal large model in each state in A to obtain a first intermediate multimodal large model list B = (B1, B2, B3); wherein B1 is an intermediate multimodal large model obtained by training the initial multimodal large model in A1, B2 is an intermediate multimodal large model obtained by training the initial multimodal large model in A2, and B3 is an intermediate multimodal large model obtained by training the initial multimodal large model in A3. The specific steps are as follows:
[0053] S210, if the state of the initial multimodal large model is A1, then use the preset first training set to train W1, W2 and W3 simultaneously to obtain B1;
[0054] S220, if the state of the initial multimodal large model is A2, then use the preset first training set to train W1 and W3 simultaneously, and do not train W2, to obtain B2;
[0055] S230, if the state of the initial multimodal large model is A3, then use the preset first training set to train W2 and W3 simultaneously, without training W1, to obtain B3.
[0056] S300, using a preset two-stage training strategy, respectively train the initial multimodal large model in each state in A to obtain a second intermediate multimodal large model list C = (C1, C2, C3); wherein C1 is an intermediate multimodal large model obtained by training the initial multimodal large model in A1, C2 is an intermediate multimodal large model obtained by training the initial multimodal large model in A2, and C3 is an intermediate multimodal large model obtained by training the initial multimodal large model in A3. The specific steps are as follows:
[0057] S310, if the state of the initial multimodal large model is A1, adjust W1 and W2 to a frozen state, and keep W3 in an unfrozen state;
[0058] S320, training W3 using a preset second training set;
[0059] S330, if the training of W3 meets the preset training conditions, W1 and W2 are adjusted to the unfrozen state, and W1, W2 and W3 are trained simultaneously using the preset first training set to obtain C1;
[0060] S340, if the state of the initial multimodal large model is A2, adjust W1 to a frozen state;
[0061] S350, training W3 using a preset second training set;
[0062] S360, if the training of W3 meets the preset training condition, W1 is adjusted to an unfrozen state, and W1 and W3 are trained simultaneously using the preset first training set to obtain C2;
[0063] S370, if the state of the initial multimodal large model is A3, adjust W2 to a frozen state;
[0064] S380, training W3 using a preset second training set;
[0065] S390, if the training of W3 meets the preset training condition, W2 is adjusted to an unfrozen state, and W2 and W3 are trained simultaneously using the preset first training set to obtain C3;
[0066] S400, obtaining the performance parameters of each intermediate multimodal large model in B to obtain a first performance parameter list α=(α1, α2, α3); and obtaining the performance parameters of each intermediate multimodal large model in C to obtain a second performance parameter list β=(β1, β2, β3); wherein α1 is the performance parameter of B1, α2 is the performance parameter of B2, α3 is the performance parameter of B3, β1 is the performance parameter of C1, β2 is the performance parameter of C2, and β3 is the performance parameter of C3; wherein α is obtained by the following steps:
[0067] S410, obtaining the training duration corresponding to each intermediate multimodal large model in B to obtain a first training duration list T=(T1, T2, T3); wherein T1 is the training duration corresponding to B1, T2 is the training duration corresponding to B2, and T3 is the training duration corresponding to B3;
[0068] S420, using each intermediate multimodal large model in B to predict the results of the preset standard sample set, and obtaining the prediction time used by each intermediate multimodal large model in B to predict the results of the preset standard sample set, so as to obtain a prediction result list U = (U1, U2, U3) and a prediction time list H = (H1, H2, H3); wherein U1 is the prediction result corresponding to B1, U2 is the prediction result corresponding to B2, and U3 is the prediction result corresponding to B3; H1 is the prediction time corresponding to B1, H2 is the prediction time corresponding to B2, and H3 is the prediction time corresponding to B3;
[0069] S430, according to U and the actual results corresponding to the preset standard sample set, determine the similarity between the predicted results corresponding to each intermediate multimodal large model in B and the actual results, so as to obtain a similarity list F = (F1, F2, F3); wherein F1 is the similarity between the predicted results corresponding to B1 and the actual results, F2 is the similarity between the predicted results corresponding to B2 and the actual results, and F3 is the similarity between the predicted results corresponding to B3 and the actual results;
[0070] S440, according to T, H and F, determine α=(α1, α2, α3); wherein,
[0071] α1=F1-λ×T1 / MAX(T)-ω×H1 / MAX(H);
[0072] α2=F2-λ×T2 / MAX(T)-ω×H2 / MAX(H);
[0073] α3=F3-λ×T3 / / MAX(T)-ω×H3 / MAX(H);
[0074] λ is the preset training duration weight, ω is the preset prediction duration weight; MAX() is the preset maximum value function; 0<λ<1, 0<ω<1;
[0075] S500, if α1<β1, α2<β2 and α3<β3, then determine the preset two-stage training strategy as the target training strategy corresponding to the initial multimodal large model;
[0076] After step S500, the method further comprises the following steps:
[0077] S600, obtaining each known type of connector to obtain a connector list L = (L1, L2, ..., Li, ..., Ln), i = 1, 2, ... n; wherein Li is the known i-th type of connector, and n is the number of known connectors;
[0078] S610, using each known connector in L to replace the initial connector corresponding to the initial multimodal large model, to obtain a candidate multimodal large model list QT = (QT1, QT2, ..., QTi, ..., QTn); wherein QTi is a candidate multimodal large model obtained by replacing the initial connector corresponding to the initial multimodal large model with Li; wherein the state of each candidate multimodal large model is the same;
[0079] S620, training each candidate multimodal large model in QT using the target training strategy to obtain a list of spare multimodal large models BT = (BT1, BT2, ..., BTi, ..., BTn); wherein BTi is a spare multimodal large model obtained by training QTi using the target training strategy;
[0080] S630, obtaining the performance parameters of each standby multimodal large model in BT to obtain a third performance parameter list η=(η1, η2, ..., ηi, ..., ηn); wherein ηi is the performance parameter corresponding to BTi;
[0081] S640, obtaining a target performance parameter QU=MAX(η); wherein MAX() is a preset maximum value function;
[0082] S650: Determine the connector corresponding to the standby multimodal large model corresponding to QU as the target connector.
[0083] So far, the present embodiment has been described in detail with reference to the accompanying drawings. Based on the above description, those skilled in the art should have a clear understanding of the present invention.
[0084] It should be noted that the implementation methods not shown or described in the drawings or the text of the specification are all forms known to ordinary technicians in the relevant technical field and are not described in detail. In addition, the above definitions of each element and method are not limited to the various specific structures, shapes or methods mentioned in the embodiments.
[0085] It should also be noted that this document may provide demonstrations of parameters containing specific values, but these parameters do not need to be exactly equal to the corresponding values, but can be approximated to the corresponding values within an acceptable error tolerance or design constraint. The directional terms mentioned in the embodiments, such as "up", "down", "front", "back", "left", "right", etc., are only reference directions of the accompanying drawings and are not intended to limit the scope of protection of the present invention. In addition, unless the steps are specifically described or must occur in sequence, the order of the above steps is not limited to those listed above, and can be changed or rearranged according to the desired design. And the above embodiments can be mixed and matched with each other or with other embodiments based on design and reliability considerations, that is, the technical features in different embodiments can be freely combined to form more embodiments.
[0086] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A method for determining a multimodal large model training strategy, characterized in that: The method is applied to train an initial multimodal large model, wherein the initial multimodal large model includes a target large language model W1, a target image encoder W2 and an initial connector W3; wherein the target large language model is used to process input text information, the target image encoder is used to process input image information, and the initial connector is used to align text information output by the target large language model with image information output by the target image encoder; The method comprises the following steps: S100, obtaining each state corresponding to the initial multimodal large model to obtain a state list A=(A1, A2, A3); wherein A1, A2 and A3 are respectively the first state, the second state and the third state corresponding to the initial multimodal large model; when the state corresponding to the initial multimodal large model is A1, W1, W2 and W3 are all in the thawed state; when the state corresponding to the initial multimodal large model is A2, W1 and W3 are both in the thawed state, and W2 is in the frozen state; when the state corresponding to the initial multimodal large model is A3, W2 and W3 are both in the thawed state, and W1 is in the frozen state; S200, using a preset one-stage training strategy, respectively train the initial multimodal large model in each state in A to obtain a first intermediate multimodal large model list B = (B1, B2, B3); wherein B1 is an intermediate multimodal large model obtained by training the initial multimodal large model in A1, B2 is an intermediate multimodal large model obtained by training the initial multimodal large model in A2, and B3 is an intermediate multimodal large model obtained by training the initial multimodal large model in A3; S300, using a preset two-stage training strategy, respectively train the initial multimodal large model in each state in A to obtain a second intermediate multimodal large model list C = (C1, C2, C3); wherein C1 is an intermediate multimodal large model obtained by training the initial multimodal large model in A1, C2 is an intermediate multimodal large model obtained by training the initial multimodal large model in A2, and C3 is an intermediate multimodal large model obtained by training the initial multimodal large model in A3; S400, obtaining the performance parameters of each intermediate multimodal large model in B to obtain a first performance parameter list α=(α1, α2, α3); and obtaining the performance parameters of each intermediate multimodal large model in C to obtain a second performance parameter list β=(β1, β2, β3); wherein α1 is the performance parameter of B1, α2 is the performance parameter of B2, α3 is the performance parameter of B3, β1 is the performance parameter of C1, β2 is the performance parameter of C2, and β3 is the performance parameter of C3; S500: If α1<β1, α2<β2, and α3<β3, the preset two-stage training strategy is determined as the target training strategy corresponding to the initial multimodal large model.
2. The method for determining a multimodal large model training strategy according to claim 1, characterized in that: Step S200 includes the following steps: S210, if the state of the initial multimodal large model is A1, then use the preset first training set to train W1, W2 and W3 simultaneously to obtain B1; S220, if the state of the initial multimodal large model is A2, then use the preset first training set to train W1 and W3 simultaneously, and do not train W2, to obtain B2; S230, if the state of the initial multimodal large model is A3, then use the preset first training set to train W2 and W3 simultaneously, without training W1, to obtain B3.
3. The method for determining a multimodal large model training strategy according to claim 1, characterized in that: Step S300 includes the following steps: S310, if the state of the initial multimodal large model is A1, adjust W1 and W2 to a frozen state, and keep W3 in an unfrozen state; S320, training W3 using a preset second training set; S330, if the training of W3 meets the preset training conditions, W1 and W2 are adjusted to the unfrozen state, and W1, W2 and W3 are trained simultaneously using the preset first training set to obtain C1; S340, if the state of the initial multimodal large model is A2, adjust W1 to a frozen state; S350, training W3 using a preset second training set; S360, if the training of W3 meets the preset training condition, W1 is adjusted to an unfrozen state, and W1 and W3 are trained simultaneously using the preset first training set to obtain C2; S370, if the state of the initial multimodal large model is A3, adjust W2 to a frozen state; S380, training W3 using a preset second training set; S390, if the training of W3 meets the preset training condition, W2 is adjusted to the unfrozen state, and W2 and W3 are trained simultaneously using the preset first training set to obtain C3.
4. The method for determining a multimodal large model training strategy according to claim 1, characterized in that: α is obtained by the following steps: S410, obtaining the training duration corresponding to each intermediate multimodal large model in B to obtain a first training duration list T=(T1, T2, T3); wherein T1 is the training duration corresponding to B1, T2 is the training duration corresponding to B2, and T3 is the training duration corresponding to B3; S420, using each intermediate multimodal large model in B to predict the results of the preset standard sample set, and obtaining the prediction time used by each intermediate multimodal large model in B to predict the results of the preset standard sample set, so as to obtain a prediction result list U = (U1, U2, U3) and a prediction time list H = (H1, H2, H3); wherein U1 is the prediction result corresponding to B1, U2 is the prediction result corresponding to B2, and U3 is the prediction result corresponding to B3; H1 is the prediction time corresponding to B1, H2 is the prediction time corresponding to B2, and H3 is the prediction time corresponding to B3; S430, according to U and the actual results corresponding to the preset standard sample set, determine the similarity between the predicted results corresponding to each intermediate multimodal large model in B and the actual results, so as to obtain a similarity list F = (F1, F2, F3); wherein F1 is the similarity between the predicted results corresponding to B1 and the actual results, F2 is the similarity between the predicted results corresponding to B2 and the actual results, and F3 is the similarity between the predicted results corresponding to B3 and the actual results; S440, according to T, H and F, determine α=(α1, α2, α3); wherein, α1=F1-λ×T1 / MAX(T)-ω×H1 / MAX(H); α2=F2-λ×T2 / MAX(T)-ω×H2 / MAX(H); α3=F3-λ×T3 / / MAX(T)-ω×H3 / MAX(H); λ is the preset training duration weight, ω is the preset prediction duration weight; MAX() is the preset maximum value function; 0<λ<1, 0<ω<1.
5. The method for determining a multimodal large model training strategy according to claim 1, characterized in that: After step S500, the method further comprises the following steps: S600, obtaining each known type of connector to obtain a connector list L = (L1, L2, ..., Li, ..., Ln), i = 1, 2, ... n; wherein Li is the known i-th type of connector, and n is the number of known connectors; S610, using each known connector in L to replace the initial connector corresponding to the initial multimodal large model, to obtain a candidate multimodal large model list QT = (QT1, QT2, ..., QTi, ..., QTn); wherein QTi is a candidate multimodal large model obtained by replacing the initial connector corresponding to the initial multimodal large model with Li; wherein the state of each candidate multimodal large model is the same; S620, training each candidate multimodal large model in QT using the target training strategy to obtain a list of spare multimodal large models BT = (BT1, BT2, ..., BTi, ..., BTn); wherein BTi is a spare multimodal large model obtained by training QTi using the target training strategy; S630, obtaining the performance parameters of each standby multimodal large model in BT to obtain a third performance parameter list η=(η1, η2, ..., ηi, ..., ηn); wherein ηi is the performance parameter corresponding to BTi; S640, obtaining a target performance parameter QU=MAX(η); wherein MAX() is a preset maximum value function; S650: Determine the connector corresponding to the standby multimodal large model corresponding to QU as the target connector.