Model training methods, devices and electronic equipment

Abstract

This application discloses a method, apparatus, and electronic device for model training. The method includes: acquiring a training dataset, wherein the training dataset is used to train a pre-trained model, and the training dataset includes image data and corresponding text data; sequentially inputting the training dataset into multiple multilayer perceptron networks (MLPNs) in the pre-trained model for training, obtaining multiple target MPNs, wherein the output of the previous MPN is used as the input of the next MPN; fusing the multiple target MPNs to obtain a fused network; training the fused network based on the training dataset to obtain a target fused network, and determining the target pre-trained model based on the target fused network. This application solves the technical problem in related technologies where visual encoder models ignore many detailed information during the encoding process, resulting in poor accuracy.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a method, apparatus, and electronic device for model training. Background Technology

[0002] In recent years, multimodal large-scale models have been widely used in the field of security surveillance image processing, such as image question answering (VQA) and ubiquitous deployment monitoring. Among related technologies, the visual encoder in multimodal large-scale models suffers from significant information loss during the information encoding process. Typically, the visual encoder encodes the input in a coarse-grained manner, ignoring many crucial details, resulting in unsatisfactory performance. Another major challenge hindering contrastive learning is feature suppression; the model captures only limited information while ignoring other potentially valuable content.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for model training, which at least solves the technical problem in related technologies that visual encoder models ignore many detailed information during the encoding process, resulting in poor accuracy.

[0005] According to one aspect of the embodiments of this application, a method for model training is provided, comprising: acquiring a training dataset, wherein the training dataset is used to train a pre-trained model, and the training dataset includes image data and corresponding text data; sequentially inputting the training dataset into multiple multilayer perceptron networks in the pre-trained model for training to obtain multiple target multilayer perceptron networks, wherein the output result of the previous multilayer perceptron network is used as the input of the next multilayer perceptron network, and the multiple target multilayer perceptron networks are multilayer perceptron networks obtained after the training of multiple multilayer perceptron networks; fusing the multiple target multilayer perceptron networks to obtain a fused network; training the fused network according to the training dataset to obtain a target fused network, and determining a target pre-trained model according to the target fused network.

[0006] Optionally, before sequentially inputting the training dataset into the multiple multilayer perceptron networks in the pre-trained model for training, the method further includes freezing all parameters in the pre-trained model except for the multiple multilayer perceptron networks.

[0007] Optionally, the training dataset is sequentially input into multiple multilayer perceptron networks in the pre-trained model for training to obtain multiple target multilayer perceptron networks, including: inputting the training dataset into a first multilayer perceptron network in the pre-trained model for a first-stage training to obtain a first target multilayer perceptron network, wherein the first multilayer perceptron network is any one of the multiple multilayer perceptron networks in the pre-trained model; and based on the first target multilayer perceptron network, performing a second-stage training on a second multilayer perceptron network in the pre-trained model to obtain a second target multilayer perceptron network, wherein the second multilayer perceptron network is an alternative to the first multilayer perceptron network. The multilayer perceptron network is trained after the second target multilayer perceptron network. Based on the second target multilayer perceptron network, the other multilayer perceptron networks in the pre-trained model are trained in corresponding stages to obtain other target multilayer perceptron networks. The other multilayer perceptron networks are multilayer perceptron networks trained after the second multilayer perceptron network. Each of the other multilayer perceptron networks corresponds to a stage of training and obtains the target multilayer perceptron network of the corresponding stage. After all multilayer perceptron networks in the pre-trained model have completed the corresponding stage training, the target multilayer perceptron networks obtained from each stage training are determined as multiple target multilayer perceptron networks.

[0008] Optionally, the training dataset is input into the first multilayer perceptron network in the pre-trained model for the first stage of training to obtain the first target multilayer perceptron network, including: Step 1: After initializing the first multilayer perceptron network, the training dataset is encoded by the image encoder and text encoder in the pre-trained model to obtain initial features, wherein the initial features include image features and text features; Step 2: The initial features are input into the first multilayer perceptron network for feature transformation and enhancement to obtain first processed features, wherein the first processed features include first image features after processing image features and first text features after processing text features; Step 3: The first contrast loss between the first image features and the first text features is determined; Step 4: Backpropagation is performed on the first multilayer perceptron network according to the first contrast loss to obtain the parameter gradient of the first multilayer perceptron network; the weights of the first multilayer perceptron network are updated according to the parameter gradient, and the updated first multilayer perceptron network is used to iteratively execute steps 2 to 4 until the condition is met and the iteration stops, thus obtaining the first target multilayer perceptron network.

[0009] Optionally, based on the first target multilayer perceptron network, a second-stage training is performed on the second multilayer perceptron network in the pre-trained model to obtain the second target multilayer perceptron network. This includes: using the first target multilayer perceptron network to infer image data in the training dataset to obtain a first image feature output; clustering the training dataset based on the first image feature output to obtain a first classification result; determining a pseudo-label for each image data in the training dataset based on the first classification result, wherein the pseudo-label is used to determine negative samples for training the second multilayer perceptron network; and performing a second-stage training on the second multilayer perceptron network based on the pseudo-label, the training dataset, and the negative samples to obtain the second target multilayer perceptron network.

[0010] Optionally, multiple target multilayer perceptron networks can be fused to obtain a fused network, including: connecting multiple target multilayer perceptron networks using a learnable routing network to obtain a fused network.

[0011] Optionally, the fusion network is trained based on the training dataset to obtain the target fusion network, including: freezing the parameters of multiple target multilayer perceptron networks and initializing the routing parameters of the routing network; inputting the training dataset into the pre-trained model, wherein the training dataset is processed by multiple target multilayer perceptron networks in the pre-trained model to obtain multiple feature representations, and the multiple feature representations are processed by the fusion network in the training dataset to obtain a fused feature representation; determining the predicted classification result corresponding to the training dataset based on the fused feature representation; determining the loss function based on the predicted classification result and the true label corresponding to the training dataset; optimizing the routing parameters based on the loss function, and obtaining the target fusion network after optimization.

[0012] According to another aspect of the embodiments of this application, a model training apparatus is also provided, comprising: an acquisition module for acquiring a training dataset, wherein the training dataset is used to train a pre-trained model, and the training dataset includes image data and corresponding text data; a training module for sequentially inputting the training dataset into multiple multilayer perceptron networks in the pre-trained model for training to obtain multiple target multilayer perceptron networks, wherein the output result of the previous multilayer perceptron network is used as the input of the next multilayer perceptron network, and the multiple target multilayer perceptron networks are multilayer perceptron networks obtained after the multiple multilayer perceptron networks have been trained; a fusion module for fusing the multiple target multilayer perceptron networks to obtain a fused network; and a determination module for training the fused network based on the training dataset to obtain a target fused network, and determining a target pre-trained model based on the target fused network.

[0013] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory for storing program instructions; and a processor connected to the memory for executing program instructions to perform the following functions: acquiring a training dataset, wherein the training dataset is used to train a pre-trained model, and the training dataset includes image data and corresponding text data; sequentially inputting the training dataset into multiple multilayer perceptron networks in the pre-trained model for training to obtain multiple target multilayer perceptron networks, wherein the output result of the previous multilayer perceptron network is used as the input of the next multilayer perceptron network, and the multiple target multilayer perceptron networks are multilayer perceptron networks obtained after the training of multiple multilayer perceptron networks; fusing the multiple target multilayer perceptron networks to obtain a fused network; training the fused network based on the training dataset to obtain a target fused network, and determining a target pre-trained model based on the target fused network.

[0014] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-described model training method by running the computer program.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions that, when executed by a processor, implement the above-described method for model training.

[0016] In this embodiment, a training dataset is obtained, which is used to train a pre-trained model. The training dataset includes image data and corresponding text data. The training dataset is then sequentially input into multiple multilayer perceptron networks in the pre-trained model for training, resulting in multiple target multilayer perceptron networks. The output of the previous multilayer perceptron network serves as the input of the next multilayer perceptron network. The multiple target multilayer perceptron networks are multilayer perceptron networks obtained after the training of the multiple multilayer perceptron networks. The multiple target multilayer perceptron networks are then fused to obtain a fused network. The fused network is then trained based on the training dataset to obtain a target fused network. The target pre-trained model is determined based on the target fused network. This achieves the goal of improving the ability to capture detailed information, thereby enhancing the flexibility and adaptability of the model. This solves the technical problem in related technologies where visual encoder models ignore many detailed information during the encoding process, resulting in poor accuracy. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a computer terminal for implementing a model training method according to an embodiment of this application;

[0019] Figure 2 This is a flowchart of a model training method according to an embodiment of this application;

[0020] Figure 3 This is a simplified schematic diagram of a single-stage multilayer perceptron network layer in a pre-trained model according to an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the overall structure of a fusion network according to an embodiment of this application;

[0022] Figure 5 This is a structural diagram of a model training apparatus according to an embodiment of this application. Detailed Implementation

[0023] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0025] The information collected in this application embodiment is information and data authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data all comply with the relevant laws, regulations and standards of the relevant regions, and necessary confidentiality measures have been taken. It does not violate public order and good morals, and provides corresponding operation entry points for users to choose to authorize or reject the automated decision results. If the user chooses to reject, the process will proceed to the expert decision-making process.

[0026] First, some nouns or terms that appear in the explanation of the embodiments of this application shall be interpreted as follows:

[0027] MLP (Multi-Layer Perceptron): A multilayer perceptron is a feedforward artificial neural network consisting of an input layer, an output layer, and one or more hidden layers. In an MLP, each neuron (except those in the input layer) has a weight used to calculate a weighted sum of the input signals, which is then processed by an activation function (such as Sigmoid, ReLU, etc.) to produce an output signal. These output signals serve as the input to the next layer of neurons, and so on, until the output layer produces the final prediction. MLPs are one of the most fundamental structures in neural networks, capable of handling complex nonlinear relationships. By adjusting the number of layers and the number of neurons in each layer, models suitable for different task requirements can be designed. In fields such as computer vision, natural language processing, and speech recognition, MLPs are often used as part of other more complex models (such as Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs)) for tasks such as feature transformation, classification, and regression.

[0028] Multistage Contrastive Learning (MCL) is an advanced method for training deep learning models, achieving significant results, particularly in self-supervised learning and computer vision. The core idea of ​​contrastive learning is to learn the intrinsic representation of data by enabling the model to distinguish between positive and negative examples. In multistage contrastive learning, this process is broken down into multiple stages, each with specific goals and strategies to progressively improve the model's representation learning ability.

[0029] MoE (Mixture of Experts) is a model architecture in machine learning and deep learning that aims to improve the overall performance and flexibility of a model by integrating multiple expert models specializing in different subtasks or subsets of data. The core idea of ​​the MoE structure is that different input data or problems may require different processing methods. Therefore, by training multiple expert models, each focusing on learning a specific part of the data or solving a specific type of problem, the model's adaptability and efficiency when facing diverse data can be improved.

[0030] A visual encoder is a model or network structure in computer vision used to convert image or video data into fixed-length vectors or feature representations. These feature vectors typically carry high-level abstract information from the original visual data and can be used for subsequent tasks such as visual recognition, classification, retrieval, and generation. The core of a visual encoder is its ability to capture meaningful visual features from pixel-level raw data and compress these features into compact, interpretable representations.

[0031] To address the problems existing in related technologies, embodiments of this application provide a model training method, which can be run on... Figure 1 The computer terminal shown is explained below.

[0032] The model training method embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal for implementing a model training method is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0033] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be implemented wholly or partially as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element in the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0034] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the model training method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby implementing the aforementioned model training method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0035] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0036] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0037] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0038] In the above operating environment, this application provides a method embodiment for model training. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0039] Figure 2 This is a flowchart of a model training method according to an embodiment of this application, such as... Figure 2As shown, the method includes the following steps:

[0040] Step S202: Obtain the training dataset, which is used to train the pre-trained model. The training dataset includes image data and corresponding text data.

[0041] In step S202 above, the two parts of the training dataset are interrelated, meaning that each image has one or more corresponding text descriptions. These text descriptions can be image titles, descriptions, labels, or descriptions of events. The pre-trained model can be, for example, an improved visual encoder model. The original visual encoder model includes, but is not limited to, models with various Transformer, CNN, and other structures, and is optimized using the training dataset. In this embodiment, by freezing most of the parameters of the original visual encoder model, the focus is on training and optimizing the multilayer perceptron (MLP) network layers to better adapt to specific tasks and datasets, thereby improving the model's ability to encode and understand visual information.

[0042] Step S204: The training dataset is sequentially input into multiple multilayer perceptron networks in the pre-trained model for training, resulting in multiple target multilayer perceptron networks. The output of the previous multilayer perceptron network is used as the input of the next multilayer perceptron network. The multiple target multilayer perceptron networks are the multilayer perceptron networks obtained after the training of multiple multilayer perceptron networks.

[0043] In step S204 above, during the multi-stage contrastive learning (MCL) process, the training dataset is used to train multiple (expert) multilayer perceptron networks (MLP_i) with different characteristics. Each expert MLP network focuses on learning different features of the input data and enhances the model's semantic understanding ability by utilizing image data and corresponding text data through a contrastive learning mechanism. In this step, the pre-trained model used (such as an improved visual encoder) has already been pre-trained on a large amount of image and text data, and possesses a certain ability to learn image and text representations.

[0044] When training the MLP layers in the model in stages, one target multilayer perceptron network (MLP_i) is trained in each stage. In the first stage, the first MLP layer is trained using the entire training dataset to optimize its parameters. After training, the first target multilayer perceptron network (MLP_1) is obtained. Starting from the second stage, the output of the target multilayer perceptron network trained in the previous stage is used as the input of the MLP layer in the next stage. This means that the MLP layer in the second stage receives not only the original image and text data, but also the processed results from MLP_1, and so on. This cascaded or chained training method allows each target multilayer perceptron network to further optimize its parameters based on the learning in the previous stage, thereby learning deeper feature representations. Through multi-stage training, multiple target multilayer perceptron networks (MLP_i) are eventually obtained, each focusing on learning different aspects of the input data and possessing its own characteristics and advantages.

[0045] Step S206: Fuse multiple target multilayer perceptron networks to obtain a fused network.

[0046] In step S206 above, after all target multilayer perceptron networks have been trained, they are connected through a learnable router network (i.e., a routing network) to form a fused multilayer perceptron network, or simply a fusion network. This router network can dynamically adjust the weights of each target multilayer perceptron network according to the characteristics of the input data and the task requirements, thereby obtaining the optimal combination of expert multilayer perceptron networks.

[0047] Step S208: Train the fusion network based on the training dataset to obtain the target fusion network, and determine the target pre-trained model based on the target fusion network.

[0048] In step S208 above, after completing the phased training of multiple multilayer perceptron networks (MLP_i), these expert MLP_i networks do not function directly as a whole, but need to be integrated through a fusion mechanism. This embodiment introduces a learnable Router network to connect and fuse the MLP_i networks trained at different stages. The Router network acts like an intelligent "decision-maker," determining how the outputs of different MLP_i networks should be weighted and combined based on the characteristics of the input data to form a more powerful and comprehensive representation. After determining the fusion network, the entire fusion network (including the Router network and all MLP_i networks) is treated as a whole and further trained using the training dataset. During this process, the parameters of other networks, except the Router network, can be frozen. This is done to solidify the learning results of previous stages and prevent the previously learned features from being destroyed or forgotten due to training the fusion network. The goal of training is to optimize the parameters of the Router network so that it can more accurately allocate weights according to the characteristics of the input data, thereby highlighting the advantages of each expert network during fusion and avoiding redundancy or conflict. Through the training described above, the Router network learns how to optimally fuse the outputs of different MLP_i to form a target fusion network. This network retains the characteristics and advantages of each expert network while dynamically adjusting its internal expert combination based on the input data, achieving diversification and complementarity in visual information encoding. The target fusion network actually constitutes the core of the target pre-trained model. Based on this, the pre-trained model combines other previously frozen components (such as image encoders and text encoders) to form a complete and efficient target pre-trained model. This model not only possesses the basic capabilities of a visual encoder model but also significantly improves its performance and adaptability on specific tasks (such as event recognition) through the enhancement of the target fusion network.

[0049] Through steps S202 to S208, the goal of improving the ability to capture detailed information is achieved, thereby enhancing the flexibility and adaptability of the model. This solves the technical problem in related technologies where visual encoder models ignore many detailed information during the encoding process, resulting in poor accuracy. The following is a further explanation.

[0050] In step S204 of the above model training method, before sequentially inputting the training dataset into the multiple multilayer perceptron networks in the pre-trained model for training, the method further includes: freezing all parameters in the pre-trained model except for the multiple multilayer perceptron networks.

[0051] In some embodiments of this application, the parameters of other components of the pre-trained model (such as convolutional layers, attention mechanism layers, etc.) are frozen, except for the multilayer perceptron network (MLP). That is, the parameters of these components remain unchanged during training, and only the MLP layer is optimized and trained.

[0052] In step S204 of the above model training method, the training dataset is sequentially input into multiple multilayer perceptron networks in the pre-trained model for training to obtain multiple target multilayer perceptron networks. This includes: inputting the training dataset into the first multilayer perceptron network in the pre-trained model for a first-stage training to obtain a first target multilayer perceptron network, wherein the first multilayer perceptron network is any one of the multiple multilayer perceptron networks in the pre-trained model; and based on the first target multilayer perceptron network, performing a second-stage training on the second multilayer perceptron network in the pre-trained model to obtain a second target multilayer perceptron network, wherein the second multilayer perceptron network is... The multilayer perceptron network is trained after the first multilayer perceptron network. Based on the second target multilayer perceptron network, the other multilayer perceptron networks in the pre-trained model are trained in corresponding stages to obtain other target multilayer perceptron networks. The other multilayer perceptron networks are multilayer perceptron networks trained after the second multilayer perceptron network. Each of the other multilayer perceptron networks corresponds to a stage of training and obtains the target multilayer perceptron network of the corresponding stage. After all multilayer perceptron networks in the pre-trained model have completed the corresponding stage training, the target multilayer perceptron networks obtained from each stage training are determined as multiple target multilayer perceptron networks.

[0053] In some embodiments of this application, firstly, the first multilayer perceptron network (MLP_1) in the pre-trained model is trained using the entire training dataset. During this stage, the parameters of other parts of the pre-trained model (such as the image encoder, text encoder, and other layers in between) are frozen, meaning these components are not updated during training; only the parameters of MLP_1 are adjusted based on data feedback. After training, MLP_1 learns representations and features specific to the training data, and this optimized MLP_1 becomes the first target multilayer perceptron network (Target MLP_1). After the first stage of training is completed, the output of the first target multilayer perceptron network (Target MLP_1) is used as one of the inputs for the second stage of training. This means that the second stage of training not only processes the original image and text data but also considers the initial processing results of the first target multilayer perceptron network. The next multilayer perceptron network (MLP_2) in the pre-trained model is then selected as the training target for the second stage. The model components except MLP_2 are frozen again, and MLP_2 is trained using updated training data (including the original data and the processing results from Target MLP_1) to optimize its parameters. After the second stage of training, the optimized MLP_2 is obtained, which is the second target multilayer perceptron network (Target MLP_2). In subsequent training stages, after each stage is completed, the output of its target multilayer perceptron network is used as one of the inputs for the next stage, forming a cascaded process. This means that as training progresses, the model not only processes the original data but also utilizes the feature representations learned in previous stages. Subsequent stages will sequentially select other multilayer perceptron networks in the pre-trained model for training, with each network corresponding to a stage of training. During training, the output from the target multilayer perceptron network of the previous stage will be considered to obtain the corresponding optimized results, i.e., other target multilayer perceptron networks (Target MLP_n). After all multilayer perceptron networks in the pre-trained model have completed their corresponding stage training, the training results of each stage (i.e., the target multilayer perceptron networks) will constitute the final set of multiple target multilayer perceptron networks. This set contains the optimized state of each multilayer perceptron network under specific training data and tasks. Each target multilayer perceptron network focuses on learning different features and perspectives of the data, providing the model with diverse representation capabilities.

[0054] In the above steps, the training dataset is input into the first multilayer perceptron network in the pre-trained model for the first stage of training to obtain the first target multilayer perceptron network, including: Step 1: After initializing the first multilayer perceptron network, the training dataset is encoded by the image encoder and text encoder in the pre-trained model to obtain initial features, wherein the initial features include image features and text features; Step 2: The initial features are input into the first multilayer perceptron network for feature transformation and enhancement to obtain the first processed features, wherein the first processed features include the first image features after processing the image features and the first text features after processing the text features; Step 3: The first contrast loss between the first image features and the first text features is determined; Step 4: Backpropagation is performed on the first multilayer perceptron network according to the first contrast loss to obtain the parameter gradient of the first multilayer perceptron network; the weights of the first multilayer perceptron network are updated according to the parameter gradient, and the updated first multilayer perceptron network is used to iteratively execute steps 2 to 4 until the condition is met and the iteration stops, thus obtaining the first target multilayer perceptron network.

[0055] In some embodiments of this application, before training begins, the first multilayer perceptron network (MLP_1) is first ensured to be in an initial state, i.e., the parameters of the first multilayer perceptron network are initialized. The training dataset is encoded using the image encoder and text encoder in the pre-trained model. This means converting each image into a set of image features and each text description into a set of text features. The initial features (i.e., image features and text features) obtained by the encoder are input into the first multilayer perceptron network (MLP_1). MLP_1 processes these features through its internal neural network structure. After processing by MLP_1, the original image features and text features are transformed and enhanced, resulting in the first image features and the first text features, respectively. These features more fully capture the intrinsic information of the input data, providing a foundation for subsequent contrastive learning optimization. After obtaining the first image features and the first text features, a first contrastive loss is determined between them. Contrastive loss is a commonly used loss function in multimodal learning; it measures the degree of matching between image features and text features. If an image and its matching text description features are closer in the embedding space, while unmatched text description features are farther apart, the contrastive loss will be smaller, and vice versa. Based on the first contrastive loss, the parameter gradient of the first multilayer perceptron network (MLP_1) is calculated using the backpropagation algorithm. The parameter gradient indicates the degree of influence of the loss function on the network parameters and is used to guide the update of the weights. The weights of the first multilayer perceptron network (MLP_1) are updated according to the calculated parameter gradient to optimize the network performance and reduce the contrastive loss. Steps 2 to 4 are executed again using the updated first multilayer perceptron network, i.e., feature transformation and enhancement, calculation of contrastive loss, backpropagation, and weight update are performed again. This process continues until a specific stopping condition is met, such as reaching a predetermined number of training epochs or the contrastive loss converging to a low level. When the stopping condition is met, the parameters and structure of the MLP_1 have been optimized, enabling it to process image and text data more effectively and showing improved performance in multimodal contrast tasks, thus becoming the first target multilayer perceptron network (Target MLP_1).

[0056] The above training process ensures that the first multilayer perceptron network (MLP_1) can learn more meaningful representations from the input data. By continuously optimizing the network weights, the contrast loss between image features and corresponding text features is reduced, enabling the model to more accurately understand and associate multimodal information.

[0057] To better understand the training of the first multilayer perceptron network in this application, the following is combined with... Figure 3 A brief introduction to the training process of the first multilayer perceptron network.

[0058] Figure 3 This is a simplified schematic diagram of a single-stage multilayer perceptron network layer in a pre-trained model according to an embodiment of this application, as shown below. Figure 3 As shown, it specifically includes the following parts:

[0059] 1. MLP_0 represents a specific stage of a multilayer perceptron network layer, such as within the transformer module of a visual encoder. MLP_0 is the first-stage MLP layer, responsible for extracting and processing features from the input data, and is a key part of the visual encoder's processing of image information.

[0060] 2. Norm: Represents a normalization layer. Its function is to adjust the distribution of input data to better match the expectations of subsequent processing layers (such as MLP and attention layers), thereby accelerating the training process and improving the stability of the model.

[0061] 3. MultiHead-Attention: This refers to part of the multi-head attention mechanism and is one of the core components of the transformer model. Multi-head attention allows the model to focus on different parts of the input from different representation subspaces, thereby enhancing the model's ability to understand complex inputs.

[0062] In the first phase, apart from MLP_0, Figure 3 The Norm and MultiHead-Attention mechanisms and their associated connections are frozen, meaning their parameters are not updated during training. Training focuses on optimizing MLP_0 by processing a large amount of pre-trained data, enabling MLP_0 to learn the features and patterns of the input data, thereby improving the model's ability to handle specific tasks.

[0063] In the above steps, based on the first target multilayer perceptron network, the second multilayer perceptron network in the pre-trained model is trained in a second stage to obtain the second target multilayer perceptron network. This includes: using the first target multilayer perceptron network to infer the image data in the training dataset to obtain a first image feature output; clustering the training dataset based on the first image feature output to obtain a first classification result; determining the pseudo-label for each image data in the training dataset based on the first classification result, wherein the pseudo-label is used to determine the negative samples for training the second multilayer perceptron network; and training the second multilayer perceptron network in a second stage based on the pseudo-label, the training dataset, and the negative samples to obtain the second target multilayer perceptron network.

[0064] In some embodiments of this application, during the second stage of training, a first target multilayer perceptron network (Target MLP_1) obtained from the first stage training is used to infer the image data in the training dataset. This means that image data is input into Target MLP_1, processed, and a new set of image feature representations, i.e., the first image feature output, is generated. Clustering analysis is performed on the first image feature output. Clustering is an unsupervised machine learning technique used to discover the structure and patterns within a dataset. Clustering algorithms can include, but are not limited to, Gaussian mixture models, hierarchical clustering, DBSCAN, etc. Through clustering, image data with similar features can be grouped into the same category, and dissimilar image data can be grouped into different categories, thus obtaining a first classification result. The first classification result reflects the distribution of image data in the feature space. Based on the first classification result, a pseudo-label is assigned to each image data in the training dataset. A pseudo-label is a label automatically generated based on model prediction or data features, which can be used to guide further learning of the model, especially when there is insufficient supervision information. In the embodiments of this application, the pseudo-label is used to determine the negative samples during the training of the second multilayer perceptron network, i.e., samples with pseudo-labels different from the current image data will be labeled as negative samples for comparative learning. In the second training phase, the first image feature output generated by the first target multilayer perceptron network, the training dataset, and negative samples determined based on pseudo-labels are used to train the second multilayer perceptron network (MLP_2) in the pre-trained model. The training objective of MLP_2 is to further learn, based on the first phase of learning, how to transform input image data into better feature representations and how to more accurately match image and text features in multimodal contrastive learning. After the second phase of training, the optimized MLP_2 can process image data more effectively and exhibits better performance in multimodal contrastive learning tasks; this MLP_2 is now the second target multilayer perceptron network (Target MLP_2). The second target multilayer perceptron network not only inherits the advantages of the first target multilayer perceptron network but also learns new feature transformation capabilities, further enhancing the model's representational diversity.

[0065] It should be noted that before the second stage of training, the parameters of other parts of the pre-trained model (such as the image encoder, text encoder, and other layers in between) need to be frozen again.

[0066] The second-stage training process emphasizes the gradual optimization of the model and the hierarchical enhancement of feature representations. By using the output of the first-target multilayer perceptron network as the input for the second-stage training, and by using pseudo-labels obtained through clustering to guide the selection of negative samples, it is ensured that the second multilayer perceptron network learns deeper image features that are different from those of the first multilayer perceptron network. This helps to build a multi-expert fusion visual encoder model with rich representational capabilities.

[0067] In each subsequent stage, cluster labels from all previous stages can be concatenated to derive pseudo-labels. This approach fully leverages the learning outcomes of previous stages, improving the accuracy of pseudo-labels. Negative samples are also determined based on previously acquired pseudo-labels.

[0068] In step S206 of the above model training method, multiple target multilayer perceptron networks are fused to obtain a fused network, including: connecting multiple target multilayer perceptron networks using a learnable routing network to obtain a fused network.

[0069] In the multi-stage training of some embodiments of this application, a series of optimized Multilayer Perceptron Networks (MLPs), namely target MLPs, are obtained. Each target MLP (Target MLP_i) focuses on extracting and processing specific aspects or features of the input data, and these networks together constitute the model's diverse processing capabilities. To effectively integrate these target MLPs, a learnable router is used. This router acts as an intelligent "allocator," dynamically determining which target MLP outputs should be given more consideration based on the characteristics of the input data, and how to weight and combine these outputs. The output of each target MLP is passed to the router. The router adjusts its internal parameters based on the characteristics of the input data and the model's current task requirements to learn how to optimally allocate weights to different target MLPs. This weight allocation mechanism ensures that for specific input data, the model can focus more on the target MLPs that provide the most relevant and valuable information. Through the connections and weight allocation of the router, the outputs of multiple target MLPs are integrated into a unified representation, forming a fusion network (which can be represented as MoE). The fusion network not only incorporates the expertise of each target's multilayer perceptron network, but also flexibly adjusts its internal expert composition based on the characteristics of the input data, achieving a comprehensive visual encoder adaptable to various tasks and data conditions. Finally, the entire fusion network, as a whole, can be optimized through further training.

[0070] In step S208 of the above model training method, the fusion network is trained based on the training dataset to obtain the target fusion network. This includes: freezing the parameters of multiple target multilayer perceptron networks and initializing the routing parameters of the routing network; inputting the training dataset into the pre-trained model, wherein the training dataset obtains multiple feature representations after passing through multiple target multilayer perceptron networks in the pre-trained model, and the multiple feature representations obtain a fused feature representation after passing through the fusion network in the training dataset; determining the predicted classification result corresponding to the training dataset based on the fused feature representation; determining the loss function based on the predicted classification result and the true label corresponding to the training dataset; optimizing the routing parameters based on the loss function, and obtaining the target fusion network after optimization.

[0071] In some embodiments of this application, the process of obtaining the target fusion network includes two fusion stages. Fusion stage 1 includes: before starting to train the fusion network, the parameters of multiple target multilayer perceptron networks (TargetMLP_i) need to be frozen. This means that the parameters of these multilayer perceptron networks will remain unchanged during training, avoiding unnecessary updates during the training of the fusion network, thus preserving the optimization results obtained in the first stage of training. Simultaneously, the routing parameters of the routing network are initialized. These parameters will be learned through training to determine how to optimally allocate and combine features from different target multilayer perceptron networks. The training dataset is input into the pre-trained model, and the data is initially encoded by an image encoder and a text encoder. Then, these encoded data are fed into multiple target multilayer perceptron networks respectively, generating multiple feature representations. Each target multilayer perceptron focuses on processing different aspects of the data, so the feature representations output by each network are unique and complementary. The feature representations obtained from the multiple target multilayer perceptron networks are input into the routing network. The routing network weights and combines these feature representations according to its initialized routing parameters to generate a fused feature representation. This fused feature representation combines the advantages of multiple target multilayer perceptron networks, resulting in a more comprehensive and powerful data representation. Based on the generated fused feature representation, the pre-trained model (which includes optimized multiple target multilayer perceptron networks and a routing network in this step) determines the predicted classification result for the training dataset. This means the model predicts the category of each data point based on the fused feature representation. The predicted classification result is compared with the true labels corresponding to the training dataset to determine a loss function. The loss function measures the difference between the model's prediction and the true label. By calculating the loss function, the current performance of the fused network can be evaluated, and directions for improvement can be identified. Based on this loss function, the parameters of the routing network are updated using backpropagation and optimization methods (such as gradient descent), i.e., the routing parameters are optimized. The purpose of this process is to teach the routing network how to more effectively allocate and combine feature representations from multiple target multilayer perceptron networks to minimize classification errors and losses. The above steps are repeated, i.e., continuously inputting training data, generating predicted classification results, calculating the loss function, and optimizing the routing parameters, until the loss function converges to a low level or a predetermined number of training epochs are reached. During this process, the parameters of the routing network are gradually optimized, and the overall performance of the fused network is continuously improved. Once the training process is complete, meaning the routing parameters have been sufficiently optimized, the fusion network at this point becomes the Target Fusion Network. While retaining the characteristics of each target multilayer perceptron network, the Target Fusion Network achieves efficient information integration through an optimized routing network, thus exhibiting superior performance in classification tasks.

[0072] Following the training process described above, a fusion phase 2 is performed. This involves releasing all neural network layers of the pre-trained model (i.e., releasing the parameters of multiple target multilayer perceptron networks, Target MLP_i), and then training all parameters of the pre-trained model using the training data. This method further optimizes the performance of the target fusion network, resulting in the final target pre-trained model (or multi-expert fusion model) of the target fusion network.

[0073] To better understand the training process of the fusion network in this application, the following is combined with... Figure 4 The above training process will be briefly explained.

[0074] Figure 4 This is a schematic diagram of the overall structure of a fusion network according to an embodiment of this application, specifically including the following parts:

[0075] 1. Input layer: The model receives various input data, which may include information from images, text, or other modalities. The input data will be passed to different expert MLP networks or models for feature extraction and processing.

[0076] 2. Multiple MLP (Multilayer Perceptron) modules: These modules are labeled MLP_0, MLP_1, ..., MLP_n, representing expert models obtained at different training stages. Each expert model, after a specific training stage, learns different features and information from the input data, possessing its own unique processing capabilities and perspectives;

[0077] 3. Routing Network: The routing network is a key component connecting all expert models. Its role is to intelligently determine which expert models should process each input data based on the characteristics of the input data and the task requirements. The parameters of the routing network are learnable, meaning it can dynamically adjust according to the characteristics of the input data, selecting the optimal combination of experts, effectively avoiding information loss and feature suppression problems, and ensuring that each input data is processed by the most suitable model.

[0078] 4. Normalization layer (Norm): After the routing network, the model includes a normalization layer. Its role is to adjust and standardize the outputs from different expert models, ensuring that they have a uniform scale and distribution when fused, which is beneficial for subsequent multi-head attention mechanisms and model output layer processing.

[0079] 5. Multi-Head Attention: Used to process multi-expert fusion features from the output of the routing network and normalization layer. Multi-head attention can focus on different parts of the input data from different perspectives, thereby enhancing the model's ability to understand and process complex inputs.

[0080] 6. Re-normalization layer (Norm): Located after the multi-head attention mechanism, it further normalizes the features to provide a stable and consistent feature representation for the final output of the model.

[0081] Figure 4 Overall, this model demonstrates how multiple independent expert models are integrated through intelligent routing network selection and multi-head attention mechanisms to form a highly efficient and multi-view visual encoder model. This model design aims to fully leverage the strengths of each expert model, improving performance and accuracy in tasks such as event recognition through dynamic routing and feature fusion.

[0082] The target pre-trained model obtained by the model training method provided in this application embodiment can improve the performance of the visual encoder, while adapting to specific tasks and different datasets. It has strong processing capabilities for different types of input data, including images with different resolutions, color modes, lighting conditions, and different language styles. Furthermore, as the training and fusion phases progress and are optimized, the model performance continuously improves, better meeting the complex and ever-changing task requirements. Through this optimization, multimodal information can be utilized more fully, information loss can be reduced, the ability to capture detailed information can be enhanced, and the overall performance and accuracy of the model can be improved. Due to its outstanding performance in tasks such as event recognition, this target pre-trained model has broad application prospects and can be applied to multiple fields such as image recognition, video analysis, and intelligent surveillance.

[0083] Figure 5 This is a structural diagram of a model training apparatus according to an embodiment of this application, such as... Figure 5 As shown, the device includes:

[0084] The acquisition module 30 is used to acquire the training dataset, which is used to train the pre-trained model. The training dataset includes image data and corresponding text data.

[0085] Training module 32 is used to sequentially input the training dataset into multiple multilayer perceptron networks in the pre-trained model for training, thereby obtaining multiple target multilayer perceptron networks. The output of the previous multilayer perceptron network is used as the input of the next multilayer perceptron network. The multiple target multilayer perceptron networks are the multilayer perceptron networks obtained after the training of multiple multilayer perceptron networks.

[0086] Fusion module 34 is used to fuse multiple target multilayer perceptron networks to obtain a fused network;

[0087] The determination module 36 is used to train the fusion network based on the training dataset to obtain the target fusion network, and to determine the target pre-trained model based on the target fusion network.

[0088] The acquisition module 30, training module 32, fusion module 34, and determination module 36 in the above-mentioned model training device achieve the goal of improving the ability to capture detailed information, thereby realizing the technical effect of enhancing the flexibility and adaptability of the model. This solves the technical problem in related technologies where visual encoder models ignore many detailed information during the encoding process, resulting in poor accuracy.

[0089] In the training module of the aforementioned model training apparatus, the training module is also used to freeze all parameters in the pre-trained model except for the multiple multilayer perceptron networks before the training dataset is sequentially input into the multiple multilayer perceptron networks in the pre-trained model for training.

[0090] In the training module of the aforementioned model training device, the training module is further configured to input the training dataset into the first multilayer perceptron network in the pre-trained model for a first-stage training to obtain a first target multilayer perceptron network, wherein the first multilayer perceptron network is any one of the multiple multilayer perceptron networks in the pre-trained model; based on the first target multilayer perceptron network, a second-stage training is performed on the second multilayer perceptron network in the pre-trained model to obtain a second target multilayer perceptron network, wherein the second multilayer perceptron network is a multilayer perceptron network trained after the first multilayer perceptron network; based on the second target multilayer perceptron network, corresponding stage training is performed on other multilayer perceptron networks in the pre-trained model to obtain other target multilayer perceptron networks, wherein the other multilayer perceptron networks are multilayer perceptron networks trained after the second multilayer perceptron network, and each of the other multilayer perceptron networks corresponds to a stage training and obtains the target multilayer perceptron network of the corresponding stage; after all multilayer perceptron networks in the pre-trained model have completed the corresponding stage training, the target multilayer perceptron networks obtained from each stage training are determined as multiple target multilayer perceptron networks.

[0091] In the training module of the above-mentioned model training device, the training module is further used to perform the following steps: Step 1: After initializing the first multilayer perceptron network, the training dataset is encoded by the image encoder and text encoder in the pre-trained model to obtain initial features, wherein the initial features include image features and text features; Step 2: The initial features are input into the first multilayer perceptron network for feature transformation and enhancement to obtain first processed features, wherein the first processed features include first image features after processing image features and first text features after processing text features; Step 3: The first contrast loss between the first image features and the first text features is determined; Step 4: Backpropagation is performed on the first multilayer perceptron network according to the first contrast loss to obtain the parameter gradient of the first multilayer perceptron network; the weights of the first multilayer perceptron network are updated according to the parameter gradient, and the updated first multilayer perceptron network is used to iteratively execute steps 2 to 4 until the condition is met and the iteration stops, thereby obtaining the first target multilayer perceptron network.

[0092] In the training module of the above-mentioned model training device, the training module is further used to infer the image data in the training dataset using a first target multilayer perceptron network to obtain a first image feature output; to cluster the training dataset based on the first image feature output to obtain a first classification result; to determine the pseudo label of each image data in the training dataset based on the first classification result, wherein the pseudo label is used to determine the negative sample for training the second multilayer perceptron network; and to perform a second-stage training on the second multilayer perceptron network based on the pseudo label, the training dataset, and the negative sample to obtain the second target multilayer perceptron network.

[0093] In the fusion module of the above-mentioned model training device, the fusion module is also used to connect multiple target multilayer perceptron networks using a learnable routing network to obtain a fusion network.

[0094] In the determination module of the above-mentioned model training device, the determination module is also used to freeze the parameters of multiple target multilayer perceptron networks, initialize the routing parameters of the routing network, input the training dataset into the pre-trained model, wherein the training dataset is processed by multiple target multilayer perceptron networks in the pre-trained model to obtain multiple feature representations, the multiple feature representations are processed by the fusion network in the training dataset to obtain fused feature representations; the predicted classification result corresponding to the training dataset is determined based on the fused feature representations; the loss function is determined based on the predicted classification result and the true label corresponding to the training dataset; the routing parameters are optimized based on the loss function, and the target fusion network is obtained after optimization.

[0095] It should be noted that, Figure 5 The apparatus shown is used to perform model training. Figure 2The model training method shown above is also applicable to the device used for training this model, and will not be repeated here.

[0096] This application also provides an electronic device, which includes a memory and a processor. The memory stores program instructions, and the processor is connected to the memory to execute program instructions that perform the following functions: acquiring a training dataset, wherein the training dataset is used to train a pre-trained model, and the training dataset includes image data and corresponding text data; sequentially inputting the training dataset into multiple multilayer perceptron networks in the pre-trained model for training to obtain multiple target multilayer perceptron networks, wherein the output of the previous multilayer perceptron network is used as the input of the next multilayer perceptron network, and the multiple target multilayer perceptron networks are multilayer perceptron networks obtained after the training of multiple multilayer perceptron networks; fusing the multiple target multilayer perceptron networks to obtain a fused network; training the fused network based on the training dataset to obtain a target fused network, and determining a target pre-trained model based on the target fused network.

[0097] It should be noted that the aforementioned electronic equipment is used to perform Figure 2 The model training method shown above also applies to this electronic device, and will not be repeated here.

[0098] This application embodiment also provides a non-volatile storage medium, which includes a stored computer program. The device containing the non-volatile storage medium executes the following model training method by running the computer program: acquiring a training dataset, wherein the training dataset is used to train a pre-trained model, and the training dataset includes image data and corresponding text data; sequentially inputting the training dataset into multiple multilayer perceptron networks in the pre-trained model for training, obtaining multiple target multilayer perceptron networks, wherein the output of the previous multilayer perceptron network is used as the input of the next multilayer perceptron network, and the multiple target multilayer perceptron networks are multilayer perceptron networks obtained after the training of the multiple multilayer perceptron networks; fusing the multiple target multilayer perceptron networks to obtain a fused network; training the fused network based on the training dataset to obtain a target fused network, and determining a target pre-trained model based on the target fused network.

[0099] It should be noted that the aforementioned non-volatile storage media is used for execution. Figure 2 The model training method shown above is also applicable to this non-volatile storage medium, and will not be repeated here.

[0100] This application also provides a computer program product, including a non-volatile computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the model training method in various embodiments of this application.

[0101] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the model training method in various embodiments of this application.

[0102] This application also provides a computer program that, when executed by a processor, implements the steps of the model training method in various embodiments of this application.

[0103] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0104] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0105] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0106] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0107] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0108] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0109] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for model training, the method comprising: The method comprises the following steps: acquiring a training data set, wherein the training data set is used for training a pre-training model, and the training data set comprises image data and corresponding text data; inputting the training data set into a plurality of multilayer perceptron networks in the pre-training model in sequence for training, to obtain a plurality of target multilayer perceptron networks, wherein the output result of a previous multilayer perceptron network is used as the input of a next multilayer perceptron network, and the plurality of target multilayer perceptron networks are obtained after the plurality of multilayer perceptron networks are trained; fusing the plurality of target multilayer perceptron networks to obtain a fusion network; training the fusion network according to the training data set to obtain a target fusion network, and determining a target pre-training model according to the target fusion network; The method further comprises the following steps:

2. The method of claim 1, wherein, using a first target multilayer perceptron network to perform inference on the image data in the training data set to obtain a first image feature output, wherein the first target multilayer perceptron network is obtained by inputting the training data set into a first multilayer perceptron network in the pre-training model for first-stage training, and the first multilayer perceptron network is any one of the plurality of multilayer perceptron networks in the pre-training model; clustering the training data set according to the first image feature output to obtain a first classification result; determining a pseudo label of each image data in the training data set according to the first classification result, wherein the pseudo label is used to determine a negative sample for training a second multilayer perceptron network, and the second multilayer perceptron network is a multilayer perceptron network trained after the first multilayer perceptron network; and performing second-stage training on the second multilayer perceptron network according to the pseudo label, the training data set and the negative sample to obtain a second target multilayer perceptron network. Before inputting the training data set into the plurality of multilayer perceptron networks in the pre-training model in sequence for training, the method further comprises the following step:

3. The method of claim 1, wherein, freezing all parameters in the pre-training model except the plurality of multilayer perceptron networks. The step of inputting the training data set into the plurality of multilayer perceptron networks in the pre-training model in sequence for training to obtain a plurality of target multilayer perceptron networks comprises the following steps: performing second-stage training on a second multilayer perceptron network in the pre-training model according to the first target multilayer perceptron network to obtain a second target multilayer perceptron network; performing corresponding stage training on other multilayer perceptron networks in the pre-training model according to the second target multilayer perceptron network to obtain other target multilayer perceptron networks, wherein the other multilayer perceptron networks are multilayer perceptron networks trained after the second multilayer perceptron network, each multilayer perceptron network of the other multilayer perceptron networks corresponds to one stage of training and obtains a target multilayer perceptron network of the corresponding stage; after all the multilayer perceptron networks in the pre-training model complete corresponding stage training, determining the target multilayer perceptron networks obtained in each stage of training as the plurality of target multilayer perceptron networks.

4. The method of claim 3, wherein, inputting the training data set into a first multi-layer perceptron network in the pre-training model for first stage training to obtain a first target multi-layer perceptron network, comprising: Step 1: after initializing the first multi-layer perceptron network, encoding the training data set through an image encoder and a text encoder in the pre-training model to obtain initial features, wherein the initial features include image features and text features; Step 2: inputting the initial features into the first multi-layer perceptron network for feature conversion and enhancement to obtain first processed features, wherein the first processed features include first image features processed from the image features and first text features processed from the text features; Step 3: determining a first contrast loss between the first image features and the first text features; Step 4: performing back propagation on the first multi-layer perceptron network according to the first contrast loss to obtain parameter gradients of the first multi-layer perceptron network; updating weights of the first multi-layer perceptron network according to the parameter gradients, and iteratively executing steps 2 to 4 using the updated first multi-layer perceptron network until the iteration is stopped after meeting a condition, to obtain the first target multi-layer perceptron network.

5. The method of claim 1, wherein, fusing the plurality of target multi-layer perceptron networks to obtain a fusion network, comprising: connecting the plurality of target multi-layer perceptron networks using a learnable routing network to obtain the fusion network.

6. The method of claim 5, wherein, training the fusion network according to the training data set to obtain a target fusion network, comprising: freezing parameters of the plurality of target multi-layer perceptron networks, and initializing routing parameters of the routing network, inputting the training data set into the pre-training model, wherein the training data set obtains a plurality of feature representations after passing through the plurality of target multi-layer perceptron networks in the pre-training model, and obtains a fusion feature representation after passing through the fusion network in the training data set; determining a predicted classification result corresponding to the training data set according to the fusion feature representation; determining a loss function according to the predicted classification result and a true label corresponding to the training data set; optimizing the routing parameters according to the loss function, and obtaining the target fusion network after optimization is completed.

7. An apparatus for model training, the apparatus comprising: comprising: an acquisition module configured to acquire a training data set, wherein the training data set is used for training a pre-training model, and the training data set includes image data and corresponding text data; The training module is configured to sequentially input the training data set into a plurality of multilayer perceptron networks in the pre-training model for training to obtain a plurality of target multilayer perceptron networks, wherein an output result of a previous multilayer perceptron network is used as an input of a next multilayer perceptron network, and the plurality of target multilayer perceptron networks are multilayer perceptron networks obtained after training of the plurality of multilayer perceptron networks is completed; further comprising: using a first target multilayer perceptron network to infer image data in the training data set to obtain a first image feature output, wherein the first target multilayer perceptron network is obtained by inputting the training data set into a first multilayer perceptron network in the pre-training model for first-stage training, and the first multilayer perceptron network is any one of the plurality of multilayer perceptron networks in the pre-training model; clustering the training data set according to the first image feature output to obtain a first classification result; determining a pseudo label of each image data in the training data set according to the first classification result, wherein the pseudo label is used to determine a negative sample for training a second multilayer perceptron network, and the second multilayer perceptron network is a multilayer perceptron network trained after the first multilayer perceptron network; and performing second-stage training on the second multilayer perceptron network according to the pseudo label, the training data set and the negative sample to obtain a second target multilayer perceptron network; The fusion module is configured to fuse the plurality of target multilayer perceptron networks to obtain a fusion network. The determination module is configured to train the fusion network according to the training data set to obtain a target fusion network, and determine a target pre-training model according to the target fusion network.

8. An electronic device, comprising: The memory is configured to store program instructions. ​ The processor is connected with the memory and used for executing program instructions to realize the following functions: obtaining a training data set, wherein the training data set is used for training a pre-training model, and the training data set comprises image data and corresponding text data; inputting the training data set into multiple multilayer perceptron networks in the pre-training model in sequence for training to obtain multiple target multilayer perceptron networks, wherein an output result of a previous multilayer perceptron network is used as an input of a next multilayer perceptron network, and the multiple target multilayer perceptron networks are the multilayer perceptron networks obtained after training of the multiple multilayer perceptron networks is completed; fusing the multiple target multilayer perceptron networks to obtain a fusion network; training the fusion network according to the training data set to obtain a target fusion network, and determining a target pre-training model according to the target fusion network; further comprising: using a first target multilayer perceptron network to infer the image data in the training data set to obtain a first image feature output, wherein the first target multilayer perceptron network is obtained by inputting the training data set into a first multilayer perceptron network in the pre-training model for first-stage training, and the first multilayer perceptron network is any one of the multiple multilayer perceptron networks in the pre-training model; clustering the training data set according to the first image feature output to obtain a first classification result; determining a pseudo label of each image data in the training data set according to the first classification result, wherein the pseudo label is used for determining a negative sample for training a second multilayer perceptron network, and the second multilayer perceptron network is a multilayer perceptron network trained after the first multilayer perceptron network; training the second multilayer perceptron network according to the pseudo label, the training data set and the negative sample to obtain a second target multilayer perceptron network.

9. A non-volatile storage medium, comprising: The non-volatile storage medium comprises a stored computer program, wherein a device in which the non-volatile storage medium is located executes the method for training a model according to any one of claims 1 to 6 by running the computer program.

10. A computer program product comprising computer instructions, characterized in that, The computer instructions are executed by the processor to realize the method for training a model according to any one of claims 1 to 6.

Images