Method and device for model training, equipment, storage medium and program product

Abstract

The invention provides a method, device and equipment for model training, a storage medium and a program product. The method comprises: performing a pre-training process on a machine learning model using a pre-training data set to obtain a pre-trained machine learning model, the machine learning model being configured to process a multi-modal query request and generate a response to the multi-modal query request; and performing a first fine tuning stage on the pre-trained machine learning model using the first training data set to obtain a fine-tuned machine learning model, the first fine tuning stage comprising: dividing the first training data set into a plurality of first training data subsets, a training sample in each first training data subset comprises a sample image, a sample text and a corresponding response label, and each sample image in the plurality of first training data subsets is divided into a plurality of image blocks according to different sizes; and utilizing the plurality of first training data subsets to finely adjust the pre-trained machine learning model to obtain an updated machine learning model.

Description

Technical Field

[0001] The exemplary embodiments of the present disclosure generally relate to the field of computers, and in particular, to methods, devices, electronic devices, computer-readable storage media, and computer program products for model training. Background Art

[0002] With the development of machine learning technology, machine learning models can be used to perform tasks in a variety of application environments. Multimodal language models have attracted much attention because of their ability to understand and generate responses guided by visual input. The model can process and fuse multiple types of data inputs and support applications across various fields, including image captioning, visual question answering, and video analysis. It is usually necessary to use a large amount of multimodal data to train multimodal language models, which leads to high training costs, long training time, and poor training efficiency. Summary of the invention

[0003] In the first aspect of the present disclosure, a method for model training is provided. The method includes: performing a pre-training process on a machine learning model using a pre-training data set to obtain a pre-trained machine learning model, the machine learning model is configured to process a multimodal query request and generate a response to the multimodal query request, the multimodal query request at least including image data and text data; and performing a first fine-tuning phase on the pre-trained machine learning model using a first training data set to obtain a fine-tuned machine learning model, wherein the first fine-tuning phase includes: dividing the first training data set into a plurality of first training data subsets, the training samples in each first training data subset include a sample image, a sample text, and a corresponding response label, and each sample image in the plurality of first training data subsets is divided into a plurality of image blocks of different sizes; using the plurality of first training data subsets to fine-tune the pre-trained machine learning model to obtain an updated machine learning model.

[0004] In the second aspect of the present disclosure, a device for model training is provided. The device includes: a pre-training execution module, configured to perform a pre-training process on a machine learning model using a pre-training data set to obtain a pre-trained machine learning model, the machine learning model is configured to process a multimodal query request and generate a response to the multimodal query request, the multimodal query request at least including image data and text data; and a fine-tuning execution module, configured to perform a first fine-tuning phase on the pre-trained machine learning model using a first training data set to obtain a fine-tuned machine learning model, wherein the first fine-tuning phase includes: dividing the first training data set into a plurality of first training data subsets, the training samples in each first training data subset include a sample image, a sample text, and a corresponding response label, and each sample image in the plurality of first training data subsets is divided into a plurality of image blocks of different sizes; using the plurality of first training data subsets to fine-tune the pre-trained machine learning model to obtain an updated machine learning model.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The electronic device includes: at least one processor; and at least one memory, the at least one memory is coupled to the at least one processor and stores instructions for execution by the at least one processor, and when the instructions are executed by the at least one processor, the electronic device executes the method according to the first aspect of the present disclosure.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor executes the method according to the first aspect of the present disclosure.

[0007] In a fifth aspect of the present disclosure, a computer program product is provided, which is tangibly stored in a computer storage medium and includes computer executable instructions, which, when executed by a device, cause the device to perform the method of the first aspect.

[0008] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0010] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0011] Figure 2 A schematic diagram showing an example environment for model training and application according to some embodiments of the present disclosure is shown;

[0012] Figure 3 An example for model training according to some embodiments of the present disclosure is shown;

[0013] Figure 4A and Figure 4B An example for model training according to some embodiments of the present disclosure is shown;

[0014] Figure 5 A flowchart of a method for model training according to some embodiments of the present disclosure is shown;

[0015] Figure 6 An exemplary structural block diagram of an apparatus for model training according to some embodiments of the present disclosure is shown; and

[0016] Figure 7 A block diagram of an electronic device in which one or more embodiments of the present disclosure may be implemented is shown. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below.

[0019] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0020] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition, use, storage or deletion of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0021] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0022] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information, so that the user can independently choose whether to provide personal information to software or hardware such as electronic devices, applications, servers or storage media that execute operations of the technical solution of the present disclosure based on the prompt message.

[0023] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information is sent to the user in a manner such as a pop-up window, in which the prompt information can be presented in text form. In addition, the pop-up window can also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0024] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0025] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning is a machine learning algorithm that processes inputs and provides corresponding outputs by using multi-layer processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0026] A "neural network" is a machine learning network based on deep learning. A neural network is capable of processing inputs and providing corresponding outputs, and typically includes an input layer and an output layer and one or more hidden layers between the input layer and the output layer. Neural networks used in deep learning applications typically include many hidden layers, thereby increasing the depth of the network. The layers of a neural network are connected in sequence so that the output of the previous layer is provided as input to the next layer, where the input layer receives the input of the neural network and the output of the output layer serves as the final output of the neural network. Each layer of a neural network includes one or more nodes (also called processing nodes or neurons), each of which processes input from the previous layer.

[0027] Generally, machine learning can be roughly divided into three stages, namely the training stage, the testing stage, and the application stage (also called the inference stage). In the training stage, a given model can be trained using a large amount of training data, and the parameter values ​​are continuously updated iteratively until the model can obtain consistent inferences that meet the expected goals from the training data. Through training, the model can be considered to be able to learn the association between input and output (also called input-to-output mapping) from the training data. The parameter values ​​of the trained model are determined. In the testing stage, the test input is applied to the trained model to test whether the model can provide the correct output, thereby determining the performance of the model. The testing stage can sometimes be integrated into the training stage. In the application or inference stage, the trained model can be used to process the actual model input based on the parameter values ​​obtained through training to determine the corresponding model output.

[0028] In the process of training a multimodal language model, in order to achieve accurate and effective alignment between features of different modalities, it is often necessary to use a very large amount of multimodal data to carry out multi-stage training of the model. This is because data of different modalities usually include different information and correspond to different features (for example, images have visual features, videos also include temporal features compared to images, and text has semantic symbol features). In order to make them adapt to each other and work together in the model, a large number of samples and multi-stage training strategies are required to allow the model to fully learn the association patterns between different modalities. Although some model training schemes have been proposed to help improve model performance, traditional schemes to increase the amount of training data usually have the problem of excessive resource consumption. It is hoped that a small amount of data can be used to facilitate the efficient training of multimodal language models.

[0029] In view of this, according to an embodiment of the present disclosure, an improved scheme for model training is provided. According to the scheme of the embodiment of the present disclosure, a pre-training process is performed on a machine learning model using a pre-training data set to obtain a pre-trained machine learning model, and the machine learning model is configured to process a multimodal query request and generate a response to the multimodal query request, and the multimodal query request includes at least image data and text data. A first fine-tuning phase is performed on the pre-trained machine learning model using a first training data set to obtain a fine-tuned machine learning model. The first fine-tuning phase includes: dividing the first training data set into multiple first training data subsets, the training samples in each first training data subset include a sample image, a sample text, and a corresponding response label, and each sample image in the multiple first training data subsets is divided into multiple image blocks of different sizes; using multiple first training data subsets to fine-tune the pre-trained machine learning model to obtain an updated machine learning model.

[0030] In this way, the machine learning model can be pre-trained and fine-tuned, and in the fine-tuning stage, the image can be divided into different granularities to extract image features of different granularities for model fine-tuning, which can help the machine learning model better understand and process image data, and help improve the model quality of the machine learning model and the accuracy of the responses output by the machine learning model.

[0031] Figure 1 A schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented is shown. In the example environment 100, an electronic device 110 can receive a query request 104 from a user 102, and use a machine learning model 120 to generate a response 130 to the query request 104.

[0032] The query request 104 may include content of at least one modality, and the at least one modality may include any appropriate modality such as video, image, text, audio, etc. As an example, the query request 104 may include an image 106 and a query text 108 for the image 106. When including data of multiple different modalities, the query request 104 may also be referred to as a multimodal query request. The image 106 may be an image that the user 102 attempts to capture via his / her corresponding image capture equipment, or may be an image that is captured in advance and uploaded in response to receiving a user selection.

[0033] The machine learning model 120 may be based on any appropriate model structure, including but not limited to a Transformer model, a convolutional neural network (CNN), a recurrent neural network (RNN), a deep neural network (DNN), etc. In some embodiments, the machine learning model 120 may be based on a language model (LM). The machine learning model 120 may also be based on other appropriate models.

[0034] The machine learning model 120 may include one or more machine learning models. It should be noted that if the machine learning model 120 includes multiple machine learning models, these multiple machine learning models may have different structures, uses, and functions, which is not limited in this disclosure. As an example only, the machine learning model 120 may include a multimodal language model (MLLM).

[0035] The machine learning model 120 may include at least a visual encoder model 122 (Visual encoder), an adapter model 124 (Adapter), and a language model 126. The visual encoder model 122 is configured to extract visual features from a received image. The adapter model 124 is configured to map the visual features extracted by the visual encoder model 122 into a portion of the input features of the language model 126, so that the language model 126 can understand and process the visual features. The language model 126 can have question-answering capabilities by learning from a large amount of corpus, and can determine specific requirements in the input based on the received input, and generate corresponding responses to meet the determined requirements.

[0036] The electronic device 110 may be any type of device with computing capabilities, including a terminal device or a server device. The terminal device may be any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, a personal communication system (PCS) device, a personal navigation device, a personal digital assistant (PDA), an audio / video player, a digital camera / camcorder, a positioning device, a television receiver, a radio broadcast receiver, an electronic book device, a gaming device or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof.

[0037] The server-side device can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks, and big data and artificial intelligence platforms. Server-side devices can include computing systems / servers such as mainframes, edge computing nodes, computing devices in cloud environments, and so on.

[0038] It should be understood that the structure and function of the various elements in the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure.

[0039] Figure 2 A schematic diagram of an example environment 200 for model training and application according to some embodiments of the present disclosure is shown. Figure 2 The example environment 200 of FIG. 1 shows three different stages of the model, including a pre-training stage 202, a fine-tuning stage 204, and an application stage 206. After the pre-training stage 202 or the fine-tuning stage 204 is completed, there may also be a testing stage, which is not shown in the figure.

[0040] In the pre-training stage 202, the model pre-training system 210 is configured to perform pre-training of the machine learning model 120 using the pre-training data set 212. At the beginning of pre-training, each component in the machine learning model 120 may have an initial parameter value. The pre-training process is to update the parameter values ​​of the machine learning model 120 to the expected value based on the data in the pre-training data set 212. The pre-training task is used to help update the parameters of the machine learning model 120.

[0041] In the pre-training stage 202, the machine learning model 120 can learn powerful generalization capabilities through a pre-training data set 212 including a large amount of data. After the pre-training is completed, the parameter values ​​of the machine learning model 120 have been updated with the pre-trained parameter values. In some embodiments, the pre-trained machine learning model 120 can more accurately extract the feature representation of the image and more accurately map the feature representation to a part of the input feature of the language model.

[0042] The pre-trained machine learning model 120 may be provided to the fine-tuning stage 204, and fine-tuned by the model fine-tuning system 220 for different downstream tasks. In the fine-tuning stage 204, the training data set 222 is further used to adjust the parameter values ​​of the machine learning model 120. In some embodiments, for the scenario of multimodal response, the pre-trained machine learning model 120 may be further updated using a data set constructed according to an embodiment of the present disclosure (including multiple images, query texts for images, and responses to query texts). As an example, the training data set 222 may include images, specific questions for images, and responses to specific questions. The specific questions may be complex questions that users may actually ask, which may include, for example, "How many XX objects does this image include?", "What is the positional relationship between object A and object B in the image?", and other questions.

[0043] During fine-tuning, the corresponding training algorithm is also used to update and adjust the parameters of the machine learning model 120. Since the model has learned a lot of knowledge from the training data in the pre-training stage, a small amount of training data can be used in the fine-tuning stage 204 to obtain a downstream task model that meets the expectations.

[0044] In some embodiments, a testing phase may be included after the fine-tuning phase 204, and the model performance of the machine learning model 120 may continue to be tested using a test data set. The data set used in the testing phase is of the same type as that in the fine-tuning phase. In some embodiments, for scenarios of multimodal responses, a test data set constructed according to an embodiment of the present disclosure (including multiple images, query texts for images, and responses to query texts) may be used to test the model performance of the trained machine learning model 120.

[0045] Of course, in addition to supporting scenarios of multimodal responses, the machine learning model 120 can also support scenarios of unimodal responses, such as scenarios where the query request only includes text or only includes images. Or in addition to image data and text, a multimodal query request can also include data in other modalities, such as audio. In this case, during the pre-training and fine-tuning stages, the training data set can include query requests of the corresponding modality, and the corresponding responses. Similarly, during the testing stage, a test data set including corresponding query requests and their responses can also be constructed to test the model performance of the machine learning model 120 in the corresponding scenario.

[0046] In the application stage 206, the obtained machine learning model 120 has trained parameter values ​​and can be provided to the model application system 230 for use. In the application stage 206, the machine learning model 120 can be used to process the corresponding query request 104 in the actual scenario and provide a corresponding response 130. In some embodiments, for a multimodal response scenario, the query request 104 includes an image and a corresponding query text.

[0047] exist Figure 2 In the embodiment, the model pre-training system 210, the model fine-tuning system 220, and the model application system 230 can be deployed at any appropriate electronic device. It is understood that the model pre-training system 210, the model fine-tuning system 220, and the model application system 230 can be deployed at the same electronic device or at different electronic devices, and the present disclosure does not limit this. As an example only, the model pre-training system 210, the model fine-tuning system 220, and the model application system 230 can all be deployed at the electronic device 110.

[0048] It should be understood that Figure 2 The components and arrangements in the example environment 200 shown are merely examples, and a computing system suitable for implementing the exemplary implementations described in the present disclosure may include one or more different components, other components, and / or different arrangements. For example, although shown as separate, the model pre-training system 210, the model fine-tuning system 220, and the model application system 230 may be integrated into the same system or device. For example, at least the model pre-training system 210 and the model fine-tuning system 220 may be integrated into a model training system or device. Implementations of the present disclosure are not limited in this respect.

[0049] Some example embodiments of the present disclosure will be described below with reference to the accompanying drawings. The model training method involved in the present disclosure can be implemented in an electronic device 110. The machine learning model to be trained (e.g., machine learning model 120) can be configured to process multimodal query requests and generate responses to multimodal query requests, and the multimodal query requests include at least image data and text data. The machine learning model 120 includes at least: a visual encoder model (e.g., visual encoder model 122), an adapter model (e.g., adapter model 124), and a language model (e.g., language model 126).

[0050] In some embodiments, the machine learning model 120 may also include a text encoder model (not shown), which is configured to extract text features of text data. In some embodiments, the text encoder model and the language model 126 can be regarded as a whole, so the text features extracted by the text encoder do not need to be feature mapped by the corresponding adapter model. The input of the language model 126 may include the feature portion mapped from the visual feature by the adapter model 124, and the text feature portion mapped by the text encoder.

[0051] In an embodiment of the present disclosure, the electronic device 110 may perform a pre-training process on the machine learning model 120 using a pre-training data set (eg, the pre-training data set 212) to obtain a pre-trained machine learning model 120. Figure 3 , Figure 3 An example architecture 300 for model training according to some embodiments of the present disclosure is shown. Figure 3 As shown, in some embodiments, the pre-training process 302 may include a stage 310 and a stage 320 after the stage 310. The pre-training process 302 may correspond to Figure 2 The pre-training stage 202 in is implemented at the model pre-training system 210.

[0052] Stage 310 may also be referred to as a first pre-training stage or an alignment pre-training stage, etc. In stage 310, the model pre-training system 210 may train the machine learning model 120 using the pre-training dataset 212. In some embodiments, in stage 310, the model pre-training system 210 may only update the parameters of the adapter model 124, keeping the parameters of the visual encoder model 122 and the language model 126 unchanged.

[0053] Stage 320 may also be referred to as a second pre-training stage or a multi-task pre-training stage, an advanced pre-training stage, etc. In stage 320, the model pre-training system 210 may train the machine learning model 120 using another pre-training data set 212. In stage 320, the model pre-training system 210 may further update the parameters of the visual encoder model 122 and the adapter model 124, and only keep the parameters of the language model 126 unchanged.

[0054] The pre-training data set used in the pre-training stage may include sample images, basic questions for the sample images, and answers to the basic questions. Basic questions may include, for example, general questions such as "What objects does this image include?" and "Does this image include a certain object?" In each pre-trained training sample, the sample image and the basic question can be regarded as the model input of the machine learning model 120. The sample image is extracted with visual features by the visual encoder model 122, and the text features corresponding to the basic questions can be extracted from the sample text by a separate text encoder model in the machine learning model 120, or can be extracted from the sample text by the text encoder in the language model 126. The visual features extracted by the visual encoder model 122 will perform feature mapping via the adapter model 124 to obtain the mapped visual features. In some embodiments, the text features and the mapped visual features can be regarded as multiple input units (also called tokens) of the language model 126.

[0055] The machine learning model 120 may be updated in an autoregressive manner. For ease of understanding, Figure 4A FIG. 1 shows how the machine learning model 120 is trained when performing a simple text prediction task. Figure 4A As shown in example 400A, the language model 126 in the machine learning model 120 can predict the first N tokens of the input (e.g., the tokens shown in the figure) at a time. <user>To Token <am>) predicts the next N+1th token (e.g. the token shown in the figure <fine>). refer to Figure 4B During training, in order to complete the training of the entire text at one time, the lower triangular attention mask shown in Example 400B can be used to constrain the calculation of the self-attention mechanism (Self-Attention) of the language model 126 to ensure that the network can only use the previous features at each token. For example, the following formula can be used to determine the cross entropy loss predicted using each token, and the model loss can be optimized based on the cross entropy loss:

[0056]

[0057] Considering the scenario of multimodal input, Figure 4A The text tokens shown can be replaced with text tokens corresponding to the question (i.e., text features) and visual tokens corresponding to the image (i.e., visual feature comments after mapping via the adapter model 124). The output of the language model 126 includes predictions of each token corresponding to the answer. The predicted tokens can be converted into corresponding text and / or images (depending on the language model 126's understanding of the input requirements and content generation). In pre-training or fine-tuning, the parameters of one or more components in the model can be continuously updated based on the difference between the answer predicted by the language model 126 and the actual answer given in the pre-training dataset or training dataset (depending on the different stages, the parameters to be updated are also different). Ultimately, the machine learning model 120 as a whole is trained to minimize the difference between the predicted answer and the actual answer (i.e., the training loss) or achieve the expected goal.

[0058] In pre-training (or subsequent fine-tuning), the model parameters are updated by minimizing or reducing the training loss at each stage. In the training algorithm, the gradient of the model loss for the model parameters to be updated can be calculated, and the model parameters can be updated by the stochastic gradient descent method, so as to minimize the loss value or continuously reduce it to the expected target. The model training process discussed above can be applied to the various pre-training or fine-tuning stages mentioned in the embodiments of the present disclosure.

[0059] It is understood that the pre-training data used to train the machine learning model 120 in stage 310 and stage 320 are different. As an example, the machine learning model 120 can be trained with a portion of the pre-training data set 212 in stage 310, and the machine learning model 120 can be trained with the other portion of the pre-training data set 212 in stage 320. The amount of data used for model training in stage 320 can be higher than the amount of data used for model training in stage 310. For example, the machine learning model 120 can be trained with millions of data in stage 310, and the machine learning model 120 can be trained with tens of millions of data in stage 320.

[0060] According to the above-mentioned pre-training strategy, in the two stages of the pre-training process 302, in the previous stage (i.e., stage 310), the model pre-training system 210 can only train the adapter model 124, and the adapter model 124 trained in this stage can more accurately map the visual features output by the visual encoder model 122 into a part of the input features of the language model 126, thereby realizing the alignment of the visual features with the text semantic features of the language model 126 itself. This can help the language model 126 to better understand the image. In the subsequent stage (i.e., stage 320), the model pre-training system 210 can train the visual encoder model 122 and the adapter model 124, which helps to further improve the accuracy of the visual encoder model 122 in extracting visual features and the accuracy of the adapter model 124 in mapping features. The machine learning model 120 pre-trained in stage 320 can have perceptual understanding capabilities such as fine-grained entity recognition, object relationship recognition, optical character recognition (OCR) information reading, and visual content summarization.

[0061] After pre-training, the electronic device 110 can use the training data set (e.g., training data set 222) to perform a fine-tuning process on the machine learning model 120 to obtain a fine-tuned machine learning model 120. The fine-tuning process may include at least one fine-tuning stage, and different fine-tuning stages may correspond to different training data sets. At least one fine-tuning stage includes at least a first fine-tuning stage, and the electronic device 110 can use the first training data set to perform a first fine-tuning stage on the pre-trained machine learning model 120 to obtain a fine-tuned machine learning model 120. The first training data set may include training samples related to multiple task types. Different task types correspond to different categories. Task types may include, for example, counting, mathematical reasoning, task understanding, task planning, and the like.

[0062] Continue to refer Figure 3 In some embodiments, the fine-tuning process 304 (which may correspond to Figure 2 The fine-tuning stage 204 in the embodiment of the present invention (implemented at the model fine-tuning system 220) includes at least stage 330. Stage 330 may also be referred to as a first fine-tuning stage or a knowledge learning stage, etc. In stage 330, the model fine-tuning system 220 may use the first training data set to fine-tune the machine learning model 120. In stage 330, the model fine-tuning system 220 may update the parameters of the visual encoder model 122, the adapter model 124, and the language model 126.

[0063] Specifically, in stage 330, the model fine-tuning system 220 can divide the first training data set into multiple first training data subsets. The training samples in each first training data subset include a sample image, sample text, and a response label corresponding to the sample text. In a multimodal response scenario, the sample text is also the query text, and the response label can indicate the response to the query text. Each sample image in the multiple first training data subsets is divided into multiple image patches according to different sizes. Different sizes can include any suitable sizes such as 48×48, 124×124, etc., and the sizes in actual applications can be flexibly set based on user needs.

[0064] The multiple first training data subsets can include any suitable number of first training data subsets. By way of example only, it can include 3 first training data subsets. In some embodiments, the divided sizes of the sample images in different first training data subsets can be different. As an example, the images in the first training data subset A can be divided into X image patches according to a larger size, the images in the second training data subset B can be divided into Y image patches according to a smaller size, and the images in the third training data subset C can be divided into Z image patches according to an even smaller size. It can be understood that the smaller the size, the more the number of divided image patches, that is, X < Y < Z. It can be understood that the number of sizes to be divided is not limited to three and can be more.

[0065] The visual encoder model 122 is configured to extract visual feature units one by one according to the divided image patches, and the visual feature unit corresponding to each image patch can be mapped by the adapter model 124 into an input unit (also called a token) of the language model 126. It can be seen that if the granularity of the image patch division is finer, it means that more visual feature units will be extracted for the same image, which in turn enables the language model 126 to obtain more feature information about the image. In the training stage, by dividing the sample images for training at multiple different sizes, different granularity feature information can be extracted for the language model 126 to process. Such a training strategy of gradually increasing the number of image divisions can enable the language model 126 to better understand visual information at different granularities, thereby improving the model robustness.

[0066] The model fine-tuning system 220 can use multiple first training data subsets to fine-tune the pre-trained machine learning model 120 to obtain an updated machine learning model 120. In some embodiments, for each training sample in the first training data subset, the model fine-tuning system 220 can use the pre-trained machine learning model 120 to extract visual features from multiple image blocks divided from the sample image in the training sample and extract text features from the sample text in the training sample. The text features can be extracted from the sample text by a separate text encoder model in the machine learning model 120, or extracted from the sample text by a text encoder in the language model 126.

[0067] The model fine-tuning system 220 can use the pre-trained machine learning model 120 to generate predicted responses based on visual features and text features, and fine-tune the pre-trained machine learning model 120 based on the difference between the predicted responses and the response labels in the training samples. The training method of the machine learning model 120 in this stage is the same as that in the above reference. Figure 4A and Figure 4B Similar to the description in .

[0068] In some embodiments, the fine-tuning process 304 may also include a stage 340 and a stage 350. Stage 340 may also be referred to as a second fine-tuning stage or an instruction learning stage, etc. Stage 350 may also be referred to as a third fine-tuning stage or a preference learning stage, etc. In stages 340 and 350, the model fine-tuning system 220 may update the parameters of the corresponding components in the machine learning model, for example, the parameters of the visual encoder model 122, the adapter model 124, and the language model 126 may be updated. That is, in these fine-tuning stages, the parameters of each component of the machine learning model 120 may be updated.

[0069] At stage 340, the model fine-tuning system 220 may use the second training data set to fine-tune the machine learning model 120 obtained after the first fine-tuning stage. The training samples in the second training data set include sample images, sample texts, and corresponding response labels. At stage 350, the model fine-tuning system 220 may use the third training data set to fine-tune the machine learning model 120 obtained after the second fine-tuning stage. The training samples in the third training data set include sample images, sample texts, corresponding response labels, and feedback information for the training samples.

[0070] The model fine-tuning system 220 may perform preference learning at stage 350, which refers to building a mathematical model of human intent by analyzing user behavior data or directly collecting preference feedback. Thus, by introducing the preference learning technology of "reinforcement learning with human feedback" (RLHF) at stage 350, the machine learning model 120 can continuously optimize its processing process through human feedback information.

[0071] In some embodiments, the data quality of the second training data set used in stage 340 is higher than that of the first training data set used in stage 330, and the data quality of the third training data set used in stage 350 is higher than that of the second training data set used in stage 340. In some embodiments, the amount of data used for model training in stage 330 and the amount of data used for model training in stage 340 are both relatively large, for example, tens of millions of data can be used to fine-tune the machine learning model 120 in stage 330 / stage 340. The amount of data used for model training in stage 350 can be less than the amount of data used for model training in stage 330 / stage 340, for example, millions of data can be used to fine-tune the machine learning model 120 in stage 350.

[0072] Therefore, in the multiple fine-tuning stages from stage 330 to stage 350, the data quality of the training data used can be improved successively, allowing the machine learning model 120 to gradually master the model inputs in rich forms and gradually improve the quality of the machine learning model 120. The fine-tuned machine learning model 120 can output complex and logical responses, which helps to improve the accuracy of the responses output by the machine learning model 120.

[0073] In some embodiments, the electronic device 110 may obtain the machine learning model 120 after the third fine-tuning stage for model reasoning. In the model reasoning stage, the electronic device 110 (specifically, for example, the model application system 230) may receive a query request (for example, a query request 104) from the user 102, and the query request 104 may include at least one modal content (for example, an image and a query text for the image). The electronic device 110 may use the trained machine learning model 120 to generate a response 130 to the query request 104.

[0074] Specifically, in response to receiving a query request, the visual encoder model in the machine learning model 120 can extract visual features of the image data, and the text encoder model / language model in the machine learning model 120 can extract text features of the query text. The adapter model in the machine learning model 120 can map the visual features extracted by the visual encoder into a part of the input features of the language model. The model input of the language model includes the visual features converted by the adapter model and the text features corresponding to the query text, and the language model can generate a response to the query text based on the received model input.

[0075] In summary, according to the embodiments of the present disclosure, the machine learning model can be pre-trained and fine-tuned, and in the fine-tuning stage, the image can be divided into different granularities to extract image features of different granularities for model fine-tuning, which can help the machine learning model better understand and process image data, and help improve the model quality of the machine learning model and the accuracy of the responses output by the machine learning model.

[0076] Figure 5 1 shows a flow chart of a method 500 for model training according to some embodiments of the present disclosure. The method 500 may be implemented in Figure 1 The electronic device 110 is referred to as Figure 1 The method 500 is described with reference to the environment 100 of FIG.

[0077] In box 510, the electronic device 110 performs a pretraining process on the machine learning model using a pretraining data set to obtain a pretrained machine learning model, where the machine learning model is configured to process a multimodal query request and generate a response to the multimodal query request, where the multimodal query request includes at least image data and text data.

[0078] In box 520, the electronic device 110 uses the first training data set to perform a first fine-tuning phase on the pre-trained machine learning model to obtain a fine-tuned machine learning model, wherein the first fine-tuning phase includes: dividing the first training data set into multiple first training data subsets, the training samples in each first training data subset include a sample image, a sample text, and a corresponding answer label, and each sample image in the multiple first training data subsets is divided into multiple image blocks of different sizes; using the multiple first training data subsets to fine-tune the pre-trained machine learning model to obtain an updated machine learning model.

[0079] In some embodiments, a plurality of first training data subsets are used to fine-tune a pretrained machine learning model to obtain an updated machine learning model, including: for each training sample in the first training data subset, using the pretrained machine learning model, extracting visual features from a plurality of image blocks divided from a sample image in the training sample and extracting text features from a sample text in the training sample; using the pretrained machine learning model, generating a predicted response based on the visual features and the text features; and fine-tuning the pretrained machine learning model based on the difference between the predicted response and the response label in the training sample.

[0080] In some embodiments, the first training data set includes training samples related to multiple task types.

[0081] In some embodiments, the machine learning model includes at least: a visual encoder model configured to extract visual features of image data; an adapter model configured to map the visual features extracted by the visual encoder model into a portion of input features of a language model; and a language model configured to process the input features to generate a response.

[0082] In some embodiments, the pre-training process includes a first pre-training stage and a second pre-training stage after the first pre-training stage, and wherein in the first pre-training stage the parameters of the adapter model are updated while the parameters of the visual encoder model and the language model remain unchanged, and in the second pre-training stage the parameters of the visual encoder model and the adapter model are further updated while the parameters of the language model remain unchanged.

[0083] In some embodiments, in a first fine-tuning phase, parameters of the visual encoder model, adapter model, and language model are updated.

[0084] In some embodiments, method 500 also includes: using a second training data set to perform a second fine-tuning stage on the machine learning model obtained after the first fine-tuning stage, wherein the training samples in the second training data set include sample images, sample texts, and corresponding answer labels; using a third training data set to perform a third fine-tuning stage on the machine learning model obtained after the second fine-tuning stage, wherein the training samples in the third training data set include sample images, sample texts, corresponding answer labels, and feedback information for the training samples; and obtaining the machine learning model after the third fine-tuning stage for model inference.

[0085] In some embodiments, during the second fine-tuning stage and the third fine-tuning stage, parameters of corresponding components in the machine learning model are updated.

[0086] In some embodiments, the data quality of the second training dataset used in the second fine-tuning stage is higher than the data quality of the first training dataset used in the first fine-tuning stage, and the data quality of the third training dataset used in the third fine-tuning stage is higher than the data quality of the second training dataset used in the second fine-tuning stage.

[0087] The embodiments of the present disclosure also provide corresponding devices for implementing the above methods or processes. Figure 6 An exemplary structural block diagram of an apparatus 600 for model training according to some embodiments of the present disclosure is shown. The apparatus 600 may be implemented as or included in the electronic device 110. Each module / component in the apparatus 600 may be implemented by hardware, software, firmware, or any combination thereof.

[0088] like Figure 6 As shown, the device 600 includes a pre-training execution module 610, which is configured to perform a pre-training process on the machine learning model using a pre-training data set to obtain a pre-trained machine learning model, and the machine learning model is configured to process a multimodal query request and generate a response to the multimodal query request, and the multimodal query request at least includes image data and text data. The device 600 also includes a fine-tuning execution module 620, which is configured to perform a first fine-tuning phase on the pre-trained machine learning model using a first training data set to obtain a fine-tuned machine learning model, wherein the first fine-tuning phase includes: dividing the first training data set into multiple first training data subsets, the training samples in each first training data subset include a sample image, a sample text and a corresponding response label, and each sample image in the multiple first training data subsets is divided into multiple image blocks of different sizes; using multiple first training data subsets to fine-tune the pre-trained machine learning model to obtain an updated machine learning model.

[0089] In some embodiments, the fine-tuning execution module 620 is further configured to: for each training sample in the first training data subset, utilize a pretrained machine learning model to extract visual features from a plurality of image blocks divided from a sample image in the training sample and extract text features from a sample text in the training sample; utilize the pretrained machine learning model to generate a predicted response based on the visual features and the text features; and fine-tune the pretrained machine learning model based on the difference between the predicted response and the response label in the training sample.

[0090] In some embodiments, the first training data set includes training samples related to multiple task types.

[0091] In some embodiments, the machine learning model includes at least: a visual encoder model configured to extract visual features of image data; an adapter model configured to map the visual features extracted by the visual encoder model into a portion of input features of a language model; and a language model configured to process the input features to generate a response.

[0092] In some embodiments, the pre-training process includes a first pre-training stage and a second pre-training stage after the first pre-training stage, and wherein in the first pre-training stage the parameters of the adapter model are updated while the parameters of the visual encoder model and the language model remain unchanged, and in the second pre-training stage the parameters of the visual encoder model and the adapter model are further updated while the parameters of the language model remain unchanged.

[0093] In some embodiments, in a first fine-tuning phase, parameters of the visual encoder model, adapter model, and language model are updated.

[0094] In some embodiments, the device 600 also includes: a second fine-tuning execution module, configured to use a second training data set to perform a second fine-tuning stage on the machine learning model obtained after the first fine-tuning stage, wherein the training samples in the second training data set include sample images, sample texts, and corresponding answer labels; a third fine-tuning execution module, configured to use a third training data set to perform a third fine-tuning stage on the machine learning model obtained after the second fine-tuning stage, wherein the training samples in the third training data set include sample images, sample texts, corresponding answer labels, and feedback information for the training samples; and an inference module, configured to obtain the machine learning model after the third fine-tuning stage for model inference.

[0095] In some embodiments, during the second fine-tuning stage and the third fine-tuning stage, parameters of corresponding components in the machine learning model are updated.

[0096] In some embodiments, the data quality of the second training dataset used in the second fine-tuning stage is higher than the data quality of the first training dataset used in the first fine-tuning stage, and the data quality of the third training dataset used in the third fine-tuning stage is higher than the data quality of the second training dataset used in the second fine-tuning stage.

[0097] The units and / or modules included in the device 600 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units and / or modules can be implemented using software and / or firmware, such as machine executable instructions stored on a storage medium. In addition to or as an alternative to machine executable instructions, some or all of the units and / or modules in the device 600 can be implemented at least in part by one or more hardware logic components. As an example and not limitation, exemplary types of hardware logic components that can be used include field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0098] It should be understood that one or more steps in the above method can be performed by a suitable electronic device or a combination of electronic devices. Such an electronic device or a combination of electronic devices may include, for example, Figure 1 The electronic device 110 in.

[0099] Figure 7 1 shows a block diagram of an electronic device 700 in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 7 The electronic device 700 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 7 The electronic device 700 shown can be used to implement Figure 1 Electronic device 110 or Figure 6 Device 600.

[0100] like Figure 7 As shown, the electronic device 700 is in the form of a general electronic device. The components of the electronic device 700 may include, but are not limited to, one or more processors or processing units 710, a memory 720, a storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760. The processing unit 710 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 720. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 700.

[0101] The electronic device 700 typically includes a plurality of computer storage media. Such media may be any accessible media that is accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may be a volatile memory (e.g., registers, caches, random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 730 may be a removable or non-removable medium, and may include a machine-readable medium, such as a flash drive, a disk, or any other medium, which may be capable of being used to store information and / or data and may be accessed within the electronic device 700.

[0102] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 720 may include a computer program product 725 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0103] The communication unit 740 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 700 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0104] The input device 750 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 760 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 700, or communicate with any device that allows the electronic device 700 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0105] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0106] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0108] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0109] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as an update, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the function involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0110] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.< / fine> < / am> < / user>

Claims

1. A method for model training, comprising: Performing a pre-training process on the machine learning model using the pre-training data set to obtain a pre-trained machine learning model, wherein the machine learning model is configured to process a multimodal query request and generate a response to the multimodal query request, wherein the multimodal query request includes at least image data and text data; as well as Performing a first fine-tuning phase on the pre-trained machine learning model using a first training data set to obtain a fine-tuned machine learning model, wherein the first fine-tuning phase comprises: Dividing the first training data set into a plurality of first training data subsets, wherein the training samples in each first training data subset include a sample image, a sample text, and a corresponding answer label, and each sample image in the plurality of first training data subsets is divided into a plurality of image blocks according to different sizes; The pre-trained machine learning model is fine-tuned using the multiple first training data subsets to obtain an updated machine learning model.

2. The method according to claim 1, wherein fine-tuning the pre-trained machine learning model using the plurality of first training data subsets to obtain an updated machine learning model comprises: For each training sample in the first training data subset, Using the pre-trained machine learning model, extracting visual features from a plurality of image blocks divided from a sample image in the training sample and extracting text features from a sample text in the training sample; generating a predicted response based on the visual features and the textual features using the pre-trained machine learning model; The pre-trained machine learning model is fine-tuned based on the difference between the predicted response and the response labels in the training samples. 3 . The method according to claim 2 , wherein the first training data set comprises training samples related to multiple task types.

4. The method according to claim 1, wherein the machine learning model comprises at least: a visual encoder model configured to extract visual features of image data; an adapter model configured to map the visual features extracted by the visual encoder model into a portion of input features of a language model; as well as The language model is configured to process input features to generate the response.

5. The method according to claim 4, wherein the pre-training process comprises a first pre-training stage and a second pre-training stage after the first pre-training stage, and In the first pre-training stage, the parameters of the adapter model are updated, while the parameters of the visual encoder model and the language model remain unchanged, and in the second pre-training stage, the parameters of the visual encoder model and the adapter model are further updated, while the parameters of the language model remain unchanged.

6. The method of claim 4, wherein in the first fine-tuning phase, parameters of the visual encoder model, the adapter model, and the language model are updated.

7. The method according to claim 1, further comprising: Performing a second fine-tuning phase on the machine learning model obtained after the first fine-tuning phase using a second training data set, wherein the training samples in the second training data set include sample images, sample texts, and corresponding answer labels; Performing a third fine-tuning phase on the machine learning model obtained after the second fine-tuning phase using a third training data set, wherein the training samples in the third training data set include sample images, sample texts, corresponding answer labels, and feedback information for the training samples; as well as The machine learning model after the third fine-tuning stage is obtained for model reasoning.

8. The method according to claim 7, wherein in the second fine-tuning stage and the third fine-tuning stage, the parameters of the corresponding components in the machine learning model are updated.

9. The method according to claim 7, wherein the data quality of the second training dataset used in the second fine-tuning stage is higher than that of the first training dataset used in the first fine-tuning stage, and the data quality of the third training dataset used in the third fine-tuning stage is higher than that of the second training dataset used in the second fine-tuning stage.

10. A device for model training, comprising: a pre-training execution module configured to perform a pre-training process on the machine learning model using the pre-training data set to obtain a pre-trained machine learning model, wherein the machine learning model is configured to process a multi-modal query request and generate a response to the multi-modal query request, wherein the multi-modal query request includes at least image data and text data; as well as A fine-tuning execution module is configured to perform a first fine-tuning phase on the pre-trained machine learning model using a first training data set to obtain a fine-tuned machine learning model, wherein the first fine-tuning phase includes: dividing the first training data set into multiple first training data subsets, the training samples in each first training data subset include a sample image, a sample text, and a corresponding answer label, and each sample image in the multiple first training data subsets is divided into multiple image blocks of different sizes; using the multiple first training data subsets to fine-tune the pre-trained machine learning model to obtain an updated machine learning model.

11. An electronic device, comprising: at least one processor; as well as At least one memory, the at least one memory is coupled to the at least one processor and stores instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 9 when executed by the at least one processor.

12. A computer-readable storage medium having computer-executable instructions stored thereon, wherein the computer-executable instructions can be executed by a processor to implement the method according to any one of claims 1 to 9.

13. A computer program product comprising computer executable instructions, wherein the computer executable instructions, when executed by a processor, implement the method according to any one of claims 1 to 9.

Images