A multimodal model training method and device

Abstract

The present disclosure provides a multimodal model training method and device, wherein the method includes obtaining a training sample set of a specified field, the training sample set includes first format data, the first format data includes at least picture information and corresponding text information, the training sample set includes second format data, the second format data includes at least instruction information and corresponding reply information, the first format data and the second format data corresponding to the first format data are used to train the multimodal model, and the parameters of the multimodal model are adjusted until the specified first format data is input to the multimodal model, and the multimodal model outputs the specified second format data. The present disclosure solves the problem that the general multimodal intelligent agent in the prior art has certain limitations when processing tasks in specific fields, improves task diversity, reduces costs, and enhances the generalization ability of the model.

Description

Technical Field

[0001] The present invention relates to the field of computer application processing technology, and in particular to a multimodal model training method, device, electronic device, non-transient computer-readable storage medium and computer program product. Background Art

[0002] With the rapid development of artificial intelligence technology, the rapid development of large language models has laid the foundation for multimodal models. By aligning the visual encoder with the large language model, the general agent based on the multimodal large language model can perform various common tasks based on pictures, which improves the application scope and practicality of the agent.

[0003] However, general multimodal agents have the following problems when dealing with specific domains, such as scientific fields and industrial applications:

[0004] First, the diversity of synthetic tasks is insufficient. Currently, the method of synthetic tasks mainly transforms existing data sets through templates written by humans or models. Although this method is cost-effective, the diversity of synthetic tasks is obviously insufficient and it is difficult to meet the needs of tasks in a wide range of fields.

[0005] Secondly, the cost of synthesizing tasks is high. Currently, although the method of task synthesis that relies on human experts or closed-source large language models can produce high-quality tasks, the cost is extremely high. Especially when dealing with large-scale domain data, this method requires a lot of resources and the participation of domain experts, which further increases the cost.

[0006] Finally, multi-stage training may harm the generalization ability of the model. In domain adaptation training, the dataset used is small in size and the task diversity is low. Existing multi-stage training methods divide the training data into two stages for separate training, which further reduces the diversity of training tasks in each stage, which may have a negative impact on the generalization ability of the model.

[0007] In summary, general multimodal agents have certain limitations when dealing with tasks in specific fields, and it is urgent to research and develop new training methods and systems to increase task diversity, reduce costs, and enhance the generalization ability of the model. Summary of the invention

[0008] In view of this, the technical problem to be solved by the present disclosure is to provide a multimodal model training method, device, electronic device, non-transitory computer-readable storage medium and computer program product to solve the problem that general multimodal agents in the prior art have certain limitations when processing tasks in specific fields.

[0009] According to one aspect of the present disclosure, a multimodal model training method is provided, the method comprising:

[0010] Acquire a training sample set of a specified field; wherein the training sample set includes data in a first format, the first format data at least includes picture information and corresponding text information; the training sample set includes data in a second format, the second format data at least includes instruction information and corresponding reply information;

[0011] Training a multimodal model using the first format data and second format data corresponding to the first format data;

[0012] The parameters of the multimodal model are adjusted until the multimodal model is input with specified data in the first format and the multimodal model outputs data in the specified second format.

[0013] Optionally, the first format data includes a “picture-description pair”, and the second format data includes a “command-long reply-short reply triplet”, wherein the long reply indicates that the amount of information is greater than a first predetermined threshold, and the short reply indicates that the amount of information is less than a second predetermined threshold;

[0014] Training the multimodal model using the first format data and the second format data corresponding to the first format data includes:

[0015] Determine whether the long reply and the short reply in the second format data are consistent to obtain a first judgment result; when the first judgment result indicates that the long reply and the short reply of the second format data are consistent, use the second format data as data for training the multimodal model, otherwise discard the second format data.

[0016] Optionally, the first format data includes a “picture-description pair”, and the second format data includes a “command-long reply-short reply triplet”, wherein the long reply indicates that the amount of information is greater than a first predetermined threshold, and the short reply indicates that the amount of information is less than a second predetermined threshold;

[0017] Training the multimodal model using the first format data and the second format data corresponding to the first format data includes:

[0018] Determine whether the long reply and / or the short reply in the second format data is an open reply, and obtain a second determination result; wherein the open reply includes an open reply without a fixed answer;

[0019] When the second judgment result is no, the second format data is used as data for training the multimodal model; otherwise, the second format data is discarded.

[0020] Optionally, the training sample set further includes data in a third format, and the data in the third format includes text information; a ratio of the amount of the data in the third format to the amount of the data in the first format is less than a third preset threshold;

[0021] The multimodal model is trained using the first format data, the third format data, and the second format data.

[0022] Optionally, adjusting the parameters of the multimodal model includes:

[0023] Calculating a prediction next-label loss of the multimodal model on a portion of the second format data;

[0024] The parameters of the multimodal model are updated using gradient backpropagation according to the predicted next tag loss; wherein the learning rate is 2e-5, the batch size is 128, the learning rate scheduler is a cosine scheduler, the maximum length of the model input is 6144 tags, the gradient decay value is 0, and the number of training rounds is 1 round.

[0025] According to another aspect of the present disclosure, a multimodal model training device is provided, the device comprising:

[0026] An acquisition module, used to acquire a training sample set of a specified field; wherein the training sample set includes data in a first format, the first format data at least includes picture information and corresponding text information; the training sample set includes data in a second format, the second format data at least includes instruction information and corresponding reply information;

[0027] A training module, configured to train a multimodal model using the first format data and second format data corresponding to the first format data;

[0028] An adjustment module is used to adjust the parameters of the multimodal model until the specified first format data is input to the multimodal model and the multimodal model outputs the specified second format data.

[0029] Optionally, the first format data includes a “picture-description pair”, and the second format data includes a “command-long reply-short reply triplet”, wherein the long reply indicates that the amount of information is greater than a first predetermined threshold, and the short reply indicates that the amount of information is less than a second predetermined threshold;

[0030] The training module includes:

[0031] A first judgment unit, configured to judge whether the long reply and the short reply in the second format data are consistent, and obtain a first judgment result;

[0032] The first processing unit is used to use the second format data as data for training the multimodal model when the first judgment result indicates that the long reply and the short reply of the second format data are consistent, and otherwise discard the second format data.

[0033] Optionally, the first format data includes a “picture-description pair”, and the second format data includes a “command-long reply-short reply triplet”, wherein the long reply indicates that the amount of information is greater than a first predetermined threshold, and the short reply indicates that the amount of information is less than a second predetermined threshold;

[0034] The training module includes:

[0035] A second judgment unit is used to judge whether the long reply and / or the short reply in the second format data is an open reply, and obtain a second judgment result; wherein the open reply includes an open reply without a fixed answer;

[0036] The second processing unit is used to use the second format data as data for training the multimodal model when the second judgment result is no, and otherwise discard the second format data.

[0037] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0038] Processor; and

[0039] Memory for storing programs,

[0040] Wherein, the program includes instructions, which, when executed by the processor, enable the processor to execute the multimodal model training method described in any one of the above embodiments.

[0041] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the multimodal model training method described in any one of the above embodiments.

[0042] According to another aspect of the present disclosure, a computer program product is provided, characterized in that the computer program product includes instructions, which, when executed, enable a computer to execute the multimodal model training method described in any one of the above embodiments.

[0043] The multimodal model training method in the disclosed embodiment obtains a training sample set of a specified field, the training sample set includes first format data, the first format data includes at least picture information and corresponding text information, the training sample set includes second format data, the second format data includes at least instruction information and corresponding reply information, the first format data and the second format data corresponding to the first format data are used to train the multimodal model, and the parameters of the multimodal model are adjusted until the specified first format data is input to the multimodal model, and the multimodal model outputs the specified second format data. The problem that the general multimodal agent in the prior art has certain limitations when processing tasks in specific fields is solved, the task diversity is improved, the cost is reduced, and the generalization ability of the model is enhanced.

[0044] It should be understood that the foregoing general description and the following detailed description are merely exemplary and illustrative and are not intended to limit the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings below are part of the specification of the present invention, and illustrate exemplary embodiments of the present invention. Together with the description of the specification, the accompanying drawings are used to explain the principles of the present invention.

[0046] Figure 1 A flowchart of a multimodal model training method provided for a specific embodiment of the present invention.

[0047] Figure 2 A schematic block diagram of a multimodal model training device is provided for a specific embodiment of the present invention.

[0048] Figure 3 A schematic block diagram of a training module provided for a specific embodiment of the present invention.

[0049] Figure 4 Another schematic block diagram of a training module provided for a specific embodiment of the present invention.

[0050] Figure 5 A structural block diagram of an exemplary electronic device that can be used to implement the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0051] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention more clearly understood, the spirit of the contents disclosed by the present invention will be clearly explained with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present invention, any technician in the relevant technical field can change and modify the techniques taught by the contents of the present invention without departing from the spirit and scope of the contents of the present invention.

[0052] The exemplary embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0053] The terms “first”, “second”, etc. used in this document do not particularly refer to an order or sequence, nor are they used to limit the present invention. They are only used to distinguish elements or operations described with the same technical terms.

[0054] The directional terms used in this document, such as up, down, left, right, front or back, etc., are only used to refer to the directions of the drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0055] The words “include,” “including,” “have,” “contain,” etc. used in this document are open-ended terms, meaning including but not limited to.

[0056] As used herein, "and / or" includes any or all combinations of the items described.

[0057] As used herein, “plurality” includes “two” and “more than two”; as used herein, “plurality of groups” includes “two groups” and “more than two groups”.

[0058] The terms "substantially" and "approximately" used herein are used to modify any quantity or error that may vary slightly, but these slight changes or errors do not change their essence. Generally speaking, the range of slight changes or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0059] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0060] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). Those skilled in the art should also understand that any transitional conjunctions and / or phrases that substantially represent two or more optional items, whether in the specification, claims, or drawings, should be understood to give the possibility of including one of these items, either of these items, or both of these items. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B", or "A and B".

[0061] At present, the adaptive methods of multimodal agents for specific fields usually follow the implementation methods of general multimodal agents: There are two common methods for synthesizing training task data: (1) Use templates written by humans / models to convert existing multimodal datasets into a format suitable for multimodal agent training. (2) Use closed-source large-scale models such as GPT-4 and GPT-4V to synthesize multimodal tasks based on pictures and / or descriptions. In terms of the training process, the two-stage training method of general multimodal agents is adopted: 1) Use "picture-description pair" data to train the agent to achieve modal alignment of visual language, and 2) Use multimodal instruction data to fine-tune the agent to improve its ability to follow human instructions to complete tasks.

[0062] However, the existing technologies have the following problems: 1) Insufficient diversity of synthetic tasks: Although the method of converting existing datasets using templates written by humans / models is cost-effective, the synthetic tasks lack diversity and are difficult to cover a wide range of domain task requirements; 2) The cost of synthetic tasks is high: Although the task synthesis methods that rely on human experts or closed-source large language models can produce high-quality tasks, the cost of such methods is quite high, especially when dealing with large-scale domain data, which requires a lot of resources and the participation of domain experts; 3) Multi-stage training may damage the generalization ability of the model: The dataset used for domain adaptive training is small in scale and has low task diversity. The existing multi-stage training method divides the training data into two categories for training in two stages, which will further reduce the diversity of training tasks in each stage, which may have a negative impact on the generalization ability of the model.

[0063] To this end, this optional embodiment provides a multimodal model training method. Figure 1 A flowchart of a multimodal model training method provided by a specific embodiment of the present invention is as follows: Figure 1 As shown, the method has the following steps:

[0064] S102: Obtain a training sample set for a specified domain. The specified domain here can be understood as a specific domain, as opposed to a general domain, where the general domain is a general domain. For example, GPT is a general agent, but if you want to create a dedicated agent in the medical field, then the medical field is the "specific field" or "specified field".

[0065] The training sample set may include first format data, which includes at least picture information and corresponding text information. In an optional embodiment, the first format data includes "picture-description pair", and the "picture-description pair" refers to a picture and a text description related to it. This description is usually used to help people understand the content of the picture, or to provide more contextual information for the picture, for example: 1) Natural scenery: Picture: a golden sand beach with a blue parasol and a deck chair on it; Description: a sunny summer beach, quiet and comfortable; 2) Animals: Picture: an orange cat curled up on the windowsill with its eyes closed; Description: a lazy house cat enjoying a leisurely afternoon in the sun; 3) Food: Picture: a plate of beautifully decorated sushi, colorful and neatly arranged; Description: freshly made Japanese sushi, with attractive and mouth-watering color combinations; 3) Technology products: Picture: a latest smartphone, with the screen lit up, showing 4) Historical buildings: Picture: An old castle with mottled stone walls surrounded by dense forests; Description: The remains of a medieval castle, witnessing the changes of history, are mysterious and solemn; 5) Artworks: Picture: An oil painting of a field of blooming tulips; Description: An Impressionist-style painting of a field of tulips with bright colors, vividly showing the vitality of spring; 6) Sports events: Picture: A football player running on the green field with the football at his feet; Description: In an exciting football match, the players go all out to show their passion for sports, etc.

[0066] For a special medical field, the first format data may be, for example: 1) Image: a human anatomy diagram showing the structure of the heart. Description: This image shows in detail the four chambers of the heart: the right atrium, right ventricle, left atrium and left ventricle, as well as the valves between them, such as the tricuspid valve and the mitral valve, and the beginning of the pulmonary artery and the aorta. 2) Image: a photo of bacteria under a microscope. Description: This image shows the morphology of streptococci under a microscope. They can be seen arranged in chains, which is an important feature in the diagnosis of streptococcal infection; 3) Image: an X-ray showing a fracture. Description: This X-ray shows a fracture of an arm. The fracture line can be clearly seen, and callus may be formed around it, which is a normal phenomenon in the healing process of a fracture; 4) Image: a photo of an operation process, showing a doctor performing an operation. Description: This image shows a surgeon performing abdominal surgery on a patient. Surgical instruments, sterile drapes and the doctor's sterile gloves can be seen, as well as the collaboration of the surgical team; 5) Image: a photo of a pathological section showing the morphology of cancer cells. Description: This picture shows a pathological section, showing the abnormal morphology of cancer cells, such as enlarged nuclei, increased chromatin, and irregular cell morphology, which are important bases for cancer diagnosis.

[0067] The training sample set may also include second format data, and the second format data includes at least instruction information and corresponding reply information. In an optional embodiment, the second format data includes "instruction-long reply-short reply triple", wherein the long reply indicates that the amount of information is greater than the first predetermined threshold, and the short reply indicates that the amount of information is less than the second predetermined threshold. Specifically, the second format data may be "instruction-long reply with higher amount of information-short reply with concise information triple", wherein the "short reply with concise information" is annotated by human experts, has less information but is more accurate, and the "long reply with higher amount of information" is generated by GPT-4V, and contains more information such as reasoning process and background knowledge.

[0068] The inventor team compiled a variety of existing multimodal data sets, including 11k data in total, covering 119 task categories. Then all the data were uniformly converted into the format required for training the "task synthesis model": the input of each data is a "picture-description pair", and the output is a "command-long reply with high information content-short reply triple".

[0069] S104: training a multimodal model using the first format data and the second format data corresponding to the first format data;

[0070] S106: Adjust the parameters of the multimodal model until the specified first format data is input to the multimodal model and the multimodal model outputs the specified second format data. After the training is completed, we input the "image-description pair" of a specific field (such as scientific field and industrial application) into the model, and extract the "command-long reply with high information content-short reply with concise information" triple from the output of the model.

[0071] Through the above steps, although "image-description pair" data (for example, pictures with text descriptions on the Internet) are relatively easy to obtain, multimodal "command-reply pair" data is relatively scarce. This optional embodiment develops a multimodal task generation model based on an open source multimodal language model, which converts the domain knowledge contained in the "image-description pair" data into "command-reply pair" data in the multimodal task. Unlike the traditional method that requires domain experts to generate domain-specific data from scratch, this optional embodiment mainly focuses on the extraction of tasks and the reconstruction of command data, which greatly reduces the need for domain experts such as human domain experts or closed source models. This optional embodiment provides a spontaneous task synthesis algorithm based on an open source multimodal language model for spontaneously synthesizing task data in a specific domain. The algorithm can effectively mine multimodal tasks from the "image-description pair" data in the target domain and convert it into "command-reply pair" data that can be used for multimodal agent training, greatly reducing the need for human experts or closed source large models, and ensuring the effectiveness and diversity of tasks. Based on the synthesized task dataset, an integrated multimodal agent domain adaptive training method is proposed, which can effectively inject target domain knowledge into the general multimodal agent and improve the agent's task performance in the target domain (such as scientific fields and industrial applications).

[0072] The above step 104 involves training the multimodal model using the first format data and the second format data corresponding to the first format data. In an optional embodiment, it is determined whether the long reply and the short reply in the second format data are consistent, and a first judgment result is obtained. If the first judgment result indicates that the long reply and the short reply of the second format data are consistent, the second format data is used as the data for training the multimodal model, otherwise the second format data is discarded. In another optional embodiment, it is determined whether the long reply and / or the short reply in the second format data are open replies, and a second judgment result is obtained, wherein the open reply includes open replies without fixed answers. If the second judgment result is no, the second format data is used as the data for training the multimodal model, otherwise the second format data is discarded. The above optional embodiment proposes a data filtering method based on reply consistency. In the absence of guidance from domain experts, it is a challenge to ensure the accuracy of the synthesized domain tasks. In order to solve this problem, we draw on the voting method in traditional machine learning and the self-consistency check in the chain-of-thought prompting, and propose a "data filtering method based on reply consistency". For each synthesized "task instruction-informative long reply-informative short reply" triple, we use an open source language model to classify the triples through a few-shot prompt, and divide them into three categories: "consistent reply", "inconsistent reply" and "open reply". Among them, "consistent reply" and "inconsistent reply" respectively represent the consistency or inconsistency between "informative long reply" and "informative short reply". "Open reply" refers to those replies that are relatively open and have no fixed answers, such as generated replies of picture descriptions or background knowledge. After classification, the triplets classified as "inconsistent reply" were removed, because this indicates that the task synthesis model has low confidence in the generation of the reply and cannot guarantee the consistency of the two replies. In addition, the triplets classified as "open reply" were also removed because the content of these replies is relatively vague, it is difficult to judge their correctness, and they highly overlap with the text description of the picture. For the retained "consistent reply" triples, we use the Chain-of-Thought format to integrate the "informative long reply" and the "informative short reply" into a complete reply: the "informative long reply" is used as the reasoning process, and the "informative short reply" is used as the final conclusion. This ensures that the generated instructions are both informative and accurate. Finally, we combine the integrated reply with the corresponding instruction to form a complete "instruction-reply" pair.

[0073] The training of general agents based on multimodal language models usually includes two stages: 1) training through picture-description data to achieve alignment between vision and language; 2) fine-tuning the model through multimodal instruction data to improve the model's ability to follow human instructions when completing tasks. This two-stage approach is also widely used in domain adaptation training of agents. However, compared with general agent training, the training data set used for domain adaptation is small in scale and limited in diversity. If the training data is divided into two independent stages for training, the diversity of tasks in each stage may be further reduced, which in turn has a negative impact on the generalization ability of the model. In order to avoid this problem, the above embodiment proposes an integrated domain adaptation training method, which integrates "picture-description pairs" and synthesized "instruction-response pairs" into one training stage to increase the diversity of training tasks. During the training process, each picture can correspond to multiple related tasks: 1) Picture description task: This task converts the original "picture-description pair" in each data into a picture description task. The task instruction is usually a question like "What's in the image?", requiring the model to describe the content of the picture, and the standard response is the text description corresponding to the picture. 2) Synthetic task: This task is generated by the task synthesizer in this optional embodiment. After passing the "data filtering method based on response consistency", the synthesized "instruction-response pair" marked as "response consistency" is retained as another training task for the image. For each piece of training data, the order of the synthetic task and the image description task is randomly disrupted to increase the diversity of the tasks. For training data without synthetic tasks, only the original "image description task" is retained.

[0074] In order to improve the robustness of the model, in an optional embodiment, the training sample set also includes data in a third format, the data in the third format includes text information but does not include image information, the ratio of the number of data in the third format to the number of data in the first format is less than a third preset threshold, and the multimodal model is trained using the first format data, the third format data, and the second format data. Specifically, 10% of the images in the training data set are randomly replaced with blank images to simulate the situation in which the task synthesis model cannot understand the images during the reasoning process, thereby training the task synthesis model to perform task synthesis using the input text description when the images cannot be understood. By introducing blank images in the training process and simulating the situation in which the task synthesis model cannot understand the images during the reasoning process, the model can be trained to perform task synthesis using the input text description when the images cannot be understood, which helps to improve the robustness, generalization ability, and multimodal fusion ability of the model, reduce the model's dependence on image quality, and thus improve the accuracy and stability of task synthesis in practical applications.

[0075] The above step S106 involves adjusting the parameters of the multimodal model. In an optional embodiment, the predicted next tag loss of the multimodal model in the second format data is calculated, and the parameters of the multimodal model are updated using gradient backpropagation according to the predicted next tag loss; wherein the learning rate is 2e-5, the batch size is 128, the learning rate scheduler is a cosine scheduler, the maximum length of the model input is 6144 tags, the gradient decay value is 0, and the number of training rounds is 1 round. Specifically, the model used is a multimodal language model (LLaVA-v1.6-Llama3-8B) with 8 billion parameters, which is a relatively large model. Generally, a larger model can capture more complex features and patterns, thereby improving the performance of the model. For each data, the "image-description pair" and "command-long response with high information content-short response with concise information" are input into the multimodal model in the above format, and then the "next-token-prediction loss" of the model in the "command-long response with high information content-short response with concise information" part is calculated, and then all parameters of the model are updated using gradient backpropagation. The "next-token-prediction loss" is used as the loss function, which is the loss function in the language model and helps the model learn long-distance dependencies in sequence data. The gradient backpropagation is used to update all parameters of the model, which is a standard deep learning training method and helps optimize model parameters. The learning rate is 2e-5, which is a relatively small learning rate and helps the model converge stably during training. The learning rate is adjusted using the cosine scheduler, which can dynamically adjust the learning rate at different stages of training, which helps the model converge quickly in the early stage and perform detailed fine-tuning in the later stage. The batch size is 128, which can strike a balance between memory consumption and computational efficiency. The maximum length of the model input is 6144 tokens, which allows the model to process longer input sequences, which is necessary for some tasks that require processing long texts. The gradient decay (weightdecay) value is 0, which means that the model parameters are not regularized during training, allowing the model to learn more complex patterns. The number of training rounds is 1, which means that the model training time is short, which is beneficial for rapid prototyping and testing. In this way, the open source multimodal language model can be fine-tuned using the training data. The training model can synthesize the "instruction-long reply with high information content-short reply with concise information" triplet based on the input "picture-description pair" data. During the training process, for each data, the fine-tuning loss is calculated only on the output part ("instruction-long reply with high information content-short reply with concise information" triplet).

[0076] This embodiment provides a multimodal model training device, such as Figure 2 As shown, the device comprises:

[0077] The acquisition module 22 is used to acquire a training sample set of a specified field; wherein the training sample set includes data in a first format, the first format data at least includes image information and corresponding text information; the training sample set includes data in a second format, the second format data at least includes instruction information and corresponding reply information;

[0078] A training module 24, configured to train a multimodal model using the first format data and second format data corresponding to the first format data;

[0079] The adjustment module 26 is used to adjust the parameters of the multimodal model until the specified first format data is input to the multimodal model and the multimodal model outputs the specified second format data.

[0080] Optionally, the first format data includes a “picture-description pair”, and the second format data includes a “command-long reply-short reply triple”, wherein the long reply indicates that the amount of information is greater than a first predetermined threshold, and the short reply indicates that the amount of information is less than a second predetermined threshold;

[0081] The training module 24 includes:

[0082] A first judgment unit 242 is used to judge whether the long reply and the short reply in the second format data are consistent, and obtain a first judgment result;

[0083] The first processing unit 244 is configured to use the second format data as data for training the multimodal model if the first judgment result indicates that the long reply and the short reply of the second format data are consistent, and otherwise discard the second format data.

[0084] Optionally, the first format data includes a “picture-description pair”, and the second format data includes a “command-long reply-short reply triple”, wherein the long reply indicates that the amount of information is greater than a first predetermined threshold, and the short reply indicates that the amount of information is less than a second predetermined threshold;

[0085] The training module 24 includes:

[0086] The second judgment unit 246 is used to judge whether the long reply and / or the short reply in the second format data is an open reply, and obtain a second judgment result; wherein the open reply includes an open reply without a fixed answer;

[0087] The second processing unit 248 is configured to use the second format data as data for training the multimodal model if the second judgment result is no, and otherwise discard the second format data.

[0088] The further functional description of each of the above modules is the same as that of the above corresponding embodiments and will not be repeated here.

[0089] The exemplary embodiment of the present invention further provides an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present invention when executed by the at least one processor.

[0090] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0091] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0092] refer to Figure 5 , a block diagram of an electronic device that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0093] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0094] A plurality of components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 may be any type of device capable of inputting information to the electronic device 500, and the input unit 506 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 507 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 may include, but is not limited to, a disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0095] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the music data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via the ROM 602 and / or the communication unit 509. In some embodiments, the computing unit 501 may be configured to perform a method according to an embodiment of the present invention in any other appropriate manner (e.g., by means of firmware).

[0096] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0097] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0098] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0099] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0100] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0101] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0102] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A multimodal model training method, characterized in that: The method comprises: Acquire a training sample set of a specified field; wherein the training sample set includes data in a first format, the first format data at least includes picture information and corresponding text information; the training sample set includes data in a second format, the second format data at least includes instruction information and corresponding reply information; Training a multimodal model using the first format data and second format data corresponding to the first format data; Adjusting parameters of the multimodal model until the multimodal model is input with specified first format data and the multimodal model outputs specified second format data; The first format data includes a "picture-description pair", and the second format data includes a "command-long reply-short reply triple", wherein the long reply indicates that the amount of information is greater than a first predetermined threshold, and the short reply indicates that the amount of information is less than a second predetermined threshold; Training the multimodal model using the first format data and the second format data corresponding to the first format data includes: Determine whether the long reply and the short reply in the second format data are consistent, and obtain a first determination result; if the first determination result indicates that the long reply and the short reply of the second format data are consistent, use the second format data as data for training the multimodal model, otherwise discard the second format data; or, The first format data includes a "picture-description pair", and the second format data includes a "command-long reply-short reply triple", wherein the long reply indicates that the amount of information is greater than a first predetermined threshold, and the short reply indicates that the amount of information is less than a second predetermined threshold; Training the multimodal model using the first format data and the second format data corresponding to the first format data includes: Determine whether the long reply and / or the short reply in the second format data is an open reply, and obtain a second determination result; wherein the open reply includes an open reply without a fixed answer; When the second judgment result is no, the second format data is used as data for training the multimodal model; otherwise, the second format data is discarded.

2. The multimodal model training method according to claim 1, characterized in that: The training sample set further includes data in a third format, wherein the data in the third format includes text information; a ratio of the amount of the data in the third format to the amount of the data in the first format is less than a third preset threshold; The multimodal model is trained using the first format data, the third format data, and the second format data.

3. The multimodal model training method according to claim 1, characterized in that: Adjusting the parameters of the multimodal model includes: Calculating a prediction next-label loss of the multimodal model on a portion of the second format data; The parameters of the multimodal model are updated using gradient backpropagation according to the predicted next tag loss; wherein the learning rate is 2e-5, the batch size is 128, the learning rate scheduler is a cosine scheduler, the maximum length of the model input is 6144 tags, the gradient decay value is 0, and the number of training rounds is 1 round.

4. A multimodal model training device, characterized in that: The device comprises: An acquisition module, used to acquire a training sample set of a specified field; wherein the training sample set includes data in a first format, the first format data at least includes picture information and corresponding text information; the training sample set includes data in a second format, the second format data at least includes instruction information and corresponding reply information; A training module, configured to train a multimodal model using the first format data and second format data corresponding to the first format data; An adjustment module, used for adjusting the parameters of the multimodal model until the multimodal model is input with specified first format data and the multimodal model outputs specified second format data; The first format data includes a "picture-description pair", and the second format data includes a "command-long reply-short reply triple", wherein the long reply indicates that the amount of information is greater than a first predetermined threshold, and the short reply indicates that the amount of information is less than a second predetermined threshold; The training module includes: A first judgment unit, configured to judge whether the long reply and the short reply in the second format data are consistent, and obtain a first judgment result; a first processing unit, configured to use the second format data as data for training the multimodal model if the first judgment result indicates that the long reply and the short reply of the second format data are consistent, and otherwise discard the second format data; or The first format data includes a "picture-description pair", and the second format data includes a "command-long reply-short reply triple", wherein the long reply indicates that the amount of information is greater than a first predetermined threshold, and the short reply indicates that the amount of information is less than a second predetermined threshold; The training module includes: A second judgment unit is used to judge whether the long reply and / or the short reply in the second format data is an open reply, and obtain a second judgment result; wherein the open reply includes an open reply without a fixed answer; The second processing unit is used to use the second format data as data for training the multimodal model when the second judgment result is no, and otherwise discard the second format data.

5. An electronic device, comprising: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-3.

7. A computer program product, characterized in that The computer program product comprises instructions which, when executed, cause a computer to perform the method according to any one of claims 1 to 3.

Images