Multi-modal large model training method, image-text generating method and device

Abstract

The invention provides a multi-modal large model training method and device and an image-text generation method and device, and relates to the technical field of artificial intelligence and large models. The model training method comprises the following steps: acquiring a first sample image and corresponding first annotation data, and acquiring a second sample image and corresponding second annotation data; obtaining a first visual feature representation of the first sample image; performing first-stage model training according to the first visual feature representation and the first annotation data, including training an initial target task model to obtain a target task model; obtaining a second visual feature representation of the second sample image; according to the second visual feature representation, the second annotation data, the query converter and the target task model, performing second-stage model training, including training the initial language model to obtain a language model; the multi-modal large model comprises a visual encoder, a target task model, a query converter and a language model. The method and the device are beneficial to improving the accuracy of the multi-modal large model output answers.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a multimodal large model training method, a graphic and text generation method, and a device. Background Art

[0002] The multimodal big model is a deep learning model that combines information from both image and text modalities. Its core concept is to fuse different media data, such as text and image data, and learn the connections between these modal data to achieve more intelligent information processing. Specifically, the multimodal big model is fed with images and text instructions, and the model can output responses to the text instructions based on the images.

[0003] In related technologies, a mainstream architecture for large multimodal models consists of three parts: a visual branch, a mapping branch, and a language branch. The visual branch uses pre-training of images and text to achieve the ability to perceive visual elements; the mapping branch maps visual features to the text domain, making it easier for the language model to understand and process them; and the language branch uses existing mature language models to understand the text mapped from the visual information and the input instructions, and outputs a response. However, the mapping branch may lose some information during the mapping process, resulting in the model outputting inaccurate responses. Summary of the Invention

[0004] The present application provides a multimodal large model training method, image and text generation method and device, which are conducive to improving the accuracy of the answers output by the multimodal large model.

[0005] In a first aspect, an embodiment of the present application provides a multimodal large model training method, comprising:

[0006] Acquire a first sample image and corresponding first annotation data in a target task scenario; and acquire a second sample image and corresponding second annotation data;

[0007] Inputting the first sample image into a visual encoder to obtain a first visual feature representation of the first sample image; the visual encoder is used to extract visual features of the input image;

[0008] Performing a first stage of model training based on the first visual feature representation and the first labeled data, wherein the first stage of model training includes training an initial target task model to obtain a target task model;

[0009] Inputting the second sample image into the visual encoder to obtain a second visual feature representation of the second sample image;

[0010] A second stage of model training is performed based on the second visual feature representation, the second annotated data, the query converter, and the target task model. The second stage of model training includes training the initial language model to obtain a language model. The multimodal large model includes the visual encoder, the target task model, the query converter, and the language model. The query converter is used to map visual features to a text domain, and the multimodal large model is used to generate corresponding answers based on the input image and text instructions.

[0011] In a second aspect, an embodiment of the present application provides a method for generating images and texts, which is applied to a large multimodal model. The large multimodal model includes a visual encoder, a query transformer, a target task model, and a language model. The method includes:

[0012] acquiring a first image;

[0013] Inputting the first image into the visual encoder to obtain a fourth visual feature representation of the first image;

[0014] inputting the fourth visual feature representation into the query transformer to obtain a third textual representation of the first image;

[0015] Inputting the fourth visual feature representation into the target task model to obtain a target prediction result of the first image;

[0016] Inputting the third text representation, the target prediction result, and the text instruction into the language model to obtain an answer to the text instruction;

[0017] Wherein, the multimodal large model is obtained according to the model training method described in the first aspect.

[0018] In a third aspect, a multimodal large model training device is provided, comprising:

[0019] An acquisition unit, configured to acquire a first sample image and corresponding first annotation data in a target task scenario; and acquire a second sample image and corresponding second annotation data;

[0020] a visual encoder configured to input the first sample image and obtain a first visual feature representation of the first sample image; the visual encoder is configured to extract visual features of the input image;

[0021] a training unit, configured to perform a first stage of model training based on the first visual feature representation and the first labeled data, wherein the first stage of model training includes performing model training on an initial target task model to obtain a target task model;

[0022] The visual encoder is further configured to input the second sample image and obtain a second visual feature representation of the second sample image;

[0023] The training unit is further used to perform a second stage of model training based on the second visual feature representation, the second annotated data, the query converter and the target task model. The second stage of model training includes training the initial language model to obtain a language model. The multimodal large model includes the visual encoder, the target task model, the query converter and the language model. The query converter is used to map visual features to a text domain, and the multimodal large model is used to generate corresponding answers based on the input image and text instructions.

[0024] In a fourth aspect, a device for generating images and text is provided, the device comprising a multimodal large model, the multimodal large model comprising a visual encoder, a query transformer, a target task model, and a language model, the device further comprising:

[0025] an acquisition unit, configured to acquire a first image;

[0026] The visual encoder is used to input the first image and obtain a fourth visual feature representation of the first image;

[0027] The query transformer is configured to input the fourth visual feature representation and obtain a third text representation of the first image;

[0028] The target task model is configured to input the fourth visual feature representation input and obtain a target prediction result for the first image;

[0029] The language model is configured to input the third text representation, the target prediction result, and the text instruction to obtain an answer to the text instruction;

[0030] Wherein, the multimodal large model is obtained according to the model training method as described in the first aspect.

[0031] In a fifth aspect, an embodiment of the present application provides an electronic device comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory, and executing the method in the first aspect, or the method in the second aspect.

[0032] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium comprising instructions, which, when executed on a computer, enables the computer to execute the method in the first aspect or the method in the second aspect.

[0033] In a seventh aspect, an embodiment of the present application provides a computer program product, comprising computer program instructions, which enable a computer to execute the method in the first aspect or the method in the second aspect.

[0034] In an eighth aspect, an embodiment of the present application provides a computer program, which enables a computer to execute the method in the first aspect or the method in the second aspect.

[0035] The above technical solution models and learns the specialized or professional capabilities that may be used in the target task scenario based on the first sample image and the corresponding first annotation data in the target task scenario, thereby obtaining a target task model, so that the target task model can perceive the visual features, and then learns the language model in the multimodal large model based on the second sample image and the corresponding second annotation data, as well as the target task model and the query converter, so that the language model can understand richer information such as the text information mapped by the visual features and the perception results obtained by task modeling, which is conducive to improving the accuracy of the answers output by the multimodal large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a system architecture of an embodiment of the present application;

[0037] Figure 2 A schematic diagram of another system architecture according to an embodiment of the present application;

[0038] Figure 3 A schematic diagram of another system architecture according to an embodiment of the present application;

[0039] Figure 4 4 is a schematic flowchart of a multimodal large model training method according to an embodiment of the present application;

[0040] Figure 5 4 is a schematic flowchart of another multimodal large model training method according to an embodiment of the present application;

[0041] Figure 6 Schematic diagram of a network architecture of a target task model according to an embodiment of the present application;

[0042] Figure 7 4 is a schematic flowchart of another multimodal large model training method according to an embodiment of the present application;

[0043] Figure 8 is an optional schematic diagram of processing visual features in a segmentation task according to an embodiment of the present application;

[0044] Figure 9 4 is a schematic flowchart of another multimodal large model training method according to an embodiment of the present application;

[0045] Figure 10 is a schematic flow chart of a method for generating images and texts according to an embodiment of the present application;

[0046] Figure 11 Schematic flow chart of a multimodal large model training device according to an embodiment of the present application;

[0047] Figure 12 is a schematic block diagram of a device for generating images and texts according to an embodiment of the present application;

[0048] Figure 13 is a schematic block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0049] The technical solutions in the embodiments of the present invention will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present invention.

[0050] It should be understood that in the embodiments of the present application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information.

[0051] In the description of this application, unless otherwise specified, "at least one" means one or more, and "plurality" means two or more than two. In addition, "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0052] It should also be understood that the first, second, etc. descriptions appearing in the embodiments of the present application are only for illustration and distinction of the description objects, and there is no order. They do not represent any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0053] It should also be understood that the specific features, structures, or characteristics associated with the embodiments in the specification are included in at least one embodiment of the present application. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0054] In addition, the terms "including" and "having" and any variations thereof are intended to cover a non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, product or device.

[0055] The embodiments of the present application are applied to the field of artificial intelligence technology.

[0056] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive field within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, pre-trained models, operating / interaction systems, and mechatronics. Pre-trained models, also known as large models or basic models, can be fine-tuned and widely applied to downstream tasks across various AI domains. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0057] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0058] The embodiments of the present application may relate to computer vision technology (CV) in artificial intelligence technology. Computer vision is a science that studies how to make machines "see". More specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further performing image processing so that the computer processing becomes an image more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to establish an artificial intelligence system that can obtain information from images or multidimensional data. Large model technology has brought important changes to the development of computer vision technology. Pre-trained models in the visual field such as swin-transformer, ViT, V-MOE, MAE, etc. can be quickly and widely applied to specific downstream tasks after fine tuning. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, etc., as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0059] Embodiments of the present application may relate to natural language processing (NLP) in artificial intelligence technology. NLP is an important field in the fields of computer science and artificial intelligence. It studies various theories and methods that can enable effective communication between humans and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology generally includes text processing, semantic understanding, machine translation, robot question answering, knowledge graphs and other technologies.

[0060] The embodiments of the present application may also relate to machine learning (ML) in artificial intelligence technology. ML is a multi-disciplinary subject that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning generally include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and formulaic learning.

[0061] The embodiments of the present application may also relate to a pre-training model (PTM) in artificial intelligence technology. PTM, also known as a cornerstone model or a large model, refers to a deep neural network (DNN) with a large number of parameters. It is trained on a large amount of unlabeled data, and the function approximation ability of the large-parameter DNN is used to enable the PTM to extract common features from the data. After fine-tuning, efficient parameter fine-tuning (PEFT), prompt-tuning and other technologies, it is suitable for downstream tasks. Therefore, the pre-training model can achieve ideal results in a few-shot or zero-shot scenario. PTM can be divided into language models (ELMO, BERT, GPT), visual models (swin-transformer, ViT, V-MOE), language models (VALL-E), multimodal models (ViBERT, CLIP, Flamingo, Gato), etc. according to the data modality processed, wherein the multimodal model refers to a model that establishes two or more data modality feature representations. Pre-trained models are important tools for outputting artificial intelligence generated content (AIGC) and can also serve as a universal interface for connecting multiple specific task models.

[0062] In related technologies, a mainstream architecture of a multimodal large model consists of three parts, including: a visual branch, a mapping branch, and a language branch. Among them, the visual branch realizes the perception of visual elements through pre-training of images and texts; the mapping branch maps visual features to the text domain to facilitate the understanding and processing of the language model; the language branch uses the existing mature language model to understand the text mapped from the visual information and the input instructions, and outputs the answer. However, the mapping branch may lose some information in the mapping process, resulting in the answer output by the model not necessarily being accurate. For example, for some more specialized or professional capabilities, such as OCR and medical image segmentation, there are still shortcomings, and it is difficult to reach the capability level of a dedicated model.

[0063] In view of this, the embodiments of the present application provide a multimodal large model training method, a graphic and text generation method and a device, which can help improve the accuracy of the answers output by the multimodal large model.

[0064] Specifically, a first sample image and corresponding first annotation data, as well as a second sample image and corresponding second annotation data in a target task scenario can be obtained; the first sample image is input into a visual encoder to obtain a first visual feature representation of the first sample image; based on the first visual feature representation and the first annotation data, a first stage of model training is performed, and the first stage of model training includes training an initial target task model to obtain a target task model; the second sample image is input into the visual encoder to obtain a second visual feature representation of the second sample image; based on the second visual feature representation, the second annotation data, the query converter and the target task model, a second stage of model training is performed, and the second stage of model training includes training an initial language model to obtain a language model; wherein the multimodal large model includes a visual encoder, a target task model, a query converter and a language model, the query converter is used to map visual features to a text domain, and the multimodal large model is used to generate corresponding answers based on the input image and text instructions.

[0065] Therefore, the embodiment of the present application models and learns the specialized or professional capabilities that may be used in the target task scenario based on the first sample image and the corresponding first annotation data in the target task scenario, thereby obtaining a target task model, so that the target task model can perceive the visual features, and then learn the language model in the multimodal large model based on the second sample image and the corresponding second annotation data, as well as the target task model and the query converter, so that the language model can understand richer information such as the text information mapped by the visual features and the perception results obtained by task modeling, which is conducive to improving the accuracy of the answers output by the multimodal large model.

[0066] In some embodiments, the target task scenario may include one task scenario, or multiple task scenarios, without limitation. In the target task scenario, for the specialized or professional capabilities that may be used in the actual application scenario, each task can be modeled and learned separately, so as to enhance the learning ability of specific tasks while retaining the general capabilities of the multimodal large model. Optionally, in a sophisticated multi-task scenario, the query converter is used to map the visual features to the text domain and the multi-task model perceives the visual features, which is then beneficial to the general capabilities of the language model, so as to achieve high-precision learning and understanding of specific task scenarios or key tasks and achieve better model expression capabilities.

[0067] For example, the target task may include various visual understanding tasks, including but not limited to image text recognition tasks, image segmentation tasks, etc. For example, the target task scenario may include image text recognition scenarios, image segmentation scenarios, etc., without limitation.

[0068] As an example, in an image text recognition scenario, the first annotation data may include text information contained in a first sample image. First, the first stage of model training is performed. Specifically, the first sample image may be input into a visual encoder to obtain a first visual feature representation of the first sample image, and then the initial image text recognition model (an example of an initial target task model) may be trained based on the first visual feature representation and the first annotation data to obtain an image text recognition model (an example of a target task model). Then, the second stage of model training is performed. Specifically, the second sample image may be input into a visual encoder to obtain a second visual feature representation of the second sample image, and then the initial language model may be trained based on the second visual feature representation and the second annotation data, as well as the query converter and the image text recognition model obtained by the first stage training to obtain a language model. After the first and second stage model training, a trained multimodal large model may be obtained, including a visual encoder, an image text recognition model, a query converter, and a language model. Since the query converter can map visual features to the text domain, the image text recognition model can perceive the text information in the image. Therefore, the embodiment of the present application can map richer visual features to the text domain, so that the language model can understand richer information, which is conducive to improving the accuracy of the answers output by the multimodal large model.

[0069] As another example, in an image segmentation scenario, the first annotation data may include the category of each pixel in the first sample image. First, the first stage of model training is performed. Specifically, the first sample image can be input into a visual encoder to obtain a first visual feature representation of the first sample image, and then the initial image segmentation model (an example of the initial target task model) can be trained based on the first visual feature representation and the first annotation data to obtain an image segmentation model (an example of the target task model). Then, the second stage of model training is performed. Specifically, the second sample image can be input into a visual encoder to obtain a second visual feature representation of the second sample image, and then the initial language model can be trained based on the second visual feature representation and the second annotation data, as well as the query converter and the image segmentation model obtained by the first stage training to obtain a language model. After the first and second stage model training, a trained multimodal large model can be obtained, including a visual encoder, an image segmentation model, a query converter and a language model. Since the query converter can map visual features to the text domain, the image segmentation model can perceive the category of each pixel in the image. Therefore, the language model in the embodiment of the present application can understand richer information such as the text information mapped by the visual features and the image segmentation results obtained by image segmentation, which is conducive to improving the accuracy of the answers output by the multimodal large model.

[0070] As another example, the target task scenario can be a multi-task scenario, in which the target task model can include both an image text recognition model and an image segmentation model. First, the first stage of model training is performed. Specifically, the initial image segmentation model and the initial image segmentation model can be trained separately to obtain an image text recognition model and an image segmentation model. Then, the second stage of model training is performed. Specifically, the second sample image can be input into the visual encoder to obtain a second visual feature representation of the second sample image. Then, based on the second visual feature representation and the second annotation data, as well as the query converter and the image segmentation model and the image segmentation model trained in the first stage, the initial language model can be trained to obtain a language model. After the first and second stages of model training, a trained multimodal large model can be obtained, including the visual encoder, the image text recognition model, the image segmentation model, the query converter, and the language model. Because the query converter can map visual features to the text domain, the image text recognition model can perceive the text information in the image, and the image segmentation model can perceive the category of each pixel in the image. Optionally, the perception results of the image segmentation model can be further mapped to the text domain. Therefore, in the embodiment of the present application, the language model can understand richer information such as text information mapped by visual features, text information in the image obtained by image text recognition, and image segmentation results obtained by image segmentation, which is conducive to improving the accuracy of the answers output by the multimodal large model.

[0071] Figure 1 This is a schematic diagram of a system architecture involved in an embodiment of the present application. Figure 1 As shown, the system architecture may include a user device 101 , a data collection device 102 , a training device 103 , an execution device 104 , a database 105 and a content library 106 .

[0072] The data acquisition device 102 is used to read training data from the content library 106 and store the read training data in the database 105. The training data involved in the embodiment of the present application includes at least a first sample image and corresponding first annotation data in the target task scenario, and a second sample image and corresponding second annotation data.

[0073] The training device 103 trains the machine learning model based on the training data maintained in the database 105. Optionally, the machine learning model may include a visual encoder, a language model, a query converter, a multi-task model, etc. In the embodiment of the present application, these machine learning models can be connected to form a multimodal large model. Optionally, the machine learning model can also be further connected to other downstream models. The model obtained by the training device 103 can be applied to different systems or devices.

[0074] In addition, reference Figure 1Execution device 104 is equipped with an I / O interface 107 for data exchange with external devices. For example, it receives images and text commands sent by user device 101 through the I / O interface. The computing module 109 in execution device 104 uses the trained multimodal large model to input the images and text commands, outputs corresponding responses, and sends the responses to user device 101 through the I / O interface.

[0075] Among them, the user device 101 may include a mobile phone, a computer, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft or other terminal devices, which is not limited in this embodiment of the present application.

[0076] The execution device 104 may be a server. For example, the server may be a computing device such as a rack server, a blade server, a tower server, or a cabinet server. The server may be a standalone server or a server cluster consisting of multiple servers, which is not limited in this embodiment of the present application.

[0077] In this embodiment, the execution device 104 is connected to the user device 101 via a network. The network can be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile Communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, Wi-Fi, or a call network.

[0078] It should be noted that Figure 1 This is merely a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationships between the devices, components, modules, etc. shown in the diagram do not constitute any limitation. In some embodiments, the data acquisition device 102, user device 101, training device 103, and execution device 104 can be the same device. The database 105 can be distributed on a single server or multiple servers, and the content library 106 can be distributed on a single server or multiple servers.

[0079] Figure 2 This is a schematic diagram of another system architecture involved in the embodiment of this application. Optionally, the system architecture can be configured in Figure 1 In the execution device 104.

[0080] like Figure 2As shown, the system architecture 200 includes a visual encoder 210, a query transformer (Q-Former) 220, a target task model 230, and a language model 240. The visual encoder 210 is used to extract visual features from an input image, the query transformer 220 is used to convert the input visual features into a text domain, the target task model 230 is used to model specific task requirements in a target task scenario to perceive the visual features and obtain a perception result, and the language model 240 is used to take the text information mapped to the visual features and the perception result obtained by task modeling as one of the inputs, and output a corresponding answer in combination with the input text instructions. Optionally, the perception result of the target task model 230 can be further mapped to a text domain.

[0081] Figure 3 This is a schematic diagram of another system architecture involved in the embodiment of this application. Optionally, the system architecture can be Figure 2 A concrete example of the system architecture in .

[0082] like Figure 3 As shown, the system architecture 300 includes a visual encoder 310, a query converter 320, a first target task model 330, a second target task model 340 and a language model 350. The first target task model 330 and the second target task model 340 are respectively Figure 2 In the specific example of the target task model 230, the visual encoder 310, the query converter 320 and the language model 350 are Figure 2 The corresponding modules in the query converter 320 are identical or similar, and can be referred to the description above. Optionally, the query converter 320, the first target task model 330, and the second target task model 340 can form a multi-task modeling module to jointly map visual features to the same text domain as one of the inputs of the language model 350.

[0083] The query converter 320 is used to map the visual features of the image to the text field to obtain text information. The first target task model 330 can be, for example, an image text recognition model, such as an OCR module, which can perform task modeling to obtain text information. <ocr>The second target task model 340 can be an image segmentation model, such as a segmentation module, which can perform task modeling and map segmentation information. <seg>In practical applications, the target task model can also be extended to other different visual understanding tasks, which is not limited in this application. The language module can map the text information of the visual features. and the perception results obtained by task modeling <ocr> 、 <seg>As one of the inputs, combine the user input question <question>, and output the appropriate answer.

[0084] The following describes the technical solutions of the embodiments of the present application in detail through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0085] Figure 4 This is a schematic flow chart of a multimodal large model training method 400 according to an embodiment of the present application. This method 400 can be executed by any electronic device with data processing capabilities. For example, the electronic device can be implemented as a server or a terminal device. For another example, the electronic device can be implemented as Figure 1 The training device 103 is not limited in this application. Figure 4 As shown, the method 400 for multimodal large model training includes steps 410 to 450.

[0086] 410 , obtaining a first sample image and corresponding first annotation data in a target task scenario; and obtaining a second sample image and corresponding second annotation data.

[0087] The first sample image and the corresponding first labeled data, namely the target task training data, are used to train a specific task, such as an image text recognition task or an image segmentation task, etc., without limitation.

[0088] In some embodiments, the target task scenario includes at least one of an image text recognition scenario and an image segmentation scenario.

[0089] When the target task scenario includes an image text recognition scenario, the first annotation data includes text information contained in the first sample image. Specifically, for image text recognition scenarios, conventional annotation data includes a detection box and corresponding text annotation results. In the embodiments of the present application, the annotation data can be simplified to include only the text annotations without the detection box.

[0090] When the target task scenario involves image segmentation, the first annotation data includes the category of each pixel in the first sample image. Specifically, for image segmentation, the object category of interest must first be determined, and then annotation is performed at the pixel level. Pixels that are not in the object category of interest can be labeled as background.

[0091] The second sample image and the corresponding second annotated data, i.e., language model training data, are used to train the language model. Optionally, they can also be used to train a related mapping layer. The second annotated data can include text from the second sample image that meets the requirements, such as a text instruction and a corresponding real answer.

[0092] Optionally, in some embodiments, a third sample image and corresponding third annotated data may be obtained. The third sample image and corresponding third annotated data may be image-text pair training data used to train the query converter Q-Former.

[0093] It should be noted that the amount of each training data can be collected according to the actual scenario, and the embodiment of the present application does not limit the amount of training data, for example, it can be more than one million, or about one hundred thousand, etc.

[0094] 420 , input the first sample image into a visual encoder to obtain a first visual feature representation of the first sample image.

[0095] Specifically, the visual encoder is used to extract visual features of the input image, that is, to extract visual features of the input image and obtain corresponding visual feature representations. Exemplarily, the visual encoder can be trained using contrastive language-image pre-training (CLIP) or two-stage optimization-boosted language-image pre-training (BLIP2). Optionally, the weights of the visual encoder model trained based on CLIP or BLIP2 can be used as the initial weights of the visual encoder.

[0096] 430. Perform a first stage of model training based on the first visual feature representation and the first labeled data. The first stage of model training includes training the initial target task model to obtain a target task model.

[0097] The target task model is used to perceive visual features based on the specific task requirements in the target task scenario and obtain perception results. Optionally, during the model training process at this stage, the weights of the visual encoder are fixed and do not participate in training.

[0098] In some embodiments, see Figure 5 When the target task scenario includes an image text recognition scenario, step 430 may further include the following steps 431 and 432.

[0099] 431, input the first visual feature representation and the first labeled data into a first initial target task model to determine a first loss function.

[0100] Specifically, when the first annotation data includes text information contained in the first sample image, thanks to the massive data and the capabilities of the multimodal large model, the text content in the first sample image can be predicted end-to-end, that is, the first visual feature representation extracted from the first sample image and the corresponding first annotation data are input into the first initial target task model, the first loss function is determined, and then the parameters of the first initial target task model are updated according to the first loss function to obtain the first target task model.

[0101] Optionally, the first initial target task model includes a first Transformer and a second Transformer. Specifically, the first visual feature representation can be input into the first Transformer, and the first labeled data can be input into the second Transformer, and the first image-text contrastive learning (ITC) loss, the first image-based text generation (ITG) loss, and the first image-text matching (ITM) loss are determined based on the output of the first Transformer and the output of the second Transformer. The first loss function includes the first ITC loss, the first ITC loss, and the first ITM loss.

[0102] The first Transformer includes a self-attention module and a cross-attention module, and the second Transformer includes a self-attention module; the first Transformer and the second Transformer share the self-attention module.

[0103] Specifically, the first Transformer is an image Transformer (ImageTransformer), which is used for visual feature extraction. The first Transformer inputs learnable queries and first models the dependencies between them through self-attention. Then, it uses cross-attention to model the dependencies between queries and visual features. Because the self-attention parameters of the first and second Transformers are shared, queries can also interact with text input. The second Transformer is a text Transformer (TextTransformer), which can serve as both a text encoder and a text decoder.

[0104] For example, see Figure 6 , the first initial target task model includes a first Transformer 601 and a second Transformer 602. The visual feature representation and the learnable query text (querying) are input into the first Transformer 601, and the input text (such as the first labeled data) is input into the second Transformer 602, and the ITC loss, ITG loss, and ITM loss are determined based on the output of the first Transformer 601 and the output of the second Transformer 602. Here, when the input visual feature representation is the first visual feature representation, the output ITC loss, ITG loss, and ITM loss correspond to the first ITC loss, the first ITG loss, and the first ITM loss, respectively.

[0105] For example, the purpose of ITC loss is to align visual features and text features to maximize their mutual information. ITC loss includes the alignment loss of visual features to text features (imagetotext, i2t) and the alignment loss of text features to visual features (texttoimage, t2i), which are respectively expressed as follows (1) and (2):

[0106]

[0107]

[0108] in, represents the loss of alignment of visual features to text features of the mth sample; represents the loss of alignment from text features to visual features of the mth sample; s() is the feature similarity function, which can generally be a matrix multiplication; I represents the visual features output by the first Transformer; T represents the text features output by the second Transformer; τ is the temperature parameter; M is the number of samples, m and M are positive integers and m is less than or equal to M.

[0109] The ITG loss is used to train the generated text for image descriptions. During the training process, queries are used to extract the information needed to generate text, which is then passed to the text tokens via the self-attention module. The first input token is changed from [cls] to [DEC] to mark the decoding task. During decoding, a mask is used to ensure that the text decoder sees all query tokens, while each text token only sees the previous text token.

[0110] ITM loss is used to align image and text representations in a more fine-grained manner, that is, to determine whether a given set of images and texts match. ITM outputs a normalized probability value and performs a cross-entropy loss calculation with the actual label.

[0111] 432. Update the parameters of the first initial target task model according to the first loss function to obtain a first target task model.

[0112] Specifically, the parameters of the first initial target task model may be updated according to the first ITM loss, the first ITG loss, and the first ITC loss to obtain the first target task model.

[0113] Therefore, the embodiment of the present application models and learns the first target task model for image and text recognition based on the first sample image and the corresponding first annotation data in the image and text recognition scenario, so that the first target task model can recognize the text information contained in the input image, thereby achieving more accurate learning and perception of the visual features of the input image.

[0114] In some embodiments, see Figure 7 When the target task scenario includes an image segmentation scenario, step 430 may further include the following steps 433 to 435.

[0115] 433. Input the first visual feature representation into the second initial target task model to determine the probability that each pixel in the first sample image belongs to each category.

[0116] Specifically, the visual feature representation output by the visual encoder can be further processed to perform segmentation tasks and obtain the probability that each pixel in the input image belongs to each category.

[0117] In some embodiments, the second initial target task model may include a transformation layer and at least one upsampling layer. In this case, step 433 may be implemented by following the steps 4331 to 4332:

[0118] 4331, the first visual feature representation is input into the transformation layer to obtain a two-dimensional feature map.

[0119] Specifically, in the visual encoder, the image is divided into several patches, such as 16×16, and features of all patches are obtained. For example, the dimension of the visual encoder output features (i.e., the first visual feature representation) can generally be HW / 256×1024, where H and W represent the height and width of the input image, respectively. After the first visual feature representation is input into the second initial target task model, it is first reshaped into a two-dimensional feature map through a transformation layer.

[0120] Figure 8 Figure 1 shows an optional schematic diagram of processing visual features in the segmentation task, which shows the output features of each layer from bottom to top. Figure 8 As shown in Figure 1, the one-dimensional visual feature of HW / 256×1024 can be reshaped into a two-dimensional feature map of H / 16×W / 16×1024 through the transformation layer.

[0121] 4332. Input the two-dimensional feature map into at least one upsampling layer to obtain the probability that each pixel in the first sample image belongs to each category; wherein the number of channels of the last upsampling layer of the at least one upsampling layer is determined according to the number of categories.

[0122] Specifically, after obtaining the two-dimensional feature map, it can be further upsampled through at least one upsampling layer to gradually increase the scale of the feature map, and predict the segmentation result, that is, the probability that each pixel in the first sample image belongs to each category. For example, the upsampling layer can be implemented by a convolutional neural network (CNN). Figure 8 For a two-dimensional feature map H / 16×W / 16×1024, four upsampling layers can be used to gradually transform the two-dimensional feature map to H / 8×W / 8×512, H / 4×W / 4×256, H / 2×W / 2×256, and H×W×(C+1). The third dimension is the number of channels of the convolutional network feature layer, which can be adjusted according to actual conditions. The number of channels in the last upsampling layer is determined by the number of categories C. Here, the number of channels in the last upsampling layer is set to (C+1), so that the probability of each pixel in the first sample image belonging to each category and the background can be obtained.

[0123] 434 , determine a second loss function based on the probability that each pixel in the first sample image belongs to each category and the first labeled data.

[0124] Specifically, the second loss function can be determined pixel by pixel using the cross entropy of the probability that each pixel belongs to each category and the true category label corresponding to the pixel. For example, the second loss function can be expressed as the following formula (3):

[0125]

[0126] Where i is the index of the pixel, j is the index of the category; N is the total number of pixels in the image; y ij is the true category j label of pixel i. If pixel i belongs to category j, then y ij =1, otherwise y ij =0;p ij is the probability predicted by the model that pixel i belongs to category j.

[0127] 435 , updating the parameters of the second initial target task model according to the second loss function to obtain a second target task model.

[0128] Therefore, the embodiment of the present application models and learns the first target task model for image segmentation based on the first sample image and the corresponding first annotation data in the image segmentation scenario, so that the first target task model can determine the probability of each pixel in the input image belonging to each category, thereby achieving more accurate learning and perception of the visual features of the input image.

[0129] 440 , input the second sample image into the visual encoder to obtain a second visual feature representation of the second sample image.

[0130] Specifically, step 440 may refer to the relevant description in step 420 .

[0131] 450. Perform a second stage of model training based on the second visual feature representation, the second annotated data, the query converter, and the target task model. The second stage of model training includes training the initial language model to obtain a language model. The multimodal large model includes a visual encoder, a target task model, a query converter, and a language model. The query converter is used to map visual features to a text domain. The multimodal large model is used to generate corresponding answers based on the input image and text instructions.

[0132] Specifically, the query converter can map the second visual feature representation to the text domain to obtain the corresponding text information. The target task model can perceive the second visual feature representation. The text information mapped to the second visual feature and the perception results obtained by the target task model can then be used as inputs to the initial language model. The initial language model is then trained in the second stage with the second annotated data to obtain a language model. The language model is used to generate corresponding answers based on the input text features.

[0133] Therefore, the embodiment of the present application models and learns the specialized or professional capabilities that may be used in the target task scenario based on the first sample image and the corresponding first annotation data in the target task scenario, thereby obtaining a target task model, so that the target task model can perceive the visual features, and then learn the language model in the multimodal large model based on the second sample image and the corresponding second annotation data, as well as the target task model and the query converter, so that the language model can understand richer information such as the text information mapped by the visual features and the perception results obtained by task modeling, which is conducive to improving the accuracy of the answers output by the multimodal large model.

[0134] For example, in a sophisticated multi-task scenario, visual features are mapped to the text domain through a query converter, and the multi-task model perceives the visual features, which then benefits the general capabilities of the language model, enabling high-precision learning and understanding of specific task scenarios or key tasks, and achieving better model expression capabilities.

[0135] In some embodiments, the query converter, target task model, and initial language model can be trained together based on the second visual feature representation and the second annotated data to obtain a trained large multimodal model. Therefore, embodiments of the present application can simultaneously optimize the weights of the query converter, target task model, and language model, which is conducive to improving the effectiveness of the large multimodal model.

[0136] Optionally, the weights of the visual encoder can be kept fixed during the second stage of model training.

[0137] Optionally, you can choose Lora to adjust the parameters of some layers in the language model, thereby reducing the number of adjustment parameters and shortening the model training time.

[0138] In some embodiments, see Figure 9 , the query converter, the target task model, and the initial language model can be jointly trained according to the following steps 451 to 454.

[0139] 451 , input the second visual feature representation into the query transformer to obtain a first text representation of the second sample image.

[0140] The query transformer is used to map the second visual feature to a text domain to obtain a first text representation of the second sample image. The first text representation output by the query transformer can be directly input into the language model.

[0141] At 452 , the second visual feature representation is input into the target task model to obtain a prediction result of the second sample image in the target task scenario.

[0142] The target task model is used to perceive the second visual feature according to the specific task requirements in the target task scenario to obtain a prediction result of the second sample image in the target task scenario.

[0143] In some embodiments, when the target task scenario is an image text recognition scenario, the prediction result output by the target task model is itself text content and can be directly input into the language model.

[0144] In some embodiments, the prediction result may also be input into a mapping layer to obtain a second text representation of the prediction result in the text feature space of the initial language model.

[0145] Specifically, when the target task scenario is image segmentation, that is, the target task model includes a second target task model for the segmentation scenario, since the prediction results themselves are the probabilities of pixels in the image belonging to each category, not the text content, it is necessary to map the prediction results to the text feature space of the language model. This allows the output of the target task model to be aligned with the text feature space.

[0146] 453 , input the first text representation, the prediction result, and the annotation instruction in the second annotation data into the initial language model to obtain a prediction answer corresponding to the annotation instruction.

[0147] Specifically, the language model takes the first text representation of the second visual feature map and the prediction result perceived by the target task model as one of the inputs, combines it with the annotation instructions in the second annotation data to understand it, and outputs the corresponding prediction answer.

[0148] Optionally, when the prediction result is mapped to the text feature space to obtain the second text representation, the first text representation, the second text representation and the annotation instruction can be input into the initial language model to obtain the prediction answer.

[0149] 454. According to the predicted answer and the true answer in the second labeled data, the parameters of the query transformer, the target task model and the initial language model are updated to obtain a trained multimodal large model.

[0150] Specifically, the loss function can be determined based on the true answers in the prediction expert and the second labeled data, and then the parameters of the query transformer, target task model and initial language model can be updated together according to the loss function to obtain a trained multimodal large model.

[0151] In some embodiments, the parameters of the query transformer, the second target task model, the mapping layer, and the initial language model may be updated based on the predicted answer and the true answer to obtain a trained multimodal large model.

[0152] In some embodiments, the query converter can directly use the weights of the open source model without retraining.

[0153] In some embodiments, a third stage of model training may be performed based on the third sample image and the corresponding third annotated data. The third stage of model training includes training the initial query converter to obtain a query converter. The third stage of model training precedes the second stage of model training.

[0154] In some embodiments, a third sample image and corresponding third annotation data can be obtained, where the third annotation data includes a text description of the third sample image; the third sample image is input into a visual encoder to obtain a third visual feature representation of the third sample image; and a third stage of model training is performed based on the third visual feature representation and the third annotation data, including model training of the initial query converter to obtain a query converter.

[0155] Optionally, a third stage of model training is performed based on the third visual feature representation and the third labeled data, which can be specifically implemented as follows:

[0156] The third visual feature representation and the third labeled data are input into the initial query transformer, a third loss function is determined; and the parameters of the initial query transformer are updated according to the third loss function to obtain a query transformer.

[0157] Optionally, the initial query transformer includes a third Transformer and a fourth Transformer; wherein the third Transformer and the fourth Transformer share a self-attention module; and the third Transformer also includes an interactive attention module. The third visual feature representation can be input into the third Transformer, and the second labeled data can be input into the fourth Transformer. A second ITC loss, a second ITG loss, and a second ITM loss are determined based on the outputs of the third and fourth Transformers. Here, the third loss function includes the second ITC loss, the second ITG loss, and the second ITM loss.

[0158] Specifically, the query converter's network architecture is identical to that of the first target task model. The difference is that the query converter predicts a textual description of the image, while the first target task model outputs the textual information contained in the image end-to-end. The query converter's training method is the same as that of the first target task model described above, and is not further detailed here.

[0159] Figure 10 This is a schematic flow chart of a method 1000 for generating text and images according to an embodiment of the present application. This method 1000 can be executed by any electronic device with data processing capabilities. For example, the electronic device can be implemented as a server or a terminal device. For another example, the electronic device can be implemented as Figure 1 The execution device 104 in the present application is not limited to this. Figure 10 As shown, the image-text generation method 1000 includes the following steps 1010 to 1050.

[0160] This method is applied to a large multimodal model, which includes a visual encoder, a query converter, a target task model, and a language model. Specifically, the large multimodal model is based on Figure 4 The multimodal large model training method shown is obtained.

[0161] 1010, acquire a first image.

[0162] 1020. Input the first image into a visual encoder to obtain a fourth visual feature representation of the first image.

[0163] 1030 : Input the fourth visual feature representation into the query transformer to obtain a third text representation of the first image.

[0164] The query transformer may align the fourth visual feature representation extracted in step 1020 to the text domain to obtain a third text representation of the first image.

[0165] 1040 , input the fourth visual feature representation into the target task model to obtain a target prediction result for the first image.

[0166] Among them, the target task model is obtained by training the initial target task model based on the sample images and corresponding annotation data in the target task scenario.

[0167] When the target task scenario is image text recognition, the target task model can complete the text recognition task in the first image based on the input fourth visual feature representation, and the obtained text information can be used as the target prediction result.

[0168] When the target scene is considered to be image segmentation, the target task model can complete the segmentation task of the first image based on the input fourth visual feature representation, and obtain the probability that each pixel in the first image belongs to each class, which can be used as the target prediction result. Optionally, the probability of each pixel in the first image belonging to each class can be further mapped to the text feature space.

[0169] 1050 , input the third text representation, the target prediction result, and the text instruction into a language model to obtain an answer to the text instruction, wherein the language model is obtained by training an initial language model based on the sample image and the corresponding annotation data.

[0170] Specifically, the third text representation obtained by mapping the first visual feature representation to the text domain in step 1030 and the target prediction result obtained by the target task model in step 1040 are both used as inputs of the language model. Combined with the text instruction (prompt) input by the user, the language model outputs a refined answer.

[0171] Therefore, the embodiment of the present application maps visual features into textual information through a query converter, and the target task model perceives the visual features, so that the language model can understand richer information such as the textual information mapped by the visual features and the perception results obtained by task modeling, which is conducive to inputting more refined answers, thereby improving the accuracy of the answers output by the multimodal large model and achieving better model expression capabilities.

[0172] The specific embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, a variety of simple modifications can be made to the technical solution of the present application, and these simple modifications all fall within the scope of protection of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner unless there is any contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, and as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0173] It should also be understood that in the various method embodiments of the present application, the order of the sequence numbers of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. It should be understood that these sequence numbers can be interchanged where appropriate, so that the embodiments of the present application described can be implemented in an order other than those shown or described.

[0174] The above describes the method embodiment of the present application in detail. Figures 11 to 13 , describe in detail the device embodiments of the present application.

[0175] Figure 11 Schematic block diagram of the multimodal large model training device 10 of the embodiment of the present application. Figure 11 As shown, the device 10 may include an acquisition unit 11, a visual encoder 12 and a training unit 13.

[0176] An acquisition unit 11 is configured to acquire a first sample image and corresponding first annotation data in a target task scenario; and acquire a second sample image and corresponding second annotation data;

[0177] A visual encoder 12 is configured to input the first sample image and obtain a first visual feature representation of the first sample image; the visual encoder is configured to extract visual features of the input image;

[0178] A training unit 13 is configured to perform a first stage of model training based on the first visual feature representation and the first labeled data, wherein the first stage of model training includes training an initial target task model to obtain a target task model;

[0179] The visual encoder 12 is further configured to input the second sample image and obtain a second visual feature representation of the second sample image;

[0180] The training unit 13 is further used to perform a second stage of model training based on the second visual feature representation, the second annotated data, the query converter and the target task model. The second stage of model training includes training the initial language model to obtain a language model; wherein the multimodal large model includes the visual encoder, the target task model, the query converter and the language model; the query converter is used to map visual features to a text domain, and the multimodal large model is used to generate corresponding answers based on the input image and text instructions.

[0181] Optionally, in some embodiments, the target task scenario includes at least one of an image text recognition scenario and an image segmentation scenario;

[0182] When the target task scenario includes the image text recognition scenario, the first annotation data includes text information contained in the first sample image;

[0183] When the target task scenario includes the image segmentation scenario, the first annotation data includes the category of each pixel in the first sample image.

[0184] Optionally, in some embodiments, when the target task scenario includes the image text recognition scenario, the training unit 13 is specifically configured to:

[0185] Inputting the first visual feature representation and the first labeled data into a first initial target task model to determine a first loss function;

[0186] The parameters of the first initial target task model are updated according to the first loss function to obtain a first target task model.

[0187] Optionally, in some embodiments, the first initial target task model includes a first Transformer and a second Transformer;

[0188] The training unit 13 is specifically configured to:

[0189] The first visual feature representation is input into the first Transformer, and the first labeled data is input into the second Transformer. A first image-text contrast learning ITC loss, a first image-based text generation ITG loss, and a first image-text matching ITM loss are determined based on the output of the first Transformer and the output of the second Transformer.

[0190] Optionally, in some embodiments, the first Transformer and the second Transformer share a self-attention module.

[0191] Optionally, in some embodiments, when the target task scenario includes the image segmentation scenario, the training unit 13 is specifically configured to:

[0192] Inputting the first visual feature representation into a second initial target task model to determine the probability that each pixel in the first sample image belongs to each category;

[0193] determining a second loss function based on the probability that each pixel in the first sample image belongs to each category and the first labeled data;

[0194] The parameters of the second initial target task model are updated according to the second loss function to obtain a second target task model.

[0195] Optionally, in some embodiments, the second initial target task model includes a transformation layer and at least one upsampling layer;

[0196] The training unit 13 is specifically configured to:

[0197] Inputting the first visual feature representation into the transformation layer to obtain a two-dimensional feature map;

[0198] The two-dimensional feature map is input into the at least one upsampling layer to obtain the probability that each pixel in the first sample image belongs to each category; wherein the number of channels of the last upsampling layer of the at least one upsampling layer is determined according to the number of categories.

[0199] Optionally, in some embodiments, the training unit 13 is specifically configured to:

[0200] Based on the second visual feature representation and the second labeled data, model training is performed on the query converter, the target task model, and the initial language model to obtain the trained multimodal large model.

[0201] Optionally, in some embodiments, the training unit 13 is specifically configured to:

[0202] Inputting the second visual feature representation into the query transformer to obtain a first text representation of the second sample image;

[0203] Inputting the second visual feature representation into the target task model to obtain a prediction result of the second sample image in the target task scenario;

[0204] Inputting the first text representation, the prediction result, and the annotation instruction in the second annotation data into the initial language model to obtain a prediction answer corresponding to the annotation instruction;

[0205] According to the predicted answer and the true answer in the second labeled data, the parameters of the query transformer, the target task model and the initial language model are updated to obtain the trained multimodal large model.

[0206] Optionally, in some embodiments, when the target task model includes a second target task model in a segmentation scenario, the training unit 13 is specifically configured to:

[0207] Inputting the prediction result into a mapping layer to obtain a second text representation of the prediction result in the text feature space of the initial language model;

[0208] Inputting the first text representation, the second text representation, and the annotation instruction into the initial language model to obtain the predicted answer;

[0209] According to the predicted answer and the true answer, the parameters of the query transformer, the second target task model, the mapping layer and the initial language model are updated to obtain the trained multimodal large model.

[0210] Optionally, in some embodiments, the acquiring unit 11 is further configured to acquire a third sample image and corresponding third annotation data, where the third annotation data includes a text description of the third sample image;

[0211] The visual encoder 12 is further configured to input the third sample image and obtain a third visual feature representation of the third sample image;

[0212] The training unit 13 is also used to perform a third stage of model training based on the third visual feature representation and the third labeled data. The third stage of model training includes performing model training on the initial query converter to obtain the query converter; wherein, the third stage of model training is before the second stage of model training.

[0213] Optionally, in some embodiments, the training unit 13 is specifically configured to input the third visual feature representation and the third labeled data into the initial query transformer to determine a third loss function;

[0214] The parameters of the initial query transformer are updated according to the third loss function to obtain the query transformer.

[0215] Optionally, in some embodiments, the initial query transformer includes a third Transformer and a fourth Transformer;

[0216] The training unit 13 is specifically configured to:

[0217] The third visual feature representation is input into the third Transformer, and the second labeled data is input into the fourth Transformer, and a second ITC loss, a second ITG loss, and a second ITM loss are determined according to the output of the third Transformer and the output of the fourth Transformer.

[0218] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 11 The multimodal large model training device 10 shown can execute the above method embodiment, and the aforementioned and other operations and / or functions of each module in the multimodal large model training device 10 are respectively for implementing the corresponding processes in the above method 400. For the sake of brevity, they will not be repeated here.

[0219] Figure 12 Schematic block diagram of the image and text generating device 20 of the embodiment of the present application. Figure 12 As shown, the apparatus 20 may include an acquisition unit 21 and a multimodal large model 22. Further, the multimodal large model 22 includes a visual encoder 221, a query transformer 222, a target task model 223 and a language model 224.

[0220] An acquisition unit 21, configured to acquire a first image;

[0221] a visual encoder 221, configured to input the first image and obtain a fourth visual feature representation of the first image;

[0222] a query transformer 222 configured to input the fourth visual feature representation and obtain a third textual representation of the first image;

[0223] The target task model 223 is configured to input the fourth visual feature representation input and obtain a target prediction result for the first image;

[0224] The language model 224 is configured to input the third text representation, the target prediction result, and the text instruction to obtain an answer to the text instruction;

[0225] Among them, the multimodal large model is obtained according to the multimodal large model training method provided in the embodiment of the present application.

[0226] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 12 The illustrated graphic and text generating device 10 can execute the above method embodiment, and the aforementioned and other operations and / or functions of each module in the graphic and text generating device 10 are respectively for realizing the corresponding processes in the above method 1000, which will not be described here for the sake of brevity.

[0227] The apparatus of the embodiment of the present application is described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, or can be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. Optionally, the software module can be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps in the above method embodiment in conjunction with its hardware.

[0228] Figure 13 It is a schematic block diagram of an electronic device provided in an embodiment of the present application.

[0229] like Figure 13 As shown, the electronic device 30 may include:

[0230] The memory 33 and the processor 32 are configured to store a computer program 34 and transmit the program code 34 to the processor 32. In other words, the processor 32 can call and run the computer program 34 from the memory 33 to implement the providing method in the embodiment of the present application.

[0231] In some embodiments, the processor 32 may call and run the computer program 34 from the memory 33 to implement the multimodal large model training method provided in the embodiments of the present application, including:

[0232] Acquire a first sample image and corresponding first annotation data in a target task scenario; and acquire a second sample image and corresponding second annotation data;

[0233] Inputting the first sample image into a visual encoder to obtain a first visual feature representation of the first sample image; the visual encoder is used to extract visual features of the input image;

[0234] Performing a first stage of model training based on the first visual feature representation and the first labeled data, wherein the first stage of model training includes training an initial target task model to obtain a target task model;

[0235] Inputting the second sample image into the visual encoder to obtain a second visual feature representation of the second sample image;

[0236] A second stage of model training is performed based on the second visual feature representation, the second annotated data, the query converter, and the target task model. The second stage of model training includes training the initial language model to obtain a language model. The multimodal large model includes the visual encoder, the target task model, the query converter, and the language model. The query converter is used to map visual features to a text domain, and the multimodal large model is used to generate corresponding answers based on the input image and text instructions.

[0237] For example, the processor 32 may be configured to execute the steps of the method 400 according to the instructions in the computer program 34 .

[0238] In some embodiments, the processor 32 may call and execute the computer program 34 from the memory 33 to implement the method for providing image-text generation in the embodiment of the present application, which is applied to a multimodal large model, wherein the multimodal large model includes a visual encoder, a query transformer, a target task model, and a language model, including:

[0239] acquiring a first image;

[0240] Inputting the first image into the visual encoder to obtain a fourth visual feature representation of the first image;

[0241] inputting the fourth visual feature representation into the query transformer to obtain a third textual representation of the first image;

[0242] Inputting the fourth visual feature representation into the target task model to obtain a target prediction result of the first image;

[0243] Inputting the third text representation, the target prediction result, and the text instruction into the language model to obtain an answer to the text instruction;

[0244] Wherein, the multimodal large model is obtained according to the model training method described in the first aspect.

[0245] For example, the processor 32 may be configured to execute the steps of the method 1000 according to the instructions in the computer program 34 .

[0246] In some embodiments of the present application, the processor 32 may include but is not limited to:

[0247] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0248] In some embodiments of the present application, the memory 33 includes but is not limited to:

[0249] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0250] In some embodiments of the present application, the computer program 34 may be divided into one or more units, which are stored in the memory 33 and executed by the processor 32 to implement the method provided by the present application. The one or more units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program 34 in the electronic device 30.

[0251] Optional, such as Figure 13 As shown, the electronic device 30 may further include:

[0252] The transceiver 33 may be connected to the processor 32 or the memory 33 .

[0253] The processor 32 can control the transceiver 33 to communicate with other devices. Specifically, it can send information or data to other devices, or receive information or data sent by other devices. The transceiver 33 may include a transmitter and a receiver. The transceiver 33 may further include an antenna, and the number of antennas may be one or more. It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes, in addition to the data bus, a power bus, a control bus, and a status signal bus.

[0254] The present application also provides a computer storage medium having a computer program stored thereon, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment. In other words, the present application also provides a computer program product containing instructions, which, when executed by a computer, enables the computer to perform the method of the above-mentioned method embodiment.

[0255] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0256] It is understandable that in the specific implementation of this application, when the above embodiments of this application are applied to specific products or technologies and involve relevant data such as user information, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0257] Those skilled in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0258] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the modules is merely a logical function division. In actual implementation, there may be other division methods, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0259] Modules described as separate components may or may not be physically separate, and components displayed as modules may or may not be physical modules, i.e., they may be located in one place or distributed across multiple network elements. Some or all of the modules may be selected based on actual needs to achieve the purpose of the present embodiment. For example, the functional modules in the various embodiments of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into a single module.

[0260] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.< / question> < / seg> < / ocr> < / seg> < / ocr>

Claims

1. A multimodal large model training method, characterized in that: include: Acquire a first sample image and corresponding first annotation data in a target task scenario; and obtaining a second sample image and corresponding second annotation data; Inputting the first sample image into a visual encoder to obtain a first visual feature representation of the first sample image; the visual encoder is used to extract visual features of the input image; Performing a first stage of model training based on the first visual feature representation and the first labeled data, wherein the first stage of model training includes training an initial target task model to obtain a target task model; Inputting the second sample image into the visual encoder to obtain a second visual feature representation of the second sample image; A second stage of model training is performed based on the second visual feature representation, the second annotated data, the query converter, and the target task model. The second stage of model training includes training the initial language model to obtain a language model. The multimodal large model includes the visual encoder, the target task model, the query converter, and the language model. The query converter is used to map visual features to a text domain, and the multimodal large model is used to generate corresponding answers based on the input image and text instructions.

2. The method according to claim 1, characterized in that The target task scenario includes at least one of an image text recognition scenario and an image segmentation scenario; When the target task scenario includes the image text recognition scenario, the first annotation data includes text information contained in the first sample image; When the target task scenario includes the image segmentation scenario, the first annotation data includes the category of each pixel in the first sample image.

3. The method according to claim 2, characterized in that When the target task scenario includes the image text recognition scenario, performing the first stage model training based on the first visual feature representation and the first annotation data includes: Inputting the first visual feature representation and the first labeled data into a first initial target task model to determine a first loss function; The parameters of the first initial target task model are updated according to the first loss function to obtain a first target task model.

4. The method according to claim 3, characterized in that The first initial target task model includes a first Transformer and a second Transformer; The step of inputting the first visual feature representation and the first labeled data into a first initial target task model and determining a first loss function includes: The first visual feature representation is input into the first Transformer, and the first labeled data is input into the second Transformer. A first image-text contrast learning ITC loss, a first image-based text generation ITG loss, and a first image-text matching ITM loss are determined based on the output of the first Transformer and the output of the second Transformer.

5. The method according to claim 4, characterized in that The first Transformer and the second Transformer share a self-attention module.

6. The method according to claim 2, characterized in that When the target task scenario includes the image segmentation scenario, performing the first stage of model training according to the first visual feature representation and the first annotated data includes: Inputting the first visual feature representation into a second initial target task model to determine the probability that each pixel in the first sample image belongs to each category; determining a second loss function based on the probability that each pixel in the first sample image belongs to each category and the first labeled data; The parameters of the second initial target task model are updated according to the second loss function to obtain a second target task model.

7. The method according to claim 6, characterized in that The second initial target task model includes a transformation layer and at least one upsampling layer; The step of inputting the first visual feature representation into the second initial target task model to determine the probability that each pixel in the first sample image belongs to each category includes: Inputting the first visual feature representation into the transformation layer to obtain a two-dimensional feature map; The two-dimensional feature map is input into the at least one upsampling layer to obtain the probability that each pixel in the first sample image belongs to each category; wherein the number of channels of the last upsampling layer of the at least one upsampling layer is determined according to the number of categories.

8. The method according to claim 1, characterized in that The performing a second phase of model training based on the second visual feature representation, the second annotated data, the query converter, and the target task model includes: Based on the second visual feature representation and the second labeled data, model training is performed on the query converter, the target task model, and the initial language model to obtain the trained multimodal large model.

9. The method according to claim 8, characterized in that The step of jointly training the query converter, the target task model, and the initial language model based on the second visual feature representation and the second annotated data to obtain the trained multimodal large model includes: Inputting the second visual feature representation into the query transformer to obtain a first text representation of the second sample image; Inputting the second visual feature representation into the target task model to obtain a prediction result of the second sample image in the target task scenario; Inputting the first text representation, the prediction result, and the annotation instruction in the second annotation data into the initial language model to obtain a prediction answer corresponding to the annotation instruction; According to the predicted answer and the true answer in the second labeled data, the parameters of the query transformer, the target task model and the initial language model are updated to obtain the trained multimodal large model.

10. The method according to claim 9, characterized in that When the target task model includes a second target task model in a segmentation scenario, inputting the first text representation, the prediction result, and the annotation instruction in the second annotation data into the initial language model to obtain a predicted answer corresponding to the annotation instruction includes: Inputting the prediction result into a mapping layer to obtain a second text representation of the prediction result in the text feature space of the initial language model; Inputting the first text representation, the second text representation, and the annotation instruction into the initial language model to obtain the predicted answer; The method includes updating the parameters of the query transformer, the target task model, and the initial language model according to the predicted answer and the true answer in the second annotated data to obtain the trained multimodal large model, including: According to the predicted answer and the true answer, the parameters of the query transformer, the second target task model, the mapping layer and the initial language model are updated to obtain the trained multimodal large model.

11. The method according to claim 1, characterized in that Also includes: Acquire a third sample image and corresponding third annotation data, where the third annotation data includes a text description of the third sample image; Inputting the third sample image into the visual encoder to obtain a third visual feature representation of the third sample image; Based on the third visual feature representation and the third labeled data, a third stage of model training is performed, wherein the third stage of model training includes performing model training on the initial query converter to obtain the query converter; wherein the third stage of model training is performed before the second stage of model training.

12. A method for generating images and texts, characterized in that: Applied to a large multimodal model, the large multimodal model includes a visual encoder, a query transformer, a target task model, and a language model, the method comprising: acquiring a first image; Inputting the first image into the visual encoder to obtain a fourth visual feature representation of the first image; inputting the fourth visual feature representation into the query transformer to obtain a third textual representation of the first image; Inputting the fourth visual feature representation into the target task model to obtain a target prediction result of the first image; Inputting the third text representation, the target prediction result, and the text instruction into the language model to obtain an answer to the text instruction; Wherein, the multimodal large model is obtained according to the model training method according to any one of claims 1-11.

13. A multimodal large model training device, characterized in that: include: An acquisition unit, configured to acquire a first sample image and corresponding first annotation data in a target task scenario; and obtaining a second sample image and corresponding second annotation data; a visual encoder configured to input the first sample image and obtain a first visual feature representation of the first sample image; the visual encoder is configured to extract visual features of the input image; a training unit, configured to perform a first stage of model training based on the first visual feature representation and the first labeled data, wherein the first stage of model training includes performing model training on an initial target task model to obtain a target task model; The visual encoder is further configured to input the second sample image and obtain a second visual feature representation of the second sample image; The training unit is further used to perform a second stage of model training based on the second visual feature representation, the second annotated data, the query converter and the target task model. The second stage of model training includes training the initial language model to obtain a language model. The multimodal large model includes the visual encoder, the target task model, the query converter and the language model. The query converter is used to map visual features to a text domain, and the multimodal large model is used to generate corresponding answers based on the input image and text instructions.

14. A graphic and text generating device, characterized in that: The apparatus includes a multimodal large model, wherein the multimodal large model includes a visual encoder, a query transformer, a target task model, and a language model, and further includes: an acquisition unit, configured to acquire a first image; The visual encoder is used to input the first image and obtain a fourth visual feature representation of the first image; The query transformer is configured to input the fourth visual feature representation and obtain a third text representation of the first image; The target task model is configured to input the fourth visual feature representation input and obtain a target prediction result for the first image; The language model is configured to input the third text representation, the target prediction result, and the text instruction to obtain an answer to the text instruction; Wherein, the multimodal large model is obtained according to the model training method according to any one of claims 1-11.

15. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores instructions, and when the processor executes the instructions, the processor executes the method according to any one of claims 1 to 12.

16. A computer storage medium, characterized in that Used to store a computer program, the computer program comprising instructions for executing the method according to any one of claims 1 to 12.

17. A computer program product, characterized in that The method comprises a computer program code, which, when executed by an electronic device, causes the electronic device to perform the method according to any one of claims 1 to 12.

Images