Multi-modal large model training method and device, image classification method and device, intelligent agent, equipment, medium and product

Abstract

The invention provides a multi-modal large model training method and device, an image classification method and device, an intelligent agent, equipment, a medium and a product, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to multi-modal recognition and classification scenes. The multi-modal large model training method comprises the steps that a training sample set is acquired, the training sample set comprises a plurality of training sample pairs, and each training sample pair comprises a sample image, a sample text and category description used for the sample image; inputting each training sample pair into a to-be-trained large model to obtain category information of each sample image; and training a to-be-trained large model by using the category description and category information of each training sample to obtain a multi-modal large model for image classification.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, and large models, and can be applied to multimodal recognition and classification scenarios. More specifically, it relates to a multimodal large model training, image classification method and device, intelligent agent, equipment, medium and product. Background Art

[0002] As large language models gradually enhance their language understanding and reasoning capabilities, multimodal large language models (MLLMs) for perceiving and understanding visual information such as images have emerged.

[0003] A multimodal large language model refers to a model based on a large language model that has the ability to receive, infer, and output multimodal information and can process and understand multiple different types of data inputs, such as text, images, audio, and video. Summary of the invention

[0004] The present disclosure provides a multimodal large model training, image classification method and device, intelligent agent, equipment, medium and product.

[0005] According to one aspect of the present disclosure, a method for training a multimodal large model is provided, comprising: obtaining a training sample set, wherein the training sample set comprises a plurality of training sample pairs, and each of the training sample pairs comprises a sample image, a sample text, and a category description for the sample image; inputting each of the training sample pairs into a large model to be trained, respectively, to obtain category information of each of the sample images; and, using the category description and category information of each of the training sample pairs, training the large model to be trained, to obtain a multimodal large model for image classification.

[0006] According to one aspect of the present disclosure, there is provided an image classification method, comprising: acquiring an image to be classified; and inputting the image to be classified into a multimodal large model to obtain an image classification result, wherein the multimodal large model is trained using a multimodal large model training method.

[0007] According to another aspect of the present disclosure, a training device for a multimodal large model is provided, comprising: a first acquisition module, used to acquire a training sample set, wherein the training sample set comprises a plurality of training sample pairs, each of the training sample pairs comprises a sample image, a sample text and a category description for the sample image; a first input module, used to input each of the training sample pairs into a large model to be trained, respectively, to obtain category information of each of the sample images; and a training module, used to train the large model to be trained using the category description and category information of each of the training sample pairs, to obtain a multimodal large model for image classification.

[0008] According to another aspect of the present disclosure, there is provided an image classification device, comprising: a second acquisition module, used to acquire an image to be classified; and a second input module, used to input the image to be classified into a multimodal large model to obtain an image classification result, wherein the multimodal large model is trained using a multimodal large model training device.

[0009] According to another aspect of the present disclosure, an artificial intelligence agent is provided, which is configured to execute a multimodal large model training method and / or an image classification method.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0011] According to another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program or instruction is stored. When the computer program or instruction is executed by a processor, the steps of the above method are implemented.

[0012] According to another aspect of the present disclosure, a computer program product is provided, including a computer program or instructions, and the steps of the above method are implemented when the computer program or instructions are executed by a processor.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0015] Figure 1The system architecture of the training method and image classification method of the multimodal large model according to the embodiment of the present disclosure is schematically shown;

[0016] Figure 2 A flowchart of a method for training a multimodal large model according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 3 An example schematic diagram schematically illustrates a process of constructing a candidate category mapping according to an embodiment of the present disclosure;

[0018] Figure 4A An example schematic diagram schematically shows a process of obtaining input features according to an embodiment of the present disclosure;

[0019] Figure 4B The following schematically shows an example of a process for obtaining category information according to an embodiment of the present disclosure;

[0020] Figure 4C An example schematic diagram schematically shows a training process of a multimodal large model according to an embodiment of the present disclosure;

[0021] Figure 5 An example schematic diagram schematically shows a training process of a multimodal large model according to another embodiment of the present disclosure;

[0022] Figure 6 The flowchart of the image classification method according to the embodiment of the present disclosure is schematically shown;

[0023] Figure 7 A block diagram of a training device for a multimodal large model according to an embodiment of the present disclosure is schematically shown;

[0024] Figure 8 A block diagram of an image classification device according to an embodiment of the present disclosure is schematically shown;

[0025] Fig. 9 A block diagram schematically shows a structure of an intelligent agent of a large model according to an embodiment of the present disclosure; and

[0026] Fig.10 A block diagram of an electronic device suitable for implementing a training method for a multimodal large model and an image classification method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0027] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0028] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of the features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0029] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0030] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0031] In one example, an image classification model or a multimodal large language model may be used to perform image classification.

[0032] Image classification models (such as MobileNet, ResNet, etc.) show high reliability and stable output probabilities in image classification tasks, and these probabilities can better reflect the true probability distribution of each category. However, these image classification models are limited to feature extraction and classification of a single modality (i.e., image), and it is difficult to support the understanding needs of image-text fusion in multimodal scenarios.

[0033] Multimodal large language models can map images and texts into a common feature space to achieve cross-modal matching and search. However, since multimodal large language models are usually trained based on contrastive learning, they output similarity scores between images and texts, which cannot directly reflect the true category probability distribution. Therefore, they are insufficient in classification accuracy and confidence interpretability.

[0034] To this end, the embodiment of the present disclosure proposes a training scheme for a multimodal large model. For example, a training sample set is obtained, wherein the training sample set includes a plurality of training sample pairs, each training sample pair includes a sample image, a sample text, and a category description for the sample image; each training sample pair is input into the large model to be trained, and the category information of each sample image is obtained; and the category description and category information of each training sample pair are used to train the large model to be trained, and a multimodal large model for image classification is obtained.

[0035] According to an embodiment of the present disclosure, by adding the respective category description of each sample image in the training sample set, the training sample set is input into the large model to be trained to obtain the respective category information of each sample image, and the respective category description and category information of each training sample are used to train the large model to be trained, so that the large model can not only learn the image features of the sample images during the training process, but also combine the semantic features of the sample text, so that the trained multimodal large model can perform category recognition more accurately, improve the consistency and reliability of its output category information, as well as the robustness and adaptability of the multimodal large model to image classification tasks, thereby helping to improve the efficiency and accuracy of image classification using the multimodal large model.

[0036] In the technical solution of the present invention, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0037] In the technical solution of the present invention, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0038] Figure 1 The system architecture of the training method and image classification method of the multimodal large model according to the embodiment of the present disclosure is schematically shown. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0039] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0040] The user may use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications may be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0041] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0042] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0043] It should be noted that the training method and image classification method of the multimodal large model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the training device and image classification device of the multimodal large model provided in the embodiment of the present disclosure can generally be set in the server 105. The training method and image classification method of the multimodal large model provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Accordingly, the training device and image classification device of the multimodal large model provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0044] Alternatively, the training method and image classification method of the multimodal large model provided in the embodiments of the present disclosure may also be executed by the first terminal device 101, the second terminal device 102 or the third terminal device 103, or may also be executed by other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103. Accordingly, the training device and image classification device of the multimodal large model provided in the embodiments of the present disclosure may also be arranged in the first terminal device 101, the second terminal device 102 or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102 or the third terminal device 103.

[0045] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0046] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0047] The above describes the system architecture of the training method for multimodal large models provided by the present disclosure. Figure 2 For example, the training process of the multimodal large model disclosed in the present invention is further explained.

[0048] Figure 2 The flowchart of the training method of the multimodal large model according to the embodiment of the present disclosure is schematically shown.

[0049] like Figure 2 As shown, the training method 200 of the multimodal large model includes operations S210~S230.

[0050] In operation S210 , a training sample set is obtained, where the training sample set includes a plurality of training sample pairs, each of which includes a sample image, a sample text, and a category description for the sample image.

[0051] In operation S220, each training sample pair is input into the large model to be trained to obtain the category information of each sample image.

[0052] In operation S230, the large model to be trained is trained using the respective category description and category information of each training sample to obtain a multimodal large model for image classification.

[0053] A training sample set refers to a data set used to train a large model to be trained, and the data set may include multiple training sample pairs. Each training sample pair may include a sample image, a sample text, and a category description for the sample image. A sample image refers to image data used for training, and a sample image may be any type of digital image, for example, JPG (Joint Photographic Experts Group), TIFF (Tag Image File Format), PNG (Portable Network Graphics), PDF (Portable Document Format), and GIF (Graphics Interchange Format). The present disclosure embodiment does not limit the file format of the sample image.

[0054] Sample text refers to text data associated with sample images. Sample text can be a description of the image content, a label, or other related information. Category description refers to information used to describe the category to which the sample image belongs. Category description can be a label, category name, or other form of identifier to guide the model's classification task.

[0055] In order to effectively guide the multimodal large model to perform image classification, the preparation of the training data set needs to take into account multimodal understanding, the accuracy of the category probability output, and the reliability of the category confidence. The method of obtaining the training sample set can be configured according to actual business needs and is not limited here. For example, an open source multimodal data set can be used as a training sample set. The open source multimodal data set can provide a large number of images and descriptions in real scenes, which can be used as basic training to ensure the basic multimodal capabilities of the model. Alternatively, an annotated multimodal data set can be used as a training sample set. The annotated multimodal data set can provide a wide range of image data with specific category descriptions, which can be used as auxiliary training to enhance the accuracy of the model category prediction. In one example, the category in the category description can be marked as a preset character, and the preset character is placed at the beginning of the category description to facilitate subsequent training.

[0056] After obtaining the training sample set, each training sample pair can be input into the large model to be trained to obtain the category information of each sample image. Category information refers to the category-related data extracted from the sample image by the large model to be trained. The category information is used for subsequent comparison with the category description to guide the large model to perform accurate classification.

[0057] The large model to be trained can be a pre-trained multimodal model. After the training sample pairs are input into the large model to be trained, the sample images can be processed for image feature extraction to obtain image features, and the sample texts can be processed for text feature extraction to obtain text features. On this basis, the image features and text features can be mapped into a shared space to obtain fused features, and based on the fused features and the category description, the category information can be output.

[0058] The image feature extraction processing method may include at least one of the following: a traditional feature extraction method and an image feature extraction method based on a deep learning model. The traditional feature extraction method may include at least one of the following: SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF), etc. The image feature extraction method based on a deep learning model may include at least one of the following: a variational autoencoder (VAE) model, a convolutional variational autoencoder (CVAE) model, and a bidirectional transformer (i.e., Bidirectional Transformer) model, etc.

[0059] The text feature extraction processing method may include at least one of the following: a text feature extraction method based on a bag-of-words (BoW) model, a text feature extraction method based on a TF-IDF model, a text feature extraction method based on an n-gram model, a text feature extraction method based on word embedding, and a text feature extraction method based on a deep learning model, etc.

[0060] The feature fusion method may include at least one of the following: a cross-modal feature determination method based on vector concatenation, a cross-modal feature determination method based on a Transformer model, a cross-modal feature determination method based on an attention mechanism, a cross-modal feature determination method based on a bidirectional transformer model, and a cross-modal feature determination method based on a multilayer perceptron (MLP).

[0061] After obtaining the respective category information of each training sample pair, for each training sample pair, the category loss value can be determined based on the category description and category information, and the category loss value can be used to adjust the model parameters of the large model to be trained until the predetermined end condition is met. Alternatively, for each training sample pair, the image loss value can be determined based on the sample image and image features, the text loss value can be determined based on the sample text and text features, and the image loss value and the text loss value can be used to adjust the model parameters of the large model to be trained until the predetermined end condition is met. The predetermined end condition may include at least one of the following: reaching the maximum number of training rounds, loss convergence, reaching a predetermined performance indicator, and the performance on the verification sample set no longer improving, etc. Thus, the model obtained when the predetermined end condition is met can be used as a multimodal large model for image classification.

[0062] According to an embodiment of the present disclosure, by adding the respective category description of each sample image in the training sample set, the training sample set is input into the large model to be trained to obtain the respective category information of each sample image, and the respective category description and category information of each training sample are used to train the large model to be trained, so that the large model can not only learn the image features of the sample images during the training process, but also combine the semantic features of the sample text, so that the trained multimodal large model can perform category recognition more accurately, improve the consistency and reliability of its output category information, as well as the robustness and adaptability of the multimodal large model to image classification tasks, thereby helping to improve the efficiency and accuracy of image classification using the multimodal large model.

[0063] In the embodiments of the present disclosure, after obtaining the training sample set, a candidate category mapping for the application scenario of the training sample set can also be constructed. Figure 3 Take as an example to illustrate the process of constructing the candidate category mapping.

[0064] Figure 3 An example schematic diagram schematically illustrates a process of constructing a candidate category mapping according to an embodiment of the present disclosure.

[0065] like Figure 3 As shown in 300, the training sample set may be determined according to the application scenario of the multimodal large model, and the training sample set may include multiple sample pairs, for example, sample pair 301_1, sample pair 301_2, ..., sample pair 301_M, where M is a positive integer greater than 1. Taking sample pair 301_2 as an example, sample pair 301_2 may include sample image 301_21, sample text 301_22, and category description 301_23.

[0066] In one example, after obtaining the training sample set, for the above multiple sample pairs, the frequency of the category description 302 can be determined according to the category descriptions included in each of the multiple sample pairs. The frequency 302 represents the probability of the same or similar category description appearing in the multiple sample pairs.

[0067] According to the frequency of the description of the category by the multiple samples, at least one candidate category 303 for the application scenario is determined. At least one candidate category 303 may refer to a category for the application scenario that needs to be trained, and is used to assist the multimodal large model in selecting the most suitable category in the classification output. At least one candidate category 303 may be adapted to the application scenario. The application scenario may be configured according to actual business needs and is not limited here.

[0068] For example, when training a multimodal big model for image classification tasks in an autonomous driving scenario, at least one candidate category 303 may include vehicles, pedestrians, buildings, and traffic signs, etc. Alternatively, when training a multimodal big model for image classification tasks in a medical scenario, at least one candidate category 303 may include X-ray imaging, magnetic resonance imaging, ultrasound imaging, and computed tomography imaging, etc. Alternatively, when training a multimodal big model for image classification tasks in an e-commerce scenario, at least one candidate category 303 may include items, models, reviews, activities, and packaging, etc. Alternatively, when training a multimodal big model for image classification tasks in an education scenario, at least one candidate category 303 may include Chinese, mathematics, and physics, etc.

[0069] After obtaining at least one candidate category 303, a category identifier 304 may be assigned to each candidate category 303. The category identifier 304 may refer to a mark (i.e., a token) used to uniquely identify the candidate category. The specific form of the category identifier 304 may be configured according to actual business requirements and is not limited here. For example, the category identifier 304 may include at least one of a number, a letter, and a character.

[0070] On this basis, a candidate category mapping 305 can be constructed according to the mapping relationship between the candidate category 303 and the category identifier 304. The candidate category mapping 305 can refer to a category vocabulary (i.e., a vocabulary). The specific form of the candidate category mapping 305 can be configured according to actual business needs and is not limited here. For example, the candidate category mapping 305 can be in the form of a list, a dictionary, an array, etc.

[0071] In a specific example, taking the training of a multimodal large model for image classification tasks in an autonomous driving scenario as an example, a candidate category mapping 306 in the form of a list can be constructed. As described above, in an autonomous driving scenario, at least one candidate category includes vehicles, pedestrians, buildings, and traffic signs. A category identifier "0001" can be configured for "vehicles", a category identifier "0002" can be configured for "pedestrians", a category identifier "0003" can be configured for "buildings", and a category identifier "0004" can be configured for "traffic signs". Based on the mapping relationship between the above candidate categories and category identifiers, a candidate category mapping 306 is constructed.

[0072] According to the embodiments of the present disclosure, the practicality and accuracy of the model are significantly improved by accurately customizing the training sample set to match the application scenario of the multimodal large model. In addition, by analyzing the frequency of descriptions of the categories included in each sample, the most representative and critical candidate categories in the application scenario can be identified, which not only optimizes the data selection process for model training, but also ensures that the model can focus on learning and identifying those most common and important categories in the application scenario. On this basis, since the candidate category mapping is constructed based on the mapping relationship between the candidate category and the category identifier, the subsequent large model can directly output the category label when processing multimodal data, thereby more accurately performing category matching and identification, and improving the accuracy of image classification of the multimodal large model in specific application scenarios.

[0073] In the embodiment of the present disclosure, after obtaining the training sample set, the training sample pairs can also be input into the large model to be trained to obtain the fusion features of the sample image and the sample text, and the category features of the category description. On this basis, the input features can be determined according to the fusion features and the category features. Figure 4A Take as an example to illustrate the process of obtaining input features.

[0074] Figure 4A An example schematic diagram of a process of obtaining input features according to an embodiment of the present disclosure is schematically shown.

[0075] like Figure 4A As shown, in 400A, taking the training sample pair 401 including the sample image 401_1, the sample text 401_2 and the category description 401_3, and the large model to be trained including the image feature extraction module 410 (i.e., Image Encoder), the text feature extraction module 420 (i.e., Text Encoder) and the multimodal projector 430 (i.e., Multimodal Projector) as an example, the process of obtaining the input features is explained.

[0076] In one example, the sample image 401_1 can be input into the image feature extraction module 410 to obtain the initial image feature 402. The image feature extraction module 410 is used to convert the input sample image 401_1 into a feature vector. In this process, the image feature extraction module 410 can be used to gradually extract the high-level features of the sample image 401_, and output the initial image feature 402 of a fixed dimension. The specific implementation of the image feature extraction module 410 can be configured according to actual business needs and is not limited here. For example, the image feature extraction module 410 may include at least one of the following: Convolutional Neural Networks (CNN) and Vision Transformer (ViT), etc.

[0077] In another example, the sample text 401_2 can be input into the text feature extraction module 420 to obtain the initial text feature 403. The text feature extraction module 420 is used to process natural language input and convert the input sample text 401_2 into a semantic embedding representation. In this process, the text feature extraction module 420 can be used to perform word segmentation and semantic embedding processing on the sample text 401_2, and generate context-related initial text features 403 based on the attention mechanism. The specific implementation of the text feature extraction module 420 can be configured according to actual business needs and is not limited here. For example, the text feature extraction module 420 can include a pre-trained language model based on Transformer.

[0078] The multimodal input is received through the image feature extraction module 410 and the text feature extraction module 420, and the multimodal input is respectively converted into a unified feature representation to respectively extract effective information of the multimodal input.

[0079] After obtaining the initial image features 402 and the initial text features 403, the multimodal projector 430 can be used to project the initial image features 402 and the initial text features 403 into the same feature space to obtain image features 404 and text features 405. The multimodal projector 430 is used to project the outputs of the image feature extraction module 410 and the text feature extraction module 420 into a shared space, so that these heterogeneous modal data are aligned in the shared space and have similar representations, so that when the multimodal large model is inferred, the image and text can be effectively compared and combined, so that they can be further used for downstream tasks.

[0080] The specific implementation of the multimodal projector 430 can be configured according to actual business needs and is not limited here. For example, the multimodal projector 430 may include at least one of the following: a fully connected layer and a multi-layer perceptron. The multimodal projector 430 can effectively utilize the information of each modality, ensure that the unique information of each modality is retained, and complement and align the information through the shared space.

[0081] After obtaining the image feature 404 and the text feature 405, the image feature 404 and the text feature 405 may be fused to obtain a fused feature 406. The fusion may include at least one of the following: concatenation and attention-based fusion. For example, the image feature 404 and the text feature 405 may be dynamically combined by using a cross-modal attention mechanism so as to adjust the association between the two modalities during the reasoning process.

[0082] The category decoder 440 is used to perform feature extraction processing on the category description 401_3 to obtain the category feature 407. In this process, the category word list based on the category decoder 440 is a candidate category mapping corresponding to the training sample set for the current application scenario, that is, the category decoder 440 can output any candidate category in the candidate category mapping. The result of the decoding process can be used as a probability distribution to represent the confidence of each candidate category after softmax. Alternatively, the category description 401_3 can also be encoded by using one-hot encoding to obtain the category feature 407.

[0083] After obtaining the fused feature 406, the input feature 408 can be determined according to the fused feature 406 and the category feature 407. For example, a preset output position can be reserved in the feature structure output by the large model to be trained to generate a category label V for characterizing the category feature 407, and the fused feature 406 and the category feature 407 are superimposed in a contact manner to obtain the input feature 408, thereby facilitating the subsequent calculation of the category probability and simplifying the model structure.

[0084] In one example, when the dimension of the category feature 407 is one, the preset output position can be the first mark bit of multiple mark bits, so that the category feature 407 can be used as the first mark bit of multiple mark bits, and the fusion feature 406 can be used as the remaining mark bits of the multiple mark bits except the first mark bit to obtain the input feature 408. In this case, the number of mark bits in the input feature 408 is 1+the feature dimension of the fusion feature 406.

[0085] In another example, when the dimension of the category feature 407 is greater than one, the preset output position may also be from the first mark bit of the multiple mark bits to the mark bits of the dimension of the category feature 407. In this case, the number of mark bits in the input feature 408 is the feature dimension of the category feature 407 + the feature dimension of the fusion feature 406.

[0086] For example, the sample image 401_1 and the sample text 401_2 represent "blue sky", at least one candidate category includes red, green and blue, etc., and the category description 401_3 is "blue". Then the category description 401_3 can be encoded using one-hot encoding, and the category feature 407 is [0,0,1]. The sample image 401_1 and the sample text 401_2 are processed using the large model to be trained, and the fusion feature 406 is [X,Y,Z]. Therefore, it can be determined that the input feature 408 is [0,0,1,X,Y,Z].

[0087] According to the embodiments of the present disclosure, by inputting the training sample pairs into the large model to be trained, the deep fusion of the sample images and sample texts and the feature extraction of the category description are realized, and more accurate fusion features and category features can be extracted. On this basis, the input features are determined by sampling the category feature as the first marker and the fusion feature as the remaining markers, so that the large model can make full use of the fusion features and category features, thereby improving the accuracy of the category information.

[0088] In the embodiment of the present disclosure, after obtaining the candidate category mapping and the input features, the output category can also be determined according to the candidate category mapping and the input features. Figure 4B Take as an example to illustrate the process of obtaining category information.

[0089] Figure 4B The following is a schematic diagram schematically showing an example of a process of obtaining category information according to an embodiment of the present disclosure.

[0090] like Figure 4B As shown, in 400B, the process of obtaining category information is explained by taking the large model to be trained also including the large language model 450 as an example.

[0091] Input feature 409 is input to large language model 450 to obtain category information. In this process, large language model 450 can generate corresponding output according to input feature 409. For example, for image classification tasks, the output of large language model 450 is category prediction; for multimodal question-answering tasks, the output of large language model 450 is a natural language answer.

[0092] In one example, the category information may include an output category 413 and an output confidence. The output category 413 refers to the category to which the sample image predicted by the large model to be trained belongs, and the output confidence refers to the possibility that the sample image predicted by the large model to be trained belongs to the output category 413.

[0093] By inputting the input feature 409 into the large language model 450, a candidate output confidence 412 of each of the at least one category identifier is obtained. For the at least one candidate output confidence 412, a maximum confidence with the largest value can be determined, and according to the candidate category mapping 411, the candidate category corresponding to the maximum confidence in the at least one candidate output confidence 412 can be determined as the output category 413, and the maximum confidence is used as the output confidence.

[0094] In one example, a confidence threshold may be preset, and when the output confidence is less than the confidence threshold, the output result of the large language model 450 may be discarded and other strategies may be used instead. The confidence threshold may also be adapted to the application scenario of the training sample set.

[0095] For example, taking a multimodal large model for an image classification task applied to an autonomous driving scenario, in this scenario, at least one candidate category includes vehicles, pedestrians, buildings, and traffic signs. The category identifier corresponding to "vehicle" is "0001", the category identifier corresponding to "pedestrian" is "0002", the category identifier corresponding to "building" is "0003", and the category identifier corresponding to "traffic sign" is "0004". The output of the large model to be trained can be "vehicle: 0.6; pedestrian: 0.1; building: 0.15; traffic sign: 0.15", and the maximum confidence can be determined to be "0.6". The candidate category corresponding to the maximum confidence is "vehicle", and thus, the output category 413 can be determined to be "vehicle" and the output confidence can be determined to be "0.6".

[0096] According to an embodiment of the present disclosure, by inputting input features into a large language model, the output confidence of at least one category identifier is calculated. On this basis, by accurately determining the candidate category corresponding to the maximum confidence as the output category according to the candidate category mapping, and using the maximum confidence as the output confidence, the accuracy of category recognition and the confidence assessment capability of the model are improved. In the above manner, the model can not only provide the most likely category prediction, but also give the corresponding confidence, thereby enhancing the reliability and practicality of the model.

[0097] In the embodiment of the present disclosure, after obtaining the category information, the large model to be trained can be trained using the category description and category information of each training sample to obtain the initial multimodal large model. Figure 4CTake as an example to illustrate the training process of the initial multimodal large model.

[0098] Figure 4C An example schematic diagram of the training process of a multimodal large model according to an embodiment of the present disclosure is schematically shown.

[0099] like Figure 4C As shown, in 400C, taking a sample pair including a sample image 414_1, a sample text 414_2 and a category description 414_3, and a large model 460 to be trained including an image feature extraction module 410, a text feature extraction module 420, a multimodal projector 430, a category decoder 440 and a large language model 450 as an example, the training process of the multimodal large model is explained.

[0100] In one example, the sample image 414_1 can be processed by the image feature extraction module 410 to obtain the initial image feature; the sample text 414_2 can be processed by the text feature extraction module 420 to obtain the initial text feature; the initial image feature and the initial text feature can be projected into the same feature space by the multimodal projector 430 to obtain the image feature and the text feature; the fusion feature can be determined based on the image feature and the text feature; the category description 414_3 can be processed by the category decoder 440 to obtain the category feature; the input feature 415 can be determined based on the fusion feature and the category feature. On this basis, the input feature 415 can be input into the large language model 450 to obtain the output category 416.

[0101] After obtaining the output category 416, a first loss value 417 can be determined based on the first loss function according to the category description 414_3 and the output category 416; a second loss value 418 can be determined based on the second loss function according to the sample image 414_1, the sample text 414_2, and the image features and the text features. On this basis, a loss function value 419 can be determined based on the first loss value and the second loss value. Based on the loss function value 419, the large model to be trained 460 is trained to obtain an initial multimodal large model.

[0102] The specific forms of the first loss function and the second loss function can be configured according to actual business needs and are not limited here. For example, the second loss function can be contrastive loss, and the first loss function can be cross-entropy loss to ensure that the trained multimodal large model can accurately classify the input image.

[0103] According to an embodiment of the present disclosure, by combining a first loss value based on a category description and an output category, and a second loss value based on image features and text features, a loss function value is determined, so that the model can simultaneously optimize the accuracy of category prediction and the alignment of image and text features during training. On this basis, by training the large model to be trained based on the loss function value, it is helpful to improve the accuracy of image classification using the initial multimodal large model.

[0104] In the embodiments of the present disclosure, after obtaining the initial multimodal large model, the validation sample set can be used to adjust the initial multimodal large model until the predetermined performance condition is met to obtain the multimodal large model. Figure 5 Take as an example to illustrate the training process of a large multimodal model.

[0105] Figure 5 An example schematic diagram of a training process of a multimodal large model according to another embodiment of the present disclosure is schematically shown.

[0106] like Figure 5 As shown, in 500, a verification sample set may be obtained. The verification sample set may include multiple sample pair sets, each sample pair set including multiple verification sample pairs. Taking sample pair set 510 as an example, the sample pair set 510 may include verification sample pair 510_1, verification sample pair 510_2, ..., verification sample pair 510_N, where N is a positive integer greater than 1.

[0107] It should be noted that the multiple verification sample pairs in the same sample pair set include at least two verification category information for the same verification image and the same verification text. The at least two verification category information are different verification confidences for the same verification category.

[0108] Taking the above sample pair set 510 as an example, the verification sample pairs 510_1, 510_2, ..., 510_N in the sample pair set 510 all include the same verification image 501, the same verification text 502, the same verification category 503, and different verification confidences. For example, in the verification sample pair 510_1, the verification confidence for the verification category 503 is verification confidence 504_1; in the verification sample pair 510_2, the verification confidence for the verification category 503 is verification confidence 504_2; and so on, in the verification sample pair 510_N, the verification confidence for the verification category 503 is verification confidence 504_N.

[0109] In one example, the method for obtaining different verification confidences for the same verification image, the same verification text, and the same verification category can be configured according to actual business needs and is not limited here. For example, the method for obtaining different verification confidences can include at least one of the following: determining different verification confidences based on temperature scaling (i.e., Temperature Scaling) and determining different verification confidences based on negative log-likelihood calibration (Negative Log-Likelihood Calibration).

[0110] For example, the method of determining different verification confidences based on temperature scaling can introduce a scaling parameter to scale the category information output by the initial multimodal large model during the training phase to adjust the probability distribution of the initial multimodal large model output so that it can more accurately reflect the actual confidence and obtain different verification confidences. Alternatively, the method of determining different verification confidences based on log-likelihood calibration can process the category information output by the initial multimodal large model during the training phase by maximizing the prediction probability of the correct category while minimizing the prediction probability of the wrong category to obtain different verification confidences.

[0111] After constructing multiple sample pair sets in the above manner, the multiple sample pair sets can be determined as verification sample sets. Using the verification sample set, the initial multimodal large model 520 is adjusted until the predetermined performance conditions are met to obtain a multimodal large model. In one example, the sample pair set 510 can be input into the initial multimodal large model 520 to obtain verification information 530; based on the verification loss function, the verification function value 540 is determined according to the sample pair set 510 and the verification information 530; using the verification function value 540, the model parameters of the initial multimodal large model 520 are adjusted until the predetermined performance conditions are met to improve the accuracy of the category probability output; the initial multimodal large model 520 obtained when the predetermined performance conditions are met is determined as the multimodal large model.

[0112] According to an embodiment of the present disclosure, since the multiple verification sample pairs in the sample pair set include at least two verification category information for the same verification image and the same verification text, and the at least two verification category information are different verification confidences for the same verification category, by utilizing the verification sample set to adjust the initial multimodal large model, it helps to improve the output probability interpretability and classification reliability of the final multimodal large model in the image classification task.

[0113] The above are only exemplary embodiments, but are not limited thereto and may also include other multimodal large model training methods known in the art, as long as they can improve the efficiency and accuracy of image classification using the multimodal large model.

[0114] The above describes the training process of the multimodal large model provided by the present disclosure. Figure 6 For example, the process of using a multimodal large model to perform image classification in the present invention is further explained.

[0115] Figure 6 The flowchart of the image classification method according to the embodiment of the present disclosure is schematically shown.

[0116] like Figure 6 As shown, the image classification method 600 includes operations S610-S620.

[0117] In operation S610, an image to be classified is acquired.

[0118] In operation S620, the image to be classified is input into the multimodal large model to obtain an image classification result. The multimodal large model is trained using a training method for the multimodal large model.

[0119] The images to be classified may belong to the same application scenario as the training sample set used to train the multimodal large model. The application scenario refers to the specific scenario in which the multimodal large model is designed and trained to handle image classification tasks. In one example, the application scenario may include autonomous driving scenarios, medical scenarios, e-commerce scenarios, and education scenarios.

[0120] During the training process of the multimodal large model, the multimodal large model is trained by utilizing a training sample set belonging to an application scenario, so that the trained multimodal large model can adapt to the needs of specific image classification tasks in the application scenario.

[0121] During the application of the multimodal large model, since the image to be classified and the training sample set belong to the same application scenario, the image to be classified is processed by utilizing the multimodal large model to perform the image classification task of a specific application scenario, thereby assigning the image to be classified to a predefined category. As a result, the obtained image classification result can also adapt to the requirements of the specific image classification task of the application scenario.

[0122] In one example, the image classification result may include the confidence of the target category and at least one candidate category. The target type refers to the most likely image category of the image to be classified obtained by the multimodal large model performing the image classification task on the image to be classified. The at least one candidate category is determined during the training process of the multimodal large model according to the frequency of descriptions of the included category by multiple samples in the training sample set.

[0123] The confidence level characterizes the prediction confidence of the multimodal large model for each candidate category. For example, the higher the confidence level, the more confident the multimodal large model is in the candidate category corresponding to the confidence level, that is, the higher the possibility that the candidate category is the target category of the image to be classified; correspondingly, the lower the confidence level, the lower the possibility that the candidate category corresponding to the confidence level is the target category of the image to be classified. The target category belongs to at least one candidate category. For example, for each confidence level of the at least one candidate category, the candidate category with the highest confidence level can be determined as the target category, and the confidence level corresponding to the target category can be used as the target confidence level.

[0124] According to the embodiments of the present disclosure, since the images to be classified are images belonging to the same application scenario as the training sample set used to train the multimodal large model, accurate image classification can be achieved. By using the confidence corresponding to the target category in the confidence of at least one candidate category included in the image classification result as the target confidence, not only the accuracy of the target category is improved, but also the ability to evaluate the confidence of the classification result is enhanced, which helps to improve the classification performance and reliability of the multimodal large model in this application scenario.

[0125] According to the embodiments of the present disclosure, since the multimodal large model is trained using the training method of the multimodal large model, by inputting the image to be classified into the multimodal large model, an image classification result that characterizes the target category to which the image to be classified belongs and the target confidence of the target category can be obtained simultaneously, thereby improving the accuracy of the target category and the interpretability of the image classification result.

[0126] The above are only exemplary embodiments, but are not limited thereto and may also include other multimodal large model training methods known in the art, as long as they can improve the accuracy of the target category and the interpretability of the image classification results.

[0127] Based on the above-mentioned multimodal large model training method, the present invention also provides a multimodal large model training device. Figure 7 The device is described in detail.

[0128] Figure 7 A block diagram of a training device for a multimodal large model according to an embodiment of the present disclosure is schematically shown.

[0129] like Figure 7 As shown, the training device 700 for a multimodal large model may include a first acquisition module 710 , a first input module 720 and a training module 730 .

[0130] The first acquisition module 710 is used to acquire a training sample set, wherein the training sample set includes a plurality of training sample pairs, and each training sample pair includes a sample image, a sample text, and a category description for the sample image.

[0131] The first input module 720 is used to input each training sample pair into the large model to be trained to obtain the category information of each sample image.

[0132] The training module 730 is used to train the large model to be trained by using the respective category description and category information of each training sample to obtain a multimodal large model for image classification.

[0133] According to an embodiment of the present disclosure, the first input module 720 may include a first input unit and a first determination unit.

[0134] The first input unit is used to input the training sample pairs into the large model to be trained to obtain the fusion features of the sample image and the sample text, and the category features of the category description.

[0135] The first determination unit is used to determine the category information according to the input feature obtained based on the fusion feature and the category feature, wherein the input feature is obtained by taking the category feature as the first marker bit of multiple marker bits and taking the fusion feature as the remaining marker bits of the multiple marker bits except the first marker bit.

[0136] According to an embodiment of the present disclosure, the training sample set is determined according to the application scenario of the multimodal large model.

[0137] The multimodal large model training device 700 may further include a first determination module and a construction module.

[0138] The first determination module is used to determine at least one candidate category for the application scenario according to the frequency of the category descriptions of the multiple samples.

[0139] The construction module is used to construct a candidate category mapping according to the mapping relationship between the candidate categories and the category identifiers.

[0140] According to an embodiment of the present disclosure, the large model to be trained includes a large language model, and the category information includes output categories and output confidences.

[0141] According to an embodiment of the present disclosure, the first determining module may include a second input unit and a second determining unit.

[0142] The second input unit is used to input the input feature into the large language model to obtain the candidate output confidence of at least one category identifier.

[0143] The second determination unit is configured to determine, according to the candidate category mapping, a candidate category corresponding to a maximum confidence among at least one candidate output confidence as an output category, and to use the maximum confidence as the output confidence.

[0144] According to an embodiment of the present disclosure, the training module 730 may include a training unit and an adjustment unit.

[0145] The training unit is used to train the large model to be trained by using the respective category description and category information of each training sample to obtain an initial multimodal large model.

[0146] An adjustment unit is used to adjust the initial multimodal large model using a verification sample set until a predetermined performance condition is met to obtain a multimodal large model, wherein the verification sample set includes a plurality of sample pair sets, each sample pair set includes a plurality of verification sample pairs, and the plurality of verification sample pairs include at least two verification category information for the same verification image and the same verification text, and at least two verification category information are different verification confidences for the same verification category.

[0147] According to an embodiment of the present disclosure, the large model to be trained also includes an image feature extraction module and a text feature extraction module; the fused features are obtained by fusing image features and text features, and the image features and text features are obtained by projecting the initial image features obtained by processing the sample image using the image feature extraction module and the initial text features obtained by processing the sample text using the text feature extraction module into the same feature space.

[0148] According to an embodiment of the present disclosure, the training unit may include a determination subunit and a training subunit.

[0149] A determination subunit is used to determine a loss function value based on a first loss value determined based on a category description and an output category, and a second loss value determined based on an image feature and a text feature.

[0150] The training subunit is used to train the large model to be trained based on the loss function value to obtain an initial multimodal large model.

[0151] Based on the above image classification method, the present invention also provides an image classification device. Figure 8 The device is described in detail.

[0152] Figure 8 The block diagram of an image classification device according to an embodiment of the present disclosure is schematically shown.

[0153] like Figure 8 As shown, the image classification device 800 may include a second acquisition module 810 and a second input module 820 .

[0154] The second acquisition module 810 is used to acquire the image to be classified.

[0155] The second input module 820 is used to input the image to be classified into the multimodal large model to obtain an image classification result, wherein the multimodal large model is trained using a multimodal large model training device.

[0156] According to an embodiment of the present disclosure, the image to be classified belongs to the same application scenario as the training sample set used to train the multimodal large model, and the image classification result includes the confidence of the target category and at least one candidate category, and the target category belongs to at least one candidate category.

[0157] According to an embodiment of the present disclosure, the image classification device 800 may further include a second determination module.

[0158] The second determination module is used to use the confidence level corresponding to the target category in the at least one confidence level as the target confidence level.

[0159] Fig. 9 The structural block diagram of the intelligent agent of the large model according to the embodiment of the present disclosure is schematically shown.

[0160] In the embodiments of the present disclosure, inspired by the von Neumann structure in modern computer theory, such as Fig. 9 As shown, the AI ​​agent 900 may include five core modules: an input module 910 , a control module 920 , a storage module 930 , a calculation module 940 and an output module 950 .

[0161] The input module 910 is responsible for receiving or sensing information such as queries, requests, instructions, signals or data from the outside world (such as users or external environment), and converting them into a format that can be understood and processed by the AI ​​agent 900. The input module 910 is the primary link for the AI ​​agent 900 to interact with the outside world. It enables the AI ​​agent 900 to efficiently and accurately obtain the necessary "sensory" information from the outside world and respond to this information.

[0162] In an example, the input module 910 may input the image to be classified as described above.

[0163] The control module 920 is the core support for the AI ​​agent 900 to handle complex tasks. The control module 920 can execute the image classification method described above.

[0164] In the example, the control module 920 will continuously interact with the storage module 930, the computing module 940 and / or the output module 950 during operation. However, it should be noted that in the embodiment of the present disclosure, the control module 920 acts as a single initiator to initiate communication with the storage module 930, the computing module 940 and / or the output module 950, and there is no communication coupling between the storage module 930, the computing module 940 and the output module 950.

[0165] In the example, the performance of the control module 920 may be closely related to the large model on which the AI ​​agent 900 is based. In order to give full play to the capabilities of the large language model, the internal structure of the control module 920 may be designed to be highly configurable and extensible so as to cope with various types of tasks and requirements in real scenarios.

[0166] The storage module 930 may be responsible for memorizing candidate category mappings for different application scenarios and the multimodal large model applicable to the application scenario. The multimodal large model as described above may be included in the storage module 930 .

[0167] In the example, after the AI ​​agent 900 obtains the image to be classified, the AI ​​agent 900 may determine to process the image to be classified to obtain an image classification result. The image classification result may be stored in the storage module 930. The AI ​​agent 900 may obtain the image classification result from the storage module 930 and feed it back to the control module 920. Then, the control module 920 may pass the fed-back image classification result to the output module 950.

[0168] The operation module 940 may be viewed as a predefined tool library. The operation module 940 may include tools for determining the frequency of category descriptions, for mapping candidate categories, etc.

[0169] In the example, when the AI ​​agent 900 needs to determine the frequency of the category description, the relevant tool for determining the frequency of the category description can be called from the operation module 940 and fed back to the control module 920. Then, the control module 920 can use the tool for determining the frequency of the category description fed back to perform statistics on multiple sample pairs, obtain at least one candidate category, and pass the at least one candidate category to the output module 950. It can be understood that although the large language model has excellent language understanding and generation capabilities, it is the same as a human being, without the help of any tools, the tasks that can be solved are very limited. When the AI ​​agent 900 is given the ability to call tools, it can achieve tasks such as completing the frequency determination of category descriptions with the help of tools for determining the frequency of category descriptions.

[0170] The output module 950 can output the image classification results described above.

[0171] The AI ​​agent 900 according to the embodiment of the present disclosure can simply and effectively improve the level of intelligence, and improve flexibility and versatility.

[0172] Fig.10A block diagram of an electronic device suitable for implementing a training method and an image classification method for a multimodal large model according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0173] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0174] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0175] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the training method of the multimodal large model and the image classification method. For example, in some embodiments, the training method of the multimodal large model and the image classification method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the training method of the multimodal large model and the image classification method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute a training method or an image classification method for a multimodal large model in any other appropriate manner (eg, by means of firmware).

[0176] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0177] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0178] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0179] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0180] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0181] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0182] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0183] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for training a multimodal large model, comprising: Acquire a training sample set, wherein the training sample set includes a plurality of training sample pairs, each of the training sample pairs includes a sample image, a sample text, and a category description for the sample image; Input each of the training sample pairs into the large model to be trained to obtain the category information of each of the sample images; and The large model to be trained is trained using the respective category description and category information of each training sample to obtain a multimodal large model for image classification.

2. The method according to claim 1, wherein: Inputting each of the training sample pairs into the large model to be trained to obtain the category information of each sample image includes: Inputting the training sample pairs into the large model to be trained, obtaining fusion features of the sample image and the sample text, and category features of the category description; and The category information is determined according to an input feature obtained based on the fused feature and the category feature, wherein the input feature is obtained by taking the category feature as the first marker bit of a plurality of marker bits and taking the fused feature as the remaining marker bits of the plurality of marker bits except the first marker bit.

3. The method according to claim 2, wherein: The training sample set is determined according to the application scenario of the multimodal large model; The method further comprises: Determining at least one candidate category for the application scenario according to the frequency of the description of the category by the multiple samples; and A candidate category mapping is constructed according to the mapping relationship between the candidate categories and the category identifiers.

4. The method according to claim 3, wherein: The large model to be trained includes a large language model, and the category information includes an output category and an output confidence; The determining the category information according to the input feature obtained based on the fusion feature and the category feature includes: Inputting the input feature into the large language model to obtain a candidate output confidence score of each of the at least one category identifier; as well as According to the candidate category mapping, a candidate category corresponding to a maximum confidence among the at least one candidate output confidence is determined as the output category, and the maximum confidence is used as the output confidence.

5. The method according to any one of claims 1 to 4, wherein: The step of using the respective category description and category information of each training sample to train the large model to be trained to obtain a multimodal large model for image classification includes: Using the respective category descriptions and category information of each of the training samples, the large model to be trained is trained to obtain an initial multimodal large model; The initial multimodal large model is adjusted by using a verification sample set until a predetermined performance condition is met to obtain the multimodal large model, wherein the verification sample set includes a plurality of sample pair sets, each of the sample pair sets includes a plurality of verification sample pairs, the plurality of verification sample pairs include at least two verification category information for the same verification image and the same verification text, and the at least two verification category information are different verification confidences for the same verification category.

6. The method according to claim 5, wherein: The large model to be trained also includes an image feature extraction module and a text feature extraction module; The fusion feature is obtained by fusing the image feature and the text feature, and the image feature and the text feature are obtained by projecting the initial image feature obtained by processing the sample image using the image feature extraction module and the initial text feature obtained by processing the sample text using the text feature extraction module into the same feature space; The step of using the respective category description and category information of each training sample to train the large model to be trained to obtain the initial multimodal large model comprises: Determine a loss function value according to a first loss value determined based on the category description and the output category, and a second loss value determined based on the image feature and the text feature; as well as Based on the loss function value, the large model to be trained is trained to obtain the initial multimodal large model.

7. An image classification method, comprising: Get the image to be classified; as well as The image to be classified is input into a large multimodal model to obtain an image classification result, wherein the large multimodal model is trained using the method described in claims 1 to 6.

8. The method according to claim 7, wherein: The image to be classified and the training sample set used to train the multimodal large model belong to the same application scenario, the image classification result includes the confidence of the target category and at least one candidate category, and the target category belongs to the at least one candidate category; The method further comprises: The confidence level among the at least one confidence level that corresponds to the target category is used as the target confidence level.

9. A training device for a multimodal large model, comprising: A first acquisition module is used to acquire a training sample set, wherein the training sample set includes a plurality of training sample pairs, each of which includes a sample image, a sample text, and a category description for the sample image; A first input module, used to input each of the training sample pairs into the large model to be trained, to obtain the category information of each of the sample images; and The training module is used to train the large model to be trained by using the respective category description and category information of each training sample to obtain a multimodal large model for image classification.

10. The device according to claim 9, wherein: The first input module comprises: A first input unit is used to input the training sample pair into the large model to be trained to obtain the fusion features of the sample image and the sample text, and the category features of the category description; and A first determination unit is used to determine the category information according to an input feature obtained based on the fusion feature and the category feature, wherein the input feature is obtained by taking the category feature as the first marker bit of a plurality of marker bits and taking the fusion feature as the remaining marker bits of the plurality of marker bits except the first marker bit.

11. The device according to claim 10, wherein: The training sample set is determined according to the application scenario of the multimodal large model; The device also includes: A first determination module, configured to determine at least one candidate category for the application scenario according to the frequency of the description of the category by the multiple samples; and The construction module is used to construct a candidate category mapping according to the mapping relationship between the candidate categories and the category identifiers.

12. The device according to claim 11, wherein The large model to be trained includes a large language model, and the category information includes an output category and an output confidence; The first determining module comprises: A second input unit, used to input the input feature into the large language model to obtain a candidate output confidence of each of the at least one category identifier; as well as A second determination unit is configured to determine, according to the candidate category mapping, a candidate category corresponding to a maximum confidence among the at least one candidate output confidence as the output category, and to use the maximum confidence as the output confidence.

13. The device according to any one of claims 9 to 12, wherein: The training module includes: A training unit, used to train the large model to be trained by using the respective category description and category information of each training sample to obtain an initial multimodal large model; An adjustment unit is used to adjust the initial multimodal large model by using a verification sample set until a predetermined performance condition is met to obtain the multimodal large model, wherein the verification sample set includes a plurality of sample pair sets, each of the sample pair sets includes a plurality of verification sample pairs, the plurality of verification sample pairs include at least two verification category information for the same verification image and the same verification text, and the at least two verification category information are different verification confidences for the same verification category.

14. The device according to claim 13, wherein: The large model to be trained also includes an image feature extraction module and a text feature extraction module; The fusion feature is obtained by fusing the image feature and the text feature, and the image feature and the text feature are obtained by projecting the initial image feature obtained by processing the sample image using the image feature extraction module and the initial text feature obtained by processing the sample text using the text feature extraction module into the same feature space; The training unit comprises: a determining subunit, configured to determine a loss function value according to a first loss value determined based on the category description and the output category, and a second loss value determined based on the image feature and the text feature; and A training subunit is used to train the large model to be trained based on the loss function value to obtain the initial multimodal large model.

15. An image classification device, comprising: A second acquisition module is used to acquire the image to be classified; as well as The second input module is used to input the image to be classified into the multimodal large model to obtain an image classification result, wherein the multimodal large model is trained using the device described in claims 9 to 14.

16. The device according to claim 15, wherein: The image to be classified and the training sample set used to train the multimodal large model belong to the same application scenario, the image classification result includes the confidence of the target category and at least one candidate category, and the target category belongs to the at least one candidate category; The device also includes: The second determination module is used to use the confidence level corresponding to the target category in the at least one confidence level as the target confidence level.

17. An artificial intelligence agent, configured to execute the method according to any one of claims 1 to 8.

18. An electronic device, comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

19. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

20. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images