Training method of image classification model, image classification method and related device

Abstract

The present application provides a kind of training method of image classification model, image classification method and related device. Among them, the training method of image classification model can be applied to training device. Training device obtains training data set, and training data set includes multiple training images. Then, training device splices prompt vector and the mask vector of pre-training model for each training image, obtains the target input vector corresponding to each training image, the pre-training model is obtained by mask vector training here. Then, training device optimizes the prompt vector based on the pre-training model and the target input vector corresponding to each training image, obtains image classification model, the parameters of the image classification model include the parameters (for example, mask vector) of pre-training model and optimized prompt vector, and the image classification model is used to execute image classification task. Through the above method, the training efficiency of image classification model can be improved, and the training cost of image classification model is reduced.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to a training method for an image classification model, an image classification method, and related apparatus. Background Technology

[0002] In recent years, pre-trained models have been widely used in image classification tasks within the field of computer vision. Currently, optimizable cue vectors can be concatenated into the input of the first layer or each layer of a pre-trained model, and a corresponding classification head can be added to the model based on the image classification task to be performed. The image classification task can then be achieved by fine-tuning the cue vectors and classification head parameters. However, for different image classification tasks, the above method requires different learning strategies (e.g., learning rate, weight decay factor) to adjust the cue vectors and classification head parameters, resulting in a significant time commitment to searching for the optimal learning strategy, thus impacting the efficiency and cost of image classification tasks. Summary of the Invention

[0003] This application provides a training method for an image classification model, an image classification method, and related apparatus, which can improve the efficiency of image classification tasks and reduce the cost of image classification tasks.

[0004] Firstly, this application provides a method for training an image classification model. This method can be applied to a training device. The training device acquires a training dataset, which includes multiple training images. Then, the training device concatenates a cue vector and a mask vector of a pre-trained model for each training image to obtain a target input vector corresponding to each training image. Here, the pre-trained model is trained using the mask vector. Afterward, the training device optimizes the cue vector based on the pre-trained model and the target input vector corresponding to each training image to obtain an image classification model that can be used to perform image classification tasks. The image classification model includes the following parameters: the parameters of the pre-trained model (e.g., the aforementioned mask vector) and the optimized cue vector.

[0005] In the technical solution provided in this application, the training device trains an image classification model using mask prediction. Specifically, it concatenates a mask vector and a cue vector into the training image to obtain the target input vector, and then trains the image classification model by predicting the target input vector. Since the pre-trained model is also trained using mask prediction, training the image classification model using the method provided in this application can narrow the gap between upstream and downstream tasks, making the learning strategy required for implementing downstream tasks more stable and applicable to different downstream tasks. Correspondingly, when using the pre-trained model to implement different downstream tasks, the same learning strategy can be used, thereby enabling the pre-trained model to be efficiently applied to multiple downstream tasks.

[0006] In one possible implementation of the first aspect, the training device concatenates a cue vector and a mask vector of the pre-trained model for each training image, including: the training device concatenates the cue vector and the mask vector of the pre-trained model with each training image. Here, concatenation means concatenating along the row direction of the vectors.

[0007] It should be understood that in practical applications, training images are usually processed as row vectors for easier processing. Furthermore, cue vectors and mask vectors are also generally row vectors. In other words, since the training image, cue vector, and mask vector are all row vectors, concatenation is a more suitable method for concatenation.

[0008] In one possible implementation of the first aspect, the training device optimizes the prompt vector based on a pre-trained model and the target input vector corresponding to each training image, including: the training device uses the pre-trained model to predict the mask vector in the target input vector corresponding to each training image to obtain the predicted visual word corresponding to each training image, then maps the predicted visual word corresponding to each training image and the category label corresponding to each training image to the same vector space, and then optimizes the prompt vector based on the similarity between the mapped predicted visual word and the category label.

[0009] In the above implementation, the training device predicts the mask vector in the target input vector corresponding to the training image, thus training the image classification model using mask prediction. Furthermore, by mapping the predicted visual word corresponding to the training image and the category label corresponding to the image to the same vector space, the distance between them can be reduced, thereby more accurately determining the difference between the predicted visual word and the category label. This allows for more accurate adjustment of the cue vector, resulting in a more accurate image classification model.

[0010] In one possible implementation of the first aspect, the training device maps the predicted visual word corresponding to each training image and the category label corresponding to each training image to the same vector space, including: the training device sets a corresponding category vector for the category label corresponding to each training image, and then performs a dimensional transformation on the predicted visual word based on the dimension of the category vector. The transformed predicted visual word has the same dimension as the category vector, that is, they are in the same vector space.

[0011] Secondly, this application provides an image classification method. This method can be applied to an inference device. The inference device acquires an image to be processed, and then classifies the image based on an image classification model to obtain the classification result. The image classification model includes the following parameters: parameters of a pre-trained model and an optimized cue vector. The parameters of the pre-trained model include a mask vector. The pre-trained model is trained based on the mask vector. The optimized cue vector is obtained by optimizing the pre-trained model and the target input vectors corresponding to multiple training images. The target input vector corresponding to each training image is obtained by concatenating the aforementioned mask vector and the unoptimized cue vector for each training image.

[0012] In the technical solution provided in this application, the image classification model is trained using mask prediction. In other words, the image classification task achieved using the image classification model is accomplished through mask prediction. This narrows the gap between the upstream and downstream tasks, allowing for more efficient application of pre-trained knowledge from the upstream task to the downstream task, thus improving the accuracy of the classification results.

[0013] In one possible implementation of the second aspect, the target input vector corresponding to each training image is obtained by concatenating the mask vector of the pre-trained model and the unoptimized cue vector with each training image. Here, concatenation refers to splicing along the rows of the vectors.

[0014] It should be understood that in practical applications, training images are usually processed as row vectors for easier processing. Furthermore, cue vectors and mask vectors are also generally row vectors. In other words, since the training image, cue vector, and mask vector are all row vectors, concatenation is a more suitable method for concatenation.

[0015] In one possible implementation of the second aspect, the inference device classifies the image to be processed based on an image classification model, including: the inference device concatenates the mask vector of the pre-trained model and the optimized cue vector with the image to obtain the target input vector corresponding to the image, and then classifies the image based on the target input vector corresponding to the image.

[0016] In one possible implementation of the second aspect, the image classification model described above also includes the following parameters: at least one category vector, and a category label corresponding to each category vector.

[0017] In one possible implementation of the second aspect, the inference device classifies the image based on the target input vector corresponding to the image to be processed, including: the inference device determines the predicted visual word corresponding to the image based on the target input vector corresponding to the image to be processed, then performs a dimensional transformation on the predicted visual word corresponding to the image based on the dimension of the at least one category vector, and then classifies the image based on the similarity between the transformed predicted visual word and the at least one category vector.

[0018] In the above implementation, the transformed predicted visual word and at least one category vector reside in the same vector space. Vectors in the same vector space are easier to distance-calculate, thus more accurately determining which category vector the transformed predicted visual word is closer to. The category label corresponding to the closer category vector is the image classification result. Therefore, the above implementation can improve the accuracy of image classification results.

[0019] Thirdly, this application provides a training apparatus. The apparatus includes an acquisition module, a stitching module, and an optimization module. The acquisition module acquires a training dataset, which includes multiple training images. The stitching module stitches a cue vector and a mask vector of a pre-trained model for each training image in the training dataset to obtain a target input vector corresponding to each training image. Here, the pre-trained model is trained using the mask vector. The optimization module optimizes the cue vector based on the pre-trained model and the target input vector corresponding to each training image to obtain an image classification model that can be used to perform an image classification task. The image classification model includes the following parameters: the parameters of the pre-trained model (e.g., the mask vector mentioned above) and the optimized cue vector.

[0020] In one possible implementation of the third aspect, the aforementioned concatenation module is used to concatenate the cue vector and the mask vector of the pre-trained model with each training image. Here, concatenation refers to stitching along the rows of the vectors.

[0021] In one possible implementation of the third aspect, the optimization module is used to predict the mask vector in the target input vector corresponding to each training image using a pre-trained model to obtain the predicted visual word corresponding to each training image, map the predicted visual word corresponding to each training image and the category label corresponding to each training image to the same vector space, and optimize the prompt vector based on the similarity between the mapped predicted visual word and the category label.

[0022] In one possible implementation of the third aspect, the optimization module is used to set a corresponding category vector for the category label of each training image, and to perform a dimensional transformation on the predicted visual word based on the dimension of the category vector. The transformed predicted visual word has the same dimension as the category vector.

[0023] Fourthly, this application provides an inference apparatus. The apparatus includes an acquisition module and a classification module. The acquisition module acquires an image to be processed. The classification module classifies the image to be processed based on an image classification model to obtain a classification result. The image classification model includes the following parameters: parameters of a pre-trained model and an optimized cue vector. The parameters of the pre-trained model include a mask vector. The pre-trained model is trained based on the mask vector. The optimized cue vector is obtained by optimizing the pre-trained model and the target input vectors corresponding to multiple training images. The target input vector corresponding to each training image is obtained by concatenating the aforementioned mask vector and the unoptimized cue vector for each training image.

[0024] In one possible implementation of the fourth aspect, the target input vector corresponding to each training image is obtained by concatenating the mask vector of the pre-trained model and the unoptimized cue vector with each training image. Here, concatenation refers to splicing along the row direction of the vectors.

[0025] In one possible implementation of the fourth aspect, the image classification model described above also includes the following parameters: at least one category vector, and a category label corresponding to each category vector.

[0026] In one possible implementation of the fourth aspect, the classification module is used to concatenate the mask vector of the pre-trained model and the optimized cue vector with the image to obtain the target input vector corresponding to the image, and classify the image based on the target input vector corresponding to the image.

[0027] In one possible implementation of the fourth aspect, the classification module is used to determine the predicted visual word corresponding to the image based on the target input vector corresponding to the image to be processed, perform dimensional transformation on the predicted visual word corresponding to the image based on the dimension of the at least one category vector, and classify the image based on the similarity between the transformed predicted visual word and the at least one category vector.

[0028] Fifthly, this application provides a computing device including a processor and a memory, wherein the processor executes computer program code in the memory to implement some or all of the methods described in the first aspect and any implementation thereof, and / or some or all of the methods described in the second aspect and any implementation thereof.

[0029] Sixthly, this application provides a computing device cluster. The computing device cluster includes at least one computing device, each computing device including a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, causing the computing device cluster to perform some or all of the methods described in the first aspect and any implementation thereof, and / or some or all of the methods described in the second aspect and any implementation thereof.

[0030] In a seventh aspect, this application provides a computer program product. The computer program product may be a software or program product containing instructions that can run on a computing device or be stored on any available medium. When the computer program product runs on a computing device or a cluster of computing devices, it causes the computing device or the cluster of computing devices to perform some or all of the methods described in the first aspect and any implementation thereof, and / or some or all of the methods described in the second aspect and any implementation thereof.

[0031] Eighthly, this application provides a computer-readable storage medium. The computer storage medium includes computer program instructions that, when executed by a computing device or a cluster of computing devices, cause the computing device or the cluster of computing devices to perform some or all of the methods described in the first aspect and any implementation thereof, and / or some or all of the methods described in the second aspect and any implementation thereof. Attached Figure Description

[0032] Figure 1 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0033] Figure 2 This is a flowchart illustrating a training method for an image classification model provided in an embodiment of this application;

[0034] Figure 3 This is a schematic diagram illustrating another application scenario provided by an embodiment of this application;

[0035] Figure 4 This is a flowchart illustrating an image classification method provided in an embodiment of this application;

[0036] Figure 5 This is a schematic diagram of an optimal learning strategy provided in an embodiment of this application;

[0037] Figure 6 This is a schematic diagram illustrating different optimal learning strategies on different datasets, provided in an embodiment of this application.

[0038] Figure 7This is a schematic diagram illustrating the test results of a different image classification model provided in an embodiment of this application;

[0039] Figure 8 This is a visual schematic diagram of a classification result provided in an embodiment of this application;

[0040] Figure 9 This is a schematic diagram of test results when using different mapping rules, provided in an embodiment of this application;

[0041] Figure 10 This is a schematic diagram of the structure of a training device provided in an embodiment of this application;

[0042] Figure 11 This is a schematic diagram of the structure of a reasoning device provided in an embodiment of this application;

[0043] Figure 12 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0044] Figure 13 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application. Detailed Implementation

[0045] To facilitate understanding of the technical solutions provided in this application, some of the terms involved in this application will be explained first.

[0046] Pre-trained model: refers to a model with visual representation capabilities that is trained on a large-scale dataset. This process is also known as the upstream task.

[0047] Masked visual token modeling (MVTM) is an implementation method for upstream tasks. It involves using mask vectors to randomly mask portions of the input image of a model (e.g., a Transformer architecture model), and then predicting the visual tokens of the masked image patches through self-supervised learning. This allows the model to learn visual representation capabilities, resulting in a pre-trained model with visual representation abilities.

[0048] Visual words: These refer to the visual features of an image, used to express the content of the image.

[0049] Self-supervised learning, also known as self-supervised training, refers to the process of automatically generating labels from unlabeled data in a given training dataset, and then determining the model's parameters based on these generated labels. For example, in mask prediction, the pre-trained model uses a visual vocabulary to encode each image patch of the input image, obtaining visual word labels corresponding to each patch. Then, the pre-trained model adjusts its parameters based on the difference between the visual word labels corresponding to the masked image patches and the predicted visual words for the masked image patches. It can be inferred that the semantic level of the visual vocabulary determines the level of visual representation ability the pre-trained model can possess. For example, if the visual vocabulary is low-level information without semantic information, the pre-trained model tends to learn the representation ability of low-level visual information. If the visual vocabulary is high-level information rich in semantic information, the pre-trained model tends to learn the representation ability of high-level semantic information. Therefore, the pre-trained model involved in this application embodiment is trained using a visual vocabulary with high-level semantic information, thus possessing higher visual representation ability.

[0050] Pre-trained models can be applied to downstream tasks related to upstream tasks. Downstream tasks refer to tasks in the field of computer vision implemented using pre-trained models; in this application's embodiments, downstream tasks include image classification tasks. For example, if a pre-trained model is a model trained on a large number of surveillance videos (including surveillance videos from multiple locations such as traffic hubs, streets, and commercial areas) that can detect specific targets (including vehicles, pedestrians, animals, etc.) in images, then this model can be used to detect vehicles passing through a certain intersection based on surveillance video of that intersection.

[0051] When using a pre-trained model to implement a downstream task, the pre-trained model needs to be trained on the training dataset of the downstream task. Compared to the large-scale dataset used in the upstream task, the training dataset used in the downstream task is smaller. Therefore, when implementing the downstream task, the pre-trained model is usually trained in a supervised manner. Supervised training (also known as supervised learning) refers to the process of determining the model's parameters based on the data in the given training dataset and the labels corresponding to each data point in the training dataset. The labels corresponding to the data in the training dataset are usually manually labeled to identify the correct answer for that data on a specific task.

[0052] During the model training phase, to improve training efficiency and post-training model performance, it's necessary to set appropriate hyperparameters (also known as learning strategies). Hyperparameters are parameters that cannot be obtained through learning from training data or that cannot be changed by training data; they are a concept relative to the parameters in the model. Model hyperparameters are typically set manually based on experience or experiments. Examples of hyperparameters include learning rate, weight decay, momentum, learning rate adjustment strategy, total epochs, and warmup epochs.

[0053] Prompt learning refers to reducing the difference between upstream and downstream tasks by designing downstream tasks to conform to the form of upstream tasks, thereby better applying pre-trained knowledge to downstream tasks, especially in scenarios with few or no samples.

[0054] Hint vectors: These are additional, optimizable parameters added to assist the pre-trained model in achieving downstream tasks.

[0055] In the field of computer vision, upstream tasks can be implemented in various ways, such as mask prediction (mentioned earlier), supervised discriminative prediction, or contrastive prediction. Similarly, downstream tasks also vary, including image classification, object detection, and image segmentation. This makes it difficult to achieve formal consistency between upstream and downstream tasks in computer vision. On the other hand, considering the large scale and numerous parameters of pre-trained models, the current primary approach is to fine-tune them using visual prompt tuning (VPT) to achieve downstream tasks (specifically image classification). Specifically, the pre-trained model is specially designed for the image classification task, for example, by concatenating learnable prompt vectors to the inputs of the first or subsequent layers of the pre-trained model, and adding a corresponding classification head (e.g., a multi-layer perceptron, MLP) based on the image classification task. Then, the designed model is trained using the dataset corresponding to the image classification task, resulting in an image classification model that can be directly used to implement the image classification task. The methods described above only require fine-tuning the cue vectors and classifier parameters based on the dataset corresponding to the image classification task. The numerous parameters included in the pre-trained model remain unchanged, allowing for the training of an image classification model. However, research and testing have revealed that different learning strategies are often needed to adjust the cue vectors and classifier parameters for different image classification tasks. In other words, these methods require a significant investment of time searching for the optimal learning strategy to obtain a suitable image classification model. This impacts the efficiency of image classification tasks and increases their cost.

[0056] For example, in security scenarios, image classification tasks may include detecting pedestrians, vehicles, and animals in surveillance videos. For these three image classification tasks, if the existing methods are used, an optimal learning strategy needs to be determined for each task. However, the optimal learning strategy differs between tasks, and the optimal learning strategy for one task often leads to extremely poor performance for another. In other words, the existing methods require determining three different optimal learning strategies to achieve the desired image classification results. Therefore, how to efficiently utilize pre-trained models to achieve downstream tasks is a pressing issue that needs to be addressed.

[0057] To address the aforementioned issues, this application provides a method for training an image classification model. This method, based on the idea of ​​cue learning, utilizes a pre-trained model to design the image classification task as a mask prediction task, thereby training the image classification model. Because this method achieves consistency in the form of the downstream task (i.e., the image classification task) and the upstream task (i.e., the training task of the pre-trained model) during image classification model training, the learning strategy obtained through this method has good stability and can be uniformly deployed across different image classification tasks. Furthermore, when the downstream and upstream tasks are consistent in form, the pre-trained knowledge can be fully applied to the downstream task. Therefore, the image classification model trained using the method provided in this application has high accuracy in image classification.

[0058] The image classification model training method provided in this application embodiment can be executed by a training device. The training device can be a software device, a hardware device, or a combination of both. When the training device is a software device, it can be a virtual machine (VM), a container, or other software with model training capabilities.

[0059] The deployment of training devices is flexible. For example, training devices can be deployed independently in a cloud-based data center, which includes a large amount of basic resources (including computing resources, storage resources, and network resources). The computing resources in a cloud-based data center can be a large number of computing devices (e.g., servers). In some embodiments, the training device can be deployed independently on servers or virtual machines in a cloud-based data center, or distributed across multiple servers or virtual machines in a cloud-based data center, or distributed across servers and virtual machines in a cloud-based data center. Figure 1 As shown, the training device is deployed by a cloud service provider on a data center in a cloud environment. Accordingly, the cloud service provider offers the functionality of the training device as a cloud service to tenants. Tenants can register an account on the cloud management platform, log in with that account, and purchase the cloud service (either pre-paid and settled based on resource usage, or purchased on an annual or monthly basis). After successful purchase, tenants can upload the training dataset corresponding to the image classification task to be implemented to the training device deployed on the data center. The training device receives the uploaded training dataset and performs image classification model training based on it. The trained image classification model can be used online or downloaded by tenants (e.g., tenants can download the parameters of the trained image classification model) to complete specific image classification tasks.

[0060] For example, the training device can also be deployed on computing devices in an edge environment (also known as edge computing devices). An edge environment is an environment geographically close to the user's terminal computing device, including edge computing devices such as edge servers and edge stations with computing capabilities. Alternatively, the training device can also be deployed on terminal computing devices, including terminal servers, smartphones, laptops, tablets, and personal desktop computers. Furthermore, the training device can be distributed and deployed in different environments (including any two or three of cloud data centers, edge computing devices, or terminal computing devices). For instance, the training device can be logically divided into multiple parts, each with different functions. One part of the training device can be deployed on computing devices in a cloud environment, and another part can be deployed on computing devices in an edge environment. The various parts of the training device deployed in different environments work together to provide the function of training the image classification model. It should be understood that the embodiments of this application do not limit the division of the various parts of the training device, nor do they limit the deployment method of the various parts of the training device. In actual applications, it can be adaptively deployed according to specific application requirements or the computing capabilities of the hardware devices.

[0061] The following is combined Figure 2 The training method of the image classification model provided in the embodiments of this application is described in detail.

[0062] S101: The training device acquires the training dataset.

[0063] The training dataset is constructed specifically for the image classification task to be implemented. It includes multiple training images and a category label for each image. The category label for each training image serves as the target for training the model (i.e., the image classification model) that can be used to achieve the image classification task. For example, to train a model that can identify different vehicles, the training dataset could include images of various vehicles, each labeled to indicate the type of vehicle it contains, such as sedan, bus, etc. In this example, the type of vehicle in each image is its category label. Similarly, to train a model that can identify different animals, the training dataset could include images of various animals, each labeled to indicate the type of animal it contains, such as cat, dog, etc. In this example, the type of animal in each image is its category label.

[0064] Optionally, the training images in the training dataset can be preprocessed images. Preprocessing includes one or more of the following operations: scaling or normalizing the image size; image verification to remove individual images that seriously affect the model training effect; and verification of the class labels corresponding to the images to remove or correct individual images whose class labels do not match the images.

[0065] In some embodiments, the training device provides an interface, such as a graphical user interface (GUI), an application programming interface (API), or a command-line interface (CLI). Users or other devices storing training datasets can upload the training dataset to the training device by calling this interface, and the training device then retrieves the training dataset.

[0066] S102: The training device processes each training image in the training dataset to obtain the target input vector corresponding to each training image.

[0067] Specifically, the training device concatenates the cue vector and the mask vector of the pre-trained model for each training image to obtain the target input vector corresponding to each training image.

[0068] The mask vector of a pre-trained model is the vector (or parameter) used in the pre-trained model to mask the training image. The pre-trained model can be a Transformer-based model, and the mask vector can be a vector used during the mask prediction phase of the pre-trained model (see the terminology explanation section above) to randomly mask portions of the input image. The cue vector refers to an additional, optimizable vector (or parameter); "optimizable" means that the vector can be dynamically adjusted during model training. In some embodiments, the cue vector can be set according to the categories in the image classification task to be implemented, or it can be other vectors used to assist the pre-trained model in achieving the image classification task, or it can be, for example, an initialization vector, a vector randomly sampled in the vector space, or a vector obtained by linearly transforming a randomly sampled vector.

[0069] The following describes how the training device obtains the target input vector corresponding to each training image in this step, taking any one of multiple training images (let's say training image x) as an example. Specifically, it can include the following steps:

[0070] 1. The training device divides the training image x into blocks to obtain multiple image blocks.

[0071] 2. The training device encodes each image block in the multiple image blocks to obtain the image block encoding sequence corresponding to the training image x.

[0072] In some embodiments, the training device can encode any one of a plurality of image patches using the following formula to obtain the encoding vector corresponding to that image patch. The image patch encoding sequence corresponding to the training image x includes the encoding vectors corresponding to multiple image patches.

[0073] e i =Embed pre (x i )

[0074] In this formula, x i Let represent any image patch of the training image x, 1 ≤ i ≤ n, where i and n are both positive integers, and n represents the number of image patches obtained after dividing the training image x into blocks. i x represents i The corresponding encoding vector. Embed pre () represents x i With e i The mapping relationship between them. Furthermore, it can be concluded that the image block coding sequence is [e1,…,e...]. n ].

[0075] 3. The training device concatenates the comprehensive classification vector of the pre-trained model with the image block encoding sequence corresponding to the training image x to obtain the original input vector corresponding to the training image x.

[0076] In some embodiments, the training device can use the following formula to combine the comprehensive classification vector of the pre-trained model with the training image x. i The corresponding image block encoding sequences are concatenated to obtain the original input vector corresponding to the training image x.

[0077] H o =[e [CLs] ,e1,…,e n ]

[0078] In this formula, H o e represents the original input vector corresponding to the training image x. [CLS] This represents the comprehensive classification vector (class token) of the pre-trained model, e [CLS] and image block encoding sequence (i.e., [e1,...,e) n Any encoded vector in ]) (e.g., e iThe number of rows is the same. This formula indicates that the original input vector corresponding to the training image x is obtained by concatenating the comprehensive classification vector of the pre-trained model before the image patch encoding sequence corresponding to the training image x. It should be understood that the implementation method represented by this formula is only an example. In practical applications, the training device can concatenate the comprehensive classification vector of the pre-trained model with the image patch encoding sequence corresponding to the training image x according to the specific situation. The method of concatenation (e.g., concatenation or parallel concatenation) and position are not limited in this application embodiment. For example, the comprehensive classification vector of the pre-trained model can be concatenated after the image patch encoding sequence corresponding to the training image x, that is, H o =[e1,…,e n ,e [CLS] Alternatively, the comprehensive classification vector of the pre-trained model can be appended to the image patch encoding sequence corresponding to the training image x, i.e., H o =[e [CLS] ;e1;…;e n In this example, the image block encoding sequence corresponding to the training image x can be [e1; ...; e n ], e [CLS] and any encoded vector in the image block encoding sequence (e.g., e i The number of columns is the same.

[0079] It is worth noting that, in the embodiments of this application, "serialization" refers to splicing along the direction of the rows of the vectors, while "parallelization" refers to splicing along the direction of the columns of the vectors.

[0080] 4. The training device concatenates the cue vector and the mask vector of the pre-trained model with the original input vector corresponding to the training image x to obtain the target input vector corresponding to the training image x.

[0081] In some embodiments, the training device can concatenate the cue vector and the mask vector of the pre-trained model with the original input vector corresponding to the training image x using the following formula to obtain the target input vector corresponding to the training image x.

[0082] H vp =[H o ,p1,…,p m ,e [MASK] ]

[0083] In this formula, H vp Let p1, ..., p be the target input vector corresponding to the training image x. m Represents the cue vector (m is a positive integer greater than 0), e [MASK] H represents the mask vector of the pre-trained model. o p1, ..., p m e [MASK]The number of rows is the same. This formula indicates that the target input vector corresponding to the training image x is obtained by concatenating the cue vector and the mask vector of the pre-trained model to the original input vector corresponding to the training image x. It should be understood that the implementation method represented by this formula is only an example. In practical applications, the training device can concatenate the cue vector and the mask vector of the pre-trained model with the original input vector corresponding to the training image x according to the specific situation. The method of concatenation (e.g., concatenation or parallel concatenation) and position are not limited in this application embodiment. For example, the cue vector and the mask vector of the pre-trained model can be concatenated before the original input vector corresponding to the training image x, that is, H vp =[p1,…,p m ,e [MASK] H o Alternatively, the cue vector can be concatenated before the original input vector corresponding to the training image x, and the mask vector of the pre-trained model can be concatenated after the original input vector corresponding to the training image x, i.e., H. vp =[p1,…,p m H o ,e [MASK] Alternatively, the cue vector can be concatenated to the original input vector corresponding to the training image x, and the mask vector of the pre-trained model can be concatenated between the original input vector corresponding to the training image x, such as H. vp =[e [CLs] ,e [MASK] ,e1,…,e n ,p1,…,p m ], or H vp =[e [CLs] ,e1,…,e [MASK] ,…,e n ,p1,…,p m Alternatively, the cue vector and the mask vector of the pre-trained model can be concatenated to the original input vector corresponding to the training image x, i.e., H. vp =[H o ;p1;...;p m ;e [MASK] In this example, the original input vector corresponding to the training image x can be H. o =[e [CLS] ;e1;…;e n ], H o p1, ..., p m e [MASK] The number of columns is the same.

[0084] It should be understood that, in order to preserve more features of the training images, the training device uses the aforementioned concatenation method, that is, concatenating the mask vector with the training image, rather than directly masking a portion of the image patch in the training image using the mask vector. This allows the model to better learn the complete features of the training image during subsequent training, resulting in a more accurate image classification model.

[0085] S103: The training device optimizes the cue vector based on the pre-trained model and the target input vector corresponding to each training image to obtain an image classification model.

[0086] Specifically, the training device uses a pre-trained model to predict the mask vector in the target input vector corresponding to each training image, thereby obtaining the predicted visual word corresponding to each training image. Then, based on the predicted visual word and the category label corresponding to each training image, the cue vector is optimized to obtain the image classification model.

[0087] In some embodiments, the training device optimizes the cue vector based on the predicted visual word and its corresponding category label for each training image. This includes mapping the predicted visual word and the category label for each training image to the same vector space, and then optimizing the cue vector based on the similarity between the mapped predicted visual word and the category label. It should be understood that the predicted visual word for the training image has no explicit semantic information. The mapping rule described above (mapping the predicted visual word and the category label for the training image to the same vector space) can shorten the distance between the predicted visual word and the category label, allowing the predicted visual word to be better supervised by the category label, thereby improving the accuracy of the classification results.

[0088] The following description, still using the training image x as an example, details this step, which may include the following steps:

[0089] 1. The training device sets a category vector for the category label corresponding to the training image x.

[0090] The training device does not impose specific restrictions on the class vectors. For example, the class vectors can be row vectors or column vectors; they can be 1*128 dimensional vectors or 1*256 dimensional vectors; and the data in the class vectors can be randomly set.

[0091] 2. The training device predicts the mask vector in the target input vector corresponding to the training image x to obtain the predicted visual word.

[0092] As shown in S102 above, the target input vector corresponding to the training image x can be divided into two parts: a masked part (i.e., a mask vector) and a non-masked part (including the cue vector and the original input vector). Therefore, in this step, the training device predicts the mask vector in the target input vector corresponding to the training image x, including: the training device predicts the masked part in the target input vector corresponding to the training image x based on the non-masked part in the target input vector corresponding to the training image x.

[0093] 3. The training device maps the predicted visual words to the vector space containing the category vectors corresponding to the category labels of the training image x.

[0094] Specifically, the training device performs a dimensional transformation on the predicted visual words based on the dimension of the category vector corresponding to the category label of the training image x. The transformed predicted visual words have the same dimension as the category vector, that is, they are in the same vector space.

[0095] In some embodiments, the training device can perform dimensional transformation on the predicted visual words using the following formula.

[0096]

[0097] In this formula, h [MASK] The above-mentioned predicted visual word, u [MASK] This refers to the transformed predicted visual word. h [MASK] The vector space in which it is located Indicate u [MASK] The vector space in which it resides. In one possible implementation, A is a mapping matrix used to map h. [MASK] Perform a dimensionality transformation to map it to the vector space containing the class vectors corresponding to the class labels of the training image x. For example, h [MASK] If A is a 1*768 dimensional vector and the category vector corresponding to the category label of the training image x is a 1*128 dimensional vector, then A can be a 768*128 dimensional matrix. Correspondingly, the transformed predicted visual word is a 1*128 dimensional vector, that is, the transformed predicted visual word and the category vector corresponding to the category label of the training image x are in the same vector space.

[0098] 4. The training device determines the similarity between the category vector corresponding to the category label of the training image x and the predicted visual word after the above transformation.

[0099] In some embodiments, the training device can determine the similarity between the category vector corresponding to the category label of the training image x and the predicted visual word after the above transformation using the following formula.

[0100] sim j =u [MASK] ·(c j ) T

[0101] In this formula, c j This represents the class vector corresponding to the class label of the training image x, (c j ) T c j The transpose vector, sim j This represents the similarity (specifically, cosine similarity) between the category vector corresponding to the category label of the training image x and the transformed predicted visual word. It should be understood that this formula is merely an example; in practical applications, the training device can also determine the similarity between the category vector corresponding to the category label of the training image x and the transformed predicted visual word by calculating the Euclidean distance, Manhattan distance, or Hamming distance, etc.

[0102] 5. The training device optimizes the cue vector in the target input vector corresponding to the training image x based on the similarity between the category vector corresponding to the category label of the training image x and the predicted visual word after the above transformation.

[0103] Specifically, the training device calculates the loss value between the predicted visual word and the category label corresponding to the training image x based on the similarity between the category vector corresponding to the category label of the training image x and the transformed predicted visual word, and then adjusts the cue vector in the target input vector corresponding to the training image x based on the loss value.

[0104] In some embodiments, the training device can calculate the loss value of the predicted visual word and the category label corresponding to the training image x using the loss function shown in the following formula.

[0105]

[0106] In this formula, N C N represents the number of categories, that is, the number of category labels corresponding to the training images in the training dataset. C N C c is a positive integer greater than 0. k This represents the category vector corresponding to any category label, where k is a positive integer, 1≤k≤N. C The loss function shown in the formula is the normalized exponential function (Softmax function). It should be understood that in practical applications, the training device can also use other loss functions (e.g., the exponential loss function) to calculate the loss value of the predicted visual word and the category label corresponding to the training image x.

[0107] The training device can iteratively adjust the cue vector using each training image in the training dataset through steps 1-5 above, until the image classification model can output the same output value as the category label corresponding to the training image with high accuracy based on the input training image. At this point, the image classification model can be used to perform image classification tasks and has high accuracy.

[0108] It should be understood that in the training process of the above image classification model, the training device designs the image classification task (i.e., the downstream task) to have the same form as the upstream task. The reason is as follows: As mentioned above, the upstream task is the training task of the pre-trained model. The pre-trained model is obtained through mask prediction; that is, a mask vector is used to randomly mask a portion of the input image, and then the visual words of the masked image blocks are predicted using the remaining image blocks in the input image (i.e., the unmasked image blocks). Then, based on the visual word labels of the masked image blocks and the predicted visual words, the parameters of the pre-trained model are continuously adjusted to obtain the pre-trained model. This process is consistent with the training process of the image classification model described in the above embodiments. Therefore, in this embodiment, the training device trains the image classification model by designing the image classification task to have the same form as the upstream task. This also reduces the difference between the image classification task and the upstream task, making the learning strategy used during training more stable and applicable to training image classification models for different image classification tasks. In addition, it can make full use of the pre-training knowledge obtained from the upstream task, so that the trained image classification model can accurately extract the low-level information that can represent the category of the training image (i.e. the predicted visual words corresponding to the training image), thereby improving the accuracy of image classification.

[0109] Furthermore, as can be seen from S103 above, in the process of training an image classification model using the method provided in this application embodiment, in addition to focusing on how to accurately extract the underlying information that can characterize the category of the training image (i.e., the predicted visual words corresponding to the training image), it also focuses on how to use the extracted underlying information for classification (i.e., by mapping the predicted visual words corresponding to the training image and the category label corresponding to the image to the same vector space to determine the classification result of the image). Since the mapped predicted visual words and category labels are in the same vector space, the classification result of the image can be determined by calculating the distance (i.e., similarity) between them, and then the cue vector can be adjusted in reverse. Compared with the prior art of directly mapping predicted visual words to category labels by adding a classification head (e.g., MLP) to achieve classification, the process of training an image classification model using the method provided in this application embodiment does not require adding an additional design (i.e., a classification head), and therefore does not require training classification head parameters. Moreover, because the distance between the predicted visual words and category labels is shortened during classification, the classification result has higher accuracy.

[0110] The image classification model obtained by performing S101-S103 above can be used to perform corresponding image classification tasks. Therefore, this application embodiment also provides an image classification method, which uses the image classification model trained above to classify the image to be processed, thereby obtaining the classification result of the image.

[0111] The image classification model training method provided in this application embodiment can be executed by an inference device. The inference device can be a software device, a hardware device, or a combination of both. When the inference device is a software device, it can be a virtual machine, a container, or software with model inference capabilities.

[0112] The deployment of inference devices is flexible. For example, inference devices can be deployed independently on a data center in a cloud environment. In some embodiments, inference devices can be deployed independently on servers or virtual machines in a cloud environment data center, or distributed across multiple servers or virtual machines in a cloud environment data center, or distributed across servers and virtual machines in a cloud environment data center. Figure 3As shown, the inference device is deployed by a cloud service provider in a data center within a cloud environment. Correspondingly, the cloud service provider offers the functionality of the inference device as a cloud service to tenants. Tenants can register an account on the cloud management platform, log in with that account, and purchase the cloud service (either pre-paid and settled based on final resource usage, or purchased on an annual or monthly basis). After successful purchase, tenants can upload images to be processed to the inference device deployed in the data center. The inference device receives the uploaded images, inputs them into an image classification model, and returns the classification result of the image to be processed to the tenant. Tenants can then use this classification result to complete specific tasks, such as prevention and early warning systems.

[0113] In other embodiments, the inference device and the training device described above can both be deployed on a data center in a cloud environment. Tenants can initially purchase only the cloud services corresponding to the training device on a cloud management platform to obtain an image classification model. When it is necessary to use the image classification model to classify images, the tenant then purchases the cloud services corresponding to the inference device to complete the image classification. Alternatively, the cloud service provider can combine the training device and the inference device into a single cloud service offered to the tenant. In this case, after purchasing this cloud service, the tenant can both train the image classification model and perform image classification on the data center in the cloud environment.

[0114] Besides data centers in cloud environments, inference devices can also be deployed on edge computing devices, terminal computing devices, or distributed across different environments (including any two or three of cloud data centers, edge computing devices, or terminal computing devices). When inference devices are distributed across different environments, they can be logically divided into multiple parts, each with different functions. One part of the inference device can be deployed on a computing device in a cloud environment, and another part can be deployed on a computing device in an edge environment. The various parts of the inference device deployed in different environments work together to provide image classification functionality. It should be understood that the embodiments of this application do not limit the division of the various parts of the inference device, nor do they limit the deployment method of the various parts of the inference device. In practical applications, they can be adaptively deployed according to specific application requirements or the computing capabilities of the hardware devices.

[0115] The following is combined Figure 4 The image classification method provided in the embodiments of this application will be described in detail.

[0116] S201: The inference device acquires the image to be processed.

[0117] Optionally, the image to be processed can be a pre-processed image, for example, an image that has been scaled or normalized according to image size specifications.

[0118] In some embodiments, the inference device provides an interface, such as a GUI, API, CLI, etc. A user or other device storing an image to be processed can upload the image to the inference device by calling this interface, and the inference device then retrieves the image.

[0119] S202: The inference device classifies the image to be processed based on the image classification model and obtains the classification result of the image.

[0120] Specifically, the inference device inputs the image to be processed into the image classification model. Correspondingly, the image classification model acquires the image to be processed and processes it to obtain the classification result. The image classification model here is the one trained through S101-S103 described above, and it includes the following parameters: parameters of the pre-trained model (e.g., mask vector) and optimized cue vectors. As mentioned earlier, the pre-trained model is trained based on the mask vector. The optimized cue vectors are obtained by optimizing the target input vectors corresponding to multiple training images (see S103 above for details). The target input vector corresponding to each training image is obtained by concatenating the mask vector of the pre-trained model and the unoptimized cue vector for each training image (see S102 above for details).

[0121] In some embodiments, the inference device classifies the image to be processed based on an image classification model, including: concatenating the mask vector of a pre-trained model and the optimized cue vector with the image to be processed to obtain a target input vector corresponding to the image to be processed. Then, the inference device classifies the image based on the target input vector corresponding to the image to be processed.

[0122] In some embodiments, the inference device classifies the image based on the target input vector corresponding to the image to be processed, including: the inference device determines the predicted visual word corresponding to the image based on the target input vector corresponding to the image to be processed, then maps the predicted visual word corresponding to the image and the category label to the same vector space, and classifies the image based on the mapped predicted visual word and category label.

[0123] In some embodiments, the inference device maps the predicted visual words and category labels corresponding to the image to be processed to the same vector space, and classifies the image based on the mapped predicted visual words and category labels. This includes: the inference device performing a dimensionality transformation on the predicted visual words corresponding to the image based on the dimension of at least one category vector, and then classifying the image based on the similarity between the transformed predicted visual words and the at least one category vector. The at least one category vector is also a parameter of the image classification model, and each category vector corresponds to a category label (see step 1 in S103 above for details).

[0124] In some embodiments, the inference device classifies the image based on the similarity between the transformed predicted visual word and at least one category vector, including: the inference device determining the similarity between the transformed predicted visual word and each category vector, and the category label corresponding to the maximum similarity is the classification result of the image to be processed.

[0125] For simplicity, this step does not describe how the inference device concatenates the mask vector of the pre-trained model and the optimized cue vector with the image to be processed. It should be understood that the specific concatenation method depends on the training process of the image classification model, as detailed in S102 above. Similarly, this step does not describe how the inference device performs dimensional transformation on the predicted visual words based on the dimension of at least one category vector, or how it calculates the similarity between the two. It should be understood that the specific implementation depends on the training process of the image classification model, as detailed in steps 3-4 of S103 above.

[0126] The inference device can perform the image classification task by executing S201-S202 as described above. Since the image classification model used in performing the image classification task is trained by mask prediction, the image classification task and the upstream task achieve formal consistency. Therefore, the pre-trained knowledge learned from the upstream task can be used more fully in the image classification process, and correspondingly, the classification result obtained has higher accuracy.

[0127] The following is combined Figures 5-9 The image classification model trained using the method provided in the embodiments of this application and its performance in actual use are further described using specific experimental data.

[0128] Experiments were conducted using the following 10 publicly available datasets: CIFAR100, CIFAR10, Flowers102, Food101, EuroSAT, SVHN, Pets, DTD, Resisc45, and PatchCame. Each dataset included both training and testing datasets. The image classification model was trained using the training dataset, and tested using the testing dataset. To facilitate understanding of the performance of the image classification model trained using the method provided in this application, the following seven baseline methods were also selected for comparison:

[0129] 1. The pre-trained model is BEITv2. The image classification model is obtained by fine-tuning BEITv2 using the training dataset. This method will be referred to as FT+BEITv2 in the following text, where FT means fine-tuning.

[0130] 2. The pre-trained model is BEITv2. Features are extracted from the training dataset using BEITv2, and then logistic regression is used to fit the class labels. This method will be referred to as LP+BEITv2 in the following text, where LP stands for linear probe.

[0131] 3. The pre-trained model is CLIP. The image classification model is obtained by fine-tuning CLIP using the training dataset. This method will be referred to as FT+CLIP in the following text.

[0132] 4. The pre-trained model is CLIP. CLIP is used to extract features from the training dataset, and then logistic regression is used to fit the class labels. This method will be referred to as LP+CLIP in the following text.

[0133] 5. The pre-trained model is CLIP. The image classification model is obtained by training CLIP with the images in the training dataset and the corresponding text of the images. This method will be referred to as TP+CLIP in the following text, where TP stands for textual prompt.

[0134] 6. The pre-trained model is CLIP, which adds learnable perturbations to the images in the training dataset as visual cues. The image classification model is trained using the images with added visual cues and the corresponding text. This method will be referred to as VP+TP+CLIP in the following text, where VP stands for visual prompt.

[0135] 7. The pre-trained model is BEITv2. VPT is used to fine-tune BEITv2 to obtain an image classification model. This method will be referred to as VPT+BEITv2 in the following text.

[0136] Please see first. Figure 5 ,like Figure 5 As shown, when using the training method of the image classification model provided in this application embodiment, the following learning strategy can be used on the above 10 datasets: the optimizer is AdamW, total epochs = 50, warmup epochs = 5, learning rate = 0.001, weight decay = 0.01, momentum = 0.9, and the scheduler is a cosine strategy. That is to say, the training method of the image classification model provided in this application embodiment supports the use of a unified learning strategy for different image classification tasks. Conversely, the baseline method VPT+BEITv2 listed above requires searching for different learning strategies for different image classification tasks. Taking the Flowers102 and Pets datasets as examples, ... Figure 6 As shown, (a) and (b) illustrate the loss (shown as Loss) and accuracy (shown as Accuracy) values ​​corresponding to training the model on Flowers102 using different learning strategies (epoch and learning rate, shown as lr) in the figure. (c) and (d) illustrate the loss (shown as Loss) and accuracy (shown as Accuracy) values ​​corresponding to training the model on Pets using different learning strategies (epoch and learning rate, shown as lr) in the figure. Figure 6 As can be seen, when training an image classification model using the baseline method VPT+BEITv2, the optimal learning strategies for the Flowers102 and Pets datasets are different. In other words, different learning strategies are required to train the model on these two datasets. Therefore, compared to the baseline method described above, the image classification model training method provided in this application does not require spending a significant amount of time searching for the optimal learning strategy for different image classification tasks, thus enabling efficient implementation of various image classification tasks.

[0137] Figure 7 The test results of image classification models trained using different methods are shown. For example... Figure 7 As shown, this test was conducted with an epoch of 50, meaning that for each dataset, all data in the test dataset were trained 50 times in the image classification model. Furthermore, since the VPT method outperforms its performance at epoch 100 at epoch 50, this test also evaluated the VPT method and the method provided in the embodiments of this application at epoch 100. Figure 7 The following conclusions can be drawn:

[0138] 1. On the above 10 datasets, the average accuracy of the image classification model trained using the method provided in the embodiments of this application is about 5% higher than the average accuracy of the image classification model trained using LP+BEI Tv2.

[0139] 2. On the first nine datasets, the average accuracy of the image classification model trained using the method provided in this application (88.65%) is close to the average accuracy of the image classification model trained using FT+CLI P (89.4%).

[0140] 3. On the first nine datasets, the average accuracies of the image classification models trained using LP+CLI P, TP+CLI P, and VP+TP+CLI P were 83%, 56.69%, and 79.67%, respectively. Therefore, on the first nine datasets, the average accuracy of the image classification models trained using these three methods was lower than the average accuracy (88.65%) of the image classification models trained using the method provided in the embodiments of this application.

[0141] 4. On the aforementioned 10 datasets, regardless of whether the epoch is 50 or 100, the average accuracy of the image classification model trained using the method provided in this application embodiment is higher than the average accuracy of the image classification model trained using VPT. Moreover, on most datasets (i.e., 8 datasets), the accuracy of the image classification model trained using the method provided in this application embodiment is higher than the accuracy of the image classification model trained using VPT.

[0142] 5. On the aforementioned 10 datasets, when the epoch is 50, the average accuracy of the image classification model trained using the method provided in this application embodiment is 87.91. A visualization of the classification results can be found in [reference needed]. Figure 8 (In this figure, black circles represent category vectors corresponding to category labels, and other discrete points represent predicted visual words obtained by the image classification model from the test dataset.) When the epoch is 100, the average accuracy of the image classification model trained using VPT is 87.91. That is, the image classification model trained using the method provided in this application can achieve the performance of the image classification model trained using VPT at epoch 100 with only epoch 50. This shows that compared to VPT, the method provided in this application can save training time for the image classification model.

[0143] Further, please see Figure 9 , Figure 9This paper illustrates the performance of the mapping rule used by the image classification model in this embodiment (i.e., determining the image classification result by mapping the predicted visual words corresponding to the image to the same vector space). To facilitate understanding of the performance of the mapping rule provided in this embodiment, the following two mapping rules were also selected for comparison in this experiment:

[0144] 1. Directly map the predicted visual words to categories. This mapping rule will be referred to as MLP-1 below.

[0145] 2. First, the predicted visual words are mapped to 128-dimensional vectors, and then the 128-dimensional vectors are mapped to categories. This mapping rule will be referred to as MLP-2 below.

[0146] On the training datasets of the aforementioned 10 datasets, image classification models were trained using MLP-1, MLP-2, and the mapping rules provided in the embodiments of this application, respectively. Then, the trained image classification models were used to test the test datasets of the aforementioned 10 datasets, yielding the results. Figure 9 The test results are shown. From Figure 9 It can be seen that:

[0147] 1. On the aforementioned 10 datasets, although the average proportion of fine-tuned parameters when using MLP-1 is lower than the average proportion of fine-tuned parameters when using the mapping rules provided in this application, the average accuracy of image classification achieved by the mapping rules provided in this application is 8.18% higher than the average accuracy of image classification achieved by MLP-1. Here, the proportion of fine-tuned parameters is the ratio of fine-tuned parameters to all parameters in the model.

[0148] 2. On the above 10 datasets, the average proportion of fine-tuned parameters when using MLP-2 is lower than the average proportion of fine-tuned parameters when using the mapping rules provided in the embodiments of this application, and the average accuracy of image classification achieved by the mapping rules provided in the embodiments of this application is higher than the average accuracy of image classification achieved by MLP-2.

[0149] It is worth noting that the mapping rules provided by MLP-1, MLP-2 and the embodiments of this application are all based on the predicted visual words corresponding to the image to determine the image category. Therefore, in order to more accurately determine the performance of each mapping rule, when training and testing the image classification model using the mapping rules provided by MLP-1, MLP-2 and the embodiments of this application, it is necessary to ensure that the predicted visual words corresponding to the images used are consistent. Here, the method provided by the embodiments of this application is used to obtain the predicted visual words corresponding to the images.

[0150] The preceding text described in detail the training methods for image classification models and the methods for performing image classification using the trained image classification models. The following section will combine... Figures 10-13 The training apparatus for performing the training method of the above-described image classification model and the inference apparatus for performing the above-described image classification method will be further described.

[0151] Figure 10 A schematic diagram of the training device is shown as an example. (e.g.) Figure 10 As shown, the training device 100 includes an acquisition module 110, a stitching module 120, and an optimization module 130. These modules work together to implement the steps performed by the training device in the above method embodiment. Specifically, the acquisition module 110 performs S101; the stitching module 120 performs S102; and the optimization module 130 performs S103.

[0152] In this embodiment, the acquisition module 110, the stitching module 120, and the optimization module 130 can all be implemented in software or in hardware. For example, the implementation of the optimization module 130 will be described below. Similarly, the implementation of the acquisition module 110 and the stitching module 120 can refer to the implementation of the optimization module 130.

[0153] As an example of a software functional unit, optimization module 130 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, optimization module 130 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0154] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0155] As an example of a hardware functional unit, the optimization module 130 may include at least one computing device, such as a server. Alternatively, the optimization module 130 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0156] The optimization module 130 includes multiple computing devices that can be distributed in the same region or in different regions. Similarly, the optimization module 130 includes multiple computing devices that can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the optimization module 130 includes multiple computing devices that can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0157] It should be noted that, in other embodiments, the acquisition module 110 can be used to execute any of the steps in S101-S103 described above; the stitching module 120 can be used to execute any of the steps in S101-S103 described above; and the optimization module 130 can be used to execute any of the steps in S101-S103 described above. The steps implemented by the acquisition module 110, the stitching module 120, and the optimization module 130 can be specified as needed. By implementing different steps in the above image classification model training method through the acquisition module 110, the stitching module 120, and the optimization module 130, all functions of the training device 100 can be realized.

[0158] Figure 11 A schematic diagram of the inference device is shown as an example. (e.g.) Figure 11 As shown, the inference device 200 includes an acquisition module 210 and a classification module 220. These modules work together to implement the steps performed by the inference device in the above method embodiments. Specifically, the acquisition module 210 performs S201; the classification module 220 performs S202.

[0159] In this embodiment, both the acquisition module 210 and the classification module 220 can be implemented in software or in hardware. For example, the implementation of the classification module 220 will be described below. Similarly, the implementation of the acquisition module 210 can refer to the implementation of the classification module 220.

[0160] As an example of a software functional unit, classification module 220 may include code running on compute instances. A compute instance may include at least one of a physical host (compute device), a virtual machine, or a container. Further, the aforementioned compute instance may be one or more. For example, classification module 220 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0161] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0162] As an example of a hardware functional unit, the classification module 220 may include at least one computing device, such as a server. Alternatively, the classification module 220 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0163] The multiple computing devices included in the classification module 220 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the classification module 220 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the classification module 220 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0164] It should be noted that, in other embodiments, the acquisition module 210 can be used to execute any step in S201-S202 described above; the classification module 220 can be used to execute any step in S201-S202 described above. The steps implemented by the acquisition module 210 and the classification module 220 can be specified as needed, and the acquisition module 210 and the classification module 220 respectively implement different steps in the above image classification method to realize all the functions of the inference device 200.

[0165] This application also provides a computing device. For example... Figure 12 As shown, the computing device 300 includes a bus 310, a processor 320, a memory 330, and a communication interface 340. The processor 320, the memory 330, and the communication interface 340 communicate with each other via the bus 310. The computing device 300 can be a server or a terminal device. It should be understood that the embodiments of this application do not limit the number of processors and memories in the computing device 300.

[0166] Bus 310 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 12 The bus 310 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 310 may include a path for transmitting information between various components of the computing device 300 (e.g., memory 330, processor 320, communication interface 340).

[0167] The processor 320 may include any one or more processors such as CPU, GPU, TPU, microprocessor (MP), or digital signal processor (DSP).

[0168] Memory 330 may include volatile memory, such as random access memory (RAM). Memory 330 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0169] The memory 330 stores executable program code. The processor 320 executes the executable program code stored in the memory 330 to implement the functions of each module in the aforementioned training device 100 (including the acquisition module 110, the stitching module 120, and the optimization module 130), thereby implementing the training method of the image classification model described above. Alternatively, the processor 320 executes the executable program code stored in the memory 330 to implement the functions of each module in the aforementioned inference device 200 (including the acquisition module 210 and the classification module 220), thereby implementing the image classification method described above. That is, the memory 330 stores instructions for executing the training method of the image classification model and / or the image classification method described above.

[0170] The communication interface 340 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 300 and other devices or communication networks. For example, the computing device 300 obtains training datasets and / or images to be processed through the communication interface 340.

[0171] This application also provides a computing device cluster. The computing device cluster includes one or more computing devices. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0172] like Figure 13 As shown, the computing device cluster 400 includes one or more computing devices 300. The memory 330 of one or more computing devices 300 in the computing device cluster 400 may store the same instructions for executing the training method and / or image classification method of the aforementioned image classification model. Alternatively, the memory 330 of one or more computing devices 300 in the computing device cluster 400 may also store partial instructions for executing the training method and / or image classification method of the aforementioned image classification model. In other words, a combination of one or more computing devices 300 can jointly execute the training method and / or image classification method of the aforementioned image classification model.

[0173] It should be noted that the memory 330 in different computing devices 300 within the computing device cluster 400 can store different instructions, which are used to execute some functions of the training device 100 and / or the inference device 200, respectively. That is, the instructions stored in the memory 330 in different computing devices 300 can implement the functions of one or more of the aforementioned acquisition module 110, splicing module 120, optimization module 130, acquisition module 210, and classification module 220.

[0174] In some embodiments, one or more computing devices 300 in the computing device cluster 400 can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc.

[0175] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions that can run on a computing device or be stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the computation graph compilation method described above.

[0176] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (such as floppy disks, hard disks, and magnetic tapes), optical media (such as DVDs), or semiconductor media (such as SSDs). The computer-readable storage medium includes instructions, wherein the instructions instruct the computing device to execute the computational graph compilation method described above, or instruct the computing device to execute the computational graph compilation method described above.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A training method for an image classification model, characterized in that, include: Obtain a training dataset, which includes multiple training images; For each training image, a cue vector and a mask vector of the pre-trained model are concatenated to obtain the target input vector corresponding to each training image. The pre-trained model is trained using the mask vector. The prompt vector is optimized based on the pre-trained model and the target input vector corresponding to each training image to obtain an image classification model. The parameters of the image classification model include the parameters of the pre-trained model and the optimized prompt vector. The image classification model is used to perform image classification tasks.

2. The method according to claim 1, characterized in that, The step of concatenating the cue vector and the mask vector of the pre-trained model for each training image includes: The cue vector and the mask vector of the pre-trained model are concatenated with each training image.

3. The method according to claim 1 or 2, characterized in that, The optimization of the prompt vector based on the pre-trained model and the target input vector corresponding to each training image includes: The pre-trained model is used to predict the mask vector in the target input vector corresponding to each training image to obtain the predicted visual word corresponding to each training image; The predicted visual word corresponding to each training image and the category label corresponding to each training image are mapped to the same vector space; The cue vector is optimized based on the similarity between the mapped predicted visual word and the category label.

4. The method according to claim 3, characterized in that, The step of mapping the predicted visual word corresponding to each training image and the category label corresponding to each training image to the same vector space includes: Set a corresponding category vector for the category label of each training image; The predicted visual word is dimensionally transformed based on the dimension of the category vector, and the transformed predicted visual word has the same dimension as the category vector.

5. An image classification method, characterized in that, include: Obtain the image to be processed; The image to be processed is classified based on an image classification model to obtain the classification result of the image to be processed. The image classification model includes the following parameters: parameters of a pre-trained model and an optimized cue vector. The parameters of the pre-trained model include a mask vector. The pre-trained model is trained based on the mask vector. The optimized cue vector is optimized based on the pre-trained model and the target input vectors corresponding to multiple training images. The target input vector corresponding to each training image is obtained by concatenating the mask vector and the unoptimized cue vector for each training image.

6. The method according to claim 5, characterized in that, The target input vector corresponding to each training image is obtained by concatenating the mask vector and the unoptimized cue vector with each training image.

7. The method according to claim 5 or 6, characterized in that, The image classification model also includes the following parameters: at least one category vector, and a category label corresponding to each category vector.

8. The method according to claim 7, characterized in that, The classification of the image to be processed based on the image classification model includes: The mask vector and the optimized cue vector are concatenated with the image to be processed to obtain the target input vector corresponding to the image to be processed; The image to be processed is classified based on the target input vector corresponding to the image to be processed.

9. The method according to claim 8, characterized in that, The classification of the image to be processed based on the target input vector corresponding to the image to be processed includes: The predicted visual word corresponding to the image to be processed is determined based on the target input vector corresponding to the image to be processed. The predicted visual word is dimensionally transformed based on the dimension of the at least one category vector; The image to be processed is classified based on the similarity between the transformed predicted visual word and the at least one category vector.

10. A training device, characterized in that, include: The acquisition module is used to acquire the training dataset, which includes multiple training images; The stitching module is used to stitch together the cue vector and the mask vector of the pre-trained model for each training image to obtain the target input vector corresponding to each training image. The pre-trained model is trained using the mask vector. An optimization module is used to optimize the prompt vector based on the pre-trained model and the target input vector corresponding to each training image to obtain an image classification model. The parameters of the image classification model include the parameters of the pre-trained model and the optimized prompt vector. The image classification model is used to perform an image classification task.

11. The apparatus according to claim 10, characterized in that, The stitching module is used to concatenate the cue vector and the mask vector of the pre-trained model with each training image.

12. The apparatus according to claim 10 or 11, characterized in that, The optimization module is used to predict the mask vector in the target input vector corresponding to each training image using the pre-trained model to obtain the predicted visual word corresponding to each training image, map the predicted visual word corresponding to each training image and the category label corresponding to each training image to the same vector space, and optimize the prompt vector based on the similarity between the mapped predicted visual word and the category label.

13. The apparatus according to claim 12, characterized in that, The optimization module is used to set a corresponding category vector for the category label of each training image, and to perform a dimension transformation on the predicted visual word based on the dimension of the category vector, so that the transformed predicted visual word has the same dimension as the category vector.

14. A reasoning device, characterized in that, include: The acquisition module is used to acquire the image to be processed; A classification module is used to classify the image to be processed based on an image classification model to obtain the classification result of the image to be processed. The image classification model includes the following parameters: parameters of a pre-trained model and an optimized cue vector. The parameters of the pre-trained model include a mask vector. The pre-trained model is trained based on the mask vector. The optimized cue vector is optimized based on the pre-trained model and the target input vectors corresponding to multiple training images. The target input vector corresponding to each training image is obtained by concatenating the mask vector and the unoptimized cue vector for each training image.

15. The apparatus according to claim 14, characterized in that, The target input vector corresponding to each training image is obtained by concatenating the mask vector and the unoptimized cue vector with each training image.

16. The apparatus according to claim 14 or 15, characterized in that, The image classification model also includes the following parameters: at least one category vector, and a category label corresponding to each category vector.

17. The apparatus according to claim 16, characterized in that, The classification module is used to concatenate the mask vector and the optimized cue vector with the image to be processed to obtain the target input vector corresponding to the image to be processed, and to classify the image to be processed based on the target input vector corresponding to the image to be processed.

18. The apparatus according to claim 17, characterized in that, The classification module is used to determine the predicted visual word corresponding to the image to be processed based on the target input vector corresponding to the image to be processed, perform dimensional transformation on the predicted visual word based on the dimension of the at least one category vector, and classify the image to be processed based on the similarity between the transformed predicted visual word and the at least one category vector.

19. A computing device, characterized in that, It includes a processor and a memory, the processor being configured to execute instructions stored in the memory to implement the method as claimed in any one of claims 1 to 9.

20. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1 to 9.

21. A computer-readable storage medium, characterized in that, It includes computer program instructions that, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1 to 9.

Images