Classification model training method, image classification method, device, medium and equipment

Abstract

The present disclosure relates to a classification model training method, an image classification method, an apparatus, a medium and an equipment. The training method comprises: pre-training a visual language model to obtain an original image classification model; and fine-tuning the original image classification model according to preset category prompt information and dynamic category prompt information learned by the original image classification model in a fine-tuning process to obtain a target image classification model. The preset category prompt information is decoupled from the image end, data enhancement on the language side is achieved, the relationship between the image and the language is better captured, the generalization and classification effect of the image classification model are improved, the model training is more stable, and the convergence is faster. In addition, a small amount of training data is used to pre-train the visual language model, and then the original image classification model obtained by pre-training is fine-tuned according to the preset category prompt information and the dynamic category prompt information. In this way, a good image classification model can be obtained with a small amount of training data.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of image processing, and in particular, to a classification model training method, an image classification method, an apparatus, a medium and an equipment. BACKGROUND

[0002] A visual language task (for example, image classification, image question answering, etc.) usually only relies on the representation ability of language information, which is not enough to complete the task well, and different image features and language representations may not match, resulting in the effect of the visual language task being affected. SUMMARY

[0003] This summary is provided to introduce a selection of concepts that are further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] In a first aspect, the present disclosure provides a method for training an image classification model, comprising:

[0005] pre-training a visual language model to obtain an original image classification model;

[0006] fine-tuning the original image classification model according to preset class prompt information and dynamic class prompt information learned by the original image classification model in a fine-tuning process to obtain a target image classification model, wherein an initial value of the dynamic class prompt information is empty.

[0007] In a second aspect, the present disclosure provides an image classification method, comprising:

[0008] obtaining an image to be classified;

[0009] generating a target classification result of the image to be classified by a pre-trained target image classification model according to the image to be classified, preset class prompt information and target class prompt information, wherein the target image classification model is trained according to the image classification model training method provided in the first aspect of the present disclosure, and the target class prompt information is dynamic class prompt information learned by the target image classification model when the training is completed.

[0010] In a third aspect, the present disclosure provides an image classification model training apparatus, comprising:

[0011] a pre-training module configured to pre-train a visual language model to obtain an original image classification model;

[0012] The fine-tuning module is configured to fine-tune the original image classification model according to preset class prompt information and dynamic class prompt information learned by the original image classification model in a fine-tuning process, to obtain a target image classification model, wherein an initial value of the dynamic class prompt information is empty.

[0013] In a fourth aspect, the present disclosure provides an image classification device, comprising:

[0014] The acquisition module is configured to acquire an image to be classified.

[0015] The classification module is configured to generate a target classification result of the image to be classified by a pre-trained target image classification model according to the image to be classified, preset class prompt information, and target class prompt information, wherein the target image classification model is obtained by the image classification model training method provided in the first aspect of the present disclosure, and the target class prompt information is dynamic class prompt information learned by the target image classification model when the training is completed.

[0016] In a fifth aspect, the present disclosure provides a computer readable medium having a computer program stored thereon, wherein the program is executed by a processing device to implement the steps of the image classification model training method provided in the first aspect of the present disclosure or the steps of the image classification method provided in the second aspect of the present disclosure.

[0017] In a sixth aspect, the present disclosure provides an electronic device, comprising:

[0018] The storage device has a computer program stored thereon.

[0019] The processing device is configured to execute the computer program in the storage device to implement the steps of the image classification model training method provided in the first aspect of the present disclosure or the steps of the image classification method provided in the second aspect of the present disclosure.

[0020] In the technical solution, first, the visual language model is pre-trained to obtain an original image classification model; then, the original image classification model is fine-tuned according to preset category prompt information and dynamic category prompt information learned by the original image classification model in a fine-tuning process to obtain a target image classification model. The preset category prompt information is decoupled from the image end, and can be enhanced on the language side, so as to better capture the relationship between the image and the language, improve the generalization and classification effect of the image classification model, and make the model training more stable and converge faster. In addition, the visual language model usually has a large scale, and a large amount of training data is required for training the model, and the training data is usually limited. Therefore, a small amount of training data can be used to pre-train the visual language model, and then the original image classification model obtained by pre-training is fine-tuned according to the preset category prompt information and the dynamic category prompt information. In this way, a good image classification model can be obtained by using a small amount of training data, and large-scale model training becomes possible.

[0021] Other features and advantages of the present disclosure will be described in detail in the following detailed description section. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent by describing in detail exemplary embodiments thereof with reference to the attached drawings in which:

[0023] Figure 1 is a flowchart of an image classification model training method according to an exemplary embodiment.

[0024] Figure 2 is a process diagram of generating a predicted classification result of each sample image by an original image classification model according to a preset category prompt vector, a dynamic category prompt vector, and each sample image according to an exemplary embodiment.

[0025] Figure 3 is a flowchart of an image classification method according to an exemplary embodiment.

[0026] Figure 4 is a block diagram of an image classification model training apparatus according to an exemplary embodiment.

[0027] Figure 5 is a block diagram of an image classification apparatus according to an exemplary embodiment.

[0028] Figure 6 is a structural diagram of an electronic device according to an exemplary embodiment. DETAILED DESCRIPTION

[0029] As discussed in the background, visual language tasks suffer from technical problems such as insufficient language representation capability, and mismatch between image and language representation. At present, these technical problems are mainly solved in the following three ways: (1) training large multi-modal language models: the model learns language representation through a large amount of language data, and combines the language representation with image features to perform visual language tasks; (2) designing visual language models: the model learns the representation of images and languages through the designed architecture and learning strategy, thereby realizing visual language tasks; (3) using adversarial learning: the model generates samples that do not match the real samples through adversarial learning, thereby strengthening the matching between image and language representation.

[0030] Although the above three schemes have achieved certain effects, they also have some defects. For example, training large multi-modal language models requires a large amount of language data, and may not be able to capture the relationship between images and languages well. The method of adversarial learning may cause the model to overfit, thereby reducing the effect of the model.

[0031] In view of this, the present disclosure provides a classification model training method, an image classification method, an apparatus, a medium and an equipment.

[0032] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms, and should not be interpreted as being limited to the embodiments set forth herein, but rather these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for exemplary purposes only, and are not intended to limit the scope of protection of the present disclosure.

[0033] It should be understood that each step described in the method embodiments of the present disclosure can be executed in different order and / or in parallel. In addition, the method embodiments can include additional steps and / or omit the execution of the steps shown. The scope of the present disclosure is not limited in this respect.

[0034] The term "comprising" and variations thereof as used herein are open-ended, that is "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Related definitions of other terms will be given in the description below.

[0035] It should be noted that the terms "first", "second", and the like mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0036] It should be noted that the terms "one", "multiple" mentioned in the present disclosure are illustrative but not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0037] The names of the messages or information exchanged between the plurality of devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0038] It can be understood that before using the technical solutions disclosed in the embodiments of the present disclosure, the type, use range, use scenario, etc. of the personal information involved in the present disclosure should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0039] For example, in response to receiving the active request of the user, the user is sent prompt information to explicitly prompt the user that the operation requested to be performed will require obtaining and using the personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the software or hardware such as electronic device, application program, server or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt information.

[0040] As an optional but non-limiting implementation manner, in response to receiving the active request of the user, the prompt information can be sent to the user in the form of a pop-up window, and the prompt information can be presented in the form of text in the pop-up window. In addition, the pop-up window can also carry selection controls for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0041] It can be understood that the above notification and user authorization process is only illustrative, and does not limit the implementation manner of the present disclosure, and other manners meeting the relevant laws and regulations can also be applied to the implementation manner of the present disclosure.

[0042] At the same time, it can be understood that the data involved in the present technical solution (including but not limited to the data itself, the acquisition or use of the data) should comply with the requirements of the relevant laws and regulations and relevant provisions.

[0043] Figure 1 is a flowchart of an image classification model training method according to an example embodiment. As shown in Figure 1 The image classification model training method can include the following S101 and S102.

[0044] In S101, the visual language model is pre-trained to obtain an original image classification model.

[0045] In S102, the original image classification model is fine-tuned according to the preset category prompt information and the dynamic category prompt information learned by the original image classification model in the fine-tuning process to obtain a target image classification model.

[0046] In the present disclosure, the preset category prompt information can be one or multiple, and similarly, the dynamic category prompt information can be one or multiple. The initial value of the dynamic category prompt information is empty, which dynamically changes in the fine-tuning process of the original image classification model.

[0047] The preset category prompt information can be prompt information used to describe the image category, which can include description information in the form of text, for example, this is an animal. The image category can be any suitable category determined according to the actual business scenario, which can include but is not limited to any one of the categories of objects contained in the image (for example, cat, dog, rabbit, tiger, etc.), the category of the scene represented by the image (for example, indoor, airport, station, shopping mall, etc.), and the state category of the target object in the image (such as defective, non-defective, etc.).

[0048] In the above technical solution, first, the visual language model is pre-trained to obtain an original image classification model; then, the original image classification model is fine-tuned according to the preset category prompt information and the dynamic category prompt information learned by the original image classification model in the fine-tuning process to obtain a target image classification model. The preset category prompt information is decoupled from the image end, which can perform data enhancement on the language side, thereby better capturing the relationship between the image and the language, improving the generalization and classification effect of the image classification model, and making the model training more stable and converging faster. In addition, the visual language model usually has a large scale, and a large amount of training data is required to train the model, while the training data is usually limited. Therefore, a small amount of training data can be used to pre-train the visual language model, and then the original image classification model obtained by pre-training is fine-tuned according to the preset category prompt information and the dynamic category prompt information. In this way, a good image classification model can be obtained through a small amount of training data, making it possible to train a large-scale model.

[0049] The specific embodiments of the pre-training of the visual language model in S101 to obtain the original image classification model are described in detail as follows.

[0050] Specifically, the visual language model can include: an image encoder, a first residual module, a text encoder, a second residual module and a mixed processing module connected in sequence, wherein the image encoder is connected with the second residual module. In the pre-training stage, the image encoder and the text encoder participate in the model pre-training, and the first residual module, the second residual module and the mixed processing module do not participate in the model pre-training. Specifically, the visual language model can be pre-trained through the following steps [1] to [5]:

[0051] Step [1]: obtaining a sample image, a labeled classification result of the sample image and preset category prompt information.

[0052] Step [2]: performing vectorization processing on the preset category prompt information to obtain a preset category prompt vector.

[0053] Specifically, the preset category prompt information is text, which can be segmented first, and then each segmented word can be mapped to a vector representation using a word embedding model to obtain the preset category prompt vector.

[0054] Step [3]: generating an image feature of the sample image through the image encoder, and generating a third text vector corresponding to each preset image category through the text encoder according to the preset category prompt vector.

[0055] Step [4]: calculating a first similarity between each third text vector and the image feature of the sample image, and determining a predicted classification result of the sample image according to each first similarity.

[0056] In the present disclosure, the predicted classification result can include a predicted image category to which the sample image belongs, and correspondingly, the labeled classification result can include a labeled image category to which the sample image belongs; or the predicted classification result can include a predicted probability that the sample image belongs to each preset image category, and correspondingly, the labeled classification result can include a labeled probability that the sample image belongs to each preset image category. When the sample image belongs to a certain preset image category, the labeled probability that the sample image belongs to the preset image category is 1, and the labeled probability that the sample image belongs to other preset image categories is 0.

[0057] When the predicted classification result includes a predicted image category to which the sample image belongs, the predicted image category to which the sample image belongs corresponding to the target text vector can be obtained, wherein the target text vector is the third text vector corresponding to the maximum value in each first similarity.

[0058] When the prediction probability of the sample image belonging to each preset image category is predicted, for each third text vector, the first similarity between the third text vector and the image feature of the sample image can be determined as the prediction probability of the sample image belonging to the preset image category corresponding to the third text vector.

[0059] Step [5]: According to the predicted classification result and the labeled classification result, the model parameter of the image encoder and the text encoder is updated.

[0060] Specifically, the model loss can be calculated according to the predicted classification result and the labeled classification result, and then the model parameter of the image encoder and the text encoder is updated according to the model loss.

[0061] In addition, the structure of the original image classification model obtained after pre-training the visual language model is the same as that of the above-mentioned visual language model. That is, the original image classification model comprises an image encoder, a first residual module, a text encoder, a second residual module and a mixed processing module connected in sequence, wherein the image encoder is connected with the second residual module.

[0062] The following detailed description of the specific implementation of the above step S102 is described in detail. Specifically, the following steps (1) to (4) can be implemented:

[0063] Step (1): Obtain a plurality of sample images and a labeled classification result corresponding to each sample image.

[0064] Specifically, a plurality of original images can be obtained first; then, for each original image, the original image is scaled to a preset size (for example, 224mm*224mm) to obtain a sample image. In this way, the original image can be adjusted to the preset size to meet the input requirements of the image encoder.

[0065] Step (2): Vectorize the preset category prompt information to obtain a preset category prompt vector, and vectorize the dynamic category prompt information to obtain a dynamic category prompt vector.

[0066] Specifically, the dynamic category prompt information is text, which can be segmented first, and then each segmented word can be mapped to a vector representation using a word embedding model to obtain a dynamic category prompt vector.

[0067] Step (3): According to the preset category prompt vector, the dynamic category prompt vector and each sample image, a predicted classification result of each sample image is generated by the original image classification model.

[0068] Step (4): updating the dynamic class hint vector and the model parameters of the original image classification model according to each predicted classification result and each labeled classification result.

[0069] The following detailed description is made with reference to the specific implementation of step (3) above, which generates a predicted classification result of each sample image through the original image classification model according to the preset class hint vector, the dynamic class hint vector, and each sample image. Specifically, the preset class hint vector and the dynamic class hint vector are both multiple, and at this time, the following steps shown in FIG. 1B (an example is taken with one sample image) can be used to achieve the implementation: Figure 2 Figure 2

[0070] First, for each preset class hint vector (four preset class hint vectors are taken for illustration in FIG. 1B), the first residual module is used to add the preset class hint vector and all dynamic class hint vectors to obtain a first text vector. Figure 2

[0071] Then, the text encoder is used to generate a text feature of each first text vector.

[0072] Finally, for each sample image, the image encoder is used to generate an image feature of the sample image; the second residual module is used to add each pair of text features and the image feature of the sample image to obtain a mixed feature; the mixed processing module is used to generate a second text vector corresponding to each preset image class according to the mixed feature; and the predicted classification result of the sample image is determined according to each second text vector and the image feature of the sample image.

[0073] The mixed processing module is used to mix the text features and the image feature in the mixed feature to prepare for image classification. For example, the mixed processing module can be a multilayer perceptron (MLP), which can include a first fully connected layer, a nonlinear activation layer, and a second fully connected layer connected in sequence.

[0074] In addition, the predicted classification result of the sample image can be determined according to each second text vector and the image feature of the sample image in the following manner:

[0075] For each second text vector, a second similarity between the second text vector and the image feature of the sample image is calculated; and then, the predicted classification result of the sample image is determined according to the second similarities.

[0076] ​​​Wherein, the predicted classification result of the sample image can be determined according to each second similarity in a similar manner to that of determining the predicted classification result of the sample image according to each first similarity in step [4] above, and details are not repeated here.

[0077] In the above embodiment, in addition to the residual learning of the preset category prompt information and the dynamic category prompt information, the feature residual learning of the image and the language is also used at the same time, so that the model training is more stable, the image feature is combined with the text feature, the registration between the view image and the language is strengthened, and the classification accuracy of the image classification model is further improved.

[0078] The specific implementation of step (4) in the above embodiment will be described in detail below.

[0079] Specifically, the following steps (41) and (42) can be used to achieve this:

[0080] Step (41): calculating the target loss of the original image classification model according to each predicted classification result and each labeled classification result.

[0081] For example, the target loss of the original image classification model can be calculated according to each predicted classification result and each labeled classification result by the following equations (1)-(5):

[0082]

[0083]

[0084]

[0085]

[0086] p' mk =p m +h k ,m={1,2,…,M},k={1,2,…,K} (5)

[0087] Wherein, L vl is the target loss; x i is the i-th sample image in the plurality of sample images; P r (c|x i ) is the predicted probability that the i-th sample image in the plurality of sample images belongs to the c-th preset image category; y icis the annotation probability of the i-th sample image in the plurality of sample images belonging to the c-th preset image category; N is the number of sample images; C is the number of preset image categories; K is the number of preset category prompt vectors; M is the number of dynamic category prompt vectors; p m is the m-th dynamic category prompt vector; h k is the k-th preset category prompt vector; p' mk is the sum of the m-th dynamic category prompt vector and the k-th preset category prompt vector; w c is the c-th preset image category vector (obtained by vectorizing the c-th preset image category); is the vector obtained after concatenating p1' k , p' 2k , …, p' Mk , and w c ; f i is the image feature of the i-th sample image in the plurality of sample images; is the text feature generated after inputting h to the text encoder; is the sum of the image feature of the i-th sample image in the plurality of sample images and K text features ; τ is a temperature coefficient, and is a pre-set hyperparameter value.

[0088] Step (42): updating the dynamic category prompt vector and the model parameters of the mixed processing module according to the target loss.

[0089] In the present disclosure, the dynamic category prompt vector and the model parameters of the mixed processing module are updated according to the target loss, while the model parameters of the text encoder and the image encoder are fixed and unchanged in the model fine-tuning stage.

[0090] In addition, the model parameters of the mixed processing module can be updated by using the target loss and an Adaptive Moment Estimation (Adam) optimizer. For example, the target loss and the Adam optimizer are used to train for 30 epochs (one epoch means that each sample image in the plurality of sample images participates in training once). The Adam optimizer is a gradient-based optimization algorithm, which has the characteristics of fast convergence, high computational efficiency, and low memory occupation.

[0091] In addition, in order to further improve the classification accuracy of the target image classification model, the original image classification model can be updated multiple times. Specifically, after the above step (4), the S102 can further include the following steps (5) and (6).

[0092] Step (5): determining whether a training stop condition is met.

[0093] In an embodiment, the training stop condition can be that the number of training rounds reaches a preset number of rounds (for example, 30 epochs), which can be set according to an actual use scenario.

[0094] In another embodiment, the training stop condition can be that the target loss of the original image classification model is less than a preset threshold, which can be set according to an actual use scenario. In the case where the target loss of the original image classification model is less than the preset threshold, it can be considered that the classification accuracy of the original image classification model meets the requirements and can accurately classify images.

[0095] If the training stop condition is not met, steps (1) to (4) are repeatedly executed until the training stop condition is met; if the training stop condition is met, the original image classification model obtained after the last model parameter update can be determined as the target image classification model, that is, the following step (6) is executed.

[0096] Step (6): Obtain the target image classification model.

[0097] Figure 3 is a flowchart of an image classification method according to an exemplary embodiment. As shown in Figure 3 The image classification method can include the following S301 and S302.

[0098] In S301, the image to be classified is obtained.

[0099] In the present disclosure, the image to be classified can be any suitable image. In implementation, a person skilled in the art can select a suitable image to be classified according to an actual application scenario, and the present disclosure embodiment is not limited specifically. For example, the image to be classified can include but is not limited to at least one of an image captured by a camera, an image frame in a video or image frame sequence, a medical image, an image generated by computer technology, etc.

[0100] In S302, according to the image to be classified, the preset category prompt information and the target category prompt information, the target classification result of the image to be classified is generated by the target image classification model.

[0101] In the present disclosure, the target image classification model is trained according to the above-mentioned image classification model training method provided by the present disclosure, and the target category prompt information is the dynamic category prompt information learned when the target image classification model is trained, which is obtained according to the dynamic category prompt vector obtained when the target image classification model is trained.

[0102] Specifically, the preset category prompt information can be subjected to vectorization processing to obtain a preset category prompt vector, and the target category prompt information can be subjected to vectorization processing to obtain a target category prompt vector. Then, the image to be classified, the preset category prompt vector, and the target category prompt vector are input into the target image classification model to obtain the target classification result of the image to be classified.

[0103] In the determination of the target classification result of the image to be classified, the preset category prompt information is referred to, so that the target image classification model can make a targeted prompt.

[0104] Figure 4 is a block diagram of an image classification model training apparatus according to an example embodiment. As shown in Figure 4 The image classification model training apparatus 400 can include:

[0105] A pre-training module 401 is configured to pre-train a visual language model to obtain an original image classification model.

[0106] A fine-tuning module 402 is configured to fine-tune the original image classification model according to preset category prompt information and dynamic category prompt information learned by the original image classification model in a fine-tuning process to obtain a target image classification model, wherein an initial value of the dynamic category prompt information is empty.

[0107] In the above technical solution, first, the visual language model is pre-trained to obtain the original image classification model. Then, the original image classification model is fine-tuned according to the preset category prompt information and the dynamic category prompt information learned by the original image classification model in the fine-tuning process to obtain the target image classification model. The preset category prompt information is decoupled from the image end, and can be subjected to data enhancement on the language side, so as to better capture the relationship between the image and the language, improve the generalization and classification effect of the image classification model, and make the model training more stable and converge faster. In addition, the visual language model usually has a large scale, and a large amount of training data is required for training the model. However, the training data is usually limited. Therefore, a small amount of training data can be used to pre-train the visual language model, and then the original image classification model obtained by the pre-training is fine-tuned according to the preset category prompt information and the dynamic category prompt information. In this way, a good image classification model can be obtained by using a small amount of training data, and the training of a large-scale model becomes possible.

[0108] Optionally, the fine-tuning module 402 includes:

[0109] An acquisition sub-module is configured to acquire a plurality of sample images and a labeled classification result corresponding to each of the sample images.

[0110] The vectorization processing submodule is configured to perform vectorization processing on the preset category prompt information to obtain a preset category prompt vector, and perform vectorization processing on the dynamic category prompt information to obtain a dynamic category prompt vector.

[0111] The classification submodule is configured to generate a predicted classification result of each sample image by using the original image classification model according to the preset category prompt vector, the dynamic category prompt vector, and each sample image.

[0112] The first updating submodule is configured to update the dynamic category prompt vector and model parameters of the original image classification model according to each predicted classification result and each labeled classification result.

[0113] Optionally, the original image classification model comprises an image encoder, a first residual module, a text encoder, a second residual module, and a hybrid processing module connected in sequence, wherein the image encoder is connected with the second residual module; and the preset category prompt vector and the dynamic category prompt vector are both multiple.

[0114] The classification submodule comprises:

[0115] The first residual submodule is configured to, for each preset category prompt vector, add the preset category prompt vector and all dynamic category prompt vectors by using the first residual module to obtain a first text vector.

[0116] The first generation submodule is configured to generate a text feature of each first text vector by using the text encoder.

[0117] The second generation submodule is configured to, for each sample image, generate an image feature of the sample image by using the image encoder; the second residual submodule is configured to add each pair of text features and image features of the sample image by using the second residual module to obtain a hybrid feature; the hybrid submodule is configured to generate a second text vector corresponding to each preset image category by using the hybrid processing module according to the hybrid feature; and the first determination submodule is configured to determine a predicted classification result of the sample image according to each second text vector and the image feature of the sample image.

[0118] Optionally, the first updating submodule comprises:

[0119] The second determination submodule is configured to calculate a target loss of the original image classification model according to each predicted classification result and each labeled classification result.

[0120] The second updating submodule is configured to update the dynamic category prompt vector and model parameters of the hybrid processing module according to the target loss.

[0121] Optionally, the fine-tuning module 402 further includes:

[0122] The triggering submodule is used to, after the first updating submodule updates the dynamic category hint vector and the model parameters of the original image classification model according to each predicted classification result and each labeled classification result, trigger the acquisition submodule to acquire multiple sample images and the labeled classification result corresponding to each sample image in response to the failure to meet the training cutoff condition.

[0123] The third determining submodule is used to obtain the target image classification model in response to the satisfaction of the training cutoff condition.

[0124] Figure 5 This is a block diagram illustrating an image classification apparatus according to an exemplary embodiment. Figure 5 As shown, the image classification device 500 includes:

[0125] The acquisition module 501 is used to acquire the image to be classified;

[0126] The classification module 502 is used to generate a target classification result of the image to be classified based on the image to be classified, preset category prompt information, and target category prompt information, using a pre-trained target image classification model. The target image classification model is trained by the image classification model training method provided in this disclosure, and the target category prompt information is dynamic category prompt information learned when the target image classification model is trained.

[0127] When determining the target classification result of the image to be classified, the preset category prompt information is referenced to facilitate the target image classification model to make targeted prompts.

[0128] This disclosure also includes a computer-readable medium having a computer program stored thereon, which, when executed by a processing device, implements the steps of the image classification model training method or the steps of the image classification method provided in this disclosure.

[0129] The following is for reference. Figure 6 The diagram illustrates a structural schematic of an electronic device (e.g., a terminal device or a server) 600 suitable for implementing embodiments of the present disclosure. The terminal device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0130] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0131] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0132] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0133] It should be noted that the computer-readable medium described above can be a computer-readable signal medium or a computer-readable storage medium or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any suitable combination of the above. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program used by or in connection with an instruction execution system, apparatus or device. In the present disclosure, the computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave in a propagated data signal, in which the computer-readable program code is contained. Such a propagated data signal can take many forms, including but not limited to, an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer-readable signal medium can also be any computer-readable medium that can send, propagate or transfer the program for use by or in connection with the instruction execution system, apparatus or device. The program code contained in the computer-readable medium can be transmitted by any suitable medium, including but not limited to, wire, cable, RF (radio frequency), etc., or any suitable combination of the above.

[0134] In some embodiments, the client, server, or both can communicate using any current known or future developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks ("LAN"), wide area networks ("WAN"), the Internet, and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any current known or future developed networks.

[0135] The computer-readable medium described above can be included in the electronic device described above; or can exist separately from the electronic device, and is not assembled into the electronic device.

[0136] The computer readable medium carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device is caused to: pre-train a visual language model to obtain an original image classification model; and fine-tune the original image classification model according to preset class prompt information and dynamic class prompt information learned by the original image classification model in a fine-tuning process to obtain a target image classification model, wherein an initial value of the dynamic class prompt information is empty.

[0137] Alternatively, the computer readable medium carries one or more programs, when the one or more programs are executed by the electronic device, the electronic device is caused to: obtain an image to be classified; and generate a target classification result of the image to be classified by a pre-trained target image classification model according to the image to be classified, preset class prompt information and target class prompt information, wherein the target image classification model is trained by the image classification model training method provided by the present disclosure, and the target class prompt information is dynamic class prompt information learned by the target image classification model when the training is completed.

[0138] Computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as "C" or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0139] The computer program product of the first aspect can include one or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform the operations of the first aspect. The computer program product of the first aspect can include a non-transitory computer-readable medium storing code that, when executed, causes a computer to perform operations for the first aspect.

[0140] The modules described in the embodiments of the present disclosure can be implemented by software, or by hardware. In some cases, the name of a module does not limit the module itself. For example, a obtaining module can also be described as a module that obtains an image to be classified.

[0141] The functions described in this specification can be implemented in part or in whole by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.

[0142] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium can include one or more lines of electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0143] The above description merely illustrates the preferred embodiment of the disclosure and a principle of applied technologies. It should be understood by those skilled in the art that the disclosed range of the disclosure is not limited to the technical solutions formed by the specific combinations of the technical features described above, and should also cover other technical solutions formed by the combinations of the technical features described above or their equivalent features without departing from the disclosed concept. For example, the technical solutions formed by the mutual replacement of the above-described features and the technical features with similar functions disclosed in the disclosure (but not limited to) can be formed.

[0144] Furthermore, although operations are depicted in a particular, sequential order, this should not be understood as requiring or implying that the operations are performed in the order illustrated or sequentially. In certain circumstances, multitasking and parallel processing can be advantageous. Likewise, although specific implementation details are contained in the above discussion, these should not be construed as limiting the scope of the disclosure. Certain features described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination.

[0145] Although the subject matter has been described in language specific to structural features and / or methodological acts, it is to be understood that the subject defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely illustrative of the example forms of implementing the claims. As to the means for performing the operations of the apparatus in the above-described embodiments, the specific manner in which the various modules perform the operations has been described in detail in the embodiments related to the method, and will not be described here in detail.

Claims

1. A method for training an image classification model, characterized in that, The method comprises the following steps: Pre-training a visual language model to obtain an original image classification model, the original image classification model comprising: an image encoder, a first residual module, a text encoder, a second residual module and a hybrid processing module connected in sequence, wherein the image encoder is connected with the second residual module; Fine-tuning the original image classification model according to preset class prompt information and dynamic class prompt information learned by the original image classification model in a fine-tuning process to obtain a target image classification model, wherein an initial value of the dynamic class prompt information is empty; The fine-tuning of the original image classification model according to the preset class prompt information and the dynamic class prompt information learned by the original image classification model in the fine-tuning process comprises: Obtaining a plurality of sample images and a labeled classification result corresponding to each sample image; Vectorizing the preset class prompt information to obtain a preset class prompt vector, and vectorizing the dynamic class prompt information to obtain a dynamic class prompt vector, wherein the preset class prompt vector and the dynamic class prompt vector are both multiple; For each preset class prompt vector, adding the preset class prompt vector and all dynamic class prompt vectors through the first residual module to obtain a first text vector; generating a text feature of each first text vector through the text encoder; for each sample image, generating an image feature of the sample image through the image encoder; adding each pair of text features and the image feature of the sample image through the second residual module to obtain a hybrid feature; generating a second text vector corresponding to each preset image class through the hybrid processing module according to the hybrid feature; determining a predicted classification result of the sample image according to each second text vector and the image feature of the sample image; Updating the dynamic class prompt vector and model parameters of the original image classification model according to each predicted classification result and each labeled classification result.

2. The method of claim 1, wherein, The updating of the dynamic class prompt vector and the model parameters of the original image classification model according to each predicted classification result and each labeled classification result comprises: Calculating a target loss of the original image classification model according to each predicted classification result and each labeled classification result; Updating the dynamic class prompt vector and model parameters of the hybrid processing module according to the target loss.

3. The method according to claim 1 or 2, characterized in that, After the step of updating the dynamic class prompt vector and the model parameters of the original image classification model according to each predicted classification result and each labeled classification result, the fine-tuning of the original image classification model according to the preset class prompt information and the dynamic class prompt information learned by the original image classification model in the fine-tuning process further comprises: in response to the training stop condition not being met, repeating the steps of obtaining a plurality of sample images and a labeled classification result corresponding to each of the sample images, and updating the dynamic class hint vector and the model parameters of the original image classification model according to each of the predicted classification result and the labeled classification result; in response to the training stop condition being met, obtaining the target image classification model.

4. An image classification method characterized by, The method comprises the following steps: obtaining an image to be classified; generating a target classification result of the image to be classified by a pre-trained target image classification model according to the image to be classified, preset class hint information and target class hint information, wherein the target image classification model is trained by the image classification model training method in any one of claims 1-3, and the target class hint information is dynamic class hint information learned by the target image classification model when the training is completed.

5. An image classification model training apparatus, characterized by comprising: The method comprises the following steps: a pre-training module is configured to pre-train a visual language model to obtain an original image classification model, the original image classification model comprising an image encoder, a first residual module, a text encoder, a second residual module and a hybrid processing module connected in sequence, wherein the image encoder is connected with the second residual module; a fine-tuning module is configured to fine-tune the original image classification model according to preset class hint information and dynamic class hint information learned by the original image classification model in a fine-tuning process to obtain a target image classification model, wherein an initial value of the dynamic class hint information is empty; wherein the fine-tuning module comprises: an obtaining sub-module configured to obtain a plurality of sample images and a labeled classification result corresponding to each of the sample images; a vectorization processing sub-module configured to vectorize the preset class hint information to obtain a preset class hint vector, and vectorize the dynamic class hint information to obtain a dynamic class hint vector, wherein the preset class hint vector and the dynamic class hint vector are both multiple; a classification sub-module configured to generate a predicted classification result of each of the sample images by the original image classification model according to the preset class hint vector, the dynamic class hint vector and each of the sample images; a first updating sub-module configured to update the dynamic class hint vector and the model parameters of the original image classification model according to each of the predicted classification result and the labeled classification result; wherein the classification sub-module comprises: a first residual sub-module configured to add all the dynamic class hint vectors to each of the preset class hint vectors by the first residual module to obtain a first text vector; a first generation sub-module configured to generate a text feature of each of the first text vectors by the text encoder; The second generation sub-module is configured to generate, for each of the sample images, image features of the sample image by the image encoder; the second residual sub-module is configured to add each pair of the text features and the image features of the sample image to obtain mixed features by the second residual module; the mixing sub-module is configured to generate, according to the mixed features, a second text vector corresponding to each of the preset image categories by the mixing processing module; and the first determination sub-module is configured to determine a predicted classification result of the sample image according to each of the second text vectors and the image features of the sample image.

6. An image classification apparatus characterized by comprising: The method comprises the following steps: An acquisition module is configured to acquire a to-be-classified image; A classification module is configured to generate a target classification result of the to-be-classified image by a pre-trained target image classification model according to the to-be-classified image, preset category prompt information, and target category prompt information, wherein the target image classification model is trained according to the image classification model training method in any one of claims 1-3, and the target category prompt information is dynamic category prompt information learned by the target image classification model when the training is completed.

7. A computer readable medium having stored thereon a computer program, characterized in that The program is executed by a processing device to implement the steps of the method in any one of claims 1-4.

8. An electronic device, comprising: The method comprises the following steps: A storage device having a computer program stored thereon; A processing device configured to execute the computer program in the storage device to implement the steps of the method in any one of claims 1-4.

Images