Image classification method, device, equipment and storage medium

Abstract

The present invention provides an image classification method, device, equipment and storage medium, the method comprising: obtaining a target image and a target text in a specific field, the target text being a visual description text of the target image; extracting general image features from the target image, and extracting general text features from the target text; obtaining target features corresponding to the target image and the target text respectively according to the general image features and the general text features, the target features corresponding to the target image are target image features integrated with text features of the target text, the target image features include general image features and image features in a specific field, the target features corresponding to the target text are target text features integrated with image features of the target image, and the target text features include general text features and text features in a specific field; determining the category of the target image according to the target features corresponding to the target image and the target text respectively. The present invention can classify images in a specific field more accurately.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an image classification method, device, equipment and storage medium. Background Art

[0002] The image classification task refers to identifying what category an image belongs to. It is an important basic task in computer vision and the basis of many visual tasks.

[0003] Most of the current image classification schemes are image classification schemes for general fields, that is, an image classification model is trained in advance using training samples from the general field, and then the trained image classification model is used to classify the images to be classified.

[0004] At present, the classification of images in specific fields is becoming more and more important. However, the image classification scheme for general fields is not very applicable to images in specific fields, that is, the classification effect is poor when the image classification scheme for general fields is used to classify images in specific fields. Summary of the invention

[0005] In view of this, the present invention provides an image classification method, device, equipment and storage medium to solve the problem that the current image classification scheme is not highly applicable to images in specific fields. The technical solution is as follows:

[0006] In a first aspect, an image classification method is provided, comprising:

[0007] Acquire a target image and a target text in a specific field, wherein the target text is a visual description text of the target image;

[0008] Extracting common image features from the target image, and extracting common text features from the target text;

[0009] According to the general image features and the general text features, target features corresponding to the target image and the target text are obtained, respectively, wherein the target features corresponding to the target image are target image features integrated with text features of the target text, and the target image features include general image features and specific domain image features of the target image, and the target features corresponding to the target text are target text features integrated with image features of the target image, and the target text features include general text features and specific domain text features of the target text;

[0010] The category of the target image is determined according to target features corresponding to the target image and the target text respectively.

[0011] Optionally, determining the category of the target image according to target features corresponding to the target image and the target text respectively includes:

[0012] Recognize entities from the target text to obtain a plurality of entities, wherein each entity is a candidate category of the target image;

[0013] Extracting features corresponding to the entities from the target features corresponding to the target text;

[0014] Calculate the correlation between the feature corresponding to each entity and the target feature corresponding to the target image;

[0015] According to the calculated correlation, the category of the target image is determined from the plurality of entities.

[0016] Optionally, extracting general image features from the target image and extracting general text features from the target text; and acquiring target features corresponding to the target image and the target text respectively according to the general image features and the general text features, comprises:

[0017] Based on a pre-trained image classification model, extracting common image features from the target image and extracting common text features from the target text;

[0018] Based on the image classification model, and taking the general image features and the general text features as a basis, obtaining target features corresponding to the target image and the target text respectively;

[0019] The image classification model is trained using training images of the specific field and visual description texts of the training images.

[0020] Optionally, the acquiring target features corresponding to the target image and the target text respectively based on the general image features and the general text features includes:

[0021] According to the general image features and the general text features, a specific domain image feature integrated with the text features of the target text is obtained to obtain a first fused feature, and a specific domain text feature integrated with the image features of the target image is obtained to obtain a second fused feature;

[0022] The first fusion feature is fused with the universal image feature to obtain a third fusion feature, and the second fusion feature is fused with the universal text feature to obtain a fourth fusion feature;

[0023] The target feature corresponding to the target image is acquired according to the third fusion feature, and the target feature corresponding to the target text is acquired according to the fourth fusion feature.

[0024] Optionally, the acquiring, based on the general image feature and the general text feature, a field-specific image feature integrated with a text feature of the target text to obtain a first fused feature, and acquiring a field-specific text feature integrated with an image feature of the target image to obtain a second fused feature, comprises:

[0025] Integrate the universal text feature into the universal image feature to obtain a first feature, and integrate the universal image feature into the universal text feature to obtain a second feature;

[0026] Performing image feature extraction on the first feature to obtain a third feature, and performing text feature extraction on the second feature to obtain a fourth feature;

[0027] Performing image feature extraction on the third feature to obtain a fifth feature, and performing text feature extraction on the fourth feature to obtain a sixth feature;

[0028] The fourth feature is integrated into the fifth feature to obtain the first fused feature, and the third feature is integrated into the sixth feature to obtain the second fused feature.

[0029] Optionally, acquiring a target feature corresponding to the target image according to the third fusion feature, and acquiring a target feature corresponding to the target text according to the fourth fusion feature, includes:

[0030] Processing the third fused feature into a feature suitable for an image classification task to obtain a fifth fused feature, and processing the fourth fused feature into a feature suitable for an image classification task to obtain a sixth fused feature;

[0031] The fifth fusion feature is fused with the general image feature to obtain a target feature corresponding to the target image, and the sixth fusion feature is fused with the general text feature to obtain a target feature corresponding to the target text.

[0032] Optionally, the image classification model includes an image encoder, a text encoder and an adapter, wherein the image encoder is pre-trained using image samples in a general field, and the text encoder is pre-trained using text samples in a general field;

[0033] The training process of the image classification model includes:

[0034] Acquire common image features of the training image based on the image encoder, and acquire common text features of visual description text of the training image based on the text encoder;

[0035] Based on the adapter, based on the general image features of the training image and the general text features of the visual description text of the training image, target features corresponding to the training image and the visual description text of the training image are obtained respectively, wherein the target features corresponding to the training image are training image features integrated with text features of the visual description text of the training image, the training image features include general image features and domain-specific image features of the training image, and the target features corresponding to the visual description text of the training image are training text features integrated with image features of the training image, and the training text features include general text features and domain-specific text features of the visual description text of the training image;

[0036] A first loss is determined according to target features corresponding to the training image and the visual description text of the training image, and parameters of the image classification model are updated according to the first loss. During the training process, parameters of the image encoder and the text encoder are not updated.

[0037] In a second aspect, an image classification device is provided, comprising: a data acquisition module, a general feature acquisition module, a target feature acquisition module and an image category determination module;

[0038] The data acquisition module is used to acquire a target image and a target text in a specific field, wherein the target text is a visual description text of the target image;

[0039] The general feature acquisition module is used to extract general image features from the target image and general text features from the target text;

[0040] The target feature acquisition module is used to acquire the target features corresponding to the target image and the target text respectively according to the general image features and the general text features, wherein the target features corresponding to the target image are target image features integrated with text features of the target text, and the target image features include general image features and specific domain image features of the target image, and the target features corresponding to the target text are target text features integrated with image features of the target image, and the target text features include general text features and specific domain text features of the target text;

[0041] The image category determination module is used to determine the category of the target image according to target features corresponding to the target image and the target text respectively.

[0042] In a third aspect, an image classification device is provided, comprising: a memory and a processor;

[0043] The memory is used to store programs;

[0044] The processor is used to execute the program to implement each step of any one of the above-mentioned image classification methods.

[0045] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, each step of any of the above-mentioned image classification methods is implemented.

[0046] The present invention provides an image classification method, firstly obtaining a target image in a specific field and a visual description text of the target image, then obtaining the universal visual features of the target image and the universal text features of the visual description text of the target image, then obtaining the target features corresponding to the target image and the target text respectively according to the universal image features and the universal text features (the target features corresponding to the target image are target image features integrated with the text features of the target text, the target image features include the universal image features of the target image and the image features of the specific field, the target features corresponding to the target text are target text features integrated with the image features of the target image, the target text features include the universal text features of the target text and the text features of the specific field), and finally determining the category of the target image according to the target features corresponding to the target image and the target text respectively. The image classification method provided by the present invention takes into account the universal features and the specific field features when classifying the target image in a specific field, on the one hand, and utilizes the multimodal features, and utilizes the multimodal features, while taking into account the universal features and the specific field features, so that the image classification method provided by the present invention can classify the images in the specific field more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 A schematic diagram of a hardware architecture involved in the present invention;

[0049] Figure 2 A schematic diagram of a flow chart of an image classification method provided by an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of the structure of an image classification model provided by an embodiment of the present invention;

[0051] Figure 4 A flowchart of a training method for an image classification model provided by an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of the structure of an adapter in an image classification model provided by an embodiment of the present invention;

[0053] Figure 6 A flowchart of another training method for an image classification model provided by an embodiment of the present invention;

[0054] Figure 7 A schematic diagram of the structure of an image classification device provided by an embodiment of the present invention;

[0055] Figure 8 A schematic diagram of the structure of an image classification device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] In view of the fact that image classification in specific fields is becoming more and more important, and the current image classification schemes (i.e., image classification schemes for general fields) are not highly applicable to images in specific fields, the inventors of this case conducted research. The initial idea was: pre-training with training samples from a specific field to obtain an image classification model for the specific field, and then classifying the images to be classified in the specific field based on the image classification model for the specific field.

[0058] However, the number of training samples in a specific field is usually small. The image classification model trained with a small number of training samples has an overfitting problem. The overfitting problem means that the image classification model has good performance on known samples but poor performance on unknown samples, that is, the generalization ability of the above scheme is poor.

[0059] In response to the problems existing in the above-mentioned ideas, the inventor of this case continued to conduct research. During the research process, it came to the idea that the number of training samples can be increased based on data enhancement technology, that is, new samples can be obtained by rotating, scaling, color jittering, affine transformation and other processing of existing training samples, and the original training samples and the new samples obtained by data enhancement technology can be used to train the image classification model.

[0060] However, the distribution of samples obtained based on data augmentation methods is close to the distribution of original training samples, which means that the image classification model trained using the original training samples and new samples obtained through data augmentation technology still does not have good generalization performance.

[0061] In view of this, the inventor of this case continued to conduct research, and through continuous research, finally proposed an image classification method with better results.

[0062] Before introducing the image classification method provided by the present invention, the hardware architecture involved in the present invention is first described.

[0063] In one possible implementation, Figure 1 As shown, the hardware architecture involved in the present invention may include: an electronic device 101 and a server 102 .

[0064] Exemplarily, the electronic device 101 can be any electronic product that can interact with a user through one or more methods such as a keyboard, touchpad, touch screen, remote control, voice interaction or handwriting device, such as a mobile phone, a laptop computer, a tablet computer, a PDA, a personal computer, a wearable device, a smart TV, a PAD, etc.

[0065] It should be noted that Figure 1 This is just an example. There are many types of electronic devices, not limited to Figure 1 Laptop in the.

[0066] Exemplarily, the server 102 may be a single server, or a server cluster consisting of multiple servers, or a cloud computing server center. The server 102 may include a processor, a memory, a network interface, and the like.

[0067] Exemplarily, the electronic device 101 may establish a connection and communicate with the server 102 via a wireless communication network; exemplary, the electronic device 101 may establish a connection and communicate with the server 102 via a wired network.

[0068] The electronic device 101 obtains the image to be classified and the visual description text of the image to be classified in a specific field, and sends the image to be classified and the visual description text of the image to be classified to the server 102. The server 102 processes the image to be classified and the visual description text of the image to be classified according to the image classification method provided by the present invention to determine the image category, and then sends the image category to the electronic device 101.

[0069] In another possible implementation, the hardware architecture of the present invention may include: an electronic device. The electronic device is a device with strong data processing capability.

[0070] The electronic device can process the to-be-classified images in a specific field and the visual description texts of the to-be-classified images according to the image classification method provided by the present invention to determine the image category.

[0071] Those skilled in the art should understand that the above-mentioned electronic devices and servers are only examples, and other existing or future electronic devices or servers that are applicable to the present invention should also be included in the protection scope of the present invention and are included here by reference.

[0072] Next, the image classification method provided by the present invention is introduced through the following embodiments.

[0073] See also Figure 2 , shows a schematic flow chart of an image classification method provided by an embodiment of the present invention, the method may include:

[0074] Step S201: Acquire a target image and target text in a specific field.

[0075] The target image is the image to be classified, and the target text is the visual description text of the target image. It should be noted that the visual description text of the target image is the text describing the image content of the target image.

[0076] Step S202: extracting common image features from the target image, and extracting common text features from the target text.

[0077] Among them, universal image features refer to image features that are commonly present or applicable in all fields. Similarly, universal text features refer to text features that are commonly present or applicable in all fields.

[0078] Step S203: According to the general image features and the general text features, target features corresponding to the target image and the target text are obtained respectively.

[0079] Among them, the target features corresponding to the target image are target image features that incorporate the text features of the target text. The target image features include two levels of image features, namely, the general image features of the target image and the specific domain image features of the target image. Similarly, the target features corresponding to the target text are target text features that incorporate the image features of the target image. The target text features include two levels of features, namely, the general text features of the target text and the specific domain text features of the target text. It should be noted that the specific domain image features are image features adapted to a specific domain, which are related to the classification task in a specific domain. Similarly, the specific domain text features are text features adapted to a specific domain, which are related to the classification task in a specific domain.

[0080] In order to ultimately obtain accurate classification results, the present invention combines general image features with general text features to determine features that include both image features and are integrated with text features, that is, target features corresponding to the target image, and combines general text features with general image features to determine features that include both text features and are integrated with image features, that is, target features corresponding to the target text.

[0081] Step S204: Determine the category of the target image according to the target features corresponding to the target image and the target text respectively.

[0082] Specifically, firstly, entities are identified from the target text to obtain several entities, and then features corresponding to several entities are extracted from the target features corresponding to the target text. Then, the correlation between the features corresponding to each entity and the target features corresponding to the target image is calculated. Finally, the category of the target image is determined from the several entities according to the calculated correlation.

[0083] When identifying entities from the target text, an existing entity recognition method may be used to identify entities from the target text, and each identified entity is a candidate category of the target image.

[0084] After calculating the correlation between the features corresponding to each entity and the target features corresponding to the target image, in one possible implementation, N (N is a positive integer, and the specific value of N can be determined according to the specific application scenario) entities whose corresponding features have the largest correlation with the target features corresponding to the target image can be determined as the categories of the target image; in another possible implementation, entities whose corresponding features have a correlation with the target features corresponding to the target image greater than a preset correlation threshold can be determined as the category of the target image; in yet another possible implementation, entities whose corresponding features have the largest correlation with the target features corresponding to the target image and are greater than a preset correlation threshold can be determined as the category of the target image.

[0085] The embodiment of the present invention provides an image classification method, firstly obtaining a target image in a specific field and a visual description text of the target image, then obtaining the general visual features of the target image and the general text features of the visual description text of the target image, then obtaining the target features corresponding to the target image and the target text respectively according to the general image features and the general text features (the target features corresponding to the target image are target image features integrated with the text features of the target text, the target image features include the general image features of the target image and the image features of the specific field, the target features corresponding to the target text are target text features integrated with the image features of the target image, the target text features include the general text features of the target text and the text features of the specific field), and finally determining the category of the target image according to the target features corresponding to the target image and the target text. The image classification method provided by the embodiment of the present invention takes into account the general features and the specific field features when classifying the target image in a specific field, on the one hand, and utilizes the multimodal features, and utilizes the multimodal features, while taking into account the general features and the specific field features, so that the image classification method provided by the embodiment of the present invention can classify the images in the specific field more accurately.

[0086] In a possible implementation, the image classification method provided in the above embodiment may be implemented based on a pre-trained image classification model, wherein the image classification model is trained using training images and visual description texts of the training images.

[0087] It should be noted that this embodiment does not limit the specific implementation form of the image classification method provided in the above embodiment, that is, this embodiment does not limit the image classification method provided in the above embodiment to be implemented based on a model, for example, it can also be implemented based on rules.

[0088] Next, the image classification method provided in the above embodiment is further introduced by taking the model-based implementation as an example.

[0089] Before introducing the process of implementing image classification based on the pre-trained image classification model, we first Figure 3 Taking the image classification model shown in the figure as an example, the training process of the image classification model is introduced.

[0090] Figure 3 The image classification model shown may include an image encoder 301I, a text encoder 301T, and an adapter 302. The image encoder 301I is pre-trained using a large number of image samples in a general field, and the text encoder is pre-trained using a large number of text samples in a general field. It should be noted that during the training process of the image classification model, the parameters of the image encoder 301I and the text encoder 301T are fixed, that is, the parameters of the image encoder 301I and the text encoder 301T are not updated.

[0091] There are many ways to train an image classification model. This embodiment provides the following two training methods.

[0092] See also Figure 4 , which shows a flow chart of a training method for an image classification model, which may include:

[0093] Step S401: obtaining original sample images and visual description texts of the original sample images from a training sample set as training images and visual description texts of the training images.

[0094] It should be noted that the training sample set includes multiple pieces of sample data, and each piece of sample data includes an original sample image and a visual description text of the original sample image.

[0095] Step S402: The image encoder 301I based on the image classification model obtains the common image features of the training image, and the text encoder 301T based on the image classification model obtains the common text features of the visual description text of the training image.

[0096] Since the image encoder 301I is pre-trained with a large number of image samples in general fields, when the training image is input into the image encoder 301I of the image classification model, the image encoder 301I can output the general image features of the training image. Similarly, since the text encoder 301T is pre-trained with a large number of text samples in general fields, when the visual description text of the training image is input into the text encoder 301T of the image classification model, the text encoder 301T can output the general text features of the visual description text of the training image.

[0097] Step S403: The adapter 302 based on the image classification model obtains target features corresponding to the training image and the visual description text of the training image respectively based on the general image features of the training image and the general text features of the visual description text of the training image.

[0098] Among them, the target features corresponding to the training images are training image features that incorporate text features of the visual description text of the training images, and the training image features include general image features and specific domain image features of the training images. The target features corresponding to the visual description text of the training images are training text features that incorporate image features of the training images, and the training text features include general text features and specific domain text features of the visual description text of the training images.

[0099] The adapter 302 based on the image classification model, based on the general image features of the training image and the general text features of the visual description text of the training image, obtains the target features corresponding to the training image and the visual description text of the training image respectively, which may include:

[0100] Step S4031, the adapter 302 based on the image classification model, based on the general image features of the training image and the general text features of the visual description text of the training image, obtains the specific domain image features integrated with the text features of the visual description text of the training image to obtain the first fusion feature, and obtains the specific domain text features integrated with the image features of the training image to obtain the second fusion feature.

[0101] Specifically, the implementation process of step S4031 may include:

[0102] Step a1, the adapter 302 based on the image classification model integrates the common text features of the visual description text of the training image into the common image features of the training image to obtain the first feature, and the adapter 302 based on the image classification model integrates the common image features of the training image into the common text features of the visual description text of the training image to obtain the second feature.

[0103] Optional, such as Figure 5 As shown, the adapter 302 of the image classification model may include: a first feature adjustment module 501, a first feature splicing module 502I and a second feature splicing module 502T. The general text features of the visual description text of the training image may be adjusted to features that can be spliced ​​with the general image features of the training image based on the first feature adjustment module 501 to obtain adjusted text features. The adjusted text features may be spliced ​​with the general image features of the training image based on the first feature splicing module 502I to obtain the first features. Similarly, the general image features of the training image may be adjusted to features that can be spliced ​​with the general text features of the visual description text of the training image based on the first feature adjustment module 501 to obtain adjusted image features. The adjusted image features may be spliced ​​with the general text features of the visual description text of the training image based on the second feature splicing module 502T to obtain the second features.

[0104] Optionally, the first feature adjustment module 501 may be composed of a linear layer (such as two linear layers) and a ReLu activation function, which adjusts the dimension of the input feature so as to be concatenated with features of another modality.

[0105] Step a2: The adapter 302 based on the image classification model performs image feature extraction on the first feature to obtain a third feature, and the adapter 302 based on the image classification model performs text feature extraction on the second feature to obtain a fourth feature.

[0106] like Figure 5As shown, the adapter 302 of the image classification model may include a first image feature extraction module 503I and a first text feature extraction module 503T. The first image feature extraction module 503I may perform image feature extraction on the first feature to obtain a third feature, and the first text feature extraction module 503T may perform text feature extraction on the second feature to obtain a fourth feature.

[0107] Optionally, the first image feature extraction module 503I may be but is not limited to a RepVGG module, which focuses on extracting image features for the first feature; and the first text feature extraction module 503T may be but is not limited to a BERT module, which focuses on extracting text features for the second feature.

[0108] It should be noted that since the first feature is the concatenation of features of two modalities, the third feature obtained by performing image feature extraction on the first feature includes not only image features but also text features, but mainly image features. Similarly, the fourth feature includes not only text features but also image features, but mainly text features.

[0109] Step a3: The adapter 302 based on the image classification model performs image feature extraction on the third feature to obtain the fifth feature, and the adapter 302 based on the image classification model performs text feature extraction on the fourth feature to obtain the sixth feature.

[0110] like Figure 5 As shown, the adapter 302 of the image classification model may include a second image feature extraction module 504I and a second text feature extraction module 504T. The image feature extraction module 504I may be used to extract the third feature to obtain the fifth feature, and the text feature extraction module 504T may be used to extract the fourth feature to obtain the sixth feature.

[0111] Optionally, the second image feature extraction module 504I may be but is not limited to a RepGhost module, and the second text feature extraction module 504T may be but is not limited to a BERT module.

[0112] Step a4: the adapter 302 based on the image classification model integrates the fourth feature into the fifth feature to obtain a first fused feature, and the adapter 302 based on the image classification model integrates the third feature into the sixth feature to obtain a second fused feature.

[0113] like Figure 5As shown, the adapter 302 of the image classification model may include a second feature adjustment module 505, a third feature splicing module 506I and a fourth feature splicing module 506T. The fourth feature may be adjusted to a feature that can be spliced ​​with the fifth feature based on the second feature adjustment module 505, and the adjusted feature may be spliced ​​with the fifth feature based on the third feature splicing module 506I to obtain a first fused feature. Similarly, the third feature may be adjusted to a feature that can be spliced ​​with the sixth feature based on the second feature adjustment module 505, and the adjusted feature may be spliced ​​with the sixth feature based on the fourth feature splicing module 506T to obtain a second fused feature.

[0114] Optionally, the second feature adjustment module 505 may be composed of a linear layer (such as two linear layers) and a ReLu activation function, which adjusts the dimension of the input feature so as to be concatenated with the features of another modality.

[0115] After steps a1 to a4, the full integration and information supplement of the two modal features can be achieved.

[0116] Step S4032: The adapter 302 based on the image classification model fuses the first fused feature with the general image feature of the training image to obtain a third fused feature, and fuses the second fused feature with the general text feature of the visual description text of the training image to obtain a fourth fused feature.

[0117] After the feature processing process of steps a1 to a4, the general features will be forgotten. In order to avoid the overfitting problem of the model, after obtaining the first fusion feature and the second fusion feature, the first fusion feature can be fused with the general image feature, and the second fusion feature can be fused with the general text feature.

[0118] Optionally, the adapter 302 of the image classification model can be based on an attention mechanism to fuse the first fused feature with the common image feature of the training image, and the adapter 302 of the image classification model can be based on an attention mechanism to fuse the second fused feature with the common text feature of the visual description text of the training image.

[0119] like Figure 5The adapter 302 of the image classification model may include a first attention module 507I and a second attention module 507T, wherein the first attention module 507I determines the weights corresponding to the first fusion feature and the general image features of the training image, and then weights the corresponding features according to the weights corresponding to the first fusion feature and the general image features of the training image, and then sums them up. The weighted sum feature is the first fusion feature. Similarly, the second attention module 507T determines the weights corresponding to the second fusion feature and the general text features of the visual description text of the training image, and then weights the corresponding features according to the weights corresponding to the second fusion feature and the general text features of the visual description text of the training image, and then sums them up. The weighted sum feature is the second fusion feature.

[0120] Step S4033, the adapter 302 based on the image classification model obtains the target feature corresponding to the training image based on the third fusion feature, and the adapter 302 based on the image classification model obtains the target feature corresponding to the visual description text of the training image based on the fourth fusion feature.

[0121] There are multiple implementation methods of step S4033: in one possible implementation method, the third fused feature can be processed into a feature suitable for the image classification task based on the adapter 302 of the image classification model, as the target feature corresponding to the training image; the fourth fused feature can be processed into a feature suitable for the image classification task based on the adapter 302 of the image classification model, as the target feature corresponding to the visual description text of the training image; in another possible implementation method, the third fused feature can be first processed into a feature suitable for the image classification task as the fifth fused feature, and the fourth fused feature can be processed into a feature suitable for the image classification task as the sixth fused feature, and then the fifth fused feature is fused with the general image feature of the training image, and the fused feature is used as the target feature corresponding to the training image, and the sixth fused feature is fused with the general text feature of the visual description text of the training image, and the fused feature is used as the target feature corresponding to the visual description text of the training image.

[0122] Optionally, the fifth fusion feature can be fused with the general image feature of the training image by using an attention mechanism based on the adapter 302 of the image classification model, and the sixth fusion feature can be fused with the general text feature of the visual description text of the training image by using an attention mechanism based on the adapter 302 of the image classification model.

[0123] For the second implementation method mentioned above, Figure 5As shown, the adapter 302 of the image classification model may include a third feature adjustment module 508I, a fourth feature adjustment module 508T, a third attention module 509I and a fourth attention module 509T. The third feature adjustment module 508I processes the third fused feature into a feature adapted to the image classification task to obtain a fifth fused feature, the fourth feature adjustment module 508T processes the fourth fused feature into a feature adapted to the image classification task to obtain a sixth fused feature, the third attention module 509I determines the weights corresponding to the fifth fused feature and the general image feature, and then weights the corresponding features according to the weights corresponding to the fifth fused feature and the general image feature, and then sums them up, and the weighted summed feature is the target feature corresponding to the training image, the fourth attention module 509T determines the weights corresponding to the sixth fused feature and the general text feature, and then weights the corresponding features according to the weights corresponding to the sixth fused feature and the general text feature, and then sums them up, and the weighted summed feature is the target feature corresponding to the visual description text of the training image.

[0124] Optionally, the third feature adjustment module 508I and the fourth feature adjustment module 508T may be composed of a linear layer (such as two linear layers) and a ReLu activation function.

[0125] Step S404: Determine a first loss according to target features corresponding to the training image and the visual description text of the training image, respectively.

[0126] Specifically, a contrast loss may be determined for a target feature corresponding to a training image and a target feature corresponding to a visual description text of the training image as the first loss.

[0127] Step S405: Update the parameters of the image classification model according to the first loss.

[0128] Update the parameters of other parts of the image classification model except the image encoder and text encoder.

[0129] The image classification model is trained multiple times according to the process of steps S401 to S405 above until the training end condition is met (for example, the set number of training times is reached).

[0130] In order to improve the feature extraction capability of the image classification model, this embodiment provides another training method for the image classification model. Figure 6 , showing a flow chart of another training method of an image classification model, which may include:

[0131] Step S601: Acquire original sample images and visual description text of the original sample images from a training sample set.

[0132] Step S602: preprocessing the original sample image, using the preprocessed image as a training image, and using the visual description text of the original sample image as the visual description text of the training image.

[0133] Among them, the preprocessing of the original sample image may include: masking some image blocks in the original sample image (for example, replacing some image blocks in the original sample image with image blocks with all pixel values ​​of 0, i.e., black image blocks), and the masked image blocks in the original sample image can be randomly selected.

[0134] Step S603: The image encoder based on the image classification model obtains the common image features of the training image, and the text encoder based on the image classification model obtains the common text features of the visual description text of the training image.

[0135] The specific implementation process of step S603 is similar to the specific implementation process of step S402, and is not described in detail in this embodiment.

[0136] Step S604: The adapter based on the image classification model obtains target features corresponding to the training image and the visual description text of the training image respectively based on the general image features of the training image and the general text features of the visual description text of the training image.

[0137] Specifically, the implementation process of step S604 may include: based on the adapter of the image classification model, integrating the general text features of the visual description text of the training image into the general image features of the training image to obtain a first feature, and integrating the general image features of the training image into the general text features of the visual description text of the training image to obtain a second feature; based on the adapter of the image classification model, performing image feature extraction on the first feature to obtain a third feature, and performing text feature extraction on the second feature to obtain a fourth feature; based on the adapter of the image classification model, performing image feature extraction on the third feature to obtain a fifth feature, and performing text feature extraction on the fourth feature to obtain a sixth feature; based on the adapter of the image classification model, integrating the fourth feature into the fifth feature to obtain a first fusion feature, and integrating the third feature into the sixth feature to obtain to the second fused feature; based on the adapter of the image classification model, the first fused feature is fused with the general image feature of the training image to obtain the third fused feature, and the second fused feature is fused with the general text feature of the visual description text of the training image to obtain the fourth fused feature; based on the adapter of the image classification model, the third fused feature is processed into a feature suitable for the image classification task to obtain the fifth fused feature, and the fourth fused feature is processed into a feature suitable for the image classification task as the sixth fused feature; based on the adapter of the image classification model, the fifth fused feature is fused with the general image feature of the training image, and the fused feature is used as the target feature corresponding to the training image, and the sixth fused feature is fused with the general text feature of the visual description text of the training image, and the fused feature is used as the target feature corresponding to the visual description text of the training image.

[0138] It should be noted that the specific implementation process of the adapter based on the image classification model to obtain the above-mentioned features can be found in the relevant part of the above-mentioned first training method, which will not be described in detail in this embodiment.

[0139] Step S605: restore the training image according to the intermediate features generated by the adapter of the image classification model in the process of acquiring the target features corresponding to the training image to obtain a restored image.

[0140] Specifically, the intermediate features may be input into a decoder for decoding to obtain a restored image. Optionally, the decoder may include a convolution module, a deconvolution operation, and an upsampling module in sequence.

[0141] The intermediate feature in this step may be the third fusion feature mentioned above.

[0142] Step S606: determining a first loss according to target features corresponding to the training image and the visual description text of the training image, respectively, and determining a second loss according to the restored image and the original sample image.

[0143] Specifically, a contrast loss can be determined for the target features corresponding to the training image and the target features corresponding to the visual description text of the training image as the first loss, and an MSE (mean square error) loss can be determined for the restored image and the original sample image as the second loss.

[0144] When determining the MSE loss for the restored image and the original sample image, in one possible implementation, the MSE loss can be determined for the entire restored image and the entire original sample image. In another possible implementation, the MSE loss can be determined for the image block at the masked position in the restored image and the image block at the masked position in the original sample image. In yet another possible implementation, the MSE losses of the above two aspects can be determined simultaneously, and then the MSE losses of the above two aspects are fused (for example, weighted summation, and the weights can be set according to the specific scenario), and the fused MSE loss is used as the final MSE loss, that is, the second loss.

[0145] Step S607: Update the parameters of the image classification model according to the first loss and the second loss.

[0146] The image classification model is trained multiple times according to the process of steps S601 to S607 above until the training end condition is met (for example, the set number of training times is reached).

[0147] In order to improve the classification ability of the image classification model on difficult samples, the above preprocessing of the original sample image can also include: replacing some image blocks in the original image sample with image blocks of difficult samples corresponding to the original image sample (the image blocks replaced in the original image sample can be random).

[0148] It should be noted that the difficult samples corresponding to the original image samples are samples that are difficult to distinguish from the original image samples. The difficult samples corresponding to the original image samples can be determined based on the image classification model. If the category of the original image sample is A, and the category determined for the original image sample based on the image classification model is B, then the image sample of category B is the difficult sample corresponding to the original image sample.

[0149] Taking into account that the classification ability of the image classification model is not good at the initial stage of training, image samples with categories different from the original image samples can be used as difficult samples corresponding to the original image samples. After multiple rounds of training (for example, 50 rounds), the difficult samples corresponding to the original image samples can be determined based on the trained image classification model.

[0150] The method of determining the difficult sample corresponding to the original image sample based on the image classification model can be: obtaining the target features corresponding to the original image sample and the visual description text of the original image sample respectively based on the image classification model; identifying entities from the visual description text of the original image sample to obtain several entities; extracting the features corresponding to each entity from the target features corresponding to the visual description text of the original image sample; calculating the correlation between the features corresponding to each entity and the target features corresponding to the original image sample; eliminating the category of the original image sample from the several entities, and taking the entity whose corresponding features among the remaining entities have the largest correlation with the target features corresponding to the original image sample as the target category, and taking the image sample of the target category as the difficult sample corresponding to the original image sample.

[0151] It should also be noted that, regardless of the first or second training method mentioned above, in the process of training the image classification model, the parameters of the image encoder and the text encoder are frozen. Freezing the parameters of the two encoders can retain the knowledge learned by the two on a large number of samples in the general field, and can also improve the training efficiency of the model.

[0152] The image classification model in the present invention not only learns the features of small samples in the specified field, but also takes into account the features learned on large samples in the general field, which enables the image classification model to have better generalization ability. In addition, the image classification model provided by the present invention can perform multi-stage interactive integration of the features of the two modalities, which enables the features of the two modalities to be fully integrated, thereby improving the model effect. Furthermore, masking part of the image blocks in the original image sample and combining the decoder and the second loss can improve the feature extraction ability of the image classification model, and replacing part of the image blocks in the original image sample with image blocks of difficult samples can improve the classification ability of the image classification model for difficult samples.

[0153] After the training is completed, the target image and target text (visual description text of the target image) in a specific field can be obtained, and the general image features of the target image and the general text features of the target text can be extracted based on the trained image classification model. Based on the trained image classification model, the target features corresponding to the target image and the target text are obtained based on the general image features and the general text features, and then the category of the target image is determined according to the target features corresponding to the target image and the target text.

[0154] Among them, the image encoder based on the image classification model can extract common image features from the target image, and the text encoder based on the image classification model can extract common text features from the target text.

[0155] Taking the image classification model trained based on the second training method as an example, based on the trained image classification model, and based on the general image features of the target image and the general text features of the target text, the process of obtaining the target features corresponding to the target image and the target text respectively may include:

[0156] Step b1, based on the image classification model, based on the general image features of the target image and the general text features of the target text, obtain the specific field image features integrated with the text features of the target text to obtain the first fusion feature, and obtain the specific field text features integrated with the image features of the target image to obtain the second fusion feature.

[0157] Specifically, first, based on the adapter of the image classification model, the universal text features of the target text are integrated into the universal image features of the target image to obtain the first feature, and the universal image features of the target image are integrated into the universal text features of the target text to obtain the second feature. Then, based on the adapter of the image classification model, image feature extraction is performed on the first feature to obtain the third feature, and text feature extraction is performed on the second feature to obtain the fourth feature. Next, based on the adapter of the image classification model, image feature extraction is performed on the third feature to obtain the fifth feature, and text feature extraction is performed on the fourth feature to obtain the sixth feature. Finally, based on the adapter of the image classification model, the fourth feature is integrated into the fifth feature to obtain the first fused feature, and the third feature is integrated into the sixth feature to obtain the second fused feature.

[0158] Step b2: Based on the image classification model, the first fusion feature is fused with the general image feature of the target image to obtain a third fusion feature, and the second fusion feature is fused with the general text feature of the target text to obtain a fourth fusion feature.

[0159] Based on the adapter of the image classification model, the first fused feature can be fused with the general image feature of the target image by using the attention mechanism, and the second fused feature can be fused with the general text feature of the target text by using the attention mechanism.

[0160] Step b3: Based on the image classification model, the target feature corresponding to the target image is obtained based on the third fusion feature, and the target feature corresponding to the target text is obtained based on the fourth fusion feature.

[0161] Specifically, first, based on the adapter of the image classification model, the third fused feature is processed into a feature suitable for the image classification task to obtain the fifth fused feature, and the fourth fused feature is processed into a feature suitable for the image classification task to obtain the sixth fused feature. Then, based on the adapter of the image classification model, the fifth fused feature is fused with the general image feature of the target image to obtain the target feature corresponding to the target image, and the sixth fused feature is fused with the general text feature of the target text to obtain the target feature corresponding to the target text.

[0162] Based on the adapter of the image classification model, the fifth fusion feature can be fused with the general image feature of the target image by using the attention mechanism, and the sixth fusion feature can be fused with the general text feature of the target text by using the attention mechanism.

[0163] It should be noted that the process of obtaining the target features corresponding to the target image and the target text respectively based on the image classification model is similar to the process of obtaining the target features corresponding to the training image and the visual description text of the training image respectively based on the image classification model. For a more specific implementation process of obtaining the target features corresponding to the target image and the target text respectively based on the image classification model, please refer to the specific implementation process of obtaining the target features corresponding to the training image and the visual description text of the training image respectively based on the image classification model, which will not be elaborated in this embodiment.

[0164] In addition, if the image classification model is trained using the first training method mentioned above, the implementation process of obtaining the target features corresponding to the target image and the target text respectively based on the image classification model can refer to the implementation process of obtaining the target features corresponding to the training image and the visual description text of the training image respectively based on the image classification model in the first training method mentioned above, and this embodiment will not be repeated here.

[0165] An embodiment of the present invention provides an image classification device. The image classification device provided by the embodiment of the present invention is described below. The image classification device described below and the image classification method described above can refer to each other.

[0166] See also Figure 7 , shows a schematic diagram of the structure of an image classification device provided in an embodiment of the present invention, the image classification device may include: a data acquisition module 701, a general feature acquisition module 702, a target feature acquisition module 703 and an image category determination module 704.

[0167] The data acquisition module 701 is used to acquire a target image and a target text in a specific field, wherein the target text is a visual description text of the target image.

[0168] The general feature acquisition module 702 is used to extract general image features from the target image and general text features from the target text.

[0169] The target feature acquisition module 703 is used to acquire the target features corresponding to the target image and the target text respectively according to the general image features and the general text features.

[0170] Among them, the target features corresponding to the target image are target image features integrated with the text features of the target text, and the target image features include general image features and specific domain image features of the target image. The target features corresponding to the target text are target text features integrated with the image features of the target image, and the target text features include general text features and specific domain text features of the target text.

[0171] The image category determination module 704 is used to determine the category of the target image according to the target features corresponding to the target image and the target text respectively.

[0172] Optionally, the image category determination module 704 may include: an entity recognition submodule, an entity feature acquisition submodule, a relevance determination submodule and an image category determination submodule.

[0173] The entity recognition submodule is used to recognize entities from the target text to obtain a plurality of entities, wherein each entity is a candidate category of the target image.

[0174] The entity feature acquisition submodule is used to extract the features corresponding to the several entities from the target features corresponding to the target text.

[0175] The relevance determination submodule is used to calculate the relevance between the feature corresponding to each entity and the target feature corresponding to the target image.

[0176] The image category determination submodule is used to determine the category of the target image from the plurality of entities according to the calculated correlation.

[0177] Optionally, when the general feature acquisition module 702 extracts the general image features from the target image and extracts the general text features from the target text, it is specifically used to:

[0178] Based on a pre-trained image classification model, common image features are extracted from the target image, and common text features are extracted from the target text.

[0179] When the target feature acquisition module 703 acquires the target features corresponding to the target image and the target text respectively according to the general image features and the general text features, it is specifically used to:

[0180] Based on the image classification model, and taking the general image features and the general text features as a basis, obtaining target features corresponding to the target image and the target text respectively;

[0181] The image classification model is trained using training images of the specific field and visual description texts of the training images.

[0182] Optionally, when the target feature acquisition module 703 acquires the target features corresponding to the target image and the target text respectively based on the image classification model and the general image features and the general text features, it is specifically used to perform based on the image classification model:

[0183] Based on the image classification model, taking the general image features and the general text features as the basis, obtaining the specific domain image features integrated with the text features of the target text to obtain the first fused features, and obtaining the specific domain text features integrated with the image features of the target image to obtain the second fused features;

[0184] Based on the image classification model, the first fusion feature is fused with the general image feature to obtain a third fusion feature, and the second fusion feature is fused with the general text feature to obtain a fourth fusion feature;

[0185] Based on the image classification model, the target feature corresponding to the target image is obtained based on the third fusion feature, and the target feature corresponding to the target text is obtained based on the fourth fusion feature.

[0186] Optionally, the target feature acquisition module 703 acquires the specific domain image features integrated with the text features of the target text based on the image classification model and based on the general image features and the general text features to obtain the first fused features, and acquires the specific domain text features integrated with the image features of the target image to obtain the second fused features, specifically for:

[0187] Based on the image classification model, the universal text feature is integrated into the universal image feature to obtain a first feature, and the universal image feature is integrated into the universal text feature to obtain a second feature;

[0188] Based on the image classification model, performing image feature extraction on the first feature to obtain a third feature, and performing text feature extraction on the second feature to obtain a fourth feature;

[0189] Based on the image classification model, performing image feature extraction on the third feature to obtain a fifth feature, and performing text feature extraction on the fourth feature to obtain a sixth feature;

[0190] Based on the image classification model, the fourth feature is integrated into the fifth feature to obtain the first fusion feature, and the third feature is integrated into the sixth feature to obtain the second fusion feature.

[0191] Optionally, when the target feature acquisition module 703 acquires the target feature corresponding to the target image based on the third fusion feature based on the image classification model, and acquires the target feature corresponding to the target text based on the fourth fusion feature, it is specifically used to include:

[0192] Based on the image classification model, the third fused feature is processed into a feature adapted to the image classification task to obtain a fifth fused feature, and the fourth fused feature is processed into a feature adapted to the image classification task to obtain a sixth fused feature;

[0193] Based on the image classification model, the fifth fusion feature is fused with the general image feature to obtain the target feature corresponding to the target image, and the sixth fusion feature is fused with the general text feature to obtain the target feature corresponding to the target text.

[0194] Optionally, when the target feature acquisition module 703 fuses the first fusion feature with the general image feature based on the image classification model to obtain a third fusion feature, and fuses the second fusion feature with the general text feature to obtain a fourth fusion feature, it is specifically used to:

[0195] Based on the image classification model, the first fusion feature is fused with the general image feature by using an attention mechanism to obtain a third fusion feature, and the second fusion feature is fused with the general text feature by using an attention mechanism to obtain a fourth fusion feature;

[0196] Optionally, when the target feature acquisition module 703 fuses the fifth fusion feature with the general image feature based on the image classification model to obtain the target feature corresponding to the target image, and fuses the sixth fusion feature with the general text feature to obtain the target feature corresponding to the target text, it is specifically used to:

[0197] Based on the image classification model, the fifth fusion feature is fused with the general image feature using an attention mechanism to obtain the target feature corresponding to the target image, and the sixth fusion feature is fused with the general text feature using an attention mechanism to obtain the target feature corresponding to the target text.

[0198] The optional image classification model includes an image encoder, a text encoder and an adapter, wherein the image encoder is pre-trained using image samples from a general field, and the text encoder is pre-trained using text samples from a general field.

[0199] The image classification device provided in the embodiment of the present invention may further include: a model training module. The model training module is used to:

[0200] Acquire common image features of the training image based on the image encoder, and acquire common text features of visual description text of the training image based on the text encoder;

[0201] Based on the adapter, based on the general image features of the training image and the general text features of the visual description text of the training image, target features corresponding to the training image and the visual description text of the training image are obtained respectively, wherein the target features corresponding to the training image are training image features integrated with text features of the visual description text of the training image, the training image features include general image features and domain-specific image features of the training image, and the target features corresponding to the visual description text of the training image are training text features integrated with image features of the training image, and the training text features include general text features and domain-specific text features of the visual description text of the training image;

[0202] A first loss is determined according to target features corresponding to the training image and the visual description text of the training image, and parameters of the image classification model are updated according to the first loss. During the training process, parameters of the image encoder and the text encoder are not updated.

[0203] Optionally, the training image is an image obtained by preprocessing an original image sample obtained from a training sample set in a specific field, wherein the preprocessing includes masking part of the image blocks in the original image sample; and the model training module is further used for:

[0204] Restoring the training image according to the intermediate features generated by the adapter in the process of acquiring the target features corresponding to the training image to obtain a restored image;

[0205] Determining a second loss according to the restored image and the original image sample;

[0206] The updating of parameters of the image classification model according to the first loss includes:

[0207] Parameters of the image classification model are updated according to the first loss and the second loss.

[0208] Optionally, the preprocessing further includes: replacing part of the image blocks in the original image sample with image blocks of difficult samples corresponding to the original image sample.

[0209] When classifying target images in a specific field, the image classification device provided by the embodiment of the present invention takes into account both general features and specific field features on the one hand, and utilizes multimodal features on the other hand. By utilizing multimodal features and taking into account both general features and specific field features at the same time, the image classification device provided by the embodiment of the present invention can classify images in a specific field more accurately.

[0210] The present invention provides an image classification device. Figure 8 , shows a schematic diagram of the structure of the image classification device, the image classification device may include: a processor 801, a communication interface 802, a memory 803 and a communication bus 804;

[0211] In the embodiment of the present invention, the number of the processor 801, the communication interface 802, the memory 803, and the communication bus 804 is at least one, and the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804;

[0212] The processor 801 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present invention;

[0213] The memory 803 may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0214] The memory stores a program, and the processor can call the program stored in the memory, wherein the program is used to:

[0215] Acquire a target image and a target text in a specific field, wherein the target text is a visual description text of the target image;

[0216] Extracting common image features from the target image, and extracting common text features from the target text;

[0217] According to the general image features and the general text features, target features corresponding to the target image and the target text are obtained, respectively, wherein the target features corresponding to the target image are target image features integrated with text features of the target text, and the target image features include general image features and specific domain image features of the target image, and the target features corresponding to the target text are target text features integrated with image features of the target image, and the target text features include general text features and specific domain text features of the target text;

[0218] The category of the target image is determined according to target features corresponding to the target image and the target text respectively.

[0219] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0220] An embodiment of the present invention further provides a readable storage medium, which may store a program suitable for execution by a processor, wherein the program is used to:

[0221] Acquire a target image and a target text in a specific field, wherein the target text is a visual description text of the target image;

[0222] Extracting common image features from the target image, and extracting common text features from the target text;

[0223] According to the general image features and the general text features, target features corresponding to the target image and the target text are obtained, respectively, wherein the target features corresponding to the target image are target image features integrated with text features of the target text, and the target image features include general image features and specific domain image features of the target image, and the target features corresponding to the target text are target text features integrated with image features of the target image, and the target text features include general text features and specific domain text features of the target text;

[0224] The category of the target image is determined according to target features corresponding to the target image and the target text respectively.

[0225] Optionally, the detailed functions and extended functions of the program may refer to the above description.

[0226] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0227] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0228] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image classification method, characterized in that: include: Acquire a target image and a target text in a specific field, wherein the target text is a visual description text of the target image; Extracting common image features from the target image, and extracting common text features from the target text; According to the general image features and the general text features, target features corresponding to the target image and the target text are obtained, respectively, wherein the target features corresponding to the target image are target image features integrated with text features of the target text, and the target image features include general image features and specific domain image features of the target image, and the target features corresponding to the target text are target text features integrated with image features of the target image, and the target text features include general text features and specific domain text features of the target text; Determining the category of the target image according to target features corresponding to the target image and the target text respectively; The step of extracting general image features from the target image and general text features from the target text; and obtaining target features corresponding to the target image and the target text respectively according to the general image features and the general text features, comprises: Based on a pre-trained image classification model, extracting common image features from the target image and extracting common text features from the target text; Based on the image classification model, and taking the general image features and the general text features as a basis, obtaining target features corresponding to the target image and the target text respectively; The image classification model is trained using training images of the specific field and visual description texts of the training images.

2. The image classification method according to claim 1, characterized in that: The determining the category of the target image according to the target features respectively corresponding to the target image and the target text comprises: Recognize entities from the target text to obtain a plurality of entities, wherein each entity is a candidate category of the target image; Extracting features corresponding to the entities from the target features corresponding to the target text; Calculate the correlation between the feature corresponding to each entity and the target feature corresponding to the target image; According to the calculated correlation, the category of the target image is determined from the plurality of entities.

3. The image classification method according to claim 1, characterized in that: The step of obtaining target features corresponding to the target image and the target text respectively based on the general image features and the general text features includes: According to the general image features and the general text features, a specific domain image feature integrated with the text features of the target text is obtained to obtain a first fused feature, and a specific domain text feature integrated with the image features of the target image is obtained to obtain a second fused feature; The first fusion feature is fused with the universal image feature to obtain a third fusion feature, and the second fusion feature is fused with the universal text feature to obtain a fourth fusion feature; The target feature corresponding to the target image is acquired according to the third fusion feature, and the target feature corresponding to the target text is acquired according to the fourth fusion feature.

4. The image classification method according to claim 3, characterized in that: The method of obtaining, based on the general image features and the general text features, a specific domain image feature integrated with the text features of the target text to obtain a first fused feature, and obtaining a specific domain text feature integrated with the image features of the target image to obtain a second fused feature includes: Integrate the universal text feature into the universal image feature to obtain a first feature, and integrate the universal image feature into the universal text feature to obtain a second feature; Performing image feature extraction on the first feature to obtain a third feature, and performing text feature extraction on the second feature to obtain a fourth feature; Performing image feature extraction on the third feature to obtain a fifth feature, and performing text feature extraction on the fourth feature to obtain a sixth feature; The fourth feature is integrated into the fifth feature to obtain a first fused feature, and the third feature is integrated into the sixth feature to obtain a second fused feature.

5. The image classification method according to claim 3, characterized in that: The step of acquiring a target feature corresponding to the target image according to the third fusion feature, and acquiring a target feature corresponding to the target text according to the fourth fusion feature, includes: Processing the third fused feature into a feature suitable for an image classification task to obtain a fifth fused feature, and processing the fourth fused feature into a feature suitable for an image classification task to obtain a sixth fused feature; The fifth fusion feature is fused with the general image feature to obtain a target feature corresponding to the target image, and the sixth fusion feature is fused with the general text feature to obtain a target feature corresponding to the target text.

6. The image classification method according to claim 1, characterized in that: The image classification model includes an image encoder, a text encoder and an adapter, wherein the image encoder is pre-trained using image samples in a general field, and the text encoder is pre-trained using text samples in a general field; The training process of the image classification model includes: Acquire common image features of the training image based on the image encoder, and acquire common text features of visual description text of the training image based on the text encoder; Based on the adapter, based on the general image features of the training image and the general text features of the visual description text of the training image, target features corresponding to the training image and the visual description text of the training image are obtained respectively, wherein the target features corresponding to the training image are training image features integrated with text features of the visual description text of the training image, the training image features include general image features and domain-specific image features of the training image, and the target features corresponding to the visual description text of the training image are training text features integrated with image features of the training image, and the training text features include general text features and domain-specific text features of the visual description text of the training image; A first loss is determined according to target features corresponding to the training image and the visual description text of the training image, and parameters of the image classification model are updated according to the first loss. During the training process, parameters of the image encoder and the text encoder are not updated.

7. An image classification device, characterized in that: include: Data acquisition module, general feature acquisition module, target feature acquisition module and image category determination module; The data acquisition module is used to acquire a target image and a target text in a specific field, wherein the target text is a visual description text of the target image; The general feature acquisition module is used to extract general image features from the target image and general text features from the target text; The target feature acquisition module is used to acquire the target features corresponding to the target image and the target text respectively according to the general image features and the general text features, wherein the target features corresponding to the target image are target image features integrated with text features of the target text, and the target image features include general image features and specific domain image features of the target image, and the target features corresponding to the target text are target text features integrated with image features of the target image, and the target text features include general text features and specific domain text features of the target text; The image category determination module is used to determine the category of the target image according to the target features corresponding to the target image and the target text respectively; The general feature acquisition module is specifically used to extract general image features from the target image and general text features from the target text: Based on a pre-trained image classification model, extracting common image features from the target image and extracting common text features from the target text; The target feature acquisition module acquires the target features corresponding to the target image and the target text respectively according to the general image features and the general text features, and is specifically used for: Based on the image classification model, and taking the general image features and the general text features as a basis, obtaining target features corresponding to the target image and the target text respectively; The image classification model is trained using training images of the specific field and visual description texts of the training images.

8. An image classification device, characterized in that: include: Memory and processor; The memory is used to store programs; The processor is used to execute the program to implement each step of the image classification method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, each step of the image classification method according to any one of claims 1 to 6 is implemented.

Images