An image classification method, apparatus, device and medium

Abstract

Embodiments of the present disclosure relate to an image classification method, device, equipment and medium, wherein the method comprises: obtaining a plurality of categories and a target image; determining corresponding category description information based on prompt words of each category and a knowledge distillation model; determining a plurality of category text features of the plurality of categories based on the plurality of category description information; extracting an image semantic feature of the target image; fusing the plurality of category text features and the image semantic feature to obtain a target feature; and performing image classification of the target image in the plurality of categories based on the target feature. Since the category description information is automatically determined by the model, the information comprehensiveness and accuracy of the classification information are effectively improved, and the efficiency is relatively high, thereby effectively improving the accuracy and efficiency of image classification.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image classification method, apparatus, device and medium. Background Technology

[0002] Image classification is a common visual algorithm task, applicable to categories such as source material, video cover images, and special effects. However, issues such as the amount of labeled data, task difficulty, and data quality often lead to unsatisfactory results. While adding category information can improve accuracy, this information is typically determined manually, which has limitations, resulting in lower accuracy and comprehensiveness, higher costs, and lower efficiency, thus requiring improvement. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides an image classification method, apparatus, device, and medium.

[0004] This disclosure provides an image classification method, the method comprising:

[0005] Acquire multiple categories and target images;

[0006] Based on the category-specific prompts and the knowledge distillation model, the corresponding category description information is determined;

[0007] Multiple category text features are determined based on multiple category description information;

[0008] Extract the image semantic features of the target image;

[0009] The target feature is obtained by fusing the multiple categories of text features with the image semantic features;

[0010] The target image is classified into multiple categories based on the target features.

[0011] This disclosure also provides an image classification apparatus, the apparatus comprising:

[0012] The acquisition module is used to acquire multiple categories and target images;

[0013] The information module is used to determine the corresponding category description information based on the prompts for each category and the knowledge distillation model;

[0014] The first feature module is used to determine multiple category text features of the multiple categories based on multiple category description information;

[0015] The second feature module is used to extract the image semantic features of the target image;

[0016] The fusion module is used to fuse the multiple categories of text features with the image semantic features to obtain the target features;

[0017] A classification module is used to classify the target image into multiple categories based on the target features.

[0018] This disclosure also provides an electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the image classification method provided in this disclosure.

[0019] This disclosure also provides a computer-readable storage medium storing a computer program for performing the image classification method provided in this disclosure.

[0020] Compared with the prior art, the technical solution provided in this disclosure has the following advantages: The image classification scheme provided in this disclosure acquires multiple categories and target images; determines corresponding category description information based on prompt words for each category and a knowledge distillation model; determines multiple category text features for multiple categories based on multiple category description information; extracts image semantic features of the target image; fuses multiple category text features with image semantic features to obtain target features; and performs image classification of the target image for multiple categories based on the target features. Using the above technical solution, category description information can be extracted through a knowledge distillation model and prompt words for each category, and the features corresponding to the category description information are fused with image semantic features to obtain feature-enhanced target features. Image classification of the target image is then achieved based on these target features. Since the category description information is automatically determined by the model, the comprehensiveness and accuracy of the classification information are effectively improved, and the efficiency is high, thereby effectively improving the accuracy and efficiency of image classification. Attached Figure Description

[0021] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0022] Figure 1 A schematic flowchart of an image classification method provided in an embodiment of this disclosure;

[0023] Figure 2 A schematic flowchart illustrating another image classification method provided in this embodiment of the present disclosure;

[0024] Figure 3A schematic diagram of an image classification process provided in an embodiment of this disclosure;

[0025] Figure 4 This is a schematic diagram of the structure of an image classification device provided in an embodiment of the present disclosure;

[0026] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0027] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0028] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0029] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0030] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0031] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0032] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0033] To address the issues of low accuracy and efficiency in image classification in related technologies, this disclosure provides an image classification method, which will be described below with reference to specific embodiments.

[0034] Figure 1 This is a flowchart illustrating an image classification method provided in an embodiment of the present disclosure. The method can be executed by an image classification device, which can be implemented in software and / or hardware, and is generally integrated into an electronic device. Figure 1 As shown, the method includes:

[0035] Step 101: Obtain multiple categories and target images.

[0036] The category can be a predefined set of labels set for the image. Image classification requires assigning the input image to different categories. The number of categories can be multiple, and the definition of the categories can be determined according to specific needs and application scenarios. For example, categories can include different sports, different kinds of plants and animals, etc. In this embodiment of the disclosure, multiple categories can be pre-determined when performing image classification to meet actual scenario needs or business requirements. The target image can be the input image that needs to be classified. The number and source of target images are not limited. For example, the target image can be an image captured in real time or an image stored locally.

[0037] Specifically, the image classification device can acquire multiple categories for the current image classification task and the target image that needs to be classified, for later use.

[0038] Step 102: Determine the corresponding category description information based on the prompt words for each category and the knowledge distillation model.

[0039] Prompts can be contextual information generated by accurately translating user needs into a form that the model can effectively understand and process. In this embodiment, category-specific prompts can be textual information set for a category that comprehensively and accurately extracts or distills its features and information. It can be an integrated prompt project, where a category's prompts can include multiple prompt statements. Each prompt statement is set from one dimension or level of that category, and the specific number of prompt statements can be set according to actual conditions. Multiple prompt statements are used to query the knowledge distillation model for category feature information. The knowledge distillation model can be a model used to distill or refine descriptive or feature information of different categories. This knowledge distillation model can be pre-trained on large-scale data, and can be obtained through a series of sub-tasks on massive amounts of data (text, images, videos, etc.). It has powerful content understanding capabilities and can be regarded as a parameterized knowledge base that can include knowledge from various fields, effectively determining category-related information. Category description information can be information that introduces or describes a category, reflecting its features and various information.

[0040] In some embodiments, determining the corresponding category description information based on the prompt words of each category and the knowledge distillation model may include: sequentially inputting multiple prompt word statements included in each category into the knowledge distillation model, and outputting multiple output texts corresponding to the multiple prompt word statements; and combining the multiple output texts of each category to determine the corresponding category description information.

[0041] The output text can be a prompt word input into the knowledge distillation model. For each category, the image classification device can obtain prompt words for that category, input multiple prompt words into the knowledge distillation model to obtain multiple output texts, and combine these multiple output texts to determine the category description information; that is, a category description information can include multiple output texts.

[0042] For example, assuming the category is badminton, the prompts could include three prompts: the first prompt could be "You are an expert in the field of sports, please describe the sport of badminton," the second prompt could be "What are the distinctive features of badminton compared to other ball sports?" and the third prompt could be "How can you identify an ongoing sport as badminton?"

[0043] In the above scheme, category description information is extracted by setting prompt words that include multiple prompt word statements, and integrated prompt word engineering processing is achieved. The knowledge distillation model can be used to distill category information comprehensively and accurately, thereby improving the comprehensiveness and accuracy of category description information determination.

[0044] Step 103: Determine multiple category text features based on multiple category description information.

[0045] Among them, the category text features can be the feature vectors corresponding to the category description information, and each category description information determines the corresponding category text features.

[0046] In some embodiments, determining multiple category text features based on multiple category description information may include: extracting features from the multiple category description information using a pre-trained language model, and determining the extracted multiple text feature vectors as multiple category text features. The specific type of the pre-trained language model is not limited and can be set according to actual conditions. For example, the pre-trained language model could be an encoder based on a converter model, which can process the entire sequence in parallel and use an attention mechanism to collect information about word-related contexts, thereby simultaneously processing words related to all other words in the sentence. An image classification device can input each category description information into the pre-trained language model for feature extraction, and determine the output text feature vector as the category text feature corresponding to that category.

[0047] Step 104: Extract the semantic features of the target image.

[0048] Image semantic features can be features that represent the meaning of image content. These features capture the semantic information in the image, that is, the meaning or significance of the data, rather than just its surface form. Depending on the processing level, image semantic features can be divided into low-level features, mid-level features, and high-level features. Low-level features can include color, texture, shape, edge, and contour, while mid-level features can include objects, parts, or attributes in the image. High-level features can include what the image expresses that is closest to human understanding, such as scenes and interactions.

[0049] In some embodiments, extracting the image semantic features of the target image may include: inputting the target image into a visual transformation model, and determining the multi-level visual features output by multiple network layers of the visual transformation model as the image semantic features of the target image.

[0050] The visual transformation model can be a deep learning model based on a transformer architecture, used to extract semantic features from an image. The visual transformation model can segment an image into a series of small image patches and linearly map these patches into a high-dimensional space, forming a sequence input. Then, the transformer's self-attention mechanism is used to process these image patches, thereby capturing the contextual dependencies at different locations in the image. In this embodiment, the visual transformation model may include multiple network layers. Different network layers output features at different processing levels, forming multi-level visual features. These multi-level visual features may include low-level features, mid-level features, and high-level features. Specifically, for a target image, the image classification device can input the target image into the visual transformation model and output the corresponding multi-level visual features, which are then determined as image semantic features.

[0051] Step 105: Fuse multiple categories of text features with image semantic features to obtain target features.

[0052] Among them, the target feature is obtained by semantic enhancement of the image semantic features of the target image through the category description information of multiple categories. The target feature can be obtained by fusing the multiple category text features corresponding to multiple categories with the image semantic features of the target image. This belongs to the enhanced multi-level visual features.

[0053] After determining the text features of multiple categories corresponding to multiple categories and the semantic features of the target image, the image classification device can use a preset fusion method to fuse the text features of multiple categories and the semantic features of the image. The preset fusion method can be, for example, similarity weighted fusion, additive fusion, etc., and can be set according to the actual situation.

[0054] For example, Figure 2This is a flowchart illustrating another image classification method provided in an embodiment of the present disclosure. In one feasible implementation, fusing multiple categories of text features with image semantic features to obtain target features may include:

[0055] Step 201: Determine the similarity between the image semantic features and multiple categories of text features.

[0056] Here, similarity can be the degree of similarity between image semantic features determined through semantic calculation and a category of text features. Image classification devices can determine the similarity between image semantic features and each category of text features. The specific calculation method is not limited; for example, it can be calculated using semantic association algorithms to obtain multiple similarities.

[0057] Step 202: Determine multiple fusion weights for text features of multiple categories based on multiple similarities.

[0058] Fusion weights are specific weights assigned to text features of a category when fusion calculations are performed with image semantic features. Fusion weights are directly proportional to similarity; the greater the similarity, the greater the fusion weight. After determining multiple similarities between image semantic features and multiple categories of text features, the image classification device uses a pre-defined mapping relationship between similarity and fusion weights to determine the fusion weight corresponding to each similarity.

[0059] Step 203: Based on multiple fusion weights, the text features of multiple categories and the semantic features of the image are weighted and fused to obtain the target features.

[0060] After determining multiple fusion weights for multiple categories of text features, an image classification device can use a weighted fusion method to fuse multiple categories of text features with image semantic features according to multiple fusion weights to obtain target features. For example, weighted fusion methods can include weighted average method, neural network method, etc. The weighted average method calculates a weighted average based on multiple fusion weights.

[0061] In the above scheme, corresponding fusion weights are assigned to the text features of the categories based on similarity, and then the text features of multiple categories are integrated into the image semantic features based on the fusion weights to obtain enhanced multi-level visual features, which effectively improves the accuracy and representativeness of feature fusion.

[0062] Step 106: Classify the target image into multiple categories based on target features.

[0063] After determining the target features, the image classification device can input the target features into the image classification network layer to classify the image and output the target category corresponding to the target image in multiple categories.

[0064] In some embodiments, classifying a target image into multiple categories based on target features may include: splitting and stitching the target features to obtain one-dimensional features; inputting the one-dimensional features into an image classification network layer, and outputting the target category of the target image in multiple categories.

[0065] When classifying a target image based on its features, an image classification device can first split the feature matrix of the target features into multiple rows and then concatenate these rows to obtain the corresponding one-dimensional features. These one-dimensional features are then input into the image classification network layer to obtain the target category corresponding to the target image. The image classification network layer can be a fully connected layer (FC), and the target category can be the category to which the final determined target image belongs among several known categories.

[0066] The image classification scheme provided in this disclosure acquires multiple categories and a target image; determines corresponding category description information based on prompt words for each category and a knowledge distillation model; determines multiple category text features for multiple categories based on the multiple category description information; extracts image semantic features of the target image; fuses the multiple category text features with the image semantic features to obtain target features; and performs image classification of the target image for multiple categories based on the target features. Using the above technical solution, category description information can be extracted through a knowledge distillation model and prompt words for each category, and the features corresponding to the category description information are fused with image semantic features to obtain feature-enhanced target features. Image classification is then performed on the target image based on these target features. Since the category description information is automatically determined by the model, the comprehensiveness and accuracy of the classification information are effectively improved, and the efficiency is high, thereby effectively improving the accuracy and efficiency of image classification.

[0067] The image classification method of this disclosure will be further illustrated by a specific example below. For example, Figure 3 This is a schematic diagram of an image classification process provided in an embodiment of the present disclosure, such as... Figure 3 As shown in the figure, taking the high jump image as an example, which includes multiple sports and multiple categories, the specific image classification process includes: inputting the target image into the visual transformation model to obtain image semantic features, which are multi-level visual features; using multiple category prompts through a knowledge distillation model to obtain multiple category description information, which is then input into a pre-trained language model to obtain multiple category text features; fusing the image semantic features and the multiple category text features through a fusion method to obtain enhanced image semantic features, which are the aforementioned target features; and inputting the target features into the image classification network layer to obtain the target category, which is the high jump sport.

[0068] When multiple categories include multiple sports, the knowledge distillation model obtains category descriptions of multiple sports. Obviously, this background and content knowledge about sports is helpful for sports recognition tasks. This knowledge is output in the form of text descriptions, ultimately obtaining a text information database about sports. Since the image classification task at this time is to classify different sports, this specific description of sports will help the model obtain more information about sports. This information is helpful for image classification; the more detailed the information, the more accurate the classification. That is, this information can be used to enhance the semantic features of the image.

[0069] The proposed image classification method based on knowledge distillation consists of three modules: 1. A knowledge distillation module, which uses cue word engineering to distill textual descriptions of image categories to construct a textual information database for image categories; 2. An image feature enhancement module, which uses knowledge from the textual information database to enhance the semantic features of images; and 3. An image recognition module, which performs image classification based on the enhanced semantic features of images.

[0070] This proposal suggests an image classification method based on knowledge distillation. It distills information relevant to image classification from a large knowledge base (the knowledge distillation model) to construct an "information base" for the image classification task. This information base contains knowledge about specific image classifications learned from massive datasets (e.g., in sports classification, the information base contains a series of textual descriptions of various sports). This knowledge is highly effective for classifying specific images, thereby improving the model's image recognition ability. It's worth noting that image classification methods are trained by collecting large amounts of labeled data. The algorithm's performance largely depends on the quantity and quality of the data. Essentially, it involves collecting more category feature information to help the model learn category features, thus improving classification accuracy. Since the knowledge distillation model is trained on massive amounts of knowledge, extracting category information from it and integrating it into the image classification model can replace a large amount of image data to achieve better classification results, achieving higher accuracy image classification with limited labeled data.

[0071] Figure 4 This is a schematic diagram of the structure of an image classification device provided in an embodiment of this disclosure. The device can be implemented by software and / or hardware, and is generally integrated into an electronic device. Figure 4 As shown, the device includes:

[0072] The acquisition module 401 is used to acquire multiple categories and target images;

[0073] Information module 402 is used to determine the corresponding category description information based on the prompt words of each category and the knowledge distillation model;

[0074] The first feature module 403 is used to determine multiple category text features of the multiple categories based on multiple category description information;

[0075] The second feature module 404 is used to extract the image semantic features of the target image;

[0076] The fusion module 405 is used to fuse the multiple categories of text features with the image semantic features to obtain target features;

[0077] The classification module 406 is used to classify the target image into multiple categories based on the target features.

[0078] Optionally, the information module 402 is used for:

[0079] The knowledge distillation model sequentially inputs multiple prompt words from each category into the knowledge distillation model, and outputs multiple output texts corresponding to the multiple prompt words.

[0080] Multiple output texts for each category are combined to determine the corresponding category description information.

[0081] Optionally, the first feature module 403 is used for:

[0082] The multiple category description information is extracted using a pre-trained language model, and the extracted text feature vectors are determined as multiple category text features.

[0083] Optionally, the second feature module 404 is used for:

[0084] The target image is input into a visual transformation model, and the multi-level visual features output by multiple network layers of the visual transformation model are determined as the image semantic features of the target image.

[0085] Optionally, the fusion module 405 is used for:

[0086] Determine multiple similarities between the image semantic features and the multiple categories of text features;

[0087] Multiple fusion weights for the multiple categories of text features are determined based on the multiple similarities;

[0088] The target feature is obtained by weighting and fusing the multiple categories of text features with the image semantic features based on the multiple fusion weights.

[0089] Optionally, the classification module 406 is used for:

[0090] The target features are split and concatenated to obtain one-dimensional features;

[0091] The one-dimensional feature is input into the image classification network layer, and the output is the target category of the target image in the multiple categories.

[0092] Optionally, the target feature is obtained by semantically enhancing the image semantic features of the target image with the category description information of the multiple categories.

[0093] The image classification apparatus provided in this disclosure can execute the image classification method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of executing the method.

[0094] This disclosure also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the image classification method provided in any embodiment of this disclosure.

[0095] Figure 5 This is a schematic diagram of an electronic device provided in an embodiment of the present disclosure. See below for details. Figure 5 The diagram illustrates a structural schematic suitable for implementing the electronic device 500 in the embodiments of this disclosure. The electronic device 500 in the embodiments of this disclosure may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0096] like Figure 5 As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0097] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0098] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the image classification method of embodiments of this disclosure.

[0099] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0100] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0101] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0102] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: acquire multiple categories and a target image; determine corresponding category description information based on prompt words for each category and a knowledge distillation model; determine multiple category text features for the multiple categories based on the multiple category description information; extract image semantic features of the target image; fuse the multiple category text features with the image semantic features to obtain target features; and perform image classification of the target image for the multiple categories based on the target features.

[0103] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0105] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0106] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0107] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0108] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0109] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0110] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0111] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An image classification method, characterized in that, include: Acquire multiple categories and target images; Based on the category-specific prompts and the knowledge distillation model, the corresponding category description information is determined; Multiple category text features are determined based on multiple category description information; Extract the image semantic features of the target image; The target feature is obtained by fusing the multiple categories of text features with the image semantic features; The target image is classified into multiple categories based on the target features.

2. The method according to claim 1, characterized in that, The determination of corresponding category description information based on category-specific prompt words and a knowledge distillation model includes: The knowledge distillation model sequentially inputs multiple prompt words from each category into the knowledge distillation model, and outputs multiple output texts corresponding to the multiple prompt words. Multiple output texts for each category are combined to determine the corresponding category description information.

3. The method according to claim 1, characterized in that, The determination of multiple category text features based on multiple category description information includes: The multiple category description information is extracted using a pre-trained language model, and the extracted text feature vectors are determined as multiple category text features.

4. The method according to claim 1, characterized in that, The extraction of image semantic features from the target image includes: The target image is input into a visual transformation model, and the multi-level visual features output by multiple network layers of the visual transformation model are determined as the image semantic features of the target image.

5. The method according to claim 1, characterized in that, The target features are obtained by fusing the multiple categories of text features with the image semantic features, including: Determine multiple similarities between the image semantic features and the multiple categories of text features; Multiple fusion weights for the multiple categories of text features are determined based on the multiple similarities; The target feature is obtained by weighting and fusing the multiple categories of text features with the image semantic features based on the multiple fusion weights.

6. The method according to claim 1, characterized in that, Based on the target features, the target image is classified into multiple categories, including: The target features are split and concatenated to obtain one-dimensional features; The one-dimensional feature is input into the image classification network layer, and the output is the target category of the target image in the multiple categories.

7. The method according to any one of claims 1-6, characterized in that, The target feature is obtained by semantically enhancing the image semantic features of the target image with the category description information of the multiple categories.

8. An image classification device, characterized in that, include: The acquisition module is used to acquire multiple categories and target images; The information module is used to determine the corresponding category description information based on the prompts for each category and the knowledge distillation model; The first feature module is used to determine multiple category text features of the multiple categories based on multiple category description information; The second feature module is used to extract the image semantic features of the target image; The fusion module is used to fuse the multiple categories of text features with the image semantic features to obtain the target features; A classification module is used to classify the target image into multiple categories based on the target features.

9. An electronic device, characterized in that, The electronic device includes: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the image classification method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the image classification method according to any one of claims 1-7.

Images